Get accuracy from quantiles — accuracy • casteval

Given a forecast and set of observations, compute the accuracy (# inside quantile interval / # total) of the forecast. Raw data and/or provided quantiles will be used to compute the quantile interval.

Usage

accuracy(fcst, obs, summarize = TRUE, quant_pairs = NULL)

Arguments

fcst

A forecast object (see output of create_forecast()).

obs

An observations data frame.

summarize

A boolean, defaults to TRUE. If TRUE, a single number will be returned as the score for the forecast. If FALSE, a data frame with columns named time, val_obs, and score will be returned, containing the scores for each individual time point. This can be used by plotting functions to colour-code observations, for example.

quant_pairs

(Optional) A list of pairs of numbers between 0 and 100, or a single pair of numbers between 0 and 100. If provided, the score for each corresponding pairs of quantiles will be calculated. If not provided, it will default to every symmetrical pair of quantiles that can be found in fcst, ordered from widest to narrowest (e.x. the 25% and 75% quantiles are symmetrical).

If summarize is FALSE, an additional column named pair will indicate which pair of quantiles each row represents. If summarize is TRUE, the output will be a vector with the same length as quant_pairs, containing the respective score for each pair.

Value

A number from 0 to 1, the rate at which the observations were inside the specified quantile interval

Examples

# forecast with raw data
fc1 <- create_forecast(dplyr::tibble(time=c(1,1,1,2,2,2,3,3,3), val=4:12))
obs1 <- data.frame(time=1:3, val_obs=c(5, 7.4, 11.6))

# calculate quantiles and accuracy from raw data
# returns 2/3
accuracy(fc1, obs1, quant_pairs=list(c(25, 75)))
#> Scoring accuracy using quantile pairs c(25, 75)
#> Used 3 time points to calculate accuracy
#> [1] 0.3333333

# forecast with quantile data
fc2 <- create_forecast(dplyr::tibble(
  time=1:3, val_q5=1:3, val_q25=4:6, val_q50=100:102, val_q75=200:202, val_q95=203:205
))
obs2 <- data.frame(time=1:3, val_obs=c(4, 202, 1000))

# infer quantile pairs from forecast data (`c(5,95)` and `c(25, 75)`)
# returns c(2/3, 1/3)
accuracy(fc2, obs2)
#> Scoring accuracy using quantile pairs c(5, 95), c(25, 75)
#> Used 3 time points to calculate accuracy
#> [1] 0.6666667 0.3333333

# return a data frame with a `time`, `pair`, `val_obs`, and `score` columns
accuracy(fc2, obs2, summarize=FALSE)
#> Scoring accuracy using quantile pairs c(5, 95), c(25, 75)
#>   time val_obs score pair
#> 1    1       4  TRUE    1
#> 2    2     202  TRUE    1
#> 3    3    1000 FALSE    1
#> 4    1       4  TRUE    2
#> 5    2     202 FALSE    2
#> 6    3    1000 FALSE    2