threshold_perf() can take a set of class probability predictions and determine performance characteristics across different values of the probability threshold and any existing groups.

threshold_perf(.data, ...)

# S3 method for data.frame
threshold_perf(.data, truth, estimate, thresholds = NULL, na_rm = TRUE, ...)

Arguments

.data

A tibble, potentially grouped.

...

Currently unused.

truth

The column identifier for the true two-class results (that is a factor). This should be an unquoted column name.

estimate

The column identifier for the predicted class probabilities (that is a numeric). This should be an unquoted column name.

thresholds

A numeric vector of values for the probability threshold. If unspecified, a series of values between 0.5 and 1.0 are used. Note: if this argument is used, it must be named.

na_rm

A single logical: should missing data be removed?

Value

A tibble with columns: .threshold, .estimator, .metric, .estimate and any existing groups.

Details

Note that that the global option yardstick.event_first will be used to determine which level is the event of interest. For more details, see the Relevant level section of yardstick::sens().

The currently calculated metrics are:

Examples

library(dplyr) data("segment_logistic") # Set the threshold to 0.6 # > 0.6 = good # < 0.6 = poor threshold_perf(segment_logistic, Class, .pred_good, thresholds = 0.6)
#> # A tibble: 4 x 4 #> .threshold .metric .estimator .estimate #> <dbl> <chr> <chr> <dbl> #> 1 0.6 sens binary 0.639 #> 2 0.6 spec binary 0.869 #> 3 0.6 j_index binary 0.508 #> 4 0.6 distance binary 0.148
# Set the threshold to multiple values thresholds <- seq(0.5, 0.9, by = 0.1) segment_logistic %>% threshold_perf(Class, .pred_good, thresholds)
#> # A tibble: 20 x 4 #> .threshold .metric .estimator .estimate #> <dbl> <chr> <chr> <dbl> #> 1 0.5 sens binary 0.714 #> 2 0.6 sens binary 0.639 #> 3 0.7 sens binary 0.561 #> 4 0.8 sens binary 0.451 #> 5 0.9 sens binary 0.249 #> 6 0.5 spec binary 0.825 #> 7 0.6 spec binary 0.869 #> 8 0.7 spec binary 0.911 #> 9 0.8 spec binary 0.937 #> 10 0.9 spec binary 0.977 #> 11 0.5 j_index binary 0.539 #> 12 0.6 j_index binary 0.508 #> 13 0.7 j_index binary 0.472 #> 14 0.8 j_index binary 0.388 #> 15 0.9 j_index binary 0.226 #> 16 0.5 distance binary 0.112 #> 17 0.6 distance binary 0.148 #> 18 0.7 distance binary 0.201 #> 19 0.8 distance binary 0.306 #> 20 0.9 distance binary 0.565
# --------------------------------------------------------------------------- # It works with grouped data frames as well # Let's mock some resampled data resamples <- 5 mock_resamples <- resamples %>% replicate( expr = sample_n(segment_logistic, 100, replace = TRUE), simplify = FALSE ) %>% bind_rows(.id = "resample") resampled_threshold_perf <- mock_resamples %>% group_by(resample) %>% threshold_perf(Class, .pred_good, thresholds) resampled_threshold_perf
#> # A tibble: 100 x 5 #> resample .threshold .metric .estimator .estimate #> <chr> <dbl> <chr> <chr> <dbl> #> 1 1 0.5 sens binary 0.667 #> 2 1 0.6 sens binary 0.593 #> 3 1 0.7 sens binary 0.444 #> 4 1 0.8 sens binary 0.333 #> 5 1 0.9 sens binary 0.185 #> 6 2 0.5 sens binary 0.821 #> 7 2 0.6 sens binary 0.75 #> 8 2 0.7 sens binary 0.679 #> 9 2 0.8 sens binary 0.5 #> 10 2 0.9 sens binary 0.214 #> # … with 90 more rows
# Average over the resamples resampled_threshold_perf %>% group_by(.metric, .threshold) %>% summarise(.estimate = mean(.estimate))
#> `summarise()` regrouping output by '.metric' (override with `.groups` argument)
#> # A tibble: 20 x 3 #> # Groups: .metric [4] #> .metric .threshold .estimate #> <chr> <dbl> <dbl> #> 1 distance 0.5 0.117 #> 2 distance 0.6 0.139 #> 3 distance 0.7 0.194 #> 4 distance 0.8 0.292 #> 5 distance 0.9 0.538 #> 6 j_index 0.5 0.541 #> 7 j_index 0.6 0.529 #> 8 j_index 0.7 0.499 #> 9 j_index 0.8 0.415 #> 10 j_index 0.9 0.254 #> 11 sens 0.5 0.723 #> 12 sens 0.6 0.656 #> 13 sens 0.7 0.575 #> 14 sens 0.8 0.468 #> 15 sens 0.9 0.270 #> 16 spec 0.5 0.819 #> 17 spec 0.6 0.874 #> 18 spec 0.7 0.924 #> 19 spec 0.8 0.947 #> 20 spec 0.9 0.985