Calibration of Binary and Multiclass Probabilities

Provides S3 calibrators, metrics, and diagnostics for binary and multiclass probability calibration. Binary methods include Platt scaling, temperature scaling, beta calibration, histogram binning, and isotonic regression. Multiclass methods include temperature scaling, vector scaling, Dirichlet calibration, and a one-vs-rest wrapper for the binary calibrators. A calibration-inference layer adds debiased calibration errors, bootstrap confidence intervals, and a kernel calibration hypothesis test for binary and multiclass predictions, including the strong (canonical) multiclass case. Methods follow Platt (1999, ISBN:9780262194488), Zadrozny and Elkan (2002) , Guo et al. (2017) < https://proceedings.mlr.press/v70/guo17a.html>, Kull et al. (2017) , Kull et al. (2019) , Widmann et al. (2019) , and Kumar et al. (2019) .

probcal

probcal calibrates predicted class probabilities after a classifier has already been fitted. Many classifiers rank observations well but assign probabilities that are too high, too low, or distorted in different parts of the range. If observations predicted near 0.80 do not occur about 80 percent of the time, the scores may still be useful for ranking, but they are unreliable for decisions that use the probability itself.

The package is designed for workflows where model fitting and probability calibration are separate steps. Fit a binary or multiclass classifier with any R modeling tool, reserve calibration data, and pass the resulting scores, logits, or probability matrices to probcal. Fitted calibrators are S3 objects: inspect them with print() or summary() and apply them to new predictions with predict().

Calibration is useful when predictions feed risk thresholds, triage rules, forecasts, or decision analyses with unequal costs. probcal provides post-hoc calibration methods, calibration metrics, reliability diagrams, and out-of-fold calibration for settings where a separate calibration split is small. Beyond these point summaries, it adds a calibration-inference layer: debiased calibration-error estimators, bootstrap confidence intervals, and a kernel calibration test that returns a p-value, so “is this model calibrated?” can be answered with a test instead of a single number.

What It Does

• Fits post-hoc calibrators for binary and multiclass probabilities.
• Works with scores, logits, probability vectors, and probability matrices.
• Uses an S3 interface: fit with cal_*(), inspect with print() or summary(), and apply with predict().
• Includes parametric methods, nonparametric methods, calibration metrics, reliability diagrams, and cross-validated calibration.
• Tests calibration: cal_test() implements the kernel calibration tests of Widmann et al. (2019) for binary and multiclass predictions, including the strong (canonical) case in which the full probability vector is assessed.
• Quantifies uncertainty: skce() provides unbiased kernel calibration-error estimators, cal_ci() bootstrap confidence intervals, and ece(debiased = TRUE) the debiased estimator of Kumar et al. (2019).
• Runs as a native R package with no Python or Julia runtime. To our knowledge, probcal is the first package to combine binary and multiclass recalibration with this calibration inference in native R.

Installation

From CRAN:

install.packages("probcal")

Development version from GitHub:

# install.packages("remotes")
remotes::install_github("prdm0/probcal")

Binary Workflow

This example simulates a classifier that is too confident. Calibration is fitted on a calibration split and evaluated on a test split.

library(probcal)
library(dplyr)

set.seed(42)
n <- 600
predictions <- data.frame(x = rnorm(n)) |>
  mutate(
    true_p = inv_logit(-0.4 + 1.1 * x),
    y = rbinom(n(), 1, true_p),
    raw_logits = 1.8 * (-0.4 + 1.1 * x),
    raw_p = inv_logit(raw_logits),
    split = sample(rep(c("calibration", "test"), each = n / 2))
  )

calibration <- predictions |>
  filter(split == "calibration")

test <- predictions |>
  filter(split == "test")

fit <- cal_beta(calibration$raw_p, calibration$y)

test <- test |>
  mutate(calibrated = predict(fit, raw_p))

test |>
  summarise(
    raw_ece = ece(raw_p, y, bins = 10),
    calibrated_ece = ece(calibrated, y, bins = 10)
  )

reliability_diagram(test$calibrated, test$y, bins = 10)

Is the Model Calibrated? Test It

The metrics above are point summaries. The calibration-inference layer turns the question into a hypothesis test and an interval. The raw overconfident probabilities are rejected; the beta-calibrated ones are not.

set.seed(1)
cal_test(test$raw_p, test$y)$p.value
cal_test(test$calibrated, test$y)$p.value

cal_ci(test$calibrated, test$y, metric = "ece", bins = 10)

cal_test() also assesses strong (canonical) calibration of a multiclass probability matrix, and skce() exposes the unbiased kernel calibration-error estimators behind the test.

Multiclass Workflow

For multiclass calibration, pass a matrix with one column per class. Temperature and vector scaling use logits. Dirichlet calibration and one-vs-rest calibration use probability matrices.

set.seed(2024)
n <- 600
k <- 3

true_prob <- matrix(runif(n * k), ncol = k)
true_prob <- true_prob / rowSums(true_prob)
labels <- apply(true_prob, 1, function(row) sample.int(k, 1, prob = row))

raw_prob <- true_prob^2
raw_prob <- raw_prob / rowSums(raw_prob)

split <- sample(rep(c("calibration", "test"), each = n / 2))

fit <- cal_dirichlet(
  raw_prob[split == "calibration", ],
  labels[split == "calibration"]
)

calibrated <- predict(fit, raw_prob[split == "test", ])

ece(raw_prob[split == "test", ], labels[split == "test"], type = "classwise")
ece(calibrated, labels[split == "test"], type = "classwise")

Included Methods

Binary calibrators:

• cal_platt(): Platt scaling with Platt target correction.
• cal_temperature(): temperature scaling for logits.
• cal_beta(): beta calibration.
• cal_isotonic(): isotonic regression.
• cal_histogram(): histogram binning with equal-width or equal-frequency bins.

Multiclass calibrators:

• cal_temperature(): multiclass temperature scaling on a logit matrix.
• cal_vector_scaling(): per-class scale and bias on a logit matrix.
• cal_dirichlet(): Dirichlet calibration with an optional ODIR penalty.
• cal_ovr(): one-vs-rest wrapper that applies a binary calibrator per class.

Metrics and diagnostics:

• ece(): Expected Calibration Error, with debiased, strategy (equal-width or equal-mass bins), and norm ("l1" or "l2") options.
• mce(): Maximum Calibration Error.
• ace(): Average Calibration Error.
• mmce(): Maximum Mean Calibration Error.
• reliability_diagram(): reliability diagram returned as a ggplot object.

Calibration inference:

• skce(): squared kernel calibration error of Widmann et al. (2019), with unbiased quadratic ("uq"), unbiased linear-time ("ul"), and biased estimators, for binary, confidence, classwise, and strong (canonical) multiclass targets.
• cal_test(): kernel calibration hypothesis test (wild bootstrap by default, asymptotic normal as a fast alternative), returning an object of class c("cal_test", "htest") with a p-value.
• cal_ci(): percentile bootstrap confidence interval for ece(), mce(), ace(), mmce(), or skce().

For small calibration sets, cal_cv() provides out-of-fold calibration. It works on supplied scores, logits, or probabilities and does not train the underlying classifier.

Choosing a Calibrator

See vignettes/choosing-a-calibrator.Rmd and vignettes/multiclass.Rmd for longer examples.

Numerical Validation

The test suite contains optional checks against external implementations. These checks are used during development and are skipped when the optional software is not available. They do not make Python or any external calibration package a runtime dependency.

Scope

probcal covers post-hoc probability calibration for binary and multiclass classification. Bayesian binning, near-isotonic ensembles, object detection calibration, regression uncertainty calibration, and neural calibration methods are not part of the current API.

Citation

Use citation("probcal") to cite the package. The citation metadata includes the package author, ORCID, version, and project URL.