A unified toolkit for binary and multiclass classification modelling that chains preprocessing, feature selection, model training and tuning, evaluation, visualization, and SHAP interpretability behind one set of objects, while keeping a strict separation between the data that fits a model and the data that judges it. Models, tasks, resampling, and tuning are 'mlr3' objects, multiple imputation is pooled with Rubin's rules, and results are exported in a layout that can be reopened outside R.
English | 简体中文
uniml is a unified predictive modeling toolkit for binary and multiclass
classification. It chains preprocessing, feature selection, model training and
tuning, evaluation, visualization, and SHAP interpretability behind one small set
of objects, while keeping a strict separation between the data that fits a model
and the data that judges it.
The package is organized as a stack of layers. Every layer can be used on its
own, and run_workflow() wires them together.
| Layer | Entry points | Purpose |
|---|---|---|
| L0 data | uniml_spec(), as_uniml_data(), make_split() |
Declare the task, roles, and splits |
| L1 task type | list_task_types(), task_type_spec() |
Extensible per-task behaviour |
| L2 preprocessing | fit_preprocessor(), impute_data(), scale_numeric(), encode_categorical(), balance_classes() |
Imputation, scaling, encoding, balancing |
| L3 selection | select_features(), select_lasso(), select_boruta(), select_rf_importance(), select_rfe(), select_univariate(), select_multivariate(), combine_selections(), remove_collinear() |
Feature selection and collinearity handling |
| L4 modeling | make_task(), resolve_learners(), train_model(), tune_model() |
mlr3 learners, tuning, cross validation |
| L5 statistics | calc_metrics(), calc_ci(), calibration_metrics(), multiclass_calibration(), calibrate_probabilities(), delong_test(), hl_test(), calc_nri_idi(), dca(), find_threshold() |
Metrics with confidence intervals, calibration numbers, tests, decision curves |
| L6 results | assemble_result(), rank_models(), export_results(), save_result(), read_result() |
Ranking, on-disk export, and a result you can hand over |
| L7 plots | plot_roc(), plot_pr_curve(), plot_calibration(), plot_dca(), plot_confusion(), theme_uniml() |
Performance figures |
| L8 SHAP | explain_shap(), plot_shap_importance(), plot_shap_beeswarm(), plot_shap_dependence(), plot_shap_waterfall(), extract_thresholds() |
Interpretability |
| L9 workflow | run_workflow(), predict_uniml() |
End-to-end run and reuse of fitted state |
uniml does not replace mlr3 or tidymodels. It is opinionated orchestration for
one job: producing a binary risk model together with a report you can defend.
Compared with using mlr3 directly. mlr3 gives you learners, tasks,
resampling, and tuning, and it lets you assemble them in any order. Order is the
part that goes wrong: fit a scaler or a feature selector on the full data frame
and every cross-validated number becomes optimistic. uniml fixes the order
once — fit on training data, apply to held-out data, never look back — turns the
remaining choices into an explicit leakage_policy object instead of an accident
of script layout, and adds the reporting layer mlr3 leaves to you: bootstrap
intervals, calibration fitted on out-of-fold predictions, DeLong and
Hosmer-Lemeshow tests, NRI/IDI, decision curves, Rubin pooling across
imputations, and a TRIPOD+AI self-audit.
Compared with tidymodels. recipes, workflows, and tune cover the
modelling mechanics well. uniml targets the clinical-prediction reporting path
— the statistics and the audit trail — and is built on the mlr3 learner
ecosystem, so everything in it is an mlr3 learner you can also use directly.
What neither gives you out of the box. Strict Rubin's rules applied across
the whole pipeline rather than to a single metric, an export layout that rebuilds
every figure outside R, and failures recorded in result$meta$failures instead
of silently dropped.
When not to use it. If you need a model and an AUC and nothing else, plain mlr3 is simpler. If you need multiclass, survival, or deep learning, this package is not it yet.
run_workflow() is binary-only: its
prediction table, thresholds, calibrators, and post-model tests assume a single
probability column, so a multiclass specification stops with a message pointing
at vignette("uniml-multiclass"). Multiclass is supported at the layer level —
specification, data, task, learners, tuning, metrics, calibration, and plots.multiclass_calibration()). Recalibration is available too, as temperature
scaling (calibrate_probabilities()): one fitted scalar, which cannot change the
predicted class. The stronger Dirichlet calibration is not, because its
K^2 + K parameters are not identifiable on the sample size this package
targets.inst/extdata come from
data-raw/simulate-cvd-mortality.R, and the 6.5% target event rate is a
simulation parameter. They make the examples runnable and reproducible. They
are not a patient cohort, and numbers computed on them say nothing about
clinical performance or about how the methods behave on real data.median_mode,
mice, missForest, or missRanger; missingness that depends on the
unobserved value is not modelled.tripod_ai_check() reports those items
as missing; writing them is your job.predict_uniml() scores a data frame in R. There
is no REST endpoint, no model registry, and no drift monitoring.The package is not on CRAN yet. Install it from GitHub:
# install.packages("remotes")
remotes::install_github("zhengyangszy/uniml")
Or from a local checkout, after R CMD build uniml:
install.packages("uniml_0.1.0.tar.gz", repos = NULL, type = "source")
library(uniml)
development <- read.csv(
system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml")
)
new_admissions <- read.csv(
system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml")
)
spec <- uniml_spec(
task_type = "binary",
outcome = "died",
positive = "yes",
id = "id"
)
spec <- set_preprocess(spec, impute = "median_mode", scale = "zscore", encode = "onehot")
spec <- set_feature_selection(spec, methods = c("univariate", "lasso"), combine = "intersect")
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_tuning(spec, folds = 5, evals = 10)
spec <- set_evaluation(
spec,
ci = "bootstrap",
calibration = "platt",
select_on = "validation"
)
split <- make_split(
as_uniml_data(development, spec),
prop = c(0.8, 0.2),
stratify = TRUE
)
result <- run_workflow(
spec,
train = split$train$data,
validation = split$validation$data,
test = new_admissions
)
result
Predictions for new data reuse the fitted preprocessing, selection, and models; they never refit anything:
predict_uniml(result, newdata = new_admissions, model = "best")
inst/extdata contains two synthetic cardiovascular cohorts:
| File | Rows | Purpose |
|---|---|---|
uniml-cvd-mortality.csv |
2000 | Development cohort; split into training and validation |
uniml-cvd-mortality-new.csv |
2000 | Independent new admissions; external test cohort |
Both cohorts target an in-hospital mortality probability of 6.5%. Their observed event rates differ slightly because the binary outcomes are sampled. The new-admission cohort is generated independently from the development cohort and has non-overlapping identifiers.
Every model in uniml is an mlr3 Learner, every data set becomes an mlr3
Task, and tuning and resampling use mlr3 objects directly. The package adds a
thin layer on top instead of hiding mlr3:
make_task() builds a TaskClassif from your data frame and specification.resolve_learners() returns mlr3 learners plus their search spaces, which you
can also tune yourself with mlr3tuning.run_workflow() stores the fitted learners in result$models[[key]]$learner,
so anything that accepts an mlr3 learner keeps working.as_uniml_binary() converts an mlr3 PredictionClassif into the object used
by the evaluation and plotting helpers, and plot_roc(),
plot_calibration(), and plot_confusion() accept an mlr3 prediction
directly.The registry provides seven stable default binary-classification learners:
| Key | mlr3 learner |
|---|---|
lr |
classif.cv_glmnet |
rf |
classif.ranger |
xgb |
classif.xgboost |
nnet |
classif.nnet |
knn |
classif.kknn |
cart |
classif.rpart |
nb |
classif.naive_bayes |
It also provides an optional set of common learners that can be requested by key but are not installed or tuned by default. These entries use their mlr3 defaults unless an explicit search space is supplied:
| Key | mlr3 learner | Optional package |
|---|---|---|
log_reg |
classif.log_reg |
stats |
svm |
classif.svm |
e1071 |
lda |
classif.lda |
MASS |
qda |
classif.qda |
MASS |
The package is not limited to those keys. Any installed mlr3 classification
learner can be passed to set_learners(). Learners outside the registry use
their mlr3 defaults and are not tuned unless a search space is supplied.
spec <- set_learners(
spec,
c("lr", "classif.log_reg", "classif.svm", "classif.lda"),
tune = TRUE
)
This is also how you use learners from packages that are not on CRAN, such as LightGBM. Install and load the providing package first, then pass the mlr3 learner key:
# remotes::install_github("mlr-org/mlr3extralearners")
library(mlr3extralearners)
spec <- set_learners(spec, "classif.lightgbm", tune = FALSE)
The retired lgb, randomForest, and gbm shortcuts now fail with a message
that points at the mlr3 learner key to use instead; nothing else changes.
For a custom search space, register the learner explicitly:
register_learner(
"svm_custom",
mlr3::lrn("classif.svm", predict_type = "prob", probability = TRUE),
paradox::ps(
cost = paradox::p_dbl(0.1, 10),
gamma = paradox::p_dbl(0.001, 1)
)
)
spec <- set_learners(spec, "svm_custom", tune = TRUE)
Every stage accepts an explicit argument list, and unknown names are rejected instead of being silently stored:
spec <- set_preprocess(
spec,
impute = "mice",
impute_args = list(m = 5, max_iter = 10),
scale_args = list(vars = c("age", "bmi")),
encode_args = list(max_levels = 20)
)
spec <- set_feature_selection(
spec,
methods = c("lasso", "boruta"),
args = list(
lasso = list(nfolds = 5, alpha = 0.8),
boruta = list(tentative = TRUE)
)
)
spec <- set_tuning(
spec,
folds = 5,
repeats = 5,
terminator = "seconds",
seconds = 600
)
spec <- set_evaluation(
spec,
calibration = "platt",
calibration_args = list(folds = 5, bins = 10, fit = "out_of_fold")
)
Validation and test data never take part in imputation, scaling, encoding,
selection, balancing, tuning, threshold selection, or calibration. Fitting
happens on the training data only, and the resulting parameters are applied to
the held-out data. set_leakage_policy() controls whether selection, balancing,
and tuning are re-run inside each cross-validation fold (strict_in), left
outside the folds (strict_out), or configured step by step (custom).
| Mode | Steps re-run inside each fold | Use it when |
|---|---|---|
strict_in |
imputation, scaling, encoding, selection, balance, tuning | you want the least optimistic cross-validation estimate and can pay for the run time |
strict_out |
none | the steps are expensive (Boruta, RFE, tuning) and the data set is large |
custom |
selection, balance, tuning | you want to name the trade-off yourself |
spec <- set_leakage_policy(
spec,
mode = "custom",
in_cv = c("selection", "balance")
)
Two cautions:
strict_out fits selection, balancing, and tuning once on the whole training
set. Those steps have then seen every training row, so per-fold scores are
optimistic for them. Cross-validation stays useful for comparing models; do
not report it as an unbiased absolute estimate.validation, test, and external are never used for
fitting, tuning, calibration, or threshold selection. strict_out relaxes what
happens inside the training set, not outside it.run_workflow() takes up to four data sets. test and external are both
optional:
run_workflow(spec, train, validation,
test = internal_holdout, # optional
external = new_admissions) # optional
validation is used for model comparison and selection; test and external
are reporting-only and never take part in fitting, tuning, calibration,
threshold selection, or ranking. Requesting select_on = "test" is allowed but
warns that the test performance becomes optimistic.
run_workflow() reports per-fold cross-validation rows and a summary;
repeats > 1 instantiates repeated cross-validation, and the summary averages
folds within a repeat before summarising across repeats. Calibration is off by
default (calibration = "none"); when requested, the calibrator is fitted on
out-of-fold training predictions and both raw and calibrated probabilities are
kept. Classification thresholds come from the same out-of-fold predictions.
Failed or downgraded steps are never silent: they are listed in
result$meta$failures.
With multiple imputation (impute = "mice", m > 1) the workflow fits each
completed data set, pools metric estimates and confidence intervals with strict
Rubin's rules (pool_mi(), Barnard-Rubin degrees of freedom), and runs
cross-validation once per imputation before pooling. Prediction probabilities
are averaged across imputations ("pooled predictions") and are labelled
separately from the Rubin-pooled metric table.
The choices that are not obvious are deliberate:
balance = "none"). Resampling a risk model
changes what the predicted probability means (van den Goorbergh et al., 2022),
so it is opt-in.T = U + (1 + 1/m)B,
Barnard-Rubin degrees of freedom, a t interval, and the interval clamped to the
metric's domain.result$meta$failures and raises a warning when the workflow returns.NRI/IDI are computed against reference_model (default: the model with the
best training AUC) with bootstrap intervals, and written to
tests/nri_idi.csv. SHAP zero-crossing thresholds are opt-in:
spec <- set_explain(
spec,
model = "best",
plots = c("importance", "beeswarm", "dependence", "waterfall"),
thresholds = list(targets = c("age", "bmi"))
)
Plots accept theme, title, subtitle, and caption, and palette can be
a ggsci name, a colour, or a colour vector. Export size and format are
controlled with export_results(..., plot_args = list(dpi = 600, device = "png")).
Everything needed to redraw or re-tabulate is exported as plain CSV plus one
RDS: predictions/predictions.csv (model, set, raw and calibrated
probabilities, truth) is enough to rebuild ROC/calibration/DCA/confusion
figures with plot_roc()/plot_calibration()/plot_dca()/plot_confusion(),
performance/thresholds.csv stores the raw and calibrated thresholds per
model, and shap/values_matrix_*.csv plus shap/feature_values_*.csv allow
custom SHAP figures. result.rds contains the complete fitted state (models,
preprocessor, calibrators) for predict_uniml() on new data.
vignette("uniml-atomic", package = "uniml")
vignette("uniml-workflow", package = "uniml")
vignette("uniml-shap", package = "uniml")
vignette("uniml-realdata", package = "uniml")
vignette("uniml-multiclass", package = "uniml")
uniml-realdata runs the whole workflow on pROC::aSAH, a published cohort of
113 patients with subarachnoid haemorrhage, and shows what the reporting layer
does when the data is small: the outcome is a collapse of gos6, so that column
has to be excluded; ratios whose bootstrap distribution is degenerate come back
as NA with a reason; and a Hosmer-Lemeshow statistic that cannot be defined
says so instead of returning NaN.
uniml-multiclass covers the three-class path: the metric conventions, the k-by-k
confusion table, one-vs-rest ROC curves, and the methods that are deliberately
not implemented for more than two classes.
uniml is at version 0.2.0. The binary and multiclass task types are
implemented; survival is registered as a placeholder only. GitHub Actions
checks the package on Windows, macOS, and Linux, and
tests/testthat/test-consistency.R pins the metric layer to independent
implementations (mlr3measures, pROC, PRROC, stats::glm(),
stats::isoreg(), ResourceSelection). See NEWS.md for changes.