Statistical Inference for Language Model Evaluations

Treats language model evaluations as statistical experiments and supplies the inference they require. Provides central limit theorem and cluster-robust standard errors for evaluation scores, paired and unpaired model comparisons, variance decomposition when several responses are drawn per question, control-variate variance reduction, multiplicity adjustment across benchmark suites, and power and minimum detectable effect calculations for planning evaluations, following Miller (2024) . For evaluations scored by a model judge, implements agreement statistics against a human gold standard and prediction-powered inference (Angelopoulos et al. 2023) with the power-tuned estimator of Angelopoulos, Bates and Jordan (2023) , so a small set of human labels debiases a large set of judge scores. Leaderboards are supported through bootstrap rank intervals and Bradley-Terry ratings (Bradley and Terry 1952) . Accepts scores from any evaluation harness.

Conformal Prediction and Uncertainty Quantification

Implements conformal prediction methods for constructing prediction intervals (regression) and prediction sets (classification) with finite-sample coverage guarantees. Methods include split conformal, 'CV+' and 'Jackknife+' (Barber et al. 2021) , 'Conformalized Quantile Regression' (Romano et al. 2019) , 'Adaptive Prediction Sets' (Romano, Sesia, Candes 2020) , 'Regularized Adaptive Prediction Sets' (Angelopoulos et al. 2021) , Mondrian conformal prediction for group-conditional coverage (Vovk, Gammerman, and Shafer 2005) , weighted conformal prediction for covariate shift (Tibshirani et al. 2019) , and adaptive conformal inference for sequential prediction (Gibbs and Candes 2021) . All methods are distribution-free and provide calibrated uncertainty quantification without parametric assumptions. Works with any model that can produce predictions from new data, including 'lm', 'glm', 'ranger', 'xgboost', and custom user-defined models.

Nonparametric Rotations for Sphere-Sphere Regression

Fits sphere-sphere regression models by estimating locally weighted rotations. Simulation of sphere-sphere data according to non-rigid rotation models. Provides methods for bias reduction applying iterative procedures within a Newton-Raphson learning scheme. Cross-validation is exploited to select smoothing parameters. See Marco Di Marzio, Agnese Panzera & Charles C. Taylor (2018) .

Tools for Trade Practitioners

A collection of tools for trade practitioners, including the ability to calibrate different consumer demand systems and simulate the effects of tariffs and quotas under different competitive regimes. These tools are derived from Anderson et al. (2001) and Froeb et al. (2003) .

Bayesian QTL Mapping Toolkit

QTL mapping toolkit for inbred crosses and recombinant inbred lines. Includes maximum likelihood and Bayesian tools.

Fuzzy Rank Tests and Confidence Intervals

Does fuzzy tests and confidence intervals (following Geyer and Meeden, Statistical Science, 2005, ) for sign test and Wilcoxon signed rank and rank sum tests.

Aster Models

Aster models are exponential family regression models for life history analysis. They are like generalized linear models except that elements of the response vector can have different families (e. g., some Bernoulli, some Poisson, some zero-truncated Poisson, some normal) and can be dependent, the dependence indicated by a graphical structure. Discrete time survival analysis, zero-inflated Poisson regression, and generalized linear models that are exponential family (e. g., logistic regression and Poisson regression with log link) are special cases. Main use is for data in which there is survival over discrete time periods and there is additional data about what happens conditional on survival (e. g., number of offspring). Uses the exponential family canonical parameterization (aster transform of usual parameterization). Unlike the aster package, this package does dependence groups (nodes of the graph need not be conditionally independent given their predecessor node), including multinomial and two-parameter normal as families. Thus this package also generalizes mark-capture-recapture analysis.

Simplify Connections to Database Sources

Connecting to databases requires boilerplate code to specify connection parameters and to set up sessions properly with the DBMS. This package provides a simple tool to fill two purposes: abstracting connection details, including secret credentials, out of your source code and managing configuration for frequently-used database connections in a persistent and flexible way, while minimizing requirements on the runtime environment.

'ViennaCL' C++ Header Files

'ViennaCL' is a free open-source linear algebra library for computations on many-core architectures (GPUs, MIC) and multi-core CPUs. The library is written in C++ and supports 'CUDA', 'OpenCL', and 'OpenMP' (including switches at runtime). I have placed these libraries in this package as a more efficient distribution system for CRAN. The idea is that you can write a package that depends on the 'ViennaCL' library and yet you do not need to distribute a copy of this code with your package.

Functions to Analyze Single System Data

Functions to visually and statistically analyze single system data.