Package: evaluatellm Title: Statistical Inference for Language Model Evaluations Version: 0.1.0 Authors@R: person("Charles", "Coverdale", email = "charlesfcoverdale@gmail.com", role = c("aut", "cre", "cph")) Description: Treats language model evaluations as statistical experiments and supplies the inference they require. Provides central limit theorem and cluster-robust standard errors for evaluation scores, paired and unpaired model comparisons, variance decomposition when several responses are drawn per question, control-variate variance reduction, multiplicity adjustment across benchmark suites, and power and minimum detectable effect calculations for planning evaluations, following Miller (2024) . For evaluations scored by a model judge, implements agreement statistics against a human gold standard and prediction-powered inference (Angelopoulos et al. 2023) with the power-tuned estimator of Angelopoulos, Bates and Jordan (2023) , so a small set of human labels debiases a large set of judge scores. Leaderboards are supported through bootstrap rank intervals and Bradley-Terry ratings (Bradley and Terry 1952) . Accepts scores from any evaluation harness. License: MIT + file LICENSE Encoding: UTF-8 Language: en-GB URL: https://charlescoverdale.github.io/evaluatellm/, https://github.com/charlescoverdale/evaluatellm BugReports: https://github.com/charlescoverdale/evaluatellm/issues Roxygen: list(markdown = TRUE) RoxygenNote: 7.3.3 Depends: R (>= 4.1.0) Imports: cli (>= 3.6.0), graphics, grDevices, stats, utils Suggests: testthat (>= 3.0.0), knitr, rmarkdown, sandwich VignetteBuilder: knitr Config/testthat/edition: 3 Repository: https://charlescoverdale.r-universe.dev Date/Publication: 2026-09-19 10:23:35 UTC RemoteUrl: https://github.com/charlescoverdale/evaluatellm RemoteRef: HEAD RemoteSha: 3ef61edaee981b31b84843678a96f6f6ddd27702 NeedsCompilation: no Packaged: 2026-10-19 11:21:43 UTC; root Author: Charles Coverdale [aut, cre, cph] Maintainer: Charles Coverdale