Pre-submission audit fixes, ahead of the first CRAN release.
ev_bootstrap() and ev_rank() called set.seed() on the global random
stream and left it altered. A user who passed seed to make one call
reproducible found every later random draw in their session silently
shifted. The seed now applies for the duration of the call only and the
user's .Random.seed is restored on exit. Seeded calls remain
reproducible and unseeded calls still vary.
ev_score() and ev_cluster() now warn when the interval has
degenerated. A slice where every item passes gives a zero standard error,
so the Wald interval collapses to a point and appears to claim certainty:
50 of 50 correct reported [1.0000, 1.0000] where Wilson gives about
[0.93, 1.00]. Too few clusters pushes a pass rate's interval outside
[0, 1]. Both are properties of the normal approximation rather than
errors, so they warn rather than abort, and the binary case quotes the
Wilson interval for comparison.
README gains a Limitations section covering the interval method and where it degrades, the cluster-count requirement, the random-sampling assumption behind prediction-powered inference, and the difference between judge agreement and judge accuracy.
First release.
Statistical inference for language model evaluations, following Miller (2024) doi:10.48550/arXiv.2411.00640 for the core standard error and experiment design results, and the prediction-powered inference literature for the model judge functions.
as_eval() builds the long-format evaluation object every other function
consumes, from a data frame or a bare vector of scores.ev_score() reports a mean score with a central limit theorem standard error
and confidence interval, switching to cluster-robust inference automatically
when a cluster column is present.ev_cluster() gives the cluster-robust standard error with the design effect
and intra-cluster correlation that explain it, and ev_icc() returns the
intra-cluster correlation on its own.ev_resample() separates between-item variance from response sampling noise
when several responses are drawn per question, and projects the standard
error achievable at any number of responses per item.ev_bootstrap() provides a cluster bootstrap for statistics that are not
means.ev_paired() compares two models on the same questions and reports the
variance saved by pairing.ev_unpaired() handles disjoint question sets with a Welch interval.ev_variance_reduction() applies a control variate from a reference model
with a known score on the full question bank.ev_multi() adjusts across a benchmark suite, pools by inverse variance, and
tests heterogeneity.ev_power() and ev_mde() size a comparison and report its minimum
detectable effect, both accounting for clustering.ev_judge_agreement() reports accuracy, Cohen's kappa, and a McNemar test
for systematic judge bias.ev_judge_debias() implements the power-tuned prediction-powered estimator,
combining many judge scores with a small human sample.ev_judge_power() sizes the human labelling budget.ev_rank() gives bootstrap rank intervals and the probability each model is
best.ev_elo() fits Bradley-Terry ratings with standard errors on the Elo scale.ev_table() and ev_plot() collect results into a table and draw them with
error bars.