Classical Test Theory

Classical Test Theory (CTT) is the boring, robust, hundred-year-old psychometric framework that the ML eval literature mostly forgot. It answers most of the diagnostic questions you actually have about a benchmark — is this item too easy? is it noise? is the whole test internally consistent? — using nothing but the raw response matrix and a handful of summary statistics.

The true-score model

CTT starts from a single decomposition. Every observed score on an item (or a test) is the sum of a true score and a measurement error :

with and — the error has zero mean and is uncorrelated with the true score. That single assumption gets you the variance decomposition , and the definition of reliability:

— the fraction of observed variance that is signal rather than measurement noise. Reliability of 1 means the test perfectly orders test-takers; reliability of 0 means it’s a coin flip. Most ML benchmarks live somewhere between.

Item difficulty:

For a binary-graded item , item difficulty is the simplest possible statistic: the fraction of test-takers who got it right.

The name is the opposite of the value — high means easy, low means hard. The item’s variance under a Bernoulli model is , which is maximal at and collapses to zero as or .

This is the first CTT diagnostic for benchmark health. Items with (everyone right) or (everyone wrong) carry no information about model differences — they have zero variance to explain. A benchmark where 60% of items have is a saturated benchmark dressed up as a hard one.

For graded items (e.g. graded relevance on a - scale) the analog is mean grade — replace “fraction correct” with “mean score” and the same logic applies: items where every model scores 3 or every model scores 0 are dead.

Item discrimination:

The second CTT diagnostic asks a different question: do the test-takers who get this item right tend to be the ones who do well overall? This is point-biserial correlation — the Pearson correlation between the binary item score and the total test score:

where is the mean total score among test-takers who got item right and is the mean among those who got it wrong. No independence or unidimensionality assumption baked in — it’s just a Pearson correlation between two columns of the response matrix.

Heuristic thresholds, well-established in the educational measurement literature:

Negative discrimination is the most diagnostic single signal in CTT. If a frontier model is less likely than a weak model to get item right, the most likely explanation is that the gold label is wrong — not that the frontier model is suddenly underperforming.

The high-low method

A poor man’s discrimination, computable in your head from a ranked spreadsheet:

— fraction correct in the top 27% of test-takers minus fraction correct in the bottom 27%. The 27% threshold is a Kelley result from 1939 that maximizes discrimination-power for normally-distributed total scores; in practice anywhere between 25% and 33% gives essentially the same answer.

is excellent, is a noise item. The high-low number is monotone with point-biserial in well-behaved cases — use it when you want to eyeball a single suspicious item without firing up Python.

Why CTT survives where parametric fits don’t

CTT’s only structural assumption is that measurement error is mean-zero and uncorrelated with true score — a much weaker condition than the parametric, independence-and-unidimensionality assumptions baked into latent-trait models. On real ML evals where items cluster by topic, test-takers are correlated through shared training data, and the item pool deforms under contamination, that gap matters.

The price: CTT can’t give you per-test-taker ability on a calibrated cross-study scale. The gain: , , and Cronbach’s are summary statistics of the response matrix, computable in five lines of NumPy on any benchmark, and they don’t depend on a model fit converging or its assumptions holding.

A worked diagnostic

Imagine a -item benchmark, models scored on it. You tabulate and for each item:

• items have — every model gets them. Drop them; they contribute no variance.
• items have — no model gets them. Drop them too, or investigate for label errors.
• items have — they correlate weakly with total score. Probably noise; drop or fix.
• items have — likely mislabeled. Re-annotate.

After the cuts, you have items doing real measurement work. Recompute the aggregate: Cronbach’s α on the trimmed test will typically rise even though dropped, because the noise items were subtracting from reliability. This is the single highest-leverage move in eval curation, and CTT is what tells you which items to cut.

Connection to the inter-rater layer

CTT diagnoses the test. Cohen’s κ diagnoses the labels the test depends on. They are complementary: a test where the item-level are healthy but judge-judge κ is is a test on top of unreliable labels — the discrimination is real but the ranking is between models that are noisily mis-scored. Always layer the two: κ to gate the labels, CTT to gate the items, then paired significance tests to gate the model comparisons.