Cronbach's Alpha

Cronbach’s α is the standard aggregate reliability statistic for a test — the single number that summarizes “do the items in this test agree they’re measuring the same thing?” It falls out of Classical Test Theory ’s variance decomposition and is, for any test with more than a handful of items, the first scalar you should report next to a leaderboard.

The formula

For a test with items, where is the variance of item across test-takers and is the variance of the total score :

The intuition is in the ratio inside the parentheses. If items are uncorrelated, exactly (sum of independent variances), the ratio is , and . If items are perfectly correlated, blows up to , the ratio shrinks to , and . The factor is a finite-sample correction.

A rearrangement makes the meaning even cleaner. Let be the average inter-item covariance and the average item variance:

— α is a function of the average inter-item covariance scaled by the test length. Items that co-vary push α up; items that don’t, drag it down.

Rule-of-thumb thresholds

The literature has a stable set of heuristic bands, originally from Nunnally:

For ML benchmarks the operational threshold is . Below that, your reported leaderboard differences are not stable to which items you happened to include.

The Spearman-Brown trap

Cronbach’s α has a built-in pathology that almost no ML eval paper acknowledges: α rises with , even when the new items are noise. The Spearman-Brown prophecy formula spells out the relationship:

where is the reliability of a -item test and is the reliability you’d predict by extending it -fold with comparable items. A -item test with extended to items predicts — a “respectable” reliability that is purely a length artifact.

The implication: two benchmarks with the same α are not equally reliable per item. A -item benchmark at may be no more diagnostic per item than a -item benchmark at . Always report alongside α — and when comparing benchmarks, compare per-item reliability via the Spearman-Brown back-calculation, not raw α.

import numpy as np

def cronbach_alpha(R):
    R = np.asarray(R, dtype=float)
    k = R.shape[1]
    item_var = R.var(axis=0, ddof=1).sum()
    total_var = R.sum(axis=1).var(ddof=1)
    return (k / (k - 1)) * (1 - item_var / total_var)

Where it fails for ML evals

Cronbach’s α was developed for tests that measure a single construct — a math test, a personality dimension, a vocabulary test. Its key statistical assumption is tau-equivalence: every item measures the same latent construct with the same loading, differing only in difficulty and noise. Most ML benchmarks violate this, sometimes flagrantly:

• Multi-domain benchmarks. MTEB covers retrieval, classification, clustering, STS — clearly not a single construct. A pooled α conflates “how well does this model embed text?” with “how well does it cluster?” and the resulting number is a length-inflated chimera. Compute α within each task family (per dataset, per task type), not over the whole suite.
• Mixed-difficulty subsets. A test that mixes elementary and frontier items often shows high α from sheer length, masking the fact that no single item-pool window covers the whole ability range. Stratify by difficulty band and recompute.
• Heterogeneous test-takers. α assumes test-takers come from a comparable population — variance contributions are stable. Mixing 0.6B-parameter models with 70B-parameter models in the same response matrix bloats and inflates α regardless of item quality.
• Skewed score distributions. α is derived under approximate normality. Bimodal distributions (models that either fully solve or fully fail an item) violate the variance decomposition; the headline number becomes hard to interpret.

A practical reporting template

Strip α reporting down to four numbers and one comparison:

1. α and . Always paired. Never report α without — that’s like reporting a p-value without sample size.
2. Per-item reliability . The Spearman-Brown-inverted single-item reliability. This is the apples-to-apples number across benchmarks.
3. Per-subdomain α if the benchmark is multi-domain. With spread.
4. Bootstrap 95% CI on α. Resample test-takers; report .
5. α-if-item-deleted for the bottom decile. Items whose removal raises α are noise items — they are reducing internal consistency. Drop them and recompute.

That last move — α-if-item-deleted — is the single most useful operational diagnostic Cronbach’s α gives you. It collapses the question “is this item helping?” into a one-number answer: if removing the item raises α, the item is hurting. Combined with low point-biserial , it’s the cleanest way to identify the items worth cutting from a benchmark.

When you have α, what do you do with it?

The same three operational uses as Cohen’s κ , transposed from the inter-rater axis to the inter-item axis:

1. Gate the benchmark. on a candidate eval set means the items don’t agree on what they’re measuring; iterate (drop noise items, add signal items, reconsider domain pooling) before publishing rankings on it.
2. Compare to the per-item-reliability ceiling. A frontier-quality benchmark targets (per-item reliability)— roughly equivalent to at . Below that, you’re doing length-driven inflation.
3. Track over time. As a benchmark ages and models converge near the top, shrinks, item co-variance dries up, and α drops — the formal signature of a saturating leaderboard. Re-measure annually; budget for a re-curation pass when α slips below .

A test you cannot certify with a published α is a test you cannot certify, period. The combination — CTT per-item diagnostics, Cronbach’s α at the test level, Cohen’s κ on the labels, paired significance tests on the model comparisons — is the minimum measurement-theory stack a serious benchmark earns its name with. See eval set quality for the consolidated checklist.