ANOVA

ANOVA (analysis of variance) is the statistical test for comparing three or more groups. The question it answers is direct: does group identity explain more variation in the data than within-group noise would predict by chance? If yes, the groups are genuinely distinguishable; if no, the apparent differences in their means are within the range of random variation.

The framework was Fisher’s contribution in the 1920s, originally for agricultural experiments — does this fertilizer yield differ across varieties of wheat? It generalized into the workhorse for any multi-group comparison: clinical trials with three or more arms, A/B/C/D product tests, multi-system benchmarks.

The variance partition

Given groups with observations each, total variation in the data decomposes into two pieces:

• — variation attributable to group means differing from the grand mean. Large when groups are pulled apart.
• — variation of observations around their own group’s mean. The “noise” within each group.

The test statistic is the ratio of mean squares — the sums of squares divided by their degrees of freedom:

where is the total number of observations. Under the null hypothesis (all group means equal), follows an F-distribution with degrees of freedom. A large observed — one in the upper tail of that distribution — rejects the null.

One-way vs repeated-measures

The two flavors of ANOVA that matter:

• One-way ANOVA — independent subjects across groups. Each observation belongs to one group only. The denominator is the pooled within-group variance.
• Repeated-measures ANOVA — every subject is observed under every condition. Same patients measured at every time point; same query set scored by every system. The subject-level variance is removed from the denominator (because it’s shared across conditions), which sharply increases sensitivity. The same underlying effect produces a much larger F.

Assumptions

ANOVA rests on three:

In practice, the dominant failure mode in evaluation is the independence assumption — running one-way ANOVA on data where every item was scored by every system (paired data masquerading as independent). The fix is always repeated-measures.

What ANOVA does NOT tell you

The F-test answers “is there some difference somewhere among the groups?” — a single global yes/no. It does not tell you which groups differ, or by how much. For those, you need:

• Pairwise comparisons — Tukey’s HSD or similar, with multiplicity correction, to identify which specific pairs differ.
• Effect size — typically (eta-squared) or partial , the share of total variance explained by group identity. Tells you how big the effect is, in a unit-free, sample-size-independent way. For the two-group special case, use Cohen’s .

Why this matters at scale

A practical note for large- evaluation: the F-test, like every parametric test, becomes trivially significant when is large enough. With items across four systems, even a effect — group identity explains half a percent of the variance — produces a wildly significant F-statistic. The test is doing exactly what it’s supposed to, but the answer it gives is “yes, the groups differ” when the practical reality is “barely.”

Always report ANOVA results with their effect-size partner. A significant F-test alone, at scale, is almost meaningless; an F-test with is a meaningful difference; an F-test with is a difference that’s real but small enough to ignore. See effect size for the calibration.