Also known as: benchmark contamination, test-set leakage, eval contamination, train-test overlap
When evaluation data leaks into training data, inflating benchmark scores without improving real capability. Detected via n-gram match against eval sets, log-probability attacks, or membership inference.
Data contamination is the silent killer of LLM benchmarks: evaluation examples that appeared in the model’s training corpus inflate its scores without reflecting real capability. The model isn’t being measured on its ability to generalize; it’s being measured on its ability to recall. Every major benchmark — MMLU, GSM8K, HumanEval, BBH, BIG-Bench, even retrieval evals like MS MARCO — has documented contamination in modern pretraining corpora , and the gap between “headline benchmark score” and “real capability” widens every year as more benchmarks become contaminated.
The contamination loop is mechanical:
The benchmark doesn’t even need to be in the corpus directly — discussion of the benchmark is enough, and that discussion is unblockable without aggressive proper-name URL filters that hurt other content. Deduplication doesn’t help because each benchmark item typically appears once or a few times, well below the threshold for near-duplicate flagging.
Three families of contamination tests, with different strengths.
The combination is more reliable than any single method. N-gram match is the default first pass because it’s cheap and produces an interpretable contamination index — “X% of MMLU items have a 13-gram match somewhere in the corpus.” Log-prob attacks and counterfactual reformulation catch what n-gram misses.
For a piece of text the model may or may not have seen, compute the per-token log-probability under the model. Sort tokens by log-probability ascending; take the bottom K% (typically 20%). Sum these. The intuition: even on text the model has seen, most tokens are highly probable; what differs is that on unseen text, some tokens will be genuinely surprising (low probability), while on seen text the model has memorized even the unusual tokens.
Shi et al. 2023 (“Detecting Pretraining Data from Large Language Models”) showed this score reliably distinguishes seen vs. unseen passages from the model’s perspective, well above chance. The method is white-box-ish — you need log-probabilities, which open-weight models give you and most API models partially expose (via logprobs in completion responses).
The limitation: min-k% works on substantial passages (a paragraph or more); single-question benchmark items are too short for the statistic to be reliable. For benchmark contamination specifically, n-gram and behavioral tests are more practical.
For a frontier-lab pretraining run, the contamination workflow is:
For a smaller fine-tuning run on top of a base model, you have an additional contamination surface: your fine-tuning data. If your fine-tuning data overlaps with your eval set, you’ll see inflated post-training metrics that don’t survive deployment. Always train/eval split fine-tuning data with the same n-gram check.
Some do — MMLU-Pro, GPQA, and FrontierMath were created partly to escape contamination of their predecessors. But every refresh has the same problem: the moment it becomes the load-bearing public benchmark, it goes on the web in discussions, leaderboards, and example outputs. The half-life resets but the dynamic is unchanged.
The structural alternatives are (a) private held-out benchmarks (Scale Eval, Vals, Anthropic’s internal evals) which require trust in the evaluator, (b) benchmarks released only with delayed answer keys, (c) benchmarks generated dynamically per query (so each model sees new items), or (d) capability-specific evals (red-teaming, coding-on-novel-tasks) that are hard to memorize. The frontier eval ecosystem in 2026 is shifting toward (a) and (d); (b) and (c) have logistical and validity problems.
Contamination is the most likely explanation when:
When you observe these, the right next step is contamination detection — not celebration.
N-gram exact-match (typically 13-gram) is the standard first pass — fast, easy to ship, low false-positive rate. Augment with min-k% probability test (Shi et al. 2023) for paraphrased contamination. For frontier-scale work, run a contamination index across every benchmark you plan to evaluate on, before training. Detection after-the-fact rarely changes anything.
Public benchmarks are on the public web; pretraining corpora are scraped from the public web. The benchmark questions, often as hyperlinked discussions, leaderboard pages, or model-card examples, appear in Common Crawl. Even with explicit blocklists, paraphrases, partial leaks, and translated versions slip through. The only robust fix is private held-out evaluation, which has its own credibility problems.
On clean cases, contamination of MMLU questions adds 5-15 percentage points; GSM8K can move 10-20 points. The 2023 contamination audits of GPT-4 and Llama-2 found MMLU contamination rates of 10-30% by item. Not every contaminated item is recovered verbatim — but the lift is large enough that benchmark deltas under 5 points across models are dominated by contamination noise, not capability differences.
