Evaluation

BEIR is a heterogeneous benchmark of 18 retrieval datasets across domains — biomedical, news, finance, scientific QA, fact-checking — designed to test zero-shot retrieval. The standard reference for whether a retriever generalizes beyond MS MARCO.

When you compare two scoring systems on the same items — index score vs reranker score, model score vs ground-truth grade — the residuals from a regression line tell you where they disagree.

Citation extraction maps each claim in an LLM-generated answer back to the supporting span in the source documents. Distinct from generation — often a small specialized model — and what makes RAG outputs auditable.

The 100-year-old psychometric toolkit that ML eval research mostly ignored. Decompose every observed score into $X = T + E$ (true score + error).

Cohen's $\kappa = (p_o - p_e) / (1 - p_e)$ — observed agreement minus chance agreement, normalized. The standard inter-annotator agreement metric. Raw % agreement is misleading on imbalanced classes; kappa is the honest version.

$\alpha = (k/(k-1)) \cdot (1 - \sum_q \sigma^2_q / \sigma^2_X)$ — the aggregate internal-consistency reliability number that falls out of CTT.

A practical diagnostic checklist for *is this benchmark actually any good?* Layer four measurement-theory tools — <Concept slug="classical-test-theory">CTT</Concept> for per-item pathologies, Cronbach's $\alpha$ for aggregate reliability.

The F1 score is the harmonic mean of precision and recall — a single number that punishes lopsided performance. Standard for classification, rare in retrieval, where ranked metrics like NDCG@K are usually the better choice.

Faithfulness is whether each claim in an LLM's answer is actually supported by the retrieved context. Distinct from relevance and from accuracy.

An LLM-as-judge configured to emit *graded* relevance — typically a 0-3 scale (irrelevant / marginal / relevant / highly relevant) rather than a binary yes/no.

Isotonic regression fits a non-parametric monotone function from raw scores to calibrated probabilities. More flexible than Platt scaling — handles any monotone miscalibration shape — at the cost of needing more labels and being prone to overfitting at the score-distribution tails.

Using a frontier LLM to score outputs — relevance, faithfulness, answer quality — at scale where human raters can't keep up. Powerful for graded labels, but introduces position bias, verbosity bias, model bias.

Mean Average Precision averages precision at each rank where a relevant document appears, then averages across queries. The older sibling of NDCG — comparable for binary relevance, weaker for graded relevance.

Mean Reciprocal Rank is the average of 1/rank across queries, where rank is the position of the first relevant document. Heavily front-loaded — only the top result really matters.

MS MARCO is Microsoft's web-search dataset of ~1M Bing queries paired with passages and human relevance judgments. The standard training corpus for retrievers and rerankers, the source of every modern dense retriever.

Massive Text Embedding Benchmark — a public benchmark covering 50+ datasets across retrieval, classification, clustering, and more. The de facto leaderboard for embedding models, despite some well-documented limitations in its retrieval portion.

Normalized Discounted Cumulative Gain at K is a ranking quality metric that rewards relevant documents appearing high in the result list, with logarithmic discounting for lower positions. The standard top-of-list quality metric for rerankers.

Platt scaling fits a logistic sigmoid on top of a model's raw scores to produce calibrated probabilities. Cheap, two parameters, the standard first-resort calibration method for SVMs, classifiers, and uncalibrated rerankers.

Precision@K is the fraction of the top-K returned documents that are relevant. The classical IR metric retrieval moved away from in favor of NDCG, but still the right choice when every position in the result list carries equal weight.

Recall@K is the fraction of queries whose relevant document appears anywhere in the top-K results. It measures whether retrieval found the right document at all — the silent ceiling on every downstream stage.

Retrieval evals report metrics like NDCG@10 averaged across queries — but each query is one sample, and most public benchmarks have hundreds, not thousands. A '+0.5 NDCG' difference is often noise.