Also known as: Spearman rho, rank correlation
Spearman's ρ is Pearson correlation computed on ranks instead of raw values. It captures any monotone relationship — linear or curved — and is the correct correlation for ranking and retrieval evaluation, where what matters is order.
Spearman correlation is what you get when you take Pearson and replace the raw values with their ranks. Sort each variable, write down each observation’s rank position, then compute Pearson on the rank sequences. The result, denoted
The conversion to ranks is the entire content of the method. It buys you robustness to the scale of the underlying variable — you only care about order — and it generalizes from linear to any monotone relationship.
where
A function
This is the property that matters for ranking evaluation. A reranker’s logits and a relevance label might be related by a sigmoid, a square root, or any other monotone curve. The reranker has done its job — it has the order right — even though Pearson on raw values understates how much it knows.
NDCG , MRR , MAP, and recall-at-k are all functions of rank position. They do not look at the magnitude of the relevance scores produced by the retriever — only at the order in which documents come out. So when you ask “do retriever A and retriever B agree?”, the right correlation is rank correlation:
(score, label) pairs is the natural per-query agreement metric. Average across queries.Both are rank-based. The split:
For most retrieval-eval purposes either works and they give correlated answers. Tau has slightly better statistical properties (smaller variance, exact null distribution) but Spearman is the older default and gets reported more often.
Graded-relevance retrieval evals have lots of ties — a long-tail of documents with label 0 (not relevant), a smaller cluster with label 1, etc. Spearman with the standard tie correction is approximately right, but the resulting null distribution is no longer cleanly tabulated.
Kendall tau-b explicitly handles ties in the denominator. For datasets with heavy tie structure (most retrieval graded-relevance evals), tau-b is the cleaner statistic and is what frameworks like pytrec_eval report under the hood when they describe rank agreement.
In practice: report Spearman on per-query metric correlations (where ties are rare because metrics are continuous), and tau-b when you correlate raw graded labels against scores (where label ties dominate).
Spearman has the same kind of t-test approximation as Pearson — replace the raw values with ranks and apply the same formula — but the assumption of bivariate normality is now bivariate normality of ranks, which is essentially never satisfied (ranks are uniform, not Gaussian). For small
The discipline: when you want to report agreement between two ranked outputs, compute Spearman and bootstrap a 95% interval. Two lines of NumPy. The headline correlation alone is not enough.
Both are rank-based and both capture monotone relationships, but they answer slightly different questions. Spearman is Pearson on ranks — it weights large rank disagreements quadratically. Kendall tau counts the fraction of concordant pairs minus discordant pairs. Tau is more interpretable ('73% of pairs agree'), more robust to small numbers of disagreements, and computes in
NDCG, MRR, and MAP are all functions of rank position, not score magnitude. If you compare two retrievers' rankings of the same documents, what you want to know is whether their orderings agree. Spearman computes exactly that. Pearson on raw scores can be high while orderings disagree (calibration without ordering), or low while orderings perfectly match (orderings without calibration).
Pure Spearman assumes distinct ranks. With ties, you average the ranks across the tied positions ('midrank' or 'fractional ranking') and apply a tie-correction term to the denominator. SciPy's spearmanr handles this automatically. For graded-relevance retrieval evals where labels are 0-3 with many ties, the correction matters; without it the reported ρ is biased upward.
