We pull queries, corpora, and original qrels from 24 MTEB datasets via HuggingFace, covering retrieval, reranking, and instruction-retrieval task types from the MTEB(Multilingual, v2) benchmark. Multi-subset datasets (e.g. MIRACL or WikipediaRetrievalMultilingual) are handled as separate subsets sharing a single evaluation.

Each of the 16 embedding models encodes every corpus and query. Rankings are computed via brute-force cosine similarity: L2-normalize both matrices, compute queries @ corpus.T, then argpartition for top-1000 per query. Models are served via their native APIs (OpenAI, Voyage, Cohere, Gemini) or self-hosted on Modal (zembed-1, qwen3, harrier, jina, bge).

For each query, we union the top-K retrieved documents across all 16 embedding models. This ensures every judge scores every document that any model might surface. This is primarily a cost-saving measure — since we care about the relative performance of models on this list, we only need to annotate documents that at least one of them retrieved, rather than scoring the entire corpus. This will mean that absolute scores for a model may change as we add more embedding models to the leaderboard, but the pairwise ranking between any two models for any given dataset will remain invariant to such additions.

Each (query, document) pair is scored independently — one pair per LLM call, no listwise comparison between documents. The prompt structure forces reasoning before scoring: judges must write their analysis first and commit to a 0-10 float last, reducing anchoring bias. The rubric explicitly anti-inflates: “Most documents should score below 7. Scores of 9-10 should be rare.”

The system prompt (rubric + query) is placed in the cacheable prefix, and work is sorted by query_id, so all documents for the same query share a cached prompt. At ~90% cache hit rate, this cuts input costs by roughly 5x. Each judge runs at up to 1024 concurrent API calls.

You are a search relevance judge. You will be given a query and a
single document. Your task is to evaluate how relevant the document
is to the query.

Score the document using a floating point number between 0 and 10:

- 10: Near-perfect match. The document directly and comprehensively
      answers the query.
- 9:  Extremely relevant. Substantial, directly useful information,
      missing only minor details.
- 8:  Highly relevant. Significant related content that would
      meaningfully help answer the query.
- 6-7: Moderately relevant. Clear connection with some useful
       information, but much is tangential.
- 4-5: Borderline to weakly relevant. Shares a general subject
       area but unclear if it helps.
- 2-3: Barely relevant. Only superficial or incidental connection
       (e.g., shared keywords, different context).
- 0-1: Irrelevant. No meaningful connection to the query.

Important:
- Most documents should score below 7.
- Scores of 9-10 should be rare and reserved for truly exceptional
  matches.
- Do not inflate scores.

Respond with JSON: {"reasoning": <string>, "score": <float>}

It is very important that you output your reasoning FIRST and the
score LAST. Think through why the document is or isn't relevant
before committing to a number.

For each of the three judges independently, we compute NDCG@k and Recall@k curves from k=1 to k=100 across all 128 system configurations (16 embeddings × 7 rerankers + 16 embed-only). Reranked configurations take the top-100 documents from each embedding model and re-sort by reranker score — meaning a reranker can reorder but never recover documents the embedder missed. Recall@100 is identical with or without a reranker; only the ranking within that window changes. Evaluation is parallelized across embedding models.

NDCG uses the full continuous scores. Recall uses per-judge thresholds calibrated to each judge’s score distribution (Gemini: 6.0, GPT-5-nano: 7.0, Grok-4-fast: 6.5) — the thresholds differ because the judges have different score distributions, but the resulting relevance sets are comparable in size (id est, they target the same percentiles). This normalization actually matters very little — we found nearly identical absolute performance metrics and relative rankings regardless of a flat 7.0 threshold, percentile based thresholding, or quantile normalization; the evaluation leaderboard was remarkably invariant to such considerations. (For example, the top three models on the average leaderboard are unchanged regardless of threshold from 0.4 to 0.9).

The per-dataset, per-judge evaluation curves are averaged in two stages: first across datasets for each judge (so “GPT-5-nano average” is the mean of that judge’s per-dataset NDCG/Recall curves), then across judges to produce the “Average” pseudo-judge shown by default on the dashboard. The dashboard lets you drill into any individual dataset or judge, switch between absolute and delta-vs-baseline views, and toggle between embed-only, reranked, or all configurations.