Also known as: multi-stage reranker, tiered reranking, ranking cascade
A cascade reranker stacks multiple rerankers from cheap-and-fast to expensive-and-accurate, with each stage filtering candidates before passing a smaller set to the next.
A cascade reranker is a pipeline of two or more rerankers arranged cheapest to most expensive, where each stage prunes candidates before the next. The economic logic: a cross-encoder is too slow on 1000 candidates per query but plenty fast on 100, and a bi-encoder rerank can knock the set from 1000 to 100 cheaply. Stack them and you get cross-encoder precision at a fraction of the cost.
The default production shape is first-pass retrieval (BM25 or dense ) returning ~500 candidates, followed by a cross-encoder reranker reducing to ~20-50. This already counts as a cascade — first-pass plays the role of cheap stage, cross-encoder is the precision stage. It’s how essentially every modern RAG pipeline ships.
Adding a third stage (a bi-encoder rerank between first-pass and cross-encoder) is worth it when first-pass surfaces 1000+ candidates and the cross-encoder budget tops out around 100. The bi-encoder rerank is fast — milliseconds per 1000 candidates — and a well-trained one preserves nearly all the relevant documents.
You could conceptually run one giant reranker on every candidate. The problem is that cross-encoder latency scales linearly in candidate count. At 50ms per pair, 500 pairs is 25 seconds — unshippable. The cascade exploits the fact that most candidates are obvious negatives that a cheap model can filter, and only the hard cases need the expensive model’s discrimination.
Three forces: upstream recall (you want it high), downstream throughput (capacity ceiling), and downstream marginal accuracy (does seeing more candidates actually improve final ranking?).
Procedure: take a labeled eval set of 100-500 queries. For each candidate count
Common cut-offs we see in production:
When stages are independent models, their score scales aren’t comparable. Fusing them via reciprocal rank fusion sidesteps this; fusing via weighted sum requires calibrated scores on each stage. Models like zerank-2 emit calibrated probabilities specifically so cascades downstream of them compose cleanly.
The other consequence: instruction-following rerankers like the instruction-following reranker are usually placed at the precision stage of the cascade, where the smaller candidate set means each instruction-conditioned forward pass is affordable. Putting an instruction-following model at stage 1 burns budget on candidates that any reranker would reject.
For most production stacks: first-pass + a single calibrated cross-encoder. Add a bi-encoder mid-stage only when first-pass recall is wide (1000+) and the cross-encoder’s per-pair cost is high. Skip three stages if two get you there — every stage is one more recall risk and one more thing to monitor.
Two is the common shape — first-pass to ~500, cross-encoder to ~50, LLM picks final. Three stages add a cheap bi-encoder rerank (BM25 to 1000, bi-encoder to 200, cross-encoder to 30) when first-pass recall is wide. Four-plus rarely pays off — each stage adds latency and a recall risk.
Tune by measuring recall@k of the upstream stage and accuracy@k of the downstream stage on your eval set. Pick the smallest upstream k where recall plateaus, then verify the downstream model has capacity to rerank that many candidates within latency budget. Typical: 1000 to 100 to 10.
Yes — if you fuse scores across stages (e.g., reciprocal rank fusion or weighted sum), uncalibrated stage outputs make the fusion arbitrary. Calibrated rerankers like zerank-2 produce probability-shaped scores that compose cleanly across stages.
