Also known as: speculative sampling, draft-and-verify, Medusa decoding
Use a small 'draft' model to predict the next several tokens, then have the big 'target' model verify them in a single forward pass. The standard latency-reduction trick for LLM inference — typically 2-4× faster generation at the same output quality.
Speculative decoding (Leviathan et al., 2023; concurrently Chen et al.) is the standard latency-reduction trick for LLM inference. The intuition: most tokens an LLM generates are easy — common syntactic glue, predictable continuations of established context. A much smaller, much faster model can predict them correctly most of the time. So: have the small “draft” model propose the next several tokens, and have the big “target” model verify them in a single forward pass that would have produced just one token.
If the draft is right on
The verification is statistically exact. Either:
You’re not approximating the target; you’re running it, just more efficiently when the easy case applies. Speculation never changes the output distribution.
Speculative decoding buys speed by exploiting the asymmetry between proposing and verifying tokens — like a code reviewer skimming a draft from a junior, only stopping to rewrite the parts that are wrong.
Speedup
Latency, mostly. Speculative decoding at batch 1 takes a 70B model’s per-token latency from ~30ms to ~10ms — the difference between feeling slow and feeling responsive in an interactive chat. For batched serving, the win is smaller (continuous batching already pushes throughput high) but speculation still reduces tail latency for low-batch traffic regimes.
Modern serving stacks (vLLM, TensorRT-LLM, SGLang) ship speculative decoding as a built-in option. The trend is toward “self-speculative” methods (Medusa, EAGLE) that avoid the separate-draft VRAM overhead — increasingly the default, with the KV cache management complexity that entails.
Speculation isn’t directly applicable to non-autoregressive models like cross-encoder rerankers — there’s no token-by-token decode loop. But the spirit — use a cheap proxy to filter, expensive verifier on survivors — is the structure of any retrieval pipeline (cheap first-pass retrieval → expensive reranker on top). Same engineering shape, different mechanism.
Decoding is memory-bandwidth-bound, not compute-bound, on modern GPUs serving large models. At batch size 1, the bottleneck is pulling the model’s weights from HBM into compute units — a 70B model at FP16 is 140GB, which dominates per-token latency at typical decoding throughput.
Verification, in contrast, runs all
This also explains why speculation breaks down at high batch sizes. With batch 32, the target’s forward pass is already pulling 32 sequences’ worth of work per weight load — it’s no longer memory-bandwidth-bound, and adding
A separate draft model has obvious overhead: extra VRAM (a 1B draft costs 2GB at FP16), an extra training pipeline, and ongoing maintenance to keep the draft aligned with the target. Self-speculative methods eliminate the separate model entirely.
Medusa (Cai et al., 2024) adds extra “Medusa heads” to the target model that predict tokens 2, 3, 4 ahead in parallel from the same residual stream. Training is a quick fine-tune (a few hundred GPU-hours) that doesn’t touch the original weights. Inference: one forward pass produces both the verified token and three speculative continuations.
EAGLE (Li et al., 2024) uses a single tiny autoregressive head — typically one transformer layer — that runs alongside the target. The head predicts the embedding of the next token (a regression target) rather than logits, then the target verifies in the next step. Higher acceptance than Medusa with smaller parameter overhead.
By 2025 these are the production default in vLLM, TensorRT-LLM, and SGLang. The maintenance cost of separate draft models killed the original speculative-decoding architecture for serious deployments — it remains useful as a research baseline and for serving setups where the draft model is already on hand.
Acceptance is exact: the verifier samples each token from the target model's true distribution, accepting the draft only when it would have produced the same token (or with calibrated probability). Output is statistically indistinguishable from greedy/sampled decoding from the target alone. Speculation buys speed; it doesn't change distribution.
2-4× for typical decode workloads with a 7B draft and a 70B target. The win depends on draft acceptance rate, which depends on how aligned the draft and target are. A draft trained via distillation from the target hits 70-80% acceptance; a generic small model hits 50-60%.
Variants that avoid the separate-draft-model overhead. Medusa adds extra heads to the target model that predict tokens 2, 3, 4 ahead in parallel; EAGLE uses a single shallow draft that runs alongside the target. Both eliminate the draft-model VRAM overhead at modest acceptance-rate cost. Increasingly the default in 2025+ serving stacks.
