Also known as: nucleus sampling, top-p, top_p
Top-p sampling restricts each step's sampling to the smallest set of tokens whose cumulative probability is at least p. Unlike top-k's fixed cutoff, the nucleus adapts to the distribution's shape.
Top-p sampling — also called nucleus sampling — is a token-selection strategy that, at each step, samples only from the smallest set of tokens whose cumulative probability sums to at least
Given the post-temperature softmax distribution
The size of the nucleus (the top
The key observation is that low-probability tokens contribute disproportionately to incoherent generation but only marginally to perplexity. The bottom 99% of tokens by probability rank contribute essentially nothing meaningful to a coherent next-token guess, but at high temperature their cumulative mass can still be a few percent — enough to occasionally sample one and produce nonsense. Sorting by probability and clipping at cumulative mass
Top-k sampling restricts to the
Top-p adapts. At the confident step, the nucleus is 1-2 tokens. At the uncertain step, the nucleus might be 200. The cutoff where cumulative probability accumulates is a more meaningful boundary than a fixed token count.
Empirically, top-p outperforms top-k on most text-quality and perplexity-on-held-out metrics. Most modern systems either use top-p alone or top-p combined with a generous top-k cap as a backstop.
The interaction with temperature matters: at low temperature the distribution is already peaky and top-p has little to do; at high temperature top-p is what prevents the flattened tail from polluting output. The standard production combination is moderate temperature (T ~ 0.7-1.0) and moderate top-p (~0.9). Pushing both knobs simultaneously into extreme territory rarely improves anything.
For tasks consumed by code — structured extraction, code generation, factual QA — temperature 0 is the correct choice and top-p is irrelevant. The whole nucleus-sampling apparatus exists to support diversity in human-facing generation; it has no role when you want a single deterministic answer.
For self-consistency ensembling and similar techniques that want multiple diverse samples, you typically want top-p around 0.9-0.95 with temperature around 0.7. The goal is samples that explore different reasoning paths without veering into incoherence.
Top-p is the universal default; top-k still ships as a knob but is rarely the primary control. Newer techniques (typical sampling, min-p, eta sampling) haven’t displaced it — top-p is good enough, every API exposes it, and the marginal improvement isn’t worth a migration. Boring-but-correct is the right place for the default to be.
Top-k uses a fixed number of tokens (say 40), regardless of how peaked the distribution is. When the model is highly confident, k=40 includes lots of tail garbage; when uncertain, k=40 might cut off legitimate alternatives. Top-p adapts: it includes 2 tokens when the top two are 99% of the mass, and 200 when the distribution is flat. The adaptive cutoff is a more principled match to the underlying distribution shape.
0.9 is the canonical default. 0.95 if you want slightly more diversity, 0.8 if you want tighter constraint. Below 0.7 it starts to feel close to greedy on confident-distribution tokens; above 0.99 it's essentially unconstrained sampling. The right value depends on temperature — at low T the distribution is already peaky and top-p has less work to do; at high T top-p is what stops the long tail from polluting output.
Sometimes — it's a belt-and-suspenders approach. Top-k caps the maximum number of tokens considered (defends against pathological flat distributions), top-p caps the cumulative probability (defends against the long tail). Using both with sensible values (k = 50, p = 0.9) works fine and is a common default. The marginal gain over top-p alone is small.
