Top-P Sampling (Nucleus Sampling)

Top-p sampling, also called nucleus sampling, restricts a model's next-token choice to the smallest set of candidates whose cumulative probability reaches a threshold p, then samples only from that set — instead of ever considering the full, long tail of unlikely tokens.

A model's probability distribution over its vocabulary always has a long tail: alongside a handful of genuinely plausible next tokens, there are tens of thousands of technically-possible-but-absurd ones, each with a tiny sliver of probability. Sampled naively, that tail occasionally gets picked and produces a token that derails the whole output. Top-p sampling's entire purpose is to make that tail unreachable.

How it works

At each generation step, sort every candidate token by probability, descending. Walk down the list, adding each token's probability to a running total, and stop the moment that total reaches or passes p. That stopped-at set, the "nucleus," is renormalized to sum to 1, and the next token is sampled only from it; everything outside is set to zero probability.

The mechanism is adaptive in a way a fixed-size cutoff is not. When the model is confident — one or two tokens hold almost all the probability mass — the nucleus is tiny, sometimes just one token, and output stays sharp. When the model is genuinely uncertain — probability spread thin across dozens of plausible continuations, common at the start of a creative sentence — the nucleus grows to include many more candidates. A related technique, top-k sampling, instead keeps a fixed number of top candidates regardless of confidence; top-p's adaptiveness is exactly what top-k lacks, which is why most production systems default to p over k, or combine both as a belt-and-suspenders cutoff.

Why it matters

Top-p sampling is usually paired with temperature rather than replacing it: temperature reshapes how peaked or flat the distribution is, and top-p then trims whatever tail remains after that reshaping. The combination gives two independent dials — how much the model favors its top guess, and how far into the tail sampling is even allowed to reach — which is why most API defaults ship both, commonly temperature around 0.7-1.0 with p around 0.9-0.95, rather than either alone.

A worked example

Suppose the sorted probabilities for the next token are 0.42, 0.23, 0.14, 0.09, 0.06, 0.04, and 0.02, summing to 1.0. With p = 0.9, walk down the list: 0.42, then 0.65, then 0.79, then 0.88 — still short of 0.9 after four tokens — then adding the fifth brings the running total to 0.94, which clears the threshold. The nucleus is those five tokens; the last two, worth 0.06 combined, are discarded entirely regardless of how the model's temperature setting shaped the rest of the distribution. Sampling now happens only inside that five-token, 94%-probability-mass nucleus — the model can still produce some variety, but it can no longer produce the near-nonsense sitting in the tail.

How Miatz teaches it

The same temperature lab in the AI Labs playground layers top-p on top of the sweep: learners watch the nucleus size shrink and grow across different prompts and settings, and complete a hand-computation drill, sorting probabilities and finding the cutoff by hand, before ever relying on an API parameter to do it for them.