Temperature & Top-p Explainer

From logits to a probability distribution

Every time a large language model predicts the next token, it doesn't pick a word directly — it produces a logit (a raw, unnormalised score) for every token in its vocabulary. Those logits are turned into a probability distribution by the softmax function, which exponentiates each score and divides by the total so everything sums to one. The example in this tool uses a small, fixed set of eight candidates so you can see the whole distribution; a real model does the same across a vocabulary of tens or hundreds of thousands of tokens. Once you have probabilities, the question becomes how to choose, and temperature, top-p and top-k are the three settings most runtimes have exposed for it.

Temperature divides every logit by the temperature value before the softmax: p = exp(u / t) ÷ Σ exp(u / t), the form Holtzman and colleagues give in the paper that introduced nucleus sampling. When temperature is high (toward 2), the gaps between logits shrink, the softmax output flattens, and unlikely tokens get a real chance of being chosen. When temperature is low, the gaps stretch and probability piles onto the highest-scoring token; the same paper notes that lowering temperature improves quality at the cost of diversity, and in its own experiments temperatures below 0.9 sharply increased repetition. As temperature approaches 0 you get greedy decoding, where the model always takes its single best guess, which is why this tool treats 0 as a pure argmax. Temperature never adds or removes candidates and never changes their ranking — it only changes how peaked or flat the curve is.

Truncation: top-p (nucleus) and top-k sampling

Temperature alone has a weakness: even at a sensible setting, the long tail of very unlikely tokens still carries a sliver of probability, and occasionally the model picks genuine nonsense. Top-k and top-p truncate the distribution before sampling. Top-k sorts the tokens by probability and keeps only the k highest. Top-p, or nucleus sampling, sorts the same way, walks down the list adding up probability, and stops at the smallest set whose total reaches p. Holtzman and colleagues proposed it because a fixed k misjudges both kinds of step: on a flat distribution a small k cuts off reasonable words, and on a peaked one a large k lets in junk. On a confident step top-p may keep one or two tokens; on an uncertain one it keeps many. After the cut, the kept tokens are renormalised so their probabilities sum back to 100%.

The order is not standard

Runtimes chain these steps differently, and the order changes which tokens survive. Hugging Face transformers applies temperature first, then top-k, then top-p, and its top-p step measures probability over whatever top-k left. llama.cpp's default chain runs top-k and top-p first, on the untempered distribution, and applies temperature last. Try temperature 2 and top-p 0.5 on the example and switch the order: the cut is measured on a different curve. OpenAI and Mistral, for their part, recommend changing temperature or top-p but not both, while llama.cpp ships all three active by default (top-k 40, top-p 0.95, temperature 0.8).

Which APIs still accept these settings

Read from each vendor's own API reference on 3 November 2026. The dials are being withdrawn on several hosted models, so check the current reference before relying on one.

The creativity-versus-determinism trade-off still holds where the dials exist: higher temperature and a looser top-p for brainstorming or varied copy, lower temperature for code and extraction. Low temperature makes output more consistent, not guaranteed: Anthropic's glossary says that even at temperature 0, identical inputs can produce different outputs across API calls.

Disclosure: RECATOOLS is written with the assistance of Claude, made by Anthropic, whose API is described on this page. Readers should weigh that interest.