Temperature & Top-p Explainer

From logits to a probability distribution

Every time a large language model predicts the next token, it doesn't pick a word directly — it produces a logit (a raw, unnormalised score) for every token in its vocabulary. Those logits are turned into a probability distribution by the softmax function, which exponentiates each score and divides by the total so everything sums to one. The example in this tool uses a small, fixed set of eight candidates so you can actually see the whole distribution; a real model does the same thing across tens of thousands of tokens at once. Once you have probabilities, the question becomes: how do you choose? Sampling parameters are the answer, and the three most important are temperature, top-p and top-k.

Temperature is the first dial, and it acts before the softmax by dividing every logit by the temperature value. When temperature is high (toward 2.0), the gaps between logits shrink, the softmax output flattens, and unlikely tokens get a real shot at being chosen — the model feels adventurous, surprising, sometimes incoherent. When temperature is low (toward 0.0), the gaps stretch, probability mass piles onto the single highest-scoring token, and output becomes repetitive but safe. At the limit of temperature 0 you get greedy decoding: the model always takes its single best guess, which is why this tool treats that case as a pure argmax. Temperature doesn't add or remove candidates — it only changes how peaked or flat the curve is.

Truncation: top-p (nucleus) and top-k sampling

Temperature alone has a weakness: even at a sensible setting, the long tail of very unlikely tokens still carries a sliver of probability, and occasionally the model rolls the dice and picks genuine nonsense. Top-k and top-p fix this by truncating the distribution before sampling. Top-k is the blunt version: sort the tokens by probability and keep only the k highest, discarding everything else. It's simple and predictable, but a fixed k can be too generous when the model is confident and too stingy when it's uncertain. Top-p, also called nucleus sampling, is the adaptive version: walk down the sorted tokens, accumulating probability, and stop as soon as the running total reaches p. On a confident step where one token holds 95% of the mass, top-p might keep just one or two tokens; on an ambiguous step where the mass is spread thin, it keeps many. The size of the surviving “nucleus” adjusts to how sure the model is — which is exactly why nucleus sampling became the default for high-quality open-ended text.

After top-k or top-p chops off the losers, the kept tokens are renormalised so their probabilities sum back to 100% — the model never samples from a truncated set that adds up to less than one. In practice these knobs stack: you set a temperature for overall flavour, then a top-p (and sometimes a top-k cap) to fence off the garbage tail. The classic trade-off is creativity versus determinism. Crank temperature up and loosen top-p for brainstorming, fiction, or varied marketing copy; pull temperature down and tighten top-p for code, extraction, factual answers, or anything where you want the same input to give a reliable, repeatable output. There's no universally correct setting — it depends entirely on whether you want the model to surprise you or to behave. Drag the sliders above and you can watch every one of these effects happen in real time, on a distribution small enough to read at a glance.