Temperature and top-p are settings that shape how a language model picks its next token from the probabilities it gives its whole vocabulary. We tried them on a made-up curve: a Zipf law with exponent 1 over 50,000 tokens, which is no model's output. With top-k off, top-p 0.9 keeps 15,996 of those tokens, about a third of the vocabulary. Holtzman and colleagues, who studied GPT-2, describe the nucleus as a small subset that "tends to range between one and a thousand candidates". The gap says more about our curve than about the method. On the paper's account, a model's distribution at one step is far more concentrated than a law fitted to a whole corpus, and top-p, the smallest set of tokens whose probabilities reach p, shrinks to match.
Holtzman's paper supplies the definitions both runtimes deal from
The paper is "The Curious Case of Neural Text Degeneration" by Holtzman, Buys, Du, Forbes and Choi (ICLR 2020). Its equation 2 defines the top-p vocabulary as "the smallest set such that" its probabilities sum to at least p. Top-k is "the set of size k" holding the most probability: exactly k tokens, with no rule for ties.
Equation 4 divides each logit by a temperature t before the softmax, an idea Holtzman credits to Ackley et al. (1985), which we could not open. The same form is equation 1 of Hinton, Vinyals and Dean's 2015 distillation paper, where T "is normally set to 1". It serves distillation there; no text is generated. The Hugging Face and llama.cpp source we read both place the top-p boundary where equation 2 does.
Temperature changes the odds on every token and discards none
Dividing logits by t never reorders tokens or zeroes one. On a Zipf curve the effect is plain algebra and no finding of ours: r^-s raised to 1/t is r^(-s/t), so temperature 0.8 turns an exponent of 1.0 into 1.25.
On our exponent-1 curve, entropy is 10.91 bits at temperature 1. It falls to 2.36 bits at 0.5 and climbs to 15.19 at 2, near the 15.61-bit ceiling for 50,000 equally likely tokens. The top token's share moves the other way, from 8.8% to 60.8% as temperature drops to 0.5.
A long made-up tail leaves thousands of tokens in the hand
Our nucleus counts on made-up Zipf curves over 50,000 tokens, temperature first, top-k off:
| Exponent, temperature | Kept at p 0.5 | Kept at p 0.9 | Kept at p 0.95 | Mass in top 40 |
|---|---|---|---|---|
| 0.8, 1.0 | 2,539 | 31,235 | 39,739 | 15.5% |
| 1.0, 1.0 | 168 | 15,996 | 28,281 | 37.5% |
| 1.2, 1.0 | 11 | 2,166 | 8,166 | 63.9% |
| 1.0, 0.7 | 3 | 107 | 460 | 84.4% |
| 1.2, 0.7 | 2 | 15 | 39 | 95.1% |
At exponent 1, the p 0.9 nucleus is 10,705 tokens of a 32,000-token vocabulary and 55,701 of 200,000. When s is 1 or less, the sum of r^-s has no finite limit as the vocabulary grows, so the nucleus keeps growing with it. At exponent 1.2 the same range only moves the count from 1,739 to 4,019.
None of this contradicts Holtzman's one to a thousand. The paper's Zipf coefficient of 0.93 for human text counts words across whole texts, and its Figure 5 shows single steps both flat and sharply peaked.
Hugging Face and llama.cpp play the same settings in a different order
Hugging Face applies temperature before the cuts; llama.cpp's default chain applies it after. On the explainer's eight-token example at temperature 2 and top-p 0.36, Hugging Face's order keeps "bear" and "dog" at about 55% and 45%, and llama.cpp's keeps "bear" alone. On our arithmetic the two final distributions differ by 45% of their probability mass.
Sweeping that example over temperature 0.05 to 2.00 in steps of 0.05 and top-p 0.01 to 0.99 in steps of 0.01, top-k and min-p off, the orders kept different counts in 1,799 of the 3,960 cells, or 45.4% of this grid only, and the median mass gap among those was 13.3%. The widest count gap, 1 token against 7 at temperature 0.05 and top-p 0.97, moves only 0.03% of the mass, because almost all of it sits on "bear".
On the exponent-1 Zipf curve at top-p 0.95, llama.cpp's order keeps 28,281 tokens at any temperature and Hugging Face's keeps 460 at 0.7, a 61-fold gap; the extra 27,821 tokens hold 4.8% of the probability. Ties differ too: Hugging Face's top-k keeps every token tied with the k-th, and llama.cpp keeps exactly k.
Min-p makes the third cut in llama.cpp's default chain
llama.cpp's common.h at commit a691b23 (8 November 2026) sets top-k 40, top-p 0.95, min-p 0.05 and temperature 0.80, run in that order. Min-p, as Nguyen and colleagues define it (2024), uses "the top token's probability as a scaling factor": a token stays if it holds at least 5% of the leader's share. Temperature comes fourth and removes nothing.
The cuts see the untempered curve, so temperature leaves these counts alone. Top-k leaves 40 tokens, and top-p 0.95 over those 40 trims them to between 29 and 35 for exponents 0.8 to 1.2. Min-p binds from exponent 0.9 up, leaving 20 survivors at exponent 1 and 12 at 1.2. It stops at rank 20^(1/s), whatever the vocabulary size.
This explainer lays both orders face up, min-p included
The temperature and top-p explainer puts both orders on a switch. Until a revision dated 8 November 2026, its llama.cpp mode left out the default min-p and kept ties at top-k as Hugging Face does. Its paste box silently read "dog 2,5" as a token "dog 2" with logit 5, and its Hugging Face row did not say that sampling is off by default.
The revision adds min-p to both orders, keeps exactly k in llama.cpp's, and refuses a comma between digits. Choose "One confident token, nineteen weak" and load llama.cpp's defaults: top-p 0.95 alone would keep 18 of the 20 tokens, and min-p 0.05 cuts them to 1. For the order case, pick "The big brown …" at temperature 2 and top-p 0.36.
House rules for a default change with the library, the model file and the API version
In transformers' generation config at commit a6f5c86 (7 November 2026 UTC), do_sample defaults to False, so the library decodes greedily and temperature does nothing until sampling is on. Then top-k falls back to 50 unless the model's own generation config sets a value, and top-p defaults to 1.0, which is off.
OpenAI's published API specification gives temperature 0 to 2, says of top_p "We generally recommend altering this or temperature but not both", and has no top-k field. Google's generateContent reference marks all three as "silently ignored on Gemini 3.6 Flash and newer models". Eight Chinese vendors' rules are in our guide to prompting Chinese language models.
Anthropic's migration guide says API version 2026-10-21 removes temperature, top_p and top_k, and a request carrying any of them gets an error even at the default value. Requests kept on version 2023-06-01 can still send non-default values to models that accept them, such as Claude Sonnet 4.6; Claude 4.7 and later accept only the defaults. Anthropic's glossary says: "Even with temperature set to 0, the results will not be fully deterministic." This guide was drafted with a model made by Anthropic, whose API is one of those described here.
View author profile → · Editorial policy
