Sampling Parameters Cheat Sheet

How a language model turns probabilities into words

A large language model does not "decide" what to say in one step. At every position it produces a probability distribution over its entire vocabulary — tens of thousands of possible next tokens, each with a score. Sampling parameters (also called decoding or generation parameters) are the controls that turn that raw distribution into a single chosen token, over and over, until the response is complete. They do not change what the model knows; they change how it picks from what it knows. That is why two requests with the identical prompt can produce a dry, deterministic answer or a wildly inventive one — the difference is entirely in the sampling settings.

The most familiar control is temperature, which stretches or flattens the distribution. Low temperature sharpens it so the top token wins almost every time, giving focused, repeatable output ideal for code and extraction. High temperature flattens it, letting unlikely tokens surface, which is what you want for brainstorming and fiction. Sitting alongside temperature are the truncation methods: top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p, while top-k keeps a fixed number of the most likely candidates. Min-p is a newer, adaptive cousin that scales its cutoff to the model's confidence. These all answer the same question — which tokens are even allowed to be picked — but with different shapes of cutoff, which is why mixing several aggressively can fight each other.

Penalties, stops, seeds and the rest of the family

Beyond choosing tokens, a second group of parameters manages repetition and control. The frequency penalty scales down a token the more often it has already appeared; the presence penalty applies a flat nudge to anything used even once, pushing the model toward new topics; and the multiplicative repetition penalty, common in open-source stacks, discourages reusing recent tokens. Used gently these cure loops and verbatim echoing; pushed too hard they wreck grammar by forcing awkward synonyms. Stop sequences end generation cleanly when a chosen string appears — invaluable for keeping structured or turn-based output from bleeding into the next role label or code fence — and max tokens caps the length of the answer, bounding both cost and latency.

Rounding out the family are the reproducibility and steering controls. A fixed seed pins the random number generator so the same inputs reproduce the same output, which is essential when you want prompt differences — not randomness — to explain a change in results. Logit bias lets you nudge specific tokens up or down, even banning a word outright or forcing a fixed vocabulary. And n asks the API for several independent completions at once, so you can generate a handful of drafts and pick the best. None of these settings is universally "correct": the right combination depends on whether you value precision or creativity, speed or thoroughness, repeatability or variety. The cheat sheet above lays each one out so you can pick deliberately instead of leaving the defaults to chance — and because every parameter interacts with the others, understanding them as a family is what separates guesswork from genuine control over a model's output.