Sampling

From logits to a choice

The model produces logits — one unnormalised score per token in the vocabulary. Softmax turns them into probabilities. The sampling parameters then cut that distribution down before a choice is made. Every knob you have acts on this one step, and nothing else.

logits        [ 2.0   1.0   0.0 ]        raw scores, any real number
  ↓ softmax(t)
probabilities [ 0.665 0.245 0.090 ]      sums to 1
  ↓ top-k / top-p
truncated     [ 0.625 0.375 0.000 ]      renormalised over what is left
  ↓ pick a random point
token

Temperature divides the logits

Temperature never changes the order of the candidates — it changes how much the leaders are favoured. Above about 1.2 the tail becomes reachable, and the tail is where the incoherence lives.

Temperature 0 is the right default for anything you intend to parse. It is usually called "deterministic", and in practice it is nearly so: batching and floating-point non-associativity on the provider’s hardware can still change a result between identical calls. Do not build a cache key on the assumption that it cannot.

softmax(logits.map((x) => x / temperature))

Top-k and top-p cut the tail

Top-k keeps a fixed number. Its weakness is that it is blind to shape: when the model is certain, 40 candidates drags in 39 bad ones; when it is genuinely uncertain, 40 may cut off good ones.

Top-p (nucleus) keeps a fixed mass, so the number of candidates adapts. On [0.9, 0.05, 0.05] at p = 0.9 it keeps one; on [0.34, 0.33, 0.33] at the same p it keeps all three. That is why it is the usual recommendation, and why most providers default to it.

They compose, and the common advice is to change one of them, not both — temperature or top-p. Two overlapping truncations are hard to reason about and easy to get into a state where nothing much is left.

topK(probs, 40)     // keep the 40 most likely, renormalise
topP(probs, 0.9)    // keep the smallest set that sums to 0.9, renormalise

What to set, by task

Two more that matter: max_tokens is a hard cut, not a hint — a reply truncated at the limit is not shorter, it is unfinished, and unfinished JSON does not parse. And stop sequences end generation when a string appears; they are checked on the decoded text, so a stop sequence that falls inside a token can behave unexpectedly.

Sampling is not reasoning

A model that gets an answer wrong at temperature 1 will often get it right on a second sample, and that is sometimes read as the model "knowing" it. What it means is narrower: the correct answer had some probability and the sampler did not pick it.

That is a real property to exploit — sample several, take the majority, which is what self-consistency does — but it is a technique with a cost, not evidence of hidden competence. And it is worth remembering that the fix for an unreliable format is temperature 0 and a schema, not more samples.