Sampling and determinism

A prompt with its instructions, its demonstrations and its format all settled will still return two different answers when it is sent twice. That looks like a defect and is in fact a deliberate setting, so it can be configured, and the configuration is one of the few genuine dials a team gets.

The randomness enters at one line of the generation loop. The model produces a probability for every entry in its vocabulary, and something then has to pick one of them. Taking the likeliest candidate every time is one option, and it has a name, which is greedy decoding. Drawing from the distribution instead is the other option, and it is the default on almost every interface, because text produced by always taking the likeliest token becomes repetitive very quickly. Ari Holtzman and colleagues documented that failure in a paper submitted on 22 April 2019 and proposed the truncation method most systems still use.

Anybody testing a feature built on a model runs into this on the first day. A prompt change looks like an improvement across five examples and the same five examples disagree an hour later. Until the sampling settings are pinned and understood, no comparison between two prompts means anything.

The sections below locate the exact step where the draw happens and work through what temperature does to the numbers. They then set out how top k and top p cut the distribution before the draw, give the settings each kind of task takes, and finish with the reason a fixed seed falls short of reproducibility.

The one line where randomness enters

Generation produces a vector of scores over the vocabulary at every step, and the softmax turns those scores into probabilities that add to one. Sampling is everything that happens between that distribution and the single integer appended to the sequence.

import math

def apply_temperature(probs, t):
    """Temperature divides the logits by t before the softmax, which is the
    same as raising each probability to the power 1/t and renormalising."""
    if t == 0:
        out = [0.0] * len(probs)
        out[probs.index(max(probs))] = 1.0   # greedy decoding
        return out
    raised = [p ** (1.0 / t) for p in probs]
    total = sum(raised)
    return [r / total for r in raised]

def nucleus(probs, p):
    """Keep the smallest set of tokens whose probabilities reach p, then
    renormalise over that set and draw from it."""
    order = sorted(range(len(probs)), key=lambda i: -probs[i])
    kept, running = [], 0.0
    for i in order:
        kept.append(i)
        running += probs[i]
        if running >= p:
            break
    total = sum(probs[i] for i in kept)
    return {i: probs[i] / total for i in kept}

Three things follow from those eleven lines. Temperature changes every number in the distribution and removes nothing, whereas top p and top k remove candidates outright and leave the survivors in the same relative proportions. Temperature zero is then a special case in the code, since dividing by zero has no meaning and the limit is the argmax.

Temperature and the shape of the distribution

Suppose that at one step the four leading candidates carry the probabilities in the first column below. Rescaling by temperature raises each probability to the power of one over the temperature and then divides by the new total.

Each column adds to 1.00, because a rescaling of a distribution is still a distribution. Halving the temperature squares the probabilities, which pushes the leader from half the mass to nearly three quarters of it and leaves the fourth candidate at three chances in a hundred. Doubling the temperature takes the square root of each one, which lifts the fourth candidate from one chance in ten to nearly one in six.

That last number is the honest description of a high temperature. The setting adds no ideas, because all it can do is make the tokens the model rated as unlikely reachable, and the model rated them unlikely for a mixture of reasons. Some of them were unusual phrasings, some were facts it half knows, and some were simply wrong.

Top k and top p as truncation

Temperature leaves the tail in place at a smaller weight. Truncation removes the tail before the draw, and two methods are in common use.

Top k keeps the k likeliest candidates and renormalises across them. At k equal to two, the table above keeps the first two candidates and redistributes their weights to 0.67 and 0.33. The weakness is that k is a fixed count applied to a distribution whose shape changes at every step. At a step where the model is certain, k of 40 admits 39 candidates that should never have been in the draw.

Top p, also called nucleus sampling, keeps the smallest set of candidates whose probabilities reach p. At p equal to 0.9 the temperature 1 column keeps exactly three candidates, since 0.50 plus 0.25 plus 0.15 reaches 0.90 on the nose and the fourth is cut. The set adapts to the step, which is why it is the default on most interfaces.

The two settings interact, and the table shows how. At temperature 2, a top p of 0.9 keeps all four candidates, because the first three now reach only 0.83 between them. Raising the temperature therefore widens the nucleus as well as flattening it, so a team that moves both at once has changed the shape of the draw twice and can attribute the result to neither. Moving one and leaving the other alone is the standard advice, and the arithmetic above is the reason for it.

The settings each kind of task takes

The question behind the settings is always the same. What consumes this output, and what does a second valid phrasing cost that consumer.

Written as configuration, three of those rows look like this.

{
  "extraction": {
    "temperature": 0,
    "top_p": 1,
    "max_tokens": 256,
    "stop": ["\n\n"],
    "note": "a parser reads this, so a second phrasing is a defect"
  },
  "grounded_answer": {
    "temperature": 0.3,
    "top_p": 0.9,
    "max_tokens": 700,
    "note": "a person reads this, and it must stay on the passages"
  },
  "draft_variants": {
    "temperature": 0.9,
    "top_p": 0.95,
    "n": 5,
    "max_tokens": 300,
    "note": "five options a person picks between"
  }
}

Two fields in there are easy to skip over and cause real incidents. The maximum token count truncates, so an answer that stops in the middle of a sentence usually means the cap was set too low and never that the model had finished. The stop sequence ends generation on a literal string, which is the cheapest way to stop a model from writing a second helpful paragraph after the JSON object the parser wanted.

Why a fixed seed does not make a call reproducible

Setting temperature to zero takes the likeliest token at every step, which should produce identical output from identical input. It does not, and the reason was set out in detail in work published on 10 September 2025 by Horace He and colleagues at Thinking Machines.

They sampled 1,000 completions of 1,000 tokens each from one open model at temperature zero, using the same prompt every time. The result was 80 different completions, with the most common of them appearing 78 times. Every one of the 1,000 was identical for the first 102 tokens and they began to separate at token 103.

The cause is arithmetic, not hardware randomness. Floating point addition is not associative, so summing the same numbers in a different order can change the final bit. Inference servers batch incoming requests together, and the kernels that do the summing choose their reduction order according to the size of that batch. A request therefore gets slightly different numbers depending on how many other people happened to be asking at the same moment, and one changed bit is enough to flip the argmax at a step where two candidates were close. The team's fix was to make three operations produce the same result at any batch size, those being the normalisation, the matrix multiplication and the attention, after which the 1,000 completions came back identical.

Almost no production system runs on kernels like that today. The consequence for a team is a practical one. A seed and a temperature of zero narrow the variation and do not remove it, so any test that asserts an exact string will go red on a day nothing changed. Tests on a model have to assert a property of the answer, and a comparison between two prompts has to run enough cases that a few flipped tokens cannot move the verdict. Module five is built on that fact.

Some prompt shapes survive this variability better than others, because they ask the model for something with fewer near ties in it. Those shapes are the subject of the next page.