Temperature, top p and top k control how a language model selects tokens during generation. They change the variation in possible responses. They do not directly measure factual accuracy, reasoning quality or confidence.
The available settings differ by model and API. Some models restrict sampling controls or expose a reasoning setting separately. Check the selected interface before copying a configuration from another model.
How temperature changes token probabilities
Temperature rescales token scores before selection. Lower positive values concentrate probability on higher scoring candidates; higher values flatten the distribution. Temperature zero is commonly implemented as greedy selection of the highest scoring token.
Consider a simplified distribution containing only four tokens:
| Candidate | Temperature 1 | Temperature 0.5 | Temperature 2 |
|---|---|---|---|
| A | 0.50 | 0.72 | 0.37 |
| B | 0.25 | 0.18 | 0.26 |
| C | 0.15 | 0.07 | 0.20 |
| D | 0.10 | 0.03 | 0.17 |
The rounded values illustrate the effect. At a higher temperature, D becomes more likely even though the model originally assigned it the lowest score. That can produce useful variation or introduce errors, depending on the task.
A lower temperature is often a reasonable experiment for extraction or classification. For writing alternatives, a higher value may produce more varied drafts. Neither is a universal optimum, and code generation does not become invalid merely because temperature is above zero.
Top p and top k sampling
Top k sampling keeps a fixed number of the highest scoring tokens. With k equal to two in the temperature one column, only A and B remain eligible.
Top p sampling, also called nucleus sampling, keeps the smallest leading set whose cumulative probability reaches a threshold. With p equal to 0.9 in the same example, A, B and C remain.
The nucleus sampling paper examined how truncating the probability distribution affects generated text. Its results concern generation methods and evaluated workloads; they are not a guarantee for a specific business task.
Temperature and truncation interact. Flattening a distribution can increase the number of tokens needed to reach a top p threshold. Change one control at a time during an experiment so the result remains interpretable.
Why seeds and temperature zero have limits
A seed can make the sampling process repeatable under controlled conditions. It cannot freeze the model weights, request routing, hardware, software or retrieved context.
Even greedy decoding can vary when small numerical differences change which of two close scores is largest. Thinking Machines Lab demonstrated the role of batching and numerical computation in Defeating Nondeterminism in LLM Inference. Their work also shows that stronger determinism is possible with deliberate inference engineering.
Record the model identifier, prompt version, settings and relevant context when comparing runs. A matching seed is useful metadata, not proof that two requests were equivalent.
Output limits and stop conditions
A maximum output setting caps generation according to the interface's accounting rules. A response can reach that limit halfway through a sentence or JSON object. Some reasoning models also account for internal reasoning within an output related budget.
Inspect the response status or finish reason before parsing content. Treat truncation as an incomplete response.
A stop sequence ends generation when a specified sequence appears, if the interface supports it. This can be useful for a deliberate text protocol, but a generic newline stop can cut off valid JSON or a legitimate multi paragraph answer. Prefer an explicit schema or completion status when appropriate.
A controlled sampling experiment
Use the same set of inputs for each candidate configuration. Include difficult cases and run each case more than once when variation matters.
For each configuration, record task success, invalid output rate, response length, cost and latency. For a classifier, compare labels against known answers. For a drafting tool, use a rubric that assesses usefulness as well as diversity.
Regression tests should assert the properties the application needs. Exact equality is appropriate for a fixed label or verified field. Open ended prose usually needs a more flexible comparison. Select settings from those results and revisit them when the model or task changes.