An LLM context window is the amount of tokenised information a model can handle within an invocation. Instructions, user input, documents, history and tool messages may all use that capacity. Generated output and internal reasoning can also count against related limits, depending on the model and interface.
A token budget allocates capacity before a request is sent. It prevents essential information from being displaced and makes growth in input size visible.
Context size and output limits are different controls
Model interfaces may specify a combined context limit, a maximum input length and a separate maximum output length. Read the selected model's rules rather than assuming that an advertised window can be filled entirely with documents.
For a simple shared window design, the planning condition is:
input tokens + reserved output tokens + safety margin <= context limit
That formula is a starting point. Message wrappers, tool schemas, images and reasoning related tokens may affect accounting. Use the provider's counting facilities or the appropriate tokenizer and compare estimates with reported usage.
Exceeding a limit can cause request rejection or an interface specific truncation behaviour. Reaching an output cap can produce an incomplete answer. Neither should be treated as a successful response.
A worked token budget
The following figures are hypothetical allowances for an assistant with a 128,000 token window. The application chooses a smaller working budget to control cost and response time.
| Component | Token allowance |
|---|---|
| Instructions and policy rules | 5,000 |
| Tool definitions | 4,500 |
| Few shot examples | 5,500 |
| Conversation history | 12,000 |
| Retrieved passages | 12,000 |
| Tool results | 4,000 |
| Current request | 1,000 |
| Output reserve | 6,000 |
| Total | 50,000 |
The remaining capacity is headroom, not a target to fill. A short request should use less than its allowance. A long uploaded document may need a different workflow or a larger authorised budget.
Record actual consumption by component. If history grows from 2,400 to 21,000 tokens, a single total does not explain the increase; a component breakdown does.
The same budget can be viewed as a capacity allocation. The bar is drawn to scale for this hypothetical shared window. Its labels separate the input already planned from the space reserved for the answer.
Handle budget overruns according to meaning
Different components need different overflow rules. Low ranking duplicate passages can be removed. Older history can be summarised while retaining corrections and commitments. Tool output can be paginated or reduced to selected fields.
Mandatory instructions require special treatment. If they exceed an approved allowance, stop or use a defined fallback while the configuration is reviewed. Silently cutting them can remove constraints the application depends on.
A generated summary must be counted again. An instruction to “summarise in 500 tokens” is not a reliable size check, and a short summary can still omit a required fact.
Reserve enough capacity for completion
Suppose a feature allows 1,500 output tokens for a short reply. After a model change, that allowance may also need to accommodate internal reasoning under the new interface's rules. The visible answer can become shorter or incomplete even though its intended length has not changed.
Inspect completion status, output usage and reasoning usage where reported. Increase limits only after identifying the cause and checking cost. A larger cap does not repair a loop or an unnecessarily verbose prompt.
Distinguish capacity from useful context length
A request can fit within the context limit and still perform poorly. Lost in the Middle found position dependent performance on the evaluated long context tasks. The practical implication is to test the intended workload at realistic lengths and placements.
Longer input can also increase cost and processing time, although caching and serving details affect the relationship. For example, 48,800 input tokens across 10,000 daily requests means 488 million input tokens. At 11,600 per request, the same traffic uses 116 million. The bill depends on applicable rates and cache behaviour.
Choose the working budget using quality evaluations and cost and latency measurements. Capacity alone cannot determine the right amount of context.