A context window is a hard limit on how many tokens one call may carry, counting the input and the generated answer together. Every part of an assembled context is a claim against that limit, and the claims arrive from different systems with no awareness of each other. Fitting them together is the last step before a call goes out, and it decides what the model never gets to see.
Window sizes went from 2,048 tokens on the model that introduced few shot prompting in 2020 to a million on several models sold today. That is a factor of roughly five hundred in five years, and it changed which failure a team meets first. A window of 2,048 tokens produced an error, at a point in the code where somebody had to deal with it. A window of a million tokens produces an invoice.
The budget is therefore an engineering artefact with an owner, a number and a metric, in the same way a latency budget is. Teams that never write one still have a budget, and it is whatever the sum of the parts happened to be this morning.
The sections below name the claims that compete for one window, work a single assistant's budget twice at six months apart, cover the room an answer needs, and finish with what a longer window does and does not fix.
The claims that compete for one window
One call on a support assistant six months after launch. Conversation history and retrieved passages are seven parts in ten between them, and the customer's actual question is 300 tokens of the 50,300. The unused three fifths of the window are the reason nothing raised an error.
Seven claims compete, and each one grows for a different reason.
- System instructions and policy grow after every incident, because adding a line is the fastest available response to a complaint.
- Tool definitions grow with every new capability, since each tool brings a name, a description and a schema that all sit in the window.
- Few shot demonstrations grow because adding one more example is the cheapest answer to a misclassification.
- Conversation history grows by use alone, with nobody deciding anything.
- Retrieved passages grow when somebody raises the number of chunks kept after a miss, and the raise is rarely lowered again.
- Tool results grow when a tool returns a list and the list gets longer.
- The user's message is the only one nobody controls and the only one that stays small.
Every growth in that list was a locally correct decision made by somebody solving a real problem. None of them was a decision about the total, because the total had no owner.
One budget worked twice
The assistant behind the figure runs on a window of 128,000 tokens. Its budget at launch and its budget six months later look like this.
| Claim on the window | At launch | Six months later |
|---|---|---|
| System instructions and policy | 1,200 | 4,800 |
| Tool definitions | 1,600 for four tools | 4,400 for eleven tools |
| Few shot demonstrations | 1,800 for four | 5,400 for twelve |
| Conversation history | 2,400 | 21,000 across forty turns |
| Retrieved passages | 3,600, four at 900 | 10,800, twelve at 900 |
| Tool results this turn | 700 | 2,100 |
| The user's message | 300 | 300 |
| Input total | 11,600 | 48,800 |
| Reserved for the answer | 1,500 | 1,500 |
| Tokens on one call | 13,100 | 50,300 |
Nothing overflowed. The input grew by a factor of 4.2 and the window never came close to filling, since 50,300 of 128,000 is under two fifths. At ten thousand requests a day the same feature now sends 488 million input tokens a day where it used to send 116 million, on answers nobody has shown to be any better. Nothing in the system raised anything, because nothing in the system was watching a number that nobody had set.
A budget makes the number explicit and gives every claim an allowance.
# A budget with an owner, checked before every call goes out.
BUDGET = {
"system": 5_000,
"tools": 4_500,
"examples": 5_500,
"history": 12_000,
"retrieved": 12_000,
"tool_results": 4_000,
"question": 1_000,
"answer": 6_000,
} # 50,000 of a 128,000 window, chosen rather than accumulated
def fit(parts, budget=BUDGET):
"""Cut each part to its own allowance. Fail the call outright before
quietly shipping a context with the policy missing from it."""
out = {}
for name, text in parts.items():
allowance, cost = budget[name], count_tokens(text)
if cost <= allowance:
out[name] = text
elif name in ("history", "tool_results"):
out[name] = summarise_to(text, allowance)
elif name == "retrieved":
out[name] = drop_lowest_ranked(text, allowance)
else:
raise BudgetExceeded(name, cost, allowance)
emit_metric("context_tokens", sum(count_tokens(t) for t in out.values()))
return out
The final branch is the one that earns the function. A system prompt over its allowance is a defect in the prompt, so the call fails loudly and somebody fixes it. History and tool results are compressible, and retrieved passages are reselectable, which is why those three have their own treatment.
The metric on the last line is what makes any of this visible later. Written out per request, it turns a doubled bill from a mystery into a line somebody can read.
One record per call, measured before fit cuts anything.
request=7f21c9 window=128000
measured system 4812 tools 4398 examples 5402
history 20961 retrieved 10783
tool_results 2094 question 297
input 48747
budget history allowance 12000, over by 8961
action history summarised to 11840
sent input 39626 output 812 total 40438
The three numbers worth alerting on are in there. Input tokens per request shows the bill, the count of allowance breaches per hour shows which claim is growing, and the identity of the claim that breached, which is history on this call, names the part of the system somebody has to change.
The room the answer needs
Input and output share the window on most interfaces, so the space for the answer has to come out of the budget before anything else is assembled. A context packed to the last token leaves the model nothing to write with, and the failure is a reply that stops in the middle of a sentence.
Two settings control this and they do different jobs. The maximum output token count is a cap, and hitting it truncates mid sentence with no warning in the body of the reply. The reserve inside the budget is an allocation, and it is what stops the assembly from spending the tokens the cap was counting on. A feature that sets a cap and never reserves has two components disagreeing about the same tokens.
On some models the reserve has to cover more than the visible reply. A model that produces its working before answering spends output tokens on that working, and the working can run several times longer than the answer it precedes. So a feature that reserved 1,500 tokens for a two paragraph reply, and then moved to a model that reasons first, begins truncating on the day of the move with nothing else about it having changed.
Why a longer window moves the problem
A longer window removes the error and leaves three costs standing.
Money. Input tokens are charged on every call, and a context four times larger is four times the input charge for the same answer. Page 16 works this through with prices.
Time. The model processes the whole input before it emits the first token, and the attention work grows with the square of the sequence length, so a long context shows up directly in the time a user waits before anything appears.
Accuracy. Performance falls as input length grows, including on tasks a model handles perfectly at short lengths. The material fits. It is simply used less well, and that finding has a name, a body of evidence behind it and a page of its own two pages from here.
A budget under pressure needs somewhere to give. There are exactly four things a team can do with a context that has grown too large, and they have been named and taught as a set since the middle of 2025.