Cost and latency

A four step chain has a bill, and every prompt, every retry, every parallel branch and every resent policy document is a line on it. Adding those lines up is what decides whether a design that works in a notebook survives a million requests a month, so cost and latency are the two quantities to hold, and both of them come out of the same unit.

That unit is the token. Providers price a request by counting the tokens that go in and the tokens that come out, usually charging separately for the two and charging more for output. The prices move often and the structure does not, so the useful thing to learn is the structure and the arithmetic that runs on it. Anybody can substitute this quarter's numbers into the calculation below.

The sections that follow explain why an output token costs several times what an input token costs, price one request end to end, and show what prompt caching needs before it hits. The last two sections cover choosing a smaller model for a smaller step, and what streaming changes about a wait that has not got any shorter.

Why an output token costs more than an input token

The asymmetry in the price list reflects an asymmetry in the hardware. A request runs in two phases and they use the machine very differently.

Prefill reads the prompt. Every input token is processed in one pass, with the work across positions done at the same time, so a graphics processor runs near its arithmetic limit and thousands of tokens clear in a fraction of a second.

Decoding writes the answer. Each output token needs its own forward pass through every layer of the model, and that pass exists to produce exactly one token. The arithmetic per pass is small, but the hardware still reads every weight in the model out of memory to perform it, so memory bandwidth sets the limit on this phase. One output token therefore occupies far more of the machine than one input token does.

Two consequences follow, and both matter more than the price ratio itself. Output length drives elapsed time almost on its own, since decoding is serial and prefill is not. And an instruction that makes a model write a long preamble before its answer costs money twice over, once in tokens and once in seconds the customer spends waiting.

One request priced end to end

The calculation below uses invented prices, stated in full so every line can be checked, and a request shaped like the support feature of the previous pages.

Prices assumed for this example, per million tokens
  input                £2.40
  output              £12.00
  cached read          £0.24     one tenth of the input price
  cache write          £3.00     one and a quarter times the input price

The shape of one request
  stable prefix      3,200 tokens   system prompt 400, tools 1,100,
                                    twelve worked examples 1,700
  variable input     3,060 tokens   six passages 2,100, history 900,
                                    the question 60
  output               320 tokens

With no caching
  input      6,260 x 2.40 / 1,000,000  =  0.0150240
  output       320 x 12.00 / 1,000,000 =  0.0038400
                                          ---------
  per request                              0.0188640

On a cache hit
  prefix     3,200 x 0.24 / 1,000,000  =  0.0007680
  variable   3,060 x 2.40 / 1,000,000  =  0.0073440
  output       320 x 12.00 / 1,000,000 =  0.0038400
                                          ---------
  per request                              0.0119520

On a cache miss, which also writes the prefix
  prefix     3,200 x 3.00 / 1,000,000  =  0.0096000
  variable   3,060 x 2.40 / 1,000,000  =  0.0073440
  output       320 x 12.00 / 1,000,000 =  0.0038400
                                          ---------
  per request                              0.0207840

Blended at a 95 per cent hit rate
  0.95 x 0.0119520  +  0.05 x 0.0207840  =  0.0123936

At 40,000 requests a day over thirty days
  no caching     40,000 x 30 x 0.0188640  =  £22,636.80
  with caching   40,000 x 30 x 0.0123936  =  £14,872.32
                                              ---------
  difference                                  £7,764.48

Three readings come out of that block. The output is 5 per cent of the tokens and 20 per cent of the cost, which is the price ratio doing its work. Caching saves 34 per cent of the total, and it saves it only on the 3,200 tokens that repeat, so the size of the saving is decided by how much of the window is genuinely stable. And a cache miss costs more than the same request with no caching at all, which is why the hit rate belongs on a dashboard beside the bill.

Prompt caching and the prefix that has to stay identical

Caching works on an exact prefix match. The provider keeps the computed state for a run of tokens from position zero forward, and a later request reuses that state only while its tokens are identical from position zero to the point where the two diverge. One differing token voids everything after it.

That single rule decides the entire design, and it is broken by accident all the time.

A window ordered so the cache never hits

  1  Request id 8f2c-41ab            18 tokens, different every request
  2  Timestamp 2026-09-25 14:07      12 tokens, different every request
  3  System prompt                  400 tokens, identical every request
  4  Tool definitions             1,100 tokens, identical every request
  5  Worked examples              1,700 tokens, identical every request
  6  Retrieved passages           2,100 tokens, different every request
  7  Conversation so far            900 tokens
  8  The question                    60 tokens

The match fails at position 0, so all 6,290 tokens are charged at the full
input rate and the 3,200 stable tokens gain nothing.

The same window reordered so the cache hits

  1  System prompt                  400 tokens  |
  2  Tool definitions             1,100 tokens  |  3,200 token stable prefix
  3  Worked examples              1,700 tokens  |
  4  Retrieved passages           2,100 tokens
  5  Conversation so far            900 tokens
  6  Request id and timestamp        30 tokens
  7  The question                    60 tokens

The first 3,200 tokens match, so they are charged at the cached read rate and
the remaining 3,090 at the full input rate.

Four habits keep a prefix stable. Volatile values go last, never first. Tool definitions are serialised in a fixed order, since a dictionary that iterates differently between processes produces a different prefix from the same tools. A prompt template carries a version so a change is deliberate. And the hit rate goes on a dashboard next to the cost, because a rewrite that moves one line upward shows up there and nowhere else.

Choosing a smaller model for a smaller step

The triage step of the previous chain reads 900 tokens and writes 12. It is a four way classification, and the largest model available will do it correctly and expensively.

Small model, at £0.15 input and £0.60 output per million tokens
  900 x 0.15 / 1,000,000  =  0.000135
   12 x 0.60 / 1,000,000  =  0.0000072
                             ---------
  per request                0.0001422

Large model, at £2.40 input and £12.00 output per million tokens
  900 x 2.40 / 1,000,000  =  0.002160
   12 x 12.00 / 1,000,000 =  0.000144
                             ---------
  per request                0.002304

Over 1.2 million requests a month
  small   £170.64
  large   £2,764.80
          ---------
  saved   £2,594.16

Sixteen times the price for a decision between four labels is the clearest saving in most pipelines, and it is available only because the chain separated the classification from the reasoning. A monolithic prompt has one model for everything, so the hardest step sets the price of every step.

The test for whether a step can move down is an evaluation set, not an opinion. A hundred triage cases with known labels settle it in an afternoon, and the same set catches the day the small model's provider changes something.

Streaming and the wait that did not get shorter

Latency has two numbers and teams often report only the second. Time to first token is how long the customer stares at nothing. Total time is how long until the answer is complete. Streaming moves the first number and leaves the second exactly where it was.

The same seven seconds feels entirely different depending on which row the interface exposes, and no engineering effort was spent on the model to get there.

Streaming carries one real cost, and it lands on everything the previous pages built. An answer cannot be validated until it is complete, so a feature that checks citations, enforces a schema or applies an output filter has to choose. It can buffer the stream, validate and then release, which gives back the whole of the gain. It can stream into a region of the interface it is able to retract, which is honest and needs design work. Or it can stream unvalidated text to a customer, which is the option that makes the checks pointless.

One number reconciles everything above, and it belongs on the same dashboard as the hit rate. Cost per successful request divides the whole bill, including retries, repairs, abandoned streams and escalations, by the count of requests that actually produced a usable answer. A feature with a cheap model and a 30 per cent repair rate is not a cheap feature.

Every figure on this page rests on one property of a chain. The number of calls is known before the request arrives, so a team can compute the bill in advance. An agent gives that up, because the model decides how many times round the loop to go, and the cost of a request stops being a number anybody can compute until after it has been paid.