How to Reduce LLM Cost and Latency: Tokens and Caching

LLM cost and latency depend on the whole request path, including input processing, generation, retrieval, tools, retries and validation. Token prices explain part of the bill. They do not capture whether the user received a correct result or how long the complete task took.

Measure cost per successful task as well as cost per call. A cheaper model that requires repeated repair can cost more than a model that finishes correctly on the first attempt.

Calculate request cost explicitly

For a simple token priced service:

request cost =
  uncached input tokens × uncached input rate
  + cached input tokens × cached input rate
  + output tokens × output rate
  + applicable cache-write, storage and tool charges

Express rates in matching units. If prices are per million tokens, divide each token count by one million.

Here is a hypothetical example, not a current provider price list:

At 100,000 identical requests, model usage costs $1,700 under these assumptions. Tool charges, retries and hosting would be additional. State those assumptions whenever presenting an estimate.

Measure input processing and generation separately

Inference commonly includes a prefill stage that processes input and a decode stage that generates output. Their performance depends on the model, hardware, batching and implementation.

Long responses often contribute substantially to completion time because generation depends on earlier tokens. Shortening unnecessary output can therefore help both cost and latency. Reducing input may also help, especially when the input is large or poorly cached.

Do not remove evidence merely to make a token chart smaller. Compare quality on the cases that depend on the removed material.

Use prompt caching where the workload permits it

Prefix caching reuses computation for a matching portion of input. Stable instructions, examples and tool definitions can be candidates. Request specific timestamps placed before stable material may reduce the reusable prefix.

Caching rules differ. Some services use explicit cache controls, minimum lengths and separate write rates; others manage caching automatically. Claude's prompt caching documentation provides one implementation example, including lifetime and usage accounting.

Measure cache hit tokens from provider usage data. A stable prefix makes a request potentially eligible; it does not guarantee a hit on every call.

Route tasks by measured capability

A smaller model may handle a narrow classification or formatting task adequately. Test it against the same quality requirements as the larger model, including difficult cases.

Account for fallback traffic. If many requests escalate to a larger model, the first call may add cost and delay without reducing much work. A routing rule should have its own evaluation and a clear definition of successful completion.

Likewise, parallel calls can reduce elapsed time for independent work while retaining most of the token cost. Include concurrency limits and retry behaviour in load testing.

Distinguish streaming from validated delivery

Time to first token measures when generation begins to arrive. Time to first useful display measures when the user sees something meaningful. Completion time measures when the result is finished.

Streaming can improve perceived responsiveness, but content shown before validation has already reached the user. If citations or decisions must be checked first, buffer the answer until those checks pass. A provisional display needs a deliberate product design; retracting text does not undo its initial exposure.

Optimise against the full service target

Track median and tail latency, such as p95, alongside task quality and cost. A good average can hide slow tool calls or repeated repairs affecting a smaller group.

Change one substantial factor at a time, such as output length, context selection, cache layout, model choice or workflow structure. Keep the change when it meets the quality threshold and improves the relevant cost or latency measure.