Chaining and decomposition

Four outcomes from one call cover a question a single lookup can settle, but real requests are rarely that tidy. A refund enquiry needs a classification, a lookup against the account, a decision against a written policy and a reply worded for the person who asked, which is four pieces of work with four ways to go wrong. Chaining is the practice of making those four separate calls and wiring the edges between them in code.

The technique predates language models by decades under other names. A compiler runs in passes, a data pipeline runs in stages, and in both cases the reason is the same. A stage with one job can be given exactly the input it needs, can be checked on its own output and can be replaced without touching anything else. What language models add is a reason to care rather more, since every stage here is probabilistic and the checks between them are the only deterministic part of the system.

The sections below set one overloaded prompt against the same work split into steps, name the four reasons the split usually wins, do the arithmetic on errors compounding across a chain, and price the extra latency. The last section draws the line between a chain and the agents of the next module.

One prompt carrying five jobs

The prompt below is the shape a feature reaches after three months of requests from stakeholders, each one adding a sentence.

You are a support assistant for the Acme billing platform.

Read the customer's message. Work out which product area it concerns. Find
anything relevant in the knowledge base below. Decide whether the customer is
entitled to a refund under the policy below. Then write a reply in the
customer's own language, at a reading age of twelve, in no more than 120
words, including a link to the relevant help article. If you are not sure
about any of this, say so.

<policy>
{{refund_policy}}
</policy>

<knowledge_base>
{{entire_knowledge_base}}
</knowledge_base>

Message: {{message}}

Five problems come with it and none of them show up in a demonstration. The knowledge base is pasted whole, so the relevant paragraph competes with everything else for attention. Five jobs share one pass of generation, and the refund decision gets the same care as the word count. The output is a single paragraph, so a wrong refund decision is invisible inside good prose and no check can reach it. Any failure reruns all five jobs at full cost. And the final instruction asks the model to flag uncertainty about a search it never actually performed.

The same work as four steps

Splitting it gives each job its own prompt, its own model and its own contract. Three of the four prompts are shown here, and the second step is a database query with no model in it at all.

Step 1   Triage                       small model, temperature 0
-------------------------------------------------------------------------
Read the customer message and return one object.

  area      one of billing, access, data_import, reporting, other
  intent    one of refund_request, how_to, bug_report, account_change
  language  the ISO 639-1 code of the language the message is written in

Message: {{message}}

Step 3   Decide the refund            large model, temperature 0
-------------------------------------------------------------------------
Apply the policy below to the facts below. Consider nothing outside them.
Return one object.

  eligible       true, false, or null when the policy does not cover this case
  policy_clause  the id of the clause you applied, e.g. "4.2(b)"
  amount_minor   the refund in pence, or null
  reasoning      one sentence naming the fact and the clause that decided it

<policy>
{{refund_policy}}
</policy>

<facts>
  plan             {{plan}}
  charged_on       {{charged_on}}
  cancelled_on     {{cancelled_on}}
  usage_after      {{usage_after_cancellation}}
</facts>

Step 4   Write the reply              small model, temperature 0.4
-------------------------------------------------------------------------
Write a reply to the customer in {{language}}, in at most 120 words.

State the decision below and the reason for it. Do not quote the clause id.
Apologise at most once. Close with the help article link and nothing after it.

  decision       {{eligible}}
  amount         {{amount_display}}
  reason         {{reasoning}}
  help_article   {{article_url}}

The code between them is where a chain earns its keep, because every gap is a place to check something.

def handle(message: Message) -> Reply:
    triage = call(TRIAGE_PROMPT, message.text,
                  model=SMALL, schema=TriageSchema, temperature=0)

    if triage.intent != "refund_request":
        return route_to(triage.area, triage.intent)

    # Step 2 is the account database. The model never guesses a fact it can
    # be told, and these four values decide the whole of step 3.
    facts = billing.facts_for(message.account_id)

    decision = call(DECISION_PROMPT, policy=POLICY, facts=facts,
                    model=LARGE, schema=DecisionSchema, temperature=0)

    if decision.policy_clause not in POLICY.clause_ids:
        metrics.increment("chain.invented_clause")
        return escalate(message, reason="invented_clause")

    if decision.eligible is None:
        return escalate(message, reason="policy_silent")

    if decision.eligible and decision.amount_minor > facts.charged_minor:
        metrics.increment("chain.refund_exceeds_charge")
        return escalate(message, reason="refund_exceeds_charge")

    return call(WRITE_PROMPT, decision=decision, language=triage.language,
                model=SMALL, schema=ReplySchema, temperature=0.4)

Three checks sit between step three and step four, and each one costs microseconds. A clause id the policy never contained means the model invented its authority. A refund larger than the original charge means it invented its arithmetic. A null eligibility means the policy is silent, which is a question for a person. In the single prompt version all three of those failures arrive as a friendly paragraph telling the customer their money is on its way.

Four reasons a chain beats one long prompt

The split pays in four ways, and all four come from the same thing, which is that a step with one job has one input, one output and one reason to fail.

Each step gets a clean window. Step 3 sees the policy and four facts. Nothing about the customer's tone, the knowledge base or the reply format competes for attention with the decision that matters most.

Each step produces something checkable. A typed object between two steps can be validated, logged and compared against a known answer. A paragraph cannot. This is also what makes a chain testable step by step, which the module on evaluating and testing depends on.

Each step can use the model it needs. Triage is a four way classification a small model does at a fraction of the price and a fraction of the latency. The policy decision needs the larger model. Running both on the large model spends the difference for no gain.

Each step fails separately. A chain that breaks at step 3 reruns step 3, and the triage result and the account facts are still in hand. A monolithic prompt reruns everything, including the parts that worked.

Error compounding across a chain

The cost of splitting is that every step is another chance to fail, and the rates multiply. A chain succeeds only when every step in it succeeds.

The last row is the one to read twice. Five steps at 99 per cent match one step at 95 per cent, so the question is never whether a chain has many steps. It is what each step's own rate is, and how much of the gap a check between the steps can close.

Three moves keep the arithmetic in hand. Validating between steps converts an undetected error into a caught one, which changes a wrong answer into a retry or a handoff. Passing typed values between steps removes a whole class of failure, since a step reading a field cannot misread a paragraph. And a step producing nothing a check can read and nothing the next step needs belongs back inside its neighbour.

Retrying helps less than it appears. A retry raises the effective rate only for faults that vary between attempts, which covers a malformed object and a truncated response. A model that misreads clause 4.2(b) reads it the same way on the second attempt, so the retry buys a second identical wrong answer at full price.

Latency budgets and fanning steps out

Steps run one after another, so their times add. A four step chain on a realistic budget looks like this.

Output tokens dominate the elapsed time, because a model produces them one at a time while it reads the input in a single pass. The refund decision costs 4.1 seconds on 180 output tokens, and the triage step costs 0.4 seconds on 12.

Steps with no dependency between them can run at once. A request that needs three documents summarised before a merge runs the three summaries in parallel and waits for the slowest.

async def summarise_all(docs: list[Document]) -> str:
    # Three independent calls. Serial they cost 2.4 s each, so 7.2 s.
    # Gathered they cost as much as the slowest one, which is 2.4 s.
    summaries = await asyncio.gather(
        *(call_async(SUMMARY_PROMPT, doc, model=SMALL) for doc in docs)
    )
    # The merge depends on all three, so it waits.
    return await call_async(MERGE_PROMPT, summaries=summaries, model=LARGE)

Serial the three summaries and the merge take 7.2 plus 1.8, or 9.0 seconds. Gathered they take 2.4 plus 1.8, or 4.2 seconds. The token cost is identical in both, since parallelism buys wall clock time and nothing else.

The line between a chain and an agent

Everything above has one property in common. A developer decided the sequence before any request arrived, so the set of possible paths through the system is finite, written down and testable. Step 1 always runs first. Step 4 never runs twice.

That property is what the next module gives up. An agent holds the same tools and the same prompts, and it chooses which to call and in what order while it runs, so the path through the system is decided at request time by the model. The gain is coverage of tasks nobody enumerated. The cost is that the number of possible paths stops being finite, which changes testing, cost control and failure analysis all at once.

Before any of that, the four step chain above has a bill. Every prompt, every retry, every parallel branch and every token of policy text resends is a line on it, and a design that looks free in a notebook can be the largest item in an infrastructure budget at a million requests a month.