Tracing and observability

A trace is the structured record of one interaction from the request arriving to the answer leaving, held as a tree of timed steps. A gate that compares two versions of a prompt across three hundred cases has to get those cases from somewhere, and a trace is both where each one comes from and what the comparison then runs against.

Distributed tracing arrived in ordinary software through a technical report Benjamin Sigelman and colleagues published at Google in April 2010, describing Dapper, the tracing system running across Google's own production services. The vocabulary of traces and spans comes from that line of work. A language model feature needs the same structure with more inside each span. The interesting part of the request is no longer only how long a call took. It is what text went into the model, which passages retrieval chose, what the model asked a tool to do and what came back.

The sections below say what a trace records and show one written out. The naming convention comes next, then the argument that a trace is a reproducible test case, then a replay against a changed prompt. The page ends with what a replay cannot reproduce and with what never gets written down at all.

What a trace records

A trace covers one user request and holds a span for each step the system took. Six kinds of span appear in almost every language model feature.

1. The request. Who asked, in what language, through what channel, with which prompt version and which model version resolved for them.
2. Retrieval. The query as it was actually issued after rewriting, the index and its build date, every chunk returned with its identifier and score, and which of those chunks made it into the window.
3. The model call. The rendered prompt after every variable was substituted, the sampling settings, the token counts in and out, and the raw completion before any parsing.
4. Tool calls. The name of each tool, the arguments the model supplied, the result or the error, and how long it took.
5. Validation. Whether the output parsed against its schema, how many repair attempts ran and what the final object looked like.
6. The outcome. What the user was shown, what they did next, and any later signal such as an escalation or a thumbs down arriving an hour afterwards.

The third item is where teams cut corners. Recording the template and the variables separately looks equivalent and is not, because the rendering code changes and a trace from August then reconstructs into something the system never sent. The rendered text is what the model saw and the rendered text is what gets stored.

One trace written out in full

The record below is a single support reply. It is long, and a shorter record would leave out exactly the fields that make it useful.

{
  "trace_id": "01J9F2Q7K3B8V0XQZ4T6M2",
  "feature": "support-reply",
  "started_at": "2026-08-19T09:14:22.104Z",
  "duration_ms": 3180,
  "outcome": "answered",
  "cost_usd": 0.0138,

  "spans": [
    {
      "span_id": "a1",
      "parent_id": null,
      "name": "support_reply.request",
      "duration_ms": 3180,
      "attributes": {
        "user.tier": "business",
        "user.locale": "en-GB",
        "channel": "email",
        "prompt.id": "support-reply",
        "prompt.version": 8,
        "gen_ai.conversation.id": "th_7712"
      }
    },
    {
      "span_id": "b1",
      "parent_id": "a1",
      "name": "retrieval.search",
      "duration_ms": 224,
      "attributes": {
        "query.original": "my delivery arrived damaged, can I get money back",
        "query.rewritten": "damaged delivery refund policy return window",
        "index.name": "policy",
        "index.built_at": "2026-08-11T02:00:00Z",
        "top_k": 5
      },
      "result": [
        { "doc": "RET-04", "chunk": 3, "score": 0.81, "tokens": 412, "used": true },
        { "doc": "RET-04", "chunk": 4, "score": 0.77, "tokens": 388, "used": true },
        { "doc": "SHIP-11", "chunk": 1, "score": 0.52, "tokens": 301, "used": true },
        { "doc": "RET-09", "chunk": 2, "score": 0.44, "tokens": 355, "used": false },
        { "doc": "FAQ-02", "chunk": 7, "score": 0.41, "tokens": 260, "used": false }
      ]
    },
    {
      "span_id": "c1",
      "parent_id": "a1",
      "name": "chat",
      "duration_ms": 2640,
      "attributes": {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "example-provider",
        "gen_ai.request.model": "example-model-2026-06-30",
        "gen_ai.response.model": "example-model-2026-06-30",
        "gen_ai.request.temperature": 0.2,
        "gen_ai.request.max_tokens": 600,
        "gen_ai.output.type": "json",
        "gen_ai.usage.input_tokens": 2914,
        "gen_ai.usage.output_tokens": 188
      },
      "rendered_prompt_ref": "blob://traces/01J9F2Q7K3B8V0XQZ4T6M2/prompt.txt",
      "completion_ref": "blob://traces/01J9F2Q7K3B8V0XQZ4T6M2/completion.txt"
    },
    {
      "span_id": "d1",
      "parent_id": "c1",
      "name": "execute_tool",
      "duration_ms": 91,
      "attributes": {
        "gen_ai.tool.name": "lookup_order",
        "gen_ai.tool.call.id": "call_004",
        "arguments": { "order_id": "88120" },
        "status": "ok",
        "result.delivered_on": "2026-07-02"
      }
    },
    {
      "span_id": "e1",
      "parent_id": "a1",
      "name": "validate_output",
      "duration_ms": 3,
      "attributes": {
        "schema": "support_reply@3",
        "valid": true,
        "repair_attempts": 0
      }
    }
  ],

  "later": {
    "shown_to_user": true,
    "thumbs": "down",
    "thumbs_at": "2026-08-19T09:21:40Z",
    "escalated_to_human": true,
    "escalated_at": "2026-08-19T11:02:09Z"
  }
}

Four details in that record pay for themselves repeatedly. The used flag on each retrieved chunk separates a retrieval failure from a reading failure, which are different bugs with different fixes. The index build date explains a trace that stopped working after a reindex. The later block arrives hours after the request and attaches to the same trace, which is the only way a thumbs down ever gets connected to the retrieval that caused it. And the rendered prompt is stored as a blob reference, because it runs to several thousand tokens and nobody wants it inline in a query result.

The attribute names two tools can agree on

Names under a common convention are what let a trace written by one library be read by a dashboard somebody else built. The OpenTelemetry project maintains semantic conventions for this, and the set covering generative AI has moved into a repository of its own. Checked on 25 September 2026, the attributes for model calls carry development stability, which means they are in wide use and the project has not frozen them.

The pair worth noticing is the request model and the response model. They are separate attributes because they disagree more often than teams expect. A request naming a floating alias is answered by whichever dated version sits behind that alias today, and the response attribute is what records which one. A team that stores only the requested name has a six month archive of traces that cannot say which model produced them, which makes every comparison across that archive worthless. That failure is the subject of the page on migration.

Conventions also carry a privacy decision. Recording the full prompt and completion is optional in the standard and switched off by default, because those fields hold whatever the user typed. Turning it on is the right choice for most product teams and it is a choice, made deliberately, with retention and redaction settled at the same time.

A trace as a reproducible test case

A trace holds everything a request consumed, which means it holds everything needed to run that request again. That is the observation everything below rests on.

An eval case, as the earlier pages defined it, needs the input, the context the system had at the time and a statement of acceptable behaviour. A trace supplies the first two exactly, with no reconstruction and no guessing. The third still takes a person, which is the work of reading traces and writing what the answer should have been.

The consequence is that building an eval set is mostly an act of selection. Yesterday's traffic already contains several thousand fully specified test cases, so the work is choosing which of them to keep and deciding what each one should have done.

The freezing matters as much as the recording. A case about a thirty day return window gives a different answer in September than it gave in August, so the trace stores the clock reading the system used and the replay serves it back. The same applies to the retrieved passages, the account state and every tool result. Anything the original run pulled from outside is stored and replayed, so the only thing that differs between the two runs is the component under test.

Replaying a trace against a change

A replay reruns one recorded interaction with one component swapped. Everything else is served from the record.

def replay(trace, prompt_version, model_version, judge):
    """Rerun one recorded interaction with a changed component.

    Retrieval, tool results and the clock are served from the trace, so
    the only difference between this run and the original is whatever
    the caller swapped. A replay that calls the live index measures the
    index as well as the prompt, and nobody can tell which moved.
    """
    rendered = render(
        prompt=load_prompt("support-reply", prompt_version),
        message=trace["input"]["message"],
        passages=[c for c in span(trace, "retrieval.search")["result"] if c["used"]],
        account=frozen_account(trace),
        now=trace["started_at"],
    )

    replayed = call_model(
        rendered,
        model=model_version,
        tools=RecordedTools(spans(trace, "execute_tool")),
    )

    return {
        "trace_id": trace["trace_id"],
        "before": read_blob(span(trace, "chat")["completion_ref"]),
        "after": replayed,
        "before_scores": trace["scores"],
        "after_scores": judge(trace["input"], rendered, replayed),
    }


class RecordedTools:
    """Serve each tool call from the trace, matched on name and arguments.

    A call the original run never made has no recorded answer. Raising
    here is deliberate: a silent stub would let the replay wander off
    the recorded path and still report a score.
    """

    def __init__(self, tool_spans):
        self.by_call = {
            (s["attributes"]["gen_ai.tool.name"],
             json.dumps(s["attributes"]["arguments"], sort_keys=True)): s
            for s in tool_spans
        }

    def call(self, name, arguments):
        key = (name, json.dumps(arguments, sort_keys=True))
        if key not in self.by_call:
            raise OffTraceCall(name, arguments)
        return self.by_call[key]["attributes"]["result"]

Run across a few hundred traces, that function prints a diff a person can read in a minute.

$ traces replay --since 2026-08-01 --sample 200 --prompt support-reply@9

200 traces replayed against support-reply@9
model held at the version each trace recorded
retrieval and tool results served from the traces

verdict changes
  fail -> pass   38
  pass -> fail    6
  unchanged     149
  off trace       7   the new prompt called a tool the original run never did

one example, trace 01J9F2Q7K3B8V0XQZ4T6M2
  grounding     2  ->  2
  outcome       1  ->  2   v9 names the 90 day damage claim route
  boundaries    2  ->  2
  answered      0  ->  2   v9 answers in the first sentence
  verdict    fail  -> pass

- Our returns policy sets a 30 day window from the delivery date, and
- order 88120 was delivered on 2 July, so the window has closed.
+ Yes, there is a route for this. Damaged deliveries are handled as a
+ claim rather than a return, and that stays open for 90 days.

The row labelled off trace is the one worth reading every time. Seven of the two hundred replays wandered somewhere the recording could not follow, and each of those is either a genuine improvement the replay cannot score or a new failure mode. Both need a person.

What a replay cannot reproduce

A replay is faithful up to the point where the world moved, and four things move.

Anything that reads a clock or a live balance. The trace freezes what the system saw. A replay therefore tests behaviour against August's account state, which is correct for the comparison and wrong for anybody who wanted to know what the system would do today.

Any path the original run never took. A changed prompt that decides to call a different tool has no recorded answer to work with. Stubbing one invents a world, so the honest handling is to stop and report the case as unscored.

The index behind retrieval. Replaying with the recorded chunks measures the prompt. Replaying against the current index measures both together. Both runs are useful and mixing them in one report is how a retrieval regression gets attributed to a prompt.

Sampling variability. Replaying once is a sample of one. The repeat count the eval runner applies belongs here too, and a replay that changes a verdict in one run out of three has found variance where somebody wanted an improvement.

Sampling, retention and redaction

Three practical decisions stop tracing from becoming its own problem.

What fraction to keep. Traces are large, mostly because of the rendered prompt. The usual arrangement keeps the span tree for everything and the full text for a sample, with the sample biased hard towards anything interesting. An error, a refusal, a schema repair, a thumbs down, an escalation or an unusual latency pulls a trace into full retention regardless of the sampling rate.

How long to keep it. An eval set built from traces outlives the traces, so a case promoted into the set copies the material it needs into the case file. Traces themselves rarely need more than thirty to ninety days, which is what the retention conversation with a privacy team usually settles on.

What never gets written. A trace holds whatever the user typed, which includes card numbers, health details and other people's names. Redaction runs before storage and never after it, the fields it covers are listed somewhere a reviewer can read, and access to full text traces is a permission somebody grants. A team that skips this builds a searchable archive of its customers' worst days.

Everything on this page treats an interaction as one request, one retrieval and one answer. An agent takes twelve steps, calls four tools, revisits a decision and arrives somewhere. Its trace has the same structure and its evaluation does not, because a correct answer reached through a broken path will fail differently tomorrow. The next page is about judging the path.