LLM observability is the ability to understand how an application behaved by inspecting its telemetry. Metrics show patterns across requests, logs record individual events and traces connect the timed steps of one interaction.
A trace is especially useful when an incorrect answer could have originated in retrieval, context assembly, generation, validation or a tool. The final response alone rarely identifies the failed stage.
Connect spans across the request
A span represents an operation such as a search, model call or tool execution. Parent child relationships show how those operations belong to a larger request.
OpenTelemetry's tracing documentation explains these core concepts. Use stable conventions supported by the instrumentation in the application, and track version changes in experimental attributes.
A useful LLM trace might contain request handling, retrieval, context assembly, model generation, validation and delivery. An agent adds repeated model and tool spans.
Record the information needed to explain a decision
Capture version identifiers for prompts, models, schemas and retrieval configuration. Record selected source IDs and versions, token usage, timing, completion status and validation outcomes.
For tools, record the proposed operation, authorised scope, execution status and durable operation ID. Sensitive arguments may need redaction or controlled references.
{
"trace_id": "run-184",
"prompt_version": "support-v7",
"model_version": "recorded-deployment-id",
"retrieval": {
"source_ids": ["POL-4.2-v9", "POL-4.7-v9"],
"status": "ok"
},
"validation": {
"schema_valid": true,
"citation_ids_valid": true
},
"outcome": "answered"
}
This is a compact teaching record, not a complete tracing standard. A production trace also needs timestamps, span relationships and the fields required by its own diagnosis needs.
Capture content selectively
Full prompts and tool results can help reproduce failures, but they may contain personal data, confidential documents or secrets. Do not assume that observability requires unrestricted content logging.
Separate operational metadata from controlled content storage. Apply redaction, access restrictions, retention limits and deletion rules. A hash can identify a version without preserving the text, although replay then requires an authorised way to retrieve that version.
Retain enough information to understand what is missing from a trace. A redacted or sampled record should not be presented as a complete reproduction package.
Replay the component under investigation
To compare two prompts, replay the same request with recorded retrieval and read only tool observations. To test retrieval, rerun it against a versioned corpus and keep the generation step controlled.
Do not replay historical write actions against live services. Use a sandbox, fixtures or a dry run execution layer. A recorded approval is not permission to repeat an old external effect.
A replay also needs fixtures matched to tool name and arguments. Returning the next recorded result regardless of the new call can make an invalid agent path appear successful.
Understand replay limits
Hosted model versions, external data and execution timing can change. Seeds do not guarantee identical output. A trace may omit redacted content or a dependency's internal state.
State what was fixed and what remained live. That scope determines whether a score change can reasonably be attributed to the component under test.
Exercise: diagnose one unsupported answer
Take a response with a citation that does not support its claim. Inspect the selected source, its version, the assembled context and the generation output.
Determine whether the passage was wrong, the applicable exception was missing or the model misread adequate evidence. Record the earliest stage where the result became incorrect.
Turn the case into an evaluation fixture. Add the fields that would have made the diagnosis possible earlier, while avoiding unnecessary capture of private content.