LLM production monitoring checks whether a deployed feature continues to meet its quality and service requirements. HTTP success rates and latency remain useful, but they cannot establish that an answer is correct or an action was appropriate.
Combine operational telemetry with sampled quality review. The operational signals arrive quickly; quality labels may be delayed and uncertain.
Define the monitored outcome
Start with the product's actual job. A support assistant may need correct policy answers, appropriate handoffs and no unauthorised actions. A summariser may need factual coverage without invented commitments.
Give each metric a denominator. “Unsupported answer rate” could mean unsupported answers among all requests, among answered requests or among reviewed answers. Those rates are not interchangeable.
Record model, prompt and release versions so changes can be associated with deployment events.
Monitor immediate operational signals
Useful signals include model and tool errors, output truncation, schema failures, invalid citations, token use, cache hit usage and latency.
For agents, add action counts, repeated calls, budget stops, approval outcomes and unknown write results. An increase in unknown outcomes may indicate a service problem requiring reconciliation.
These metrics do not require a model judge. They should remain available even if the quality grading service is unavailable.
Sample answer quality deliberately
Use a representative sample to estimate the traffic level quality rate and targeted samples for important risk segments. Preserve the distinction or apply explicit weights.
A calibrated LLM judge can assist, but retain human review for disagreement, high impact failures and calibration checks. User feedback adds useful evidence but is self selected and often sparse.
Track the number and age of labelled examples. A dashboard based on yesterday's 20 reviewed cases should not appear to describe today's entire workload precisely.
Alert on actionable changes
An alert should name the affected metric, release, segment, sample size and response owner. Link it to example traces.
Suppose the quality objective permits a 2% failure rate. An observed 6% rate is three times that allowance under the same definition. That ratio is a burn rate style signal, but its interpretation depends on the sample and labelling delay.
Google's SRE monitoring guidance explains the value of actionable alerts and user relevant signals. For sampled LLM quality, include minimum evidence requirements and avoid treating every short term fluctuation as an incident.
Critical events may warrant immediate action even when rare. Do not average an observed unauthorised action into a large volume of ordinary successful replies.
Diagnose drift before choosing a fix
A rising abstention rate can reflect missing documents, new user intents, retrieval failure or changed model behaviour. More expensive requests can reflect longer sessions, retries or a failed cache.
Inspect the traces behind the metric. Compare the source collection, request mix and release configuration before assuming that the prompt is responsible.
When the grading model changes, run calibration comparisons. Otherwise judge drift can appear as application drift.
Exercise: design an on call dashboard
Create dashboard panels for task success, unsupported claims, tool or contract failures, cost per successful task and tail latency. For each, write the denominator, data source, update delay and owner.
Choose one failure and write a response procedure. Inspect traces, identify the affected version, decide whether to pause actions or roll back, and preserve cases for evaluation.
The dashboard should help an operator decide what to do next. A large collection of unowned scores does not provide that capability.