Monitoring a language model feature means watching what it says, because the failure mode is a fluent answer that happens to be wrong. Getting a change into production safely is where a release process stops. A feature that passed its gate, rolled out cleanly and sat at 0.93 on the held out set can still be failing for a class of user nobody sampled, and nothing in that process will say so.
Ordinary monitoring was built for systems that fail loudly. A service returns a 500, a queue backs up, a disk fills, and each of those is a discrete event somebody can count. A support assistant that invents a refund policy returns a 200 in four seconds with a well formed body. Every existing dashboard shows a healthy service.
The sections below explain why an error rate says so little here, then set out the signals worth watching and how a sample of live traffic reaches the judge. An alert condition follows, adapted for a measurement that arrives slowly, then the signals that need no judge at all. The page ends with what somebody does when a number moves.
Why an error rate says almost nothing about this feature
Three properties make the usual measures blind to the failures that matter.
The failure is a successful request. Nothing throws. The model produced tokens, the schema validated, the response left in four seconds. The only thing wrong with it is the content, and the content is what no status code describes.
The failure is not reproducible on demand. The same input produces a different answer on the next run, so an engineer told about a bad reply often cannot make it happen again. Without the trace that recorded the original run, the investigation ends there.
The failure is concentrated. A change rarely degrades everything by two points. It breaks one language, one intent, one account type or one input shape, and the affected group is usually small enough that an aggregate moves by less than the noise. The complaints segment falling twenty points in the worked comparison on regression testing moved the overall figure by four.
What follows from those three is that the monitoring has to sample content, score it, keep the score attached to the trace it came from, and report it cut by the same segments the eval set uses.
The signals worth watching
Six signals between them cover most of what goes wrong. Two need a judge and four are free.
| Signal | What it catches | What usually moves it |
|---|---|---|
| Sampled judge score, per segment | Quality falling for a group | A prompt change, a model update, a reindex |
| Citation and grounding rate | Answers drifting away from the retrieved material | Retrieval returning less, or the window filling up |
| Refusal rate | The feature declining work it should do | A tightened instruction, a safety filter, an ambiguous input class |
| Fallback and repair rate | Structured output breaking | A model update, a schema change, longer inputs |
| Empty retrieval rate | Questions arriving that the index cannot serve | A new product, a new market, a stale index |
| Tail latency and cost per request | A loop wandering or a window growing | More retrieved chunks, longer histories, an agent taking more steps |
The pairing worth watching is refusal rate against judge score. A change that makes the model more cautious usually raises the score, because a refusal fails fewer rubric criteria than a wrong answer does. A feature can improve on every quality measure while quietly becoming useless, and the refusal rate is the only line that says so.
Sampling live traffic through the judge
Scoring every response doubles the cost of the feature and buys very little, because most traffic is ordinary. A stratified sample with a rule for what is always scored gives a better picture for a few per cent of the spend.
import random
# Sample rates chosen by consequence and never by volume. Complaints are
# 6 per cent of traffic and get scored at eight times the rate of
# billing, because a bad answer there costs the most.
SAMPLE_RATES = {
"complaint": 0.25,
"billing": 0.03,
"order": 0.03,
"product": 0.05,
"other": 0.10,
}
ALWAYS_SCORE = ("refused", "schema_invalid", "tool_error",
"thumbs_down", "escalated", "retrieval_empty")
def selection(trace):
"""Return why this trace was chosen, or None.
The two reasons are kept apart on purpose. A trace pulled in because
something was already flagged cannot be counted in a rate, because
including it makes the sample worse than the population it stands
for. Those traces are a reading queue and nothing else.
"""
if any(trace["flags"].get(flag) for flag in ALWAYS_SCORE):
return "flagged"
if random.random() < SAMPLE_RATES.get(trace["segment"], 0.05):
return "random"
return None
def score_window(traces, judge, judge_version):
random_sample, flagged = [], []
for trace in traces:
reason = selection(trace)
if reason is None:
continue
result = judge(trace["input"], trace["context"], trace["output"])
# Written back onto the trace, so a number that moves can be
# opened and read. A score that lives only in a metrics store
# tells somebody that quality fell and nothing about which
# answers fell or why.
annotate(
trace["trace_id"],
judge_version=judge_version,
verdict=result["verdict"],
criteria=result["scores"],
sampled_as=reason,
)
row = {
**result,
"segment": trace["segment"],
"prompt_version": trace["prompt_version"],
"model_version": trace["model_version"],
}
(random_sample if reason == "random" else flagged).append(row)
return {
# The only figures that estimate the population.
"rates": aggregate(random_sample,
by=("segment", "prompt_version", "model_version")),
# A queue for a person, carrying no rate at all.
"to_read": sorted(flagged, key=lambda r: r["total"])[:50],
}
The separation between the two return values is the part most teams get wrong. Pulling every thumbs down into the scored set and then reporting a pass rate over the whole set produces a number that falls whenever users complain more, which is not the same thing as quality falling. The random portion carries the rate. The flagged portion is work for somebody on Tuesday morning.
The cut by prompt version and model version costs nothing and settles most arguments. A score that fell on Thursday is either a version the team shipped or a version it did not, and those two facts lead to entirely different afternoons.
An alert condition for a variable output
An alert on a single bad response is noise, because a variable system produces bad responses at a rate that never reaches zero. What is worth paging somebody about is a rate moving fast enough to matter.
The established shape for this comes from the Site Reliability Workbook, published by Google in 2018. An objective is stated over a window, the budget is what the objective allows to fail, and an alert fires on how fast that budget is being consumed. The recommended parameters in its chapter on alerting set out three tiers. A burn rate of 14.4 measured over one hour pages, because at that speed two per cent of a thirty day budget goes in a single hour. A burn rate of 6 over six hours pages at five per cent. A burn rate of 1 over three days raises a ticket. Each tier also carries a short window one twelfth the length of the long one, so an alert resets promptly once the burn stops.
The shape transfers to a quality objective with one adjustment, which is sample size.
# alerts/support-reply.yaml
slo:
name: support reply quality
objective: 0.90 # share of judged responses with verdict "pass"
window: 30d
measured_on: >
the random portion of the live sample only, scored by
judge/support-rubric@4, excluding traces pulled in because they were
already flagged
burn_rate_alerts:
# The burn rates and the budget fractions are the Site Reliability
# Workbook's table 5-8. The windows are longer. At 80,000 requests a
# day and the sample rates above, five minutes yields about 14 scored
# responses, which cannot distinguish anything from anything.
- severity: page
burn_rate: 14.4
long_window: 6h # 1h in the original
short_window: 30m
min_scored_in_long_window: 150
- severity: page
burn_rate: 6
long_window: 24h # 6h in the original
short_window: 2h
min_scored_in_long_window: 500
- severity: ticket
burn_rate: 1
long_window: 3d
short_window: 6h
min_scored_in_long_window: 1500
# Evaluated per segment as well as overall, because the whole argument
# for segments is that an aggregate hides a group.
per_segment:
enabled: true
min_scored_in_long_window: 60
severity: ticket
absolute_alerts:
- severity: page
condition: schema_invalid_rate > 0.02 sustained over 15m
- severity: page
condition: p95_latency_ms > 12000 sustained over 15m
- severity: ticket
condition: refusal_rate > 2x the trailing 7 day median over 6h
- severity: ticket
condition: empty_retrieval_rate > 0.10 over 6h
- severity: ticket
condition: cost_per_request > 1.5x the trailing 7 day median over 24h
Three decisions in that file are the ones a team should make deliberately.
The minimum sample in the window. An alert that can fire on fourteen observations will fire on fourteen observations, repeatedly, at three in the morning. Refusing to evaluate the rule below a floor is what makes a quality page believable.
The windows lengthened from the original. Google's table was written for request level availability, where a busy service produces thousands of observations a minute. A sampled quality signal produces a few hundred an hour. Keeping the burn rates and stretching the windows preserves the arithmetic and matches the data rate.
Cost and refusal alert to a ticket and never to a page. Both are real signals and neither is an emergency. A page that arrives for something nobody will act on before Monday teaches the team to ignore pages.
The signals that need no judge at all
Four measurements cost nothing, arrive immediately and catch a surprising share of real incidents.
Schema validation failures. Structured output breaking is the earliest signal that a model version changed underneath the feature. It is a boolean per request, it needs no sampling and it moves within minutes.
Empty or weak retrieval. The share of requests where retrieval returned nothing, or returned nothing above the score threshold, is a direct measure of questions arriving that the index cannot serve. It rises when a company launches something the documentation has not caught up with.
Refusal and fallback. Every path the feature has for declining, apologising or handing over to a person is counted separately. A fallback rate that doubles overnight is a defect somewhere upstream, and the aggregate quality score will rise while it happens.
Token counts and cost per request. Input tokens rising steadily is a context window filling up with history, retrieved chunks or an agent's accumulated scratchpad. It shows up on the invoice a month later and on this chart the same day.
Those four belong on the same dashboard as the judge score, because three of the four move before the quality signal does.
Reading the traces behind a number that moved
A monitoring system that reports a number and nothing else produces meetings about the number. The point of writing the judge score back onto the trace is that every movement opens into the responses behind it.
The sequence that works takes about twenty minutes. Somebody takes the segment and window where the score fell, pulls the twenty lowest scoring traces from it, and reads them with the judge's evidence quotations beside each one. Most of the time a pattern appears in the first five, because a real regression affects a class of input and a class of input looks the same.
What comes out of that reading is one of three things, and each has a different next step. A genuine regression becomes a rollback and then a case in the eval set. A shift in traffic, meaning users asking something new, becomes a change to the prompt or to the index and a new segment in the set. And a judge disagreeing with people becomes a revalidation against human labels, because the grader has drifted and every number since it drifted is suspect.
The third outcome is the one worth expecting, because the judge sits on a model the team does not control. So does the feature. A provider can update a hosted model, deprecate the version a profile pins, or price it differently, and none of those arrive as a change to any file in the repository. The next page is about the dependency nobody controls.