A regression test for a prompt runs the held out set against a changed version and compares the result with the version currently serving production. A grader worth trusting is the last missing piece, so once a validated judge, a set built from real traffic and a comparison that runs automatically are all in place, an eval stops being a document and becomes a gate.
Nothing about this is new as software practice. A test suite that runs on every change and blocks a merge has been ordinary for decades. What changes is that the suite gives a different answer each time it runs, so the gate compares two distributions and decides whether the difference between them is real.
The sections below treat a prompt as a versioned artefact, then show the comparison a pipeline prints and the configuration that runs it. The gate rules follow, with the segment a passing average hides, and the page ends with what a regression run cannot catch.
A prompt as a versioned artefact
A prompt held in a database row that somebody edits through an admin screen cannot be regression tested, because there is no previous version to compare with and no record of what changed. The first requirement is a file with a version on it.
prompts/support-reply.md
---
id: support-reply
version: 8
owner: support-ai
supersedes: 7
created: 2026-09-18
model: resolved from config/models.yaml, never named in this file
changelog: |
7 -> 8 Added an explicit instruction to name the return window and
cite the policy document. Written after the August trace
reading found 14 replies in 100 answering from the model's
own knowledge with no citation.
evidence:
set: held-out
cases: 300
last_run: 2026-09-19
overall: 0.860
segments:
billing: 0.917
order: 0.900
product: 0.917
complaints: 0.650
---
You are a support assistant for an online retailer. Answer using only
the policy passages supplied below...
Three fields carry weight beyond the obvious. supersedes makes the chain
readable, so anybody can walk back through the versions to find when a
behaviour appeared. model points at a separate file, which is what allows the
model version to be held constant while the prompt changes and changed on its
own later. And evidence records the last measurement with the set it came
from, so a version is never separated from the number that justified shipping
it.
The comparison a build runs
With a versioned prompt on both sides, the comparison is mechanical. The output below is what a pipeline prints on a pull request that changes the prompt.
$ evals compare --baseline prompts/support-reply@v7 \
--candidate prompts/support-reply@v8 \
--set held-out --repeats 3
support-reply held out set, 300 graded cases, 3 repeats each
plus a 60 case adversarial block, reported in the gate
model pinned at the version v7 was last measured on
judge pinned at judge/support-rubric@v4
segment cases v7 v8 change moved better worse
------------------ ----- ------ ------ ------- ----- ------ -----
billing 120 85.0% 91.7% +6.7 14 11 3
order status 60 80.0% 90.0% +10.0 8 7 1
product questions 60 85.0% 91.7% +6.7 6 5 1
complaints 60 85.0% 65.0% -20.0 12 0 12
------------------ ----- ------ ------ ------- ----- ------ -----
overall 300 84.0% 86.0% +2.0 40 23 17
sign test on the 40 cases whose verdict moved
23 better, 17 worse, two sided p = 0.43
gate
overall floor 0.85 86.0% pass
no segment may fall > 3.0 -20.0 FAIL complaints
adversarial cases all pass 60/60 pass
result: FAIL
The three columns on the right are the ones worth arguing over. A change of plus two points on an average of three hundred cases is well inside the noise of a set that size, and the sign test says so directly with a probability of 0.43. The same run moved forty individual verdicts, which is where the information actually is, and twelve of those moves were one segment breaking.
The instruction in version 8 told the assistant to name the return window and cite the policy. On billing and order questions that is exactly right. On a complaint it produces a reply that opens with a policy citation to somebody who has just described a damaged delivery, and the rubric criterion about answering the question asked marks every one of them down.
The configuration that runs it
The comparison runs on the change, in the pipeline, with no human deciding whether today is a day for evals.
# .github/workflows/eval.yml
name: eval
on:
pull_request:
paths:
- "prompts/**"
- "eval/**"
- "src/retrieval/**"
- "config/models.yaml"
jobs:
held-out:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- name: Resolve the baseline
# The baseline is whatever is serving production right now, read
# from the registry. Comparing against the previous commit on
# this branch measures the last hour of work and not the change.
run: echo "BASELINE=$(prompts current support-reply)" >> $GITHUB_ENV
- name: Run the held out set against both versions
env:
MODEL_VERSION: ${{ vars.PINNED_MODEL }}
JUDGE_VERSION: ${{ vars.PINNED_JUDGE }}
run: |
evals compare \
--baseline "prompts/support-reply@$BASELINE" \
--candidate "prompts/support-reply@working" \
--set held-out --repeats 3 \
--report eval-report.json
- name: Apply the gate
run: evals gate eval-report.json --rules eval/gate.yaml
- uses: actions/upload-artifact@v4
if: always()
with:
name: eval-report
path: eval-report.json
Two lines in that file do more than they look like. The paths filter names
retrieval and the model configuration alongside the prompts, because a change
to what retrieval returns moves the output as surely as a change to the
instruction does. And the baseline comes from the registry, so the comparison
is always against production and never against whatever the branch happened to
contain an hour ago.
The gate rules and why each one exists
A gate is a short function with rules a team agreed before it had a result in front of it. Agreeing them afterwards is how a failing run becomes a conversation about whether the set is any good.
from math import comb
GATE = {
"overall_floor": 0.85, # absolute, agreed before the change
"segment_drop_limit": 0.03, # no segment may fall further than this
"adversarial_floor": 1.00, # every adversarial case, every repeat
"min_cases_per_segment": 30, # below this a segment is reported only
}
def sign_test(better, worse):
n = better + worse
if n == 0:
return 1.0
k = max(better, worse)
return min(1.0, 2 * sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n)
def gate(baseline, candidate, rules=GATE):
"""Return a verdict and the reasons behind it.
Every reason names a number and the rule it broke, because a gate
that prints "quality regression" gets switched off within a month.
"""
blocking, warnings = [], []
if candidate["overall"] < rules["overall_floor"]:
blocking.append(
f"overall {candidate['overall']:.3f} is below the floor "
f"{rules['overall_floor']:.3f}"
)
for segment, score in candidate["segments"].items():
cases = candidate["cases"][segment]
drop = baseline["segments"][segment] - score
if cases < rules["min_cases_per_segment"]:
warnings.append(f"{segment} has only {cases} cases, reported only")
continue
if drop > rules["segment_drop_limit"]:
blocking.append(
f"{segment} fell {drop:.3f} against a limit of "
f"{rules['segment_drop_limit']:.3f}"
)
if candidate["segments"]["adversarial"] < rules["adversarial_floor"]:
blocking.append("an adversarial case failed on at least one repeat")
# Reported and never blocking. A true improvement of one point on a
# 300 case set cannot clear a significance test, and blocking on this
# would stop every small honest gain while letting a large regression
# through on the same logic.
p = sign_test(candidate["better"], candidate["worse"])
if p > 0.05:
warnings.append(
f"the net change is indistinguishable from noise, p = {p:.2f}"
)
return ("fail" if blocking else "pass"), blocking, warnings
Four decisions in that function are the ones a team argues about.
The overall floor is absolute. It is a number somebody agreed, not the previous run plus a tolerance. A floor defined relative to last week drifts downwards one acceptable step at a time.
The segment limit measures movement. A segment that has always sat at seventy per cent is allowed to stay there. What the rule catches is a fall against the baseline, because a fall is what this change caused.
Adversarial cases are all or nothing. There is no percentage on a case that exists because somebody tried to make the system leak a document. One failure blocks.
Statistical significance warns and never blocks. A real improvement worth two points on three hundred cases will not clear a significance test, and a gate that demands one would refuse every honest small gain. The number is printed so nobody claims a win the set cannot support.
Making the complaints instruction conditional and running the comparison again gives a different verdict. Version 9 scored 91.7 per cent overall against version 7's 84.0, thirty seven verdicts moved, thirty of them improved and the two sided probability of that split is 0.0002. Every segment rose. That is what a change worth shipping looks like on this output, and it looks nothing like plus two points.
The segment a passing average hides
The complaints row above is the whole argument for reporting per segment, and it is worth stating as arithmetic rather than as advice.
An average over a set is a weighted mean, and the weights are the case counts. Billing carried a hundred and twenty of three hundred cases and rose by 6.7 points, which alone lifts the overall figure by 2.7. Complaints carried sixty and fell by 20, which costs 4.0. Order status and product questions between them added 3.3. The four movements sum to plus two, and the number a team would have seen with no breakdown is plus two.
That arithmetic has a direct consequence for how a set is built. A segment needs enough cases for its own rate to mean something, which is the reason the previous page set a floor of thirty to sixty cases per segment instead of sampling in proportion to traffic. A segment with eight cases produces a rate that jumps twelve points when one case flips, so the gate reports it and refuses to block on it.
The second consequence is about which segments to carve out at all. The useful cuts are the ones where the feature plausibly behaves differently, which means intent, language, channel, account type, input length and whether retrieval returned anything. A cut by day of week tells nobody anything.
What a regression run cannot catch
A gate is a filter and never a proof, and three things pass straight through it.
A failure mode nobody has written a case for. The set contains the organisation's history of being wrong. A change that introduces a new way of being wrong scores well on every existing case and ships.
Anything that depends on scale. Latency at the ninety ninth percentile, cost per thousand requests and behaviour under a queue do not appear in a three hundred case run on a build machine. Those belong to the page on monitoring.
A drift in the judge itself. If the provider updates the judge model between the baseline measurement and the candidate measurement, the comparison has compared two graders while appearing to compare two prompts. Pinning the judge version in the pipeline is the defence, and rerunning the judge against its human labels is what confirms the pin still means what it meant.
Each of those gaps has the same underlying shape. The gate can only compare what somebody recorded, and the quality of every comparison on this page depends on having a faithful record of what the system did on each case. That record has a name and a structure, and the next page is about it.