Prompt regression testing compares a proposed change with an established baseline on the same evaluation cases. It identifies improvements, new failures and trade offs before the change reaches all users.
The release unit is larger than the prompt text. Model configuration, examples, retrieval settings, schemas and tool contracts can all change behaviour. Record them together so the comparison has a clear meaning.
Freeze the baseline and evaluation conditions
Use immutable versions or snapshots for both systems. These can live in source control or a versioned registry; the essential property is recoverable history.
Fix the dataset and grader versions. When isolating a prompt change, keep retrieved evidence and tool fixtures stable. When evaluating the whole application, run the actual pipeline against a controlled environment and label that broader scope.
Save outputs as well as scores. A failed comparison should be inspectable without paying for another run that may produce different text.
Compare results by case and segment
The overall score can conceal damage to a smaller group.
| Segment | Cases | Baseline passes | Candidate passes |
|---|---|---|---|
| Routine requests | 90 | 81 | 86 |
| Disputes | 10 | 9 | 5 |
| Total | 100 | 90 | 91 |
The candidate improves from 90% to 91% overall while disputes fall from 90% to 50%. The small dispute sample also carries substantial uncertainty, but the observed regressions warrant inspection.
Report baseline only successes and candidate only successes. Those paired outcomes show which cases changed and support a more appropriate comparison than two unrelated percentages.
Define the gate before running the candidate
A gate should encode the requirements that justify release. Example conditions include a minimum success rate in each critical segment, no observed forbidden actions and a maximum cost or latency increase.
Set a minimum sample requirement for important segments. A candidate should not pass merely because a difficult segment happened to contain no cases.
Use a separate review outcome when the evidence is inconclusive. Repeatedly running the same candidate until one result passes selects favourable noise.
Handle variable output without hiding failures
Repeat cases where stability matters, and keep every trial in the report. Do not retry a failed evaluation until it succeeds and then record only the success.
Distinguish application retries from evaluator reruns. If the product includes one repair attempt, evaluate that policy as part of the system and include its cost. An extra evaluator rerun answers a different question about variability.
Exact comparisons remain useful for fixed labels and verified fields. Open ended responses need task specific criteria or a calibrated model judge.
Integrate the comparison with release review
A useful report names the versions, cases, grader and environment, then lists score changes, segment results, critical violations, cost and latency. Link failures to their inputs and outputs.
Run cheap deterministic checks early. Schedule expensive behavioural suites according to the release process, while ensuring the required suite completes before release approval.
A passing gate establishes performance on the tested conditions. It does not replace a controlled rollout and production monitoring.
Exercise: write a release decision
Use the table above to draft a one paragraph release recommendation. Name the regression, its denominator and the additional evidence needed.
Then add a rule that would block the candidate despite its better overall score. Confirm that the rule would also block a candidate whose dispute cases were missing entirely. That second check prevents incomplete data from appearing as success.