Versioning a prompt means giving it an identifier, a number, an owner and a record of what changed, so that a release can be described, measured and undone. The controls around a feature all change what it does in production. Narrowing a tool scope breaks a request that used to work, a gate in the wrong place makes a product unusable, and a validation rule that refuses too often becomes a support queue. All of them reach production the same way a prompt change does.
Teams arrive at this late, because a prompt looks like text. It is stored in a string, it reads like a paragraph, and editing it feels closer to fixing a typo than to deploying code. The behaviour it controls says otherwise. One clause added to a system prompt moved a whole segment twenty points in the worked comparison on regression testing, and that change went through a build gate precisely because it had a version on both sides.
The sections below treat the prompt as an artefact and show the registry entry a request resolves against. Sticky assignment to a share of traffic follows, then the model version held constant while the prompt moves, then the rules that stop a rollout on their own. The page ends with what has to change together.
The registry entry a request resolves against
A registry is a store of prompt versions with their status, their measurements and their rollout state. It can be a directory of files in the repository, and for most teams it should be, because a file has review, history and a diff already.
# prompts/support-reply/registry.yaml
id: support-reply
owner: support-ai
description: Answers a customer support email using retrieved policy.
model_profile: support-reply # resolved in config/models.yaml
schema: support_reply@3
judge: judge/support-rubric@4
current: 9
canary:
version: 10
share_per_mille: 50 # 5 per cent of users
started_at: 2026-09-24T08:00:00Z
ends_at: 2026-09-26T08:00:00Z
stop_if:
judge_score_below: 0.86 # current sits at 0.931
refusal_rate_above: 0.04
schema_invalid_rate_above: 0.01
p95_latency_ms_above: 6000
versions:
- version: 9
status: current
created: 2026-09-19
supersedes: 8
author: r.okafor
changelog: |
8 -> 9 Made the policy citation conditional on the message being a
question about policy. Version 8 opened every reply to a
complaint with a citation and lost 20 points on that segment.
evidence:
set: held-out
cases: 360
run: 2026-09-19T14:22:00Z
overall: 0.931
segments: { billing: 0.917, order: 0.900, product: 0.917,
complaint: 0.933, adversarial: 1.000 }
- version: 10
status: canary
created: 2026-09-24
supersedes: 9
author: r.okafor
changelog: |
9 -> 10 Added a sentence naming the damage claim route, which the
August trace reading found missing in 9 of 100 replies.
evidence:
set: held-out
cases: 360
run: 2026-09-23T17:05:00Z
overall: 0.939
segments: { billing: 0.925, order: 0.900, product: 0.933,
complaint: 0.950, adversarial: 1.000 }
- version: 8
status: retired
retired_on: 2026-09-19
reason: regression on complaints, see the comparison in run 2026-09-18
Five things in that file are worth arguing for individually.
One current version and at most one canary. Two experiments running at once on the same prompt produce a result neither of them can claim. The constraint is annoying and it is what makes a measurement attributable.
The model profile resolved elsewhere. The prompt names a profile and the profile names a model. That indirection is what allows the prompt to change while the model is held, and the model to change while the prompt is held.
Evidence attached to each version. A version is never separated from the measurement that justified it, so anybody asking why version 9 ships gets the run date, the set and the per segment figures.
A retired version with a reason. Version 8 stays in the file with the sentence explaining why it went. Six months later, somebody proposing the same idea reads that sentence before writing the code.
Stop conditions written before the rollout starts. Those four numbers are the whole of the automatic rollback, and agreeing them in advance is what stops the conversation during an incident from being about whether the number is bad enough.
Sticky assignment to a share of traffic
A canary sends a fraction of traffic to the new version. The fraction has to be stable per user, because a customer who gets a different assistant on each message is a customer nobody can help and a complaint nobody can reproduce.
import hashlib
from datetime import datetime
def resolve_version(registry, prompt_id, user_id, now=None):
"""Return the prompt version this request should use.
Assignment is a hash of the user and the prompt id, so a person sees
one version for the whole of a rollout and for any rollout of the
same prompt afterwards. Drawing at random per request would give one
customer three assistants in a single conversation.
"""
now = now or datetime.utcnow()
entry = registry[prompt_id]
canary = entry.get("canary")
if not canary or now >= canary["ends_at"] or canary.get("stopped"):
return entry["current"]
digest = hashlib.sha256(f"{prompt_id}:{user_id}".encode()).hexdigest()
bucket = int(digest[:8], 16) % 1000
return canary["version"] if bucket < canary["share_per_mille"] else entry["current"]
Two details in that function are the ones teams get wrong.
The hash includes the prompt identifier, so the same user is not permanently in every canary the organisation ever runs. Hashing the user alone produces a cohort of people who receive every experimental version of everything, and their experience stops representing anybody.
The version is resolved once per request and recorded in the trace. Resolving it again inside the same request, after a retry or a fallback, produces a conversation assembled from two prompts, and no trace will explain the result.
Holding the model version constant while the prompt changes
One variable at a time is an old rule and this is where it earns its keep. The model profile lives in its own file for exactly this reason.
# config/models.yaml
# A dated version on every line. An alias here means the provider
# decides when this feature's behaviour changes.
profiles:
support-reply:
model: example-model-2026-06-30
temperature: 0.2
max_tokens: 600
timeout_ms: 20000
support-judge:
model: example-judge-2026-05-12
temperature: 0
max_tokens: 400
summariser:
model: example-small-2026-07-14
temperature: 0.0
max_tokens: 250
With the two files separate, every release changes one thing and the measurement says which.
| What is changing | What is held | What the change has to pass |
|---|---|---|
| The prompt | Model profile, retrieval index, judge version | The held out set against the current prompt |
| The model | Every prompt version, retrieval, judge | The held out set for every prompt on that profile |
| Retrieval or chunking | Prompts, model, judge | The held out set, with retrieval scored separately |
| The judge | Nothing ships on the judge alone | Revalidation against the human labels |
The fourth row is the one that trips people. A new judge version rescores the entire history, so yesterday's 0.917 and today's 0.902 may describe an unchanged system. A judge change is a measurement change and it is announced as one, with the old and new judge run over the same held out set so the offset is known.
The rules that stop a rollout on their own
A canary that needs a person to notice it is failing is a canary that fails for as long as the person is at lunch. The four conditions in the registry entry are evaluated on a schedule against the canary cohort alone.
$ rollout status support-reply
canary version 10, 5.0% of users, 14h into a 48h window
cohort 3,812 requests from 402 users
control 76,190 requests from 7,930 users
canary control stop rule state
judge score, sampled 0.921 0.918 below 0.860 ok
refusal rate 0.019 0.021 above 0.040 ok
schema invalid rate 0.002 0.003 above 0.010 ok
p95 latency 4,180ms 4,050ms above 6,000ms ok
cost per request $0.0141 $0.0138 reported only +2.2%
no stop condition met. next evaluation in 15 minutes.
Three properties of that report make it usable. It compares the canary against the control cohort over the same window, so a site wide problem moves both columns and stops nothing. It names the rule next to the number, so anybody reading it knows what would have to happen. And it reports cost without a threshold, because a cost rise is a decision for a person and never a reason to roll back automatically.
Sample size decides how long the window has to be. Five per cent of traffic on a feature serving eighty thousand requests a day gives roughly four thousand canary requests in fourteen hours, and a judge sampled at one in twenty of those gives about two hundred scored responses. Two hundred is enough to catch a collapse and nowhere near enough to resolve a two point difference. The window is set by the size of the drop worth catching, and a rollout that wants to detect small differences either runs longer or takes more traffic.
Rolling back
A rollback is the cheapest safety property available and it stays cheap only if it is exercised. The registry makes it one operation, because the current version is a field.
$ rollout stop support-reply --reason "refusal rate 0.061 on the canary"
support-reply
canary version 10 stopped at 2026-09-24T22:11:04Z
all traffic resolves to version 9 from the next request
version 10 marked: stopped, reason recorded, evidence retained
in flight conversations: 214
these finish on version 10, because switching a prompt mid
conversation produces a reply that contradicts the one before it
The last clause is the part that needs a decision in advance. Draining an in flight conversation on the version it started with is almost always right for a chat feature and almost always wrong for a security fix, where the point is to stop the behaviour immediately. Both behaviours belong in the tooling, and which one applies is a flag on the rollback and never an argument at the time.
Two habits keep a rollback real. A version stays deployable for at least a month after it is superseded, because the fastest fix for a problem discovered on Friday is the version from Tuesday. And the rollback path runs on a schedule whether or not anybody needs it, for the same reason a backup gets restored on a schedule.
What has to change together
Some things cannot move independently however much a team would like them to.
The prompt and the output schema. A prompt that starts producing a new field and a consumer that starts requiring it have to arrive in an order that works. The order that works is the consumer first, accepting both shapes, then the prompt, then the removal of the old shape once no traffic produces it.
The prompt and the tool definitions. A tool description is read by the model as an instruction, so renaming a tool or rewording its description is a prompt change wearing different clothes. It goes through the same gate.
The prompt and the retrieval index. A prompt written around passages that were four hundred tokens long behaves differently against passages of eight hundred. A reindex with a new chunking strategy runs the held out set for every prompt that reads from it.
Everything above gets a change safely into production and says nothing about what happens afterwards. A feature that passed its gate, rolled out cleanly and sat at 0.93 on the held out set can still be failing quietly for a class of user nobody sampled, and it will fail by producing something plausible rather than by throwing an error. Watching for that is a different discipline, and it is the next page.