A prompt and context practice is the set of artefacts and habits that lets a team keep a language model feature working while everything around it moves. Everything around it does move, because the traffic changes, the index goes stale, the judge drifts against its labels, the provider retires the model and the people who built the thing move to something else. None of that is an event, and all of it is a slow condition.
A first version of a feature can be built by one person in a fortnight, and keeping it working for two years is a different job. Effort is not what separates the two, because the decisions somebody made in that fortnight only survive if they were written down in a form the next person can read, argue with and measure.
The sections below name the four artefacts a practice rests on and take each one in turn, with what it stops and where it lives. The rhythm that keeps them alive follows, then the question of who owns each piece, and the page ends by returning to where the course began.
The four artefacts a practice rests on
Four things, each of which stops a specific failure that costs a team months.
| Artefact | What it stops | Where it lives | Who keeps it |
|---|---|---|---|
| A prompt registry | A behaviour change nobody can attribute or undo | The repository, reviewed like code | The team owning the feature |
| An eval set that grows | A regression reaching a customer twice | The repository, added to every week | The same team |
| A trace review habit | A failure mode nobody has words for yet | A calendar entry and a rota | Everybody, in turn |
| An experiment log | The same experiment being run twice | The repository, one file per question | Whoever opened the entry |
None of the four is expensive. A registry is a directory of files, a set is a file of cases, a review is forty five minutes on a Thursday and a log entry is a page somebody writes in ten minutes. What makes them hard is that each one costs something today and pays back in a quarter, which is exactly the shape of work that never gets scheduled.
The prompt registry as the one place a prompt lives
Every prompt the organisation runs, including the judges, the classifiers and the small summarising steps nobody thinks of as prompts, has one home with a version, an owner and a changelog. The format is the registry entry from the page on versioning, and the discipline is that nothing else assembles a prompt.
Three rules make the difference between a registry and a directory.
No prompt reaches a model without passing through it. A string built inline in a service, a template in a database row and a system message hard coded in a notebook are all prompts, and each one is a behaviour nobody can version. The test is whether somebody can list every prompt in production from one place.
A tool description counts as a prompt. The model reads it as an instruction, so rewording it changes behaviour. It goes in the registry, it gets a version, and a change to it runs the eval set.
A retired version keeps its reason. The sentence explaining why version 8 was withdrawn is the cheapest thing in the whole practice and the one that saves the most time, because it is read by whoever proposes the same idea in March.
An eval set that grows from production
The set built on the earlier pages was a photograph of one month. A practice keeps it current, and the mechanism is a rule somebody can point at.
- No incident closes until its input is a case. The trace that caused it, the expected behaviour written out, the segment tagged. This one rule does more than any other to make a set resemble the system it tests.
- No argument about an answer closes until it is a case. Two people disagreeing about whether a reply was acceptable means the expected behaviour was never defined. Settling it produces one case and one clause.
- Fifty fresh traces are read every week. Anything that surprises the reader is promoted. This is what catches the intent that did not exist last quarter.
- The whole set is re examined quarterly. Segments are redrawn against current traffic, cases about behaviour the product no longer has are retired, and the quota per segment is set again.
The fourth rule is the one teams skip, and skipping it produces a set that tests a product from two years ago. A case is retired with a note saying why, in the same way a prompt version is, because a deleted case is an argument somebody will have again.
The trace review habit
A number on a dashboard tells a team that something moved. Reading the traces behind it is what tells them what moved, and no amount of instrumentation substitutes for a person reading what the feature actually said to somebody.
Thursday trace review
45 minutes, four people, one of them on a rota
prepared in advance by whoever holds the rota this week
1 The twenty lowest judge scores from the week, with the judge's
evidence quotations beside each response. 15 min
2 Everything flagged refused, schema invalid, tool error or
escalated, grouped by reason rather than read one at a time.
10 min
3 Five traces drawn at random from ordinary traffic, read in full,
by somebody who did not write the prompt. This item is the one
that finds what nobody was looking for. 10 min
4 Deciding what each finding becomes. A case, a ticket, an
experiment or nothing. 10 min
Output, written into the repository before the meeting ends
- new eval cases with expected behaviour filled in, not "todo"
- one experiment log entry per question worth answering
- a name against each, because an action with no name is a wish
Item three earns its ten minutes more often than anybody expects. The low scores and the flags are failures the system already knows how to recognise. Reading an ordinary successful response in full is how a team discovers that every reply has been opening with a sentence nobody would write, or citing the same document whether or not it applies, or quietly dropping the second half of two part questions. None of those trip a single measure on the dashboard.
The rota matters as much as the agenda. A review run by the person who wrote the prompt finds what that person already suspects. Rotating it through the team, including somebody from support, produces the reading that catches something new.
A written record of what was tried
The most expensive thing a team does is run the same experiment twice, and it happens because the first result lived in a chat thread that scrolled away.
experiments/2026-09-18-EXP-041-conditional-citation.md
---
id: EXP-041
prompt: support-reply
opened: 2026-09-18
closed: 2026-09-19
owner: r.okafor
status: adopted, with a change
supersedes: EXP-019 (April 2026, same question, different team)
---
## The question
Does naming the return window and citing the policy document on every
reply raise the grounding score without costing anything else?
## What was predicted, written before the run
Grounding rises on billing and order questions. Nothing else moves.
Agreed in advance: overall must not fall, and no segment may fall by
more than 3 points.
## What happened
overall 84.0% -> 86.0% +2.0
billing 85.0% -> 91.7% +6.7
order 80.0% -> 90.0% +10.0
product 85.0% -> 91.7% +6.7
complaints 85.0% -> 65.0% -20.0 blocked the gate
40 verdicts moved, 23 better, 17 worse, two sided p = 0.43
## What was learned
An instruction that is right for a question is wrong for a complaint.
Opening a reply to a damaged delivery with a policy citation fails the
rubric criterion about answering the question asked. The rule needed a
condition and never a stronger statement.
## What shipped
Version 9, with the citation conditional on the message being a
question about policy. 91.7% overall on the graded cases, every segment
above its baseline. Version 8 retired the same day.
## What not to try again
An unconditional citation instruction on this prompt. Measured twice
now, in April by the platform team and in September here, with the same
result both times.
The last section is the reason the file exists. Everything above it is a record, and that section is an instruction to the future. A team with twenty of those files has a map of what does not work on its own product, which is knowledge nobody else has and nobody can buy.
Two habits keep the log usable. The prediction is written before the run, which is what turns a result into evidence rather than a story assembled afterwards. And an entry is opened for questions that were abandoned as well as for those that were answered, because the fact that somebody tried and stopped is itself worth knowing.
The rhythm that keeps all four alive
Artefacts decay unless something touches them on a schedule. The schedule below is what a team of five running two model features can carry.
| Cadence | What happens |
|---|---|
| Every change | The held out set runs in the build and the gate reads the segments |
| Every day | Somebody on rota reads the dashboard, including refusal rate and cost |
| Every week | The trace review, new cases promoted, the live sample scored |
| Every month | The judge is revalidated against human labels and the rollback path is exercised |
| Every quarter | Segments redrawn, stale cases retired, the model lifecycle file reconciled against what the providers have published |
The monthly row carries the two things nobody wants to do. Revalidating the judge means somebody grading eighty responses by hand for an afternoon, and exercising a rollback means deliberately taking a working system backwards on a Tuesday. Both are skipped for months at a time and both fail silently while they are skipped, which is precisely why they belong on a calendar and never on a list of good intentions.
Who owns each piece
A practice with no named owner belongs to whoever most recently felt guilty about it, which is a rotation with no schedule.
The team that ships the feature owns the prompt registry, the eval set and the experiment log. Those three are part of the product and they are reviewed the way the product's code is reviewed.
The trace review is owned by a rota that includes somebody who talks to customers. That person reads a response differently from the person who wrote the prompt, and the difference is the whole value of the session.
The judge, the eval runner and the lifecycle file are shared across features, so they belong to whoever owns the shared platform. If no such team exists, they belong to the feature team that built them first, and the important thing is that somebody's name is on the file.
What the whole course was about
The first page of this course described a mechanism. A model reads a sequence of tokens and predicts the next one, conditioned on everything already in the window and on nothing else. It holds no memory of yesterday's conversation, it cannot see a document it was not shown, and it has no way to tell an instruction the team wrote from a sentence a stranger planted in a retrieved passage. Everything else in this course follows from those three facts.
Every technique across the thirty three pages before this one is a decision about what the window contains at the moment of prediction. A few shot example is material placed there to demonstrate a shape the instruction could not describe. Retrieval fetches material and puts it there because the answer depends on facts the model was never trained on. Compression takes material out to make room, and isolation gives a subtask a window of its own so that one task's material cannot crowd out another's. A tool definition is text saying what the model may ask the world to do, and the model reads it exactly as it reads everything else.
Module five and module six are the same idea seen from the other end. A trace is a record of what was in the window and what came out. An eval asks whether what was in the window was enough to produce an acceptable answer. A judge reads that answer against a rubric, which is itself a window somebody assembled. Prompt injection is a stranger putting something in the window, and every defence against it limits what the model can do with what it found there. A model migration changes the thing reading the window, which is why the prompts stop working.
That is the skill the course set out to teach. A team that can say what is in the window on any given request, why each piece is there, what it cost, what it displaced and how anybody would find out if it stopped working has the whole of it. The practice on this page is what keeps that answer true after the traffic changes, the model moves and the people rotate.