What replaces the unit test when the same input can produce a different output. Writing an eval, building a set from real traffic rather than from imagination, using a model to grade a model and checking the grader, catching a regression before a release, the traces that make any of it possible, and judging an agent on its trajectory as well as its answer.
A garden equipment retailer is writing its first eval for a refund assistant. One case carries the expected behaviour below, and the reviewer refuses it on the grounds that nobody can grade it. What does the strong version of the same case do?
AIt replaces the description with the exact answer the assistant should produce, so an equality check settles the case and no judgement is needed.
BIt raises the threshold from a majority of the repeats to all of them, so a loosely worded requirement cannot be satisfied by a partial answer.
CIt turns the one vague requirement into separate clauses, covering what the reply must contain, what it must not contain and what it must cite, each of which somebody can check by reading.
DIt moves the grading to a model against a rubric, since two reviewers will disagree about helpful and about accurate while a model applies one standard every time.