Building an eval set

An eval set is the collection of cases a team runs its feature against, and it is the one part of an eval that cannot be reasoned into existence. The shape of a case is settled easily enough, since it is inputs, expected behaviour, a grader and a threshold, but that shape leaves open the question of which inputs, and the answer to it decides everything the score is worth. A perfect grader applied to cases somebody made up at a desk measures how well the feature handles what that person imagined.

The material to build a set from already exists, because every request the feature has served is a case with the answer attached. Getting at it takes a record of what happened, which is why tracing earns as much attention as grading does. Teams that skip the record end up writing cases from memory, and memory is where the bias lives.

The sections below say where cases come from and how a team reads a sample of traffic to find them. A quota per segment follows, then the cases nobody would have invented, then one case written out in the structure a runner reads. The page ends with the three sets a team keeps and with the arithmetic that decides how large a set has to be.

Where the cases come from

Production traffic is the source. Real requests carry phrasing nobody on the team would produce, context the team did not anticipate and combinations that only arrive at volume. A support assistant meets an order number pasted with a trailing space, a question asked in two languages in one sentence and a customer who describes a refund without using the word.

Four places supply cases, in rough order of how much each one is worth.

1. Failures somebody already noticed. A thumbs down, an escalation to a human, a support ticket about the feature itself, an incident write up. Every one of these is a case with a known verdict.
2. Silent failures found by reading. Requests where the system returned something, nobody complained, and the answer was wrong. These are the most valuable and the only way to find them is for a person to read.
3. Ordinary traffic sampled for coverage. The cases that keep a change from breaking the common path while fixing the rare one.
4. Adversarial cases written by hand. The only category that is invented, and it is invented on purpose, because an attacker's input is rare in traffic and cheap to imagine.

The ordering is worth stating because teams usually work it backwards. They write the adversarial cases first, since those are the fun ones, and then never get to the reading that produces the other three.

Reading a sample of traces and naming the failures

Finding the silent failures is a manual exercise and there is no substitute for it. The method borrows from qualitative research and takes an afternoon.

Somebody takes a hundred traces sampled from the last month, reads the whole interaction in each one and writes a short free text note about what went wrong, using no fixed categories. After a hundred notes, the reader sorts them into groups, names each group and counts it. The counting is the point. A team that begins with a list of failure modes will find the failure modes on the list, because a category supplied in advance tells the reader what to notice and quietly tells them what to skip. A team that begins with the traces finds the faults the feature actually has.

The output of that afternoon is a table, and it usually surprises whoever built the feature.

Thirty nine faults in a hundred traces is a high number and a normal one for a first reading. Nobody expects it. The top row is where the next sprint goes, and those fourteen traces are also fourteen cases for the set, copied across with the context they arrived with.

Sampling for coverage

Raw traffic is a poor sample because it is dominated by whatever the feature is used for most. A set drawn in proportion to traffic spends most of its cases on the easy path and leaves the rare intents with two examples each, which is the same as leaving them untested.

So a quota replaces the proportion. Each segment gets a floor that makes a change measurable for that segment on its own, and the high volume segments are capped so they cannot swamp the set.

Complaints take sixty cases on seven per cent of traffic because that is the segment where a bad answer costs the most. Billing takes a hundred and twenty, which is a smaller share of the set than of the traffic, because a regression there is visible within an hour anyway. The quota encodes a judgement about consequence, and writing it down is what lets somebody argue with the judgement later.

The query that draws the sample caps each stratum in one pass.

-- Stratified sample of production traces for the support reply feature.
-- ROW_NUMBER partitioned by segment is what enforces the cap: high volume
-- intents cannot contribute more than the quota however much traffic they
-- carry.
WITH recent AS (
  SELECT
    trace_id,
    captured_at,
    intent,
    locale,
    channel,
    thumbs_down,
    escalated_to_human,
    schema_invalid,
    tool_error,
    refused
  FROM traces
  WHERE feature = 'support-reply'
    AND captured_at >= now() - interval '30 days'
),
ranked AS (
  SELECT
    recent.*,
    -- Anything already marked as a failure sorts to the front of its
    -- stratum, so the quota fills with known faults before it fills with
    -- ordinary traffic.
    ROW_NUMBER() OVER (
      PARTITION BY intent
      ORDER BY
        (thumbs_down OR escalated_to_human OR schema_invalid
          OR tool_error OR refused) DESC,
        random()
    ) AS rank_in_segment
  FROM recent
)
SELECT trace_id, captured_at, intent, locale, channel
FROM ranked
WHERE (intent = 'billing'    AND rank_in_segment <= 120)
   OR (intent = 'order'      AND rank_in_segment <= 60)
   OR (intent = 'product'    AND rank_in_segment <= 60)
   OR (intent = 'complaint'  AND rank_in_segment <= 60)
ORDER BY intent, rank_in_segment;

The ordering clause inside the window function is the part worth copying. It sorts the traces already flagged as problems to the front of every segment, so a quota of sixty fills with sixty known faults where sixty exist and tops up with random traffic where they do not.

The cases nobody would have invented

Two sources produce cases a team would never write from imagination, and both are free.

Every incident becomes a case. The rule is that no incident closes until the input that caused it sits in the set with the behaviour that would have avoided it. A team that follows this rule for a year has an eval set shaped exactly like its own history of being wrong, which is the most useful shape available. A team that does not has the same incident twice.

Every disagreement becomes a case. When two people argue about whether a particular answer was acceptable, the argument is evidence that the expected behaviour was never defined. Settling it produces one case and one clause in the definition, and the clause is worth more than the case.

Adversarial cases are the exception that gets invented, and the reason is arithmetic. An injection attempt that appears in one request in fifty thousand will not appear in a sample of three hundred, so waiting for it in traffic means waiting for the incident. Somebody writes twenty by hand in an hour, drawing on the published attack patterns and on whatever the feature happens to have access to.

One case in the structure a runner reads

A case stored as a sentence in a document cannot be run. The structure below holds everything the previous page called the four parts, plus the provenance that makes the case defensible a year later.

{
  "id": "refund-window-de-0041",
  "split": "held-out",
  "added": "2026-08-21",
  "owner": "support-ai",

  "source": {
    "trace_id": "01J9F2Q7K3B8V0XQZ4T6M2",
    "captured_at": "2026-08-19T09:14:22Z",
    "why_kept": "shipped answer promised a refund outside the policy window"
  },

  "segment": {
    "intent": "billing",
    "locale": "de-DE",
    "channel": "email",
    "account_tier": "business"
  },

  "input": {
    "message": "Die Lieferung kam beschädigt an. Bekomme ich mein Geld zurück? Bestellung 88120.",
    "history": []
  },

  "context": {
    "account": { "order_id": "88120", "delivered_on": "2026-07-02" },
    "retrieved": [
      { "doc": "RET-04", "chunk": 3, "score": 0.81 },
      { "doc": "RET-04", "chunk": 4, "score": 0.77 },
      { "doc": "SHIP-11", "chunk": 1, "score": 0.52 }
    ],
    "now": "2026-08-19T09:14:22Z"
  },

  "expected": {
    "must_contain": [
      "the 30 day return window",
      "that this order falls outside it",
      "the damage claim route, which stays open for 90 days"
    ],
    "must_not_contain": ["any refund approval", "any amount", "any date promise"],
    "must_cite": ["RET-04"],
    "language": "de"
  },

  "grader": "clause-assertions",
  "repeats": 3,
  "pass_if": "all clauses satisfied on every repeat"
}

Four fields in that record do work that a looser format loses. The trace identifier means the case can be replayed against the exact context the system had. The captured timestamp and the frozen now field keep a case about a thirty day window from changing its own answer as the calendar moves. The segment block is what produces a score per segment instead of one average. And why_kept is the sentence that stops somebody deleting the case in eighteen months because they cannot see what it was for.

Three sets and what each is for

One set cannot do three jobs, so a team keeps three.

The third row is the one most teams lack and the one that keeps the other two honest. A held out set built in March is a photograph of March. By September the traffic has moved, a new intent has appeared and a whole segment of questions the set never contained now accounts for a tenth of requests. Drawing fifty fresh traces every week and scoring them by hand is what reveals that, and it costs an hour.

Cases move between the three in one direction only. A live sample case that exposes a failure is promoted into the held out set, and a held out case never moves back. Once a case has been read during development it has told the team what it knows and it cannot surprise anybody again.

How large a set has to be before a difference means anything

A score with no interval around it invites a decision the number cannot support. The arithmetic is ordinary and it is worth doing once.

For a set of a hundred cases scoring eighty five per cent, the ninety five per cent interval runs about seven points either side, so the true rate lies somewhere between seventy eight and ninety two. At three hundred cases the interval narrows to about four points. At a thousand it is about two. A team that wants to detect a three point improvement needs the thousand, which is usually out of reach.

The way out is that an eval set is not two samples. The same cases run through both versions of the prompt, so the comparison is paired, and a paired comparison only counts the cases where the two versions disagreed.

Suppose the three hundred graded cases run against both versions and forty of them change verdict. If twenty four improved and sixteen got worse, the two sided probability of a split that lopsided under a fair coin is 0.27, which is nothing. If twenty seven improved and thirteen got worse, the probability is 0.04 and the change is evidence. The threshold sits at twenty seven of forty, and a team that knows this stops arguing about a set average that moved by one point.

from math import comb

def sign_test(better: int, worse: int) -> float:
    """Two sided probability of a split this lopsided under a fair coin.

    Cases where both versions agreed carry no information about which
    version is better, so they are excluded before this is called.
    """
    n = better + worse
    if n == 0:
        return 1.0
    k = max(better, worse)
    tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
    return min(1.0, 2 * tail)

sign_test(24, 16)   # 0.268  noise
sign_test(27, 13)   # 0.038  evidence

Counting only the cases that moved is also what makes a small set usable. A hundred well chosen cases where thirty change verdict will settle a question that a thousand cases compared as two independent averages would not.

Every number above assumed somebody could say whether a case passed. For a label or a citation that is an assertion anybody can write. For an open ended answer the grader is a person reading a response against a rubric, which works beautifully for twenty cases and collapses at three hundred on every change. The next page is about the substitute, about the three biases it brings with it and about the step that makes its scores worth using.