An LLM evaluation measures how well a model or application performs a defined task. It combines inputs, expected behaviour and a grading method. A release evaluation also includes criteria for accepting or rejecting a change.
Evals complement ordinary software tests. Unit tests still check parsers, permissions, calculations and state transitions. Behavioural evaluations address outputs that may vary or require judgement, such as whether a summary preserves the important facts.
Start with a decision the evaluation must support
“Which prompt should ship?” is a useful question if success has been defined. “Is this model good?” is too broad until the task, users and consequences are specified.
For an order support assistant, define whether the evaluation measures correct eligibility, citation support, appropriate clarification or all three. Keep critical violations separate from an average quality score.
Public benchmarks can help shortlist models. Application evaluations establish performance on the actual instructions, tools and data. Both are measurements, but they support different decisions.
Write a case with enough context to grade
A case should record the request, relevant state or sources, expected behaviour and grader. Store provenance and a stable ID so failures can be investigated.
{
"id": "cancel-dispatched-01",
"segment": "dispatched_orders",
"input": "Can I cancel order A-4471?",
"context": {
"order_status": "dispatched",
"policy": "Cancellation is available only before dispatch."
},
"expected": {
"eligible": false,
"must_not_claim_action_completed": true
},
"grader": "cancellation_contract_v1"
}
The case checks a particular decision. It does not require one exact sentence when several clear explanations would be acceptable.
If the intended evaluation includes retrieval, store the starting corpus and request so retrieval can run. If it isolates generation, provide fixed passages. State which system boundary is under test.
Match the grader to the requirement
Use exact checks for closed labels, numerical results, schema validity and known IDs. Use a rubric for open ended qualities such as completeness or explanation quality. Human review can establish labels and calibrate automated grading.
A model grader can apply a rubric at scale, but it adds its own error. Validate it on human reviewed examples, especially examples of unacceptable output that look fluent.
Avoid giving every criterion the same weight by default. An unauthorised action may block release even when wording and formatting scores improve.
Keep sampling and reliability measures explicit
Run repeated trials when variability affects the decision. Report the average per attempt success rate and, when useful, the proportion of cases that pass every trial.
Those are different metrics. If a case passes twice in three attempts, its per attempt rate is 2/3 and its all trials result is failure. The latter is deliberately stricter and changes when the number of trials changes.
Anthropic's agent evaluation discussion distinguishes tasks, trials, grading and outcomes. The same discipline helps single call evaluations.
Define release criteria before inspecting results
A release rule can require a minimum overall score, minimum scores on important segments and no observed critical violation. Add cost and latency limits when they are product requirements.
For example, an illustrative gate could require 90% overall success, 85% in each named segment and zero unauthorised actions in the suite. Those numbers are a teaching example. Actual thresholds need a risk and workload rationale.
Report denominators and uncertainty. A small test set can expose defects, but it cannot establish a precise production error rate.
A prompt comparison uses the same evaluation cases and grading rules for both versions. The point is to inspect which cases changed, as well as the overall result.
Build a first evaluation deliverable
Choose one narrow task and write an initial set covering normal cases, ambiguity, missing information and known failures. Implement deterministic checks first, then add a rubric only where necessary.
Save the inputs, model configuration, outputs, grader version and per case results. Compare a baseline with one candidate change. Keep development cases separate from a held out release set to reduce repeated tuning to the same examples.
The deliverable is a reviewable comparison with failure details. A single percentage without the cases and grading rules cannot explain whether the change is useful.