An LLM evaluation dataset is a collection of inputs and expected behaviours used to measure a system. Its value depends on which situations it represents, how labels were established and whether development has leaked into the test set.
Build the dataset from several sources. Production traffic reveals real phrasing and context, while expert written cases cover requirements and rare failures. Carefully reviewed synthetic cases can expand coverage before sufficient traffic exists.
Separate typical use from targeted risk coverage
A traffic weighted sample estimates performance on the observed request mix, subject to sampling limitations. A deliberately balanced or adversarial set answers different questions about coverage.
Keep the distinction visible. If rare cancellation disputes make up half an evaluation, its overall score is not a direct estimate of the production success rate.
A practical dataset can contain a representative sample plus labelled challenge segments. Report each separately or use explicit weights whose meaning is documented.
Turn failures into durable cases
Review complete interactions, including retrieved evidence and tool results where relevant. Name the failure precisely, distinguishing missing policy exception, incorrect order identity, unsupported citation or duplicated action.
Remove unnecessary personal information and replace identifiers consistently. Preserve the relationships that caused the error. Replacing every account ID with the same placeholder can accidentally erase an ownership bug.
Store the source of the case, its label rationale and the system conditions needed to reproduce it.
Use designed and synthetic cases deliberately
Before launch, expert written cases are essential for requirements that have not appeared in traffic. Include empty input, ambiguous requests, conflicting sources and attempts to cross permission boundaries.
Synthetic variations can test spelling, phrasing and language, but they may be less diverse than they appear. Review whether a variation changes the expected answer.
Do not use the same model's unchecked output as both the input generator and the authority for the answer. Labels need a source of truth appropriate to the task.
Prevent leakage between splits
A development set supports iteration. A held out set supports less frequent release decisions. A separate challenge suite can preserve known high impact failures.
Split related cases together. Near duplicate messages, multiple turns from one conversation and questions generated from the same source passage can leak information across a random row split.
When a held out failure becomes a development example, replace or refresh the held out coverage. Repeatedly inspecting and tuning to the same set weakens its independence.
Interpret sample size without false precision
A score of 85 passes in 100 independent representative cases is an estimate. A 95% Wilson interval is approximately 76.7% to 90.7%. It is not a guarantee that production lies in that range, especially if the sample was deliberately selected or cases are correlated.
For two versions evaluated on the same cases, use a paired comparison. Record which cases improved and which regressed. Separate intervals for each overall score do not by themselves settle whether the difference is meaningful.
The NIST discussion of binomial proportion intervals provides statistical background. Choose the method according to the sampling design.
Report coverage alongside the score
A useful report includes the number of cases, their segments, data sources, label process and exclusions. State whether the score is traffic weighted or a challenge suite result.
Small suites are useful for rapid feedback and finding clear failures. Larger independent samples support narrower uncertainty, but no dataset size repairs incorrect labels or missing task coverage.
Exercise: build a dataset manifest
Create a manifest with case ID, segment, source, expected behaviour, label reviewer and split. Add one near duplicate pair and verify that both enter the same split.
Then count coverage for each important user intent and failure condition. Identify at least one requirement that has no case. Add a reviewed case for that requirement before increasing the dataset through bulk generation.