This lab builds a small evaluation for a support ticket routing prompt. The deliverable is a report showing which cases each prompt handles correctly, where the candidate still fails and whether it meets a stated release rule.
The starter includes two prompts, eight labelled cases and a deterministic grader. It runs offline with authored outputs so the mechanics are reproducible. Optional live mode replaces those fixtures with actual model responses.
Download and run the starter
Download the Python lab bundle, extract it and open a terminal in the extracted folder. Python 3.10 or later is required. The labs use the standard library, so no package installation is needed.
python prompt_eval.py --demo
The script creates a new lab output/prompt directory containing results.jsonl, report.json and prompts.json. It refuses to overwrite an existing output directory. For another run, pass a new path with the out option.
The demo outputs are teaching fixtures, not measured model performance. The script labels them authored_demo and records an authored demo fixture model value.
Understand the task before editing the prompt
The classifier routes each ticket to BILLING, DELIVERY or CLARIFY. Billing takes priority when both billing and delivery issues appear. CLARIFY applies when the message does not identify either issue.
The dataset contains four ordinary cases, two mixed cases and two ambiguous cases. The baseline names the labels but leaves their boundaries underspecified. The candidate adds the priority and missing information rules.
A case such as “My parcel is late and I was charged twice” should route to BILLING under this task definition. The correct label comes from the product rule, not from a general truth about customer support.
Inspect the first report
The authored baseline passes five of eight cases. The authored candidate passes seven of eight, or 87.5%. Its ambiguous segment still passes only one of two cases.
The teaching gate requires at least 85% overall and at least 75% in every segment. The candidate therefore fails despite its improved overall score.
baseline overall 5 / 8
candidate overall 7 / 8
candidate ambiguous 1 / 2
teaching gate fail
Open results.jsonl and locate the candidate's failed case. Each record includes the expected label, response, segment, prompt hash and grade. Live responses also include provider usage and elapsed time.
Eight cases are enough to explain the mechanism and expose these defects. They are not enough to estimate production reliability precisely.
Review the grader's limits
The grader strips surrounding whitespace, checks membership in the three permitted labels and compares with the expected label.
A response such as “The answer is BILLING” fails the format requirement even if a person can infer the decision. That is deliberate because the consuming program expects a label.
This grader suits a closed classification task. An open ended support reply would require separate checks for factual accuracy, source use and response quality. Do not reuse exact string matching as a universal measure of answer quality.
Run actual model responses
Live mode uses the Anthropic Messages API through the small llm_client.py adapter. Set ANTHROPIC_API_KEY through a local environment or secret manager, then use a model ID available to that account.
python prompt_eval.py --live --model YOUR_MODEL_ID --out lab-output/prompt-live-01
This makes paid API requests. One repeat runs eight cases against two prompts, for 16 requests, each capped at 128 output tokens. The script makes no automatic retries and records failed attempts as failures.
For repeated trials:
python prompt_eval.py --live --model YOUR_MODEL_ID --repeats 3 --out lab-output/prompt-live-02
Three repeats make 48 requests. The resulting rate is per attempt. Inspect case level variability as well as the aggregate, and do not interpret three repeated demo fixtures as three independent trials.
Change one instruction and test the result
Edit only the candidate prompt. Add a boundary example that clarifies when CLARIFY applies, then run into another output directory.
Before looking at the result, record the predicted improvement and which segments must not regress. Compare the actual outputs, prompt hashes and per segment scores. If the same case still fails, inspect whether its expected label is well specified.
Add at least four new cases, including a mixed issue, an ambiguous request and a ticket containing an instruction to ignore the routing task. Review the expected labels before generating responses.
Keep a separate set for the final comparison once prompt tuning begins. Repeatedly optimising against the eight starter cases demonstrates development progress, not independent generalisation.
Acceptance criteria and submitted artefacts
Submit the revised prompt, case definitions, baseline and candidate outputs, a report and a short release decision. The decision should explain each remaining failure and the evidence needed before wider use.
A successful lab demonstrates a justified comparison, even when the candidate is rejected. Passing the teaching gate is not a reason to discard inconvenient cases.
Run the included offline contract checks after editing the harness:
python -m unittest -v test_labs.py
The tests check grading and the other starter components without contacting a model service. For another provider, replace the adapter while preserving its text, model, usage and seconds return fields.