AI agent evaluation measures whether an agent completed the task, respected constraints and used acceptable resources. It examines both the final state and the actions taken to reach it.
An agent saying that a file was saved is weaker evidence than checking the file. An agent reaching the right final state after an unauthorised action still violates the task requirements.
Define the task through observable outcomes
Record the starting state, goal, available tools, user authority and completion conditions. Run the task in a controlled environment that can verify the resulting state.
For an order cancellation task, the evaluator might require the intended order to be cancelled once, every other order to remain unchanged and the final message to match the service result.
Keep the test environment separate from production. Reset state between independent trials so one trial's action does not change another trial's starting conditions.
Evaluate constraints without prescribing every step
A reference trajectory can describe required dependencies. Authenticate before reading a private record, verify eligibility before changing it, and obtain any required approval before execution.
Several valid tool sequences may satisfy those conditions. An exact sequence match can unfairly reject an efficient alternative.
Use exact ordering only when order is a requirement. Otherwise evaluate required actions, forbidden actions, argument validity and partial order constraints.
Keep outcome and trajectory scores separate
A useful report includes:
| Measure | What it checks |
|---|---|
| Task success | The required final state or answer |
| Constraint compliance | Permissions, approvals and prohibited actions |
| Tool correctness | Appropriate tools and valid, relevant arguments |
| Efficiency | Time, tokens, calls and unnecessary retries |
| Recovery | Behaviour after a controlled failure |
A single blended score can hide a severe violation behind good formatting or speed. Define which failures block success independently.
Anthropic's evaluation guidance for agents discusses combining graders and checking actual outcomes. Use the environment's evidence wherever it can settle the question.
Measure reliability across repeated trials
Repeated trials reveal whether success is stable. Record all attempts, including failures.
Two metrics are often confused. Pass at k asks whether at least one of k attempts succeeds. Pass to the k asks whether all k attempts succeed. Under a simplified independent trial model with success probability p, these are 1 minus (1 minus p) to the power k, and p to the power k respectively.
At p = 0.8 and k = 3, the corresponding probabilities are 99.2% and 51.2%. One metric describes success with multiple chances; the other describes consistency. Real trials may be correlated, so report observed results and the assumptions behind any formula.
The tau bench paper uses repeated interactions to study agent reliability.
Include failure injection in the evaluation
Test a transient read failure, an invalid identifier, a duplicate write retry, a budget limit and a changed assumption. Verify whether the agent recovers or stops appropriately.
For a timed out write, the correct behaviour may be to reconcile an unknown outcome. A second action with a fresh identity can be a failure even if the final message looks correct.
Exercise: specify a cancellation task
Write a fixture with two orders owned by the same customer, only one of which is eligible for cancellation. Define the request, permitted tools and expected final state.
Add a trial in which the cancellation succeeds but its response is lost. The evaluator should confirm one cancellation, a reused operation identity and an accurate final message.
Then add an unauthorised order. The agent should not read or change it. This checks the execution boundary as well as the model's willingness to follow instructions.