Evaluating an agent

Evaluating an agent means scoring the path it took as well as the answer it produced. A trace is what makes that possible, because it holds every step the agent decided to take, every tool it called and every argument it supplied, and reading only the last span of one discards most of what was recorded.

The reason to look at the path is that an agent has many ways to arrive at the right answer and only some of them will keep working. A run that called a write tool twice and happened to be idempotent, or that guessed an argument and recovered, produces an output indistinguishable from a clean run. It will fail differently tomorrow, and the outcome score gives no warning at all.

The sections below separate the two questions an agent evaluation answers, then set a reference trajectory against a real run. The rules for matching a run against that reference follow, with a scorer that keeps four numbers apart, then the measures beyond correctness, and the page ends with reliability across repeats.

The two questions an agent evaluation answers

An agent evaluation asks two things and a team needs both answers.

Did it reach the right outcome. This is the ordinary eval question from earlier in the module, applied to whatever the agent was for. The refund was created, the ticket was routed to the right queue, the report contained the four figures somebody asked for. It is graded by an assertion against the resulting state where the state is checkable, and by a judge against a rubric where the output is text.

Did it get there sensibly. Whether the tools it called were the right ones, whether the arguments were correct, whether the order made sense, how many steps it took and what it cost. This is graded against a reference path a person who knows the domain would approve.

The two come apart in both directions, and both directions are informative. An agent that reaches a wrong outcome through a sensible path usually has a missing tool or a missing piece of context, which is a cheap fix. An agent that reaches the right outcome through a chaotic path has a reliability problem nobody has measured, which is an expensive one.

A reference trajectory set against a real run

The reference is written once, by somebody who knows what the task requires, and stored with the case. The comparison below is a single run against it.

task
  "Order 88120 arrived damaged. Refund it if it qualifies, otherwise
   explain what the customer can do instead."

reference trajectory, written by a support lead
  1  lookup_order(order_id="88120")
  2  get_policy(topic="damaged_delivery")
  3  create_claim(order_id="88120", reason="damaged")
  4  final answer, naming the claim reference and the 90 day window

actual trajectory, repeat 3 of 5
  step  call                                          verdict
  ----  --------------------------------------------  ------------------
   1    lookup_order(order_id="88120")                 matches step 1
   2    search_orders(query="88120")                   extra
   3    get_policy(topic="returns")                    wrong argument
   4    get_policy(topic="damaged_delivery")           matches step 2
   5    create_claim(order_id="88120", reason="damaged")  matches step 3
   6    create_claim(order_id="88120", reason="damaged")  DUPLICATE WRITE
   7    final answer, naming the claim reference and the 90 day window

scores
  outcome           pass    the answer is correct and complete
  coverage          pass    every reference call appears, in order
  argument accuracy 0.80    4 of 5 calls to a reference tool were right
  extra steps       +3      7 steps against a reference of 4
  duplicate writes  1       create_claim ran twice with identical arguments
  cost              $0.21   against a reference path costing $0.09
  trajectory        FAIL    duplicate write on a tool that creates money

The row that matters is the sixth step. The claim was created twice with identical arguments, and the agent then described one claim reference to the customer because that is what the last tool result contained. Nothing in the answer is wrong. A finance team will find two open claims against one order next week and nobody will connect them to this conversation, because the outcome score for this run said pass.

One earlier step is worth noticing for a different reason. At step two the agent called a search tool it did not need, which cost a round trip and a few hundred tokens and changed nothing. One of those per run is a rounding error. The same redundant call on every run of a feature serving a hundred thousand requests a day is a line item.

Rules for matching a run against a reference path

A comparison against a reference needs a rule for what counts as matching, and the right rule depends on whether the order of the steps carries meaning.

Exact match is tempting and it is usually the wrong rule. It fails a run that did everything required and also checked something sensible, which turns the measure into a test of conformity to one person's habits. The rule that fits most agents is in order with extras tolerated, paired with a separate count of how many extras there were, so tolerance does not mean silence.

Argument correctness is scored on its own and never folded into the call comparison. An agent that calls the right tool with a wrong argument has a different problem from one that calls the wrong tool, and the two have different fixes. The first is usually a tool description that does not say what the parameter means. The second is usually a tool description that does not say what the tool is for.

The scorer that keeps four numbers apart

One number for a trajectory hides which part moved, so the scorer returns four.

from dataclasses import dataclass

WRITE_TOOLS = {"create_claim", "issue_refund", "send_message", "cancel_order"}


@dataclass(frozen=True)
class Call:
    name: str
    args: dict

    def key(self):
        return (self.name, tuple(sorted(self.args.items())))


def covers(reference_keys, actual_keys, mode):
    """Whether the actual run contains every reference call."""
    if mode == "exact":
        return reference_keys == actual_keys

    if mode == "in_order":
        position = 0
        for key in actual_keys:
            if position < len(reference_keys) and key == reference_keys[position]:
                position += 1
        return position == len(reference_keys)

    remaining = list(reference_keys)
    for key in actual_keys:
        if key in remaining:
            remaining.remove(key)
    return not remaining


def score_trajectory(reference, actual, mode="in_order"):
    """Four separate numbers describing how a run reached its answer.

    Averaging these into one score is what hides a duplicate write
    behind three other measurements that all improved.
    """
    reference_keys = [c.key() for c in reference]
    actual_keys = [c.key() for c in actual]
    reference_names = {c.name for c in reference}

    called_a_reference_tool = [c for c in actual if c.name in reference_names]
    with_right_arguments = [c for c in called_a_reference_tool
                            if c.key() in set(reference_keys)]

    writes = [c for c in actual if c.name in WRITE_TOOLS]

    return {
        "coverage": covers(reference_keys, actual_keys, mode),
        "argument_accuracy": (
            len(with_right_arguments) / len(called_a_reference_tool)
            if called_a_reference_tool else 0.0
        ),
        "extra_steps": len(actual) - len(reference),
        # A write repeated with identical arguments is the failure that
        # costs money. It is counted on its own and it blocks on its own.
        "duplicate_writes": len(writes) - len({w.key() for w in writes}),
    }


def trajectory_verdict(scores):
    """Any duplicate write fails, whatever the other three numbers say."""
    if scores["duplicate_writes"] > 0:
        return "fail", "a write tool ran twice with identical arguments"
    if not scores["coverage"]:
        return "fail", "a required call is missing"
    if scores["argument_accuracy"] < 0.9:
        return "fail", f"argument accuracy {scores['argument_accuracy']:.2f}"
    if scores["extra_steps"] > 4:
        return "warn", f"{scores['extra_steps']} steps above the reference"
    return "pass", ""

The hard gate on duplicate writes deserves its own line in the code because it deserves its own line in the review. Every other measure on this page is a matter of degree. A write repeated with identical arguments is a defect whatever else the run did well, and a percentage that averages it away is worse than no percentage.

The measures beyond correctness

Four more numbers come free from the trace and every one of them predicts a problem before a user reports it.

Step count. The distribution matters more than the mean. A feature whose runs cluster at four steps with a tail at nineteen has a class of input that sends the loop wandering, and finding that class is a filter and a read.

Cost per completed task. An agent that solves a task in eleven calls solves it. Whether it should is an arithmetic question that the outcome score cannot answer, and the number is the same number that appears on the invoice.

Latency to the first useful thing. A run that takes forty seconds and shows the user nothing for the first thirty five has a different problem from one that takes forty seconds and streams progress.

Recovery behaviour. A tool returns an error on perhaps one call in fifty in any real system. What the agent does next is a measurement in its own right, and the three outcomes are retrying sensibly, giving up cleanly and inventing a result. The third one is the reason error paths belong in the eval set.

Reliability across repeats

An agent that succeeds once has not been shown to succeed. This is the single largest difference between evaluating an agent and evaluating a single call, and the vocabulary for it comes from a benchmark published in June 2024 by Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan at Sierra.

Their benchmark put an agent into simulated conversations with a user and a set of domain tools, in a retail setting and an airline setting, with written policies the agent was required to follow. Alongside the ordinary success rate they reported a measure they wrote as pass to the power of k, which is the share of tasks an agent solved on every one of k attempts.

The gap between the two measures was the finding. A leading model of the time, used with function calling, reached about sixty one per cent on the retail tasks and about thirty five per cent on the airline tasks when each task was attempted once. Required to succeed on all eight attempts at the same retail tasks, it fell to roughly twenty five per cent. The same agent and the same tasks, with a difference of more than thirty points depending only on how many times it was asked.

def pass_hat_k(runs_by_task, k):
    """Share of tasks that succeeded on every one of k repeats.

    The familiar pass@k asks whether a task succeeded at least once,
    which flatters an agent that is mostly guessing. pass^k asks whether
    it succeeded every time, which is the question somebody relying on
    the agent is actually asking.
    """
    return sum(all(runs[:k]) for runs in runs_by_task.values()) / len(runs_by_task)


def reliability_curve(runs_by_task, max_k=8):
    """The shape that tells a team whether an agent is ready.

    A curve that falls off a cliff between k=1 and k=3 describes a
    feature that will work in a demonstration and generate support
    tickets in production.
    """
    return {k: pass_hat_k(runs_by_task, k) for k in range(1, max_k + 1)}

The practical instruction is to run every agent case at least five times and report the curve rather than the first number. A team that has only ever run each case once has measured something other than whether the feature works.

A correct answer reached by a broken path

Everything above comes down to one claim worth stating plainly. An agent's output is the end of a process, and a process that produced the right answer by an accident of ordering will produce a wrong one when the ordering changes. Model versions change the ordering. So does a slower tool, a longer input, a retry and a full context window.

That is why the trajectory measures are the leading indicators and the outcome score is the lagging one. Duplicate writes, argument errors, wandering step counts and a reliability curve that collapses at three repeats all appear before the incident. A team watching only the outcome score learns about them from a customer.

Every measurement so far has been taken against cases a team chose, with inputs a team froze, in an environment a team controlled. Real users send things nobody sampled, at volumes no test set reaches, and some of them are trying to make the system do something it was never meant to do. The next module is about what happens then.