All glossary terms
A Agent engineering Reliability

Agent evaluation

Agent evaluation is establishing whether an agent is doing its job, judged on the route as well as the result. Four different activities share the name, and most teams do one of them. The instruments for the most-quoted one turn out to be exploitable.

Definition

Agent evaluation means checking whether an AI agent is doing its job well and how it got there. The tricky part is that ‘agent evaluation’ is an umbrella term for four different types of checks, and most teams only run one of them. To make it worse, the most used method also turns out to be the easiest one to game, since agents can pass its tests without genuinely doing good work.

What is agent evaluation?

Agent evaluation establishes whether an agent is working, and it has to judge the route as well as the result. An agent that reaches the right answer through a dangerous or wasteful sequence of tool calls has still failed. That is why the practice looks nothing like model testing.

This is explained in more detail under AgentOps. In short, a model can be scored against a labelled dataset because there's one correct answer. An agent completing a multi-step task usually has several acceptable paths and no single right one. So evaluation shifts from checking correctness to judging quality.

That shift is what this page is really about. Without one expected output to check against, you need:

  • A way to score the path the agent took
  • A way to define what good looks like
  • A way to keep checking after launch
  • A way to know if your scoring method itself is reliable

These are four separate problems, each with its own answer.

There is no single agent quality score, and anyone offering one has hidden a weighting decision from you. Task completion, trajectory validity, tool-call correctness, cost, latency, and intervention rate can move in opposite directions. An agent that completes more tasks by taking twice as many steps has improved on one axis and regressed on another. Collapsing that into one number requires deciding how much a step is worth relative to an outcome, which is a business judgement rather than a measurement.

What are the four things called agent evaluation?

Capability benchmarking, pre-release testing, production measurement, and incident review. They answer different questions, use different methods, and have different owners. Confusing them is why a team can hold an impressive benchmark number and still have no idea whether its own agent works.

Separating the four is the most useful thing on this page, because vendors typically sell one of them and buyers usually need all four.

A timeline with four activities. Before you build, capability benchmarking compares models. Before you ship, pre-release testing runs a golden set against your own tools and data. While it runs, production measurement samples live traffic. After it fails, incident review reconstructs a single trajectory. A note reads that most teams do one of the four and assume they have covered the others.
The four distinct activities called agent evaluation, when each happens, and what each cannot tell you
Activity When and what it answers Method What it cannot tell you
Benchmarks Before you build. Can a model do this class of task at all? Public suites, standardized tasks, leaderboards Whether your agent works on your data with your tools
Pre-release Before you ship. Does our agent do the job we built it for? A curated case set of your own, scored on outcome and trajectory How it behaves on traffic nobody anticipated
Production While it runs. Is it still working, and is it drifting? Sampled scoring of live runs, plus outcome telemetry Why a specific run went wrong
Incident After it fails. What happened in this particular run? Trace reconstruction across the full span tree Whether the problem is systemic or a one-off

Read the last column downward and the dependency becomes obvious. Each activity has a blind spot that the next one covers, which is why doing one thoroughly does not substitute for the others. The most common configuration in practice is production monitoring without systematic pre-release testing, which means teams can watch their agents failing and have no mechanism for preventing the failures.

Why do agent benchmark scores mislead?

Three reasons: benchmarks test a different workload than yours, high scores don't guarantee the agent will succeed on repeated attempts, and the scoring methods themselves can be gamed. That third point is a recent finding, and it changes how much trust a published score deserves.

Benchmarks are still useful for choosing a model. The mistake is treating them as proof that a deployed agent will work, and there are now three separate reasons that assumption breaks down.

  • They are not your workload. Public suites use clean inputs, predictable tool responses, and controlled environments. Production agents face ambiguous requests, flaky third-party APIs, rate limits, and adversarial input. No benchmark was built to test your schemas, your tools, or your compliance rules.
  • A high score is not reliability. Most benchmarks report whether a task was solved once. Production retries the same task across different sessions, and success on one attempt is a much weaker claim than success on every attempt. Metrics that measure success across repeated attempts, such as the pass-at-k family, are considerably closer to what production reliability means, and few benchmarks report them.
  • The instruments have been shown to be gameable. In April 2026, researchers at Berkeley's Center for Responsible Decentralized Intelligence reported that every major agent benchmark they examined contained exploitable evaluation mechanisms, allowing near-perfect scores without genuinely completing the tasks. Examples included validation answers being publicly available and reference answers being readable from task configuration. At least one major lab has stopped reporting scores on an affected suite after confirmed evaluation-set leakage.

A useful research development is worth noting alongside this. Work published by the UK AI Safety Institute in mid-2026 suggests agent capability is better understood as a curve across compute budget than as a fixed score, which implies benchmark numbers need a stated budget to be interpretable at all. Expect published scores to become more heavily qualified rather than more comparable.

What should you measure when evaluating an agent?

Six measures cover most cases: task completion, trajectory validity, tool-call correctness, intervention rate, cost per completed task, and reliability across repeated attempts. Accuracy is not among them, because a multi-step task rarely has a single correct output to be accurate against.

The set below is deliberately small. Instrumenting six things well is worth considerably more than reporting twenty, and each of these connects to a decision somebody actually makes.

Six measures for agent evaluation, what each captures, and the decision each one informs
Measure What it captures Why it matters
Completion Share of runs that finished the task end to end, not per-step accuracy The only measure a business stakeholder recognizes
Trajectory Whether the path taken was a reasonable one, scored against a reference or a rubric A right answer by a dangerous route is still a failure
Tool calls Correct tool selected, correct arguments, graceful recovery on failure Where most agent errors actually originate
Intervention Proportion of runs needing a human, and whether it is falling The clearest signal of whether the agent is earning its place
Cost Spend and latency per completed task, not per model call Per-call cost hides agents that succeed expensively
Reliability Success across repeated attempts at the same task, not a single pass Production retries; a one-shot score overstates it

Two of these deserve emphasis because they are the ones most often missing. Cost per completed task catches the agent that improved its completion rate by taking three times as many steps, which per-call cost reporting hides entirely. And reliability across repeated attempts is the measure that separates a demo from a deployment, because the same task arriving twice should not succeed once.

Why does an LLM judge need its own evaluation?

Because a model scoring another model is a measurement instrument with an unknown error rate. Most teams adopt LLM-as-judge and never check its agreement with human judgement, which means they are managing an agent by a number whose reliability nobody has established.

This is the gap that undermines more evaluation programmes than any other, and it is rarely discussed because the alternative is uncomfortable: validating a judge requires human labour, which is the thing the judge was adopted to avoid.

The problem is not that judges are bad. Used carefully they are the only tractable way to score trajectories at volume. The problem is that they have documented, systematic biases and are treated as neutral.

  • Length bias. Judges reward longer responses independently of quality, which quietly pushes an agent toward verbosity.
  • Position bias. In side-by-side comparisons, the order in which options appear affects which one wins.
  • Self-preference bias. A judge tends to prefer output produced by the same model family, which matters if you judge with the model you build on.
  • Non-determinism. The same trajectory scored twice can receive different scores, so a change in your metric may reflect nothing at all.

Research in this area is direct about the consequence. Reviews of automated evaluation note that most published work has not calibrated LLM-as-judge quality against human raters, which leaves the reliability of the auto-evaluator itself in question. The teams that do calibrate report the agreement rate as a figure alongside their results, which is what turns a judge from an assumption into an instrument.

If you use a model as a judge, the judge needs an eval too. The practice is straightforward and almost nobody does it: take a sample of runs, have humans score them against the same rubric, and measure how often the judge agrees. That agreement rate is the confidence interval on every other number you report. Re-measure it whenever the judge model, the rubric, or the agent changes, because all three move. Two mitigations help alongside it: require the judge to state its reasoning before scoring, which reduces the biases above, and use deterministic checks wherever a rule can express the requirement, because a function returning zero or one has no opinions to be biased by.

How do you build an agent evaluation practice?

Layer your checks by cost. Run deterministic checks on everything, use an AI judge on a sample of cases, and have humans review a smaller subset to keep that judge calibrated. Build your test cases from real failures, not hypothetical ones. And treat evaluation as an ongoing process, not a one-time check before release.

The layering matters because evaluation economics determine evaluation practice. A full suite costs many model calls per run, so teams that ignore the economics quietly reduce frequency until evaluation happens once before launch.

A pyramid of three evidence layers. The wide base is deterministic checks, cheap enough to run on every run. The middle layer is LLM-as-judge scoring, run on a sample. The narrow top is human review, expensive and run on a small sample, with an arrow showing that human review calibrates the judge layer beneath it. Axes on the side show cost rising and coverage falling toward the top.

  1. Put deterministic checks at the base

    Anything expressible as a rule should be a function returning pass or fail: schema validity, required fields present, a forbidden tool not called, spend under a ceiling, no loop beyond a step count. These are cheap enough to run on every single run, and they have no biases to correct for. Build these before anything else.

  2. Use a judge for what rules cannot express

    Trajectory reasonableness, tone, and whether an answer addressed the actual question need judgement. Run the judge on a sample rather than every turn, because a full model call per judgement is too expensive to apply universally and adds latency to production traffic.

  3. Reserve humans for calibration and the hard cases

    Human review is the most expensive layer and has two jobs that nothing else can do: establishing what good behaviour means for your users, and measuring whether the judge agrees with them. Sample small and deliberately, weighted toward ambiguous and high-risk runs.

  4. Build the case set from real failures

    Three sources, in ascending order of value. Hand-built cases covering your most critical scenarios form the anchor set, and a small number of well-constructed ones is worth more than many careless ones. Production traces of runs that actually failed give the highest ecological validity. Synthetic variations expand coverage cheaply once the anchors exist. Constructing a single good agent case with its task, tool environment, and expected trajectory can take hours, so expect this to be the real cost of the programme.

  5. Score spans, not just outcomes

    Capture the full nested tree of model calls, tool calls, and arguments, then score at the span level as well as the run level. This is what connects a failing score to the specific step that caused it, which is the difference between knowing something is wrong and being able to fix it. The tracing requirements are covered under AgentOps.

  6. Run it continuously, and refresh it

    A suite that passed at release describes the day it ran, because the model, the tools, and the data all move underneath the agent. Run against live traffic on a sample, alert on movement rather than on absolute thresholds, and add every production failure to the case set so the same regression cannot ship twice.

Which category of tooling supports agent evaluation

Three categories, and most teams end up with two. Evaluation platforms provide case management, judge orchestration, and trajectory scoring, usually with tracing built in or integrated. Observability platforms increasingly add evaluation on top of the traces they already collect, which suits teams whose operations already sit there. Agent management platforms cover the release gate and runtime policy that evaluation results should feed into, which is where an evaluation finding becomes a decision rather than a dashboard. Standard telemetry conventions for generative AI are worth insisting on in all three cases, so your evaluation data is not locked to one vendor's schema.

Frequently asked questions about agent evaluation

If your question is not here, our team will answer it directly.


Talk to a Specialist →
What is agent evaluation?

Start with what makes it different from testing ordinary software. Conventional tests assert that a known input produces a known output. An agent given the same task twice may take two different valid routes, so there is often no single expected result to assert against. Evaluation therefore shifts from checking correctness to judging quality: was the outcome right, and was the path a reasonable way to reach it. That judgement has to be repeated continuously, because an agent's behaviour changes without anyone deploying anything.

How is agent evaluation different from model evaluation?

A model produces an output that can be compared against a labelled answer. An agent produces a trajectory: a sequence of decisions, tool calls, and intermediate results leading to an outcome. Two different sequences can both be correct, and one sequence can reach the right outcome through steps nobody would accept, such as calling a tool it should not have touched. So model evaluation measures accuracy against ground truth, while agent evaluation scores both the outcome and the path, which requires judgement rather than comparison.

What metrics should you use to evaluate an AI agent?

Six cover most cases. Task completion, meaning the share of runs that finished the job end to end. Trajectory validity, meaning whether the path was reasonable. Tool-call correctness, covering the right tool, right arguments, and graceful recovery. Intervention rate, the proportion of runs needing a human. Cost and latency per completed task rather than per model call. And reliability across repeated attempts rather than a single pass. Accuracy is deliberately absent, because a multi-step task rarely has one correct output.

Are AI agent benchmarks reliable?

Treat them as claims about a model, not about your system. Three problems apply. Benchmarks use clean inputs and controlled environments while production faces ambiguity, flaky APIs, and adversarial input. Most report success on a single attempt, whereas production retries, so a high score overstates reliability. And in April 2026 researchers at Berkeley's Center for Responsible Decentralized Intelligence reported that every major agent benchmark they examined had exploitable evaluation mechanisms allowing near-perfect scores without solving tasks. Benchmarks remain useful for model selection.

What is trajectory evaluation?

Trajectory evaluation scores the path an agent took rather than only its final answer. It compares the sequence of steps against a reference path or a rubric, and looks for specific pathologies: wrong tool selected, incorrect arguments, looping, unnecessary steps, and whether the agent recovered when something failed. It matters because outcome-only scoring passes an agent that got lucky. An agent reaching the right answer through a dangerous or wildly inefficient sequence of tool calls is a production failure that outcome metrics record as a success.

What is LLM-as-judge, and can you trust it?

LLM-as-judge uses a model to score another model's output or trajectory against a rubric. It is the only tractable way to evaluate trajectories at volume, and it has documented systematic biases: it rewards longer responses, is influenced by the order options appear in, tends to prefer output from its own model family, and is non-deterministic, so the same trajectory can score differently twice. It is trustworthy to the extent you have measured its agreement with human raters, and most teams have not measured that at all.

How do you validate an LLM judge?

Take a sample of runs, have humans score them against the same rubric the judge uses, and measure how often the two agree. That agreement rate is effectively the confidence interval on every other number the judge produces, and it should be reported alongside them. Re-measure whenever the judge model, the rubric, or the agent changes, since all three move. Two mitigations help: require the judge to state its reasoning before scoring, which reduces the known biases, and use deterministic checks wherever a rule can express the requirement.

How many test cases do you need to evaluate an agent?

Fewer than people expect for confidence in a specific scenario, and more than expected for confidence in an aggregate number. A focused anchor set of carefully hand-built cases covering your most critical scenarios is enough to gate a release, and current practice suggests several hundred cases before aggregate metrics mean much. The binding constraint is construction cost rather than volume: a single good agent case, with its task, tool environment, expected trajectory, and gold output, can take hours to build correctly.

Should agent evaluation run before or after deployment?

Both, and the second is the one most often missing. Pre-release testing establishes that the agent does the job you built it for. Production evaluation establishes that it still does, which matters because four of the five things determining an agent's behaviour can change without anyone deploying anything: the model version, the instructions, the tool set, and the data it retrieves. A suite that passed at release describes the day it ran. Surveys suggest most teams monitor production and fewer than half test systematically beforehand.

Can agent evaluation be fully automated?

Mostly, and not entirely, and the part that resists automation is the part that defines what you are measuring. Deterministic checks and judge scoring can run continuously without people. Two things cannot: deciding what good behaviour means for your users, and establishing whether your automated scoring agrees with human judgement. Both require human review on a sample. A fully automated programme is one that has stopped checking whether its own instruments are accurate, which is a different thing from being efficient.

How much does agent evaluation cost to run?

Enough that the economics shape the practice, which is why layering matters. Deterministic checks are effectively free and can run on every request. Judge scoring costs a full model call per judgement, so running it on every production turn is usually prohibitive and sampling is standard. Human review is the most expensive per case and the only source of calibration. Teams that ignore this quietly reduce evaluation frequency until it becomes a one-off release gate, which is the failure mode the layering is designed to prevent.

Who owns agent evaluation?

The engineers building the agents own running it; whoever owns the business outcome owns defining what good means. That split matters. Engineering can build the harness, the judge, and the tracing, and cannot decide whether a particular trajectory is acceptable to a customer or a regulator. Domain owners can make that call and cannot instrument it. Programmes fail in both directions: an engineering-only practice measures what is easy to measure, and a business-only one produces quality definitions nobody has implemented.

Keep reading

Terms that come up in the same conversation

Browse the full glossary
A

AgentOps

The wider practice this sits inside, including the seven artifacts you have to trace to score a run.
A

AI guardrails

Evaluation tells you what happened. Guardrails are what prevent it happening again at runtime.
C

Context engineering

A common root cause behind a failing evaluation score, and one that outcome metrics never surface.

From a score to a decision
What happens when an evaluation says no?

An evaluation result only matters if something acts on it. CAMS connects the two: promotion gates through Dev, QA and Production that a failing result can block, per-span tracing to locate the step that failed, and a kill switch for when a live agent starts regressing.

Book a demo Explore CAMS