What Is Agent Evaluation?

Agent evaluation is how you measure whether an AI agent does the job. You build a set of test tasks, run the agent on them, grade the results, and track the score over time.

It is the difference between “the demo looked good” and “this agent resolves the right cases at a rate we can defend.”

The parts of an eval

Anthropic’s January 2026 engineering post, Demystifying evals for AI agents, gives a clean vocabulary:

Task. One test with defined inputs and success criteria.
Trial. One attempt at a task. You run several, because agents vary.
Grader. Logic that scores some aspect of performance.
Transcript. The full record of a trial: outputs, tool calls, reasoning, intermediate results.
Outcome. The final state of the environment when the trial ends.

That last one matters. If the agent says “refund issued,” the eval should check the payment system, not the agent’s claim.

Graders come in three types: code-based, model-based, and human. See LLM as a judge for the model-based kind. Model graders carry known biases toward position, verbosity, and their own style, documented in the MT-Bench paper, so check them against human labels.

Two kinds of eval

The same post separates two purposes:

Capability evals ask what the agent can do well. They should start with a low pass rate and give you something to improve.
Regression evals ask whether the agent still handles what it used to. They should pass close to 100% and protect against backsliding.

I keep both. The capability set tells me where to work. The regression set tells me whether a change is safe to ship.

Reliability is its own metric

Agents are not deterministic. Run the same task ten times and you may get ten different paths. A single pass tells you little.

The tau-bench paper (June 2024) tests agents in simulated customer conversations with domain tools and policy rules. It compares the database state at the end of a conversation to the goal state. It also introduced pass^k, a metric for reliability across repeated trials. Their headline: even the strongest function-calling agents they tested succeeded on under 50% of tasks, and pass^8 was under 25% in the retail domain.

Anthropic’s post explains the distinction. pass@k is the chance of at least one success in k tries. pass^k is the chance that all k tries succeed. For a customer-facing agent, pass^k is the honest number, because customers do not get to retry.

What to measure

For business agents I track:

Task success. Did the final state match the goal?
Policy compliance. Did it follow the rules, even when the user pushed?
Tool correctness. Right tool, right arguments.
Escalation quality. Did it hand off when it should have?
Cost and latency per task.

Where test cases come from

Start with real work. Pull past tickets, emails, or cases. Add the ugly ones: missing fields, angry users, conflicting documents, attempts to break the rules. Ten to twenty good cases beat a hundred synthetic ones. That is my judgment, not a measured result.

A short example

Say an agent books internal meeting rooms. The team writes 20 tasks, including a double-booking attempt, a request for a room that does not exist, and a user asking it to cancel someone else’s booking. Each task runs five times. A code grader checks the calendar state. A model grader checks that refusals were polite and clear. The team reports task success, and pass^5 for the cancel-someone-else case, which must be 100%.

Why it matters for production agents

Evals are what let you change prompts, models, and tools without fear. They also catch silent decay. When a vendor updates a model, your regression suite tells you in minutes whether anything broke. Without it, users tell you.

Common mistakes

Grading the transcript, not the outcome. The agent says it worked. The system disagrees.
One trial per task. You miss variance.
Only happy-path tasks.
No human spot-check. Graders drift and miss new failure types.
Never refreshing. The test set no longer looks like real traffic.

A minimal starting plan

If you have nothing today, start small:

Pick one workflow and write down what success means in one sentence.
Collect 20 real cases, including five that are messy or adversarial.
For each, record the expected final state, not the expected wording.
Write code graders for anything checkable. Add a model grader only for what code cannot judge.
Run each case several times and record the pass rate.
Save the results. Rerun after every prompt, model, or tool change.
Review failures by hand every week and add new cases from real incidents.

That takes a day or two and gives you more signal than months of eyeballing outputs.

LLM as a judge covers model-based grading. Tool calling is where many failures start. Human in the loop review often supplies the ground-truth labels. For a pilot-sized approach, see the one-week AI pilot methodology.