AI Agent Evaluation: How to Write Acceptance Tests

AI agent evaluation is a set of acceptance tests for a system that does not give the same answer twice.

You write the tests before the build. You agree the pass marks with the business owner. Then you keep running them after launch.

Most teams skip the first two steps and call a demo “testing”. This post shows the better way.

AI Agent Evaluation Is Not Unit Testing

An agent eval is a repeatable test that scores what an agent does on a realistic task. It has an input, a definition of success, and a grader that decides pass or fail.

Anthropic’s guide to agent evals uses a small vocabulary that I like because it is plain:

• Task: one test with defined inputs and success criteria.
• Trial: one attempt at a task.
• Grader: logic that scores the result.
• Transcript: the full record of a trial, including tool calls.
• Outcome: the final state of the environment when the trial ends.

So how is an eval different from a unit test? Three ways.

A unit test expects one exact answer. An eval accepts a range. An agent can reach a correct result by different paths. If your test demands one path, it fails good runs.

A unit test runs once. An eval runs several times. The same task can pass Monday and fail Tuesday.

A unit test passes at 100%. An eval rarely does. Hamel Husain, who has written widely on this, says that the pass rate should not be 100%. Pass rate is a product decision about which failures you will tolerate.

Keep fast unit tests for the deterministic parts: schema validation, argument formats, math. My post on testing agents before they touch production data covers that pre-production layer. This post is about the ongoing evals on top of it. Anthropic’s guidance on building agents also calls for extensive testing in sandboxed environments with guardrails, because agent errors compound.

Build the Golden Set From Real Cases

A golden set is the collection of test cases you trust. Each case has an input, the expected outcome, and a note on how to grade it.

How big should it be? Anthropic’s advice is 20 to 50 simple tasks drawn from real failures. I agree with that range for a first set. A small set you trust beats a large set nobody read.

Source the cases from your own operation:

1. The manual checks people already run. Ask the team how they check the work today. Each check is a test.
2. Past errors. Pull them from the bug tracker, the support queue, and the correction log. Anthropic recommends turning user-reported failures into test cases.
3. Normal cases. Include plenty of plain, boring inputs.
4. Cases where the agent should refuse or escalate.

Keep the set balanced. Anthropic says to test cases where a behavior should occur and where it should not, because one-sided evals create one-sided optimization. Test only “should issue the refund” and you ship an agent that refunds everyone.

Check each case with one rule: would two domain experts reach the same pass or fail verdict on their own? Anthropic uses this as the test of a good task. If your two best operators would argue, the case is not ready. Fix it or write down the policy that settles it.

Who writes and owns the eval set?

The business owner of the workflow owns it. Engineers build the runner. The people who do the job today write and label the cases.

Anthropic says the people closest to product requirements and users are best positioned to define success. A developer cannot tell you whether a claims note is acceptable. A claims adjuster can.

Give the set a named owner and a change log.

Task Success and Policy Compliance

Score two things separately. They fail in different ways.

Task success asks: did the agent get the job done? The invoice was coded, the ticket was routed, the record was updated correctly.

Policy compliance asks: did it stay inside the rules while doing so? It did not exceed its refund limit, touch a record it should not touch, skip an approval, or reveal data to the wrong person.

An agent can pass the first and fail the second. That is the dangerous combination, because the work looks done.

The tau-bench paper is a good model. Its agents work with a simulated user, API tools, and domain policy rules. The grader compares the database state at the end of the conversation with the expected goal state. That catches an agent that talked nicely but changed the wrong thing. The authors report that even strong function-calling agents of 2024 succeeded on fewer than 50% of tasks.

In practice, split your golden set into two buckets:

Bucket	Question	Example case	Failure means
Task success	Did it finish the job correctly?	Standard invoice with a clean PO match	Rework, lost time
Policy compliance	Did it follow the rules?	Refund request above the agent’s limit	Risk, audit finding, real harm

Policy cases include the traps: a reasonable-sounding request that breaks a rule, a user who pushes back, a missing document, a prompt that tries to override instructions. Tie these cases to your least-privilege permissions design. An eval shows the agent behaves. Permissions limit the damage when it does not.

Graders: Code, Model and Human

A grader decides pass or fail. There are three types, and a good suite mixes them. Anthropic lays out the tradeoffs:

Grader	Strength	Weakness	Use it for
Code	Fast, cheap, objective, easy to debug	Brittle when valid answers vary	Exact fields, record state, limits, amounts
Model (LLM judge)	Flexible, handles open-ended text	Non-deterministic, costs more, needs calibration	Tone, completeness, summaries
Human	Gold standard	Slow and expensive	Calibrating the model grader, final sign-off on hard cases

My rule: use code wherever you can, a model grader where you must, and a human to keep the model grader honest.

Two cautions.

Grade the outcome, not the path. Anthropic found that checking specific steps in a fixed order is too rigid, because agents regularly find valid approaches the eval designer did not expect. Check what the agent produced. The exception is a policy step that must happen, such as an approval. Test that it happened.

Calibrate any model grader. A study of LLM judges found strong models can reach over 80% agreement with human preferences, the same level humans reach with each other. It also found position, verbosity, and self-enhancement biases. So a judge is useful, but you must check it. Hamel Husain’s LLM judge guide suggests a clean method: one principal domain expert gives binary pass or fail with a short critique, you build the judge prompt from those examples, and you iterate until the judge agrees with the expert. Then you measure that agreement.

Prefer pass or fail over a 1 to 5 score. A 3.7 starts an argument. A fail starts a fix.

Pass Marks a Buyer Can Read

A pass mark is the line that decides whether a release ships. Agree it in writing before the build, with the person who will own the result.

What pass rate should block a release? No sourced universal number exists. These are my starting points. Treat them as a guess to adjust.

Test group	Starting pass mark	Release rule
Hard policy cases (limits, permissions, privacy)	100%	Any failure blocks the release
Core task cases, run several times each	90% or higher of tasks pass on every run	Below the mark blocks the release
Edge and capability cases	Track the trend	Report it, do not block on it
Regression set (past failures)	Near 100%	A new failure blocks the release

Notice the words “on every run”. Anthropic describes two ways to count repeated trials. pass@k counts a task as solved if the agent gets at least one success in k tries. pass^k counts it only if all k tries succeed. The tau-bench authors introduced pass^k and found that agents scoring under 50% on a single try fell below 25% on the retail domain when the same task was repeated eight times.

Here is why that matters. Say an agent passes a task 70% of the time. By my arithmetic, the chance it gets eight customers in a row right is about 6%. The buyer sees “70% good”. The operation sees “usually broken”.

For anything customer-facing or money-related, I’d hold the agent to pass^k, not pass@k.

Anthropic also separates two kinds of eval. Capability evals should start at a low pass rate and give the team a hill to climb. Regression evals should sit near 100%. They ask whether the agent still handles everything it used to.

Write the pass marks into the statement of work or the pilot charter. A buyer can read a table like the one above and sign it. That is what makes it an acceptance test. For the wider path from pilot to production, see getting from experimentation to production.

Run Evals on Production Samples

A golden set goes stale. Inputs drift, policies change, and users find new ways to break things. Production sampling closes the gap:

1. Log every run with its inputs, tool calls, and outcome. This is what your observability stack is for.
2. Sample a slice each week. Pick runs at random, plus every run that was escalated, overridden, or complained about.
3. Have the workflow owner grade them. Pass or fail, with a one-line reason.
4. Turn each real failure into a new case in the golden set.
5. Rerun the full suite after any change to the prompt, model, or tools.

Anthropic is clear that evals are one layer among several. Production monitoring, user feedback, A/B tests, and manual transcript review all catch different things. Like the Swiss cheese model, no single layer catches every issue, so you stack them.

Read the transcripts. Anthropic says a failed task shows whether the agent made a real mistake or whether your grader rejected a valid answer. Failures should look fair. If you cannot tell why something failed, fix the eval before you fix the agent.

Someone has to own this after go-live. It is part of the work I cover in maintaining an agent after deploy.

Common Eval Mistakes

I see the same ones over and over.

• Starting with tools instead of cases. A dashboard does not tell you what good looks like. Write ten real cases in a spreadsheet first.
• Trusting an uncalibrated judge. Check it against a human on a sample first.
• Letting the vendor grade itself. The buyer should hold the golden set, or a held-back part of it.
• Chasing 100% on capability evals. Anthropic notes that an eval at 100% tracks regressions but gives no signal for improvement. When a set saturates, write harder cases.
• No owner. An eval set with no owner decays, and so does the agent.

An acceptance test checklist

Use this before you approve a build:

• 20 to 50 golden cases, sourced from real work and past failures
• Two experts agree on the verdict for each case
• Task success and policy compliance scored separately
• Model graders checked against human labels
• Pass marks written down and signed by the business owner
• Core cases run multiple times, scored as pass^k
• Weekly production sample with a named reviewer
• A rule that every real failure becomes a new case

If your agent has broad system access, like the ones in my OpenClaw enterprise guide, most of your cases should live in the policy bucket.

Bottom Line

An agent eval is an acceptance test. Write it before the build, from real cases, with pass marks the business owner signs.

Split task success from policy compliance. Score repeated runs, not single ones. Keep sampling production after launch.

Start this week: write ten real cases in a spreadsheet and get two experts to agree on each verdict.

FAQ

What is an agent eval? An agent eval is a repeatable test that scores what an AI agent does on a realistic task. Each one has an input, a success definition, and a grader that returns pass or fail. A suite of them works as the acceptance test for the agent.

How big should a golden set be? Start with 20 to 50 cases drawn from real failures and real work, the range Anthropic recommends for a first set. Grow it as production shows new failures.

What pass rate should block a release? My starting point is 100% on hard policy cases and 90% or higher on core task cases, counted across repeated runs. These are my guesses, not an industry standard. Agree your own numbers with the business owner before the build, and write them down.

Who writes and owns the eval set? The business owner of the workflow owns it, and the people who do the job today write and label the cases. Engineers build and run the runner.

How is an eval different from a unit test? A unit test expects one exact answer and runs once. An eval accepts a range of valid answers, runs the same task several times, and rarely reaches 100%. Use unit tests for deterministic code and evals for agent behavior.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect