Agent Evals in CI: Regression Gates for AI Agents

Agent evals in CI are a test suite for behavior. Every change to a prompt, tool, or model runs a fixed set of cases, and the build fails if quality drops.

Four things should fail a build: a pass-rate drop, a critical case that breaks, a step or token budget overrun, and any policy violation.

Below is how I set those gates, how many cases you need, and what to do with flaky results.

Why unit tests are not enough

A unit test says: given this input, the function returns that output. Same answer every time. Agents do not work that way.

An agent chooses its own path. Change one line in a prompt and a task that passed yesterday can fail today. Swap a model version and a dozen things shift at once. Your normal tests will stay green because the code did not change. The behavior did.

Two more problems:

• Outputs vary. Anthropic’s guide to agent evals notes that model outputs differ between runs, so they run multiple trials per task to get consistent results.
• Reliability decays fast. The same guide explains the pass^k metric: the chance that all k trials succeed. At a 75% success rate per trial, three trials all passing is about 42%. The tau-bench paper introduced the metric and found that even strong function-calling agents of its time succeeded on under 50% of tasks and were inconsistent: pass^8 was under 25% in the retail domain.

So CI needs checks built for behavior. The Anthropic guide puts evals in CI/CD, running on each agent change and model upgrade, as a first line of defense. I agree. If you do not yet have a case set, start with our agent evaluation guide and come back.

What to gate on

Not every metric should block a merge. Split them into hard gates and soft warnings.

Check	Type	Example rule
Critical cases	Hard	Every must-pass case passes, no exceptions
Policy violations	Hard	Zero (leaked data, banned action, missing approval)
Overall pass rate	Hard	Fails if it drops more than X points from baseline
Step budget	Hard or soft	Fails if a case exceeds its step limit
Token budget	Hard or soft	Fails if median tokens per task rise more than Y%
Latency (P90)	Soft	Warn if P90 time rises more than Z%
New failure categories	Soft	Warn and review the transcript

A few rules I follow.

Compare to a baseline, not to perfect. Store the pass rate from your main branch. Gate on the change from that number. A fixed bar like “must be 95%” breaks the first time you add harder cases.

Make critical cases binary. Pick 10 to 20 cases that must never fail: refunds over a limit need approval, PII is never sent out, a destructive tool is never called without a check. A single failure blocks the merge.

Treat policy as zero tolerance. One violation is a fail, not a percentage.

Use the right kind of eval. The Anthropic guide separates capability evals, which start at a low pass rate and give you a hill to climb, from regression evals, which should sit near 100%. Only the regression set should gate merges. Capability evals belong in a separate, non-blocking job.

When a gate trips, read the transcript. The same guide says to check whether the agent made a real mistake or your grader rejected a valid answer. If you grade with a model, calibrate it first. See LLM as a judge for agent evals.

Step and token budgets as failures

An agent can pass every case and still be worse. It may take twice the steps, loop on a tool, or burn three times the tokens. Users feel that as slowness and finance feels it as cost.

So treat budgets like tests. For each case, record steps taken, tokens used, and time. The Anthropic guide lists latency, token usage, cost per task, and error rates as metrics you can track on a static bank of tasks. Then fail the build on overruns.

A starting config might look like this (my suggested shape, not a standard):

gates:
  critical_cases: all_pass
  policy_violations: 0
  pass_rate:
    max_drop_vs_main: 5   # points
  budgets:
    max_steps_per_case: 15
    median_tokens_vs_main: +20%   # percent
    p90_seconds_vs_main: +25%

How to set the numbers:

1. Run the suite on main five or more times. Record the spread.
2. Set limits above that natural spread, or you will get false alarms.
3. Use a hard per-case cap on steps (a runaway loop should fail loudly) and a softer rule on medians.
4. Revisit quarterly. Tighten only when the suite is stable.

Cap the runaway case in code too, not just in CI. A step limit inside the agent protects production, and the eval proves the limit holds. For why latency matters, see why agents are slow and how to cut latency.

Dataset size and tiers

How many cases are enough? There is no magic number, but there is a sensible start. The Anthropic guide says 20 to 50 simple tasks drawn from real failures is a great start. I agree, and I would build tiers so most runs stay cheap.

Tier	Size	When it runs	Purpose
Smoke	10 to 20 cases	Every pull request	Catch obvious breaks fast
Core	100 to 200 cases	Pull requests that touch prompts, tools, or models; nightly	The real regression gate
Full	300+ cases, multiple trials	Before release; weekly	Deep check, edge cases, trend

Source cases from real failures: support tickets, bad outputs, near misses. Every production bug becomes a new case. Add a case, never delete one without a reason in the commit.

Size changes how much a drop means

Small suites are noisy. Here is my arithmetic. Say your baseline pass rate is 90% and you fail the build on a 5-point drop. With no real change, random variation alone would trip that gate about this often (normal approximation, baseline treated as known):

Cases	Chance of a false alarm
50	about 12%
100	about 5%
200	about 1%

At 50 cases, roughly one clean run in eight blocks someone for nothing. People then learn to rerun until green, and the gate is dead. That is why the core tier needs 100 or more cases. For deeper reading, the paper Adding Error Bars to Evals argues evals are experiments and should be analyzed that way.

Two ways to cut noise without more cases: compare the same cases on both branches and look only at the ones that flipped, and run each case more than once.

Flaky cases

A flaky case passes sometimes and fails sometimes with no change. Agents produce plenty of them, because the model is not deterministic. Classic software has the same disease. Google’s testing team reported that about 1.5% of its test runs were flaky and about 16% of its tests showed some flakiness, and they handled it with retries and quarantine.

Do the same, with care:

1. Find the cause first. Read the transcripts. Is the grader unstable? Is the task ambiguous? Does the case depend on live data that changed? Most flakiness is a bad case or a bad grader, not model noise.
2. Run more than one trial for suspect cases. Use pass^k thinking for anything critical: if a refund-approval rule only holds two times out of three, that is a failure, not a flake.
3. Retry once, and log it. A pass on retry is a warning, not a clean pass. Track the retry rate.
4. Quarantine. Move a case that stays flaky into a separate non-blocking job with an owner and a deadline. Never leave it silent.
5. Fix, then restore. Tighten the task, pin the data, or repair the grader.
6. Freeze inputs. Mock or snapshot external tools so the world does not change under your test.

One warning: never “fix” flakiness by loosening a critical rule until it passes. That hides the exact risk the case exists to catch.

Run time and cost

Evals call a model many times, so cost and time grow quickly. Plan for both.

Simple math: cases times model calls per case times seconds per call, divided by how many you run in parallel. For example, 100 cases at 12 calls each and 6 seconds per call is 72 seconds per case. Run 10 at once and the suite takes about 12 minutes. At 1 at a time it takes two hours. Concurrency matters most, but watch provider rate limits.

How to keep it short:

• Tier it. Smoke on every pull request, core only when it matters.
• Run only on relevant changes. In GitHub Actions, the paths filter runs a workflow only when certain files change. Promptfoo’s CI guide shows this pattern for prompt files, plus caching keyed to those files.
• Cache what has not changed. If a case’s inputs and prompt are the same, reuse the result.
• Set a timeout. Use timeout-minutes so a hung job fails instead of running for hours. GitHub’s default is 360 minutes.
• Use smaller models for grading where your calibration shows they hold up.
• Track spend per run. Put cost in the report so nobody is surprised.

My target for a pull request check: under 10 minutes. Past that, people stop waiting for it.

Reporting to non-engineers

A product manager or compliance lead will not read a log. Give them one page per run:

• Verdict: pass or fail, in one word at the top.
• What changed: prompt version, model, tool set.
• Quality: pass rate versus main, with the number of cases.
• Critical and policy checks: count passed out of count.
• Cost and speed: tokens, steps, P90 time versus main.
• What got worse: the three cases that flipped from pass to fail, in plain language.
• Flaky or quarantined: how many, and who owns them.
• Recommendation: ship, fix first, or needs human review.

Link every case to its transcript. Pair this with prompt versioning so each result ties to a specific change. See prompt change control.

Bottom Line

Treat agent behavior like code. Keep a regression set, run it on every change, and fail the build on critical cases, policy violations, quality drops, and budget overruns.

Use tiers: a small smoke set for every pull request and a 100-plus case core set for the real gate. Handle flaky cases by finding the cause, not by loosening rules.

Next step: turn your last five production failures into cases, add two hard gates, and run them on your next pull request.

FAQ

How do you regression test an agent? Keep a fixed set of real cases with known good outcomes. Run it on every change to prompts, tools, or models. Compare the pass rate, critical cases, steps, and tokens to the main branch, and block the merge if they get worse.

What should fail a build? Any critical case failure, any policy violation, a pass rate that drops past your threshold, and a step or token budget overrun. Latency increases can warn first, then block once you trust the numbers.

How many cases are enough? Start with 20 to 50 cases from real failures. For a gate you can trust, use 100 or more, since a 50-case suite can trip a 5-point drop about 12% of the time by chance alone.

How do I handle flaky cases? Read the transcripts to find the cause. Fix unclear tasks and unstable graders, run critical cases several times, and retry once with logging. If a case stays flaky, quarantine it in a non-blocking job with an owner.

How long can an eval run take? For a pull request check, aim for under 10 minutes using a small smoke set and parallel runs. Run the larger core and full sets nightly or before release, and set a job timeout so hung runs fail fast.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect