AI Agent Hallucination Budget: How Much Error a Process Can Take

You will not get an AI agent to zero hallucinations. So stop chasing zero.

Set a budget instead. For each workflow, decide how much error is tolerable, which errors are expensive, and where a hard check or a human goes. That is AI agent hallucination mitigation as an operating practice, not a prompt trick.

This post gives you the budget method, the tools that keep an agent inside it, and a worksheet to run with the process owner.

Why Zero Error Is the Wrong Target

Humans make errors too. A process that “never fails” usually has an error rate nobody measures.

Here is the problem with chasing zero. The cost of getting close rises fast, and some of that cost is the agent refusing to answer. A system tuned never to be wrong will say “I don’t know” constantly. Then you have an expensive search box.

The real question is simple: what does one wrong answer cost, and how many can this process absorb?

It also helps to know why models guess at all. OpenAI’s research, Why language models hallucinate (September 2025), argues that standard evaluations reward guessing over admitting uncertainty. Its example compares two models on a question-answering test:

Model	Abstained	Right	Wrong
gpt-5-thinking-mini	52%	22%	26%
o4-mini	1%	24%	75%

Accuracy is nearly the same. The error rate is not. The model that said “I don’t know” more often was wrong far less. OpenAI’s recommendation is to penalize confident errors more than uncertainty. Your budget does the same thing for your own process.

NIST uses the term confabulation (AI 600-1, July 2024) for confidently presented, false content. The word matters less than the point: the output sounds sure either way, so you cannot judge it by tone.

Cost of an Error by Workflow

Start with money and harm, not model quality.

For each workflow, estimate three numbers:

• Volume. Outputs per month.
• Escaped error rate. The share of wrong outputs that get past every check and reach a customer, a ledger, or a decision. Budget on this one, not on raw model error.
• Cost per escaped error. Refund, rework, fine, lost customer, legal time. Be honest. Include the time to find and fix it.

Expected monthly cost of error is volume x escaped error rate x cost per escaped error.

A hypothetical: an invoice-coding agent handles 2,000 invoices a month. Say 1% escape, and each costs $150 to find and correct. That is 20 x $150, or $3,000 a month. If a reviewer sampling 10% of invoices costs less than the errors they catch, add the review. If not, accept the budget.

Costs can also be legal. In Moffatt v. Air Canada, a chatbot gave a customer the wrong bereavement fare policy. The British Columbia tribunal held the airline responsible for what its chatbot said, as reported by the ABA in February 2024. The award was small, about CAD 650 plus interest and fees. The lesson was not the amount. It was that “the bot said it” is not a defense.

Here are the three tiers I use as a starting point. The percentages are my defaults, not industry benchmarks.

Tier	Example workflows	Escaped error budget	Controls
1. Low stakes	Internal summaries, draft emails, meeting notes	Around 5%, if the user sees it before it matters	Grounding, user can see sources
2. Correctable	Customer FAQ answers, ticket triage, data entry with a downstream check	1% to 2%	Required citations, abstain rule, sampled review
3. High stakes	Payments, legal or clinical content, regulated disclosures, anything irreversible	Near zero escapes	Human approval or a hard rule check on every action, abstain by default

If you cannot name a tier for a workflow, the workflow is not ready for an agent.

AI Agent Hallucination Mitigation: Grounding and Required Citations

Grounding means the agent answers from documents or data you supply, not from the model’s memory. The usual method is retrieval-augmented generation. For the build-or-tune choice, see fine-tuning vs RAG. The glossary entries on grounding and hallucination cover the basics.

Does RAG fix hallucination? No. It lowers it.

Stanford researchers tested commercial legal research tools built on RAG. Stanford HAI reported in May 2024 that Lexis+ AI and Ask Practical Law AI gave incorrect information more than 17% of the time, and Westlaw’s AI-Assisted Research more than 34%. Their words: “even RAG systems are not hallucination-free.”

A Vectara paper from May 2025 found the same pattern at a general level. Models still frequently add unsupported claims or contradictions even when given relevant context.

So grounding is the floor, not the fix. Add these on top:

1. Require a citation for every factual claim. The agent must point to a source passage. No source, no claim.
2. Verify the citation with code, not with the model. Check that the quoted text actually appears in the retrieved document, using a string match. Check that any ID, amount, or date exists in the system of record. This is a hard check. It does not hallucinate.
3. Extract quotes first on long documents. Anthropic’s guidance on reducing hallucinations suggests having the model pull word-for-word quotes before answering, then cite them. It also suggests making the model retract any claim it cannot back with a quote.
4. Check ground truth in the data. NIST’s profile includes an action to verify that data used for retrieval is itself grounded and traceable (MS-2.5-005). A wrong source document produces a confident, cited, wrong answer.

Citations make errors findable. That matters because the cheapest error is the one a reviewer can catch in ten seconds by clicking a link.

Abstain Rules

An agent that can say “I don’t know” is worth more than one that always answers.

You get abstention from layers, not one setting:

• Permission in the prompt. Anthropic’s guidance says to explicitly allow the model to say it does not know. It is a small change with a real effect.
• Retrieval floor. If no retrieved passage scores above a set threshold, do not answer. Route to a human or ask a clarifying question.
• Citation rule. If the agent cannot attach a verified citation, it abstains.
• Consistency check. Ask the same question several times and compare. Anthropic’s docs call this best-of-N verification: disagreement across runs can signal a hallucination. It costs more tokens, so use it on Tier 2 and 3 only.
• Verifier step. A second pass that tests each claim against the sources and flags what is unsupported.

Then set the threshold with data. Run your eval set at several retrieval or confidence cutoffs. For each, record the escaped error rate and the abstain rate. Pick the loosest cutoff that stays inside the budget.

The price of abstaining is work for humans. If 20% of questions go to a person, that is a staffing number. Put it in the business case, using the method in how to calculate AI agent ROI.

Measuring With an Eval Set

You cannot manage what you cannot count. Build a small, real eval set before you tune anything.

What goes in it:

• 50 to 200 real cases from the actual process. Not invented ones.
• A correct answer for each, labeled by the process owner or a subject expert.
• Cases with no right answer in the sources, where the correct behavior is to abstain. Without these you will never measure abstention.
• Hard cases: stale documents, conflicting sources, ambiguous questions, out-of-scope requests.

What to measure:

Metric	What it tells you
Answer accuracy	Share of answers that are correct
Faithfulness	Share of claims supported by the retrieved context. Ragas defines it as supported claims divided by total claims, from 0 to 1
Citation validity	Share of citations that pass the string and ID checks
Abstain precision and recall	Did it refuse when it should, and answer when it should
Escaped error rate	Wrong outputs that got past all checks. This one maps to your budget

Two cautions on small samples.

First, zero errors in 100 cases does not mean zero errors. A rough rule: if you see no errors in n independent cases, the true rate could still be as high as about 3 divided by n at 95% confidence. With 100 cases, that is about 3%. If your Tier 2 budget is 1%, you need roughly 300 clean cases to show it.

Second, an LLM judge can grade faithfulness at scale, but spot-check it against human labels. The Vectara paper above exists because automated hallucination detectors have limits.

Re-run the set on every prompt, model, retrieval, or document change. The method is covered in the AI agent evaluation guide and AI agent testing strategies.

Where a Human Check Goes

A human check is expensive and slow. Put it where it earns its cost.

Use this order of preference:

1. A hard check first. If a rule can verify the output (totals match, ID exists, value in allowed range, quote found), use code. It is cheap and consistent.
2. Sampled review second. For Tier 2, a person reviews a random slice, plus every output the agent flagged as low confidence. Track the error rate in the sample. If it rises, review more.
3. Approval third. For Tier 3, a person approves before the action happens. Design the screen so the reviewer sees the answer, the cited source, and the check results side by side. If they must hunt for the evidence, they will click approve.

Watch out for rubber-stamping. A reviewer who approves 99.8% of items for weeks stops reading. Seed known-bad items into the queue now and then to test whether they catch them.

For design patterns, see human-in-the-loop AI agents.

Setting the Budget With the Process Owner

The process owner decides the budget. Not the model vendor, and not the builder.

Run a 45-minute session. Bring the worksheet:

1. Workflow and output. What does the agent produce? Who or what consumes it?
2. Volume. Outputs per month.
3. Worst plausible error. Describe the most expensive wrong output. What happens next?
4. Cost per escaped error. Dollars, plus anything you cannot price (trust, compliance).
5. Human baseline. What is the current error rate of the manual process? The agent has to beat a number, not perfection.
6. Tier and budget. Pick Tier 1, 2, or 3. Set the escaped error rate.
7. Controls. Grounding source, citation rule, abstain rule, hard checks, human check.
8. Eval set owner and size. Who labels, how many cases.
9. Review trigger. What measured result forces more review or a pause?

Get the process owner to sign the page. When an error happens later, you will have an agreed number to compare it against, not an argument.

Bottom Line

Hallucination is a risk to budget, not a defect to eliminate.

Set an escaped error rate for each workflow. Ground the agent, require verified citations, give it permission to abstain, and measure on a real eval set. Put people where hard checks cannot reach.

My recommendation: pick your highest-volume workflow this week and fill in the nine-line worksheet before you tune a single prompt.

FAQ

How do you stop an AI agent from hallucinating? You cannot stop it fully. You reduce it and catch what is left. Ground the agent in your own documents, require a citation for every claim, verify those citations with code, let the agent say “I don’t know,” and add human review where the cost of an error is high.

Does RAG fix hallucination? No. RAG lowers it. A May 2024 Stanford study found legal research tools built on RAG still produced incorrect information more than 17% of the time in one case and more than 34% in another. Models can also add unsupported claims even when the right documents are in front of them.

How much error is acceptable for an AI agent? It depends on what one wrong output costs. As a starting point, I use roughly 5% for low-stakes drafts a user will read, 1% to 2% for correctable customer-facing work, and near zero for payments, legal, clinical, or irreversible actions. Your process owner should set the final number.

How do I measure hallucination? Build an eval set of 50 to 200 real cases with correct answers, including cases where the right move is to abstain. Measure answer accuracy, faithfulness to sources, citation validity, and escaped error rate. Re-run it after every change to the prompt, model, or documents.

When do I need a human check? Use a hard rule check first when one can verify the output. Add sampled human review for correctable work. Require human approval before any irreversible or regulated action. If the reviewer cannot see the source evidence quickly, the check will not work.

What is an abstain threshold? It is the cutoff below which the agent declines to answer and routes to a person. You set it by running your eval set at different cutoffs and picking the loosest one that keeps the escaped error rate inside your budget.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect