AI Pilot Success Criteria: A Scale-or-Stop Charter

Write your AI pilot success criteria before anyone builds anything.

That means four things on one page: a measured baseline, a target, a stop rule with a date, and one named person who makes the call. Without them, the pilot ends in a demo and a debate.

The rest of this post is how to fill in each of the four, and how to score the result.

Why Pilots Drift Without a Charter

Most pilots do not fail loudly. They drift.

The demo works. The team reports “promising early results.” Nobody can say if the agent beat the old process, so nobody scales it and nobody kills it. That is pilot purgatory.

The numbers on this are rough, but they all point the same way:

• IDC research with Lenovo found that for every 33 AI proofs of concept a company launched, only four reached production. The same article cites unclear ROI, weak data, and pressure to experiment as causes.
• Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027. The causes it names are escalating costs, unclear business value, and inadequate risk controls.
• In McKinsey’s 2025 survey, 23% of respondents said they were scaling an agentic AI system somewhere in the company. Nearly two-thirds had not begun scaling AI across the enterprise.

Look at the causes again. Unclear value. Unclear ROI. Costs that grow. Those are charter problems, not model problems. A charter fixes them on day one, when changing course is cheap. I covered the wider failure pattern in why AI pilots fail.

Baseline First

You cannot show improvement without a “before.” This is the step teams skip most.

NIST’s AI RMF Playbook says the same thing in risk language. Its Measure function (subcategory MEASURE 3.1) calls for comparing the AI system against simpler models and human baseline performance.

A usable baseline has five parts:

1. The unit of work. One ticket, one invoice, one claim, one lead. Define it so two people would count it the same way.
2. Volume. Units per week or month, from system data, not from memory.
3. Time per unit. Measure a real sample. Twenty to fifty units is enough for a first read. Do not ask people to estimate.
4. Quality. The current error, rework, or reopen rate. Humans make mistakes too, and you need that number.
5. Loaded cost per unit. Labor minutes times loaded hourly rate, plus tools, plus the cost of rework.

Here is a hypothetical, to show the math. Say a support team handles 4,000 tickets a month. A sample shows 9 minutes of handling time per ticket. Loaded labor runs $38 an hour. Cost per ticket is 9 / 60 x $38, or $5.70. The reopen rate is 6%.

Those two numbers, $5.70 and 6%, are now the bar. Everything else gets judged against them.

Pick the baseline owner now. It is usually the process owner or someone in finance, not the builder. The builder has an incentive to find a low baseline.

If you cannot measure a baseline in a week, pause the pilot. A pilot with no baseline cannot prove value, and I would not scale from its results.

Target and Stop Rule

A target says what winning looks like. A stop rule says what losing looks like. You need both, and you need them in writing.

The target. Pick one hero metric. Set a minimum target and an aspiration, and let the minimum decide. Using the hypothetical above:

• Hero metric: all-in cost per resolved ticket.
• Minimum target: $4.00 or less, about 30% below baseline.
• Guardrail: reopen rate at or below 6%. No new category of high-severity error.

“All-in” matters. It includes model and tool spend, plus the human minutes spent reviewing and fixing what the agent did. An agent that “saves” 5 minutes and creates 4 minutes of review has not saved much.

The stop rule. Annie Duke calls these kill criteria. In an interview with David Epstein, she says to think in advance about the signals that would tell you to walk away, then commit to walking away when you see them. She adds that it is not only a state of the world. It is also a date.

I use four kinds of stop rule. A pilot should have at least two.

Stop rule	Example wording
Result at checkpoint	By day 30, if the agent resolves under 40% of tickets with no human edit, stop or pivot.
Budget cap	If total pilot spend passes $X before the checkpoint, pause and review.
Safety trip wire	One severity-1 error (wrong refund, data exposure) pauses the pilot until the cause is fixed.
Date	On the decision date, a decision is made. No automatic extension.

The checkpoint and the date do the most work. Keep the pilot window short. A short window forces a decision, and it caps the cost of being wrong. TMA’s one-week pilot methodology is built around that idea.

Write the stop rule while everyone is still neutral. After three weeks of effort, people defend the pilot. That is human, and it is the reason you decide early.

Quality, Speed and Cost Metrics

Pick few metrics. One hero metric, two or three supporting ones, and guardrails. More than six and the decision meeting turns into a metrics tour.

Group them like this:

Group	Metrics to consider	Watch for
Quality	Accuracy against a labeled sample, error rate, reopen rate, escalation rate	Agent scored only on cases it chose to handle
Speed	Cycle time, queue age, time to first response	Faster output that needs slower review
Cost	All-in cost per completed case, human correction minutes per case, run cost per month	Pilot prices that will not hold at full volume
Adoption	Share of eligible cases routed to the agent, user override rate	A tool people quietly route around
Risk	Severity-1 incidents, policy violations, data access exceptions	Rare events that a small sample cannot show

Two traps come up often.

Counting only the wins. Cost per successful action should divide all pilot spend by the actions that passed your quality bar, not by every action the agent attempted. Failed attempts cost money too.

Pilot economics that do not scale. A pilot on 200 cases with heavy hand-holding does not predict cost at 20,000. Ask the builder what changes at full volume: model cost, review staffing, exception handling.

For ROI math beyond the pilot, use the method in how to calculate AI agent ROI. The metrics here feed that model. They do not replace it.

The Decision Meeting

Hold the decision meeting on the date written in the charter. Run it as a criteria review, not a results presentation.

The agenda is short:

1. Read the charter’s target and stop rule aloud.
2. Show the measured result against each. No new metrics added.
3. List any guardrail breaches and any severity-1 incidents.
4. Choose one of four outcomes: scale, iterate, pivot, stop.
5. Record the decision, the reason, and the next date.

Who makes the call? One named person. I recommend the executive owner of the process the agent touches, the person with budget and authority over it. Not the builder. Not a committee.

Others have roles:

• Process owner: confirms the baseline and the measured results.
• Finance partner: signs the cost numbers. This is the CFO’s check, and it is what makes the result credible. For the finance view, see the CFO guide to AI agent investment.
• Risk or security: reports on guardrails and incidents.
• Builder: presents data and answers questions. Does not vote.

NIST’s Playbook points the same way. Subcategory MEASURE 2.8 asks organizations to document “go” and “no-go” decisions made by accountable parties. A written, signed decision is also your audit record.

A Rubric for Scale, Iterate, Pivot, Stop

Four outcomes cover almost every pilot. Define them before the build, with thresholds.

The thresholds below are my defaults. Adjust them for your risk and your cost of delay.

Outcome	Use it when	What happens next
Scale	The hero metric meets the minimum target on a representative sample. No guardrail breached. Run cost at full volume is known. An owner is staffed.	Write the production plan. Fund it. See POC vs pilot vs production for what changes.
Iterate	The result is within reach of the target (I use at least 70% of the targeted gain), and the gap has a named, fixable cause.	One extension, fixed length, same metrics. Only one. A second iteration is a stop or pivot.
Pivot	The target is missed, but the data shows value in a different scope, workflow, or user group.	Write a new charter with a new baseline. Treat it as a new pilot.
Stop	The result falls below the stop line at the checkpoint. Or a guardrail breach has no fix. Or no one owns the process.	Shut it down. Record what you learned. Redirect the budget.

Two rules keep the rubric honest.

First, a missed minimum target is a “no” even if the secondary findings are interesting. Interesting is a reason to pivot, not to scale.

Second, a pilot that hits its target on a curated sample with a dedicated team has shown the agent can work. It has not shown your organization can run it. Check that the scale decision includes who operates it, who reviews exceptions, and what it costs per month.

Stopping is a good outcome when the charter did its job. A stopped pilot that was scoped well costs little. The expensive pilot is the one nobody can stop.

The One-Page Charter Template

Copy this. Fill it in before the build starts. Get the decider and a finance partner to sign.

AI Pilot Charter

1. Workflow and unit of work: one sentence. What is a “case”?
2. Decider: name and title. The person who makes the scale-or-stop call.
3. Process owner and finance partner: names.
4. Baseline (measured, with date): volume, time per unit, quality rate, loaded cost per unit. Sample size and source.
5. Hero metric: one, with units.
6. Minimum target and aspiration: for the hero metric.
7. Supporting metrics: two or three, with targets.
8. Guardrails: limits that cannot be crossed (error types, data access, escalation rules).
9. Stop rules: checkpoint result, budget cap, safety trip wire.
10. Pilot window and decision date: start, checkpoint, decision.
11. Rubric: thresholds for scale, iterate, pivot, stop.
12. Scale conditions: run cost at full volume, owner, review staffing, support model.
13. Signatures and date.

If a field is blank, the pilot is not ready to start.

Bottom Line

A pilot needs a baseline, a target, a dated stop rule, and one decider. Write them first.

Score the result against the charter on the decision date. Scale, iterate, pivot, or stop. Then record why.

My recommendation: do not start a build until the baseline is measured and the charter is signed. Use the template above, and book the decision meeting on day one.

FAQ

How do I decide whether to scale an AI pilot? Compare the measured result to the target written in the charter. Scale if the hero metric meets the minimum target on a representative sample, no guardrail was breached, and you know the run cost at full volume. If any of those is missing, iterate once, pivot, or stop.

What baseline do I need? Measure the current process on the same unit of work the agent will handle. You need volume, time per unit, quality rate, and loaded cost per unit, from a real sample. Estimates from memory are not enough. Without a baseline, you cannot prove improvement.

What is a good stop rule? One with a number and a date. For example: by day 30, if fewer than 40% of cases resolve without human edits, stop or pivot. Add a budget cap and a safety trip wire. Write the rule before the build, while nobody is attached to the result.

Which metrics prove ROI? All-in cost per completed case is the core one. It includes model and tool spend plus human review time. Pair it with a quality metric (error or reopen rate) and a speed metric (cycle time). Then project the numbers at full volume, not pilot volume.

Who makes the call? One executive who owns the process and its budget. A finance partner signs the cost numbers, the process owner confirms results, and the builder presents data. Committees drift. A named decider does not.

How long should an AI pilot run? Short enough to force a decision. The checkpoint and decision date go in the charter. A pilot with no end date has no stop rule.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect