What Is Pilot Purgatory?

Pilot purgatory is what happens when an AI pilot never ends. The demo worked. The team is “iterating.” Months later, there is no production release, no business case, and no decision to stop.

It is the most common way AI projects fail, because nobody has to admit failure. The pilot just keeps going.

What the research says

Be careful with big numbers here, because they measure different things.

Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027. Its reasons: escalating costs, unclear business value, or inadequate risk controls. Gartner also said many vendors engage in “agent washing,” relabeling assistants, RPA, and chatbots as agents, and estimated only about 130 of the thousands of agentic AI vendors are real.

A report from MIT’s NANDA initiative, covered by Fortune in August 2025, said 95% of the generative AI pilots in its dataset were falling short, with little measurable effect on profit and loss. It pointed to a “learning gap” in tools and organizations, meaning flawed integration into enterprise workflows, rather than model quality. It is one report with its own definition of success. I treat it as a direction, not as a precise failure rate.

Reliability is part of the problem too. The tau-bench study found that even strong function-calling agents were inconsistent across repeated trials, with pass^8 under 25% in its retail domain. A demo shows one good run. Production needs the same result every time.

Both point to the same causes: cost, unclear value, weak integration, weak controls. None of them are model problems.

Warning signs

The pilot has no end date.
Nobody can state the metric the pilot must move, in one sentence.
The “success criteria” were written after the demo.
The test data is cleaner than real data.
A vendor or team lead says “we need a few more weeks” for the third time.
No one owns the production decision.
Security, legal, or IT first see the project after the build.
Users in the pilot are volunteers who like AI, not the people who do the work.

Why pilots stall

No baseline. Without today’s cycle time, error rate, or cost, you cannot show improvement.
Vague scope. “Help with customer service” has no finish line. “Draft first replies for billing tickets” does.
Prototype data. The pilot ran on a clean export. Production has missing fields and odd formats.
No evaluation. The team judges quality by reading examples, so nobody can say whether a change made things better or worse.
Risk reviews arrive last. Security finds the agent has broad access, and the project resets.
No owner after launch. Nobody is paid to keep it working.

I covered these in more detail in why AI pilots fail.

How to get out

Pick one workflow. One input, one output, one owner.
Write the hero metric before building. For example, “median time to first response drops from X to Y.” Get the baseline first.
Set a short clock. A fixed window, with a go or no-go at the end. See the one-week AI pilot methodology.
Use real data from day one. Messy is fine. It is the truth.
Build an evaluation set early. Twenty real cases with expected outcomes beat opinions.
Bring security and the process owner in during week one.
Define the production path. Who runs it, who is paged, what is the rollback.
Allow a no. A pilot that ends with “this does not pay back” is a success. You saved a year.

For the business case side, see AI agent ROI and executive buy-in for AI agents.

A short example

Say a finance team starts an “AP automation pilot.” Three months in, it handles invoices well in demos. But nobody recorded how many invoices an analyst clears per hour today, and the agent still needs a person to check every line. The team resets the pilot. They choose one vendor class, measure the current handling time, set a two-week window, and require 95% match accuracy on a held-out set of real invoices before any rollout. At the end they have a number, and a decision.

The 95% is a hypothetical target in this example, not a benchmark.

Common mistakes

Treating the pilot as a science project.
Letting a champion’s enthusiasm replace evidence.
Expanding scope mid-pilot.
Running without a comparison group or baseline.
Never killing anything.

A go or no-go template

Before a pilot starts, write this down and have the process owner sign it:

Workflow: one named process.
Metric and baseline: what we measure today.
Target: the result that justifies production.
Window: start date and decision date.
Data: where the real inputs come from.
Risks reviewed: security, legal, privacy, with names.
If it passes: who runs it and who pays.
If it fails: what we stop.

At the decision date, compare results to the target. No new scope, no extra weeks, unless the sponsor signs a new version.

An AI agent is the thing being piloted. Agentic workflows are the multi-step processes it runs. Human in the loop review is often the bridge from pilot to production. For more on the Gartner numbers, see Gartner’s 40 percent agentic AI prediction and AI agent predictions for 2026.