AI Agent Demo vs Production: Red Flags

A demo shows the best case. Production has to survive the worst one.

If a vendor will not run their agent on your data, show you a failure, open the logs, and name the person on call, you are looking at a demo. Not a product.

This post gives you five asks, a fair live test, and a one-page scorecard to use in the room.

Why demos look better than reality

A demo is built to succeed. The inputs are chosen. The prompts are tuned to those inputs. The tools behind it are connected to test data that never changes. Nothing in the room is allowed to go wrong.

Production is the opposite. Inputs are messy. Systems are slow or down. Users ask things nobody planned for. The same input can give a different answer on a different day.

That last point matters most for agents. The τ-bench paper from Sierra tested agents on multi-turn tasks with real tool calls and policy rules. It found that even strong function-calling agents completed fewer than half the tasks and were inconsistent: on retail tasks, the agent rarely got the same task right across eight repeated trials. The paper calls this metric pass^k. That test used 2024 models, so treat the exact numbers as dated. The lesson holds: a demo is one run. Production is thousands.

The market also makes this worse. Gartner said in June 2025 that many vendors are “agent washing,” meaning they relabel assistants, RPA, and chatbots as agents. Reporting on the release says Gartner estimates only about 130 of the thousands of agentic AI vendors are real, and predicts over 40% of agentic AI projects will be canceled by the end of 2027. That is a forecast from one analyst firm. Either way, it is hard to tell the real ones from the polished ones from the outside.

And the failure rate after the demo stage is high. IDC research with Lenovo, reported by CIO, found 88% of observed proofs of concept did not reach wide deployment. If you want the wider picture on why, start with why AI pilots fail.

Ask for a run on your data

First ask: “Run it on a sample of our data, live, while we watch.”

Not your data after the vendor has seen it for a week. Not a cleaned file. A sample you pick that morning.

What a fair live test looks like:

You choose 20 to 50 real past cases. Include the ugly ones: scanned pages, missing fields, odd formats, angry customers.
You know the right answers from your own records. The vendor does not see them.
The vendor runs the agent on the batch live, or you run it yourself.
You compare results to your answers, case by case. Not a summary slide.
You run a few cases twice. Do you get the same answer?

If the vendor says they need two weeks to prepare, that tells you something. They may need that time for a real integration. Or they may need it to tune the demo. Ask which one.

Red flags:

“We can only demo on our sample data.”
“Your data needs cleaning first.” (Maybe true. But then the demo shows nothing about your case.)
A big accuracy number with no test set, no method, and no error list.

Ask for a failure

Second ask: “Show me a case where it fails, and what happens next.”

Every real system fails. What you want to see is what it does when it fails. Anthropic’s own guidance on building agents recommends extensive testing in sandboxed environments, stopping conditions such as a maximum number of iterations, and pausing for human feedback at checkpoints or blockers. A vendor who has built for production will have all three.

Failure cases that matter for a buyer:

Failure	What good looks like
Input is missing a required field	Agent stops and asks, or routes to a person. It does not guess.
Source system is down or slow	Agent retries a set number of times, then queues the work and alerts someone.
Two sources disagree	Agent flags the conflict and shows both.
Question is out of scope	Agent says so and hands off.
Agent is unsure	It shows low confidence and routes to review.
Someone tries to push it off task	It refuses, and the attempt is logged.
Wrong answer reaches a customer	There is a recall or correction path, and an owner.

That last row is not theoretical. In February 2024 a Canadian tribunal held Air Canada responsible for wrong refund advice given by its website chatbot, rejecting the argument that the bot was a separate entity. It was a small claims decision and not binding precedent. But it shows where the liability lands. It lands on you, the company that deployed it, not on the vendor.

If the vendor cannot show a failure, they have not looked for one.

Ask to see logs and approvals

Third ask: “Show me the log for one run, start to finish.”

You want to see, for a single case:

the input as received
each tool or system the agent called, and what came back
the output
which model and prompt version produced it
who approved it, and when, if a person was involved
cost and time for the run

If you cannot rebuild why the agent did something, you cannot fix it, defend it, or audit it. I go deeper on this in what to log for an AI agent audit trail.

Also ask how approvals work. OWASP’s 2025 list names “excessive agency” as a core risk for LLM apps, driven by too much functionality, too many permissions, or too much autonomy. Its mitigations include least-privilege access, narrow tools, and human approval of high-impact actions. So ask:

What can the agent read? What can it write?
Which actions need a human to approve first?
Does the downstream system check permissions, or does it trust the agent?
Can I turn the agent off in one step?

A vendor who answers “it has admin access so it can do anything” has given you a red flag.

Ask who is on call

Fourth ask: “Who gets paged when this breaks, and what is the response time?”

A production system has an owner. You want a name or a role with a schedule, not “our support team.” Ask for:

the support hours, in your time zone
how incidents are classified and how fast each class is answered
how model or prompt changes are tested and released
how you are told before a change that could alter behavior
how they roll back a bad release
what happens to your data and logs if you leave

Model providers change and retire models. If the vendor depends on one, ask what happens when it is updated. The honest answer includes a regression test set. My notes on that are in AI agent testing strategies.

Ask for production references

Fifth ask: “Give me two customers running this in production, in a workflow like mine.”

Pilots do not count. Case study PDFs do not count. You want a call with someone who runs it.

Questions for the reference:

How long has it been live, with real volume?
What share of cases does it handle without a person?
What went wrong in the first 90 days?
Who owns it on your side, and how many hours a week does that take?
Did the cost match the quote?
Would you buy it again?

The Omdia 2025 survey, reported by The Register, found only 9% of enterprises said more than half of their PoCs were accepted into production. So a vendor with several real production customers is rarer than the marketing suggests. If the references are all “in pilot,” that is your answer.

Scoring the demo

After the demo, score it. Do it the same day, with the whole buying team, before the sales follow-up shapes your memory.

Give each row 0, 1, or 2.

0: not shown, or the answer was vague
1: shown, with gaps
2: shown clearly, on your data or with evidence

Then decide using the rules below the table.

A one-page demo scorecard

#	Check	0 / 1 / 2	Notes
1	Ran live on our sample, chosen by us
2	Results compared to our answers, case by case
3	Same input twice gave a consistent result
4	Showed a real failure and the handling
5	Showed a full log for one run
6	Can say what it reads, what it writes, and what needs approval
7	One-step off switch or kill path
8	Named owner and on-call process
9	Change and rollback process described
10	Two production references, similar workflow, real volume
11	Pricing includes upkeep, not just build
12	Data stays where we need it to

How I read the total (24 is the maximum). This is my own rubric, not a standard:

20 to 24: strong. Move to a bounded pilot.
14 to 19: promising but unproven. Ask for the missing items in writing before you proceed.
Under 14: it is a demo. Do not sign a long contract on it.

One override: if rows 4, 5, or 8 score 0, stop. No failure handling, no logs, or no owner means you cannot run this safely, whatever the total says.

For the next step, use the questions in what to ask an AI agent vendor and the criteria in how to choose an AI agent development company.

Bottom Line

A demo proves a vendor can make a good video. It does not prove the system will hold up on your data.

Make five asks: a live run on your sample, a failure, the logs, the on-call owner, and production references. Score them the same day.

If they will not show a failure or a log, walk away. Then run a bounded pilot, with one metric, before you commit.

FAQ

How do I tell a demo from a production system? Ask for a live run on your own sample data, a real failure case, a full log of one run, and the name of the person on call. A demo can fake the first. It struggles with the last three. Also ask for production customers with real volume.

What should I ask to see in an AI vendor demo? Five things: a run on your data, a failure and how it is handled, logs and approvals for one case, the on-call and change process, and two production references in a similar workflow. Score each one on the same day.

What failure cases matter most? Missing input, a source system that is down, conflicting data, out-of-scope questions, low confidence, and attempts to push the agent off task. The key test is whether it stops, flags, or hands off, rather than guessing.

What is a fair live test? Pick 20 to 50 real past cases yourself, including ugly ones. Keep the right answers hidden from the vendor. Run the batch while you watch, compare case by case, and repeat a few cases to check for consistent answers.

What do production references look like? A customer who has run the system with real volume for months, can say what share of cases it handles alone, can describe what broke early on, and names an internal owner. Pilot customers and marketing case studies are not production references.

Is a polished demo a bad sign? No. A polished demo with nothing behind it is. If the vendor also passes the failure, log, and on-call checks, polish is a bonus.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect

AI Agent Demo vs Production: Red Flags to Check