10 Questions to Ask Before Hiring an AI Agent Builder

How to choose an AI agent development company comes down to ten questions. Most vendors cannot answer them with evidence.

I run one of these companies, so weigh that. I wrote the questions as a buyer would use them, and I’d want them used on us too.

Short version: ask for production proof, insist on owning what gets built, and judge vendors on a small paid build, not a slide deck.

Why self-ranked top-10 lists mislead

Search for this topic and you get lists. Agencies rank themselves first. Directories rank by who paid for placement or who has the most reviews. None of it tells you if a team can ship an agent that survives contact with your data.

The market makes this worse. Gartner describes “agent washing”: vendors rebranding assistants, RPA, and chatbots as agents. It estimates only about 130 of the thousands of agentic AI vendors are real.

That is a prediction, not a measurement. But it matches what I see. A lot of “agent” work is a chatbot with a nicer logo.

The upside is real too. Fortune’s coverage of the MIT NANDA report says purchases from specialized vendors and partnerships succeed about 67% of the time, and internal builds do much worse. So buying help is often right. Buying the wrong help is the expensive part.

If you are still deciding whether to buy at all, start with build vs buy for AI agents. If you have decided, here is the scorecard.

Questions 1 to 3: proof of production work

1. Which agents are running in production today, and for how long?

You want a count, a start date, and a workflow description. “Running in production” means real users, real data, and someone paged when it breaks. A demo environment does not count.

Red flag: a portfolio of prototypes, hackathon results, or “pilots in progress.”

2. Can I talk to a technical owner at a reference, not the executive sponsor?

Sponsors say nice things. The engineer or operator who lives with the agent tells you about the 2 a.m. failures.

Many good vendors cannot name clients because of NDAs. That is fine. See the reference section below for how to check without a name.

Red flag: references that are always C-level, always the same two companies, or “available after contract.”

3. What did you ship that failed, and what changed after?

Every team has a failure story. The question tests honesty and learning. A good answer names the cause (bad data, vague success metric, a tool permission too wide) and the process fix.

Red flag: “we have never had a failed project.” Or a failure that was always the client’s fault.

How to check a reference without a client name

Ask the vendor for these instead:

• Industry, company size range, and workflow, with names removed.
• A redacted architecture diagram of the production system.
• A redacted weekly report or eval summary from the live agent.
• A call with someone who ran it, under NDA terms they set.
• Permission to ask the reference one question: “Is it still running, and would you rebuy?”

If a vendor can supply none of these, they likely have nothing in production.

Questions 4 to 6: data, access, and ownership

4. Where does my data live, and where does it get processed?

Get the full path: your systems, the vendor’s systems, the model provider, any logging or tracing tool. Ask for a diagram. For regulated work, ask if the build can run inside your own cloud account. This is the core of data sovereignty.

Then ask about the model provider. Terms differ by vendor and tier. For example, OpenAI states that since March 1, 2023, API data is not used to train its models unless you opt in, and it offers a Zero Data Retention option. Your builder should know the terms of every model they plan to call, and put them in writing.

Red flag: “it’s all secure.” Or a vendor who cannot name their subprocessors.

5. Who owns the code, prompts, evals, and configuration at the end?

My recommendation: you own all of it. That includes source code, prompt files, evaluation datasets, tool definitions, and infrastructure scripts. Delivered in your repo, not theirs, from week one.

Watch for these:

• “License to use” language on custom work.
• Prompts stored only inside the vendor’s platform.
• Evaluation sets the vendor keeps.
• A proprietary runtime you cannot host yourself.

Any of these is lock-in. Sometimes lock-in is a fair trade for speed. But make it a decision, not a surprise. Ask what you would need to rebuild if the vendor vanished tomorrow.

6. What access does the agent need, and how do you limit it?

An agent that can act can also act wrongly. OWASP names “excessive agency” as a top LLM risk and traces it to excessive functionality, permissions, and autonomy. Its prevention advice is to minimize the tools an agent gets and what each tool can do.

Good answers include read-only access first, scoped service accounts, human approval on irreversible actions, and an audit log of every tool call.

Red flag: “we’ll need admin access to get started.”

For the full control set, see the enterprise AI security checklist.

Questions 7 and 8: pilot scope and acceptance test

7. What ships in week one?

This is the question I weight most. A serious team can show something running on real or realistic data inside the first week. It will be thin. One workflow, one path, one narrow tool set. But it runs.

A week-one deliverable should look like this:

• A working slice of the agent on your data, in your environment.
• A written success metric with a baseline number.
• A first evaluation set of 20 to 50 real cases from your team.
• A list of what is out of scope.
• A decision date for what happens next.

Red flag: week one is “discovery,” “workshops,” or “architecture review.” Those are fine in week zero. They are not a deliverable. I wrote more on this in our one-week pilot methodology.

8. How do we agree on “done” before you start?

You need an acceptance test written before the build. Not “users like it.” Something like: on these 50 historical cases, the agent matches a human reviewer at least X percent of the time, and every miss routes to a person.

Pick X with your operators, not with the vendor. Good vendors push back and ask for a baseline. That is a good sign.

Red flag: success defined as “deployed,” or measured by the vendor’s own dashboard only.

Questions 9 and 10: support and exit

9. Who is on call after launch, and what is the response time?

Agents drift. Source systems change, models get updated, inputs shift. Someone must watch quality after launch.

Ask for names or roles, hours, a response-time commitment in writing, and what monitoring ships with the agent. Ask what happens when a model provider changes or retires a model version. Your builder should have a plan, not a shrug.

Red flag: “support is billed hourly as needed” with no monitoring. You will find problems from user complaints.

10. How do I leave?

Ask for the exit plan on day one. What do you hand over, in what format, and how fast? Who pays for the transition? Is knowledge transfer included?

A confident vendor answers in two minutes. An insecure one gets defensive.

Red flag: termination fees larger than the pilot, auto-renewals with long notice windows, or no handover clause.

Scorecard: how to choose an AI agent development company

Score each vendor 0 to 2 per question. Zero means a red flag. One means partial. Two means a specific answer with evidence.

#	Question	Strong answer	Red-flag answer
1	Production agents	Count, dates, workflow described	Prototypes only
2	Technical reference	Operator call under NDA	Sponsor only, “later”
3	A failure	Named cause, process fix	”Never failed”
4	Data path	Diagram, named subprocessors	”It’s secure”
5	Ownership	You own code, prompts, evals	License to use
6	Access limits	Scoped, read-first, audit log	Admin access needed
7	Week one	Running slice on your data	Discovery workshops
8	Acceptance test	Written, measurable, pre-build	”Users like it”
9	Post-launch	Named on-call, monitoring	Hourly, no monitoring
10	Exit	Handover in writing	Fees, no handover clause

Total out of 20. My rough cutoffs, which are judgment and not research: 16 or more is a real candidate. 11 to 15 needs follow-up on the zeros. Under 11, move on.

Treat any zero on questions 5, 6, or 8 as a stop. Those three decide who controls your outcome.

Certifications help but do not replace these answers. A SOC 2 report covers controls against the AICPA’s trust services criteria such as security, availability, and confidentiality. ISO/IEC 42001 is a management system standard for organizations that develop, provide, or use AI. The NIST AI Risk Management Framework gives you a common vocabulary (Govern, Map, Measure, Manage) for risk talks. Ask what each one covers and which system it was scoped to. A certificate for the company’s HR system says nothing about the agent.

How to run the comparison in two weeks

You do not need a six-month RFP. Here is the plan I’d run.

Days 1 to 3: shortlist. Pick three vendors. Send the same ten questions in writing. Ask for written answers, not a call. Score them blind if you can.

Days 4 to 5: reference checks. Run the checks above on your top two. Drop anyone who stalls.

Days 6 to 12: a paid mini-build. Give your top two the same small task with the same data and the same acceptance test. Keep it to one workflow. Pay both. Paying filters out vendors who would rather sell than build, and it gives you fair output to compare.

Days 13 to 14: decide. Compare on four things: accuracy on your test cases, how clearly they explained misses, what you now own, and how it felt to work with them. Pick one, or walk away.

If your problem is a plain rules-based flow, you may not need an agent at all. AI agents vs traditional automation helps you check that before you spend anything. And if the pilot works, the next hurdle is moving it into daily use. See from experimentation to production.

Bottom Line

Ten questions, one scorecard, one paid mini-build.

Vendors who ship production agents answer with evidence, dates, and diagrams.

Insist on owning the code, prompts, and evals.

Next step: send the ten questions to three vendors this week and score the written answers.

FAQ

How do I evaluate an AI agent development company? Ask for production proof, data handling details, ownership terms, a week-one deliverable, and an exit plan. Score the answers on a simple 0 to 2 scale. Then test your top two with a small paid build on the same task. Accuracy on your own cases beats any pitch.

What should a week-one deliverable look like? A working slice of one workflow on real or realistic data, a written success metric with a baseline, a first set of 20 to 50 test cases, and a list of what is out of scope. It will be narrow. It should run. Workshops and diagrams alone are not a deliverable.

What are red-flag answers? “It’s all secure” with no data diagram. A license to use instead of ownership. Admin access required to start. Success defined as “deployed.” Support billed hourly with no monitoring. Termination fees with no handover clause. Any one of these deserves a hard follow-up.

Who should own the code and prompts? You should. That includes source code, prompt files, evaluation sets, tool definitions, and infrastructure scripts, delivered to your repository from the first week. If the vendor keeps any of it, get the reason in writing and know what you would rebuild if they left.

How do I check a reference without a client name? Ask for the industry, company size, and workflow with names removed, a redacted architecture diagram, a redacted eval summary, and a call with the person who operates the agent under NDA terms. Then ask one question: is it still running, and would they rebuy?

Should I pick a big firm or a small specialist? Neither size is safe by default. Judge on the ten questions. Big firms often have stronger security paperwork and slower delivery. Small teams often ship faster but need a closer look at on-call coverage and exit terms.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect