Questions to Ask an AI Agent Vendor Before You Sign

The questions to ask an AI vendor are not the same as the questions you ask a SaaS vendor. An agent does not just store your data. It reads it, decides something, and acts.

So the questionnaire needs to cover what the agent can touch, who approves what, and what gets written down.

Below is the list I’d use. It is grouped by who on your team should own the answer. Each question has a good answer and a dodge. The last section is a one-page version you can print.

Why generic software questionnaires fall short

Your security team already has a vendor questionnaire. It was built for software that hosts data or processes transactions. It asks about encryption, uptime, and access control. Those still matter.

It does not ask what happens when a model reads a poisoned document and then calls a tool with write access.

Three gaps show up again and again:

• The model is a supply chain. NIST’s Generative AI Profile (July 2024) lists “value chain and component integration” as its own risk. It says the many third-party components in these systems can make it “difficult to attribute issues in a system’s behavior to any one of these sources.” Your vendor may sit on top of a model provider, a vector database, and three other services.
• The system acts. OWASP calls the failure mode excessive agency. It has three root causes: too much functionality, too many permissions, and too much autonomy.
• The behavior changes without a release. A model swap or a prompt edit can change outputs while the contract stays the same.

Questions to ask an AI vendor: business owner

The business owner owns the workflow. These questions test whether the vendor understands your work and will be measured against it.

1. What is the single metric this agent will move, and how will we measure it? Good answer: a named metric, a baseline step, and a plan to measure on your data. Dodge: “customers see big efficiency gains.” If the vendor will not commit to a measure, they cannot be held to one.

2. Which parts of the workflow does the agent do, and which parts stay with my team? Good answer: a clear split, with the handoff points named. Dodge: “it handles everything end to end.” Nothing handles everything end to end on day one.

3. What does a pilot look like, and what happens at the end of it? Good answer: a short pilot on real data with a decision date and written stop criteria. Dodge: a six-month “discovery phase.” See how to choose an AI agent development company for what to expect from a credible process.

4. Who on your side is accountable when it gets something wrong? Good answer: a named role and an escalation path. Dodge: “the model is probabilistic, so errors are expected.” Errors are expected. Ownership of them is still the vendor’s job to define.

5. What does it cost per completed task at my volume, and what drives that number up? Good answer: a worked example with token usage, tool calls, and review time. Dodge: a per-seat price with no usage story. Usage-based systems can surprise you.

Security and data questions

These go to your security lead and your privacy counsel. They overlap with the standard questionnaire, but the answers have AI-specific traps.

6. Is my data used to train your models? This is the question everyone asks. Ask it precisely: are inputs, outputs, files, and feedback used to train, fine-tune, or improve any model, yours or a third party’s?

Good answer: a written exclusion in the contract, not only on a web page. For reference, OpenAI’s enterprise privacy page (updated January 2026) says business data is not used for training by default unless you opt in, and that the opt-in covers things like feedback mechanisms. A good vendor can tell you which of their own toggles count as an opt-in.

Dodge: “we don’t train on your data” with no mention of feedback buttons, telemetry, or what the underlying model provider does.

7. Where does inference run, and where does data rest? Good answer: a region, a cloud, and a diagram. The diagram should show every hop, including the model provider. Dodge: “it’s in the cloud.” If you have data residency rules, ask for the answer per environment. My post on data sovereignty for AI agents covers what to pin down.

8. How long is data retained, and can retention be set to zero? Good answer: a retention period you control, and a zero-retention option in writing for the endpoints you will use. OpenAI’s page, for example, says API inputs and outputs may be retained up to 30 days for most endpoints, with zero data retention available for eligible endpoints and qualifying use cases. Notice the “eligible” and “qualifying.” Ask the same of your vendor and of their model provider.

9. Who are your subprocessors, and how do I find out when the list changes? Good answer: a published list, a notice period, and a right to object. Dodge: “available on request.”

10. What certifications do you hold, and what is their scope? SOC 2 Type 2 is table stakes. ISO/IEC 42001, published in December 2023, is the AI management system standard. Treat it as a plus, not a gate. Ask whether the scope covers the agent product you are buying or only the corporate network.

11. How do you test for prompt injection? OWASP’s guidance is plain that indirect injection, where the model reads hostile content in a web page or file, is a core risk. Good answer: they name the controls (segregating external content, least-privilege tools, human approval for risky actions) and describe adversarial testing. Dodge: “our model is trained to resist it.”

If you want a deeper security list, I wrote one in the enterprise AI security checklist. The Cloud Security Alliance also publishes an AI-CAIQ, a yes/no control questionnaire for AI systems.

Agent-specific questions: tools, permissions, approvals

This is the section the GRC checklists skip. It decides whether the agent is safe to run.

12. What tools can the agent call, and can I restrict each one? OWASP’s mitigations for excessive agency are a useful rubric: minimize the number of extensions, minimize what each one can do, avoid open-ended tools, and minimize their permissions. Good answer: a tool list, with scopes you can turn off one by one. Dodge: a single “connect to your systems” integration with admin credentials.

13. Whose identity does the agent act under? Good answer: the agent runs in the end user’s context or under a scoped service identity, and authorization is checked in the downstream system, not left to the model. OWASP recommends exactly that. Dodge: a shared super-user account.

14. Which actions need a human approval, and can I change that? Good answer: a policy you configure, with approval required by default for anything that writes, pays, deletes, or sends. Dodge: “approvals slow down the value.” Start strict and loosen with evidence. See human-in-the-loop for how that looks in practice.

15. What is the kill switch and how fast does it work? Good answer: a documented way to pause the agent or revoke its credentials in minutes, tested on a schedule. Dodge: “just contact support.”

16. What does the vendor log? Good answer: every model call, tool call, input, output, approval, and error, tied to a run ID, exportable to your systems. NIST’s profile suggests policies to “record and track GAI system reported errors, near-misses, and negative impacts.” If you are a deployer of a high-risk system in the EU, Article 26 of the AI Act requires you to keep the logs under your control for at least six months. So ask who holds the logs and whether you can get them. Dodge: “we log for debugging” with no export.

17. How do you change prompts, models, and tools, and how do I find out? Good answer: versioned changes, a regression eval before release, and advance notice for model swaps. Dodge: “we continuously improve the system.” Continuous change with no notice is a risk by itself.

Legal and exit questions

Counsel owns these. Sit in on the call anyway. The operational detail matters.

18. How do I exit the contract? This is the question buyers forget until it hurts. Good answer: data export in a standard format, deletion confirmed in writing, a transition period, and no penalty for taking your prompts, configs, and eval sets with you. Dodge: “you can always cancel,” with no word on what you keep.

19. Who owns the outputs, the prompts, and anything built on my data? Good answer: you own inputs and outputs, and the contract says what the vendor keeps (usually the platform, not your configuration). OpenAI’s page states that you “retain all rights to the inputs you provide” and own outputs “to the extent permitted by law.” Ask your vendor for the same language, then ask about derived assets such as fine-tuned models and embeddings.

20. What happens if the vendor changes its terms? The FTC warned AI companies in January 2024 that quietly changing terms of service to use customer data for training can break the law. That protects you in some cases. It is not a substitute for a clause that requires notice and your consent for any change to data use.

21. Who is liable when the agent acts wrongly? Say a vendor’s agent sends a wrong payment instruction because it misread an invoice. Who eats the cost? Good answer: a clear split tied to your approval design. If you approved the action, you probably own it. If the agent bypassed an approval, they do. Dodge: silence on the subject.

Reading the answers

You will get a lot of text back. Read for three things.

Specificity. Good answers have nouns: a region name, a retention number, a role title, a document name. Dodges have adjectives: “enterprise-grade,” “industry-leading,” “secure by design.”

Where the commitment lives. A promise on a web page can change. A promise in the contract or the DPA cannot change without your consent. For every important answer, ask “where is that written?”

The follow-up. Ask the same question a second way. If “we don’t train on your data” becomes “well, we use de-identified usage data” on the second pass, you learned something.

Score each answer as Pass (specific, in writing), Soft (specific, verbal only), Gap (vague), or Fail (wrong). Any Fail on data, tools, or exit stops the process until fixed. Many Gaps in the agent section means the vendor has not thought about running agents.

A one-page version to print

If you only have 30 minutes, ask these ten. They are my minimum.

#	Question	Owner
1	What metric moves, and how is it measured on our data?	Business
2	What is the pilot, and what are the stop criteria?	Business
3	Is our data used to train any model, in writing?	Security
4	Where does inference run and data rest?	Security
5	What is retained, for how long, and can it be zero?	Security
6	What tools can the agent call, with what scopes?	Security / Ops
7	Which actions need approval, and who configures that?	Ops
8	What is logged, and can we export it?	Ops
9	How are prompts and models changed, and how are we told?	Ops
10	How do we exit, and what do we keep?	Legal

Send these ahead of the call. Ask for written answers. Then use the call to probe the soft spots.

Bottom Line

Ask about the agent, not only the software. What it can touch, who approves, what is logged, and how you leave.

Judge answers by specificity and by where the promise is written.

My recommendation: send the ten-question table first, then go deep on the section where the vendor was vaguest.

FAQ

What should I ask an AI vendor before buying? Start with the metric they will move, then data use, retention, and where inference runs. Then ask what tools the agent can call, which actions need approval, what gets logged, and how you exit. The ten-question table above is a fast starting point.

Is my data used to train your models? Ask for the answer in the contract, not on a web page. Cover inputs, outputs, files, feedback, and telemetry, and ask the same about the vendor’s model provider. Many large providers say they do not train on business data by default, but opt-in features can change that.

Where does inference run? Ask for a region, a cloud, and a diagram of every hop, including the model provider. If you have residency rules, get the answer in writing for each environment you will use.

How do I exit the contract? Ask for data export in a standard format, written confirmation of deletion, a transition period, and the right to keep your prompts, configs, and eval sets. If the vendor cannot say what you keep, assume you keep less than you think.

What does the vendor log? It should log every model call, tool call, input, output, approval, and error, tied to a run ID. Ask whether you can export the logs and for how long they are kept. Under the EU AI Act, deployers of high-risk systems must keep logs under their control for at least six months.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect