Validate an AI Product Idea With Three Customers

You can validate an AI product idea without writing code. Do the job by hand for three real customers.

Time it. Grade it. Write down what they push back on. Then ask them to pay or commit to the next step.

If that goes well, you know what to automate. If it does not, you spent three weeks, not six months and a build budget.

Why Build Last

The build is the expensive part. It is also the part where you learn the least about the customer.

Most AI ideas from domain experts are not wrong about the problem. They are wrong about the scope, the data, the exceptions, or who pays. A manual test shows you all four before a developer touches the project.

Paul Graham’s classic YC essay, Do Things that Don’t Scale, makes the case for starting with manual work. He writes that “the most common unscalable thing founders have to do at the start is to recruit users manually.” He also describes a more extreme version, where you “don’t just use your software, but are your software.”

That is the idea here. You are the product for three weeks. AI tools can help you do the task faster, but you stay in the loop and you watch what breaks.

Two names for this approach come up often:

• Concierge test. You manually help users reach their goals to learn whether they need what you offer. The customer knows a person is doing the work.
• Wizard of Oz test. Nielsen Norman Group defines it as a method where a user interacts with an interface that appears autonomous but is fully or partly controlled by a human. It suits technology that is hard to test with a static prototype.

I prefer the concierge version for B2B work. Business buyers are fine with “a specialist does this for you, with AI tools.” Hiding the human adds risk and teaches you less.

If you are a domain expert starting from scratch, this post sits inside a bigger path. See how to build an AI startup as a domain expert and turn your expertise into an AI product.

Pick Three Willing Customers

Three is a floor, not a statistical sample. It is enough to spot a pattern or kill a weak idea cheaply. It is not enough to prove a market. I will say it plainly so nobody oversells the result later.

How do I find three customers? Start where you already have standing.

• Former colleagues and vendors who still do the job you used to do.
• People in your industry’s LinkedIn groups, Slack or Discord communities, and Reddit forums. YC’s guide to talking to users names LinkedIn, Reddit, Slack or Discord communities, and in-person events as the most common places founders find people to interview.
• Trade events and association meetings.

Pick three that meet these tests:

1. They do the task now. Weekly at least. Not “we’ve been meaning to”.
2. They have the pain in their own budget or time. Someone who owns the cost, or whose team does.
3. They will share real work. Real documents, real tickets, real emails. Samples cleaned up for a demo hide the exceptions that break products.
4. They are not your friends. Friends compliment. Strangers with a problem tell you the truth.

Avoid three copies of the same company. Pick differences in size or setup, so you can see if the workflow holds across them.

Set the scope in writing before you start. One workflow. A fixed window, say three weeks. A fixed volume, say 50 to 100 units each. A short agreement on data handling. If a customer will not let you touch real data, that is a finding. Write it down.

Do the Task by Hand

Run the work the way the finished product would, using whatever tools you want behind the scenes.

Say you spent 12 years running accounts payable at a distribution company. Your idea is an agent that codes vendor invoices and flags exceptions. For the manual test, each customer sends you their invoices as they arrive. You code them using a general AI chat tool plus your own review, then send the results back in the format they use today. You do not build a platform. You build a spreadsheet and a routine.

Keep these rules:

• Use the real inputs. Whatever arrives in their inbox or queue, in the state it arrives.
• Use AI where you plan to use it. If the product will draft with a model, draft with a model. You are testing the real workflow, including how often the draft is wrong.
• Review every output before it goes back. You are the quality gate. Note every fix you make.
• Keep the turnaround honest. If the product promises same-day, deliver same-day.
• Do not demo features. Do the job. Features are what you build later.

On discovery calls, use the same discipline YC recommends. Gustaf Alströmer’s advice is to avoid introducing your idea until late in the call, or not at all, because it biases answers. Ask how they do the task today, what is hardest, why, how often, and what they do about it now. Look for behavior, not opinions.

Record Time, Quality and Objections

If you do not log it, you only have a feeling. Keep one sheet per customer with the same columns.

Column	What to record
Unit	One invoice, one ticket, one claim. Same definition for all three customers
Their time today	Minutes their team spends per unit now (ask, then watch if you can)
Your time	Minutes you spent, split into prep, AI-assisted work, and review
AI draft quality	Right, fixed by you, or wrong. Tag the reason for each fix
Exception type	Missing data, odd format, policy judgment, system lookup, other
Customer feedback	Their words, not your summary
Objection	What they pushed back on
Ask	Did they offer money, data access, or an introduction?

After the window, add it up:

• Time per unit, split into the part a model did well and the part you had to fix.
• First-pass accuracy. How often the AI draft needed no fix.
• Top five exception types by count and by minutes.
• Repeat rate. How many units across all three customers looked the same.
• Objections, ranked. Count how many customers raised each.

Watch the “fix” column closely. It is your future error budget. If you fix one of every three drafts, the product needs more than a prompt. Anthropic’s guide to building effective agents says the key to success is “measuring performance and iterating on implementations.” You are doing the measuring early, with no code.

What Proof Is Enough

Compliments are not proof. “This is great, let us know when it’s ready” costs the customer nothing.

Proof is a customer giving up something real. Ranked from weakest to strongest:

Signal	Strength
Praise, “interesting”, “send me a deck”	None
Agreed to a second call	Weak
Gave you more real data without being asked	Medium
Introduced you to a colleague or their boss	Medium
Signed a letter of intent or agreed in writing to a paid pilot	Strong
Paid for the manual service, or asked to keep it running	Strongest

Charge for it if you can, even a small amount. In an a16z conversation, Pilot’s Waseem Daher said his company charged from day one because it is “the clearest signal that you are actually solving a real problem.”

Here is my pass mark. These are my defaults for a three-customer test, not an industry standard:

• At least two of three agree to pay, or commit in writing to a paid pilot.
• The manual workflow cut their time per unit by about 30% or more, counting your review time.
• The same workflow shape repeated across all three. Fewer than half the units were one-off.
• No objection that you cannot design around (for example, “legal will not allow data to leave our network”).

When do I stop? Stop, or change the idea, if two of three would not pay, if every customer’s process was different, or if the top objection is something a product cannot fix. Set a date in advance for that call. Do not extend the test because you like the idea. For the same stop-rule logic on a bigger pilot, see AI pilot success criteria.

Decide What to Automate

Now the logs tell you where to build. Do not automate the whole job. Automate the pieces with the best return.

Rank each step in the workflow on three things:

1. Minutes per unit. Where does the time go?
2. Repeatability. How similar is the step across units and customers?
3. Cost of a wrong answer. What happens if the automated step is wrong?

Automate the steps that are high in minutes, high in repeatability, and low in error cost. Keep a human on the steps where an error is expensive or where judgment varies. Leave the rest manual for now.

Also ask if you need an agent at all. Anthropic’s guidance is to find “the simplest solution possible, and only increasing complexity when needed.” Sometimes a template, a rules check, and one model call beat a multi-step agent. For help choosing the first target, use which workflows to automate first.

Your logs also give you a ready-made test set. The cases where you fixed the AI draft are exactly the cases a builder should test against.

Brief a Build Partner

A good build partner will ask what proof you have. Show up with it.

Bring this package:

• A one-page workflow map. Inputs, steps, outputs, who touches each step.
• The three logs. Time, accuracy, exceptions, objections.
• A set of real examples. At least 30 to 50 units with the correct outcome marked, including the messy ones. This becomes the test set.
• Error cost. What a wrong output costs at each step.
• The commercial proof. Who agreed to pay, how much, and what they signed.
• Data and access limits. Where the data lives, who can see it, and what must stay inside the customer’s environment.
• Your role. What you will own after launch: customer relationships, exception review, domain judgment.

Be wary of any partner who will build without asking for any of this. Also be wary of a partner who asks for proof but offers no view of cost or timeline in return.

If you want to understand how partnerships like this are structured, read what an AI venture studio is.

Bottom Line

Do not build first. Do the work by hand for three customers, log it, and ask for money or a written commitment.

If two of three commit and the logs show a repeatable workflow with real time savings, you have a case. If not, you found out cheaply.

Next step: pick your three customers this week and write the one-page scope.

FAQ

How do I test an AI idea cheaply? Run a concierge test. Deliver the service by hand to a few real customers, using AI tools behind the scenes and checking every output yourself. Log your time, the AI’s accuracy, and the customer’s reaction. It costs your time and some tool fees, not a development budget.

What counts as proof that an AI idea will work? Commitment, not praise. A customer who pays for the manual service, signs a letter of intent for a paid pilot, or brings you more data and an introduction is giving you real evidence. A compliment is free, so it tells you very little.

How do I find three customers? Start with people who currently do the task: former colleagues, vendors, and peers. Then look in industry LinkedIn groups, Slack or Discord communities, Reddit forums, and trade events. Pick people who do the task weekly and who will share real work, and avoid close friends.

What do I measure? Per unit of work: their time today, your time, how often the AI draft needed a fix, and why. Also count exception types, repeat rate across customers, and objections. Add whether each customer offered money, data, or an introduction.

When do I stop? Set the date before you start. Stop or rework the idea if most customers will not pay, if every customer’s process differs, or if the top objection cannot be solved by a product. Do not extend the test because the idea feels promising.

Is three customers enough? For a cheap first read, yes. It will not prove a market. It will show a pattern, expose weak assumptions, and give you evidence to take to a build partner. Plan to test with more customers once you have a prototype.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect