August 12, 2026

How to vet an AI vendor's sales pitch: 7 questions that separate signal from hype

Every AI vendor demo looks incredible. The bot handles the tricky question, the dashboard shows beautiful metrics, and the salesperson has an answer for everything. Then you sign, go live, and discover the demo was the product. If you're a small business owner sitting through pitches right now, these seven questions will tell you within an hour whether you're talking to a real vendor or a well-rehearsed pitch deck.

The pattern to watch for is simple: vendors with working systems talk about failure modes, monitoring, and unit costs. Vendors selling hype talk about capabilities, roadmaps, and vision. Force the conversation onto the first track and see who's still standing.

Question 1: "Can I talk to a customer who has been running this in production for over six months?"

Six months is the magic number. Anything less and you're talking to someone still in their honeymoon phase, before the edge cases start piling up and the novelty wears off. Real production systems break in interesting ways between month three and month six.

Green flag answer: "Yes, here are three references. Two are in your industry. I'll email intros today." A confident vendor with real customers has references queued up before you ask.

Red flag answer: "We can arrange an intro after signing." That's not how references work. References exist to reduce your risk before you commit, not validate your decision after. Also watch for: "Our customers are under NDA," "We're in stealth with our biggest accounts," or a single reference who turns out to be an investor, advisor, or the founder's college roommate.

If they cite a Fortune 500 logo on their site, ask specifically which team uses it and for how long. "Company X is a customer" often means one pilot in one department that quietly got shelved.

Question 2: "When's the last time this system got something wrong in production, and how did you catch it?"

This is the single most revealing question you can ask. Every AI system gets things wrong. If a vendor tells you theirs doesn't, they either don't have real usage or they're not looking at their logs.

Green flag answer: A specific story. "Two weeks ago our support agent misclassified a refund request as a shipping question because the customer used an unusual phrase. We caught it because our confidence-scoring flagged it for human review, and we added the phrasing pattern to our eval set the same day." Notice the structure: specific incident, specific detection mechanism, specific fix.

Red flag answer: "Honestly, it just works." Or a vague answer about "occasional hallucinations we've mostly solved." Or a pivot to talking about how their model is more accurate than a competitor's. If they can't name a real failure from the past month, they're either not monitoring or not being honest.

Real operators have battle scars and they're happy to show them. It's a form of expertise flex — they know the failure modes because they've fixed them.

Question 3: "What's your evaluation process? How do you know a new model version is better than the old one?"

This one weeds out the majority of AI vendors instantly. The models underneath (GPT-4, Claude, Gemini, whatever) get updated constantly. Vendors have to decide whether to upgrade, and that decision requires an eval set — a stable collection of test cases with known correct answers that the system runs against every time something changes.

Green flag answer: "We maintain a golden dataset of about [some number] real conversations with graded outcomes. Every model change or prompt change gets scored against it before we ship. Here's roughly what a row in that dataset looks like." They may not show you the whole thing, but they can describe its structure and how it grows over time.

Red flag answer: "We test it before we deploy." Meaning what, exactly? Someone poked at it for ten minutes and it seemed fine? "Our model is very accurate" — accurate on what data, measured how? If they can't tell you the size of their eval set or how they add cases to it, they're guessing. And guessing at production AI means you're the QA team.

Question 4: "What happens on day one after go-live if the model has a bad day?"

AI systems have bad days. A model provider pushes an update, a prompt regression sneaks in, a weird spike in traffic surfaces cases no one anticipated. The question is what happens when that occurs at 2pm on a Tuesday and the system is answering your customers.

Green flag answer: A concrete incident response plan. Something like: "We have a rollback path to the previous prompt version, we can flip a feature flag to route everything to human queue in under a minute, and we have alerts on error rate, latency, and confidence-score distribution." They should mention a kill switch — a way for you, not just them, to disable the system without a support ticket.

Red flag answer: "That won't happen, our system is very reliable." Or "Just let us know and we'll take a look." That's an incident response time measured in hours or days, not seconds. If they don't have a kill switch, you don't have control of your own customer experience.

Question 5: "What data do you retain, for how long, and can I export all of it on demand?"

Your customer conversations, your internal tickets, your lead data — all of it flowing through their system. Ownership and portability of that data determines whether you have a vendor or a landlord.

Green flag answer: "You own the data. We retain conversation logs for [X] days for debugging, but you can export the full dataset in JSON or CSV at any time through the dashboard or an API endpoint. Here's a sample export." Bonus points if they proactively bring up SOC 2 status, data residency, and whether your data is used to train their models (the correct answer is no, unless you explicitly opt in).

Red flag answer: Anything vague. "We handle all that on our side." "Data export is available on enterprise plans." "We can pull that together if you need to leave." Vendors who make it hard to export are locking you in. The moment your bill triples at renewal, you'll wish you'd asked.

Get the export process in writing before you sign. Ideally, run an export during the pilot to confirm it actually works.

Question 6: "What's the real cost per unit at my scale — per conversation, per lead, per resolved ticket?"

Monthly pricing pages are marketing. What you need is the unit economics. If you're deploying a support agent, what does one resolved ticket cost you all-in? If you're deploying a lead-triage agent, what does one qualified lead cost?

Green flag answer: "At your volume — call it 5,000 conversations a month — you're looking at roughly $X per conversation, which breaks down into $Y of model API costs and $Z of platform fee. Here's a spreadsheet." They can talk about how the cost changes as you scale up or down and where the breakpoints are.

Red flag answer: Hand-waving. "Most customers see great ROI." "It's very cost-effective compared to hiring." "We'll work out pricing that makes sense." That means either they don't know their own unit economics — which is terrifying — or they know but don't want you to know because the number looks bad at your scale.

Ask specifically about model API pass-through costs. Some vendors mark those up 3–5x, some pass them through at cost. That difference compounds fast at scale.

Question 7: "Who owns the prompts, the training data, and any fine-tuned model artifacts?"

This is the question no vendor volunteers. If they build a custom system for you, who owns the intellectual property that resulted?

Green flag answer: "You own the prompts we write for you, the data you provide, and any fine-tuned artifacts trained on your data. If you leave, we'll hand over the prompt library. Here's the clause in our MSA." They may reserve rights to their platform code and general methodology, which is fair. But the artifacts built for and on your business should be yours.

Red flag answer: "All prompts and configurations are proprietary to our platform." Translation: when you leave, you start from zero. Every prompt they refined over 18 months of your usage walks out the door with them. That's a huge switching cost, and switching costs are how vendors quietly raise prices at renewal.

The answer to this question changes your negotiating leverage more than any other. If you own the artifacts, you can migrate. If you don't, you're captured.

The pattern behind all seven

Look back over these questions and you'll notice something. None of them are about what the AI can do. They're all about what happens when it doesn't — failure modes, evals, rollbacks, exports, unit costs, ownership.

That's not an accident. In my experience running security and AI work for small businesses, the vendors who last are obsessed with the boring stuff: monitoring, incident response, cost per unit, data hygiene. The vendors who don't last talk about capability and vision. Capability is the easy part now. Anyone can wire up an API and demo something impressive. What separates a working system from a demo is the machinery around it that catches errors, controls costs, and keeps you in the driver's seat.

If a vendor bristles at these questions, that's your answer. Legitimate operators appreciate technical scrutiny because it means you'll be a serious customer who understands what you're buying. Hype vendors want to move fast and get past the questions before you notice the gaps.

One more thing: get a second opinion in the room

Vendor calls are asymmetric. They have a trained salesperson who does this every day; you have a business to run. If you're evaluating a pitch that would involve real money or real customer data, it's worth having someone technical sit in on the call whose job is to ask the uncomfortable questions.

If you'd like a second set of ears on a vendor call — someone who's built these systems and knows where the bodies are buried — get in touch. I sit in on vendor evaluations as a technical second opinion, no long engagement required. An hour of scrutiny before you sign is worth more than a year of buyer's remorse after.