Check the terms first, then test the work
An AI vendor evaluation checklist has two halves, and a common buying mistake is letting one stand in for the other. The first asks whether you can live with how the vendor handles your data, changes its models, and lets you leave. Answer it with documents, not sales-call answers, and drop any vendor that cannot meet the terms your data requires, however well it performs. The second asks whether the product does your work. Answer it by testing each remaining tool on 30 to 50 of your own examples.
A demo answers neither question, because the vendor chose the inputs. The method works for a chatbot, an AI receptionist, or an AI feature inside software you already pay for. If you have not yet decided to buy at all, build vs buy AI compares buying with configuring a workflow or commissioning custom work.
This is a buyer's working checklist, not legal advice. Contract terms belong with your counsel, and the vendor practices cited here are as of October 2026.
Write down the job before you call vendors
Before the first demo, write one paragraph that names:
- The task, in your staff's words: "draft a first reply to a quote request from the web form."
- The data the tool will see: customer contact details, job notes, pricing, health or financial information, employee records.
- Who is affected if the output is wrong, and who catches it before a customer does.
- Who makes the buying decision, and who owns the tool after launch.
That paragraph sets which checklist rows are must-pass:
| If the tool | Must-pass rows |
|---|---|
| Sees only public or non-sensitive internal content | Retention and training; Export and exit |
| Sees customer, employee, or financial information | The rows above, plus Security assurance, Data processing, Subprocessors, Incident notification, and Access control |
| Talks to customers, publishes under your name, or runs a step the business depends on | The rows for its data, plus Model changes, Uptime and support, and Output terms |
This split is an editorial starting point, not a standard; regulated data, such as health or financial records, can add requirements of its own. If staff already use unapproved tools for this task, shadow AI is often the clearest evidence of which task is worth buying for.
The AI vendor evaluation checklist
Send every vendor the same questions in writing and collect the answers in one sheet. The evidence column matters most: an answer no document backs is a promise, not a term. Screen on the must-pass rows before any bake-off.
| Area | Question to ask | A good answer sounds like | Evidence to request |
|---|---|---|---|
| Security assurance | Has an independent auditor examined your controls, including the AI feature we would use? | Names the report and its period; the product is in scope | SOC 2 report or SOC 3 summary, plus a bridge letter if the period ended months ago; failing that, an equivalent written answer |
| Data processing | What do you do with our data, and on whose instructions? | Uses it only to provide the service, under a signed agreement | Data processing agreement (DPA) |
| Subprocessors | Which other companies receive our data, including the model provider? | A current list with each company's purpose and region | Subprocessor list; change-notice process |
| Retention and training | How long are prompts, files, and outputs kept, and do they train any model? | Specific retention periods; training off for business accounts, in writing | Retention schedule; the training clause in the terms, not the FAQ |
| Model changes | Which model runs the feature, and how much notice do we get before it changes? | Names the model and a notice period; re-tests before switching | Deprecation policy; changelog |
| Uptime and support | What availability do you commit to, and what happens if you miss it? | Stated uptime, support hours, and service credits | Service level agreement (SLA); status page history |
| Incident notification | How fast will you tell us about a security incident or serious AI failure that affects us? | A notice window in the contract and a named contact | Incident clause in the DPA or terms |
| Access control | Can we control who uses it, see what they did, and remove access quickly? | Single sign-on or managed accounts, roles, and an exportable activity log | Admin documentation; a sample log |
| Export and exit | If we leave, what do we get back, in what format, and when is our data deleted? | Full export in a standard format; confirmed deletion | Exit and deletion terms; a test export |
| Output terms | Who owns the outputs, and do you defend us if one infringes someone's rights? | Clear ownership; any indemnity and its conditions stated plainly | Output and indemnity clauses, for counsel |
| Claims | What test supports your accuracy or time-saved claims? | Describes the test set, the grading, and its limits | The evaluation method, not just the headline number |
What the documents prove, and what they don't
A SOC report is an independent auditor's examination of a service provider's controls. The AICPA describes SOC 3 reports (opens in a new tab) as covering the same control areas as SOC 2 (security, availability, processing integrity, confidentiality, and privacy) in less detail, which is why a SOC 3 can be freely distributed. Check the report's scope: it may not include an AI feature added after the audit period.
Subprocessor lists matter more for AI tools, because the model often belongs to someone else. Microsoft's Copilot privacy documentation (opens in a new tab), for example, names Anthropic and OpenAI as subprocessors for some experiences and notes that Anthropic models are currently excluded from its EU Data Boundary. Ask for that level of detail.
AI accuracy and savings claims are claims like any other. In September 2024 the FTC announced Operation AI Comply (opens in a new tab), a crackdown on deceptive AI claims and schemes, and then-Chair Lina M. Khan said there is "no AI exemption from the laws on the books." When a vendor quotes an accuracy rate or hours saved, ask what was tested, on what data, and who graded it. A testimonial is not evidence about your work.
Model changes are a contract question
Many AI products run on another company's model, and model providers retire models on a schedule. As of October 2026, OpenAI's deprecation policy (opens in a new tab) states at least six months' notice for generally available models, at least three months for specialized variants, and much shorter notice, such as two weeks, for previews, and says safety or compliance concerns can shorten these timelines. Anthropic's deprecation page (opens in a new tab) commits to at least 60 days' notice before retiring publicly released models and advises testing replacements well before the retirement date.
A vendor built on those models inherits the schedule. Ask how it passes notice on, whether it re-tests before switching, and whether you can stay on the current model while you re-test your examples. A tool that answered well in March can answer differently in September with nothing changed on your side.
NIST's Generative AI Profile (AI 600-1, July 2024) (opens in a new tab) points the same way. Its suggested action GV-6.2-007 covers contracts that consider system changes over time, serious-incident notification, and service level agreements; GV-6.1-007 covers an inventory of third parties with access to your content and an approved provider list. The profile is voluntary guidance, not a certification.
Run a bake-off on 30 to 50 of your own examples
Before you sign, run every vendor that passed the document screen through the same set of your own examples.
Collect the examples. Pull 30 to 50 real inputs from recent months: emails, intake forms, call notes, quote requests. Include awkward ones: missing details, an upset customer, a request outside your services, a question the tool should decline. Remove or replace personal details before anything leaves your systems.
Write the grading rule first. For each example, note what an acceptable output must contain and what makes it a fail: a wrong price, an invented policy, a missed request, a tone you would not send. Pass, needs edits, or fail is enough.
Run every vendor on the same inputs. Use the same instructions and settings, record the model and configuration, and save every output.
Grade without vendor names. The person who reviews this work day to day grades a shuffled set with the names removed.
Run a sample twice. Rerun a handful of examples a day later. A tool that passes once and fails the second time has not passed.
Time the fixes. Time how long each needs-edits output takes to correct. Fast drafts that need long corrections may save nothing; the worksheet in how to measure AI ROI counts that time against a baseline.
Anthropic's guidance on building evaluations (opens in a new tab) recommends test cases that mirror your real task distribution, including edge cases. OpenAI's evaluation best practices (opens in a new tab) notes that the same input can produce different outputs and calls test sets that do not reflect real traffic an anti-pattern.
Score each example on a sheet like this:
| Example | Type | Tool A | Tool B | Tool C | What went wrong |
|---|---|---|---|---|---|
| 1 | Ordinary quote request | Pass, edit, or fail | |||
| 2 | Missing address | ||||
| 3 | Service you don't offer | ||||
| 4 | Upset customer | ||||
| Totals | Passes, edits, fails | ||||
| Rerun | Same result on the second run? | ||||
| Fix time | Typical minutes per edit |
Thirty to fifty examples is a screen, not a benchmark, so do not read the totals as an error rate. The sheet shows how each tool fails on your work, and whether the failures are cheap (a clumsy sentence) or expensive (a wrong price sent to a customer).
Contract questions to take to counsel
These are questions for your lawyer, not legal conclusions; the answers depend on your contract, industry, and states.
- Can the vendor change its terms or data practices mid-contract, and how will you be told? FTC staff wrote in February 2024 (opens in a new tab) that quietly adopting more permissive practices for consumers' data, such as AI training, through a surreptitious, retroactive change to terms may be unfair or deceptive. That post concerns consumers; as a business customer, rely on your contract.
- Who owns outputs, and does any indemnity for third-party claims come with conditions?
- What is the liability cap, and does it cover an incident at a subprocessor?
- What are the renewal, termination, and data return terms, and do they survive an acquisition?
- If you handle regulated data such as health, financial, student, or children's information, will the vendor sign the agreement your obligations require?
Make the decision and write it down
A gap in a must-pass row rules a vendor out, whatever the bake-off shows. Among the rest, choose on the bake-off: how many examples passed, how costly the failures were, and how long the fixes took. Any other gap is a risk you are choosing to accept, so have someone with authority accept it in writing.
Record the decision, the evidence, the open gaps, and what triggers a recheck (a model change, a contract renewal, or a change in the task) as a row in your AI risk register. Treat the first weeks after purchase as a pilot; why AI pilots stall covers the pass and stop conditions to set before launch. For phone tools, the AI receptionist guide adds call-specific questions.
If you want a second opinion on a shortlist, Cleland & Co.'s AI Strategic Advisor engagement includes vendor evaluation: the shortlist, the questions to ask, and the failure modes to test before signing.
