The demo is a different test
A pilot demo is a curated event. The inputs are representative, the reviewer is sympathetic, and the failure cases are either absent or explained away as edge cases to be handled later. None of that is dishonest — it is what a demo is for. The problem is that the demo's success is then treated as evidence about production, and it is not evidence about production at all.
Production asks four questions the demo did not. How often is it wrong, and how wrong. What happens downstream when it is. What it costs at real volume rather than at demo volume. And who is responsible when the answer to any of those turns out to be unacceptable. A pilot that never defined acceptable thresholds for those four cannot pass or fail — it can only stall, which is what it does.
Three specific failure patterns
Success was never falsifiable
The most common one by a wide margin. The pilot's goal was 'improve efficiency in support' or 'explore AI for document review' — statements that no result could contradict. When there is no criterion that could turn out false, there is no moment at which the pilot is done, so it neither ships nor gets killed. It just gradually stops being discussed.
The fix is unglamorous and takes an afternoon: write down, before building, the specific number and the threshold. Not 'reduce handling time' but 'median handling time below four minutes on the top three ticket categories, with resolution quality no worse than the current baseline as scored by the existing QA rubric.' That is a statement that can lose.
Nobody owned the transition
The pilot had an owner — usually the person who championed it. Production has a different owner, or several, or none. The handoff between them is where the work dies, because the receiving team inherits a system with no evaluation harness, no runbook, no monitoring, and no budget line, and reasonably declines to take responsibility for it.
The unit economics were never checked at scale
A pilot runs on a few thousand requests. Production runs on a few million. Costs that were invisible at pilot scale — tokens, retries, embedding refreshes, human review of low-confidence outputs — become the dominant line item, and someone discovers late that the business case inverts at real volume. This is arithmetic that can be done on day one and usually is not.
None of these three are technology problems. All of them are planning problems, and all of them are cheaper to solve before the build than after it.
What to do instead
Run the pilot as a rehearsal rather than a demonstration. The difference is the intent: a demonstration is designed to succeed, a rehearsal is designed to find out where the plan breaks while breaking it is still cheap. Practically, that means four things before anyone builds.
- State success as a number with a threshold, and state the failure condition that would cause you to stop.
- Name the production owner before the pilot starts, and have them define what they need in order to accept the handoff.
- Do the volume arithmetic on day one, including retries, review, and the cost of being wrong.
- Reserve time to attack the system deliberately — bad inputs, missing dependencies, and the ways busy people will actually misuse it.
The fourth is the one that gets cut, and it is the one that separates a pilot that ships from one that stalls. A system nobody has tried to break is a system whose failure modes will be discovered by a customer.