Why AI Pilots Succeed and AI Deployments Fail
Reading note
Essays for people who want the pattern behind the pattern.
This page is designed to read like a quiet, deliberate argument rather than a feed item.
There’s a pattern I keep seeing: an AI pilot gets built in 6 weeks, impresses stakeholders, and then takes 6 months to deploy — if it deploys at all.
The pilot succeeds because it’s built in a controlled environment with clean data, a single use case, and no compliance requirements. The deployment fails because it hits reality: messy data, multiple user types, security reviews, integration with existing systems, and the question nobody asked during the pilot — “who maintains this after launch?”
The failure isn’t technical. It’s architectural. Specifically:
Pilots optimize for capability. Deployments require operability. “Can the model do the thing?” is a different question from “can the team operate the thing at scale?” Most pilots only answer the first question.
Identity and access matter more than model selection. I’ve watched teams spend weeks evaluating GPT-4 vs. Claude vs. Gemini and then deploy without thinking about who has access to the data the model can see. The model choice is usually the least consequential architectural decision.
Cost modeling happens too late. The pilot runs on a corporate card. The deployment needs a budget line. And the per-token cost at production volume is always higher than the pilot suggested, because the pilot didn’t account for retries, logging, redundancy, or the fact that users ask the same question six different ways.
The fix isn’t better pilots. It’s building the production concerns into the pilot from day one — even if it slows the demo timeline. An AI system that can’t pass a security review isn’t a system. It’s a prototype with ambitions.