The demo works beautifully. The steering committee is impressed. Someone says "let's roll this out next quarter." And then — nothing happens for six months.
I've watched this play out enough times to recognise the pattern. AI pilots rarely fail because the model isn't good enough. They stall because the organisation wasn't ready to absorb what the pilot proved. Here's what I now insist on before anyone writes a prompt.
1. A metric with an owner
"Improve customer experience" is not a metric. "Reduce handling time on order-status cases by 30%, owned by the Head of Service" is. If no business leader is willing to put their name next to the number, the pilot is a science project — and science projects get defunded.
2. A definition of "done" that the operations team wrote
Technologists define success as "the agent answered correctly." Operations teams define it as "the case is closed and the customer didn't come back." Those are different things. I ask the people who run the process today to write the acceptance criteria — and they always add cases the technical team missed.
3. The data conversation, in week one
Every AI initiative is a data initiative in disguise. Before building, I want answers to three questions:
- Where does the data the agent needs actually live — and who owns it?
- How good is it? (Not how good the data team thinks it is.)
- What is the agent allowed to see, and what must be masked?
4. Guardrails designed, not bolted on
Trust is the real product. Leadership needs to know what the AI will never do, as much as what it will do. That means explicit scope (topics it handles and those it refuses), clear escalation rules, PII masking, audit trails and a human in the loop for anything high-risk. Designing these up front turns the security review from a blocker into a formality.
5. A test library, not a demo script
A demo has five happy-path conversations. Production has thousands of messy ones. I build a library of real (anonymised) conversations — including the awkward, ambiguous and adversarial — and run every change against it. That library becomes the most valuable asset of the programme.
6. Change management for the humans
If the people whose work is changing see the AI as a threat, adoption will quietly fail. The most successful rollout I've been part of shipped AI assistance for human agents alongside automation — summaries, classification, suggested responses. Agents saw it make their day better before they saw it take volume away.
The short version
Pilots prove the technology. Production proves the organisation. Plan for the second from day one.
Pranav Sanvatsarkar is a Solution Architect with 14+ years across telecom, healthcare, public sector and non-profit, focused on AI, agentic automation and Salesforce architecture. Say Hello!