Most AI pilots impress in a demo and fail in production. The demo runs on ten hand-picked examples with someone watching. Production runs on ten thousand messy ones with nobody watching. The gap between those two is where budgets disappear.
Before we hand anything over, it passes three checks. A pilot that fails any of them stays a pilot, and we say so.
One: it has an evaluation suite, not a vibe
An evaluation suite is a fixed set of real historical cases with known correct answers, run automatically on every change. For the Nordwind claims workbench that was 1,200 historical claims. Without it you cannot tell whether a prompt change helped or quietly broke a category of edge case.
If a team cannot tell you their accuracy number and how it was measured, they do not have one.
Two: it knows what it does not know
Every output carries a confidence signal, and anything below the threshold routes to a person with the context attached. The threshold is a business decision, not an engineering one: how often are you willing to be wrong, and what does being wrong cost?
Systems that answer everything with equal confidence are the ones that erode trust fastest. One bad answer with no hedge undoes fifty good ones.
Three: someone owns it on Monday
Handover is not a zip file. It is documentation, a runbook, an on-call owner, and a person on the client side who has changed a prompt, run the evaluation and deployed it while we watched.
We build the 30-day support window in for exactly this. If nobody on your side has touched it by day 30, it is not adopted, and we would rather fix that than close the invoice.
Scopes the audit week, writes the plans and owns handover. Product lead on the Nordwind claims workbench and the Coastline order run.