Why your AI pilot never reached production

Usually one of three things, none of which is model quality: nobody agreed a baseline so improvement could not be shown, nobody owned it after the demo, or it never had write access to the system it was supposed to change. All three are decided before a pilot starts.

JarvisBitz

A pilot worked, everyone was impressed, and eighteen months later it is still a pilot. This is common enough that a small industry has grown around explaining it, usually by citing a striking failure statistic and concluding that the problem is data readiness.

A note on those statistics before we go further, because they are load bearing in most articles on this subject and they should not be. The widely quoted figures come from a small number of studies, they are frequently reproduced without dates, their methodology counts are reported inconsistently across outlets, and at least one of the headline numbers has been revised substantially by its own publisher within a year. We tried to verify several at source for this piece and could not open the primaries. So we are not going to quote any of them at you, and you should be mildly suspicious of anyone who does without saying when it was measured and of what.

What we can talk about is what we see when we are called in to a stalled pilot, which is a smaller and much more consistent set of causes than the literature suggests.

Nobody agreed a baseline

The most common single cause, and the most preventable. The pilot ran, it produced output, everyone agreed the output looked good, and there is no way to demonstrate it was better than what happened before, because nobody measured what happened before.

This kills projects at exactly the wrong moment. The pilot succeeded technically and then cannot be defended in a budget conversation, because the honest answer to what did it save is that nobody knows. A finance team is not being obstructive when they decline to fund the rollout of something with no measured benefit. They are doing their job, and the project deserved better preparation.

Nobody owned it after the demo

Pilots are usually run by whoever was interested: an innovation function, a technical lead with slack, an enthusiastic manager. Production systems need an owner in the operational line, with a budget line and a name against them.

The transition between those two is where projects quietly stop. The pilot ends, the person who ran it returns to their actual job, and the system needs someone to answer for it in exactly the way nobody was assigned to. Nothing is cancelled. It just stops having anyone whose problem it is.

The fix is unglamorous and organisational: name the production owner before the pilot starts, and have them agree in advance what result would make them take it. If no one in the operational line will put their name to it, that is worth knowing at the beginning, because it usually means the problem being solved is not one they consider theirs.

It never had write access

The third cause is architectural and specific. Pilots are frequently built read only, because that is faster and safer and gets to a demo. Read only systems produce recommendations, and recommendations require a person to act on them.

So the pilot demonstrates that the AI can identify the right action, and the production version requires integration, permissions, approval design and an audit trail, none of which were in the pilot. The gap between demo and production is not polish, it is a different system, and the estimate everyone is working from was for the first one.

If the value depends on the system doing something rather than suggesting something, build the write path in the pilot, even narrowly, even behind an approval gate. We cover how to do that safely in testing an agent before production and where the gates belong in human approval. A pilot that has written one real record to one real system has retired the risk that actually kills these projects.

The failure mode nobody names

There is a fourth outcome that is worse than cancellation and gets recorded as success: the permanent pilot. It runs, a handful of people use it, it is nobody's system, and it is reported as live because it has users. It consumes a little budget and some goodwill indefinitely, and it occupies the slot a real version would need.

The cause is having no stopping condition in either direction. A pilot that cannot fail also cannot conclude, so it persists at whatever scale it reached on the day the enthusiasm ran out. Agreeing in advance what result means adopt and what result means stop is the cheapest possible protection against it, and it is uncomfortable precisely because it makes failure a legitimate outcome rather than an embarrassment.

The things usually blamed instead

• Data readiness. Genuinely a problem sometimes, and it is a reason a pilot performs badly rather than a reason a successful pilot fails to ship. If the pilot worked, the data was good enough.
• Model quality. Rarely the blocker now. If the demo was convincing, the model was adequate.
• Change resistance. Frequently a symptom of the second cause. People resist systems nobody in their line has taken responsibility for.

What to decide before the next one

• Write down the current cost or time, measured, before anything is built.
• Name the person who will own it in production, and get their criterion for adopting it.
• Include one real write to one real system, however narrow.
• Agree what result ends the pilot, in either direction. A pilot with no stopping condition becomes a permanent pilot, which is the failure mode nobody names.

If you have a pilot that worked and stalled, the diagnosis is usually quick, because it is nearly always one of the three above. Our free AI audit covers which one, and how we run delivery covers how we avoid all three on the next one.

Related reading

AI vendor security questions: what a good answer sounds like

The standard questionnaire will not tell you what you need. SOC 2 covers how a vendor runs their own company, not what the agent they build for you is able to do when it is talked into the wrong action. The questions that separate a serious builder are about permissions, credentials, and what capability they gave up to get there.

Build vs buy an AI agent: the arithmetic that decides it

You cannot answer it until you know how many billable actions one conversation takes, because every pricing model now meters actions rather than conversations. That makes your bill a consequence of how the agent is designed, and it means the same use case can land either side of the line depending on choices you have not made yet.

What you should own when the AI build ends

The code is the least valuable thing you funded. Name the prompts, the eval set, the embeddings and index configuration, the fine-tuned weights if any, and the infrastructure definition, in usable formats. The eval set matters most, because it is the artifact a successor cannot cheaply rebuild.