The demo went well. The model flagged the right batches on last year’s data, the slide showed a nice accuracy number, and everyone agreed it was “promising”. Six months later nobody on the line opens it. No one decided to kill it. It just never made it to the shop floor.
After 24 years in manufacturing and an M.Sc. in industrial engineering, I am convinced that the model is rarely the problem. The gap sits between the notebook and the daily routine of the people who should use the output. Gartner said it plainly in 2024: it predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, because of poor data quality, weak risk controls, rising costs or unclear business value. None of those reasons is about the algorithm.
In this article I go through the seven places where pilots usually get stuck, an illustrative worked example that shows how value leaks away, a production readiness scorecard you can fill in this week, and a short checklist.
Seven gaps between a pilot and a tool people use
- Data plumbing. In the pilot, someone exported a CSV by hand. In production, the data has to arrive on its own, on time, every shift. If the decision is made at the machine within minutes, a daily export is useless.
- Ownership. The pilot belongs to a data person or a vendor. When they move on, nobody in operations owns the result, the thresholds or the budget.
- Process change. The model gives a signal, but the work instruction never changed. Nobody wrote down who reacts, how fast, and what “react” means.
- Operator trust. If the first week brings a few false alarms with no explanation, people stop looking. Trust is earned on the line, not in the steering committee.
- The wrong success metric. Accuracy or F1 looks good in a report, but the plant manager cares about scrap, downtime, hours or cash. If you did not measure a baseline before the pilot, you cannot prove anything after it.
- Maintenance (MLOps). Materials, suppliers, recipes and machines change. A model that nobody monitors slowly gets worse. Researchers at Google described this years ago as hidden technical debt in machine learning systems: the quick win is never free.
- Integration with MES and ERP. The output lives in a separate dashboard or browser tab. Operators work in MES, planners work in ERP. If the alert does not show up where the work happens, it does not exist.
If you are still choosing a first use case, start with my article on applied AI use cases that pay in manufacturing. For a low-risk first pilot, see the AI document assistant (RAG) pilot or predictive maintenance on one critical asset. Two of the gaps above are really data-discipline problems, which I covered in master data ownership before ERP–MES integration and the KPI dictionary.
Worked example: where the value leaks (illustrative)
The numbers below are an illustrative example, not data from a real plant. They show why a good model can deliver almost nothing.
A line produces 40 batches a day, and about 5% fail final inspection, so roughly 2 bad batches a day. Catching a bad batch early saves about 1.5 hours of rework. In the back-test, the model catches 70% of bad batches. On the slide, that is 1.4 batches a day and about 2.1 hours saved every day.
On the floor, the real catch rate is the model multiplied by everything around it:
| Link in the chain (illustrative) | Pilot as deployed | Production-ready |
|---|---|---|
| Model catches bad batches (back-test) | 70% | 70% |
| Data arrives before the batch is finished | 30% (manual export) | 95% (automatic feed from MES) |
| Alert is seen where the operator works | 50% (separate dashboard) | 90% (on the MES screen) |
| Operator acts on the alert | 60% (no instruction, low trust) | 85% (reaction plan, feedback button) |
| Real share of bad batches caught | 70% × 30% × 50% × 60% ≈ 6% | 70% × 95% × 90% × 85% ≈ 51% |
| Rework hours saved per day (2 bad batches × 1.5 h × share) | ≈ 0.2 h | ≈ 1.5 h |
Same model, same data scientist, same accuracy. In the first column the pilot saves about 0.2 hours a day, and someone in a meeting will rightly say “AI doesn’t work here”. In the second column it saves about 1.5 hours a day, roughly eight times more, and the only things that changed were plumbing, placement, instructions and trust. That is where the effort of going from pilot to production should go.
A production readiness scorecard
Before you scale a pilot, or even before you start one, score each line 0, 1 or 2. It takes about an hour with the line owner, IT and the person who built the model in the same room.
| Dimension | What “2” looks like |
|---|---|
| Business owner | A named person in operations owns the result, the thresholds and the budget. |
| Success metric | One business KPI (scrap, downtime, hours, cash) with a baseline measured before the pilot. |
| Data pipeline | Data arrives automatically, with a delay that fits the decision; no manual exports. |
| Integration | The output appears in MES, ERP or the screen people already use. |
| Process change | The work instruction says who reacts, how fast and how; operators were trained. |
| Operator trust | Alerts come with a simple reason; operators can mark “useful / false alarm”; the false-alarm rate was agreed with them. |
| Monitoring and maintenance | Someone checks performance monthly, knows when to retrain, and every model version is recorded. |
| Fallback | If the model or feed is down, the line knows what to do, and nobody stops production waiting for it. |
From pilot to production, step by step
- Write down the decision. One sentence: which decision the model supports, who makes it, and how quickly.
- Measure the baseline first. Agree the business KPI and measure it for a few weeks before anything goes live.
- Replace the manual export. Connect the data source properly, even if it is only one machine or one table at first.
- Put the output where the work happens. An MES screen, an ERP field, a message to the shift leader. Not a new portal.
- Change the work instruction with the operators. Let them help write the reaction plan. People trust what they helped build.
- Run in shadow mode for two to four weeks. The model makes predictions, people work as usual, and you compare. Fix false alarms before they meet the line.
- Name the maintenance owner. Monitoring, retraining triggers and model versions get a name and a monthly slot.
- Decide: scale, fix or stop. Use the scorecard and the KPI against the baseline. Stopping a weak pilot early is a good result, not a failure.
The payoff
When the gaps are closed, an AI pilot stops being a demo and becomes part of how the line runs. You get a number the plant manager believes, operators who use the alerts because they helped shape them, and a model that someone looks after. You also get a repeatable path, so the second use case takes weeks instead of months.
Checklist before you scale an AI pilot
- The decision, the decision-maker and the reaction time are written down.
- One business KPI has a baseline measured before go-live.
- Data flows automatically, fast enough for the decision.
- The output appears in MES, ERP or the screen operators already use.
- The work instruction and reaction plan were updated together with operators.
- A shadow run compared predictions with reality, and false alarms were reduced.
- Monitoring, retraining and fallback each have a named owner.
- The scorecard is at 12 or above, with no zero on owner, metric or data.
Have a pilot that never reached the line?
If you have a model that looked good in the demo but nobody uses on the shop floor, send me a short message. We can go through the scorecard together, find the weakest link and agree a realistic plan to bring it into daily work, or decide honestly to stop it.
Contact me Read: 5 AI use cases that pay Read: predictive maintenance, one asset
Sources
- Gartner (July 29, 2024). Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025.
- Sculley, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS 2015.