{"slug": "5-things-to-agree-before-any-ai-pilot", "title": "5 things to agree before any AI pilot", "summary": "A developer published a five-point pre-pilot checklist for AI adoption, arguing that most pilots fail because success criteria are never defined before work begins. To illustrate the approach, the developer built a demo ticket-triage system on the public Banking77 dataset, fixing a 97% accuracy target on automatically handled tickets before selecting a confidence threshold on a validation split; on the held-out test set, 75.9% of tickets were handled automatically at 97.7% accuracy, versus 90.2% accuracy if everything had been automated. The developer stresses the result is a public benchmark demo, not a client outcome, and that the automation level fell out of the pre-agreed accuracy target with a human fallback for uncertain cases.", "body_md": "Most AI pilots fail before they start. Not because the model is weak, but because nobody wrote down what \"success\" means. Three months later one person says \"it works pretty well\", another says \"it makes mistakes\", and there is no number to settle the argument.\n\nHere are five things to agree on in one meeting, before any code is written. If you can't answer all five, the pilot isn't ready.\n\nPick a single process, not \"customer experience\". For example: incoming support requests, invoices from suppliers, or questions about internal documents. Then name one person who owns it and can say \"yes, this result is acceptable\".\n\nWithout an owner, nobody can accept the pilot. Without a single process, nobody can measure it.\n\nHow long does the work take today? How often does it go wrong? Take a real sample, even 100 items, and measure by hand.\n\nThis is the step teams skip most often. If you don't know that a person spends 6 minutes per invoice with a 3% error rate, you can't say whether the pilot improved anything. You can only say it \"feels\" faster.\n\nWrite the target as a number before you see any results. Examples:\n\nTwo numbers usually work better than one: how much is automated, and how accurate that automated part is. They pull against each other, and the pilot is really about choosing the point between them.\n\nWhich systems, which fields, which time period? Is personal data involved? Can a third-party model API see it, or does it have to stay inside your infrastructure?\n\nSettle this early. It is the most common reason a \"successful\" pilot dies in legal review a month later.\n\nPut the decision date in the calendar and name the three outcomes in advance: scale it, fix specific problems and re-test, or stop. \"Stop\" has to be an allowed answer. A pilot that cannot be stopped is just a project with a smaller budget.\n\nI built a small demo to show the shape of an agreed target. It routes customer-support messages (Banking77, a public dataset, 77 intents). The acceptance target was fixed first: at least 97% accuracy on automatically handled tickets. The confidence threshold was then chosen on a validation split to meet that target, and the test set was used once for reporting.\n\nOn the held-out test set: 75.9% of tickets were handled automatically at 97.7% accuracy, and the rest went to a human. If everything had been automated, accuracy would have been 90.2%.\n\nTo be clear: this is a public benchmark and a demo on test data, not a client result. The point is the structure. The automation level was not guessed, it fell out of an accuracy target that was agreed first, and there is a human fallback for uncertain cases. Raise the target and you get fewer errors and less automation; that is the trade-off the owner in point 1 has to decide on.\n\nCode and the numbers are in the repo: [https://github.com/TerrKot/WorkSearchHepler](https://github.com/TerrKot/WorkSearchHepler) (folder `demos/ticket-triage`).\n\nPut the five questions on one slide and ask the stakeholders to answer them in the meeting. If the answers take a week of emails, you have found the real first task of the pilot, and it was cheap to find out.\n\nI write short notes like this about practical, measurable AI adoption in business, with demos on test data and no invented case studies:", "url": "https://wpnews.pro/news/5-things-to-agree-before-any-ai-pilot", "canonical_source": "https://dev.to/slava_kotenev_159/5-things-to-agree-before-any-ai-pilot-4mem", "published_at": "2026-10-25 10:41:06+00:00", "updated_at": "2026-10-25 10:43:24.730585+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools"], "entities": ["Banking77", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/5-things-to-agree-before-any-ai-pilot", "markdown": "https://wpnews.pro/news/5-things-to-agree-before-any-ai-pilot.md", "text": "https://wpnews.pro/news/5-things-to-agree-before-any-ai-pilot.txt", "jsonld": "https://wpnews.pro/news/5-things-to-agree-before-any-ai-pilot.jsonld"}}