# When to Remove an AI Chatbot: A Decision Procedure > Source: > Published: 2026-08-12 17:05:34+00:00 Most chatbot removals are argued from irritation rather than from measurement, which is why they take six months and get reversed. There are four numbers that settle it, and the honest version of the decision usually turns out to be about the *interface* rather than about the model behind it. “Should we remove the chatbot?” conflates two independent decisions: whether the underlying capability is useful, and whether an open-ended conversational box is the right way to expose it. They have different answers surprisingly often. A model that answers product questions accurately can be worth keeping while the free-text input that fronts it is the thing destroying the experience. So the decision splits in two, and only the second half is about removal: If the answer to the first is no, remove it. If the answer to the first is yes and the second is no, you are replacing an interface, which is a much smaller and much more reversible project. The general argument against defaulting to chat is in [why chat is often the wrong interface](https://multigrid.ai/learn/chat-antipattern). | Measurement | Description | |---|---| | Containment rate | Share of sessions that ended without the user escalating to a human, a search, or abandonment — and where the user did not return with the same intent within 48 hours. The return window is what makes this honest; without it, a user who gave up counts as contained. | | Escalation quality | For escalated sessions, how much of the human's time the bot saved or wasted. A transcript that gives the agent the account, the intent and the failed attempt saves time. One that makes the agent read six turns of small talk costs time, and this is usually negative. | | Second-session rate | Of users who used it once, how many used it again within 30 days. Novelty produces a strong first number and a collapsing second one, so first-use adoption is the metric most likely to justify keeping something that is not working. | | Support burden created | Tickets caused by the bot: wrong answers to correct, promises it made that the product does not keep, and confusion about whether the user was talking to a person. This is a cost line, not an incident count. | The fourth is the one teams do not have instrumented, and it is frequently large enough to reverse the decision on its own. Categorise it before you argue about it — [the support load an AI feature creates](https://multigrid.ai/learn/ai-support-load) sets out the categories. There is no universal containment threshold, and anyone quoting one is quoting a vendor. What can be stated generally is the *form* of the threshold, which is a comparison rather than an absolute. That third comparison is the one that most often decides it, and it is cheap to run: cluster a month of successful transcripts by intent and look at the head. The third comparison above — whether a handful of intents accounts for most successful sessions — is the one that usually decides the question, and it is a half-day of work rather than a project. Two cautions. Run it on successful and failed sessions separately, or the two distributions average into a shape that describes neither. And be aware that the distribution is partly an artefact of the interface: an open box attracts the questions people think it can answer, so a flat distribution may reflect users exploring rather than users needing. Removing a chatbot without a replacement returns users to whatever they were doing before, which was usually worse. Three replacement patterns keep the capability and drop the open-ended box. | Pattern | Description | |---|---| | Intent buttons over the same model | The head intents become named actions. Each runs a fixed prompt with a fixed schema, so the output can be validated and rendered as structure rather than as prose. Failure modes collapse from unbounded to a handful. | | Inline assistance | The capability moves to where the task is — a summarise action on the record, a draft action in the reply box. Discovery improves because the feature is next to the work, and abandonment falls because there is no blank prompt to compose. | | Better search with generated answers | Keep retrieval, drop the conversation. A ranked list with a short generated summary and citations answers most informational intents and degrades gracefully when retrieval finds nothing — which a chatbot does not. | All three narrow the input, which is the actual change. A constrained input is what makes [structured output](https://multigrid.ai/learn/structured-output) and [output validation](https://multigrid.ai/learn/output-validation-repair) tractable, and it removes most of the [prompt injection](https://multigrid.ai/learn/prompt-injection) surface at the same time. Chat is the correct interface when the input genuinely cannot be enumerated and the user genuinely knows what they want to say. That is rarer than product roadmaps assume, but it is real: The measurements above still apply in all three cases. “The interface is appropriate” is not the same claim as “the feature is earning its cost”, and [what an unused feature actually costs](https://multigrid.ai/learn/feature-nobody-used) is the second half of the decision.