It’s past time to end AI-based automated customer responses

Anthropic's automated chatbot dismissed a security vulnerability report from Wiz researchers, claiming the issue fell outside its threat model, even though Anthropic had already patched the flaw. The incident highlights the risks of AI-based customer response systems generating incorrect and confident rejections, prompting calls to limit bots to pre-approved scripts.

An automated chatbot working for Anthropic this month shot down a Wiz researcher’s security hole report, saying that it “falls outside of the Claude Code threat model.” That was news to the security researchers at Wiz https://www.wiz.io . It also turned out to be news to Anthropic execs, who had a very different view. In reality, Anthropic was one of many victims of the hole — including Amazon, Google and Cursor, among others https://www.csoonline.com/article/4195235/ai-coding-tool-hole-illustrates-a-big-problem-with-human-in-the-loop.html . But what makes the incident so bizarre is that, far from dismissing the threat, Anthropic had detected it before the security researchers and had even patched it before the researchers alerted them. As these AI bots are wont to do, the bot didn’t merely reject the request. It confidently explained its rationale, even though its reasoning was wrong. “This falls outside our current threat model,” the chatbot said, according to a report by Wiz https://www.wiz.io/blog/ghostapproval-a-trust-boundary-gap-in-ai-coding-assistants . “When the user first starts Claude Code in a directory, they must confirm that they trust the directory prior to starting the session. The scenario you describe involves a user explicitly confirming a permission prompt inside of a directory containing a malicious symlink, which falls outside of the Claude Code threat model.” That researchers said Anthropic management later clarified the situation: “The symlink warning in the Edit/Write permission dialog shipped in v2.1.32 Feb 5, 2026 , nine days before this report was submitted to us. It was added as part of proactive security hardening based on internal review. The decline to comment was an autoreply from our triage system.” An autoreply from our triage system? How many other make-believe replies did this system send? And what level of damage is Anthropic exposing itself to? This is not just an Anthropic issue. There have been numerous enterprise bot glitches in communications with customers. Some of my favorites include: Let’s be clear, here: Bots should be limited to relaying only pre-approved scripts. Generative AI allows for far greater chatbot sophistication, but that also means the chance of far greater errors. This is untenable in any business function. And when the app is pretending to be a human — and interacting with human customers — it’s even more unacceptable.