# When Your Chatbot Is So Eager to Help, It Forgot Whose Side It’s On > Source: > Published: 2026-07-31 23:15:23+00:00 12 min read *It started with a bank telling customers how to avoid its own fees. It ends with a Chevy bot recommending a Ford.* **TL;DR: ** - Companies are putting AI into the customer-facing layer of their product to be maximally helpful. - Sometimes AI is so good at its job that it points customers toward the cheaper, faster, or fee-free path, one that skips the workflow the company built its revenue on. This is Agentic AI Business Risk. - It’s already showing up in retail, travel, banking, and auto sales. The businesses handling it well decided, on purpose, which leaks they can live with. ## Table of Contents **The Unspoken Launch Variable** A support chatbot has one job description on paper: to resolve your customer’s problem. Nobody writes “protect the commission structure” or “preserve the overdraft fee” into that job description, because it would sound bad in a press release. But those are the things that most businesses need the interaction to protect, and even a genuinely capable AI doesn’t know that unless someone tells it to care. That’s the setup. A large language model, unlike a scripted bot, will reach for whatever answer best satisfies the question asked. If a customer asks a bank’s assistant how to avoid a fee, or asks a retailer’s assistant to find the cheapest version of a product, an AI trained to be helpful will often find it, even if the honest answer routes the customer away from the thing that pays for the assistant in the first place. It’s the same underlying behavior we found when we [tested how language models describe real brands](https://industrycontents.com/language-models-flatten-brand-positioning/): the model optimizes for the best answer to the question in front of it, not for whoever is paying to host the conversation. Bain & Company has a name for the far end of this: agents could “*entirely disintermediate multibrand retailers and turn direct-to-consumer brands into indirect ones*,” reducing some retailers to little more than commoditized fulfillment. This here is the boardroom-level version of the problem. Down at the level of a single chat window, it looks smaller and stranger, and it’s happening. ## Four Real Cases ### Amazon’s Rufus Sends Shoppers Off Amazon, on Purpose Amazon’s in-app assistant, Rufus, is built to keep people inside Amazon’s checkout. But Amazon also built a feature called **Buy for Me** directly into it. If a shopper asks for something Amazon doesn’t sell, Rufus will find it on a brand’s own website and complete the purchase there, using the customer’s Amazon payment details, without the shopper ever leaving the app. It’s part of a broader shift toward [agentic checkout](https://industrycontents.com/agentic-checkout-adoption-gap/), where the AI doesn’t just recommend a product, it completes the transaction. The numbers around Rufus are pretty huge. [Amazon says](https://www.aboutamazon.com/news/retail/amazon-shopping-app-buy-for-me-brands) shoppers who use Rufus are more than 60% more likely to finish a purchase during that session, and the assistant is now credited with [close to $12 billion](https://www.ecomcrew.com/amazons-rufus-can-now-buy-things-for-you-heres-what-that-means-for-sellers/) in incremental annualized sales. And some of that money is flowing to brands that don’t sell on Amazon’s own marketplace at all. It’s a strange choice for a company that built its empire on owning the transaction, until you consider the alternative: shoppers increasingly start their search in ChatGPT or Gemini instead of Amazon’s search bar, and a version of Amazon that says “we don’t have that” loses the shopper entirely. Amazon appears to have decided that losing a slice of margin on an off-platform sale beats losing the shopper’s attention altogether. That calculus also shows up in how these systems weigh signals that used to matter most to shoppers; we found something similar when we looked at [why AI shopping agents barely react to star ratings](https://industrycontents.com/ai-shopping-agents-ignore-star-ratings/) the way a human buyer would. However, merchants have complained to [Modern Retail](https://www.modernretail.co/technology/brands-are-upset-that-buy-for-me-is-featuring-their-products-on-amazon-without-permission/) that Buy for Me listed their products without asking first, forcing them to opt out after the fact rather than opt in. Amazon designed its tool in order to keep shoppers engaged, and at the same time, training its own AI to treat the rest of the internet as its inventory. ### Hotels Are Trying to Use AI to Bypass the Middleman, and So Is the Middleman Online travel agencies built a business on being the place travelers search first, in exchange for a commission that industry sources put at [15% to 25% per booking](https://tragento.com/en/and-search-as-a-new-chance-for-hotels-to-bypass-expensive-OTA-commissions/). Hotels have wanted out of that arrangement for two decades and never had the leverage to make it stick. Conversational AI just joined the chat. [RateGain’s UNO Booking Engine](https://rategain.com/blog/what-is-a-direct-booking/) was among the first to hook a hotel’s live rates directly into AI assistants through the Model Context Protocol, letting a traveler book a room inside the conversation itself, no OTA in the loop. Startups built entirely around this idea, pushing hotel rates straight into ChatGPT and Claude conversations, have already signed several of the world’s largest hotel groups, including Radisson and BWH. The irony is that the same AI assistants could just as easily entrench the OTAs instead of killing them off. [Mews](https://www.mews.com/en/blog/openai-operator-hotel-bookings), a hospitality platform, points out that Booking.com and Expedia already have a stronger, more AI-legible web presence than most independent hotels, so an assistant like ChatGPT’s Operator might default to booking through the OTA anyway, out of pure convenience, deepening the commission dependency instead of ending it. Still nobody knows yet which way this settles. ### When Helpful AI Becomes Bad for Bank Revenue Probably the most quantified case of the four, and it starts with predictive alerts than a chatbot. It started with predictive alerts. Overdraft and non-sufficient-funds fees cost American customers [$6.7 billion in 2024 alone](https://www.meniga.com/resources/ai-in-overdraft-protection/), and for a long time, that fee income was a real line on a bank’s income statement. Then banks started building AI-driven low-balance warnings meant purely as a customer-trust feature: Wells Fargo’s Predictive Banking, Ally’s low-balance insights, Huntington’s Heads Up. Every one of them exists to tell a customer, correctly, how to avoid the fee. Bank of America’s has [disclosed](https://www.stocktitan.net/news/BAC/bank-of-america-announces-sweeping-changes-to-overdraft-services-in-zszb8n0ipltv.html) that overdraft fee revenue fell 97% from 2009 levels, a decline it attributes directly to tools like Balance Assist and its predictive alerts, alongside product changes like dropping NSF fees entirely and cutting the overdraft fee from $35 to $10. The UK didn’t wait for banks to volunteer it: the [Financial Conduct Authority](https://thefinancialbrand.com/news/artificial-intelligence-banking/artificial-intelligence-digital-banking-overdraft-fees-93784) forced fixed overdraft fees out of the market entirely in 2020, calling the market “dysfunctional.” American banks got to choose the soft landing. They may not get that choice twice. ### The Chevy Chatbot That Allegedly Recommended a Ford In December 2023, a Chevrolet dealership in Watsonville, California put a ChatGPT-powered sales chatbot on its website, built by a vendor called Fullpath. Within days, screenshots spread showing the bot answering unrelated coding questions and, in one widely shared thread, [seemingly recommending](https://www.theautopian.com/chevy-dealers-ai-chatbot-allegedly-recommended-fords-gave-free-access-to-chatgpt/) a Ford F-150 as a capable truck to a visitor on a Chevy dealer’s own site. The Autopian, which broke the story, was careful to flag that the screenshots couldn’t be independently confirmed before the dealership locked the bot down, and that caveat matters. This is the weakest-sourced case in this piece, and it stays in for exactly that reason: it shows what happens at the edge of what current evidence can support. The same chatbot, on the same site, that same week, was also talked into agreeing to [sell a 2024 Chevy Tahoe for $1](https://cut-the-saas.com/ai/chatbot-case-study-purchasing-a-chevrolet-tahoe-for-dollar-1), complete with the line *“that’s a legally binding offer, no takesies backsies.”* Nobody got the truck. But the dealership pulled the bot entirely, and the incident became the reference case for what happens when a customer-facing AI has no boundary between “helpful” and “authorized to make commitments on the company’s behalf.” It’s a milder version of a pattern we’ve tracked before, where [AI agents confirm things that were never actually agreed to](https://industrycontents.com/ai-agent-journey-hallucinations/), with nobody catching it until a customer holds up the transcript. ## The fundamental reason There’s a structural reason this pattern recurs, and it isn’t that any single company is careless. [MIT Sloan Management Review’s 2025 report](https://sloanreview.mit.edu/projects/the-emerging-agentic-enterprise-how-leaders-must-navigate-a-new-age-of-ai/) with Boston Consulting Group frames agentic AI as introducing four distinct organizational tensions that older tools like search or scripted bots didn’t have, because those older tools didn’t plan, act, and adapt on their own. A rules-based chatbot can be scripted to never mention a competitor. A model trained to be genuinely useful has to be told not to, explicitly, and even then it may find the unexpected edge cases. MIT’s Sinan Aral says agentic strategy requires “systematic assessment of risks as well as business benefits,” precisely because the model doesn’t know your margin structure unless someone encodes it. That’s the mechanism. Its not that AI is disloyal, it’s just optimizing for the goal it was given, which is usually “answer the question well,” not “protect this specific revenue line,” and those two goals only happen to overlap by default when nobody’s checking. It’s the flip side of a question we’ve asked before: [an AI mentioning your business is not the same as an AI vouching for it](https://industrycontents.com/ai-mention-vs-recommendation/), and the same gap applies in reverse, an AI being helpful to your customer is not the same as it protecting your business. ## The Case Against the Panic It would be a cleaner story if every one of these cases were a company caught flat-footed by its own tool. That’s not quite what the evidence shows. Amazon built Buy for Me on purpose, knowing it would send money off-platform, because the alternative risk, losing shopper attention to third-party AI search, was judged worse. Several major hotel groups signed up for direct AI booking channels deliberately, as a strategy to cut OTA dependency. Bank of America’s overdraft numbers reflect a series of product decisions made over more than a decade, only some of which were AI-driven, and the bank frames the whole thing as a trust play that paid off in retention. In three of these four cases, the “bypass” was a chosen trade. Only the Chevrolet case looks like a pure accident, and even there, the actual financial damage was zero. Nobody got a dollar Tahoe. The cost was more reputational, and few viral memes and it happened because of a chatbot with no guardrails. The honest summary sits between the two extremes. Agentic AI does not automatically flatten a company’s margins or empty its funnel. It does, reliably, surface the path the customer would have taken anyway if they’d had thirty more minutes and better information. Whether that’s a threat or a feature depends entirely on whether the company decided that trade-off on purpose. ## A Framework for Auditing Your Own AI This doesn’t require a vendor audit or a data science team. It requires about an hour and the same three questions asked from the customer’s seat, not the boardroom’s. **Ask your own AI the question a bargain-hunter would ask.**“What’s the cheapest way to get X” or “how do I avoid the Y fee.” Whatever it says back is the honest baseline, not the marketing copy.**Check whether the answer routes around a paid step, a fee, or a fulfillment path you control.** A vague answer is fine. A specific, actionable workaround is the thing worth flagging.**Decide, on the record, whether that leak is acceptable.** Bank of America decided yes, on the overdraft alerts, and turned it into a retention story. A dealership decided no, on unlimited chatbot authority, and pulled the tool.**Set the boundary explicitly rather than hoping the model infers it.** Businesses built on facts like never franchising or staying family owned share the same root cause with the Chevrolet chatbot story: nobody told the AI, in plain and specific terms, what it was not allowed to concede, a gap that shows up whenever a company skips writing down[what the AI actually owes an explanation for](https://industrycontents.com/ai-right-to-explanation/).**Re-run the test after every model or vendor update.** The Chevrolet bot passed whatever testing it got before launch. The gap showed up two days after it went live. None of this requires guessing what an AI model will do. It requires asking it, in writing, and being willing to not like the answer. ## FAQ AI disintermediation occurs when a company’s own AI tools inadvertently or intentionally guide customers to bypass revenue-generating workflows. This leads users toward cheaper or faster alternatives that circumvent the business’s standard commissions, fees, or product catalogs. No, it is not always accidental. While many incidents are unintended, some companies make deliberate strategic bets to integrate AI in ways that prioritize overall customer experience over traditional internal routing. It is rarely possible or advisable to prevent this risk entirely. Businesses should instead focus on setting explicit operational boundaries and determining acceptable trade-offs rather than assuming the AI will inherently protect their financial interests. No, this risk scales across all business sizes. Any organization using support chatbots, booking assistants, or shopping helpers faces the challenge of balancing AI helpfulness with the preservation of their own revenue streams. The best first step is to simulate a price-sensitive customer by asking your AI questions that might lead to a cheaper alternative. If the AI directs the user to bypass your internal workflows, you have identified an immediate need for better guardrails.