{"slug": "building-a-support-agent-that-refuses-to-make-things-up", "title": "Building a support agent that refuses to make things up", "summary": "An engineer building after-sales support agents for e-commerce shops has developed an architecture that prevents AI hallucination by routing every request to a closed set of intents, each mapped to a typed action with an explicit contract. The system uses a whitelist of facts and a natural-language layer that only phrases responses, ensuring the model cannot generate unverified information. Escalation is treated as a normal outcome with distinct reasons for hand-off to human agents.", "body_md": "Most \"AI customer service\" demos fall apart the same way. You ask a question the\n\nmodel can't answer from data, and instead of stopping, it produces something\n\nplausible. In a chat toy that's a curiosity. In after-sales support it's a\n\npromise your company now has to honour — a refund that was never approved, a\n\ndelivery date that doesn't exist, a return window that isn't your policy.\n\nI build after-sales agents for e-commerce shops, and almost all the engineering\n\ngoes into that one problem: making the agent's honesty a property of the\n\narchitecture, not of the prompt. Here's how that's actually put together.\n\nThe naive design is one model, one big prompt, and a pile of documents in a\n\nvector store. Ask \"where is my order 41822?\" and the retrieval layer returns the\n\nthree chunks that look most like the question. If none of them contain order\n\n41822 — because it's a live database row, not a document — the model still gets\n\na context window full of *order-shaped text*. It will answer. It will be wrong.\n\nThe fix isn't a better prompt. It's removing the model's ability to answer that\n\nclass of question at all.\n\nEvery request the agent handles is routed to exactly one of a fixed set of\n\nintents — order status, delivery delay, return, refund status, exchange, invoice,\n\nproduct question, cancellation. That set is closed. There is no fallback intent\n\nthat means \"answer anyway\".\n\nEach intent maps to a typed action with an explicit contract:\n\n```\n@dataclass(frozen=True)\nclass OrderStatus:\n    \"\"\"Reads the order system of record. Never generates a status.\"\"\"\n    order_ref: str\n\n    def resolve(self, ctx: Ctx) -> Resolution:\n        order = ctx.commerce.get_order(self.order_ref)   # Shopify / WooCommerce / API\n        if order is None:\n            return Resolution.escalate(\n                reason=Reason.NOT_FOUND,\n                say=\"I can't find that order number on this account.\",\n            )\n        return Resolution.answer(\n            template=\"order_status\",\n            facts=order.public_facts(),   # only whitelisted fields\n        )\n```\n\nTwo things matter here.\n\n`facts`\n\nis a whitelist, not the row.`public_facts()`\n\nreturns the carrier,\n\nthe tracking number, the shipped-at timestamp, the current state. It does not\n\nreturn the margin, the internal notes, the customer's other orders, or the\n\nfraud score. The model literally cannot leak a field it was never given.\n\n**The natural-language layer only phrases.** The model receives the resolved\n\nfacts and a template intent, and its job is to write one or two sentences in the\n\nshop's tone of voice. It is not asked what the status is. It is told, and asked\n\nto say it nicely. Hallucination has no surface to attach to, because there is no\n\nquestion left open at generation time.\n\n`Resolution.escalate`\n\nisn't an error path. It's a normal, expected outcome with\n\nits own quality bar — arguably the most important one.\n\n```\nclass Reason(Enum):\n    NOT_FOUND       = auto()   # no matching record\n    OUT_OF_SCOPE    = auto()   # intent not in the closed set\n    POLICY_UNCLEAR  = auto()   # rule exists but doesn't cover this case\n    LOW_CONFIDENCE  = auto()   # intent classification below threshold\n    HUMAN_REQUESTED = auto()   # customer asked for a person\n    EMOTIONAL       = auto()   # anger / distress detected\n```\n\nEach reason produces a different hand-off: a different message to the customer,\n\na different priority in the human queue, and a different summary attached to the\n\nticket so the agent who picks it up doesn't restart from zero.\n\nThe `EMOTIONAL`\n\nbranch is worth calling out. A furious customer is not a\n\nretrieval failure — the system may have every fact it needs. It's routed to a\n\nhuman anyway, because \"technically resolvable\" and \"should be resolved by a\n\nmachine\" are different questions. Getting that distinction wrong is how\n\nautomation projects lose the trust they were meant to build.\n\n`LOW_CONFIDENCE`\n\ndeserves a real threshold, calibrated per shop, not a\n\nhard-coded `0.7`\n\n. And the threshold should be *asymmetric*: the cost of a wrong\n\nrefund is not the cost of an unnecessary escalation.\n\nReading an order is safe. Issuing a refund is not. Those two live behind\n\ndifferent gates, and the gate is configuration owned by the merchant, not a\n\nprompt:\n\n```\nactions:\n  order_status:   { mode: auto }\n  return_label:   { mode: auto, max_value_eur: 80 }\n  refund:         { mode: propose }        # drafts it, a human clicks send\n  cancel_order:   { mode: propose }\n  address_change: { mode: auto, before_dispatch_only: true }\n```\n\n`propose`\n\nmode is what makes the first month of a deployment survivable. The\n\nagent does the work and produces the action; a human approves it. You watch the\n\napproval rate. When a given action has been approved without edit often enough,\n\n*that's* the evidence to flip it to `auto`\n\n— not a vendor's claim, and not a\n\nnumber in a slide.\n\nI run each shop on its own instance, in France, on a French model\n\n([Mistral](https://mistral.ai/)). That reads like a marketing line, so here's the\n\nengineering reason.\n\nSupport conversations are among the most sensitive data a shop holds. Not\n\nbecause of the order numbers — because of what customers write around them.\n\nAddresses. Health reasons for a return. Financial difficulty. Complaints about a\n\nperson. Once that transits through a shared multi-tenant pipeline in another\n\njurisdiction, \"where is my data\" stops being a question you can answer, and\n\nbecomes a question you forward to a vendor.\n\nA dedicated instance also makes reversibility real rather than contractual. If\n\nthe merchant leaves, the export is a database and a config file, not a support\n\nticket asking for their data back.\n\nThe EU AI Act's transparency obligation (Article 50) pushes the same way: the\n\ncustomer has to know they're talking to a machine. That's simpler to guarantee\n\nwhen the disclosure is in your own message-composition layer than when it's a\n\nsetting in someone else's dashboard.\n\nHonesty about the trade-off: this design resolves fewer conversations than a\n\nmodel with a free hand. A closed intent set can't handle the long tail. Bounded\n\nactions can't improvise. `propose`\n\nmode needs a human in the loop for weeks.\n\nThat's the trade I'd make every time. The failure mode of the permissive design\n\nisn't \"slightly worse answers\" — it's a confident wrong statement that a real\n\nperson acts on, in a channel where the shop is legally the one who said it.\n\nAn agent that says \"I'll get a human on this\" is a mildly disappointing\n\nexperience. An agent that invents a refund policy is an incident.\n\n*I'm Amine, founder of Bynevo Labs — we build\nsovereign after-sales AI agents\nfor French e-commerce, hosted in France on an open-source stack. Happy to talk\narchitecture in the comments.*", "url": "https://wpnews.pro/news/building-a-support-agent-that-refuses-to-make-things-up", "canonical_source": "https://dev.to/bynevolabs/building-a-support-agent-that-refuses-to-make-things-up-4g4l", "published_at": "2026-08-10 15:03:59+00:00", "updated_at": "2026-08-10 15:17:43.924635+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-products", "developer-tools"], "entities": ["Shopify", "WooCommerce"], "alternates": {"html": "https://wpnews.pro/news/building-a-support-agent-that-refuses-to-make-things-up", "markdown": "https://wpnews.pro/news/building-a-support-agent-that-refuses-to-make-things-up.md", "text": "https://wpnews.pro/news/building-a-support-agent-that-refuses-to-make-things-up.txt", "jsonld": "https://wpnews.pro/news/building-a-support-agent-that-refuses-to-make-things-up.jsonld"}}