{"slug": "the-more-powerful-the-ai-the-more-the-architecture-matters", "title": "The More Powerful the AI, the More the Architecture Matters", "summary": "A developer built an LLM-powered customer support agent that uses AI only for intent detection and document retrieval while deterministic software enforces refund eligibility, access scoping, and risk tiers, with every guarantee backed by a test that fails if the boundary breaks. The project, documented across fourteen posts with a companion repository, runs without an API key and deliberately omits multi-tenancy, real persistence, and LLM-as-judge evaluation. The developer notes unresolved gaps including no rate limiting on proposals and no protection against duplicate refund execution on retry.", "body_md": "*The boundaries I designed, the gaps I haven't solved, and why the difference matters*\n\nPart 14 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The [companion repo](https://github.com/antoniolopescorreia/reliable-ai-support) contains the full code.\n\nIf someone asked me over a coffee what I've been building, the short version is this.\n\n**A support agent that can work out whether you're owed a refund, and cannot give you one.**\n\nThe AI reads the message and finds the documents. Software decides eligibility, owns the risk tiers, scopes every lookup to whoever is actually logged in, and parks anything expensive in a queue for a human. Fourteen posts, one repo, and the whole thing runs with no API key.\n\n``` php\nflowchart LR\n    M[\"Customer message\"] --> AI[\"AI: intent, retrieval\"]\n    AI --> SW[\"Software: scoping,<br/>eligibility, risk tier\"]\n    SW --> G{\"Gate\"}\n    G -->|\"LOW\"| X[\"Runs\"]\n    G -->|\"MEDIUM to HIGH\"| H[\"Human approves\"]\n    G -->|\"VERY HIGH\"| P[\"Proposed only,<br/>a person executes\"]\n    classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f\n    classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f\n    classDef good fill:#ecf2ed,stroke:#93b39d,color:#3d5344\n    class M,AI,X step\n    class SW,H,P good\n    class G decision\n```\n\nEvery guarantee in there has a test that fails when it stops being true — the gate, the scoping, the risk tiers. The one I'd argue about longest is the dullest of them:\n\n```\n/** The only packages a domain class may reference: itself and the JDK. */\nprivate static final String[] DOMAIN_ALLOWLIST = {\n        \"dev.tonal.support.domain..\", \"java..\", \"javax..\"\n};\n\n@Test\nvoid domainDependsOnlyOnItselfAndTheJdk() {\n    classes().that().resideInAPackage(\"dev.tonal.support.domain..\")\n            .should().onlyDependOnClassesThat()\n            .resideInAnyPackage(DOMAIN_ALLOWLIST)\n            .check(production);\n}\n```\n\nAn allowlist, not a blacklist: the business rules may reference themselves and the JDK, and nothing else. No enumerating of forbidden layers, which would need updating every time I add a package. Add a JSON library to a policy class and the build goes red — I checked, by breaking it on purpose and watching it fail.\n\nIt exists because this kind of architecture decays quietly. Nobody announces the afternoon they import a framework into the business rules. It just happens, and a few months later \"the AI doesn't decide business questions\" isn't true any more — and nobody noticed the day it stopped being true.\n\n**A sentence in a README can't catch that. A failing build can.**\n\nQuite a lot. The interesting question is which parts were decisions.\n\nMulti-tenancy was. Per-tenant risk policies mean the tier lookup becomes runtime configuration, and \"who gets to decide\" is the last thing I want configurable by a form. I'd want evidence that a second tenant genuinely needs different tiers before I'd trade that away.\n\nReal persistence was too, though it sounds lazier. The approval queue and the audit trail are ports with in-memory adapters behind them, and the ports already spell out what a real implementation owes you: durability, notification, permanence. Writing that adapter is an afternoon. Deciding what it must guarantee was the part worth the thinking.\n\nAnd no LLM-as-judge in the eval suite, which I keep being tempted by. A model grading a model has its own variance and its own bad days, so it would need its own eval before I'd believe a word of its verdicts. That's a project, not a checkbox.\n\n**There's a difference between \"I decided not to\" and \"I haven't got to it\".**\n\nSaying which is which out loud is the only thing stopping every gap from becoming retroactive wisdom.\n\nNothing rate-limits proposals, so somebody patient could bury a reviewer in plausible-looking refund requests until approving becomes reflex. An approved refund executed twice after a retry isn't prevented. And nothing notices if the knowledge base quietly gets worse while every test stays green. Those sit in the failure-mode catalogue marked Planned, which is a better place for them than my head.\n\nHere's the part that would make me sceptical if I were reading someone else's series.\n\nIt has never served a real customer. Every scenario in the eval suite is one I invented, which makes it good at catching regressions and useless against the thing I didn't think of. Real users are more imaginative than I am, and they aren't trying to be fair.\n\nThe suite also grades deterministic classifiers, so anyone can clone the repo and get my exact numbers. That's a genuine win for reproducibility and a genuine limit on what I've shown: the version swap I graded was one component against another, not one provider against another. And I've never measured end-to-end latency with a hosted model in the loop.\n\nSo: a design I believe in, tests for the parts that are testable, and no war stories. If it ever does meet production traffic, the thing I most want to know is whether the audit trail catches what the evals missed.\n\nWhat's the gap in your system that everyone calls deliberate?", "url": "https://wpnews.pro/news/the-more-powerful-the-ai-the-more-the-architecture-matters", "canonical_source": "https://dev.to/tonal/-446k", "published_at": "2026-09-19 12:37:01+00:00", "updated_at": "2026-09-19 12:54:26.409818+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "developer-tools"], "entities": ["GitHub"], "alternates": {"html": "https://wpnews.pro/news/the-more-powerful-the-ai-the-more-the-architecture-matters", "markdown": "https://wpnews.pro/news/the-more-powerful-the-ai-the-more-the-architecture-matters.md", "text": "https://wpnews.pro/news/the-more-powerful-the-ai-the-more-the-architecture-matters.txt", "jsonld": "https://wpnews.pro/news/the-more-powerful-the-ai-the-more-the-architecture-matters.jsonld"}}