{"slug": "where-should-ai-stop-and-code-start", "title": "Where Should AI Stop and Code Start?", "summary": "A developer's cost experiment comparing an LLM-powered support agent against deterministic code found that a refund-eligibility check run 50,000 times a day costs roughly $14,600 to $73,000 per year via Claude models versus about $0.000013 per year for a Java method, a gap of roughly a billion to one. The findings argue that high-frequency decisions with a right answer belong in software, while expensive model calls should be reserved for judgment calls that run once per conversation, such as parsing intent. The author notes the boundary was drawn as a correctness decision, not a cost one, and that the bill simply agrees.", "body_md": "*Why some decisions belong in AI—and others belong in five lines of code.*\n\nPart 13 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The [companion repo](https://github.com/antoniolopescorreia/reliable-ai-support) contains the full code.\n\n\"Is this order eligible for a refund?\" is four rules: delivered, paid, inside the return window, belongs to the customer.\n\nAn LLM can answer that. It would probably answer correctly almost every time. The interesting question isn't whether it can — it's what it costs to ask, multiplied by how often you ask.\n\nRefund eligibility gets checked on every refund request, every status enquiry that mentions a return, and every retry. Say 50,000 checks a day for a mid-sized shop.\n\n```\n$ ./gradlew checkCost\n\nOne refund-eligibility check, 50,000 times a day\n\nPATH                       PER CALL        PER DAY       PER YEAR\nClaude Opus 5             $0.004000        $200.00     $73,000.00\nClaude Sonnet 5           $0.001600         $80.00     $29,200.00\nClaude Haiku 4.5          $0.000800         $40.00     $14,600.00\nJava method               $7.09e-13      $3.54e-08      $0.000013\n\nMeasured: ~70 ns per deterministic check\n```\n\nThe model rows are published per-token prices times an estimated prompt: the policy as a system prompt, the order as JSON, the request, a structured verdict back. Call it 500 tokens in, 60 out.\n\nThe Java row is `RefundEligibility.evaluate` measured in a warmed-up loop and costed as rented CPU time. Seventy nanoseconds at $0.036 per vCPU-hour.\n\nThe gap is about a billion to one. Not a percentage — a factor with nine zeros. The deterministic check's entire annual compute bill is roughly one thousandth of a cent.\n\nNothing clever, which is the point:\n\n```\npublic static double perCall(TokenPrice price, PromptSize prompt) {\n    return prompt.inputTokens() / PER_MILLION * price.inputPerMillion()\n            + prompt.outputTokens() / PER_MILLION * price.outputPerMillion();\n}\n```\n\nThe prompt estimate lives in a value called `PromptSize`, not as a literal inside a formula, precisely so you can disagree with my token count and re-run the comparison with yours. Halve it and the cheapest model still costs $7,300 a year. There is no token estimate that makes this a close call.\n\n``` php\nflowchart LR\n    D{\"How often does this<br/>decision run?\"}\n    D -->|\"once per conversation\"| AI[\"AI: intent, retrieval<br/>$0.004 is a bargain\"]\n    D -->|\"per request, per retry,<br/>per rule\"| SW[\"Software: policy, eligibility,<br/>risk tiers, scoping\"]\n    AI --> B[\"The boundary from ADR 001\"]\n    SW --> B\n    classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f\n    classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f\n    classDef same fill:#ecf2ed,stroke:#93b39d,color:#3d5344\n    class AI,SW step\n    class D decision\n    class B same\n```\n\n$0.004 for a judgement call that genuinely needs judgement is cheap. Reading \"the shoes don't fit, can I send them back?\" and turning it into a structured request is worth every cent, and no rules engine I'd want to maintain does it as well.\n\nThat call happens **once per conversation**. Eligibility happens per request, per retry, per rule evaluation. Same price tag, wildly different bill.\n\nSo cost doesn't tell you \"AI expensive, code cheap\". It tells you where the boundary from post 1 pays for itself: the components that ended up in `domain` are exactly the ones that run at high frequency and have a right answer. That wasn't a cost decision when I drew it — it was a correctness decision. The bill just happens to agree.\n\nThe same arithmetic runs anywhere cheap-per-unit meets high-volume. A fraud model scoring every transaction, versus a rules pre-filter that rejects the obvious ones first. A vision system inspecting every part on a line, versus a dimension check that catches most defects for free.\n\nPut the expensive judgement where judgement is needed. Let the cheap deterministic thing handle the rest.\n\nThree things, in rough order of likelihood:\n\nThat last point is the one I'd defend longest. Even at zero cost, I'd keep eligibility in a method: it's testable, it's inspectable in an audit, and it can't have a bad day. Cost isn't the reason for the boundary. It's just the easiest reason to put on a slide.\n\n*What decision in your system runs 50,000 times a day, and do you know what each one costs?*", "url": "https://wpnews.pro/news/where-should-ai-stop-and-code-start", "canonical_source": "https://dev.to/tonal/where-should-ai-stop-and-code-start-1gcp", "published_at": "2026-09-16 08:14:15+00:00", "updated_at": "2026-09-16 08:43:00.368586+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Claude Opus 5", "Claude Sonnet 5", "Claude Haiku 4.5", "Java", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/where-should-ai-stop-and-code-start", "markdown": "https://wpnews.pro/news/where-should-ai-stop-and-code-start.md", "text": "https://wpnews.pro/news/where-should-ai-stop-and-code-start.txt", "jsonld": "https://wpnews.pro/news/where-should-ai-stop-and-code-start.jsonld"}}