Where Should AI Stop and Code Start?

A developer's cost experiment comparing an LLM-powered support agent against deterministic code found that a refund-eligibility check run 50,000 times a day costs roughly $14,600 to $73,000 per year via Claude models versus about $0.000013 per year for a Java method, a gap of roughly a billion to one. The findings argue that high-frequency decisions with a right answer belong in software, while expensive model calls should be reserved for judgment calls that run once per conversation, such as parsing intent. The author notes the boundary was drawn as a correctness decision, not a cost one, and that the bill simply agrees.

Why some decisions belong in AI—and others belong in five lines of code. Part 13 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo https://github.com/antoniolopescorreia/reliable-ai-support contains the full code. "Is this order eligible for a refund?" is four rules: delivered, paid, inside the return window, belongs to the customer. An LLM can answer that. It would probably answer correctly almost every time. The interesting question isn't whether it can — it's what it costs to ask, multiplied by how often you ask. Refund eligibility gets checked on every refund request, every status enquiry that mentions a return, and every retry. Say 50,000 checks a day for a mid-sized shop. $ ./gradlew checkCost One refund-eligibility check, 50,000 times a day PATH PER CALL PER DAY PER YEAR Claude Opus 5 $0.004000 $200.00 $73,000.00 Claude Sonnet 5 $0.001600 $80.00 $29,200.00 Claude Haiku 4.5 $0.000800 $40.00 $14,600.00 Java method $7.09e-13 $3.54e-08 $0.000013 Measured: ~70 ns per deterministic check The model rows are published per-token prices times an estimated prompt: the policy as a system prompt, the order as JSON, the request, a structured verdict back. Call it 500 tokens in, 60 out. The Java row is RefundEligibility.evaluate measured in a warmed-up loop and costed as rented CPU time. Seventy nanoseconds at $0.036 per vCPU-hour. The gap is about a billion to one. Not a percentage — a factor with nine zeros. The deterministic check's entire annual compute bill is roughly one thousandth of a cent. Nothing clever, which is the point: public static double perCall TokenPrice price, PromptSize prompt { return prompt.inputTokens / PER MILLION price.inputPerMillion + prompt.outputTokens / PER MILLION price.outputPerMillion ; } The prompt estimate lives in a value called PromptSize , not as a literal inside a formula, precisely so you can disagree with my token count and re-run the comparison with yours. Halve it and the cheapest model still costs $7,300 a year. There is no token estimate that makes this a close call. php flowchart LR D{"How often does this<br/ decision run?"} D -- |"once per conversation"| AI "AI: intent, retrieval<br/ $0.004 is a bargain" D -- |"per request, per retry,<br/ per rule"| SW "Software: policy, eligibility,<br/ risk tiers, scoping" AI -- B "The boundary from ADR 001" SW -- B classDef step fill: eef2f6,stroke: 8fa3b8,color: 24313f classDef decision fill: f7f4ec,stroke: b3a988,color: 24313f classDef same fill: ecf2ed,stroke: 93b39d,color: 3d5344 class AI,SW step class D decision class B same $0.004 for a judgement call that genuinely needs judgement is cheap. Reading "the shoes don't fit, can I send them back?" and turning it into a structured request is worth every cent, and no rules engine I'd want to maintain does it as well. That call happens once per conversation . Eligibility happens per request, per retry, per rule evaluation. Same price tag, wildly different bill. So cost doesn't tell you "AI expensive, code cheap". It tells you where the boundary from post 1 pays for itself: the components that ended up in domain are exactly the ones that run at high frequency and have a right answer. That wasn't a cost decision when I drew it — it was a correctness decision. The bill just happens to agree. The same arithmetic runs anywhere cheap-per-unit meets high-volume. A fraud model scoring every transaction, versus a rules pre-filter that rejects the obvious ones first. A vision system inspecting every part on a line, versus a dimension check that catches most defects for free. Put the expensive judgement where judgement is needed. Let the cheap deterministic thing handle the rest. Three things, in rough order of likelihood: That last point is the one I'd defend longest. Even at zero cost, I'd keep eligibility in a method: it's testable, it's inspectable in an audit, and it can't have a bad day. Cost isn't the reason for the boundary. It's just the easiest reason to put on a slide. What decision in your system runs 50,000 times a day, and do you know what each one costs?