Why some decisions belong in AI—and others belong in five lines of code.
Part 13 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.
"Is this order eligible for a refund?" is four rules: delivered, paid, inside the return window, belongs to the customer.
An LLM can answer that. It would probably answer correctly almost every time. The interesting question isn't whether it can — it's what it costs to ask, multiplied by how often you ask.
Refund eligibility gets checked on every refund request, every status enquiry that mentions a return, and every retry. Say 50,000 checks a day for a mid-sized shop.
$ ./gradlew checkCost
One refund-eligibility check, 50,000 times a day
PATH PER CALL PER DAY PER YEAR
Claude Opus 5 $0.004000 $200.00 $73,000.00
Claude Sonnet 5 $0.001600 $80.00 $29,200.00
Claude Haiku 4.5 $0.000800 $40.00 $14,600.00
Java method $7.09e-13 $3.54e-08 $0.000013
Measured: ~70 ns per deterministic check
The model rows are published per-token prices times an estimated prompt: the policy as a system prompt, the order as JSON, the request, a structured verdict back. Call it 500 tokens in, 60 out.
The Java row is RefundEligibility.evaluate measured in a warmed-up loop and costed as rented CPU time. Seventy nanoseconds at $0.036 per vCPU-hour.
The gap is about a billion to one. Not a percentage — a factor with nine zeros. The deterministic check's entire annual compute bill is roughly one thousandth of a cent.
Nothing clever, which is the point:
public static double perCall(TokenPrice price, PromptSize prompt) {
return prompt.inputTokens() / PER_MILLION * price.inputPerMillion()
+ prompt.outputTokens() / PER_MILLION * price.outputPerMillion();
}
The prompt estimate lives in a value called PromptSize, not as a literal inside a formula, precisely so you can disagree with my token count and re-run the comparison with yours. Halve it and the cheapest model still costs $7,300 a year. There is no token estimate that makes this a close call.
flowchart LR
D{"How often does this<br/>decision run?"}
D -->|"once per conversation"| AI["AI: intent, retrieval<br/>$0.004 is a bargain"]
D -->|"per request, per retry,<br/>per rule"| SW["Software: policy, eligibility,<br/>risk tiers, scoping"]
AI --> B["The boundary from ADR 001"]
SW --> B
classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f
classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f
classDef same fill:#ecf2ed,stroke:#93b39d,color:#3d5344
class AI,SW step
class D decision
class B same
$0.004 for a judgement call that genuinely needs judgement is cheap. Reading "the shoes don't fit, can I send them back?" and turning it into a structured request is worth every cent, and no rules engine I'd want to maintain does it as well.
That call happens once per conversation. Eligibility happens per request, per retry, per rule evaluation. Same price tag, wildly different bill.
So cost doesn't tell you "AI expensive, code cheap". It tells you where the boundary from post 1 pays for itself: the components that ended up in domain are exactly the ones that run at high frequency and have a right answer. That wasn't a cost decision when I drew it — it was a correctness decision. The bill just happens to agree.
The same arithmetic runs anywhere cheap-per-unit meets high-volume. A fraud model scoring every transaction, versus a rules pre-filter that rejects the obvious ones first. A vision system inspecting every part on a line, versus a dimension check that catches most defects for free.
Put the expensive judgement where judgement is needed. Let the cheap deterministic thing handle the rest.
Three things, in rough order of likelihood:
That last point is the one I'd defend longest. Even at zero cost, I'd keep eligibility in a method: it's testable, it's inspectable in an audit, and it can't have a bad day. Cost isn't the reason for the boundary. It's just the easiest reason to put on a slide.
What decision in your system runs 50,000 times a day, and do you know what each one costs?