Drawing the Line: What Deserves an LLM and What Doesn't

A developer building an LLM-powered support agent outlines a decision framework for determining which components should use AI versus deterministic software, emphasizing that AI interprets intent while software enforces policy. The approach uses a static risk-tier lookup to prevent prompt injection from reclassifying actions, and retrieval is treated as a hybrid where fuzzy results sit behind a hard, testable contract.

The three questions I use to decide what stays deterministic Part 2 of an ongoing experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo grows with the series. Every AI feature reaches the same fork. This piece here — does it get a model, or does it get a method? Go wrong one way and you've built a rules engine that can't read a sentence. Go wrong the other and a language model is deciding whether someone gets their money back. "The AI interprets intent, software enforces policy" is easy to say — it's the principle this series started from https://dev.to/tonal/an-agent-on-a-leash-or-why-my-ai-agent-doesnt-make-business-decisions-1o1 . Applying it to a specific component on a specific Tuesday is the hard part. This post is the ruler I use for that. For every component, in order: php flowchart LR A "New component" -- B{"Is the answer a fact?"} B -- "no" -- AI "AI" B -- "yes" -- C{"Costs money or trust?"} C -- "no" -- P "Either" C -- "yes" -- D{"Can a test pin it today?"} D -- "yes" -- SW "Deterministic software" D -- "no" -- HY "Software contract around AI" classDef box fill: eef2f6,stroke: 8fa3b8,color: 24313f classDef decision fill: f7f4ec,stroke: b3a988,color: 24313f classDef ai fill: eef0f4,stroke: 8fa3b8,color: 24313f class A,AI,P,HY box class SW box class B,C,D decision Applied to Post 1's components: intent interpretation goes to the AI. Refund eligibility to software. Retrieval targets end up hybrid the interesting case, below . Refund execution is software plus a human gate. Policy exceptions are a job for a rule engine. Retrieval deserves its own paragraph, because it's where most teams get tripped up. "Let the AI find the right knowledge base article" sounds like an AI decision. It isn't — at least not entirely. The results are fuzzy: similarity search returns plausible articles, ranked, sometimes wrong. You cannot write assert search "refund" == refundsArticle and mean it. But the call is rigid: which function runs, with what arguments, against which index. That part is plain software with a typed signature, and it's fully unit-testable: // The results are probabilistic. The invocation isn't. public interface KnowledgeBase { / Returns up to k articles ranked by semantic relevance. Ranking quality is evaluated statistically, never asserted exactly. / List<KnowledgeArticle search Query query ; } That split — fuzzy contents behind a hard contract — is what makes retrieval safe to hand to the model as a tool . The agent decides when to search; it never gets to redefine what searching means. Here's the part that took me longest to appreciate. Deciding who decides is itself a decision — so who makes that one? If the answer is "the LLM classifies each action's risk tier at runtime," the whole architecture collapses: the boundary becomes another probabilistic output that can be talked into moving. Prompt injection doesn't need to break a rule if it can reclassify the action the rule applies to. So in this system, the classification of every action is a static lookup — code, not judgment: // dev/tonal/support/domain/RiskPolicy.java public enum RiskTier { LOW, MEDIUM, HIGH, VERY HIGH } public final class RiskPolicy { private static final Map<ActionType, RiskTier TIERS = Map.of ActionType.SUMMARIZE TICKET, RiskTier.LOW, ActionType.DRAFT RESPONSE, RiskTier.LOW, ActionType.CLASSIFY TICKET, RiskTier.MEDIUM, ActionType.PROCESS REFUND, RiskTier.HIGH, ActionType.MODIFY ORDER, RiskTier.HIGH, ActionType.UPDATE PERMISSIONS, RiskTier.HIGH, ActionType.CANCEL SUBSCRIPTION, RiskTier.VERY HIGH, ActionType.DELETE DATA, RiskTier.VERY HIGH ; public static RiskTier tierFor ActionType action { return TIERS.get action ; // null = unclassified = fails closed } } Two deliberate choices in there: Map.of over clever logic.And the tests are about the map itself, not just lookups: @Test void everyActionMustHaveATier { for ActionType action : ActionType.values { assertThat RiskPolicy.tierFor action .as "action %s must be classified", action .isNotNull ; } } @Test void destructiveActionsAreNeverLowRisk { assertThat RiskPolicy.tierFor ActionType.DELETE DATA .isEqualTo RiskTier.VERY HIGH ; assertThat RiskPolicy.tierFor ActionType.CANCEL SUBSCRIPTION .isEqualTo RiskTier.VERY HIGH ; } The first test is the important one: it pins exhaustiveness. Nobody can silently add an action type next sprint and forget to give it a risk tier — CI fails until they classify it. The boundary enforces itself. The same three questions classify decisions in any domain where models meet consequences: clinical triage systems route judgment but never prescribe fact, high cost ; loan underwriting separates scoring models from disbursement logic; industrial safety controllers treat perception as input but interlocks as law. Wherever you look, the durable systems aren't the ones with the smartest model — they're the ones where nobody had to trust the model on a question that has a right answer. Previous posts in this series: