Drawing the Line: What Deserves an LLM and What Doesn't

The three questions I use to decide what stays deterministic

Part 2 of an ongoing experiment: building an LLM-powered support agent with deterministic boundaries. The

[companion repo]grows with the series.

Every AI feature reaches the same fork. This piece here — does it get a model, or does it get a method?

Go wrong one way and you've built a rules engine that can't read a sentence. Go wrong the other and a language model is deciding whether someone gets their money back.

"The AI interprets intent, software enforces policy" is easy to say — it's the principle this series started from. Applying it to a specific component on a specific Tuesday is the hard part.

This post is the ruler I use for that.

For every component, in order:

flowchart LR
    A["New component"] --> B{"Is the answer a fact?"}
    B -- "no" --> AI["AI"]
    B -- "yes" --> C{"Costs money or trust?"}
    C -- "no" --> P["Either"]
    C -- "yes" --> D{"Can a test pin it today?"}
    D -- "yes" --> SW["Deterministic software"]
    D -- "no" --> HY["Software contract around AI"]
    classDef box fill:#eef2f6,stroke:#8fa3b8,color:#24313f
    classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f
    classDef ai fill:#eef0f4,stroke:#8fa3b8,color:#24313f
    class A,AI,P,HY box
    class SW box
    class B,C,D decision

Applied to Post 1's components: intent interpretation goes to the AI. Refund eligibility to software. Retrieval targets end up hybrid (the interesting case, below). Refund execution is software plus a human gate. Policy exceptions are a job for a rule engine.

Retrieval deserves its own paragraph, because it's where most teams get tripped up. "Let the AI find the right knowledge base article" sounds like an AI decision. It isn't — at least not entirely.

The results are fuzzy: similarity search returns plausible articles, ranked, sometimes wrong. You cannot write assert(search("refund") == refundsArticle)

and mean it.

But the call is rigid: which function runs, with what arguments, against which index. That part is plain software with a typed signature, and it's fully unit-testable:

// The results are probabilistic. The invocation isn't.
public interface KnowledgeBase {
    /** Returns up to k articles ranked by semantic relevance.
     *  Ranking quality is evaluated statistically, never asserted exactly. */
    List<KnowledgeArticle> search(Query query);
}

That split — fuzzy contents behind a hard contract — is what makes retrieval safe to hand to the model as a tool. The agent decides when to search; it never gets to redefine what searching means.

Here's the part that took me longest to appreciate. Deciding who decides is itself a decision — so who makes that one?

If the answer is "the LLM classifies each action's risk tier at runtime," the whole architecture collapses: the boundary becomes another probabilistic output that can be talked into moving. Prompt injection doesn't need to break a rule if it can reclassify the action the rule applies to.

So in this system, the classification of every action is a static lookup — code, not judgment:

// dev/tonal/support/domain/RiskPolicy.java
public enum RiskTier { LOW, MEDIUM, HIGH, VERY_HIGH }

public final class RiskPolicy {

    private static final Map<ActionType, RiskTier> TIERS = Map.of(
            ActionType.SUMMARIZE_TICKET,    RiskTier.LOW,
            ActionType.DRAFT_RESPONSE,      RiskTier.LOW,
            ActionType.CLASSIFY_TICKET,     RiskTier.MEDIUM,
            ActionType.PROCESS_REFUND,      RiskTier.HIGH,
            ActionType.MODIFY_ORDER,        RiskTier.HIGH,
            ActionType.UPDATE_PERMISSIONS,  RiskTier.HIGH,
            ActionType.CANCEL_SUBSCRIPTION, RiskTier.VERY_HIGH,
            ActionType.DELETE_DATA,         RiskTier.VERY_HIGH);

    public static RiskTier tierFor(ActionType action) {
        return TIERS.get(action); // null = unclassified = fails closed
    }
}

Two deliberate choices in there:

Map.of

over clever logic.And the tests are about the map itself, not just lookups:

@Test
void everyActionMustHaveATier() {
    for (ActionType action : ActionType.values()) {
        assertThat(RiskPolicy.tierFor(action))
                .as("action %s must be classified", action)
                .isNotNull();
    }
}

@Test
void destructiveActionsAreNeverLowRisk() {
    assertThat(RiskPolicy.tierFor(ActionType.DELETE_DATA)).isEqualTo(RiskTier.VERY_HIGH);
    assertThat(RiskPolicy.tierFor(ActionType.CANCEL_SUBSCRIPTION)).isEqualTo(RiskTier.VERY_HIGH);
}

The first test is the important one: it pins exhaustiveness. Nobody can silently add an action type next sprint and forget to give it a risk tier — CI fails until they classify it. The boundary enforces itself.

The same three questions classify decisions in any domain where models meet consequences: clinical triage systems route judgment but never prescribe (fact, high cost); loan underwriting separates scoring models from disbursement logic; industrial safety controllers treat perception as input but interlocks as law. Wherever you look, the durable systems aren't the ones with the smartest model — they're the ones where nobody had to trust the model on a question that has a right answer.

Previous posts in this series: