An Agent on a Leash, or why my AI agent doesn't make business decisions

This post kicks off an ongoing experiment: building an LLM-powered support agent you can actually trust, one decision at a time. Everything described here ships in the companion repo

[(]reliable-ai-support

[code as of this post: tag]), which grows as the series does. Where I mention future topics, read them as current intentions, not contracts — the whole point of building in public is that the plan bends to what actually works.post-001

Picture this: It's 2 AM. A customer messages our support channel, frustrated because their recent order never arrived. They want a refund, or at minimum, an explanation. They've been emailing back and forth for three days with no resolution.

A support agent is assigned. But it's 2 AM. There's no one on call. So... does the AI just handle it?

That's the question I couldn't let go of — the one this whole series exists to answer.

The fantasy is seductive. Build an LLM-powered agent, give it access to the right tools, and let it handle issues end-to-end. It markets itself: "AI-powered 24/7 customer support!" Users love it. Everyone wins.

Except... what happens when the AI decides to process a refund to the wrong person? What happens when it accesses data it shouldn't? What happens when it "hallucinates" an order ID and tries to charge it?

The first thing I had to accept when designing this system: an LLM's confidence is not a reliability metric. It's a correlation metric. And correlation alone isn't enough.

LLMs are great at understanding intent, surfacing information, writing responses. The problem is what happens when you hand them the keys to the kingdom.

So I built a table. A boundary that says "up to here, and no further."

The AI interprets intent. The software enforces policy. That's the split, and it's the thesis of everything that follows.

If you remember one picture from this series, make it this one:

flowchart LR
    A[User input] --> B[Intent classification<br/>AI]
    B --> C{Deterministic validation<br/>software}
    C -->|low / medium risk| D[AI-assisted action]
    C -->|high risk| E[Human approval<br/>with audit trail]
    C -->|very high risk| F[Proposed only —<br/>human executes manually]
    D --> G[Execution or rollback]
    E --> G
    F --> H[Manual execution]

Every action passes through the deterministic gate first. What happens after the gate depends on the risk tier: low and medium risk actions proceed with AI assistance, high risk actions for human approval with a full audit trail, and very high risk actions are never executed by the system at all — the AI proposes, a human does the actual work. The tier-specific gates deserve their own walkthrough later; for now, remember just this: nothing executes without passing deterministic validation. No "let me just try it and see" energy allowed.

The gate either opens or closes based on hard-coded rules. The AI never holds the key.

I like to think of this as a budget. You have a certain amount of "AI agency" to spend, and once it's spent, the human takes over.

Low risk: AI proposes, human disposes. Example: summarizing a support ticket, generating a response draft, highlighting relevant knowledge base articles. The AI does the heavy lifting, the human signs off.

Medium risk: AI routes, human confirms. Example: directing a user to the right self-service option, suggesting the right KB article, classifying the ticket type.

High risk: AI proposes action with audit trail, human approves. Example: processing a refund, updating a user's permissions, modifying an order. Money or critical data is involved.

Very high risk: AI never executes. Proposes only. Human performs action manually. Example: canceling a subscription, changing a user's email address, deleting data. The AI can suggest what to do, but the actual action? That's on a human. Period. No exceptions.

The higher the risk, the more hands are in the cookie jar. And that's exactly how it should be.

This design isn't free, and pretending otherwise would make this a sales pitch instead of engineering notes.

What would change my mind? If eval suites get good enough that an AI's judgment on specific action classes is measurably more accurate than the rules — with error rates we can pin and monitor — then some of these gates could safely open. And building that evidence is exactly what I want this series to be about.

So what actually happens to our insomniac customer now?

Their message hits the pipeline: intent classification understands "refund for undelivered order." The deterministic gate runs RefundEligibility.evaluate()

— order not delivered, refund eligible. The system drafts a response, proposes the refund, files it in the approval queue with a full audit trail. At 8 AM, a human opens the queue, sees the proposal with every fact attached, and clicks approve. The customer gets their money back before their first coffee.

No one decided anything at 2 AM. The system prepared everything; a human decided something at 8. That gap between prepare and decide is the whole architecture.

If you're building AI-powered systems, you've probably felt the tension between "move fast and break things" and "don't break customers' lives." This tension doesn't go away just because you're using fancier models.

The deterministic/non-deterministic split is where this series starts. As the system grows, other tensions will demand their own answers: what the agent's mistakes cost, how you test behavior that's different every run, which failures you design for versus prevent. Every trade-off gets documented — ADRs included — so you can watch the thinking evolve.

The autonomy budget translates directly beyond customer service bots: healthcare triage must route critical cases to physicians, financial chatbots can't move money without verification, autonomous vehicle perception must never execute without deterministic gatekeeping. The same pattern — AI interprets intent, software enforces policy — shows up wherever probabilistic intelligence meets real consequences.

The split between AI and software becomes concrete the moment you actually need to check whether a refund can happen. Here's the shape of the class that does it — a plain Java class living in app/domain

, whose entire job is to answer one yes/no question: can this order be refunded under policy? Notice what's absent: no LLM call, no prompt, no model config. Just facts in, verdict out.

// app/domain/RefundEligibility.java
public EvaluationResult evaluate(Order order, RefundRequest request) {
    // Deterministic business rules — no AI involved
    if (!order.isDelivered()) {
        return EvaluationResult.notEligible("Order must be delivered before refund");
    }
    if (!order.isPaid()) { /* ... */ }
    if (order.ageInDays() > RETURN_WINDOW_DAYS) { /* ... */ }

    // Eligible ≠ executed: actually issuing the refund goes through
    // a separate, risk-tiered authorization gate.
    return EvaluationResult.eligible(order.getId(), order.getCustomerId());
}

Two things worth noticing:

eligible(...)

is not an instruction to move money.The complete implementation — records, all rules, and the test suite that proves the determinism claim — lives in the companion repo. Walking through it line by line, side by side with what an LLM-decided version would have looked like, is high on my list for an upcoming post. For now, hold onto the shape: this is what "software enforces policy" looks like when it's not a slide.

Next up (working title): "Drawing the Line: What Deserves a Model and What Doesn't" — where I break down exactly why refund eligibility is a software problem, not an AI problem.

And if you disagree with any of this — great. That means you're thinking critically about the problem. Leave a comment, let's hash it out.