Claudeor GPT-4 and ask, "Does this comply?"
Sure, it works most of the time. But for an audit tool, "most of the time" is a failure.
The problem is that when the LLM produces the verdict, the verdict inherits all the flaws of the LLM. It's non-deterministic; a plan might pass on Monday and get flagged on Tuesday. It's unfalsifiable because you can't see the actual threshold the model used—only a probability distribution. Most importantly, it's unauditable. If you tweak the prompt to fix one edge case, you have no way of knowing what other "judgments" shifted in the process. You aren't getting evidence; you're getting a second opinion with no paper trail.
My solution was a hard constraint: no LLM call is allowed in the judgment path.
Splitting the workflow into Zones #
The trick is to separate language tasks from logic tasks. LLMs are incredible at extraction but mediocre at strict rule enforcement.
Zone A (Probabilistic): This is where the LLM lives. It takes free-form English and extracts structured data (e.g., "3 sets, 8-10 reps, 75% of 1RM").Zone B (Deterministic): This is where Python lives. It takes that structured data and compares it against a hard-coded range (e.g., is 75% between 70% and 85%?).
The architecture looks like this:
Free Text
→ [LLM Extraction]
→ Structured Data
→ [Python Rules]
→ Verdict
When the boundary leaks #
In a real-world AI workflow, this boundary leaks constantly. I hit a bug where the extractor correctly pulled age_years: 14
from a request, but the routing logic failed. I had asked the model to also provide a minor
boolean. Since the input text didn't explicitly use the word "minor," the model left it null. The system then cheerfully evaluated a 14-year-old's plan against adult guidelines.
The fix was a simple line of code, not a better prompt:
if plan.get("age_years") is not None and plan["age_years"] < 18:
plan["minor"] = True
The lesson here is that any time you "hope" a model will infer something that follows mechanically from data you already possess, your boundary is in the wrong place.
This approach is objectively slower to build. Writing a "check this" prompt takes an afternoon; building a structured extraction pipeline and a rule engine takes weeks. But the result is a system that is actually verifiable. If a plan is rejected, you can point to the exact line of code and the exact extracted value that triggered the fail. That's the only way to build a professional-grade LLM agent system that people can actually trust.
Next Claude 3.5 Sonnet beats GPT-4o at designing complex file systems →