Originally published on tamiz.pro.
In the rapidly expanding ecosystem of Large Language Model (LLM) applications, a specific architectural anti-pattern has emerged that is quietly draining engineering budgets and system reliability. It is the practice of using a general-purpose, non-deterministic reasoning engine (the LLM) to perform tasks that are fundamentally deterministic and cheap to compute. We call this "If-Statements with a GPU Bill."
You see it in the latest agent frameworks: a system prompt that says, "If the user asks for a refund, check the database. If the user is asking about the weather, call the weather tool." Or an agent graph where a planner LLM decides to route to a simple API endpoint that requires zero reasoning. When you pay for that inference call, you aren't just paying for the token compute; you are paying for the latency, the complexity, and the probability of hallucination that comes with asking a transformer to act as a state machine.
This article dissects why this pattern exists, how it breaks in production, and how to refactor it back into solid, maintainable engineering. We will move from the "black box" of an agent's internal thought process to the white box of explicit control flow.
To understand where the failure is, we must map the anatomy of a typical over-engineered agent workflow. Consider a support agent built on a modern orchestration library (like LangGraph, CrewAI, or custom Python).
The Standard Bloat Pattern:
get_order_status tool."{ "status": "delayed", "reason": "Shipping hold", "eta": "2 days" }.
Where is the inefficiency? Steps 2 and 5. The LLM did not need to "decide" to use the tool; the user's intent ("order late") mapped directly to a specific query. The LLM did not need to "polite-ify" the response; a template does that better.
However, in a multi-step agent, the bloat compounds. If the agent has a "memory" module, it might invoke an LLM to decide what to store. If it has a "search" module, it might invoke an LLM to write a query for the vector database. Every single edge in this graph represents a decision that could likely be made by a human developer in 5 lines of code.
Before you can refactor, you must identify which parts of your agent are "deterministic" and which are genuinely "probabilistic."
A task is probabilistic if the solution space is too large to enumerate, or if it requires semantic understanding of unstructured data. Examples: "Summarize this 50-page legal document," "Write a poem in the style of Shakespeare," or "Route this ambiguous ticket to the correct department based on subtle context."
A task is deterministic if the logic can be expressed in boolean operators, database queries, or simple state transitions. Examples: "Check if the user is an admin," "Call the API to get the current time," "Extract the phone number from the text."
When reviewing your agent's logs or prompt engineering, look for these diagnostic markers:
The core principle of refactoring over-engineered agents is to push deterministic logic down the stack, away from the LLM. We want the LLM to be the "cognitive core"—handling ambiguity, intent, and synthesis—while the surrounding infrastructure handles precision and routing.
Instead of asking the LLM "Which tool should I use?", design a pre-flight interceptor.
Before (LLM-Heavy):
get_weather tool.
After (Rule-Light):
This reduces latency by 50-70% and removes a potential failure point (where the LLM might get confused by a similar-sounding word).
Often, developers use LLMs to parse user input into structured data. While LLMs are great at handling messy, unstructured input (e.g., "I need a 20% discount on the blue shirt I bought in March"), they should not be trusted to validate the result.
Refactor:
{"item": "blue shirt", "time": "March", "discount": 0.2}.
This "Code-First Validation" ensures that you are not paying GPU cycles to catch a simple type error.
Let's look at how this refactor changes the code. We will assume a Python context with a hypothetical Agent class.
class OverEngineeredAgent:
def handle_request(self, user_input: str):
prompt = f"""You are an agent. User said: {user_input}.
Decide if we should check order status or check weather.
Return tool name."""
decision = self.llm.generate(prompt) # Expensive & Slow
if "order" in decision:
data = self.order_service.get(user_input)
elif "weather" in decision:
data = self.weather_service.get(user_input)
final_prompt = f"""Tool returned: {data}. Write a polite email."""
response = self.llm.generate(final_prompt)
return response
Critique: The LLM is used to "decide" simple logic and "write" a simple template. Both are low-value LLM tasks.
import re
class EfficientAgent:
def handle_request(self, user_input: str):
intent = self._determine_intent(user_input)
if intent == "order_status":
data = self.order_service.get(user_input)
elif intent == "weather":
data = self.weather_service.get(user_input)
else:
decision = self._llm_resolve_ambiguity(user_input)
data = self._dispatch_tool(decision, user_input)
prompt = f"""The user requested an update. The raw data is: {json.dumps(data)}.
Context: {self._get_context()}.
Write a natural, helpful response."""
response = self.llm.generate(prompt)
return response
def _determine_intent(self, text: str) -> str:
"""Fast, deterministic intent detection."""
if "order" in text.lower() and "status" in text.lower():
return "order_status"
elif "weather" in text.lower():
return "weather"
return "unknown"
Analysis:
_determine_intent method runs in microsecond-level. unknown intent, which is a rare edge case.
Over-engineering extends beyond routing to state management. A common mistake is treating the LLM as a memory bank for facts that should be in a database.
Some developers argue, "But if I put the database data into the prompt, the LLM will answer better!" While true, doing so indiscriminately leads to "context blindness." The LLM struggles to focus when the signal-to-noise ratio drops.
When a tool returns a massive JSON (e.g., a 500-item inventory list), do not dump it into the prompt.
prompt = f"Here is the inventory: {json}"
stock = db.get_stock('X') -> returns True/False. Item X is in stock: True.
The LLM never sees the 500 items. It only sees the answer. This is the "Push Down" principle applied to data.
When you refactor these workflows, debugging becomes significantly easier, but it also requires a new mindset.
In a fully LLM-driven agent, if the output is wrong, you often have no idea why. Did it fail because it chose the wrong tool? Because it hallucinated the tool parameters? Because it got confused by the context?
Tools for Debugging Logic-First Agents:
Log.info("Routing to 'Order' via regex").
Because you are using code for routing, you can now implement
a strict validation layer that checks the LLM's output against ground truth data before it even reaches the user. Instead of asking the model to "verify its own work" (which is prone to sycophancy), we use code to assert that specific fields exist, types are correct, and values fall within expected ranges.
def validate_agent_output(raw_response: str, expected_schema: dict):
"""
Deterministic validation of the LLM's JSON output.
If this fails, we do not trust the model's reasoning and trigger a retry loop.
"""
try:
parsed = json.loads(raw_response)
except json.JSONDecodeError:
return False, "Invalid JSON format"
for key in expected_schema.keys():
if key not in parsed:
return False, f"Missing required field: {key}"
for key, value in expected_schema.items():
if type(parsed.get(key)) != value:
return False, f"Field '{key}' is {type(parsed.get(key))}, expected {value}"
if "quantity" in parsed and parsed["quantity"] < 0:
return False, "Quantity cannot be negative"
return True, "Validation passed"
success, error_msg = validate_agent_output(llm_output, schema={"order_id": str, "status": str, "quantity": int})
if not success:
correction_prompt = f"Your previous output failed validation: {error_msg}. Retry with correct JSON."
llm_output = generate_response(correction_prompt)
This pattern shifts the burden of "correctness" from the probabilistic model to the deterministic code. The LLM is allowed to be creative in its reasoning, but the code is the final arbiter of structural integrity.
One of the most insidious aspects of over-engineered agent workflows is the "infinite loop" or "rabbit hole" behavior. An agent gets stuck in a planning loop, re-reading the same tool documentation, or making the same API call repeatedly because it doesn't understand why it's failing.
In traditional software, this might just be a busy loop. In LLM applications, it is a financial disaster.
We need to implement a circuit breaker that monitors the semantic progress of the agent, not just the number of steps.
Instead of counting iterations, we calculate the cosine similarity between the current state of the agent's plan and the previous state. If the similarity is above a threshold (e.g., 0.95) for two consecutive steps, we assume the agent is stuck.
import numpy as np
from sentence_transformers import SentenceTransformer
class CircuitBreaker:
def __init__(self, model_name='all-MiniLM-L6-v2', similarity_threshold=0.95, max_steps=10):
self.embedder = SentenceTransformer(model_name)
self.similarity_threshold = similarity_threshold
self.max_steps = max_steps
self.history = []
self.step_count = 0
def record_step(self, plan_text: str) -> bool:
"""
Records the current plan and checks for stagnation.
Returns True if the agent should continue, False if circuit breaks.
"""
self.step_count += 1
if self.step_count > self.max_steps:
print(f"Circuit Breaker: Max steps ({self.max_steps}) exceeded. Forcing termination.")
return False
current_embedding = self.embedder.encode(plan_text)
if len(self.history) > 0:
previous_embedding = self.history[-1]
similarity = np.dot(current_embedding, previous_embedding) / (np.linalg.norm(current_embedding) * np.linalg.norm(previous_embedding))
if similarity > self.similarity_threshold:
if len(self.history) > 1:
prev_prev_embedding = self.history[-2]
prev_sim = np.dot(previous_embedding, prev_prev_embedding) / (np.linalg.norm(previous_embedding) * np.linalg.norm(prev_prev_embedding))
if prev_sim > self.similarity_threshold:
print(f"Circuit Breaker: Semantic stagnation detected. Similarity {similarity:.2f}.")
return False
self.history.append(current_embedding)
return True
When the circuit breaks, you should not simply kill the agent. Instead, you should inject a "Meta-Prompt" that asks the model to reflect on why it is stuck.
"You have repeated the same planning step three times. Please stop. Analyze your previous attempts. Identify the specific constraint or error that is preventing progress, and propose a fundamentally different approach."
This forces the model to shift from execution mode to diagnostic mode, which often uncovers the hidden misunderstanding that was causing the loop.
The biggest source of technical debt in LLM systems is the "Prompt Ladder"—a deeply nested sequence of prompts where each step depends on the output of the previous one, and the context window is stuffed with intermediate reasoning.
Consider this anti-pattern:
This is brittle. If Step 1 is slightly off, the error propagates through every subsequent step. The context window bloats, and the cost scales linearly with depth.
The Refactor: Consolidate into a Single-Reasoner with Tool Loops
Instead of a sequential chain, use a single agent with a loop that has access to tools. The key is to move the "reasoning" out of the prompt chain and into the agent's internal state.
class RefactoredAgent:
def __init__(self, llm_client, tools):
self.llm = llm_client
self.tools = tools
def run(self, user_request):
messages = [
{"role": "system", "content": "You are a coding assistant. Use tools to solve tasks. Reason step-by-step in <thought> tags, then call tools."},
{"role": "user", "content": user_request}
]
while True:
response = self.llm.chat(messages)
if "<tool_call>" in response:
tool_name, args = self.parse_tool_call(response)
result = self.execute_tool(tool_name, args)
messages.append({"role": "assistant", "content": response})
messages.append({"role": "tool", "content": f"Result: {result}"})
else:
return response
Notice the difference:
If you find yourself writing prompts that say "Based on the previous analysis...", you are over-engineering. The LLM is the analysis. Let it handle the context management.
You cannot debug what you cannot see. In traditional backend services, we log requests, responses, and errors. In agent workflows, we need to log the cognitive process.
Create a unified logging schema that captures:
import logging
import json
class AgentLogger:
def __init__(self, session_id):
self.session_id = session_id
self.log_file = f"agents/{session_id}.jsonl"
def log_step(self, step_type, content, metadata=None):
entry = {
"session_id": self.session_id,
"timestamp": datetime.utcnow().isoformat(),
"step_type": step_type, # 'thought', 'tool_call', 'tool_result', 'final'
"content": content,
"metadata": metadata or {}
}
with open(self.log_file, 'a') as f:
f.write(json.dumps(entry) + "\n")
logger.log_step("thought", "I need to query the database for user 123", {"tokens_in": 150, "tokens_out": 40})
logger.log_step("tool_call", "query_db(user_id=123)", {"latency_ms": 45})
By dumping these logs to a central store (like Elasticsearch or Datadog), you can build dashboards that show where agents fail. Do they fail at the planning stage? Do they pick the wrong tool? Do they hallucinate data? This data is invaluable for iterative improvement.
The tension in AI engineering is that LLMs are probabilistic and creative, while software systems are deterministic and strict. The over-engineered workflow attempts to use determinism to control creativity, which results in fragile, costly, and hard-to-debug systems.
The solution is not to remove the LLM, but to remove the fragility from the LLM's responsibilities.
Debugging an AI agent is less like debugging a C++ program and more like debugging a new hire. You need clear instructions, strict validation of their work, and the humility to admit when your instructions were ambiguous. By enforcing these boundaries in code, you build workflows that are not only more reliable but significantly cheaper and faster to maintain.