Originally published on tamiz.pro.
You've seen it in production: an AI agent confidently fabricates a bank balance that doesn't exist, invents a file path that isn't real, or claims a function succeeded when it silently failed. These aren't user errors or bad prompts — they're the natural output of self-interpreting language models working in complex loops. This isn't just a "hallucination problem" in the colloquial sense. It's a structural engineering failure in how agents observe, reason, and act on their own outputs.
The core issue is that today's dominant agent architecture — the ReAct loop (Reason + Act) — asks a stateless LLM to simultaneously think through a problem, call tools, observe results, and revise its mental model, all within a single streaming context window. The model has no ground truth anchor between turns. Its past actions are just text tokens in its context. It cannot verify whether what it says happened actually happened. This is why agents lie to themselves with alarming consistency.
Understanding the mechanics of agent self-deception is prerequisite to building defenses that actually work. Let's go under the hood.
At the fundamental level, an agent's "memory" of its own prior actions is nothing more than tokens in a context buffer. When an agent calls get_balance(account_id="12345")
and the tool returns {"balance": 94200}
, both the action and the observation are encoded as text. The next turn, the model reads those tokens and generates its next thought. There is no separate, verified execution trace. The model could easily misread its own observation, conflate it with prior observations, or fabricate a new one entirely.
This is especially dangerous in long-horizon tasks where the context window grows to thousands of tokens. The signal-to-noise ratio for the model's own prior actions degrades dramatically. Empirical studies have shown hallucination rates climbing from ~5% in single-turn tool use to 20–40% in multi-turn agent loops exceeding 10 steps.
In a well-designed system, an action produces an observable effect in the world, and the observation is grounded in that effect. In a standard ReAct agent, this coupling is purely textual. The model generates an action string like read_file(path="/data/config.json")
, the runtime executes it, and the result is appended to context. But the model itself has no causal link to the execution. It doesn't know the file was read — it only sees the text that follows its own token.
When the tool response is large, noisy, or ambiguous, the model frequently performs interpretive reconstruction: it summarizes, paraphrases, or even invents what it thinks the response should say, rather than faithfully representing the actual output. This is not a bug in the model's training — it's an inherent property of autoregressive generation trying to compress high-dimensional reality into low-dimensional text.
LLMs are optimized for fluency and plausibility, not truthfulness. Their loss function rewards generating the next token that makes the sequence coherent, not the next token that is factually accurate. When an agent is pressed through multiple reasoning steps, confidence compounds. A model that starts with a mild misreading will generate increasingly confident (but increasingly wrong) subsequent thoughts, because each new token is conditioned on the previous flawed reasoning.
This creates a feedback loop: the agent becomes more certain of its incorrect model as it generates more text around it. The longer the chain of reasoning, the harder it is for the agent to self-correct — not because it lacks the capability, but because the context is dominated by its own authoritative-sounding but fabricated traces.
Sandboxing doesn't prevent an agent from lying — it prevents the lie from causing damage. The key insight is architectural separation: isolate the agent's execution environment so that even a fully hallucinated action sequence can't reach production resources with unmitigated authority.
A sandbox creates a bounded execution context with enforced boundaries:
Without these boundaries, a single hallucinated tool call (e.g., delete_database(database="production")
) executed with broad credentials causes irreversible damage. With proper sandboxing, that same hallucinated call either fails at the policy layer or operates only in an isolated test environment.
Here's a production-grade sandbox pattern using a policy-enforced tool executor:
import json
import hashlib
from datetime import datetime
from contextlib import contextmanager
from typing import Any, Callable
class SandboxPolicy:
def __init__(self, allowed_paths: list[str], allowed_hosts: list[str], max_retries: int = 3):
self.allowed_paths = [p.rstrip('/') for p in allowed_paths]
self.allowed_hosts = allowed_hosts
self.max_retries = max_retries
self.audit_log: list[dict] = []
def resolve_path(self, request_path: str) -> str:
"""Ensure the requested path is within an allowed directory."""
base = self.allowed_paths[0] if self.allowed_paths else "."
resolved = f"{base}/{request_path}"
real = hashlib.sha256(resolved.encode()).hexdigest()[:16]
if not any(resolved.startswith(p) for p in self.allowed_paths):
raise PermissionError(f"Path {request_path} not in allowed sandboxes")
return resolved
def check_host(self, host: str) -> bool:
return any(host.endswith(h) for h in self.allowed_hosts) or host in self.allowed_hosts
def log(self, action: str, tool: str, args: dict, result: Any, timestamp: datetime):
self.audit_log.append({
"timestamp": timestamp.isoformat(),
"action": action,
"tool": tool,
"args": args,
"result_summary": self._summarize(result),
"result_ok": isinstance(result, dict) and result.get("ok", True),
})
def _summarize(self, result: Any) -> str:
if isinstance(result, str):
return result[:200] + ("..." if len(result) > 200 else "")
if isinstance(result, dict):
return json.dumps(result, sort_keys=True)[:200]
return str(result)[:200]
class SandboxToolExecutor:
def __init__(self, policy: SandboxPolicy):
self.policy = policy
self._retry_count: dict[str, int] = {}
@contextmanager
def execute(self, tool_name: str, tool_args: dict, fn: Callable):
"""Execute a tool call within sandbox policy constraints."""
if tool_name not in self._retry_count:
self._retry_count[tool_name] = 0
if self._retry_count[tool_name] >= self.policy.max_retries:
return {
"ok": False,
"error": f"{tool_name}: exceeded max retries ({self.policy.max_retries})",
"policy_enforced": True,
}
safe_args = {}
for key, value in tool_args.items():
if key == "path" or key.endswith("_path"):
safe_args[key] = self.policy.resolve_path(str(value))
elif key == "url" or key.endswith("_url"):
parsed = str(value)
if not self.policy.check_host(parsed.split("://")[-1].split("/")[0]):
return {
"ok": False,
"error": f"Host {parsed} not in allowed list",
"policy_enforced": True,
}
safe_args[key] = parsed
else:
safe_args[key] = value
try:
result = fn(**safe_args)
self._retry_count[tool_name] = 0 # Reset on success
return result
except Exception as e:
self._retry_count[tool_name] += 1
return {
"ok": False,
"error": str(e),
"retry_remaining": self.policy.max_retries - self._retry_count[tool_name],
"policy_enforced": False,
}
def read_file(path: str) -> dict:
with open(path, "r") as f:
return {"ok": True, "content": f.read(), "bytes": len(f.read())}
def write_file(path: str, content: str) -> dict:
with open(path, "w") as f:
f.write(content)
return {"ok": True, "bytes_written": len(content)}
def query_service(url: str, params: dict) -> dict:
return {"ok": True, "data": {"status": "simulated_response"}}
policy = SandboxPolicy(
allowed_paths=["/tmp/agent_workspace"],
allowed_hosts=["api.internal.example.com", "db.internal.example.com"],
max_retries=3,
)
executor = SandboxToolExecutor(policy)
This pattern ensures that every tool call passes through a policy enforcement layer before touching real resources. The critical design choice: the agent never sees the policy decisions directly in its tool results — the sandbox transparently resolves paths and blocks disallowed hosts, returning controlled error responses the model can reason about without escalating to broader permission requests.
It's important to be precise about the boundaries. Sandboxing contains damage but doesn't improve the quality of the agent's reasoning. A sandboxed agent can still hallucinate successfully within its bounded environment — it might confidently invent data, make incorrect API calls that succeed in the sandbox, or build flawed plans based on fabricated observations. The lies are still happening; they're just contained.
If sandboxing is the brake system, audit trails are the dash cam. They don't prevent the accident, but they make it possible to understand exactly what happened, when, and why — which is the foundation for any remediation or model improvement.
A production-grade audit trail for AI agents needs to capture five dimensions:
// TypeScript type definitions for an agent audit trail
interface AuditAction {
id: string;
timestamp: ISO8601;
step: number;
actor: "agent" | "system" | "human";
// The model's stated thought/reasoning
thought: string;
thoughtTokens?: number;
thoughtConfidence?: number; // Log-probability of generated tokens
// Tool interaction
toolCall?: {
name: string;
arguments: Record<string, unknown>;
argumentHash: string; // SHA-256 of serialized args
};
toolResult?: {
content: string;
contentHash: string; // SHA-256 of actual result
durationMs: number;
status: "success" | "error" | "timeout" | "policy_denied";
policyViolation?: string;
};
// Model's interpretation of the result (may differ from ground truth)
modelInterpretation?: string;
interpretationDrift?: boolean; // True if interpretation diverges from raw result
// Context snapshot
contextSnapshot: {
tokenCount: number;
windowStart: number; // Step index
windowEnd: number;
keyPriorActions: Array<{step: number; tool: string; resultSummary: string}>;
};
}
interface AuditReport {
traceId: string;
agentId: string;
startTime: ISO8601;
endTime: ISO8601;
totalSteps: number;
actions: AuditAction[];
validationChecks: ValidationCheck[];
summary: {
hallucinationFlags: number;
policyViolations: number;
interpretationDrifts: number;
totalToolCalls: number;
failedToolCalls: number;
};
}
interface ValidationCheck {
actionId: string;
checkType: "ground_truth_match" | "schema_valid" | "value_range" | "dependency_consistent";
passed: boolean;
expected?: string;
actual?: string;
severity: "critical" | "warning" | "info";
}
The most valuable pattern in an audit trail is interpretation drift detection. Between the raw tool result and the model's stated understanding of that result, there is often a gap. The model might receive a JSON response {"balance": 94200, "currency": "USD"}
but later claim "the balance is ninety-four thousand two hundred dollars" when the actual response was {"balance": 942, "currency": "USD"}
— a 100x error introduced through textual compression and reconstruction.
import hashlib
import json
from difflib import SequenceMatcher
def detect_interpretation_drift(
raw_result: str,
model_interpretation: str,
sensitivity: float = 0.15
) -> dict:
"""Detect when the model's interpretation diverges significantly from raw output."""
raw_hash = hashlib.sha256(raw_result.encode()).hexdigest()
interp_hash = hashlib.sha256(model_interpretation.encode()).hexdigest()
raw_entities = extract_entities(raw_result)
interp_entities = extract_entities(model_interpretation)
mismatches = []n for key in set(raw_entities.keys()) | set(interp_entities.keys()):
raw_val = raw_entities.get(key)
interp_val = interp_entities.get(key)
if raw_val != interp_val:
mismatches.append({
"entity": key,
"raw": raw_val,
"interpreted": interp_val,
})
similarity = SequenceMatcher(None, raw_result, model_interpretation).ratio()
drift_detected = len(mismatches) > 0 or similarity < (1.0 - sensitivity)
return {
"raw_hash": raw_hash,
"interpretation_hash": interp_hash,
"drift_detected": drift_detected,
"similarity_score": round(similarity, 4),
"mismatches": mismatches,
"severity": "critical" if len(mismatches) > 2 else "warning",
}
def extract_entities(text: str) -> dict:
"""Extract key-value entities from structured or semi-structured text."""
entities = {}
try:
parsed = json.loads(text)
if isinstance(parsed, dict):
return parsed
except json.JSONDecodeError:
pass
import re
for match in re.finditer(r'(\w+\w*):\s*([^\n,}]+)', text):
entities[match.group(1).strip()] = match.group(2).strip()
return entities
When drift is detected, the system can flag the action for human review, suppress downstream actions that depend on the drifted interpretation, or trigger a context reset that forces the model to re-read the raw result.
Sandboxing and auditing are defensive — they limit damage and increase visibility. Honest agent design is offensive: it changes the architecture so the agent is structurally less likely to fabricate in the first place.
The ReAct pattern interleaves thinking and acting in a single loop, which means the model's reasoning is contaminated by its own prior actions (which may be hallucinated). A cleaner architecture separates the planning layer from the execution layer:
┌─────────────────────┐ ┌─────────────────────┐
│ Planner (think) │────▶│ Executor (act) │
│ - No tool access │ │ - Tool execution │
│ - Creates plan │ │ - Returns results │
│ - Requests tools │ │ - Immutable record │
└─────────────────────┘ └─────────────────────┘
│ │
│◀────────────────────────┘
│ Verified results only
▼
┌─────────────────────┐
│ Reasoner (reflect) │
│ - Sees only ground │
│ truth results │
│ - Can question plan │
│ - Cannot fabricate │
│ execution details │
└─────────────────────┘
In this three-component design, the Planner generates tool calls based on the current goal but has no way to verify the results. The Executor runs the tools and returns raw, uninterpreted results. The Reasoner receives only the verified results and updates the model's understanding. This separation means the agent can never confuse a planned action with an executed one, and can never misremember a tool result because it never directly observes it — it only reasons about what the Executor reports.
Require the agent to cite its sources for every factual claim. This isn't a prompt engineering trick — it's an architectural constraint enforced by the executor.
class AttributionEnforcedAgent:
def __init__(self, llm_client, executor: SandboxToolExecutor):
self.llm = llm_client
self.executor = executor
self.source_registry: dict[str, str] = {} # claim_id -> source_ref
self.claims: list[dict] = []
def generate_with_attribution(self, query: str, context: list[dict]) -> dict:
"""Generate a response where every factual claim must cite a source."""
prompt = f"""
Answer the following query. For every factual claim, you MUST cite
the source using the format [source:{id}]. If a claim cannot be
sourced from the context, state "UNSUPPORTED" explicitly.
Query: {query}
Available sources:
{self._format_sources(context)}
Rules:
1. Every number, name, date, or factual statement must have a [source:X] citation
2. If you cannot cite a source, write "UNSUPPORTED" instead of guessing
3. Do not fabricate source IDs — only reference sources that exist
4. If multiple sources conflict, state the conflict explicitly
"""
response = self.llm.generate(prompt, temperature=0.1) # Low temp for factual tasks
attributed_claims = self._extract_claims(response)
for claim in attributed_claims:
if claim["citation"] and claim["citation"] not in self.source_registry:
claim["status"] = "UNSUPPORTED"
elif claim["citation"]:
claim["verified"] = self._verify_claim_against_source(claim)
else:
claim["status"] = "NO_CITATION"
return {
"response": response,
"claims": attributed_claims,
"unverified_count": sum(1 for c in attributed_claims if not c.get("verified")),
"supported_count": sum(1 for c in attributed_claims if c.get("verified")),
}
def _verify_claim_against_source(self, claim: dict) -> bool:
"""Cross-reference the claim against the cited source."""
source_id = claim["citation"]
source = self.source_registry.get(source_id, {})
return self._check_claim_consistency(claim["text"], source)
By forcing the agent to declare when it cannot support a claim, you convert the default behavior from "generate plausible text" to "generate only what can be sourced." This dramatically reduces confident fabrication because the model is explicitly rewarded for saying "I don't know" rather than inventing an answer.
For high-stakes actions, require the agent to pass through independent verification gates before the action is executed. This is particularly effective for destructive or irreversible operations.
from enum import Enum
from dataclasses import dataclass
class ActionRiskLevel(Enum):
LOW = "low" # Read-only, reversible
MEDIUM = "medium" # Write operations, user-facing changes
HIGH = "high" # Destructive operations, production changes
CRITICAL = "critical" # Irreversible, wide-impact operations
@dataclass
class VerificationGate:
risk_level: ActionRiskLevel
required_checks: list[str]
approver_required: bool = False
cooldown_seconds: int = 0
VERIFICATION_PATTERNS: dict[str, VerificationGate] = {
"delete": VerificationGate(
risk_level=ActionRiskLevel.CRITICAL,
required_checks=["rollback_available", "owner_confirmed", "dependency_check"],
approver_required=True,
cooldown_seconds=300,
),
"deploy": VerificationGate(
risk_level=ActionRiskLevel.HIGH,
required_checks=["test_passed", "config_validated", "canary_ready"],
approver_required=True,
cooldown_seconds=60,
),
"create_user": VerificationGate(
risk_level=ActionRiskLevel.MEDIUM,
required_checks=["unique_email", "role_valid", "quota_check"],
approver_required=False,
),
"read": VerificationGate(
risk_level=ActionRiskLevel.LOW,
required_checks=["permission_check"],
approver_required=False,
),
}
class AgentOrchestrator:
def __init__(self, agent, verification_policy: dict):
self.agent = agent
self.policy = verification_policy
self.last_action_times: dict[str, datetime] = {}
def execute_with_gates(self, action_type: str, args: dict) -> dict:
gate = self.policy[action_type]
if gate.cooldown_seconds > 0:
last_run = self.last_action_times.get(action_type)
if last_run:
elapsed = (datetime.utcnow() - last_run).total_seconds()
if elapsed < gate.cooldown_seconds:
return {
"status": "throttled",
"reason": f"{action_type} on cooldown ({gate.cooldown_seconds}s)",
"retry_after": gate.cooldown_seconds - elapsed,
}
check_results = {}
for check_name in gate.required_checks:
check_fn = self._resolve_check(check_name)
result = check_fn(args)
check_results[check_name] = result
if not result["passed"]:
return {
"status": "blocked",
"gate": action_type,
"failed_check": check_name,
"reason": result["message"],
"risk_level": gate.risk_level.value,
}
if gate.approver_required:
approval = self._request_approval(action_type, args, check_results)
if not approval["granted"]:
return {
"status": "rejected",
"gate": action_type,
"approver": approval["approver"],
"reason": approval["reason"],
}
self.last_action_times[action_type] = datetime.utcnow()
result = self._run_action(action_type, args)
return {
"status": "executed",
"gate": action_type,
"checks_passed": list(check_results.keys()),
"result": result,
"risk_level": gate.risk_level.value,
}
This gate system ensures that no single agent hallucination can bypass the verification pipeline. Each check is an independent operation — some may call different services, use different models, or require human judgment. The agent's fabricated reasoning about what should happen is irrelevant; the gates validate what actually happens.
Different failure modes require different layers of defense. Here's a decision framework:
| Failure Mode | Primary Defense | Secondary Defense |
|---|---|---|
| Hallucinated tool arguments (wrong path, missing field) | Sandbox policy enforcement | Ground-truth validation gate |
| Misinterpreted tool results | Audit trail with drift detection | Attribution-enforced reasoning |
| Fabricated facts in final output | Source attribution requirement | Multi-step verification |
| Recursive self-reinforcement of errors | Separated planner/executor | Human-in-the-loop gates |
| Credential overreach from hallucinated actions | Sandbox credential scoping | Least-privilege policy engine |
| Long-context memory degradation | Periodic context compaction with verification | Structured memory stores |
The honest agent design isn't about making the model "more truthful" — it's about building systems where truthfulness is structurally enforced rather than probabilistically hoped for. A model that hallucinates 5% of the time in isolation becomes a reliability nightmare when those hallucinations compound across 20 sequential tool calls. The defenses described above don't eliminate hallucination; they make the system robust to it.
Sandboxing contains the blast radius of each individual lie. Audit trails make every lie traceable and analyzable. Honest design patterns (separation of concerns, forced attribution, verification gates) reduce the probability that a lie reaches execution in the first place. Together, they form a defense-in-depth strategy that transforms an inherently unreliable reasoning component into a production-safe system.
The hard truth is that no amount of prompt engineering or fine-tuning will produce an agent that never fabricates. The architecture has to assume fabrication as a baseline condition and build controls around it. That's the difference between hoping your agent tells the truth and engineering a system where honesty is the path of least resistance.
Q: Can I just use better prompts to reduce agent hallucinations?
A: Prompts can reduce the frequency of hallucinations somewhat, but they cannot eliminate the structural cause. A well-prompted agent in a ReAct loop will still misremember tool results, conflate context windows, and generate plausible-looking but fabricated outputs — especially over long horizons. Prompts are a tuning parameter, not an architectural fix. For production systems, combine prompt guidance with the structural defenses above.
Q: How do I balance agent autonomy with verification overhead?
A: Use risk-tiered verification. Low-risk read operations can proceed with minimal gates (permission check only). Medium-risk writes need source attribution and basic validation. High-risk and critical operations require full verification gates including human approval and cooldown periods. This lets agents move quickly on safe operations while adding friction proportional to the potential damage of a hallucination.
Q: Does this approach work with any LLM provider?
A: Yes — these are architectural patterns, not provider-specific techniques. The sandbox executor, audit trail, and verification gates are all implemented at the application layer, independent of whether you're using OpenAI, Anthropic, or open-weight models. The key is that your agent runtime (not the model itself) enforces these constraints. This is actually the recommended approach: keep the model simple and fluent, and put the reliability logic in the surrounding system. Read more about production agent architectures on Tamiz's Insights.
Q: How do I handle the latency overhead of verification gates?
A: Verification gates are typically sub-second for automated checks (schema validation, permission lookups, dependency resolution). Human approval gates are the bottleneck and should only be required for genuinely critical operations. For most agent workflows, the overhead is 10–50ms per gate. The tradeoff is between speed and the cost of a single hallucinated destructive action — and in production, the latter almost always outweighs the former. Consider async verification for non-blocking operations where the agent can continue reasoning while checks complete in parallel.