Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation

A developer describes replacing fragile prompt chains with structured workflows, typed schemas, validation gates, and per-step evaluation to build AI agents that don't hallucinate. The approach achieved 94% task success compared to a 60% baseline.

How we replaced fragile prompt chains with typed schemas, validation gates, and evaluation at every step — 94% task success vs 60% baseline January 2024. We built a "research agent" — 12 prompts chained together: It worked 60% of the time. The other 40%: Debugging meant reading 12 LLM calls' worth of logs. Adding a step broke three others. We moved from prompt chains to structured workflows with: ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Decompose │──▶│ Search │──▶│ Extract │──▶│ Synthesize │ │ Question │ │ Planning │ │ Facts │ │ Answer │ │ │ │ │ │ │ │ │ │ In: Query │ │ In: Plan │ │ In: Results │ │ In: Facts │ │ Out: SubQ │ │ Out: Steps │ │ Out: Fact │ │ Out: Answer │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ ▼ ▼ ▼ ▼ Schema Schema Schema Schema Guardrail Guardrail Guardrail Guardrail Eval: 0.9 Eval: 0.85 Eval: 0.9 Eval: 0.95 python agent eval/schemas.py from pydantic import BaseModel, Field from typing import Literal, Any class DecomposeInput BaseModel : user query: str context: dict = Field default factory=dict class DecomposeOutput BaseModel : sub questions: list str = Field min length=1, max length=5 requires tools: bool reasoning: str class PlanInput BaseModel : sub questions: list str available tools: list str class Step BaseModel : query: str source: Literal "web", "internal", "api" priority: int = Field ge=1, le=3 class PlanOutput BaseModel : steps: list Step = Field min length=1 estimated confidence: float = Field ge=0, le=1 class Fact BaseModel : claim: str evidence: str source url: str confidence: float = Field ge=0, le=1 class ExtractInput BaseModel : tool results: list Any original query: str class ExtractOutput BaseModel : facts: list Fact = Field min length=1 gaps: list str = Field default factory=list confidence: float class Citation BaseModel : text: str source url: str class SynthesizeInput BaseModel : facts: list Fact user query: str tone: Literal "professional", "casual", "technical" = "professional" class SynthesizeOutput BaseModel : answer: str citations: list Citation confidence: float warnings: list str = Field default factory=list python agent eval/agent.py class StructuredAgent: def init self, steps: list AgentStep , guardrails: list Guardrail , evaluator: Evaluator : self.steps = steps self.guardrails = guardrails self.evaluator = evaluator async def run self, input: BaseModel - AgentResult: context = input.model dump step results = for step in self.steps: 1. Execute step with structured output extraction output = await self. execute step step, context 2. Validate schema validated = step.output model.model validate output 3. Run guardrails blocking for guardrail in self.guardrails: if not await guardrail.check validated, context : return AgentResult blocked=True, reason=guardrail.violation 4. Evaluate step quality non-blocking, for observability eval result = await self.evaluator.evaluate step step.name, context, validated step results.append StepResult step=step.name, output=validated, eval=eval result context.update validated.model dump return AgentResult steps=step results, final output=context python agent eval/guardrails.py from abc import ABC, abstractmethod class Guardrail ABC : @abstractmethod async def check self, output: BaseModel, context: dict - bool: ... class CitationValidator Guardrail : """Every claim in answer must have a citation from retrieved docs.""" async def check self, output: SynthesizeOutput, context: dict - bool: retrieved docs = context.get "retrieved docs", doc text = " ".join d.text for d in retrieved docs for citation in output.citations: if citation.text not in doc text: return False Hallucinated citation return True class ConfidenceGate Guardrail : """Block low-confidence outputs.""" def init self, threshold: float = 0.7 : self.threshold = threshold async def check self, output: BaseModel, context: dict - bool: return getattr output, "confidence", 1.0 = self.threshold class FormatEnforcer Guardrail : """Ensure structured output matches schema exactly.""" async def check self, output: BaseModel, context: dict - bool: try: type output .model validate output.model dump return True except ValidationError: return False class SafetyGuardrail Guardrail : """PII, harmful content, policy violations.""" def init self : self.detector = instructor.from openai AsyncOpenAI async def check self, output: BaseModel, context: dict - bool: class SafetyCheck BaseModel : safe: bool violations: list str result = await self.detector.chat.completions.create model="gpt-4o-mini", response model=SafetyCheck, messages= { "role": "user", "content": f"Check for PII, harmful content, policy violations:\n{output.model dump json }" } , temperature=0.0, return result.safe python agent eval/evaluation.py class StepEvaluator: def init self, judges: list Judge : self.judges = judges async def evaluate step self, step name: str, input: dict, output: BaseModel - StepEvalResult: results = {} for judge in self.judges: if judge.applies to step name : result = await judge.evaluate input, output results judge.name = result return StepEvalResult step=step name, judge results=results Judges per step type DECOMPOSE JUDGES = LLMJudge "completeness", "All aspects of query covered?", threshold=0.8 , LLMJudge "no hallucination", "Sub-questions answerable from available tools?", threshold=0.9 , PLAN JUDGES = LLMJudge "feasibility", "Plan executable with available tools?", threshold=0.85 , LLMJudge "efficiency", "Minimal steps to answer?", threshold=0.7 , EXTRACT JUDGES = LLMJudge "faithfulness", "Facts supported by tool results?", threshold=0.9 , LLMJudge "completeness", "All relevant info extracted?", threshold=0.8 , SYNTHESIZE JUDGES = LLMJudge "accuracy", "Answer matches extracted facts?", threshold=0.9 , LLMJudge "citation quality", "Citations precise and relevant?", threshold=0.85 , LLMJudge "tone adherence", "Matches requested tone?", threshold=0.8 , python agent eval/structured output.py import instructor from openai import AsyncOpenAI from pydantic import BaseModel, ValidationError class StructuredExtractor: def init self, model="gpt-4o-mini", max retries=3 : self.client = instructor.from openai AsyncOpenAI self.model = model self.max retries = max retries async def extract self, response model: type BaseModel , prompt: str, system: str = None, context: dict = None - BaseModel: messages = if system: messages.append {"role": "system", "content": system} if context: messages.append {"role": "system", "content": f"Context:\n{json.dumps context }"} messages.append {"role": "user", "content": prompt} last error = None for attempt in range self.max retries : try: return await self.client.chat.completions.create model=self.model, response model=response model, messages=messages, temperature=0.0, except ValidationError as e: last error = e messages.append {"role": "assistant", "content": f"Validation failed: {e}"} messages.append {"role": "user", "content": "Fix the validation errors. Output ONLY valid JSON."} raise last error | Metric | Prompt Chain | Structured Agent | Improvement | |---|---|---|---| | Task success rate | 60% | 94% | +34 pp | | Format validity | 72% | 99.8% | +27.8 pp | | Hallucination rate | 23% | 3% | -20 pp | | Avg steps to complete | 4.2 | 2.8 | -33% | | Debug time per failure | 45 min | 8 min | -82% | | CI catch rate regressions | 12% | 87% | +75 pp | | Prompt Chain | Structured Agent | |---|---| | "Write a prompt that works" | "Define the I/O contract for each step" | | Test on 5 examples | Golden set with 200+ stratified cases | | "Add safety to prompt" | Guardrail as typed, testable code | | Debug by reading logs | Debug by failed step + judge scores | | Hope it generalizes | Regression test on every change | The upfront cost schemas, guardrails, eval pays off at step 3. By step 5 it's mandatory. pip install agent-eval-framework python from agent eval import StructuredAgent, DecomposeStep, PlanStep, ExtractStep, SynthesizeStep from agent eval.guardrails import CitationValidator, ConfidenceGate, SafetyGuardrail from agent eval.judges import create judge ensemble agent = StructuredAgent steps= DecomposeStep , PlanStep , ExtractStep , SynthesizeStep , , guardrails= CitationValidator , ConfidenceGate 0.7 , SafetyGuardrail , , evaluator=StepEvaluator judges=create judge ensemble , result = await agent.run DecomposeInput user query="How do I reset my 2FA?" All MIT licensed: Code: github.com/yourname/agent-eval-framework | Discussion: Hacker News | Follow: @yourname