{"slug": "building-ai-agents-that-don-t-hallucinate-structured-workflows-guardrails-and", "title": "Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation", "summary": "A developer describes replacing fragile prompt chains with structured workflows, typed schemas, validation gates, and per-step evaluation to build AI agents that don't hallucinate. The approach achieved 94% task success compared to a 60% baseline.", "body_md": "*How we replaced fragile prompt chains with typed schemas, validation gates, and evaluation at every step — 94% task success vs 60% baseline*\n\nJanuary 2024. We built a \"research agent\" — 12 prompts chained together:\n\nIt worked 60% of the time. The other 40%:\n\nDebugging meant reading 12 LLM calls' worth of logs. Adding a step broke three others.\n\nWe moved from **prompt chains** to **structured workflows** with:\n\n```\n┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐\n│  Decompose  │──▶│   Search    │──▶│  Extract    │──▶│ Synthesize  │\n│  Question   │   │   Planning  │   │   Facts     │   │  Answer     │\n│             │   │             │   │             │   │             │\n│ In: Query   │   │ In: Plan    │   │ In: Results │   │ In: Facts   │\n│ Out: SubQ[] │   │ Out: Steps  │   │ Out: Fact[] │   │ Out: Answer │\n└──────┬──────┘   └──────┬──────┘   └──────┬──────┘   └──────┬──────┘\n       │                 │                 │                 │\n       ▼                 ▼                 ▼                 ▼\n  [Schema]          [Schema]           [Schema]           [Schema]\n  [Guardrail]       [Guardrail]        [Guardrail]        [Guardrail]\n  [Eval: 0.9]       [Eval: 0.85]       [Eval: 0.9]        [Eval: 0.95]\npython\n# agent_eval/schemas.py\nfrom pydantic import BaseModel, Field\nfrom typing import Literal, Any\n\nclass DecomposeInput(BaseModel):\n    user_query: str\n    context: dict = Field(default_factory=dict)\n\nclass DecomposeOutput(BaseModel):\n    sub_questions: list[str] = Field(min_length=1, max_length=5)\n    requires_tools: bool\n    reasoning: str\n\nclass PlanInput(BaseModel):\n    sub_questions: list[str]\n    available_tools: list[str]\n\nclass Step(BaseModel):\n    query: str\n    source: Literal[\"web\", \"internal\", \"api\"]\n    priority: int = Field(ge=1, le=3)\n\nclass PlanOutput(BaseModel):\n    steps: list[Step] = Field(min_length=1)\n    estimated_confidence: float = Field(ge=0, le=1)\n\nclass Fact(BaseModel):\n    claim: str\n    evidence: str\n    source_url: str\n    confidence: float = Field(ge=0, le=1)\n\nclass ExtractInput(BaseModel):\n    tool_results: list[Any]\n    original_query: str\n\nclass ExtractOutput(BaseModel):\n    facts: list[Fact] = Field(min_length=1)\n    gaps: list[str] = Field(default_factory=list)\n    confidence: float\n\nclass Citation(BaseModel):\n    text: str\n    source_url: str\n\nclass SynthesizeInput(BaseModel):\n    facts: list[Fact]\n    user_query: str\n    tone: Literal[\"professional\", \"casual\", \"technical\"] = \"professional\"\n\nclass SynthesizeOutput(BaseModel):\n    answer: str\n    citations: list[Citation]\n    confidence: float\n    warnings: list[str] = Field(default_factory=list)\npython\n# agent_eval/agent.py\nclass StructuredAgent:\n    def __init__(self, steps: list[AgentStep], guardrails: list[Guardrail], evaluator: Evaluator):\n        self.steps = steps\n        self.guardrails = guardrails\n        self.evaluator = evaluator\n\n    async def run(self, input: BaseModel) -> AgentResult:\n        context = input.model_dump()\n        step_results = []\n\n        for step in self.steps:\n            # 1. Execute step with structured output extraction\n            output = await self._execute_step(step, context)\n\n            # 2. Validate schema\n            validated = step.output_model.model_validate(output)\n\n            # 3. Run guardrails (blocking)\n            for guardrail in self.guardrails:\n                if not await guardrail.check(validated, context):\n                    return AgentResult(blocked=True, reason=guardrail.violation)\n\n            # 4. Evaluate step quality (non-blocking, for observability)\n            eval_result = await self.evaluator.evaluate_step(step.name, context, validated)\n\n            step_results.append(StepResult(step=step.name, output=validated, eval=eval_result))\n            context.update(validated.model_dump())\n\n        return AgentResult(steps=step_results, final_output=context)\npython\n# agent_eval/guardrails.py\nfrom abc import ABC, abstractmethod\n\nclass Guardrail(ABC):\n    @abstractmethod\n    async def check(self, output: BaseModel, context: dict) -> bool: ...\n\nclass CitationValidator(Guardrail):\n    \"\"\"Every claim in answer must have a citation from retrieved docs.\"\"\"\n    async def check(self, output: SynthesizeOutput, context: dict) -> bool:\n        retrieved_docs = context.get(\"retrieved_docs\", [])\n        doc_text = \" \".join(d.text for d in retrieved_docs)\n\n        for citation in output.citations:\n            if citation.text not in doc_text:\n                return False  # Hallucinated citation\n        return True\n\nclass ConfidenceGate(Guardrail):\n    \"\"\"Block low-confidence outputs.\"\"\"\n    def __init__(self, threshold: float = 0.7):\n        self.threshold = threshold\n\n    async def check(self, output: BaseModel, context: dict) -> bool:\n        return getattr(output, \"confidence\", 1.0) >= self.threshold\n\nclass FormatEnforcer(Guardrail):\n    \"\"\"Ensure structured output matches schema exactly.\"\"\"\n    async def check(self, output: BaseModel, context: dict) -> bool:\n        try:\n            type(output).model_validate(output.model_dump())\n            return True\n        except ValidationError:\n            return False\n\nclass SafetyGuardrail(Guardrail):\n    \"\"\"PII, harmful content, policy violations.\"\"\"\n    def __init__(self):\n        self.detector = instructor.from_openai(AsyncOpenAI())\n\n    async def check(self, output: BaseModel, context: dict) -> bool:\n        class SafetyCheck(BaseModel):\n            safe: bool\n            violations: list[str]\n\n        result = await self.detector.chat.completions.create(\n            model=\"gpt-4o-mini\",\n            response_model=SafetyCheck,\n            messages=[{\n                \"role\": \"user\",\n                \"content\": f\"Check for PII, harmful content, policy violations:\\n{output.model_dump_json()}\"\n            }],\n            temperature=0.0,\n        )\n        return result.safe\npython\n# agent_eval/evaluation.py\nclass StepEvaluator:\n    def __init__(self, judges: list[Judge]):\n        self.judges = judges\n\n    async def evaluate_step(self, step_name: str, input: dict, output: BaseModel) -> StepEvalResult:\n        results = {}\n        for judge in self.judges:\n            if judge.applies_to(step_name):\n                result = await judge.evaluate(input, output)\n                results[judge.name] = result\n\n        return StepEvalResult(step=step_name, judge_results=results)\n\n# Judges per step type\nDECOMPOSE_JUDGES = [\n    LLMJudge(\"completeness\", \"All aspects of query covered?\", threshold=0.8),\n    LLMJudge(\"no_hallucination\", \"Sub-questions answerable from available tools?\", threshold=0.9),\n]\n\nPLAN_JUDGES = [\n    LLMJudge(\"feasibility\", \"Plan executable with available tools?\", threshold=0.85),\n    LLMJudge(\"efficiency\", \"Minimal steps to answer?\", threshold=0.7),\n]\n\nEXTRACT_JUDGES = [\n    LLMJudge(\"faithfulness\", \"Facts supported by tool results?\", threshold=0.9),\n    LLMJudge(\"completeness\", \"All relevant info extracted?\", threshold=0.8),\n]\n\nSYNTHESIZE_JUDGES = [\n    LLMJudge(\"accuracy\", \"Answer matches extracted facts?\", threshold=0.9),\n    LLMJudge(\"citation_quality\", \"Citations precise and relevant?\", threshold=0.85),\n    LLMJudge(\"tone_adherence\", \"Matches requested tone?\", threshold=0.8),\n]\npython\n# agent_eval/structured_output.py\nimport instructor\nfrom openai import AsyncOpenAI\nfrom pydantic import BaseModel, ValidationError\n\nclass StructuredExtractor:\n    def __init__(self, model=\"gpt-4o-mini\", max_retries=3):\n        self.client = instructor.from_openai(AsyncOpenAI())\n        self.model = model\n        self.max_retries = max_retries\n\n    async def extract(self, response_model: type[BaseModel], prompt: str, \n                      system: str = None, context: dict = None) -> BaseModel:\n        messages = []\n        if system:\n            messages.append({\"role\": \"system\", \"content\": system})\n        if context:\n            messages.append({\"role\": \"system\", \"content\": f\"Context:\\n{json.dumps(context)}\"})\n        messages.append({\"role\": \"user\", \"content\": prompt})\n\n        last_error = None\n        for attempt in range(self.max_retries):\n            try:\n                return await self.client.chat.completions.create(\n                    model=self.model,\n                    response_model=response_model,\n                    messages=messages,\n                    temperature=0.0,\n                )\n            except ValidationError as e:\n                last_error = e\n                messages.append({\"role\": \"assistant\", \"content\": f\"Validation failed: {e}\"})\n                messages.append({\"role\": \"user\", \"content\": \"Fix the validation errors. Output ONLY valid JSON.\"})\n\n        raise last_error\n```\n\n| Metric | Prompt Chain | Structured Agent | Improvement |\n|---|---|---|---|\n| Task success rate | 60% | 94% | +34 pp |\n| Format validity | 72% | 99.8% | +27.8 pp |\n| Hallucination rate | 23% | 3% | -20 pp |\n| Avg steps to complete | 4.2 | 2.8 | -33% |\n| Debug time (per failure) | 45 min | 8 min | -82% |\n| CI catch rate (regressions) | 12% | 87% | +75 pp |\n\n| Prompt Chain | Structured Agent |\n|---|---|\n| \"Write a prompt that works\" | \"Define the I/O contract for each step\" |\n| Test on 5 examples | Golden set with 200+ stratified cases |\n| \"Add safety to prompt\" | Guardrail as typed, testable code |\n| Debug by reading logs | Debug by failed step + judge scores |\n| Hope it generalizes | Regression test on every change |\n\nThe upfront cost (schemas, guardrails, eval) pays off at step 3. By step 5 it's mandatory.\n\n```\npip install agent-eval-framework\npython\nfrom agent_eval import StructuredAgent, DecomposeStep, PlanStep, ExtractStep, SynthesizeStep\nfrom agent_eval.guardrails import CitationValidator, ConfidenceGate, SafetyGuardrail\nfrom agent_eval.judges import create_judge_ensemble\n\nagent = StructuredAgent(\n    steps=[\n        DecomposeStep(),\n        PlanStep(),\n        ExtractStep(),\n        SynthesizeStep(),\n    ],\n    guardrails=[\n        CitationValidator(),\n        ConfidenceGate(0.7),\n        SafetyGuardrail(),\n    ],\n    evaluator=StepEvaluator(judges=create_judge_ensemble()),\n)\n\nresult = await agent.run(DecomposeInput(user_query=\"How do I reset my 2FA?\"))\n```\n\nAll MIT licensed:\n\n*Code: github.com/yourname/agent-eval-framework |\nDiscussion: Hacker News |\nFollow: @yourname*", "url": "https://wpnews.pro/news/building-ai-agents-that-don-t-hallucinate-structured-workflows-guardrails-and", "canonical_source": "https://dev.to/imus_d7584cbc8ee9b0336256/building-ai-agents-that-dont-hallucinate-structured-workflows-guardrails-and-per-step-evaluation-5gg1", "published_at": "2026-07-21 09:43:52+00:00", "updated_at": "2026-07-21 10:01:12.887723+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety", "developer-tools", "large-language-models"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/building-ai-agents-that-don-t-hallucinate-structured-workflows-guardrails-and", "markdown": "https://wpnews.pro/news/building-ai-agents-that-don-t-hallucinate-structured-workflows-guardrails-and.md", "text": "https://wpnews.pro/news/building-ai-agents-that-don-t-hallucinate-structured-workflows-guardrails-and.txt", "jsonld": "https://wpnews.pro/news/building-ai-agents-that-don-t-hallucinate-structured-workflows-guardrails-and.jsonld"}}