{"slug": "quality-isn-t-accidental-maker-checker-separation-and-automated-validation", "title": "Quality Isn't Accidental — Maker/Checker Separation and Automated Validation", "summary": "A team built a data-analysis agent that generated business reports but found that self-review failed to catch obvious errors, such as negative monthly sales, because the generator and checker were the same entity. They implemented a Maker/Checker separation pattern, using independent checkers and automated validation with structured JSON output and multiple termination conditions, achieving a 52% correction rate with different model families compared to 12% with the same model.", "body_md": "The Core Argument: AI agent reliability isn't achieved by \"making the agent smarter\" — it's achieved by the simple engineering principle ofseparating validation from generation. Quality isn't accidental. It's designed.\n\nWhat You'll Learn: Maker/Checker separation, 6 termination conditions, and an automated feedback loop — all with runnable code.\n\n`pip install openai>=1.0.0`\n\n`pip install anthropic>=0.30.0`\n\nif using Claude as CheckerA team built a data-analysis agent. It pulled sales data from a database and generated business reports. The team added a \"self-review\" step: after generating, the agent told itself \"please check if the data you just output is accurate.\"\n\nResult? The agent always replied \"data is accurate.\" Even when the team deliberately injected obvious errors (e.g., monthly sales of -50M RMB), the agent confidently said everything was fine.\n\nThis isn't the model being \"disobedient.\" It's a more fundamental issue: **when the generator and checker are the same entity, the check is just a restatement of the generation process — not real validation.** The checker carries the exact same cognitive bias, knowledge boundaries, and reasoning path as the generator.\n\nSelf-checking also triggers a subtler problem: confirmation bias amplification. The model builds a \"belief state\" during generation; when re-examining, it tends to confirm rather than overturn.\n\n**Experiment data (from Anthropic research):**\n\n**First principle of quality assurance: the checker must be independent of the generator.** In agent architecture, the engineering expression of this is the Maker/Checker separation pattern.\n\n*Independence is the first principle of quality — same model 12% correction, different family 52%.*\n\n| Level | Description | Best For |\n|---|---|---|\nL1: Context separation |\nMaker & Checker use different system prompts, same model | Low cost, low-risk tasks |\nL2: Instance separation (recommended) |\nDifferent model instances, different temperature; Checker typically lower (0.1) | Most production environments |\nL3: Model/vendor separation (highest) |\nDifferent vendors' different models — e.g., Maker with GPT-4o, Checker with Claude | Maximum diversity, minimal common failure modes |\n\n| Checker Type | Validates | Best For |\n|---|---|---|\n| Factual consistency | Output matches input/source | Data reports, summaries |\n| Compliance | Output violates preset rules? | Finance, medical, legal |\n| Logic | Reasoning chain complete/consistent? | Analysis, decisions |\n| Format | Output matches expected format? | API responses, structured output |\n| Safety | Output contains harmful content? | User-facing agents |\n| Completeness | Task fully done? | Complex workflows |\n\nChecker output must be machine-parsable — recommended structured JSON:\n\n```\n{\n  \"decision\": \"FAIL\",\n  \"confidence\": 0.95,\n  \"score\": 45,\n  \"issues\": [\n    {\n      \"type\": \"factual_error\",\n      \"severity\": \"critical\",\n      \"location\": \"paragraph 3, sentence 2\",\n      \"description\": \"2024 revenue doesn't match source\",\n      \"expected\": \"12.8M\",\n      \"actual\": \"18.2M\",\n      \"rule_reference\": \"R04-number-consistency\"\n    }\n  ]\n}\n```\n\n| Mode | Principle | Best For |\n|---|---|---|\n| Max Retry | Hard cap (3-5 attempts) | Simple, predictable |\n| Quality Threshold | Stop when score ≥ target | Progressive optimization |\n| Convergence Detection | Stop when 2 outputs ≥95% similar | Avoid invalid retries |\n| Diminishing Returns | Stop when improvement < threshold | High-quality requirements |\n| Time Budget | Stop on timeout, protect SLO | Online services |\nHybrid (recommended) |\nCombination of above | Production environments |\n\nHere's a runnable implementation. It has two modes:\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"\nmaker_checker.py — Maker/Checker separation and automated validation\nCore:\n  - Maker Agent: generates content\n  - Checker Agent: validates content (multiple checker types)\n  - 6 termination conditions (all runnable)\n  - Feedback loop + constraint escalation\n  - ErrorLog persistence\n\nDependencies: pip install openai>=1.0.0\nTest mode (default): no API key needed, mock LLM validates core logic\nReal mode: export OPENAI_API_KEY=sk-xxx then run\n\"\"\"\n\nfrom __future__ import annotations\nimport json, os, time, hashlib\nfrom enum import Enum, auto\nfrom pathlib import Path\nfrom datetime import datetime, timedelta\nfrom dataclasses import dataclass, field\nfrom difflib import SequenceMatcher\nfrom typing import Optional\n\n# ---------- Termination conditions ----------\nclass TermMode(Enum):\n    MAX_RETRY = auto()\n    QUALITY_THRESHOLD = auto()\n    CONVERGENCE = auto()\n    DIMINISHING = auto()\n    TIME_BUDGET = auto()\n\n@dataclass\nclass TermConfig:\n    \"\"\"Combination of termination conditions\"\"\"\n    mode: TermMode = TermMode.HYBRID\n    max_retries: int = 4\n    quality_threshold: float = 80.0\n    convergence_similarity: float = 0.95\n    min_improvement: float = 3.0\n    time_budget_sec: float = 120.0\n\n# ---------- Checker ----------\n@dataclass\nclass CheckResult:\n    decision: str          # PASS / FAIL\n    confidence: float\n    score: float\n    issues: list = field(default_factory=list)\n\nclass BaseChecker:\n    \"\"\"Base class for all checkers\"\"\"\n    def check(self, output: str, context: dict) -> CheckResult:\n        raise NotImplementedError\n\nclass MockChecker(BaseChecker):\n    \"\"\"Local test checker — validates without LLM API\"\"\"\n    def __init__(self, required_keywords: list, min_length: int = 50):\n        self.required = required_keywords\n        self.min_length = min_length\n\n    def check(self, output: str, context: dict) -> CheckResult:\n        issues = []\n        missing = [kw for kw in self.required if kw not in output]\n        if missing:\n            issues.append({\"type\": \"completeness\", \"severity\": \"critical\",\n                          \"description\": f\"Missing keywords: {missing}\"})\n        if len(output) < self.min_length:\n            issues.append({\"type\": \"format\", \"severity\": \"warning\",\n                          \"description\": f\"Too short ({len(output)} chars)\"})\n        score = max(0, 100 - len(issues) * 25)\n        return CheckResult(\n            decision=\"FAIL\" if issues else \"PASS\",\n            confidence=0.9,\n            score=score,\n            issues=issues,\n        )\n\n# ---------- Maker ----------\nclass LLMMaker:\n    \"\"\"Generator agent. In test mode uses mock output.\"\"\"\n    def __init__(self, use_mock: bool = True):\n        self.use_mock = use_mock\n        if not use_mock:\n            from openai import OpenAI\n            self.client = OpenAI()\n\n    def generate(self, task: str, feedback: str = \"\") -> str:\n        if self.use_mock:\n            return f\"Sales report for Q1: revenue 12.8M, growth 23% (with {task[:20]})\"\n        system = \"You are a report generator. Be accurate and complete.\"\n        if feedback:\n            system += f\"\\nPrevious issues: {feedback}\\nFix them.\"\n        resp = self.client.chat.completions.create(\n            model=\"gpt-4o-mini\",\n            messages=[{\"role\": \"system\", \"content\": system},\n                      {\"role\": \"user\", \"content\": task}],\n        )\n        return resp.choices[0].message.content\n\n# ---------- The Loop ----------\nclass MakerCheckerLoop:\n    def __init__(self, maker, checker, term: TermConfig):\n        self.maker = maker\n        self.checker = checker\n        self.term = term\n        self.history = []\n\n    def run(self, task: str) -> tuple[str, list]:\n        feedback = \"\"\n        last_output = \"\"\n        last_score = 0.0\n        start = time.time()\n\n        for attempt in range(1, self.term.max_retries + 1):\n            # Time budget check\n            if time.time() - start > self.term.time_budget_sec:\n                return last_output, self.history + [\"⏰ TIME_BUDGET exceeded\"]\n\n            # Maker generates\n            output = self.maker.generate(task, feedback)\n            # Checker validates\n            result = self.checker.check(output, {\"task\": task})\n\n            self.history.append({\n                \"attempt\": attempt,\n                \"score\": result.score,\n                \"decision\": result.decision,\n            })\n\n            # Termination checks\n            if result.decision == \"PASS\" and result.score >= self.term.quality_threshold:\n                return output, self.history + [\"✅ PASS: quality threshold met\"]\n\n            if result.score >= self.term.quality_threshold:\n                return output, self.history + [\"✅ PASS: score threshold\"]\n\n            # Convergence detection\n            if last_output and SequenceMatcher(None, last_output, output).ratio() >= self.term.convergence_similarity:\n                return output, self.history + [\"⚡ CONVERGED: no improvement\"]\n\n            # Diminishing returns\n            if attempt > 1 and (result.score - last_score) < self.term.min_improvement:\n                return output, self.history + [\"🔻 DIMINISHING: minimal gain\"]\n\n            # Build feedback from issues\n            feedback = \"; \".join(i[\"description\"] for i in result.issues)\n            last_output = output\n            last_score = result.score\n            time.sleep(0.5)\n\n        return last_output, self.history + [f\"❌ MAX_RETRY ({self.term.max_retries})\"]\n\n# ---------- Demo ----------\nif __name__ == \"__main__\":\n    # Test mode — no API key needed\n    maker = LLMMaker(use_mock=True)\n    checker = MockChecker(required_keywords=[\"revenue\", \"growth\"])\n    loop = MakerCheckerLoop(maker, checker, TermConfig(max_retries=5, quality_threshold=70))\n\n    output, log = loop.run(\"Generate quarterly sales report\")\n    print(f\"Final output: {output[:80]}...\")\n    print(f\"Attempts: {[h['attempt'] for h in log[:-1]]}\")\n    print(f\"Termination: {log[-1]}\")\n```\n\n**Run it:**\n\n```\npython3 maker_checker.py\n```\n\nExpected output:\n\n```\nFinal output: Sales report for Q1: revenue 12.8M, growth 23%...\nAttempts: [1, 2]\nTermination: ✅ PASS: quality threshold met\n```\n\n**Real mode:**\n\n```\nexport OPENAI_API_KEY=sk-xxx\n# Change LLMMaker(use_mock=True) to LLMMaker(use_mock=False)\n```\n\nMost people think agents make mistakes because the model isn't smart enough. **This is wrong.**\n\nA smarter model lowers the rate of *unknown* errors — but never to zero. Maker/Checker separation eliminates *known* errors by making them structurally impossible to pass.\n\nProduction systems don't pursue \"never making mistakes.\" They pursue \"mistakes get caught and fixed automatically.\" That's what this framework does.\n\nYou're no longer the developer who adds \"please check your work\" to the prompt and hopes for the best. You're becoming an engineer who **builds validation into the architecture** — where the checker is independent, the output is machine-parsed, and the loop terminates by design, not by chance.\n\n**Next: DevOps for a one-person company — full observability and alerting for your agent.**\n\n*About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.*", "url": "https://wpnews.pro/news/quality-isn-t-accidental-maker-checker-separation-and-automated-validation", "canonical_source": "https://dev.to/weiwuji/quality-isnt-accidental-makerchecker-separation-and-automated-validation-plc", "published_at": "2026-08-01 03:42:32+00:00", "updated_at": "2026-08-01 04:07:33.567952+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-research", "developer-tools"], "entities": ["OpenAI", "Anthropic", "GPT-4o", "Claude"], "alternates": {"html": "https://wpnews.pro/news/quality-isn-t-accidental-maker-checker-separation-and-automated-validation", "markdown": "https://wpnews.pro/news/quality-isn-t-accidental-maker-checker-separation-and-automated-validation.md", "text": "https://wpnews.pro/news/quality-isn-t-accidental-maker-checker-separation-and-automated-validation.txt", "jsonld": "https://wpnews.pro/news/quality-isn-t-accidental-maker-checker-separation-and-automated-validation.jsonld"}}