# Quality Isn't Accidental — Maker/Checker Separation and Automated Validation > Source: > Published: 2026-08-01 03:42:32+00:00 The Core Argument: AI agent reliability isn't achieved by "making the agent smarter" — it's achieved by the simple engineering principle ofseparating validation from generation. Quality isn't accidental. It's designed. What You'll Learn: Maker/Checker separation, 6 termination conditions, and an automated feedback loop — all with runnable code. `pip install openai>=1.0.0` `pip install anthropic>=0.30.0` if using Claude as CheckerA team built a data-analysis agent. It pulled sales data from a database and generated business reports. The team added a "self-review" step: after generating, the agent told itself "please check if the data you just output is accurate." Result? The agent always replied "data is accurate." Even when the team deliberately injected obvious errors (e.g., monthly sales of -50M RMB), the agent confidently said everything was fine. This isn't the model being "disobedient." It's a more fundamental issue: **when the generator and checker are the same entity, the check is just a restatement of the generation process — not real validation.** The checker carries the exact same cognitive bias, knowledge boundaries, and reasoning path as the generator. Self-checking also triggers a subtler problem: confirmation bias amplification. The model builds a "belief state" during generation; when re-examining, it tends to confirm rather than overturn. **Experiment data (from Anthropic research):** **First principle of quality assurance: the checker must be independent of the generator.** In agent architecture, the engineering expression of this is the Maker/Checker separation pattern. *Independence is the first principle of quality — same model 12% correction, different family 52%.* | Level | Description | Best For | |---|---|---| L1: Context separation | Maker & Checker use different system prompts, same model | Low cost, low-risk tasks | L2: Instance separation (recommended) | Different model instances, different temperature; Checker typically lower (0.1) | Most production environments | L3: Model/vendor separation (highest) | Different vendors' different models — e.g., Maker with GPT-4o, Checker with Claude | Maximum diversity, minimal common failure modes | | Checker Type | Validates | Best For | |---|---|---| | Factual consistency | Output matches input/source | Data reports, summaries | | Compliance | Output violates preset rules? | Finance, medical, legal | | Logic | Reasoning chain complete/consistent? | Analysis, decisions | | Format | Output matches expected format? | API responses, structured output | | Safety | Output contains harmful content? | User-facing agents | | Completeness | Task fully done? | Complex workflows | Checker output must be machine-parsable — recommended structured JSON: ``` { "decision": "FAIL", "confidence": 0.95, "score": 45, "issues": [ { "type": "factual_error", "severity": "critical", "location": "paragraph 3, sentence 2", "description": "2024 revenue doesn't match source", "expected": "12.8M", "actual": "18.2M", "rule_reference": "R04-number-consistency" } ] } ``` | Mode | Principle | Best For | |---|---|---| | Max Retry | Hard cap (3-5 attempts) | Simple, predictable | | Quality Threshold | Stop when score ≥ target | Progressive optimization | | Convergence Detection | Stop when 2 outputs ≥95% similar | Avoid invalid retries | | Diminishing Returns | Stop when improvement < threshold | High-quality requirements | | Time Budget | Stop on timeout, protect SLO | Online services | Hybrid (recommended) | Combination of above | Production environments | Here's a runnable implementation. It has two modes: ``` bash #!/usr/bin/env python3 """ maker_checker.py — Maker/Checker separation and automated validation Core: - Maker Agent: generates content - Checker Agent: validates content (multiple checker types) - 6 termination conditions (all runnable) - Feedback loop + constraint escalation - ErrorLog persistence Dependencies: pip install openai>=1.0.0 Test mode (default): no API key needed, mock LLM validates core logic Real mode: export OPENAI_API_KEY=sk-xxx then run """ from __future__ import annotations import json, os, time, hashlib from enum import Enum, auto from pathlib import Path from datetime import datetime, timedelta from dataclasses import dataclass, field from difflib import SequenceMatcher from typing import Optional # ---------- Termination conditions ---------- class TermMode(Enum): MAX_RETRY = auto() QUALITY_THRESHOLD = auto() CONVERGENCE = auto() DIMINISHING = auto() TIME_BUDGET = auto() @dataclass class TermConfig: """Combination of termination conditions""" mode: TermMode = TermMode.HYBRID max_retries: int = 4 quality_threshold: float = 80.0 convergence_similarity: float = 0.95 min_improvement: float = 3.0 time_budget_sec: float = 120.0 # ---------- Checker ---------- @dataclass class CheckResult: decision: str # PASS / FAIL confidence: float score: float issues: list = field(default_factory=list) class BaseChecker: """Base class for all checkers""" def check(self, output: str, context: dict) -> CheckResult: raise NotImplementedError class MockChecker(BaseChecker): """Local test checker — validates without LLM API""" def __init__(self, required_keywords: list, min_length: int = 50): self.required = required_keywords self.min_length = min_length def check(self, output: str, context: dict) -> CheckResult: issues = [] missing = [kw for kw in self.required if kw not in output] if missing: issues.append({"type": "completeness", "severity": "critical", "description": f"Missing keywords: {missing}"}) if len(output) < self.min_length: issues.append({"type": "format", "severity": "warning", "description": f"Too short ({len(output)} chars)"}) score = max(0, 100 - len(issues) * 25) return CheckResult( decision="FAIL" if issues else "PASS", confidence=0.9, score=score, issues=issues, ) # ---------- Maker ---------- class LLMMaker: """Generator agent. In test mode uses mock output.""" def __init__(self, use_mock: bool = True): self.use_mock = use_mock if not use_mock: from openai import OpenAI self.client = OpenAI() def generate(self, task: str, feedback: str = "") -> str: if self.use_mock: return f"Sales report for Q1: revenue 12.8M, growth 23% (with {task[:20]})" system = "You are a report generator. Be accurate and complete." if feedback: system += f"\nPrevious issues: {feedback}\nFix them." resp = self.client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "system", "content": system}, {"role": "user", "content": task}], ) return resp.choices[0].message.content # ---------- The Loop ---------- class MakerCheckerLoop: def __init__(self, maker, checker, term: TermConfig): self.maker = maker self.checker = checker self.term = term self.history = [] def run(self, task: str) -> tuple[str, list]: feedback = "" last_output = "" last_score = 0.0 start = time.time() for attempt in range(1, self.term.max_retries + 1): # Time budget check if time.time() - start > self.term.time_budget_sec: return last_output, self.history + ["⏰ TIME_BUDGET exceeded"] # Maker generates output = self.maker.generate(task, feedback) # Checker validates result = self.checker.check(output, {"task": task}) self.history.append({ "attempt": attempt, "score": result.score, "decision": result.decision, }) # Termination checks if result.decision == "PASS" and result.score >= self.term.quality_threshold: return output, self.history + ["✅ PASS: quality threshold met"] if result.score >= self.term.quality_threshold: return output, self.history + ["✅ PASS: score threshold"] # Convergence detection if last_output and SequenceMatcher(None, last_output, output).ratio() >= self.term.convergence_similarity: return output, self.history + ["⚡ CONVERGED: no improvement"] # Diminishing returns if attempt > 1 and (result.score - last_score) < self.term.min_improvement: return output, self.history + ["🔻 DIMINISHING: minimal gain"] # Build feedback from issues feedback = "; ".join(i["description"] for i in result.issues) last_output = output last_score = result.score time.sleep(0.5) return last_output, self.history + [f"❌ MAX_RETRY ({self.term.max_retries})"] # ---------- Demo ---------- if __name__ == "__main__": # Test mode — no API key needed maker = LLMMaker(use_mock=True) checker = MockChecker(required_keywords=["revenue", "growth"]) loop = MakerCheckerLoop(maker, checker, TermConfig(max_retries=5, quality_threshold=70)) output, log = loop.run("Generate quarterly sales report") print(f"Final output: {output[:80]}...") print(f"Attempts: {[h['attempt'] for h in log[:-1]]}") print(f"Termination: {log[-1]}") ``` **Run it:** ``` python3 maker_checker.py ``` Expected output: ``` Final output: Sales report for Q1: revenue 12.8M, growth 23%... Attempts: [1, 2] Termination: ✅ PASS: quality threshold met ``` **Real mode:** ``` export OPENAI_API_KEY=sk-xxx # Change LLMMaker(use_mock=True) to LLMMaker(use_mock=False) ``` Most people think agents make mistakes because the model isn't smart enough. **This is wrong.** A smarter model lowers the rate of *unknown* errors — but never to zero. Maker/Checker separation eliminates *known* errors by making them structurally impossible to pass. Production systems don't pursue "never making mistakes." They pursue "mistakes get caught and fixed automatically." That's what this framework does. You're no longer the developer who adds "please check your work" to the prompt and hopes for the best. You're becoming an engineer who **builds validation into the architecture** — where the checker is independent, the output is machine-parsed, and the loop terminates by design, not by chance. **Next: DevOps for a one-person company — full observability and alerting for your agent.** *About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.*