Quality Isn't Accidental — Maker/Checker Separation and Automated Validation

A team built a data-analysis agent that generated business reports but found that self-review failed to catch obvious errors, such as negative monthly sales, because the generator and checker were the same entity. They implemented a Maker/Checker separation pattern, using independent checkers and automated validation with structured JSON output and multiple termination conditions, achieving a 52% correction rate with different model families compared to 12% with the same model.

The Core Argument: AI agent reliability isn't achieved by "making the agent smarter" — it's achieved by the simple engineering principle ofseparating validation from generation. Quality isn't accidental. It's designed. What You'll Learn: Maker/Checker separation, 6 termination conditions, and an automated feedback loop — all with runnable code. pip install openai =1.0.0 pip install anthropic =0.30.0 if using Claude as CheckerA team built a data-analysis agent. It pulled sales data from a database and generated business reports. The team added a "self-review" step: after generating, the agent told itself "please check if the data you just output is accurate." Result? The agent always replied "data is accurate." Even when the team deliberately injected obvious errors e.g., monthly sales of -50M RMB , the agent confidently said everything was fine. This isn't the model being "disobedient." It's a more fundamental issue: when the generator and checker are the same entity, the check is just a restatement of the generation process — not real validation. The checker carries the exact same cognitive bias, knowledge boundaries, and reasoning path as the generator. Self-checking also triggers a subtler problem: confirmation bias amplification. The model builds a "belief state" during generation; when re-examining, it tends to confirm rather than overturn. Experiment data from Anthropic research : First principle of quality assurance: the checker must be independent of the generator. In agent architecture, the engineering expression of this is the Maker/Checker separation pattern. Independence is the first principle of quality — same model 12% correction, different family 52%. | Level | Description | Best For | |---|---|---| L1: Context separation | Maker & Checker use different system prompts, same model | Low cost, low-risk tasks | L2: Instance separation recommended | Different model instances, different temperature; Checker typically lower 0.1 | Most production environments | L3: Model/vendor separation highest | Different vendors' different models — e.g., Maker with GPT-4o, Checker with Claude | Maximum diversity, minimal common failure modes | | Checker Type | Validates | Best For | |---|---|---| | Factual consistency | Output matches input/source | Data reports, summaries | | Compliance | Output violates preset rules? | Finance, medical, legal | | Logic | Reasoning chain complete/consistent? | Analysis, decisions | | Format | Output matches expected format? | API responses, structured output | | Safety | Output contains harmful content? | User-facing agents | | Completeness | Task fully done? | Complex workflows | Checker output must be machine-parsable — recommended structured JSON: { "decision": "FAIL", "confidence": 0.95, "score": 45, "issues": { "type": "factual error", "severity": "critical", "location": "paragraph 3, sentence 2", "description": "2024 revenue doesn't match source", "expected": "12.8M", "actual": "18.2M", "rule reference": "R04-number-consistency" } } | Mode | Principle | Best For | |---|---|---| | Max Retry | Hard cap 3-5 attempts | Simple, predictable | | Quality Threshold | Stop when score ≥ target | Progressive optimization | | Convergence Detection | Stop when 2 outputs ≥95% similar | Avoid invalid retries | | Diminishing Returns | Stop when improvement < threshold | High-quality requirements | | Time Budget | Stop on timeout, protect SLO | Online services | Hybrid recommended | Combination of above | Production environments | Here's a runnable implementation. It has two modes: bash /usr/bin/env python3 """ maker checker.py — Maker/Checker separation and automated validation Core: - Maker Agent: generates content - Checker Agent: validates content multiple checker types - 6 termination conditions all runnable - Feedback loop + constraint escalation - ErrorLog persistence Dependencies: pip install openai =1.0.0 Test mode default : no API key needed, mock LLM validates core logic Real mode: export OPENAI API KEY=sk-xxx then run """ from future import annotations import json, os, time, hashlib from enum import Enum, auto from pathlib import Path from datetime import datetime, timedelta from dataclasses import dataclass, field from difflib import SequenceMatcher from typing import Optional ---------- Termination conditions ---------- class TermMode Enum : MAX RETRY = auto QUALITY THRESHOLD = auto CONVERGENCE = auto DIMINISHING = auto TIME BUDGET = auto @dataclass class TermConfig: """Combination of termination conditions""" mode: TermMode = TermMode.HYBRID max retries: int = 4 quality threshold: float = 80.0 convergence similarity: float = 0.95 min improvement: float = 3.0 time budget sec: float = 120.0 ---------- Checker ---------- @dataclass class CheckResult: decision: str PASS / FAIL confidence: float score: float issues: list = field default factory=list class BaseChecker: """Base class for all checkers""" def check self, output: str, context: dict - CheckResult: raise NotImplementedError class MockChecker BaseChecker : """Local test checker — validates without LLM API""" def init self, required keywords: list, min length: int = 50 : self.required = required keywords self.min length = min length def check self, output: str, context: dict - CheckResult: issues = missing = kw for kw in self.required if kw not in output if missing: issues.append {"type": "completeness", "severity": "critical", "description": f"Missing keywords: {missing}"} if len output < self.min length: issues.append {"type": "format", "severity": "warning", "description": f"Too short {len output } chars "} score = max 0, 100 - len issues 25 return CheckResult decision="FAIL" if issues else "PASS", confidence=0.9, score=score, issues=issues, ---------- Maker ---------- class LLMMaker: """Generator agent. In test mode uses mock output.""" def init self, use mock: bool = True : self.use mock = use mock if not use mock: from openai import OpenAI self.client = OpenAI def generate self, task: str, feedback: str = "" - str: if self.use mock: return f"Sales report for Q1: revenue 12.8M, growth 23% with {task :20 } " system = "You are a report generator. Be accurate and complete." if feedback: system += f"\nPrevious issues: {feedback}\nFix them." resp = self.client.chat.completions.create model="gpt-4o-mini", messages= {"role": "system", "content": system}, {"role": "user", "content": task} , return resp.choices 0 .message.content ---------- The Loop ---------- class MakerCheckerLoop: def init self, maker, checker, term: TermConfig : self.maker = maker self.checker = checker self.term = term self.history = def run self, task: str - tuple str, list : feedback = "" last output = "" last score = 0.0 start = time.time for attempt in range 1, self.term.max retries + 1 : Time budget check if time.time - start self.term.time budget sec: return last output, self.history + "⏰ TIME BUDGET exceeded" Maker generates output = self.maker.generate task, feedback Checker validates result = self.checker.check output, {"task": task} self.history.append { "attempt": attempt, "score": result.score, "decision": result.decision, } Termination checks if result.decision == "PASS" and result.score = self.term.quality threshold: return output, self.history + "✅ PASS: quality threshold met" if result.score = self.term.quality threshold: return output, self.history + "✅ PASS: score threshold" Convergence detection if last output and SequenceMatcher None, last output, output .ratio = self.term.convergence similarity: return output, self.history + "⚡ CONVERGED: no improvement" Diminishing returns if attempt 1 and result.score - last score < self.term.min improvement: return output, self.history + "🔻 DIMINISHING: minimal gain" Build feedback from issues feedback = "; ".join i "description" for i in result.issues last output = output last score = result.score time.sleep 0.5 return last output, self.history + f"❌ MAX RETRY {self.term.max retries} " ---------- Demo ---------- if name == " main ": Test mode — no API key needed maker = LLMMaker use mock=True checker = MockChecker required keywords= "revenue", "growth" loop = MakerCheckerLoop maker, checker, TermConfig max retries=5, quality threshold=70 output, log = loop.run "Generate quarterly sales report" print f"Final output: {output :80 }..." print f"Attempts: { h 'attempt' for h in log :-1 }" print f"Termination: {log -1 }" Run it: python3 maker checker.py Expected output: Final output: Sales report for Q1: revenue 12.8M, growth 23%... Attempts: 1, 2 Termination: ✅ PASS: quality threshold met Real mode: export OPENAI API KEY=sk-xxx Change LLMMaker use mock=True to LLMMaker use mock=False Most people think agents make mistakes because the model isn't smart enough. This is wrong. A smarter model lowers the rate of unknown errors — but never to zero. Maker/Checker separation eliminates known errors by making them structurally impossible to pass. Production systems don't pursue "never making mistakes." They pursue "mistakes get caught and fixed automatically." That's what this framework does. You're no longer the developer who adds "please check your work" to the prompt and hopes for the best. You're becoming an engineer who builds validation into the architecture — where the checker is independent, the output is machine-parsed, and the loop terminates by design, not by chance. Next: DevOps for a one-person company — full observability and alerting for your agent. About the author: Wu Ji 无记 — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.