The Core Argument: AI agent reliability isn't achieved by "making the agent smarter" — it's achieved by the simple engineering principle ofseparating validation from generation. Quality isn't accidental. It's designed.
What You'll Learn: Maker/Checker separation, 6 termination conditions, and an automated feedback loop — all with runnable code.
pip install openai>=1.0.0
pip install anthropic>=0.30.0
if using Claude as CheckerA team built a data-analysis agent. It pulled sales data from a database and generated business reports. The team added a "self-review" step: after generating, the agent told itself "please check if the data you just output is accurate."
Result? The agent always replied "data is accurate." Even when the team deliberately injected obvious errors (e.g., monthly sales of -50M RMB), the agent confidently said everything was fine.
This isn't the model being "disobedient." It's a more fundamental issue: when the generator and checker are the same entity, the check is just a restatement of the generation process — not real validation. The checker carries the exact same cognitive bias, knowledge boundaries, and reasoning path as the generator.
Self-checking also triggers a subtler problem: confirmation bias amplification. The model builds a "belief state" during generation; when re-examining, it tends to confirm rather than overturn.
Experiment data (from Anthropic research):
First principle of quality assurance: the checker must be independent of the generator. In agent architecture, the engineering expression of this is the Maker/Checker separation pattern.
Independence is the first principle of quality — same model 12% correction, different family 52%.
| Level | Description | Best For |
|---|---|---|
| L1: Context separation | ||
| Maker & Checker use different system prompts, same model | Low cost, low-risk tasks | |
| L2: Instance separation (recommended) | ||
| Different model instances, different temperature; Checker typically lower (0.1) | Most production environments | |
| L3: Model/vendor separation (highest) | ||
| Different vendors' different models — e.g., Maker with GPT-4o, Checker with Claude | Maximum diversity, minimal common failure modes |
| Checker Type | Validates | Best For |
|---|---|---|
| Factual consistency | Output matches input/source | Data reports, summaries |
| Compliance | Output violates preset rules? | Finance, medical, legal |
| Logic | Reasoning chain complete/consistent? | Analysis, decisions |
| Format | Output matches expected format? | API responses, structured output |
| Safety | Output contains harmful content? | User-facing agents |
| Completeness | Task fully done? | Complex workflows |
Checker output must be machine-parsable — recommended structured JSON:
{
"decision": "FAIL",
"confidence": 0.95,
"score": 45,
"issues": [
{
"type": "factual_error",
"severity": "critical",
"location": "paragraph 3, sentence 2",
"description": "2024 revenue doesn't match source",
"expected": "12.8M",
"actual": "18.2M",
"rule_reference": "R04-number-consistency"
}
]
}
| Mode | Principle | Best For |
|---|---|---|
| Max Retry | Hard cap (3-5 attempts) | Simple, predictable |
| Quality Threshold | Stop when score ≥ target | Progressive optimization |
| Convergence Detection | Stop when 2 outputs ≥95% similar | Avoid invalid retries |
| Diminishing Returns | Stop when improvement < threshold | High-quality requirements |
| Time Budget | Stop on timeout, protect SLO | Online services |
| Hybrid (recommended) | ||
| Combination of above | Production environments |
Here's a runnable implementation. It has two modes:
#!/usr/bin/env python3
"""
maker_checker.py — Maker/Checker separation and automated validation
Core:
- Maker Agent: generates content
- Checker Agent: validates content (multiple checker types)
- 6 termination conditions (all runnable)
- Feedback loop + constraint escalation
- ErrorLog persistence
Dependencies: pip install openai>=1.0.0
Test mode (default): no API key needed, mock LLM validates core logic
Real mode: export OPENAI_API_KEY=sk-xxx then run
"""
from __future__ import annotations
import json, os, time, hashlib
from enum import Enum, auto
from pathlib import Path
from datetime import datetime, timedelta
from dataclasses import dataclass, field
from difflib import SequenceMatcher
from typing import Optional
class TermMode(Enum):
MAX_RETRY = auto()
QUALITY_THRESHOLD = auto()
CONVERGENCE = auto()
DIMINISHING = auto()
TIME_BUDGET = auto()
@dataclass
class TermConfig:
"""Combination of termination conditions"""
mode: TermMode = TermMode.HYBRID
max_retries: int = 4
quality_threshold: float = 80.0
convergence_similarity: float = 0.95
min_improvement: float = 3.0
time_budget_sec: float = 120.0
@dataclass
class CheckResult:
decision: str # PASS / FAIL
confidence: float
score: float
issues: list = field(default_factory=list)
class BaseChecker:
"""Base class for all checkers"""
def check(self, output: str, context: dict) -> CheckResult:
raise NotImplementedError
class MockChecker(BaseChecker):
"""Local test checker — validates without LLM API"""
def __init__(self, required_keywords: list, min_length: int = 50):
self.required = required_keywords
self.min_length = min_length
def check(self, output: str, context: dict) -> CheckResult:
issues = []
missing = [kw for kw in self.required if kw not in output]
if missing:
issues.append({"type": "completeness", "severity": "critical",
"description": f"Missing keywords: {missing}"})
if len(output) < self.min_length:
issues.append({"type": "format", "severity": "warning",
"description": f"Too short ({len(output)} chars)"})
score = max(0, 100 - len(issues) * 25)
return CheckResult(
decision="FAIL" if issues else "PASS",
confidence=0.9,
score=score,
issues=issues,
)
class LLMMaker:
"""Generator agent. In test mode uses mock output."""
def __init__(self, use_mock: bool = True):
self.use_mock = use_mock
if not use_mock:
from openai import OpenAI
self.client = OpenAI()
def generate(self, task: str, feedback: str = "") -> str:
if self.use_mock:
return f"Sales report for Q1: revenue 12.8M, growth 23% (with {task[:20]})"
system = "You are a report generator. Be accurate and complete."
if feedback:
system += f"\nPrevious issues: {feedback}\nFix them."
resp = self.client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "system", "content": system},
{"role": "user", "content": task}],
)
return resp.choices[0].message.content
class MakerCheckerLoop:
def __init__(self, maker, checker, term: TermConfig):
self.maker = maker
self.checker = checker
self.term = term
self.history = []
def run(self, task: str) -> tuple[str, list]:
feedback = ""
last_output = ""
last_score = 0.0
start = time.time()
for attempt in range(1, self.term.max_retries + 1):
if time.time() - start > self.term.time_budget_sec:
return last_output, self.history + ["⏰ TIME_BUDGET exceeded"]
output = self.maker.generate(task, feedback)
result = self.checker.check(output, {"task": task})
self.history.append({
"attempt": attempt,
"score": result.score,
"decision": result.decision,
})
if result.decision == "PASS" and result.score >= self.term.quality_threshold:
return output, self.history + ["✅ PASS: quality threshold met"]
if result.score >= self.term.quality_threshold:
return output, self.history + ["✅ PASS: score threshold"]
if last_output and SequenceMatcher(None, last_output, output).ratio() >= self.term.convergence_similarity:
return output, self.history + ["⚡ CONVERGED: no improvement"]
if attempt > 1 and (result.score - last_score) < self.term.min_improvement:
return output, self.history + ["🔻 DIMINISHING: minimal gain"]
feedback = "; ".join(i["description"] for i in result.issues)
last_output = output
last_score = result.score
time.sleep(0.5)
return last_output, self.history + [f"❌ MAX_RETRY ({self.term.max_retries})"]
if __name__ == "__main__":
maker = LLMMaker(use_mock=True)
checker = MockChecker(required_keywords=["revenue", "growth"])
loop = MakerCheckerLoop(maker, checker, TermConfig(max_retries=5, quality_threshold=70))
output, log = loop.run("Generate quarterly sales report")
print(f"Final output: {output[:80]}...")
print(f"Attempts: {[h['attempt'] for h in log[:-1]]}")
print(f"Termination: {log[-1]}")
Run it:
python3 maker_checker.py
Expected output:
Final output: Sales report for Q1: revenue 12.8M, growth 23%...
Attempts: [1, 2]
Termination: ✅ PASS: quality threshold met
Real mode:
export OPENAI_API_KEY=sk-xxx
Most people think agents make mistakes because the model isn't smart enough. This is wrong.
A smarter model lowers the rate of unknown errors — but never to zero. Maker/Checker separation eliminates known errors by making them structurally impossible to pass.
Production systems don't pursue "never making mistakes." They pursue "mistakes get caught and fixed automatically." That's what this framework does.
You're no longer the developer who adds "please check your work" to the prompt and hopes for the best. You're becoming an engineer who builds validation into the architecture — where the checker is independent, the output is machine-parsed, and the loop terminates by design, not by chance.
Next: DevOps for a one-person company — full observability and alerting for your agent.
About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.