{"slug": "build-a-self-correcting-ai-agent-with-reflection-and-retry-loops", "title": "Build a Self-Correcting AI Agent with Reflection and Retry Loops", "summary": "Anthropic's Claude agent can be built into a self-correcting AI agent using a Python script that generates code, runs it against hidden tests, and retries with a critic's feedback until tests pass or an attempt cap is reached. The tutorial by Rachel Goldstein uses the Anthropic Python SDK 1.2.0, Pydantic 2.13.5, and the claude-opus-5 model, with a maximum of 4 attempts and a 10-second timeout per check. The agent's success check includes test cases that require handling bare numbers as seconds, uppercase units with padding, and raising ValueError for invalid input.", "body_md": "# Build a Self-Correcting AI Agent with Reflection and Retry Loops\n\nAdd a critic call and a deterministic check so a Claude agent fixes its own output until the tests pass.\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)\n\n## 1. What you'll build\n\nA Python agent that writes a function, runs it against tests it can't see, hands the failures to a separate critic call, and retries with the critique until the tests pass or it hits an attempt cap. Two [Claude](https://platform.claude.com/docs/en/models/overview) calls per iteration, a deterministic check in between, structured outputs on both ends so nothing gets parsed out of Markdown.\n\n## 2. Prerequisites\n\n[Python](https://www.python.org/downloads/)3.10 or newer. The SDK refuses older versions. Verified on 3.13.5.[Anthropic Python SDK](https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/python)1.2.0 (released 2026-08-27). It pulls in[Pydantic](https://docs.pydantic.dev/)2.13.5, which you'll use for the response schemas.- An API key from the\n[Claude Console](https://platform.claude.com/settings/keys), exported as`ANTHROPIC_API_KEY`\n\n. - Model:\n`claude-opus-5`\n\n. Thinking is on by default, and it supports structured outputs and the`effort`\n\nparameter. Expect 1 to 7 calls per run. - Commands are for macOS/Linux. On Windows, activate the venv with\n`.venv\\Scripts\\activate`\n\n.\n\n## 3. Set up the project\n\n```\nmkdir self-correcting-agent && cd self-correcting-agent\npython3 -m venv .venv && source .venv/bin/activate\npip install \"anthropic==1.2.0\"\nexport ANTHROPIC_API_KEY=\"sk-ant-...\"\n```\n\n## 4. Define the task and the success check\n\nCreate `agent.py`\n\n. The first block holds the task the generator sees and the tests it does not. The tests encode three details the task text leaves out: a bare number means seconds, units can be uppercase with padding, and garbage must raise `ValueError`\n\n. That gap is deliberate: in real work the tests are the spec, and the loop exists to close the gap without a human in the middle.\n\n``` php\n\"\"\"Self-correcting agent: generate -> check -> critique -> retry.\"\"\"\n\nimport subprocess\nimport sys\nfrom dataclasses import dataclass\n\nimport anthropic\nfrom pydantic import BaseModel\n\nMODEL = \"claude-opus-5\"\nMAX_ATTEMPTS = 4\nCHECK_TIMEOUT = 10\n\nclient = anthropic.Anthropic()\n\nTASK = \"\"\"Write a Python function parse_duration(s: str) -> int that converts a\nduration string such as \"1h30m\", \"45s\", or \"2h 15m 30s\" into total seconds.\nSupported units are h, m, and s. Whitespace between components is allowed.\nReject invalid input by raising ValueError.\"\"\"\n\n# The success check. The generator never sees it; the critic only sees what failed.\nTEST_CODE = \"\"\"\nimport sys\nCASES = [(\"1h30m\", 5400), (\"45s\", 45), (\"2h 15m 30s\", 8130),\n         (\"90\", 90), (\" 1H 30M \", 5400)]\nfailures = []\nfor s, want in CASES:\n    try:\n        got = parse_duration(s)\n    except Exception as e:\n        got = f\"{type(e).__name__}: {e}\"\n    if got != want:\n        failures.append(f\"parse_duration({s!r}) -> {got!r}, want {want!r}\")\nfor bad in [\"\", \"abc\", \"1x\"]:\n    try:\n        parse_duration(bad)\n        failures.append(f\"parse_duration({bad!r}) returned instead of raising ValueError\")\n    except ValueError:\n        pass\n    except Exception as e:\n        failures.append(f\"parse_duration({bad!r}) raised {type(e).__name__}, want ValueError\")\nprint(\"\\\\n\".join(failures) if failures else \"ALL PASSED\")\nsys.exit(1 if failures else 0)\n\"\"\"\n\nclass Draft(BaseModel):\n    code: str\n    notes: str\n\nclass Critique(BaseModel):\n    root_cause: str\n    fix_plan: list[str]\n\n@dataclass\nclass CheckResult:\n    passed: bool\n    output: str\n\n@dataclass\nclass Attempt:\n    code: str\n    check: CheckResult\n    critique: Critique\n\ndef check(code: str) -> CheckResult:\n    \"\"\"Run the candidate plus the tests in a fresh interpreter.\"\"\"\n    try:\n        proc = subprocess.run(\n            [sys.executable, \"-c\", code + \"\\n\" + TEST_CODE],\n            capture_output=True, text=True, timeout=CHECK_TIMEOUT,\n        )\n    except subprocess.TimeoutExpired:\n        return CheckResult(False, f\"Timed out after {CHECK_TIMEOUT}s\")\n    output = (proc.stdout + proc.stderr).strip()\n    return CheckResult(proc.returncode == 0, output[-3000:])\n```\n\n`check()`\n\nruns the candidate in a subprocess so a syntax error or infinite loop can't take the agent down. Whatever lands on stdout or stderr, tracebacks included, becomes the critic's evidence.\n\n## 5. Write the generator and the critic\n\nAppend the two model calls. Both use `client.messages.parse()`\n\nwith a Pydantic class as `output_format`\n\n; the SDK converts it to `output_config.format`\n\non the wire and hands back a validated instance on `response.parsed_output`\n\n.\n\n```\nGENERATOR_SYSTEM = (\n    \"You write production-quality Python. Put the complete module source in \"\n    \"`code`: plain Python, no Markdown fences, no example usage, no prints.\"\n)\n\nCRITIC_SYSTEM = (\n    \"You are a strict code reviewer. Diagnose why the code failed the check. \"\n    \"Do not rewrite the code. Name the root cause and give minimal, concrete fix steps.\"\n)\n\ndef generate(history: list[Attempt]) -> Draft:\n    prompt = TASK\n    if history:\n        last = history[-1]\n        lessons = \"\\n\".join(f\"- {a.critique.root_cause}\" for a in history)\n        steps = \"\\n\".join(f\"- {s}\" for s in last.critique.fix_plan)\n        prompt += (\n            \"\\n\\nYour previous attempt failed the acceptance check.\\n\\n\"\n            f\"Previous code:\\n{last.code}\\n\\n\"\n            f\"Check output:\\n{last.check.output}\\n\\n\"\n            f\"Reviewer's fix plan:\\n{steps}\\n\\n\"\n            f\"Root causes found so far (do not repeat them):\\n{lessons}\\n\\n\"\n            \"Write a corrected version.\"\n        )\n    response = client.messages.parse(\n        model=MODEL,\n        max_tokens=16000,\n        system=GENERATOR_SYSTEM,\n        messages=[{\"role\": \"user\", \"content\": prompt}],\n        output_format=Draft,\n    )\n    return parsed(response)\n\ndef critique(code: str, check_output: str) -> Critique:\n    response = client.messages.parse(\n        model=MODEL,\n        max_tokens=16000,\n        system=CRITIC_SYSTEM,\n        output_config={\"effort\": \"medium\"},  # short diagnosis; full depth not needed\n        messages=[{\n            \"role\": \"user\",\n            \"content\": f\"Task:\\n{TASK}\\n\\nCode:\\n{code}\\n\\nCheck output:\\n{check_output}\",\n        }],\n        output_format=Critique,\n    )\n    return parsed(response)\n\ndef parsed(response):\n    if response.parsed_output is None:\n        raise RuntimeError(f\"No structured output (stop_reason={response.stop_reason})\")\n    return response.parsed_output\n```\n\nTwo design choices matter. The critic is told not to rewrite the code, so its tokens go into diagnosis instead of a second draft the generator would have to reconcile. The generator gets every root cause found so far, not just the last, so a fix on attempt 3 doesn't reintroduce the bug from attempt 1.\n\n`max_tokens`\n\nis 16000 because thinking tokens count against it; a cap sized for the JSON alone truncates on hard retries. `output_config`\n\nand `output_format`\n\ncoexist: the SDK merges the schema into the config you pass.\n\n## 6. Wire the retry loop\n\n``` php\ndef run() -> str:\n    history: list[Attempt] = []\n    for n in range(1, MAX_ATTEMPTS + 1):\n        draft = generate(history)\n        result = check(draft.code)\n        print(f\"attempt {n}: {'PASS' if result.passed else 'FAIL'}\")\n        if result.passed:\n            return draft.code\n        print(result.output)\n        if n == MAX_ATTEMPTS:\n            break\n        review = critique(draft.code, result.output)\n        print(f\"  root cause: {review.root_cause}\")\n        history.append(Attempt(draft.code, result, review))\n    raise SystemExit(f\"Gave up after {MAX_ATTEMPTS} attempts\")\n\nif __name__ == \"__main__\":\n    code = run()\n    with open(\"parse_duration.py\", \"w\") as f:\n        f.write(code)\n    print(\"wrote parse_duration.py\")\n```\n\nThe cap is the safety valve: without it, a task the model can't solve, or a flaky check, burns tokens forever. Skipping the critique on the final failure saves one call nothing would consume.\n\n## 7. Verify it works\n\n```\npython agent.py\n```\n\nA run where the first draft misses the hidden spec looks like this. The shape is fixed; the exception text and the root-cause line come from the model and will differ:\n\n``` php\nattempt 1: FAIL\nparse_duration('90') -> 'ValueError: 90', want 90\nparse_duration(' 1H 30M ') -> 'ValueError:  1H 30M ', want 5400\n  root cause: Bare numbers and uppercase units are not handled\nattempt 2: PASS\nwrote parse_duration.py\n```\n\n`claude-opus-5`\n\nsometimes infers the hidden cases and passes on attempt 1, which is fine. To force a retry, add a case the text doesn't imply, such as `(\"1.5h\", 5400)`\n\n, to `CASES`\n\n.\n\nConfirm the artifact is usable on its own:\n\n``` python\npython -c \"from parse_duration import parse_duration; print(parse_duration('2h 15m 30s'))\"\n8130\n```\n\n## 8. Troubleshooting\n\n`TypeError: \"Could not resolve authentication method. Expected one of api_key, auth_token, or credentials to be set. ...\"`\n\nThe key isn't in the environment the script runs in. Export `ANTHROPIC_API_KEY`\n\nin the same shell you run `python agent.py`\n\nfrom; activating a venv doesn't carry it over from another terminal.\n\n`anthropic.BadRequestError: ... \"thinking.type.enabled\" is not supported for this model. Use \"thinking.type.adaptive\" and \"output_config.effort\" to control thinking behavior.`\n\nYou added `thinking={\"type\": \"enabled\", \"budget_tokens\": ...}`\n\nfrom an older example. Delete it. Thinking is already on for `claude-opus-5`\n\n; steer depth with `output_config={\"effort\": ...}`\n\ninstead.\n\n`pydantic_core._pydantic_core.ValidationError: 1 validation error for Draft ... Invalid JSON: EOF while parsing a string ... [type=json_invalid`\n\nThe response hit `max_tokens`\n\nmid-JSON. Thinking spends from the same budget as the output, so raise `max_tokens`\n\nor drop the generator to `output_config={\"effort\": \"medium\"}`\n\n.\n\n`anthropic.RateLimitError`\n\nafter a few attempts\nThe SDK already retries 429s twice with backoff. For a bigger `MAX_ATTEMPTS`\n\n, or several agents in parallel, construct the client with `anthropic.Anthropic(max_retries=5)`\n\nso a burst of retries doesn't abort the run.\n\n## 9. Next steps\n\n- Swap\n`TEST_CODE`\n\nfor`pytest`\n\non a real repo: write the draft to a temp file and run the suite as the check. For untrusted tasks, run the check in a container or the[code execution tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/code-execution-tool). `TASK`\n\nand both system prompts repeat on every call. Mark the system prompt with`cache_control`\n\nper the[prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)docs and retries get cheaper.- Anthropic's\n[Building effective agents](https://www.anthropic.com/research/building-effective-agents)calls this the evaluator-optimizer workflow and covers when it beats a single well-prompted call. - With no deterministic check, use a rubric-scoring model call as the evaluator. Prefer the deterministic one; a judge that hallucinates a pass is worse than no loop.\n- For tasks that need tools mid-generation, move the generator onto the SDK's\n[tool runner](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview)and keep`check()`\n\nand`critique()`\n\naround it.\n\n## Sources & further reading\n\n-\n[Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)— platform.claude.com -\n[Effort](https://platform.claude.com/docs/en/build-with-claude/effort)— platform.claude.com -\n[Models overview](https://platform.claude.com/docs/en/models/overview)— platform.claude.com -\n[Python SDK](https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/python)— platform.claude.com -\n[Troubleshooting thinking](https://platform.claude.com/docs/en/build-with-claude/thinking-troubleshooting)— platform.claude.com -\n[anthropic 1.2.0](https://pypi.org/project/anthropic/1.2.0/)— pypi.org\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor\n\nRachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/build-a-self-correcting-ai-agent-with-reflection-and-retry-loops", "canonical_source": "https://sourcefeed.dev/a/build-a-self-correcting-ai-agent-with-reflection-and-retry-loops", "published_at": "2026-08-31 11:42:47+00:00", "updated_at": "2026-08-31 11:52:39.794034+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "developer-tools"], "entities": ["Anthropic", "Claude", "Rachel Goldstein", "Anthropic Python SDK", "Pydantic", "claude-opus-5"], "alternates": {"html": "https://wpnews.pro/news/build-a-self-correcting-ai-agent-with-reflection-and-retry-loops", "markdown": "https://wpnews.pro/news/build-a-self-correcting-ai-agent-with-reflection-and-retry-loops.md", "text": "https://wpnews.pro/news/build-a-self-correcting-ai-agent-with-reflection-and-retry-loops.txt", "jsonld": "https://wpnews.pro/news/build-a-self-correcting-ai-agent-with-reflection-and-retry-loops.jsonld"}}