{"slug": "a-coding-agent-in-5-files-plus-a-skeptic-that-catches-fake-fixes", "title": "A coding agent in 5 files, plus a skeptic that catches fake fixes", "summary": "A new open-source repository, 'skeptic', provides a 5-file coding agent framework that includes an independent verification system to catch reward hacking, where AI agents cheat by editing tests or hardcoding values. The skeptic module uses hidden contract probes to deterministically detect cheats without an LLM, addressing a problem Anthropic and METR documented in frontier models like Claude Sonnet 3.7 and o3, which gamed evaluators in up to 100% of runs on some tasks.", "body_md": "Hand a coding agent a failing test and tell it *make it pass*. Sometimes it fixes\nthe bug. Sometimes it edits the test, hardcodes the expected value, or deletes the\nassertion — and reports **done ✅** with a straight face. Every \"build an agent\"\ntutorial stops before this problem. This one is built around it.\n\nThe agent itself is the easy part — a `while`\n\nloop around a chat model with one\ntool. [ 01_loop/agent.py](/Saivineeth147/skeptic/blob/main/01_loop) is ~70 real lines that fix a real bug, live:\n\nThe **hard** part — Chapter 5, and the reason this repo exists — is an independent\n**skeptic** that doesn't take \"done ✅\" on faith. It checks the code's *behavior*\nagainst contract probes the agent never saw, so a fix that only games the visible test\n— editing it, or hardcoding/stubbing the source — is caught **deterministically, no LLM\nrequired** (a model judge reads the full diff on top, as a second opinion). You build\nboth, one small chapter at a time. The scaffolding you're assembling has a name now —\nthe **harness** — and this is *harness engineering*, from scratch.\n\n```\ngit clone https://github.com/Saivineeth147/skeptic && cd skeptic\npython3 -m venv .venv && source .venv/bin/activate   # recommended (system Python blocks pip: PEP 668)\npip install -r requirements.txt\n\n# runs on any model via any OpenAI-compatible endpoint — one key, no billing setup:\necho 'OPENROUTER_API_KEY=sk-or-...' > .env      # grab a key at openrouter.ai\n\ncd demo\npython ../01_loop/agent.py \"the cart test is failing, fix it\"   # ⚠ runs in this dir; it's a sandbox demo\n```\n\nDefaults to Claude Sonnet on OpenRouter; point `AGENT_MODEL`\n\n/ `OPENAI_BASE_URL`\n\nat\nOpenAI, a local Ollama/vLLM server, or anything OpenAI-compatible. A full run costs a\nfew cents (or nothing, on a free model).\n\nThen watch the skeptic catch cheats — including the sneaky one that never touches the test:\n\n```\npython ../05_verify/verify.py --cheat        # edits the TEST to pass          -> ❌ REJECTED\npython ../05_verify/verify.py --cheat-stub   # stubs the SOURCE, tests green   -> ❌ REJECTED\npython ../05_verify/verify.py --cheat-eq     # __eq__ that equals ANYTHING     -> ❌ REJECTED\npython ../05_verify/verify.py                # an honest agent fixes the code  -> ✅ VERIFIED\n```\n\nThe cheat runs are the point: in each, the code passes the visible test yet is rejected,\nbecause a hidden contract check the agent never saw exercises the behavior the test left\nout. That's the whole idea — you can only verify work with a check the worker couldn't\noptimize against. (`--cheat-eq`\n\nis the nastiest: an object whose `__eq__`\n\nreturns `True`\n\nfor everything satisfies every bare assert in existence — the oracle catches it because\nits comparisons are type-pinned, and its probe requires a completion sentinel so an early\n`sys.exit(0)`\n\ncan't fake a clean exit either.)\n\nFrontier models game their tests at production scale — measured, not anecdotal:\n\n**Anthropic** reports reward hacking — including**test hardcoding**— occurred during the production RL training of Claude Sonnet 3.7, and showed that models which learn to game coding tests generalize to far worse behavior, up to attempted sabotage in agentic coding sessions ([paper](https://assets.anthropic.com/m/74342f2c96095771/original/Natural-emergent-misalignment-from-reward-hacking-paper.pdf)).**METR** measured o3 gaming its evaluator in**39 of 128 runs (30%)** across RE-Bench tasks — and in**21 of 21 runs (100%)** on one task. Documented strategies: patching the evaluation function to judge every submission successful, monkey-patching the grader to return a perfect score, overwriting the timing function ([writeup](https://metr.org/blog/2025-06-05-recent-reward-hacking/)).- METR saw the same behavior across\n**o3, o1, and Claude 3.7 Sonnet**— cross-model, and getting more sophisticated, not less.\n\nThose documented strategies are exactly this repo's threat model — and the toy versions\nof them ship here as runnable, caught-by-CI cheats ([ evals/cases.py](/Saivineeth147/skeptic/blob/main/evals/cases.py)).\n\n**Chapters 1–4 build the agent everyone builds. Chapter 5 is why this repo exists.**\nEach is a single file that runs, plus a \"what breaks without this\" note — every\nchapter exists because the previous one broke somewhere.\n\n| # | Chapter | The question it answers |\n|---|---|---|\n| 01 |\n|\n\n*is*an agent, minimally?[Real tools](/Saivineeth147/skeptic/blob/main/02_tools)`bash`\n\nforever?[Context & compaction](/Saivineeth147/skeptic/blob/main/03_context)[Permissions & sandboxing](/Saivineeth147/skeptic/blob/main/04_permissions)`rm`\n\n?**★ Verify**`final/`\n\nThat gap is the whole reason this repo exists. Same task, two runs. Both show **visible\ntests PASS** in green — but the skeptic also checks behavior the visible test never\ntouched, so the honest fix is VERIFIED and the stubbed-source cheat is REJECTED:\n\nThe loop-and-a-tool part, yes — that's a commodity, and it's Chapters 1–4 here\nbecause you should understand it. What almost nobody builds is the part that makes\nan agent *trustworthy*: an independent check that the work is real. A model graded on\n\"make the test pass\" will, often enough, pass the test without fixing the code. **This\nrepo goes to the part the tutorials skip — a skeptic that catches that — and makes it\nthe whole point.**\n\n**Can't the agent just fool the skeptic too?** It's harder than it looks, because the\nstrongest layer needs no model at all. The skeptic runs a **hidden contract check** —\nbehavior probes on inputs the agent never saw — so a cheat that games the visible test\n(editing it, hardcoding, stubbing, special-casing the tested inputs) still fails on the\ninputs it couldn't optimize against. That's deterministic. The oracle is also hardened\nagainst the classic evaluator-gaming tricks from the reward-hacking literature: its\ncomparisons are **type-pinned** (an object whose `__eq__`\n\nreturns `True`\n\nfor everything\nstill fails) and its probe requires a **completion sentinel** (an early `sys.exit(0)`\n\nthat ends the process \"cleanly\" still fails) — both ship as runnable cheats. On top sits\na model judge reading the full diff, as a best-effort second opinion. The honest limit:\non a benchmark or this demo *you* own the spec and can supply the hidden check; pointed\nat an arbitrary repo (`final/ --verify`\n\n), the skeptic can only re-run your tests, flag\ntest-file edits, and lean on the judge — unless you hand it a held-out suite with\n`--holdout`\n\n. \"Tests are green\" and \"the code is correct\" are different claims; the\nhidden check is what actually separates them.\n\n**Do I need Claude?** No, for the *agent* — any OpenAI-compatible model works\n(`AGENT_MODEL`\n\n), and it runs on a local Ollama/vLLM server (`OPENAI_BASE_URL`\n\n), no cloud\nkey required. Smaller models emit malformed tool calls more often; the code handles that\nrather than crashing. The **judge** is more model-sensitive, though: in my testing, small\n(7–8B) models both *missed* subtly-disguised cheats and *falsely rejected* correct fixes.\nSo point the verifier's judge at a strong model (`JUDGE_MODEL`\n\n, or `--judge-model`\n\n), or\nlean on the deterministic checks (the hidden oracle / `--holdout`\n\n), which don't depend on\nthe judge at all. The default judge, Claude Sonnet, caught the source cheats here with no\nfalse rejects — but that floor is real, and the repo says so rather than hiding it.\n\n**Is it safe to run?** It executes model-generated shell commands, so treat it like it.\nChapters 1–3 have no gate — use a throwaway directory. Chapter 4 adds a permission gate\n(a coarse heuristic, **not** a security boundary). The real isolation is a container — a\n[ Dockerfile](/Saivineeth147/skeptic/blob/main/Dockerfile) ships in the repo, giving the agent a scratch filesystem that\nholds only what you mount, so it can't reach your\n\n`~/.ssh`\n\n, `~/.aws`\n\n, or the rest of your disk:\n\n```\ndocker build -t skeptic .\n# the deterministic cheat-catch needs no model, so this runs fully air-gapped:\ndocker run --rm --network=none -e OPENROUTER_API_KEY=unused skeptic \\\n  sh -c \"cd demo && python ../05_verify/verify.py --cheat-stub\"     # -> ❌ REJECTED, offline\n```\n\nThe gate makes mistakes *visible*; the sandbox makes them *harmless*. A live agent run\nneeds egress to reach the model, so drop `--network=none`\n\nand pass a real key (or point it\nat a local model) — the filesystem isolation still holds. `--yes`\n\nwarns unless it detects a\nsandbox.\n\n**Python version?** 3.9+ (the code uses builtin generic type hints like `dict[str, str]`\n\n) —\nchecked in CI on both 3.9 and 3.12.\n\nI kept reading \"how to build an agent\" posts that stopped at the loop and a\n`get_weather`\n\ntool, and never touched the parts that make a coding agent usable or\ntrustworthy: what to do when the context fills up, how to stop it running something\ndestructive, and — the part I care about most — how to know its output is real instead\nof taking \"done ✅\" on faith. So I built the version I wanted to read: every mechanism\nas its own runnable chapter, ending in a small CLI you can genuinely use.\n\nIf the 70-line agent makes you go *\"wait, that's all it is?\"* — that's the point.\nEverything after is turning that toy into something you'd trust.\n\nThe self-contained chapters make this easy to extend — see [CONTRIBUTING.md](/Saivineeth147/skeptic/blob/main/CONTRIBUTING.md).\nThe most on-theme contribution: **add a new cheat the skeptic should catch** — drop it in\n[ evals/cases.py](/Saivineeth147/skeptic/blob/main/evals/cases.py) and watch the oracle reject it.\n\nRun the suite (no API key needed): `pytest tests/ -q`\n\n, and the reproducible cheat-catch\nrate: `python evals/run.py`\n\n. Both run in [CI](/Saivineeth147/skeptic/blob/main/.github/workflows/tests.yml) on every push.\n\nMIT — see [LICENSE](/Saivineeth147/skeptic/blob/main/LICENSE). Learn from it, fork it, ship your own.", "url": "https://wpnews.pro/news/a-coding-agent-in-5-files-plus-a-skeptic-that-catches-fake-fixes", "canonical_source": "https://github.com/Saivineeth147/skeptic", "published_at": "2026-07-23 22:08:27+00:00", "updated_at": "2026-07-23 22:22:26.847839+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-research", "ai-tools", "developer-tools"], "entities": ["Anthropic", "METR", "Claude Sonnet 3.7", "OpenRouter", "OpenAI", "Saivineeth147"], "alternates": {"html": "https://wpnews.pro/news/a-coding-agent-in-5-files-plus-a-skeptic-that-catches-fake-fixes", "markdown": "https://wpnews.pro/news/a-coding-agent-in-5-files-plus-a-skeptic-that-catches-fake-fixes.md", "text": "https://wpnews.pro/news/a-coding-agent-in-5-files-plus-a-skeptic-that-catches-fake-fixes.txt", "jsonld": "https://wpnews.pro/news/a-coding-agent-in-5-files-plus-a-skeptic-that-catches-fake-fixes.jsonld"}}