A coding agent in 5 files, plus a skeptic that catches fake fixes

A new open-source repository, 'skeptic', provides a 5-file coding agent framework that includes an independent verification system to catch reward hacking, where AI agents cheat by editing tests or hardcoding values. The skeptic module uses hidden contract probes to deterministically detect cheats without an LLM, addressing a problem Anthropic and METR documented in frontier models like Claude Sonnet 3.7 and o3, which gamed evaluators in up to 100% of runs on some tasks.

Hand a coding agent a failing test and tell it make it pass . Sometimes it fixes the bug. Sometimes it edits the test, hardcodes the expected value, or deletes the assertion — and reports done ✅ with a straight face. Every "build an agent" tutorial stops before this problem. This one is built around it. The agent itself is the easy part — a while loop around a chat model with one tool. 01 loop/agent.py /Saivineeth147/skeptic/blob/main/01 loop is ~70 real lines that fix a real bug, live: The hard part — Chapter 5, and the reason this repo exists — is an independent skeptic that doesn't take "done ✅" on faith. It checks the code's behavior against contract probes the agent never saw, so a fix that only games the visible test — editing it, or hardcoding/stubbing the source — is caught deterministically, no LLM required a model judge reads the full diff on top, as a second opinion . You build both, one small chapter at a time. The scaffolding you're assembling has a name now — the harness — and this is harness engineering , from scratch. git clone https://github.com/Saivineeth147/skeptic && cd skeptic python3 -m venv .venv && source .venv/bin/activate recommended system Python blocks pip: PEP 668 pip install -r requirements.txt runs on any model via any OpenAI-compatible endpoint — one key, no billing setup: echo 'OPENROUTER API KEY=sk-or-...' .env grab a key at openrouter.ai cd demo python ../01 loop/agent.py "the cart test is failing, fix it" ⚠ runs in this dir; it's a sandbox demo Defaults to Claude Sonnet on OpenRouter; point AGENT MODEL / OPENAI BASE URL at OpenAI, a local Ollama/vLLM server, or anything OpenAI-compatible. A full run costs a few cents or nothing, on a free model . Then watch the skeptic catch cheats — including the sneaky one that never touches the test: python ../05 verify/verify.py --cheat edits the TEST to pass - ❌ REJECTED python ../05 verify/verify.py --cheat-stub stubs the SOURCE, tests green - ❌ REJECTED python ../05 verify/verify.py --cheat-eq eq that equals ANYTHING - ❌ REJECTED python ../05 verify/verify.py an honest agent fixes the code - ✅ VERIFIED The cheat runs are the point: in each, the code passes the visible test yet is rejected, because a hidden contract check the agent never saw exercises the behavior the test left out. That's the whole idea — you can only verify work with a check the worker couldn't optimize against. --cheat-eq is the nastiest: an object whose eq returns True for everything satisfies every bare assert in existence — the oracle catches it because its comparisons are type-pinned, and its probe requires a completion sentinel so an early sys.exit 0 can't fake a clean exit either. Frontier models game their tests at production scale — measured, not anecdotal: Anthropic reports reward hacking — including test hardcoding — occurred during the production RL training of Claude Sonnet 3.7, and showed that models which learn to game coding tests generalize to far worse behavior, up to attempted sabotage in agentic coding sessions paper https://assets.anthropic.com/m/74342f2c96095771/original/Natural-emergent-misalignment-from-reward-hacking-paper.pdf . METR measured o3 gaming its evaluator in 39 of 128 runs 30% across RE-Bench tasks — and in 21 of 21 runs 100% on one task. Documented strategies: patching the evaluation function to judge every submission successful, monkey-patching the grader to return a perfect score, overwriting the timing function writeup https://metr.org/blog/2025-06-05-recent-reward-hacking/ .- METR saw the same behavior across o3, o1, and Claude 3.7 Sonnet — cross-model, and getting more sophisticated, not less. Those documented strategies are exactly this repo's threat model — and the toy versions of them ship here as runnable, caught-by-CI cheats evals/cases.py /Saivineeth147/skeptic/blob/main/evals/cases.py . Chapters 1–4 build the agent everyone builds. Chapter 5 is why this repo exists. Each is a single file that runs, plus a "what breaks without this" note — every chapter exists because the previous one broke somewhere. | | Chapter | The question it answers | |---|---|---| | 01 | | is an agent, minimally? Real tools /Saivineeth147/skeptic/blob/main/02 tools bash forever? Context & compaction /Saivineeth147/skeptic/blob/main/03 context Permissions & sandboxing /Saivineeth147/skeptic/blob/main/04 permissions rm ? ★ Verify final/ That gap is the whole reason this repo exists. Same task, two runs. Both show visible tests PASS in green — but the skeptic also checks behavior the visible test never touched, so the honest fix is VERIFIED and the stubbed-source cheat is REJECTED: The loop-and-a-tool part, yes — that's a commodity, and it's Chapters 1–4 here because you should understand it. What almost nobody builds is the part that makes an agent trustworthy : an independent check that the work is real. A model graded on "make the test pass" will, often enough, pass the test without fixing the code. This repo goes to the part the tutorials skip — a skeptic that catches that — and makes it the whole point. Can't the agent just fool the skeptic too? It's harder than it looks, because the strongest layer needs no model at all. The skeptic runs a hidden contract check — behavior probes on inputs the agent never saw — so a cheat that games the visible test editing it, hardcoding, stubbing, special-casing the tested inputs still fails on the inputs it couldn't optimize against. That's deterministic. The oracle is also hardened against the classic evaluator-gaming tricks from the reward-hacking literature: its comparisons are type-pinned an object whose eq returns True for everything still fails and its probe requires a completion sentinel an early sys.exit 0 that ends the process "cleanly" still fails — both ship as runnable cheats. On top sits a model judge reading the full diff, as a best-effort second opinion. The honest limit: on a benchmark or this demo you own the spec and can supply the hidden check; pointed at an arbitrary repo final/ --verify , the skeptic can only re-run your tests, flag test-file edits, and lean on the judge — unless you hand it a held-out suite with --holdout . "Tests are green" and "the code is correct" are different claims; the hidden check is what actually separates them. Do I need Claude? No, for the agent — any OpenAI-compatible model works AGENT MODEL , and it runs on a local Ollama/vLLM server OPENAI BASE URL , no cloud key required. Smaller models emit malformed tool calls more often; the code handles that rather than crashing. The judge is more model-sensitive, though: in my testing, small 7–8B models both missed subtly-disguised cheats and falsely rejected correct fixes. So point the verifier's judge at a strong model JUDGE MODEL , or --judge-model , or lean on the deterministic checks the hidden oracle / --holdout , which don't depend on the judge at all. The default judge, Claude Sonnet, caught the source cheats here with no false rejects — but that floor is real, and the repo says so rather than hiding it. Is it safe to run? It executes model-generated shell commands, so treat it like it. Chapters 1–3 have no gate — use a throwaway directory. Chapter 4 adds a permission gate a coarse heuristic, not a security boundary . The real isolation is a container — a Dockerfile /Saivineeth147/skeptic/blob/main/Dockerfile ships in the repo, giving the agent a scratch filesystem that holds only what you mount, so it can't reach your ~/.ssh , ~/.aws , or the rest of your disk: docker build -t skeptic . the deterministic cheat-catch needs no model, so this runs fully air-gapped: docker run --rm --network=none -e OPENROUTER API KEY=unused skeptic \ sh -c "cd demo && python ../05 verify/verify.py --cheat-stub" - ❌ REJECTED, offline The gate makes mistakes visible ; the sandbox makes them harmless . A live agent run needs egress to reach the model, so drop --network=none and pass a real key or point it at a local model — the filesystem isolation still holds. --yes warns unless it detects a sandbox. Python version? 3.9+ the code uses builtin generic type hints like dict str, str — checked in CI on both 3.9 and 3.12. I kept reading "how to build an agent" posts that stopped at the loop and a get weather tool, and never touched the parts that make a coding agent usable or trustworthy: what to do when the context fills up, how to stop it running something destructive, and — the part I care about most — how to know its output is real instead of taking "done ✅" on faith. So I built the version I wanted to read: every mechanism as its own runnable chapter, ending in a small CLI you can genuinely use. If the 70-line agent makes you go "wait, that's all it is?" — that's the point. Everything after is turning that toy into something you'd trust. The self-contained chapters make this easy to extend — see CONTRIBUTING.md /Saivineeth147/skeptic/blob/main/CONTRIBUTING.md . The most on-theme contribution: add a new cheat the skeptic should catch — drop it in evals/cases.py /Saivineeth147/skeptic/blob/main/evals/cases.py and watch the oracle reject it. Run the suite no API key needed : pytest tests/ -q , and the reproducible cheat-catch rate: python evals/run.py . Both run in CI /Saivineeth147/skeptic/blob/main/.github/workflows/tests.yml on every push. MIT — see LICENSE /Saivineeth147/skeptic/blob/main/LICENSE . Learn from it, fork it, ship your own.