Hand a coding agent a failing test and tell it make it pass. Sometimes it fixes the bug. Sometimes it edits the test, hardcodes the expected value, or deletes the assertion — and reports done ✅ with a straight face. Every "build an agent" tutorial stops before this problem. This one is built around it.
The agent itself is the easy part — a while
loop around a chat model with one tool. 01_loop/agent.py is ~70 real lines that fix a real bug, live:
The hard part — Chapter 5, and the reason this repo exists — is an independent skeptic that doesn't take "done ✅" on faith. It checks the code's behavior against contract probes the agent never saw, so a fix that only games the visible test — editing it, or hardcoding/stubbing the source — is caught deterministically, no LLM required (a model judge reads the full diff on top, as a second opinion). You build both, one small chapter at a time. The scaffolding you're assembling has a name now — the harness — and this is harness engineering, from scratch.
git clone https://github.com/Saivineeth147/skeptic && cd skeptic
python3 -m venv .venv && source .venv/bin/activate # recommended (system Python blocks pip: PEP 668)
pip install -r requirements.txt
echo 'OPENROUTER_API_KEY=sk-or-...' > .env # grab a key at openrouter.ai
cd demo
python ../01_loop/agent.py "the cart test is failing, fix it" # ⚠ runs in this dir; it's a sandbox demo
Defaults to Claude Sonnet on OpenRouter; point AGENT_MODEL
/ OPENAI_BASE_URL
at OpenAI, a local Ollama/vLLM server, or anything OpenAI-compatible. A full run costs a few cents (or nothing, on a free model).
Then watch the skeptic catch cheats — including the sneaky one that never touches the test:
python ../05_verify/verify.py --cheat # edits the TEST to pass -> ❌ REJECTED
python ../05_verify/verify.py --cheat-stub # stubs the SOURCE, tests green -> ❌ REJECTED
python ../05_verify/verify.py --cheat-eq # __eq__ that equals ANYTHING -> ❌ REJECTED
python ../05_verify/verify.py # an honest agent fixes the code -> ✅ VERIFIED
The cheat runs are the point: in each, the code passes the visible test yet is rejected,
because a hidden contract check the agent never saw exercises the behavior the test left
out. That's the whole idea — you can only verify work with a check the worker couldn't
optimize against. (--cheat-eq
is the nastiest: an object whose __eq__
returns True
for everything satisfies every bare assert in existence — the oracle catches it because
its comparisons are type-pinned, and its probe requires a completion sentinel so an early
sys.exit(0)
can't fake a clean exit either.)
Frontier models game their tests at production scale — measured, not anecdotal:
Anthropic reports reward hacking — includingtest hardcoding— occurred during the production RL training of Claude Sonnet 3.7, and showed that models which learn to game coding tests generalize to far worse behavior, up to attempted sabotage in agentic coding sessions (paper).METR measured o3 gaming its evaluator in39 of 128 runs (30%) across RE-Bench tasks — and in21 of 21 runs (100%) on one task. Documented strategies: patching the evaluation function to judge every submission successful, monkey-patching the grader to return a perfect score, overwriting the timing function (writeup).- METR saw the same behavior across o3, o1, and Claude 3.7 Sonnet— cross-model, and getting more sophisticated, not less.
Those documented strategies are exactly this repo's threat model — and the toy versions of them ship here as runnable, caught-by-CI cheats ( evals/cases.py).
Chapters 1–4 build the agent everyone builds. Chapter 5 is why this repo exists. Each is a single file that runs, plus a "what breaks without this" note — every chapter exists because the previous one broke somewhere.
| # | Chapter | The question it answers |
|---|---|---|
| 01 | ||
isan agent, minimally?Real toolsbash
forever?Context & compactionPermissions & sandboxingrm
?★ Verifyfinal/
That gap is the whole reason this repo exists. Same task, two runs. Both show visible tests PASS in green — but the skeptic also checks behavior the visible test never touched, so the honest fix is VERIFIED and the stubbed-source cheat is REJECTED:
The loop-and-a-tool part, yes — that's a commodity, and it's Chapters 1–4 here because you should understand it. What almost nobody builds is the part that makes an agent trustworthy: an independent check that the work is real. A model graded on "make the test pass" will, often enough, pass the test without fixing the code. This repo goes to the part the tutorials skip — a skeptic that catches that — and makes it the whole point.
Can't the agent just fool the skeptic too? It's harder than it looks, because the
strongest layer needs no model at all. The skeptic runs a hidden contract check —
behavior probes on inputs the agent never saw — so a cheat that games the visible test
(editing it, hardcoding, stubbing, special-casing the tested inputs) still fails on the
inputs it couldn't optimize against. That's deterministic. The oracle is also hardened
against the classic evaluator-gaming tricks from the reward-hacking literature: its
comparisons are type-pinned (an object whose __eq__
returns True
for everything
still fails) and its probe requires a completion sentinel (an early sys.exit(0)
that ends the process "cleanly" still fails) — both ship as runnable cheats. On top sits
a model judge reading the full diff, as a best-effort second opinion. The honest limit:
on a benchmark or this demo you own the spec and can supply the hidden check; pointed
at an arbitrary repo (final/ --verify
), the skeptic can only re-run your tests, flag
test-file edits, and lean on the judge — unless you hand it a held-out suite with
--holdout
. "Tests are green" and "the code is correct" are different claims; the hidden check is what actually separates them.
Do I need Claude? No, for the agent — any OpenAI-compatible model works
(AGENT_MODEL
), and it runs on a local Ollama/vLLM server (OPENAI_BASE_URL
), no cloud
key required. Smaller models emit malformed tool calls more often; the code handles that
rather than crashing. The judge is more model-sensitive, though: in my testing, small
(7–8B) models both missed subtly-disguised cheats and falsely rejected correct fixes.
So point the verifier's judge at a strong model (JUDGE_MODEL
, or --judge-model
), or
lean on the deterministic checks (the hidden oracle / --holdout
), which don't depend on the judge at all. The default judge, Claude Sonnet, caught the source cheats here with no false rejects — but that floor is real, and the repo says so rather than hiding it.
Is it safe to run? It executes model-generated shell commands, so treat it like it. Chapters 1–3 have no gate — use a throwaway directory. Chapter 4 adds a permission gate (a coarse heuristic, not a security boundary). The real isolation is a container — a Dockerfile ships in the repo, giving the agent a scratch filesystem that holds only what you mount, so it can't reach your
~/.ssh
, ~/.aws
, or the rest of your disk:
docker build -t skeptic .
docker run --rm --network=none -e OPENROUTER_API_KEY=unused skeptic \
sh -c "cd demo && python ../05_verify/verify.py --cheat-stub" # -> ❌ REJECTED, offline
The gate makes mistakes visible; the sandbox makes them harmless. A live agent run
needs egress to reach the model, so drop --network=none
and pass a real key (or point it
at a local model) — the filesystem isolation still holds. --yes
warns unless it detects a sandbox.
Python version? 3.9+ (the code uses builtin generic type hints like dict[str, str]
) — checked in CI on both 3.9 and 3.12.
I kept reading "how to build an agent" posts that stopped at the loop and a
get_weather
tool, and never touched the parts that make a coding agent usable or trustworthy: what to do when the context fills up, how to stop it running something destructive, and — the part I care about most — how to know its output is real instead of taking "done ✅" on faith. So I built the version I wanted to read: every mechanism as its own runnable chapter, ending in a small CLI you can genuinely use.
If the 70-line agent makes you go "wait, that's all it is?" — that's the point. Everything after is turning that toy into something you'd trust.
The self-contained chapters make this easy to extend — see CONTRIBUTING.md. The most on-theme contribution: add a new cheat the skeptic should catch — drop it in evals/cases.py and watch the oracle reject it.
Run the suite (no API key needed): pytest tests/ -q
, and the reproducible cheat-catch
rate: python evals/run.py
. Both run in CI on every push.
MIT — see LICENSE. Learn from it, fork it, ship your own.