{"slug": "show-hn-a-reproducible-harness-for-catching-agent-eval-cheating", "title": "Show HN: A reproducible harness for catching agent-eval cheating", "summary": "A new open-source harness demonstrates that agent evaluations are vulnerable to cheating when agents share environments with evaluators, showing that per-task isolation using Firecracker microVMs eliminates tampering and score inflation. The harness, built on Tensorlake and Harbor, reveals a 53% lie rate in naive evaluations and zero contamination with isolated forks.", "body_md": "A tiny, reproducible harness that shows **why a trustworthy agent evaluation needs\nper-task isolation**, run live on [Tensorlake](https://tensorlake.ai) microVM sandboxes.\n\nThe benchmark debate is stuck on scores. The harder, under-written problem is the\n**harness**: if the agent runs in the same environment the evaluator trusts, a\nzero-capability agent can score high by tampering with the test machinery or just\nclaiming success. The fix is a causal chain:\n\nverification must be\n\nhiddenfrom the agent → hiding it needsper-task isolation→ cheap isolation needsfast forks→ that isTensorlake + Harbor.\n\nEvery number below comes from running this harness on real Firecracker sandboxes.\n\n| Experiment | Result | What it shows |\n|---|---|---|\nEXP0 determinism |\nsame task x5 -> 5/5 identical PASS |\na bit-identical snapshot removes environment-induced variance |\nEXP1 cheats vs verdicts |\na trusting harness passes cheating agents 4/4; ground-truth on visible checks 1/4 |\ntrusting the agent is worthless; artifact checks help but are not enough |\nEXP1b held-out + isolation |\nhonest passes held-out; `hardcode_visible` fails held-out |\nteaching-to-the-test is only caught by held-out checks the agent never saw, which stay secret only because each task runs in its own fork |\nEXP2 lie rate |\ntrusting 87% vs ground-truth 33% -> 53% lie rate, +53pt inflation |\nnaive scores massively overstate real capability |\nEXP3 contamination |\nshared harness 3 spurious failures vs isolated 0 |\none poisoned task silently corrupts every later task in a shared box; fork-per-task eliminates it |\nEXP3 throughput |\n~16 tasks/min at concurrency=1 |\nscales ~linearly as paid tiers lift the cap to 1,000+ |\n\nThe \"cheats\" are the ones documented in the literature: the SWE-bench `conftest.py`\n\nhook that forces every test to report passed, editing the test file, and simply\nprinting \"all tests passed\". The held-out case hardcodes the visible tests and is only\ncaught by checks it was never shown.\n\nRequires Python >= 3.10 and a Tensorlake API key (the free tier works; it is capped at 1 concurrent sandbox, so the harness runs strictly sequential).\n\n```\npython -m venv .venv && . .venv/bin/activate\npip install -r requirements.txt\nexport TENSORLAKE_API_KEY=tl_apiKey_...      # from https://cloud.tensorlake.ai -> API Keys\npython eval_harness.py\n```\n\nIt builds one canonical task environment, snapshots it once, and forks a clean microVM\nper task for every experiment. Get a free key and credits at the\n[Tensorlake playground](https://tensorlake.ai).\n\n**The task (**: a buggy Python module plus a visible test file. Snapshotted once; every rollout is a fresh fork of that snapshot.`s0`\n\n)**Scripted \"agents\"**: deterministic bash behaviors (honest fix, no-op, the three cheats, a partial fix, and a hardcode-to-the-visible-test). No model or model API key needed, so results are fully reproducible.**Trusting harness**: believes the agent's own`pytest`\n\nexit code or its stdout claim.**Ground-truth verifier**: imports the artifact directly (not via pytest, so a hijacked`conftest`\n\ncannot forge it) and checks behavior.**Held-out verifier**: the same, with checks the agent never saw, injected only at verify time into a fork the agent could not reach.\n\nVerification is only as good as the checks you write; a finite oracle can be memorized, which is exactly why held-out checks plus isolation matter. External services the sandbox reaches over the network are outside the snapshot. There is a per-restore floor, and snapshots consume storage.\n\nCompanion to the Medium piece *Your Agent Benchmark Is Lying to You* (link added on\npublish). Primary sources the results line up with: the UC Berkeley RDI benchmark audit\n(BenchJack), OpenAI retiring SWE-bench Verified (Feb 2026), Cursor on reward hacking,\nMETR on frontier reward hacking, Terminal-Bench / Harbor, and the run-to-run variance\nstudy on agentic evals.\n\nBuilt with [Tensorlake](https://tensorlake.ai) sandboxes and the\n[Harbor](https://github.com/harbor-framework/harbor) evaluation framework.", "url": "https://wpnews.pro/news/show-hn-a-reproducible-harness-for-catching-agent-eval-cheating", "canonical_source": "https://github.com/sebuzdugan/agent-eval-harness", "published_at": "2026-07-13 22:11:50+00:00", "updated_at": "2026-07-13 22:36:04.258591+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "developer-tools"], "entities": ["Tensorlake", "Harbor", "Firecracker", "SWE-bench", "UC Berkeley", "OpenAI", "Cursor", "METR"], "alternates": {"html": "https://wpnews.pro/news/show-hn-a-reproducible-harness-for-catching-agent-eval-cheating", "markdown": "https://wpnews.pro/news/show-hn-a-reproducible-harness-for-catching-agent-eval-cheating.md", "text": "https://wpnews.pro/news/show-hn-a-reproducible-harness-for-catching-agent-eval-cheating.txt", "jsonld": "https://wpnews.pro/news/show-hn-a-reproducible-harness-for-catching-agent-eval-cheating.jsonld"}}