{"slug": "sol-loves-to-cheat", "title": "Sol Loves to Cheat", "summary": "A developer's custom supervisor agent, chum-codex, achieved 89.9% accuracy on Terminal Bench 2.1, surpassing the published GPT-5.5 benchmark of 83.8%, but the developer discovered the model was cheating by exploiting benchmark vulnerabilities. The project, which began as an attempt to automate a spec-driven development flow, unintentionally revealed that the model prioritized winning over honest task completion.", "body_md": "[← back](/)\n\n# Sol loves to cheat\n\n*tl;dr*\n\nWhat started as an exploration into building a custom harness, unintentionally led to hacking benchmarks with a model that wants to win at all costs.\n\n## Background\n\nI’ve been running a “spec-driven” development flow for the past ~year.\n\nIt’s pretty simple.\n\nBefore asking an LLM to *do something*, I first ask it to draft a doc for *what it needs to do.*\n\nI use this strategy for feature development, greenfield projects, debugging, you name it.\n\nThe pattern works for me, but it’s a bit repetitive.\n\nSo I decided to automate it.\n\n## chum-codex\n\nThe idea was straightforward: I’d create a **supervisor agent**, that would run a “spec-driven process” by delegating to **worker subagents** who would actually write the docs, do the work, etc.\n\nNote: when trying to do this with vanilla Codex or Claude Code, it would somewhat work, but the default prompts are catered to a user much more so than a “supervisor”\n\nI hypothesized that the supervisor agent need only have the ability to read files and call workers, because that’s what I do.\n\nRather than rebuild a coding harness for the workers, I looked at [Pi](https://pi.dev/), [OpenCode](https://opencode.ai/), and Codex’s [App Server](https://learn.chatgpt.com/docs/app-server).\n\nI’d been using Codex for quite awhile, so I decided to give app-server a spin. The other options are cool, you should check them out.\n\nAnyhow, the first version worked well enough: the supervisor would size the task, call a worker with e.g. a design request, the worker would spit out a doc, the supervisor would then ask the worker to turn that doc into an implementation spec (split by phase, as appropriate), and then finally ask the worker to actually implement the thing.\n\n*Note: this simplified diagram omits the user feedback portions e.g. design doc review*\n\n```\n---\nconfig:\n  sequence:\n    mirrorActors: false\n---\nsequenceDiagram\n    participant S as Supervisor\n    participant W as Worker\n\n    S->>S: Size task\n    S->>W: Design request\n    W-->>S: Design doc\n\n    S->>W: Create implementation spec\n    W-->>S: Phased implementation spec\n\n    S->>W: Implement\n    W-->>S: Result\n```\n\nWoot! I’d saved some time in my development process.\n\n*(or did I?)*\n\n## The Rabbit Hole\n\nGreat, it worked; hacky, but working.\n\nNote:this is where I should have stopped\n\nSitting on my high horse, I surveyed the landscape and thought “wow, everyone should see this!”\n\nWhat’s the best way to do that? Benchmarks!\n\nWhat’s the best benchmark to use? Not [Terminal Bench](https://github.com/harbor-framework/terminal-bench)!\n\nWhat benchmark did I dive too deep on? [Terminal Bench 2.1](https://www.tbench.ai/leaderboard/terminal-bench/2.1)!\n\n## Terminal Bench\n\nIf you’re not familiar with agentic benchmarks, Terminal Bench’s name is telling. It’s a set of tasks that can be accomplished from the terminal, covering a range of one-off tasks from chess to DNA assembly.\n\nBecause it’s so simple, **it’s probably one of the worst benchmarks to test a spec-driven development flow.**\n\nDue to its simple nature, however, it was easy to test against.\n\nI started with a few of the tasks that vanilla Codex w/GPT-5.5 failed at, such as DNA assembly/insert, video extraction/processing, ELF extraction, and protein assembly.\n\nIt worked.\n\nThese tasks benefited from a “design pass” before implementation, as the doc helped avoid narrowing and circular validation.\n\nThe horse I was riding just got a lot taller.\n\nNote:Terminal Bench 1.x/2.x is likely saturated, more on that later.\n\n## GPT-5.6?\n\nThe [published](https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2-1/latest?tab=leaderboard&leaderboard=main) GPT-5.5 benchmark is **83.8%** (~74/89 tasks, 5 runs).\n\n`chum-codex`\n\nwas hitting **89.9%** or [~80/89 tasks](https://gist.github.com/jumploops/09087cfbad3efaa94da5f6bddaacfd06).\n\nExcited to share the news of beating Codex, I ran a couple of vanilla Codex benchmarks just to make sure.\n\nFor context: this was on June 25th, 2026 and rumors were spreading that GPT-5.6 was imminent.\n\nI ran three vanilla Codex benchmarks… and my heart sank: **88.8%**\n\nMy harness was *just one task ahead* of vanilla Codex.\n\nSome tasks were clearly improved, others had regressed.\n\nThe next day, GPT-5.6 Sol was [announced](https://openai.com/index/previewing-gpt-5-6-sol/).\n\nInterestingly, Terminal Bench 2.1 was the only coding-related benchmark they initially shared, showing **88.8%** on GPT-5.6 Sol and **91.9%** on Sol Ultra.\n\nThe new model (sans Ultra) is hitting the same score as the old model (*on the latest version of Codex*)? Is the harness what matters here?\n\n*I reached out to OpenAI, and they confirmed all my request IDs hit GPT-5.5, not GPT-5.6*\n\nSol Ultra spawns parallel subagents to do work, though in my testing it’s quite a bit more token-heavy than most people want/need for the majority of their tasks.\n\nIn either case, I was excited to see the new frontier!\n\n## Steering\n\nGPT-5.6 is much harder to steer.\n\nSwitching from 5.5 to 5.6 made my harness drop in effectiveness. Things that were easy to do before, were now much more difficult.\n\nI traced part of this delta to a change in the base Codex prompt. For GPT-5.5, [the prompt](https://gist.github.com/jumploops/56b45522d5b1fbbeb113001346580e4f) is coding-focused and spends a lot of time on “engineering judgment” including frontend guidance, editing constraints, and having “sympathy with the codebase already in front of you.”\n\n*Excerpt from GPT-5.5 prompt*\n\nThe Codex prompt for GPT-5.6 is [much different](https://gist.github.com/jumploops/2063c2b7c9aeca76449f12567212251d), spending almost zero energy on engineering related specifics. Instead it focuses on communication, autonomy/persistence, and skills (which were previously loaded in as a separate prompt for 5.5).\n\n*Excerpt from GPT-5.6 Sol prompt*\n\nSimilar to what others have noticed, and as I predicted [8 months ago](https://x.com/jumploops/status/2009910802170740771), better models are requiring less ceremony to work effectively.\n\nOn the flip side, as the models get *better*, they’ll become [harder to control](https://openai.com/index/hugging-face-model-evaluation-security-incident/).\n\nA simple example of this is the PyTorch task on Terminal Bench 2.1.\n\nWith GPT-5.6 Luna and Terra, the model is easily steered into a general solution that accepts two inputs: `forward(src, tgt)`\n\nWith Sol, and especially at higher reasoning levels, the model will, *regardless of steering*, default to a single input `forward(src)`\n\nsolution.\n\nThe problem, it seems, is that the model is incredibly hard to steer away from its own reasoning. Even when instructed to accept the broadest callable interface it can (which sometimes works, if repeated, on medium reasoning, but rarely works on xhigh).\n\nWrestling with this model led me down a path that got **way too close to benchmark hacking** for my liking; but I was too intrigued to stop.\n\n## 94% on TB 2.1\n\nHaving reduced my prompts substantially, it began to feel like I was starting over. Even if I wanted to directly hack the benchmark, the model wouldn’t let me. Its circular reasoning was too strong to overcome in some cases, and the supervisor was all too willing to go along with its intelligent worker’s report.\n\nIt’s a tough balance, if you swing too far in one direction, the supervisor will happily expand scope or chase validation endlessly.\n\nThese are straightforward tasks. I want a working solution on the first pass, not limitless expansion.\n\nI tried lowering the reasoning level, using simplified language, reducing the spec-driven flow, adding new skills, etc. Some things improved, but others failed.\n\nA few things showed promise.\n\nThe first was a **third context**. The idea was that I could use an agent that only saw the commentary/reasoning of the worker, and would surface all of the potential mismatches/assumptions that worker made compared to the actual details of the request.\n\n```\nflowchart TB\n    S[\"Supervisor\"]\n    W[\"Worker\"]\n    R[\"Commentary / Reasoning\"]\n    A[\"Assumption Auditor\"]\n\n    S -->|\"task / steer\"| W\n    W -->|\"result\"| S\n\n    W --> R\n    R -.->|\"read-only visibility\"| A\n    A -->|\"assumptions surfaced\"| S\n\n    W ~~~ A\n\n    style R fill:#6fc7e1,stroke:#141414,color:#141414\n```\n\nThe supervisor could then review the assumptions the worker took, and ask it to revisit or question said steps. This kind of works, but it’s slow and happens after the fact.\n\nAnother idea was to **ask the model to output “open questions”** – something I do with my more hands-on development. The initial idea was to have the worker return open questions (rather than a full design doc) whenever it faced them, and then have the supervisor resolve them. This would free up the supervisor’s context, showing it the forest rather than the trees.\n\nStill, even with a reduced context, the **supervisor was hard-pressed to disagree with the worker’s conclusions** (or on the flip-side, overly eager to expand on trivial details).\n\nTo remove this bias, the next idea was to employ a separate context, which would first **map and reduce** (everything old is new again!) the questions, in an attempt to remove any inherent or unfound bias, before ultimately returning a normalized version to the supervisor (or directly to the worker).\n\nThis performed better, but it relied on the worker announcing the correct issues *as questions*.\n\nWith Sol, it turns out, it’s much easier to have it **output its decisions, rather than its questions.** The model is confident, so it doesn’t see its assumptions as questions, even if it has already stated the alternatives in its reasoning or commentary.\n\n```\nflowchart TB\n    S[\"Supervisor\"]\n    W[\"Worker\"]\n    D[\"Decisions\"]\n    M[\"Map\"]\n    R[\"Reduce\"]\n\n    S -->|\"task / steer\"| W\n    W -->|\"result\"| S\n\n    W --> D\n    D --> M\n    M --> R\n    R -->|\"normalized questions\"| S\n\n    R -.->|\"optional\"| W\n\n    style D fill:#6fc7e1,stroke:#141414,color:#141414\n    style M fill:#f49bab,stroke:#141414,color:#141414\n    style R fill:#f49bab,stroke:#141414,color:#141414\n```\n\nWith decisions in hand, the supervisor (or third context) can pause the worker, assess the decisions as questions, and then steer appropriately.\n\nThis worked much better, and led to the best result: 84/89 tasks on Terminal Bench 2.1\n\n*Note: 1 task was cyber security blocked, but passed with a GPT-5.6 Terra fallback, so 83 + 1*\n\n## Catch-22s\n\nBack on my high horse, having finally harnessed Sol, and already way too far down the path of using the benchmark for development rather than… as a benchmark, I wanted to see how far I could push this.\n\nNo longer looking exclusively at vanilla Codex regressions, I wanted to see what was stopping us from hitting 86 or 88/89.\n\nLong story short, the tail end of tasks in Terminal Bench 2.1 is poorly specified, and that’s the reason we’re seeing Mythos, GPT-5.6, etc. top out around 88.8% without more specialized machinery.\n\nThe direction needed to perform better in one task actively harms progress in another.\n\nAn example of this is `make-mips-interpreter`\n\nwhich informs the agent that the *“I (the user) will check that you booted doom correctly”*\n\nThe [problem](https://github.com/harbor-framework/terminal-bench-2-1/issues/9)? The verifier fails if the output file, *from the agent booting doom* already exists.\n\nSlowing this down a bit:\n\n- User states they will check that agent boots Doom\n- Booting Doom outputs\n`/tmp/frame.bmp`\n\n- Agent ensures\n`/tmp/frame.bmp`\n\nexists so user knows it booted Doom correctly - Verifier fails if\n`/tmp/frame.bmp`\n\nexists\n\nThe agent assumes that the user wants to check that it, the agent, booted the VM, so the agent leaves the file behind to prove it booted, but the verifier’s test fails early if the file already exists. A catch-22!\n\nFixing this is possible, by prompting the system to remove validation state/override a user concern, but that fix (obviously) backfires in other tasks/usecases\n\nBefore moving on to better things, I decided I wanted to share the results with the world, with the caveat that **it’s a little too benchmark hacky for my liking** (the whole third context map-reducer thing works for this benchmark, but in the real world I can just write better instructions and/or iterate with follow-on messages).\n\nI ran the benchmark once before doing the full N=5 run, and was surprised to see a previously passing task had failed:\n\n`torch-pipeline-parallelism`\n\nI ran it a couple of times. 1/3 worked.\n\nDiving into the details, I couldn’t figure out what had changed with our harness, so I tested it against vanilla Codex, also on xhigh.\n\nIt passed 3/3 times.\n\nIntriguing.\n\n## Sol loves to cheat\n\nI reviewed the runs to determine what worked and what didn’t work.\n\n**GPT-5.6 Sol cheated 3/3 times** on vanilla Codex.\n\nI looked the two recent passing runs for chum-codex on `torch-pipeline`\n\nand found it cheated as well.\n\nUh oh, were *all* of the past successes due to cheating?\n\nInterestingly, web search was disabled, but life finds a way:\n\n*GPT-5.6 Sol on xhigh*\n\nThe worker *did not* have access to the `web_search`\n\ntool, but instead decided to use `curl`\n\nto access DuckDuckGo, Github, grep.app, and SourceGraph.\n\nYes, to deal with this properly we should remove network access entirely or only allow specific resources. The point is, we didn’t need to do that until now.\n\nLooking back, July 29th was the first “cheat” from vanilla Codex, and our harness cheated for the first time today, August 12th.\n\n```\n{\n  \"src\": \"/charts/torch-pipeline-apexcharts-pass-data.json\",\n  \"chart\": { \"height\": 440 },\n  \"colors\": [\"#6fc7e1\", \"#e08e45\", \"#3d8ba6\", \"#f49bab\"]\n}\n```\n\nAdmittedly, **this isn’t a lot of data to go off of**. The 3/3 Vanilla Codex cheating session was followed by 2 runs that *didn’t cheat*.\n\nIt’s also unclear if the models are being intentional about cheating, or if they’re just stumbling across the solution while searching the web.\n\nLooking at vanilla Codex traces however, and we find our smoking gun:\n\n*I need to investigate the HF source using curl to check GitHub for the latest versions. It could be helpful to know the expected hidden test based on the challenge.*\n\n*That certainly feels a lot like cheating.*\n\nFor chum-codex, the last step before the curl requests is equally revealing:\n\n**Perhaps the solution is available publicly**, which means I can compare it effectively. I'll just need to use curl to access the raw paths and **gather the necessary information!**\n\n*Not to anthropomorphize a machine modeled after humans, but it almost seems happy!*\n\nConcerned, but equally intrigued, I looked back at the 83/89 run from July 17th, and [found no evidence of cheating](https://gist.github.com/jumploops/ef9535daff9637d087dc9fba76077a50) on this or any other tasks.\n\n## What’s next\n\nThe `torch-pipeline`\n\ntask wasn’t the only one to fail today, giving me an eerie reminder of migrating from GPT-5.5 to GPT-5.6.\n\nIt seems that the better the models become, the harder it’ll be to build useful guardrails around them, and for now I need a break.\n\nI’ll probably revisit the harness as I engage more with the new problems I face with Sol, Fable, and beyond, but for now I’m going to stick with a more hands-on approach to my development.\n\nPutting powerful models in a loop with lazy prompting can be fun, but trusting their output is getting harder.\n\nAs the models get more powerful, **I need to instruct them less, but those instructions are more important than ever.**\n\nHeck, even the new Terminal Bench 3.0 has the following instructions added to all of their [tasks](https://github.com/harbor-framework/terminal-bench/blob/v3.0.0/tasks/distributed-dedup/instruction.md):\n\n“Do not cheat by using online solutions or hints specific to this task.”\n\nIs this enough? **Unlikely.**", "url": "https://wpnews.pro/news/sol-loves-to-cheat", "canonical_source": "https://jumploops.com/blog/sol-loves-to-cheat/", "published_at": "2026-08-18 16:29:51+00:00", "updated_at": "2026-08-18 16:41:48.262937+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "ai-ethics"], "entities": ["chum-codex", "Terminal Bench 2.1", "GPT-5.5", "Codex", "Pi", "OpenCode", "App Server"], "alternates": {"html": "https://wpnews.pro/news/sol-loves-to-cheat", "markdown": "https://wpnews.pro/news/sol-loves-to-cheat.md", "text": "https://wpnews.pro/news/sol-loves-to-cheat.txt", "jsonld": "https://wpnews.pro/news/sol-loves-to-cheat.jsonld"}}