I ported my coding-agent benchmark to Kaggle, and the first bugs I found were mine

A developer ported six cli-bench coding-agent tasks to Kaggle Benchmarks and found that two of the tasks were themselves broken, accounting for 5 of the 13 failed trials in an earlier leaderboard run. The sec/patch-xss task could not be passed by a correct html.escape fix because its tests banned the words 'alert' and 'onerror' while its probe required the escaped payload text, and the data/log-analysis task's question sheet defined error_rate differently from the verifier. The Kaggle versions fix both by checking for raw markup instead of words and by stating the rule the verifier actually checks.

This is a submission for the Kaggle Benchmarking Challenge https://dev.to/challenges/kaggle-2026-09-23 I maintain cli-bench https://github.com/arjunkshah12345-hash/cli-bench , an open benchmark for coding agents that work in a terminal. Each task is a small repository with a bug to fix, a feature to build, or a function to speed up, plus a verifier script that decides pass or fail. Nothing is graded by another model. Either the tests pass and the extra checks hold, or the run fails. For this challenge I ported six cli-bench tasks to Kaggle Benchmarks: | Kaggle task | What the model has to do | What the verifier checks | |---|---|---| | cb-debug-wrong-answer | Find the bug in a small stats library | The shipped pytest suite passes | | cb-refactor-deadcode | Delete the three unused functions and nothing else | Dead functions gone, six live ones still defined, tests pass | | cb-feature-rate-limiter | Write a thread-safe token bucket from an interface spec | Tests for burst, refill, atomic rollback, and 100 threads at once | | cb-perf-hot-loop | Make a pair counter at least 10x faster in pure Python | Stdlib only, same signature, randomized equivalence, 10x on uniform, clustered, and gridded points | | cb-data-log-analysis | Answer seven exact questions about an application log it never sees | The model writes solve.py ; the task runs it and compares every answer to the log | | cb-sec-patch-xss | Close an XSS hole in a comment board | Exploit tests pass and a fresh payload renders as text | In cli-bench, an agent gets a shell and a time budget. On Kaggle the model gets one shot: the prompt holds every file in the repo, and the model answers with whole files in FILE: path blocks. The task writes those files into a temporary directory and runs the original cli-bench verifier gates on the result. A run passes only if every gate passes. If the reply rewrites a test file or an input, that write is thrown away and the run fails, the same way cli-bench treats sabotage. So the question this benchmark asks is: how much of a coding agent's score comes from the model reading code carefully, and how much comes from the agent loop of running tests and trying again? I already had one agent run on the full suite Codex with gpt-5.6-luna, 23 of 36 trials passed . The Kaggle version takes the loop away. Porting meant reading every verifier line by line, and checking each task with a reference solution and a few wrong ones. Two tasks did not hold up. sec/patch-xss could not be passed by a correct fix. Two shipped tests assert that the words alert and onerror appear nowhere in the rendered page. The verifier's own probe then requires the escaped payload text, including alert , to still be in the page. Escaping the comment with html.escape is the textbook fix, and it fails the tests, because <script alert ... still contains the word alert . Deleting the text passes the tests and fails the probe. In the Codex run, two of the three trials used html.escape and failed the tests, and the third stripped the text and failed the probe. I had written those failures up as the model's fault. data/log-analysis described one rule and graded another. The question sheet defined error rate as the "fraction of lines with level ERROR". The verifier divides ERROR request lines by request lines, and about one log line in seven is not a request. Both of the Codex trials that failed this task were off on error rate and nothing else 0.0434 vs 0.05, 0.038 vs 0.0441 . Those are exactly the numbers you get by following the text. That is 5 of the 13 failed trials in my first leaderboard run that came from the benchmark, not the model. The Kaggle versions fix both: the XSS tests now check for raw markup instead of words, and the question sheet states the rule the verifier checks. | Model Kaggle slug | Why it is in the lineup | |---|---| | gpt-5.6-luna | The same model my Codex agent run used, so one shot and agent loop can be compared directly. | | claude-opus-5-5-default | Anthropic's top tier on Kaggle's model list. | | gemini-3.1-pro-preview | Google's Pro tier on Kaggle's model list. | | qwen3-coder-480b-a35b-instruct | An open-weights model built specifically for code. | | gpt-oss-120b | An open-weights general model you can run on your own hardware. | | gemini-3.7-flash | Not picked: Kaggle runs its default model when a task is pushed, so it came along for free. | Each model ran each task once, through Kaggle's model proxy at the SDK's default temperature. | task | Opus 5.5 | Gemini 3.1 Pro | GPT-5.6 Luna | Qwen3 Coder 480B | gpt-oss-120b | Gemini 3.7 Flash | |---|---|---|---|---|---|---| | cb-debug-wrong-answer | pass | pass | pass | pass | pass | pass | | cb-refactor-deadcode | pass | pass | pass | pass | pass | pass | | cb-feature-rate-limiter | pass | pass | pass | pass | pass | pass | | cb-perf-hot-loop | pass | pass | pass | fail | pass | pass | | cb-data-log-analysis | pass | pass | pass | pass | pass | pass | | cb-sec-patch-xss | pass | pass | pass | pass | pass | pass | | total | 6/6 | 6/6 | 6/6 | 5/6 | 6/6 | 6/6 | 1. One shot was enough for these six tasks. 35 of 36 runs passed. With the files in front of them and no way to run anything, every model fixed the stats bug, pruned exactly the three dead functions, wrote a token bucket that rolls back atomically and survives 100 threads, and escaped the comment board correctly. These tasks are too easy to separate frontier models in one shot. That is a result too: the difficulty I measured in cli-bench was not coming from these tasks. 2. The same model did better without the agent loop. GPT-5.6 Luna passed all six in one shot. As a Codex agent it passed hot-loop 1 time in 3, with a worst-case speedup of 1.2x on the gridded dataset. In one shot it wrote a spatial grid that, timed on my machine against the same verifier, ran 42.5x faster on uniform points, 26.5x on gridded, and 14.6x on clustered. The caveats are real: one run against three, one dataset seed against three, and different machines. But it is the opposite of what I expected. Having a shell did not help the agent find the fast solution, and may have pulled it toward measuring and patching a slow one. 3. Every model reached for the same idea on hot-loop, and the only failure was invisible to the unit tests. All six bucketed points into a grid. Clustered points were the worst case for every passing model 14.6x to 41x on my machine , not gridded. Qwen3 Coder's grid passed all nine shipped unit tests and still undercounted: on one random case it found 84 pairs where there are 123. Only the randomized comparison against the naive version caught it. If my verifier had stopped at the shipped tests, that bug would have scored as a pass. 4. Most of my first-round failures were my harness, again. The first batch of Kaggle runs had 9 failures. Seven were mine: tmp path , monkeypatch , so all six models "failed" the XSS task. Every one of them had escaped the output correctly. FILE: solve.py and rejected gpt-oss-120b's FILE: solve.py . I ran its script by hand against the same log and every answer was correct. I fixed both, pushed new versions of those two tasks, and re-ran every model on them. The table shows the re-runs. The other two first-round failures were real. One is the hot-loop bug above. In the other, Qwen3 Coder's log script called statistics.quantiles with a method name that does not exist and crashed. On the re-run it wrote a different script and passed. That is one run at the default temperature, so treat any single pass or fail as a sample, not a verdict. 5. The corrected XSS task is passable by the textbook fix. All six models used HTML escaping, and all six passed the corrected tests and the probe. In cli-bench 0.9.1, the same approach failed. That confirms the problem was the tests, not the models. What surprised me: across both versions of this benchmark, bugs on my side two task specs, a test runner, a parser caused more failures than the models did. What I would measure next: The lesson I am keeping: a benchmark that has never been run against a known-correct answer is not a benchmark yet. Every task in this port now ships with a reference solution, an empty reply, and a test-rewriting reply. A task only counts when the first passes and the other two fail, and that has to hold in the environment where the models actually run, not just on my laptop. Kaggle benchmark: https://www.kaggle.com/benchmarks/aks1321/cli-bench-one-shot https://www.kaggle.com/benchmarks/aks1321/cli-bench-one-shot The six public tasks each page has the full task code in its published notebook : cli-bench is Apache-2.0: https://github.com/arjunkshah12345-hash/cli-bench https://github.com/arjunkshah12345-hash/cli-bench