{"slug": "a-benchmark-shipped-2700-agent-trajectories-i-recomputed-its-leaderboard", "title": "A benchmark shipped 2,700 agent trajectories. I recomputed its leaderboard.", "summary": "A developer independently recomputed the LSCBench leaderboard from the benchmark's shipped 2,700 agent traces, finding that five of six model rows match the paper's Table 10 to the decimal, including Claude Fable 5's 25.1% functional-and-secure rate. The only discrepancy was Kimi-K3's functional pass rate, 92.2% in the trace catalog versus 91.3% in the paper — a four-task gap that changes no ranking. The developer argued that shipping per-run outcomes, rather than only summary tables, is what makes a benchmark score a reproducible measurement.", "body_md": "Most eval papers give you a table and a PDF. LSCBench gives you the runs.\n\n[LSCBench](https://arxiv.org/abs/2610.14766) is a secure-code benchmark out of NUS and the Qwen team: 150 real vulnerabilities cut into 450 tasks across seven languages, at three granularities (fill in a function, write a whole file, rebuild a repo's source tree). The task spec never mentions security. Hidden functional and security suites grade the result. Six frontier agents run it under Claude Code or Codex, with a 500-turn cap, interactive tools off, and no network access.\n\nThe headline number is that the best model, Claude Fable 5, produces code that is both functional and secure on 25.1% of tasks. Functional completion alone runs 73.6% to 91.3%. In other words, the code mostly works, and mostly keeps the vulnerability.\n\nWhat got me to actually open the thing: the repo ships the traces. [alibaba/last-secure-code-benchmark](https://github.com/alibaba/last-secure-code-benchmark) has a `traces/` directory with 2,700 trace JSONs, 450 per model, plus a `catalog.json` that carries the verdict for every single run. So instead of trusting the table, I recomputed it.\n\nI pulled `catalog.json` and counted, per model over the 450 tasks: `functional`, `secure`, and both. Here is my number next to the paper's Table 10.\n\n| Model | FuncPass (mine) | Func ∧ Sec (mine) | Func ∧ Sec (paper) | \n|---|---|---|---|\n| Claude Fable 5 | 91.1% | 25.1% | 25.1% | \n| GPT-5.6-Sol | 89.1% | 24.7% | 24.7% | \n| Qwen3.8-Max | 89.3% | 22.0% | 22.0% | \n| GLM-5.3 | 87.1% | 21.3% | 21.3% | \n| DeepSeek-V4-Pro | 73.6% | 18.0% | 18.0% | \n| Kimi-K3 | 92.2% | 16.4% | 16.4% | \n\nFive rows match to the decimal. The scenario-level column matches too: I get 12.0% for Claude Fable 5 on scenarios secured at all three granularities, which is exactly what the paper reports. The pooled union matches as well, 198 of 450 tasks secured by at least one model (44.0%), 19 tasks secured by all six, 62 secured by exactly one.\n\nOne row does not match. Kimi-K3's FuncPass is 92.2% in the catalog (415 of 450) and 91.3% in the paper. That is a four-task gap. It changes no ranking and no conclusion. It is also the only difference I found across 2,700 runs.\n\nThat is a good result for the benchmark, and it is the reason the shipped artifact matters at all.\n\nA number you cannot recompute is a claim. A number you can recompute is a measurement. The difference shows up the moment someone else's run disagrees with yours, which is exactly when you need raw outcomes instead of a summary.\n\nI have written before about why [offline benchmark scores and online review performance diverge](https://aicodereview.io/blog/ai-code-review-benchmarks-offline-vs-online-evals/), and the mechanism is always the same: the summary drops the conditions. Shipping per-run outcomes is the cheap fix. It costs a tarball and a catalog file. Compare that to benchmarks where [the headline claim rots in months](https://aicodereview.io/blog/martians-code-review-bench-the-1-claim-rots-in-months/) because nobody can re-derive it.\n\nThe four-task gap is small enough to be version skew, or a difference in how a missing verdict gets counted. I cannot tell which from outside. That is the point. With the traces shipped, I can at least see that there is a disagreement. With only the table, I would have no way to know one existed.\n\nEvery trace carries `reward`, and it is 1 only when the run is both functional and secure. So a run that rebuilt the module and kept the vulnerability intact scores the same 0 as a run that never produced a compiling build.\n\nThose are different events. One is a security miss, the other is a capability miss, and they call for different responses from anyone reading the results. The paper is careful about this and reports FuncPass, SecPass and Func ∧ Sec separately. The shipped `reward` column collapses all three back into one bit.\n\nConcretely: 574 of 2,700 runs scored reward 1. If you filter traces by `reward` to find \"the agent gave up\", you get the wrong set, because giving up and shipping the bug are indistinguishable there. Sort by `reward` and you cannot rank a security regression against a build regression. You have to go back to the two booleans.\n\nThe catalog also records `exit_reason`, which is the field I did not expect and the one I ended up caring about most.\n\n| exit_reason | runs | of which reward 1 | \n|---|---|---|\n| (none) | 2,580 | 559 | \n| NonZeroAgentExitCodeError | 41 | 1 | \n| AgentMaxTurnsError | 32 | 6 | \n| ApiRateLimitError | 25 | 5 | \n| AgentTimeoutError | 16 | 3 | \n| VerifierTimeoutError | 5 | 0 | \n| AgentProcessSignalError | 1 | 0 | \n\n120 of 2,700 runs, 4.4%, ended on something other than the agent deciding it was done. Six hit the 500-turn cap. Twenty-five died on an API rate limit. Sixteen timed out. Five failed inside the verifier itself, which is not the agent's fault at all.\n\nNow the part that matters. Fifteen of those errored runs still scored reward 1. A run that hit the turn cap or died on a rate limit can still have left a correct, secure implementation behind, so the pass is real. But 2.6% of all passes in the dataset come from a run that never cleanly finished. If you are ranking models on a difference of a couple of points, that is the same order of magnitude as your margin.\n\nThe five verifier timeouts are the cleanest example. They are recorded as `functional: false, secure: false`, which is the same treatment the paper gives a missing verdict by design (\"counted as failures when missing\"). That is a defensible convention. It is also a convention that turns an infrastructure failure into a model failure, and you can only see it because the traces are there.\n\nFour things, and each takes minutes when the traces ship:\n\n`functional` and `secure` separately rather than using the collapsed reward. The reward bit merges a security miss with a build miss.`exit_reason` on every run and see how many passes came from a run that errored out. If a model's lead is smaller than its count of errored-but-passed runs, you do not have a ranking yet.\nWhen a benchmark ships its traces, recompute it before you cite it. You will usually find that the paper is right, and occasionally you will find the one row that is not. Both are more useful than trusting a table, and the second one is only findable if the artifact exists.\n\nIf you maintain an eval, the cheapest credibility you can buy is publishing the per-run outcomes next to the summary, including the failure reasons. If you consume one, the cheapest sanity check is counting the rows, splitting the reward bit, and reading the exit reasons. That is three filters between you and a number that sounds better than it is.", "url": "https://wpnews.pro/news/a-benchmark-shipped-2700-agent-trajectories-i-recomputed-its-leaderboard", "canonical_source": "https://dev.to/cole_halton_42f71d71b809b/a-benchmark-shipped-2700-agent-trajectories-i-recomputed-its-leaderboard-pgf", "published_at": "2026-10-20 00:30:02+00:00", "updated_at": "2026-10-20 00:47:18.790444+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "ai-agents", "large-language-models"], "entities": ["LSCBench", "Claude Fable 5", "GPT-5.6-Sol", "Qwen3.8-Max", "GLM-5.3", "DeepSeek-V4-Pro", "Kimi-K3", "Alibaba"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-benchmark-shipped-2700-agent-trajectories-i-recomputed-its-leaderboard", "markdown": "https://wpnews.pro/news/a-benchmark-shipped-2700-agent-trajectories-i-recomputed-its-leaderboard.md", "text": "https://wpnews.pro/news/a-benchmark-shipped-2700-agent-trajectories-i-recomputed-its-leaderboard.txt", "jsonld": "https://wpnews.pro/news/a-benchmark-shipped-2700-agent-trajectories-i-recomputed-its-leaderboard.jsonld"}}