Who benchmarks the benchmark?

A new audit of the EnterpriseOps Gym benchmark found that fixing environment issues in the 'Teams' domain raised GPT-5.6 Luna's score from 26.2% to 100% on 61 tasks, revealing that many agent failures are actually benchmark failures. The audit identified contradictions, misleading observations, and misleading descriptions as primary causes, with examples including a function that returns None after a successful write and documentation that contradicts server behavior. This follows broader findings that 7 of 10 widely used agentic benchmarks violate task validity, and OpenAI stopped reporting SWE-bench Verified after finding 59.4% of its models' failures were due to broken problems.

2026-08-18 Agent failures are often environment failures disguised as model failures https://rywalker.com/context-engineering-hard-problem . To detangle the failure modes, we build benchmarks also called gyms that control the environment so we can better study and improve the model in isolation. In this post, I seek a method to score benchmarks themselves. Gyms are typically made up of these 7 components: That’s a lot of points of failure However, only 3 measures the model itself. When you investigate a failing gym scenario, these are common symptoms you may witness: The problem is, it’s not always easy to attribute causes to the symptom. Click around to see what I’m talking about: click a node The Agentic Benchmark Checklist https://arxiv.org/html/2507.02825v3 assessed 10 widely used agentic benchmarks and found 7 violating task validity and 7 violating outcome validity. A separate audit broke 8 benchmarks https://moogician.github.io/blog/2026/trustworthy-benchmarks-cont/ without solving a single task, most of them to a near-perfect score. SciCode-Verified https://arxiv.org/html/2608.04975v1 found 262 defects across 63 of its 64 problems, 192 of them rejecting correct solutions. Famously, OpenAI stopped reporting SWE-bench Verified after finding that 59.4% of the problems its models failed were themselves broken https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ , with 35.5% carrying tests so narrow they reject functionally correct submissions. Let’s zoom in on one. The EnterpriseOps Gym https://enterpriseops-gym.github.io measures an agent’s ability to follow complex instructions to operate a set of real-world MCP tools. It’s 1150 expert-curated tasks across 8 domains, 7 to 30 steps each, running against live containerized MCP servers with real state. The results are in the chart below. Fable 5 scores 52% on the “Teams” domain. 52% reads like the goal to beat, but you may be surprised to learn that I took the teams / oracle split 61 tasks from 26.2% to 100% with gpt-5.6-luna by fixing various issues in the gym itself. Here’s the breakdown: Contradictions are the largest single source of failure. Well, there are two types: 1 Misleading observations. For example, create virtual event townhall commits the row and then returns None , which results in an error, even though the write succeeded: Failed to create townhall: schemas.virtual event townhall.VirtualEventTownhallResponse argument after must be a mapping, not NoneType The confusing error message causes the agent to retry, and strict verifiers fail this scenario. This one was reported in March https://github.com/ServiceNow/EnterpriseOps-Gym/issues/4 with the affected task ids and acknowledged by the maintainers, and it is still open. 2 Misleading descriptions. For example, add channel member documents that the operation is “allowed only for channels with a membershipType value of private or shared” . That’s true of real Teams and false of this server, which accepts standard channels and writes the row. An agent that believed the documentation correctly skipped the call and was graded wrong. This second case is the more damaging one, because it rewards agents that don’t follow instructions . The benchmark’s own system prompt orders the agent to “never infer” and to “abort with a reason”, and then the environment penalizes exactly that compliance. One more example, just for good measure: the tab tools ship an examples value pointing at app id 06805b9e-77e3-4b93-ac81-525eb87513b8 , and the server rejects it: Teams app '06805b9e-77e3-4b93-ac81-525eb87513b8' not found in organization app catalog. Five tasks need a tab app id and have no app-listing tool in their oracle set, so the documentation is the only available source, and it is wrong. Sometimes verifiers are satisfiable, but only by luck aka flaky rather than broken . These come in a few forms: 1 A value that appears nowhere. Four verifiers require callback uri = 'https://meetings.techcorp.com/api/calls' . That string is in no prompt and in no seed database. An agent that refuses to invent values cannot pass; one that fabricates a plausible URL cannot pass either, unless it guesses this exact one. 2 A sentence with two readings. One task promotes Bob to owner, then says “add the other owners of the team excluding me as co-organizers” , leaving open whether the just-promoted Bob counts. Across four runs the tool sequences were identical , producing different results: | Run | co-organizers sent | Result | |---|---|---| | 1 | alice, bob | pass | | 2 | alice | fail | | 3 | alice, bob | pass | | 4 | alice | fail | By the way, this same class of problems occurs when the agent generates prose. For example, one verifier requires the literal %leadership channel% ; the agent posted the task’s text verbatim but bolded a word, The <b Leadership</b channel , using the HTML content type, and failed. Another only passed when the agent wrote “the weekly TechCorp Weekly Release Readiness townhall” ; every run that phrased it naturally, matching the instruction word for word, failed. The system prompt ships inside the dataset row , so it is task data, and it instructs: “When identifiers such as names or IDs are missing, perform exactly one lookup per entity type…Never inferuser/team/channel data … If a request violates access control or schema constraints,abortwith a reason.” The database contains TechCorp Solutions Team . The model queried displayName eq 'TechCorp Solutions' , got , and, permitted one lookup and forbidden to infer, aborted. 18 of 61 tasks ended with 3 or fewer tool calls. The tool’s own filter documentation advertises startswith and contains , either of which returns the team. So does a single unfiltered call. Correcting that one clause and changing nothing else recovered +9.8 points and eliminated 13 of the 18 early aborts. The clause is present in all 5 prompt variants across all 61 tasks , so it plausibly suppresses every model on the published leaderboard. There are some assertions that are unsatisfiable by any behavior, causing a hard cap on every model’s score. Fortunately, these can sometimes be found by static analysis without running an agent at all. 1 Impossible SQL. For example, one welcome-message verifier requires LOWER m.body json LIKE '%Alice Johnson%' , comparing a lowercased column against a capitalized literal, so it matches nothing, ever. 2 A type bug in the comparison engine. expected value is stored as the string "1" and compared with == against SQL’s integer 1 . 1 == "1" is False , so those verifiers fail regardless of what the agent does. This affects 144 verifiers across the public split, including 20 tasks in which every verifier is string-typed , unwinnable for every model including every entry currently on the leaderboard . While in there I found a third: verifier results are silently dropped when two verifiers share a name https://github.com/ServiceNow/EnterpriseOps-Gym/issues/23 , so only the last one is ever checked. PR 24 https://github.com/ServiceNow/EnterpriseOps-Gym/pull/24 fixes it. The repo is open and they do merge these: an earlier report of 14 broken CSM tasks landed as revised task data https://github.com/ServiceNow/EnterpriseOps-Gym/pull/16 . The “oracle” mode in the benchmark is meant to hand the agent exactly the tools its task needs. list team apps ; the server exposes list teams apps , one letter apart. update team is absent from its tool list. The instruction the task closes on cannot be carried out as written.Some tasks assume relative dates “next Friday”, “next quarter” , yet the scenarios don’t define what now is. The real wall clock doesn’t work as a substitute: the scenarios are written around 2025-11 to 2026-01, so a later “today” makes every task read as historical and a careful agent correctly refuses to schedule. The corpus is also internally inconsistent about time , so no single simulated date is correct for all of it. Some verifiers bake this in directly: one requires call records within datetime 'now', '-60 days' when the newest fixture row is 2025-12-29. It passed when authored and fails permanently afterwards, and the cap widens as the dataset ages. BetterBench https://arxiv.org/pdf/2411.12990 does score benchmarks, on 46 criteria, but they are about documentation and process. A gym can score full marks and still ship twenty faulty tasks. So here is a different score. Instead of running a model, you replay a task’s known-correct solution and ask whether the gym accepts it, which makes the answer a property of the gym rather than of whoever ran it: C is the one that catches an environment lying about itself, which in my split was 42% of the gap. We can actually make it checkable by giving every task a certificate: a replayable trace, closed over the task’s declared tools, conformant with the docs and the policy. WebForge https://arxiv.org/pdf/2604.10988 and SkillsBench https://arxiv.org/html/2602.12670v1 already run solvability certificates in CI. τ²-bench makes this argument better than I can. It ships an expected action trace for every task, and the audit’s fixes include removing incorrect ones. Maybe we should start publishing SCR scores? I can try to derive them: | Gym | S | C | R | SCR | |---|---|---|---|---| EnterpriseOps teams / oracle | 0.74 | 0.56 | 0.64 | 0.26 | | SciCode | 0.82 | 0.65 | 0.86 | 0.46 | | τ²-bench retail and airline | ? | ? | ? | ≤ 0.68 | | SWE-bench Verified | 1.00 | 0.84 | 1.00 | ≤ 0.84 | SciCode and SWE-bench hand you the split for free, since both audits already sort their defects by mechanism, though SWE-bench’s two 1.00s mean unmeasured rather than clean. τ² publishes only a total 53 documented fixes across 164 tasks , which is plenty for the composite the factors are conditional, so everything cancels except blocked-over-total but not enough to fill the columns. Column C is a new measurement I think, and it deserves a deeper dive than this post can give it. Regardless, it’s time we start benchmarking the benchmarks.