{"slug": "who-benchmarks-the-benchmark", "title": "Who benchmarks the benchmark?", "summary": "A new audit of the EnterpriseOps Gym benchmark found that fixing environment issues in the 'Teams' domain raised GPT-5.6 Luna's score from 26.2% to 100% on 61 tasks, revealing that many agent failures are actually benchmark failures. The audit identified contradictions, misleading observations, and misleading descriptions as primary causes, with examples including a function that returns None after a successful write and documentation that contradicts server behavior. This follows broader findings that 7 of 10 widely used agentic benchmarks violate task validity, and OpenAI stopped reporting SWE-bench Verified after finding 59.4% of its models' failures were due to broken problems.", "body_md": "2026-08-18\n\nAgent failures are often [environment\nfailures disguised as model failures](https://rywalker.com/context-engineering-hard-problem).\n\nTo detangle the failure modes, we build benchmarks (also called gyms) that control the environment so we can better study and improve the model in isolation.\n\nIn this post, I seek a method to score benchmarks themselves.\n\nGyms are typically made up of these 7 components:\n\nThat’s a lot of points of failure! However, only #3 measures the model itself.\n\nWhen you investigate a failing gym scenario, these are common symptoms you may witness:\n\nThe problem is, it’s not always easy to attribute causes to the symptom. Click around to see what I’m talking about:\n\nclick a node\n\nThe [Agentic Benchmark\nChecklist](https://arxiv.org/html/2507.02825v3) assessed 10 widely used agentic benchmarks and found 7\nviolating task validity and 7 violating outcome validity. A separate\naudit [broke\n8 benchmarks](https://moogician.github.io/blog/2026/trustworthy-benchmarks-cont/) without solving a single task, most of them to a\nnear-perfect score. [SciCode-Verified](https://arxiv.org/html/2608.04975v1) found\n262 defects across 63 of its 64 problems, 192 of them rejecting correct\nsolutions. Famously, OpenAI stopped reporting SWE-bench Verified after\nfinding that [59.4%\nof the problems its models failed were themselves broken](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/), with 35.5%\ncarrying tests so narrow they reject functionally correct\nsubmissions.\n\nLet’s zoom in on one. The [EnterpriseOps Gym](https://enterpriseops-gym.github.io)\nmeasures an agent’s ability to follow complex instructions to operate a\nset of real-world MCP tools. It’s 1150 expert-curated tasks across 8\ndomains, 7 to 30 steps each, running against live containerized MCP\nservers with real state. The results are in the chart below. Fable 5\nscores 52% on the “Teams” domain.\n\n52% reads like the goal to beat, but you may be surprised to learn\nthat I took the `teams`\n\n/`oracle`\n\nsplit (61 tasks)\nfrom 26.2% to 100% with `gpt-5.6-luna`\n\nby fixing various\nissues in the gym itself.\n\nHere’s the breakdown:\n\nContradictions are the largest single source of failure. Well, there are two types:\n\n**(1) Misleading observations.** For example,\n`create_virtual_event_townhall`\n\ncommits the row and\n*then* returns `None`\n\n, which results in an error, even\nthough the write succeeded:\n\n```\nFailed to create townhall: schemas.virtual_event_townhall.VirtualEventTownhallResponse()\nargument after ** must be a mapping, not NoneType\n```\n\nThe confusing error message causes the agent to retry, and strict\nverifiers fail this scenario. This one was [reported\nin March](https://github.com/ServiceNow/EnterpriseOps-Gym/issues/4) with the affected task ids and acknowledged by the\nmaintainers, and it is still open.\n\n**(2) Misleading descriptions.** For example,\n`add_channel_member`\n\ndocuments that the operation is\n*“allowed only for channels with a membershipType value of private or\nshared”*. That’s true of real Teams and false of this server, which\naccepts standard channels and writes the row. An agent that believed the\ndocumentation correctly skipped the call and was graded wrong.\n\nThis second case is the more damaging one, because it rewards\n**agents that don’t follow instructions**. The benchmark’s\nown system prompt orders the agent to “never infer” and to “abort with a\nreason”, and then the environment penalizes exactly that compliance.\n\nOne more example, just for good measure: the tab tools ship an\n`examples`\n\nvalue pointing at app id\n`06805b9e-77e3-4b93-ac81-525eb87513b8`\n\n, and the server\nrejects it:\n\n```\nTeams app '06805b9e-77e3-4b93-ac81-525eb87513b8' not found in organization app catalog.\n```\n\nFive tasks need a tab app id and have no app-listing tool in their oracle set, so the documentation is the only available source, and it is wrong.\n\nSometimes verifiers are satisfiable, but only by luck (aka\n*flaky* rather than broken). These come in a few forms:\n\n**(1) A value that appears nowhere.** Four verifiers\nrequire\n`callback_uri = 'https://meetings.techcorp.com/api/calls'`\n\n.\nThat string is in no prompt and in no seed database. An agent that\nrefuses to invent values cannot pass; one that fabricates a plausible\nURL cannot pass either, unless it guesses this exact one.\n\n**(2) A sentence with two readings.** One task promotes\nBob to owner, then says *“add the other owners of the team (excluding\nme) as co-organizers”*, leaving open whether the just-promoted Bob\ncounts. Across four runs the tool sequences were\n**identical**, producing different results:\n\n| Run | co-organizers sent | Result |\n|---|---|---|\n| 1 | `alice, bob` |\npass |\n| 2 | `alice` |\nfail |\n| 3 | `alice, bob` |\npass |\n| 4 | `alice` |\nfail |\n\nBy the way, this same class of problems occurs when the agent\ngenerates prose. For example, one verifier requires the literal\n`%leadership channel%`\n\n; the agent posted the task’s text\nverbatim but bolded a word,\n`The <b>Leadership</b> channel`\n\n, using the HTML\ncontent type, and failed. Another only passed when the agent wrote\n*“the weekly TechCorp Weekly Release Readiness townhall”*; every\nrun that phrased it naturally, matching the instruction word for word,\nfailed.\n\nThe system prompt ships **inside the dataset row**, so\nit is task data, and it instructs:\n\n“When identifiers such as names or IDs are missing, perform\n\nexactly one lookup per entity type…Never inferuser/team/channel data … If a request violates access control or schema constraints,abortwith a reason.”\n\nThe database contains `TechCorp Solutions Team`\n\n. The model\nqueried `displayName eq 'TechCorp Solutions'`\n\n, got\n`[]`\n\n, and, permitted one lookup and forbidden to infer,\naborted. **18 of 61 tasks ended with 3 or fewer tool\ncalls.** The tool’s own `_filter`\n\ndocumentation\nadvertises `startswith()`\n\nand `contains()`\n\n, either\nof which returns the team. So does a single unfiltered call.\n\nCorrecting that one clause and changing nothing else recovered +9.8\npoints and eliminated 13 of the 18 early aborts. The clause is present\nin **all 5 prompt variants across all 61 tasks**, so it\nplausibly suppresses every model on the published leaderboard.\n\nThere are some assertions that are unsatisfiable by any behavior, causing a hard cap on every model’s score. Fortunately, these can sometimes be found by static analysis without running an agent at all.\n\n**(1) Impossible SQL.** For example, one welcome-message\nverifier requires\n`LOWER(m.body_json) LIKE '%Alice Johnson%'`\n\n, comparing a\nlowercased column against a capitalized literal, so it matches nothing,\never.\n\n**(2) A type bug in the comparison engine.**\n`expected_value`\n\nis stored as the string `\"1\"`\n\nand\ncompared with `==`\n\nagainst SQL’s integer `1`\n\n.\n`1 == \"1\"`\n\nis `False`\n\n, so those verifiers fail\nregardless of what the agent does. This affects **144 verifiers\nacross the public split, including 20 tasks in which every\nverifier is string-typed**, unwinnable for every model (including\nevery entry currently on the leaderboard).\n\nWhile in there I found a third: verifier results are [silently\ndropped when two verifiers share a name](https://github.com/ServiceNow/EnterpriseOps-Gym/issues/23), so only the last one is\never checked. [PR\n#24](https://github.com/ServiceNow/EnterpriseOps-Gym/pull/24) fixes it. The repo is open and they do merge these: an earlier\nreport of 14 broken CSM tasks landed as [revised\ntask data](https://github.com/ServiceNow/EnterpriseOps-Gym/pull/16).\n\nThe “oracle” mode in the benchmark is meant to hand the agent exactly the tools its task needs.\n\n`list_team_apps`\n\n; the server exposes\n`list_teams_apps`\n\n, one letter apart.`update_team`\n\nis absent from its tool list. The instruction\nthe task closes on cannot be carried out as written.Some tasks assume relative dates (“next Friday”, “next quarter”), yet\nthe scenarios don’t define what *now* is. The real wall clock\ndoesn’t work as a substitute: the scenarios are written around 2025-11\nto 2026-01, so a later “today” makes every task read as historical and a\ncareful agent correctly refuses to schedule.\n\nThe corpus is also **internally inconsistent about\ntime**, so no single simulated date is correct for all of it.\nSome verifiers bake this in directly: one requires call records within\n`datetime('now', '-60 days')`\n\nwhen the newest fixture row is\n2025-12-29. It passed when authored and fails permanently afterwards,\nand the cap widens as the dataset ages.\n\n[BetterBench](https://arxiv.org/pdf/2411.12990) does score\nbenchmarks, on 46 criteria, but they are about documentation and\nprocess. A gym can score full marks and still ship twenty faulty tasks.\nSo here is a different score. Instead of running a model, you replay a\ntask’s known-correct solution and ask whether the gym accepts it, which\nmakes the answer a property of the gym rather than of whoever ran\nit:\n\n**C** is the one that catches an environment lying about\nitself, which in my split was 42% of the gap. We can actually make it\ncheckable by giving every task a certificate: a replayable trace, closed\nover the task’s declared tools, conformant with the docs and the policy.\n[WebForge](https://arxiv.org/pdf/2604.10988) and [SkillsBench](https://arxiv.org/html/2602.12670v1) already run\nsolvability certificates in CI.\n\nτ²-bench makes this argument better than I can. It ships an expected action trace for every task, and the audit’s fixes include removing incorrect ones.\n\nMaybe we should start publishing SCR scores?\n\nI can try to derive them:\n\n| Gym | S | C | R | SCR |\n|---|---|---|---|---|\nEnterpriseOps `teams` /`oracle` |\n0.74 | 0.56 | 0.64 | 0.26 |\n| SciCode | 0.82 | 0.65 | 0.86 | 0.46 |\n| τ²-bench retail and airline | ? | ? | ? | ≤ 0.68 |\n| SWE-bench Verified | 1.00 | 0.84 | 1.00 | ≤ 0.84 |\n\nSciCode and SWE-bench hand you the split for free, since both audits already sort their defects by mechanism, though SWE-bench’s two 1.00s mean unmeasured rather than clean. τ² publishes only a total (53 documented fixes across 164 tasks), which is plenty for the composite (the factors are conditional, so everything cancels except blocked-over-total) but not enough to fill the columns.\n\nColumn **C** is a new measurement I think, and it\ndeserves a deeper dive than this post can give it.\n\nRegardless, it’s time we start benchmarking the benchmarks.", "url": "https://wpnews.pro/news/who-benchmarks-the-benchmark", "canonical_source": "https://shukla.io/blog/2026-08/gym.html", "published_at": "2026-08-18 01:41:06.016992+00:00", "updated_at": "2026-08-18 01:41:08.585325+00:00", "lang": "en", "topics": ["ai-research", "ai-ethics", "ai-agents"], "entities": ["EnterpriseOps Gym", "OpenAI", "SWE-bench Verified", "SciCode-Verified", "Fable 5", "GPT-5.6 Luna", "ServiceNow"], "alternates": {"html": "https://wpnews.pro/news/who-benchmarks-the-benchmark", "markdown": "https://wpnews.pro/news/who-benchmarks-the-benchmark.md", "text": "https://wpnews.pro/news/who-benchmarks-the-benchmark.txt", "jsonld": "https://wpnews.pro/news/who-benchmarks-the-benchmark.jsonld"}}