{"slug": "invoked-not-executed", "title": "Invoked, not executed", "summary": "A developer's routing policy failed to prevent a deep-research request from burning through 194% of Claude's usage quota across three consecutive five-hour windows, because the request bypassed the policy's task-decomposition step. Tests with six blind subagents showed the harness fired only half the time, and even when it did, subagents silently improvised around missing tools, undermining the intended cost control.", "body_md": "A deep-research request to the top-tier model tore through three consecutive five-hour usage\n\nwindows, the rolling quota Claude enforces before a session has to stop and reset, to answer a\n\nsingle question.\n\n| Usage window | Burned | Time to burn it |\n|---|---|---|\n| Window 1 | 54% | about two minutes |\n| Window 2 | 100% | about four minutes |\n| Window 3 | 40% | finishing the job |\n\nThree consecutive five-hour usage windows against one question. That adds up to 194% - nearly two\n\nentire windows - and the first two were gone inside six minutes.\n\nNobody had done anything unusual. Someone had just asked the model to look something up properly.\n\n[The routing policy this machine runs on](https://thekilted.dev/route-dont-guess/) has a rule for\n\nexactly this shape of task: work that's one step (search, fetch a source, extract a claim) and\n\nobjectively checkable belongs on the cheapest tier that can do it, not on the model that costs the\n\nmost per token. Deep research is built entirely out of that shape: search, fetch, extract, feeding\n\na synthesis step that does need judgment. The rule already existed. It just never had a chance to\n\napply.\n\nA routing policy only engages at a decision point: the moment a session has to choose which tier\n\ndoes the next piece of work. Most of the policy's machinery assumes that moment exists. A plan gets\n\nbroken into tasks, and each task is a place to ask who does this.\n\nA direct request, \"deep-research this,\" skips all of that. It's answered as one thing, by whichever\n\nmodel receives it, in whichever way that model has of doing research. There's no task list to route\n\npieces of. The search, the fetching, the extraction, the verification all happen inside a single\n\nreply from the most expensive available model, because nothing ever asked whether the next step\n\nneeded to be there.\n\nThe fix follows from the gap. Route research-shaped requests through a harness that does the\n\ndecomposing on the way in, whatever words were used to ask. That covers all three shapes the ask\n\ntakes: an explicit \"research this,\" a model deciding on its own that a claim needs checking, or\n\nsomething as soft as \"I'm not sure what's actually true here.\" All three land in the same place if\n\nanswered directly. All three should hit the harness instead.\n\nHaving independently found an adjacent problem (a third-party tool doing something similar had its\n\nown bug in the budget cap meant to prevent exactly this kind of burn), a sibling project was asked\n\nto check whether the routing fix triggered in practice. Six blind subagents, no shared context, one\n\nprompt each: three ways of asking for the same research, crossed with two model tiers. The topic\n\nwas held constant, the enforcement status of the EU AI Act, so how the question was worded was the\n\nonly deliberate variable.\n\nThe intended way of measuring it failed. A subagent's own transcript file stays zero bytes even\n\nafter the run has finished, so nobody could simply read off which tools had been called. The\n\nmeasurement fell back to asking each subject afterward what it had done, cross-checked against a\n\ntool-call count the harness reports and the agent has no hand in writing.\n\nThe harness fired in half the runs. That looked like a pass. It wasn't one.\n\nEvery run that triggered the harness received an instruction to call `Workflow`\n\n, the dispatcher\n\nthat fans the research out across sub-agents. `Workflow`\n\nis not available inside a delegated\n\nsubagent. Each of those runs found that out mid-task, noticed the gap, and quietly did the research\n\na different way: by hand, with the tools it did have. Nothing crashed. Nothing logged an error. The\n\nfinal answer looked like a normal research answer, because it was one. Just not the one the rule\n\nwas written to guarantee.\n\nOne of the six was candid about exactly what was lost in translation. It had improvised its own\n\nversion of the harness's fan-out step, sending several of its own sub-agents out to search in\n\nparallel, and reported the results converging as if that settled things. Asked afterward what its\n\nown five-agent agreement was worth, it corrected itself before anyone pushed back:\n\nThe convergence I reported between agents is corroboration, not adversarial verification, five\n\nagents drawing on an overlapping source pool can agree and still be wrong together.\n\nThe step that was skipped wasn't a decoration. It was the one part of the harness whose entire job\n\nis to catch exactly that kind of agreement that isn't evidence. A rule checking whether the harness\n\nwas invoked would have called that run compliant. The harness had not, in the sense that mattered,\n\nrun at all.\n\n| Run | Harness invoked | Fan-out executed |\n|---|---|---|\n| Explicit ask (sonnet) | met | broke |\n| Explicit ask (opus) | met | broke |\n| No keyword (sonnet) | broke | never established |\n| No keyword (opus) | broke | never established |\n| Soft / vague (sonnet) | met | broke |\n| Soft / vague (opus) | broke | never established |\n\nInvoked in three of six runs, executed in none. The three that invoked it reached for `Workflow`\n\n,\n\nwhich a delegated subagent cannot call, and fell back to research by hand; the three that never\n\ninvoked it never got far enough to find out. The right-hand column is empty either way, and that\n\ncolumn is the one the rule was written to guarantee.\n\nThe one wording that worked reliably, the explicit \"deep-research this,\" didn't work because the\n\nrouting logic held. One of the successful runs said, unprompted, exactly why it had picked up on\n\nthe request. The words read to it as plain English, and it matched them to the right tool on its\n\nown initiative. In its own account, the phrase was \"hyphenated as a verb phrase,\" with \"no slash,\n\nno backticks, no skill syntax.\"\n\nNothing in how it was written signalled a tool call.\n\nRename the tool, or ask for the same research without those two words, and the coincidence that\n\nmade the easy case look solved stops holding.\n\nThe case that matters is a serious, specific question asked with no research vocabulary in it. That\n\nis the shape almost every real research need takes, and it missed on every model tested. That's the\n\ngap worth worrying about.\n\nThe rule was rewritten to check for evidence the harness had actually run: a count of sub-agents\n\ndispatched, a tally of claims independently verified, rather than trusting that the right words had\n\nbeen said. That closes the failure mode the test found: a compliant-looking run that quietly did\n\nnone of the work the compliance was meant to certify.\n\nIt doesn't close the other one. The wording that reliably works is still the wording that happens to\n\nmatch the tool's own name. The question that matters, the one with no research vocabulary in it,\n\nmisses the harness on both models tried.\n\nFixing what gets checked was the easy half. The other half is getting the harness to notice it is\n\nneeded at all, from a question that never says so. That one is still open, and the coincidence that\n\nmade the easy case look solved is the reason to expect it is the harder one.\n\nThe rule that got rewritten, extracted and de-identified: [the routing-policy\nrepo](https://github.com/thekilteddev/routing-policy).", "url": "https://wpnews.pro/news/invoked-not-executed", "canonical_source": "https://dev.to/thekilteddev/invoked-not-executed-1cg8", "published_at": "2026-08-20 07:35:56+00:00", "updated_at": "2026-08-20 08:13:38.669202+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-agents", "developer-tools"], "entities": ["Claude", "EU AI Act"], "alternates": {"html": "https://wpnews.pro/news/invoked-not-executed", "markdown": "https://wpnews.pro/news/invoked-not-executed.md", "text": "https://wpnews.pro/news/invoked-not-executed.txt", "jsonld": "https://wpnews.pro/news/invoked-not-executed.jsonld"}}