{"slug": "what-your-coding-agent-lets-you-control-and-three-things-we-got-wrong", "title": "What Your Coding Agent Lets You Control — and Three Things We Got Wrong", "summary": "A new governance scorecard for coding agents, published by an unnamed tool builder, found that its own product fails three of nine control questions, including whether its hook can grant permission in Claude Code and Antigravity. The index, which only measures product capabilities rather than model behavior, downgraded the Claude Code claim to 'unknown' and confirmed that Antigravity treats 'allow' as non-authoritative. The findings highlight that many agent products lack true programmatic control over tool calls.", "body_md": "# What Your Coding Agent Lets You Control — and Three Things We Got Wrong\n\nHere is a question you can answer about your coding agent in about five minutes, without running a single agent task:\n\n**Can a program of yours see a tool call before it happens, and say no?**\n\nNot *“will the model behave sensibly.”* Not *“how good is it at refusing bad instructions.”*\nJust: does the product let your code get a word in first. It is a question about the\nsoftware, not about the model, and it has a yes-or-no answer that will still be true in\nsix months.\n\nAlmost everything published about agent safety measures the first kind of question. Almost nothing measures the second. So we wrote down nine of the second kind, answered them for the tools we actually use, and published the answers.\n\nThen the answers marked us down three times. That is the part worth writing about.\n\n## The awkward part first\n\nWe build a governance tool for coding agents. That is a problem for anyone we ask to trust\na scorecard we wrote, because a scorecard that rewards *“has a hook my product can plug\ninto”* is an advertisement wearing a table’s clothes.\n\nSo the index carries three rules, and they constrain us rather than excuse us:\n\n**No question may require our tooling to answer.** If you cannot check a cell without ever having heard of us, it does not belong in the index — however useful it would be.**No question about model behaviour.** Only about what the product permits. That is what keeps a result from March still true in November.**No single score.** Per-question answers only. One number would hide exactly the one of the nine you should have cared about.\n\nAnd a fourth that turned out to matter most: **“we looked and didn’t find one” is not a\nmeasurement.** Unknown is a publishable answer. A guess is a defect.\n\nThose rules cost us something immediately.\n\n## Three claims of ours that the measurement corrected\n\n**1. “Our hook can grant permission.” Downgraded to unknown.**\n\nOne of the nine questions is whether your code can return *allowed* and have the host skip\nits own approval prompt. This is the difference between an overlay and an authority. Without\nit, your policy can only ever add friction on top of the host’s permission list; with it,\nyour policy can *be* the list.\n\nWe had that marked yes for Claude Code. Our evidence was our own demo, which shows that\n**our hook emits an “allow” decision**. Read that again, because we didn’t for a while: it\nshows what our code sends, not what the host does with it. Those are different claims and we\nhad only tested the first.\n\nBy the index’s own rule — that “documented” means the *vendor’s* documentation — our demo\nnever qualified. The cell is now **unknown**.\n\n**2. “Our hook can grant permission on Antigravity.” Measured. It cannot.**\n\nSame claim, different tool, and this time we could actually test it, because Antigravity’s\nheadless mode has a useful property: it cannot show you a prompt, so when something *would*\nhave prompted, it refuses and tells you why. “Would have prompted” becomes an observable\noutcome with nobody sitting there to click.\n\nA hook returning `allow`\n\nleft the command refused, with the same message as a hook returning\nnothing at all.\n\nThat result only means something because of the control we ran next: the same hook returning\n`deny`\n\n**did** stop a command, and the tool told the model *“blocked by a system hook”*,\nquoting our reason text. So the hook is being consulted and obeyed. `deny`\n\nis authoritative;\n`allow`\n\nis not. Overlay, not authority — which is exactly what the question was asking.\n\nOur own documentation said otherwise. It has been corrected.\n\n**3. “Configuration is partly file-based.” It is worse than partly.**\n\nAnother question asks whether everything the agent can reach is written in files you can read, diff and commit — or whether some of it exists only in a settings screen.\n\nWe had this at *partial*. Then we counted. In one working session there were **six MCP\nservers live**. **Exactly one** was declared in a configuration file.\n\nFour of the others appear on disk only inside a list recording what has *ever* been\nconnected — which is history, not configuration. It included a connector that was not\nrunning, and editing it changes nothing about what loads. The sixth left no trace anywhere.\n\nA surface you cannot enumerate from disk is one you cannot review, diff, or put in version\ncontrol. One out of six is not *partial*. The cell now reads **no**.\n\n## What we learned about the tools themselves\n\nSince the point of all this is the practitioner’s question, the useful findings:\n\n**You can refuse a built-in tool, but you cannot remove one.** A deny rule makes the call\nfail; the tool is still there, still offered, still callable. We also tried the other\nmechanism — a `disallowedTools`\n\nlist in the settings file — and it did nothing at all. Not\nan error, not a warning: the call simply succeeded. A setting that is accepted and ignored\nis worse than one that is rejected.\n\n**There is a post-execution hook, and it lets you count what you stopped.** A pre-execution\nhook records what was *proposed*; a post-execution hook records what actually *ran*, with the\nresult and how long it took. Subtract one from the other and you have a direct count of calls\nthat were proposed and never happened. We tested it with three deliberately-blocked commands:\nthe pre-hook logged four events, the post-hook logged one, and the three refusals appear in\nno post-hook line at all. Most governance tooling can only estimate that number. Here you can\njust count it.\n\n**A denial is a message to the model, not a log line.** When a hook refuses a call, the\nreason text you write is handed straight to the assistant. That makes the wording part of the\ninterface. “Denied by policy” teaches the model nothing; “denied: this path is outside the\nproject root” tells it what to do differently.\n\n## The method, which is the transferable part\n\nThree separate times during this work, a single run looked like a clean answer, and the control run destroyed it.\n\nThe clearest case: we installed a hook that granted permission, ran a command, and it worked\nwith no prompt. That looks like proof. Then we removed the hook, ran the identical command,\nand it *also* worked with no prompt. Nothing had been demonstrated — the command was going to\nsucceed either way, and the hook was irrelevant to the outcome we’d just observed.\n\n**The measured quantity is the difference between two runs, not the outcome of one.** All\nthree near-misses would have published a confident, wrong claim about somebody else’s\nproduct.\n\nThere is a second failure worth naming because it is embarrassing and easy to repeat. One measurement appeared to show a tool call bypassing our hook entirely — a dramatic result. It was not real. The command being tested shared a line with the command that cleared the log, so the hook faithfully wrote its record and the test then deleted it. The instrument was standing in its own output. When a result looks dramatic, suspect your instrument first.\n\n## What we don’t know\n\nTwo of the nine questions are still unanswered for Claude Code, and not for lack of trying.\nBoth need an action the tool will *prompt* about, and on the machine we tested, nothing\nprompted: an unrecognised command, the same with the sandbox disabled, and a file write\noutside every permitted directory all ran silently. A hook explicitly asking for a prompt\ndidn’t produce one either. Those cells need a differently-configured machine, not more\neffort, so they say **unknown** and explain why.\n\nTwo whole columns — Codex CLI and Copilot — are entirely unknown. We have not run the procedures there. Listing them with guesses would be worse than listing them empty.\n\n## Take it and check us\n\nThe parameter definitions and the results table are\n[in the ai2rules repo](https://github.com/sv-pro/ai2rules/blob/main/docs/GOVERNABILITY-INDEX.md).\nThe step-by-step procedures — what to type and what to look for, including the ones we ran\nand the ones we haven’t — are in\n[agentic-coding-lab](https://github.com/sv-pro/agentic-coding-lab/tree/main/artifacts/governability-probe).\nThe procedures are MIT; the index lives in a repo dual-licensed MIT or Apache-2.0. Copy\neither. None of it requires our software.\n\nEvery cell carries a date and a version number, because these products ship weekly and a result without them isn’t a result. A cell changes when somebody runs the procedure and publishes what they saw — including when what they saw contradicts us. That has already happened three times, and all three times it was us doing the contradicting.\n\nThat is the whole design. An index that has never cost its author anything isn’t measuring; it’s marketing.", "url": "https://wpnews.pro/news/what-your-coding-agent-lets-you-control-and-three-things-we-got-wrong", "canonical_source": "https://ai2rules.dev/blog/what-your-coding-agent-lets-you-control/", "published_at": "2026-08-08 00:00:00+00:00", "updated_at": "2026-08-15 07:43:07.510204+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "developer-tools"], "entities": ["Claude Code", "Antigravity"], "alternates": {"html": "https://wpnews.pro/news/what-your-coding-agent-lets-you-control-and-three-things-we-got-wrong", "markdown": "https://wpnews.pro/news/what-your-coding-agent-lets-you-control-and-three-things-we-got-wrong.md", "text": "https://wpnews.pro/news/what-your-coding-agent-lets-you-control-and-three-things-we-got-wrong.txt", "jsonld": "https://wpnews.pro/news/what-your-coding-agent-lets-you-control-and-three-things-we-got-wrong.jsonld"}}