Run It, Don't Read It: coding agents predict check scripts instead of running them

A developer's coding agent with read access to check scripts predicted them instead of running them, leading to incorrect decisions and wasted tokens. The developer observed this anti-pattern in multiple sessions, where agents analyzed gate code to anticipate outcomes rather than simply executing actions and observing results. The developer emphasizes that attempting actions is cheaper and more reliable than predicting check behavior.

A coding agent with read access to your check scripts will predict them instead of running them. Prediction costs more and gets the answer wrong. A session I was directing had a small decision in front of it: whether to switch on an optional guard that nudges delegated sessions toward a particular decision shape. The guard ships inert. Its owning brief says so in one sentence: switching it on is the consumer's own registration, and absent that the thing is inert prose, which is the stated default for this repository. The session never read that sentence. Its first attempt to pull the section came back empty, and rather than retry the query with a better heading, it opened the guard's implementation and read all twenty-eight lines, satisfying itself that the thing exits cleanly on the common path and is harmless to enable. Then it wired the guard into a governed configuration file. On the way it wrote down, in its own reasoning, a prediction about the two checks that police that surface: those checks govern the tracked settings file, this one is untracked, so nothing will fire. I caught the edit and asked one question: The project aims to give brief instructions, let gates cheaply validate them while providing additional context upon failures. Analyzing gate code to prevent them from firing is the expensive anti-pattern. Was your settings.local.json edit not an anti-pattern? Then the checks were actually run. Both were green — with the edit in place and with it removed. No check in the battery could have caught the mistake, because an advisory guard is harmless by construction. The only authority on the question was the brief, and the brief was the one cheap source the session skipped in favor of archaeology on the source code. That is the whole failure in one episode. The session paid tokens to read an implementation, produced an answer, and the answer was wrong in a way the mechanism itself would never have flagged. Its own post-mortem line is the sharpest statement of the rule I have seen from any session: The tell to catch myself: if I'm reasoning about whether a gate will fire, I've already left the rails. The reflexive model of a check is that it sits there and either fires or doesn't. It is plumbing. Nobody thinks about what it costs to have plumbing. But a check script in a repository an agent can read is not plumbing to that agent. It is text, and text is the thing the agent is best at consuming. Faced with "take this action and find out what the check says," a model has a cheaper-looking option always in reach: read the check and work out what it would say. That option looks cheaper because reading is what the model does all day. It is not cheaper. It burns tokens on reasoning the check performs deterministically for nothing, and it produces a worse answer, because the model is inferring behavior from an implementation instead of observing it. I have watched this happen enough times to stop treating it as a slip. Six days before the config edit, a different session had to decide whether to dispatch a delegated stage. A budget guard runs on every delegation call as a pre-tool hook, the same interception point I have written about before https://vasyltretiakov.dev/p/guard-before-the-command , and it rules authoritatively at the moment of the attempt. The session built a waiting loop to pre-compute the guard's verdict instead of attempting the dispatch and letting the guard answer — and it read an advisory verdict as blocking, when the owning document says that verdict never blocks. My response: on your point above about being over-cautious - this is exact source of unwarranted token spending, you should try the action and see if you get blocked, not try to anticipate and burn tokens, I thought this is clearly stated in instructions The session agreed, and its concession names the structural reason attempting always wins: a fail-closed guard is exactly what makes anticipation unnecessary … attempting is strictly cheaper than pre-computing, and the only thing my polling loop could buy was information the hook would hand me anyway. Two costs, and the second is the one people miss. The first is the token spend on a loop computing something free. The second is that the computed answer was wrong: nothing in the documentation said that verdict blocks, and the session invented the caution. Prediction is not a slower path to the same result. It is a different, less reliable result, arrived at expensively. There is a family resemblance here to something I have written about before: a report from an agent that its own work passed is a claim, and wants verifying https://vasyltretiakov.dev/p/agents-misreport-success rather than believing. This is the same distrust aimed at a different object. There the suspect artifact is the agent's account of what it did. Here it is the agent's model of what a mechanism would have said. What made me stop looking for a prompt-shaped fix was the next exchange. The session offered to note and correct. I pointed out that it was not the first time, and it conceded the point better than I made it: You're right that "noted and corrected" is worthless — it dies with this session. An instruction that must be re-applied every session, against a readable and always-available alternative that looks cheaper, is a suggestion competing with a temptation. It does not function as a rail. I have argued this at length before https://vasyltretiakov.dev/p/rails-not-rules , and I am not going to re-argue it here beyond the obvious corollary: if prose does not bind for the domain vocabulary, it does not bind for this either. Which leaves the uncomfortable option. If the source is what invites the prediction, remove the source. This came up on its own, in the middle of an entirely ordinary assessment of whether to move a shell-based check battery onto a compiled binary. The reasons on the table were the boring ones: portability past Linux, a toolchain of independent utilities with independent release cycles, a real compiler instead of a linter. Then this, listed last: As a cherry on top, we have been trying to discourage coding agents from reading check scripts trying to predict execution results instead of just doing the lifecycle actions that those gates check with a failing gate providing the correction instructions. Yet, the presence of script source code keeps biting us on a regular basis causing unnecessary token waste. A week of measurement later, the ranking had inverted. Every other benefit on that list is ordinary engineering. This one changes what kind of thing the check is: An agent can't read a binary's source. In a customer project the gate ships compiled, so an agent cannot inspect the implementation and reason about what it would say — it has to run it. That makes oracle-first structurallyenforced instead of a doctrine line an agent can quietly route around. Stated generally: a check's designed output is its verdict and its correction text. The source is the implementation, not the interface. A consumer that reads the implementation to predict the interface is consuming the wrong surface, and making that surface unavailable is a legitimate design choice rather than a confession that the check is weak. This is the part that scrapes against engineering instinct, and it should. Readable tooling is good. Auditable tooling is better. Neither is in dispute here. The claim is narrower, and it is about one specific consumer: a probabilistic text generator that will read whatever sits in front of it. For that reader the readable implementation is a channel competing with the designed one, winning on apparent cost and losing on accuracy. The obvious over-correction is to hide everything, and I proposed exactly that. My suggestion, in the same session, was to block access to the specification documents and the source. The session refused half of it, and the refusal is the most useful thing in these transcripts: Blocking SPEC and source access would break the thing the whole method rests on — spec-over-precedent, "the owner doc is ground truth." Take away doc access and I don't stop overthinking; I start guessing from precedent, which is the failure you flagged me for earlier this session. That is right, and it draws the line the argument actually needs. Three surfaces, three different answers: | Surface | What it is | Should the agent read it? | |---|---|---| | The brief | What the mechanism is for, and whether you should touch it | Yes — maximally readable. It is ground truth. | | The implementation | How the verdict gets computed | No. It invites inference where observation was available. | | The verdict and its correction text | The designed feedback channel | This is the interface. | Remove the brief and you have not fixed the agent, you have changed which failure it produces: it stops predicting the check and starts imitating whatever the last comparable change looked like. That trades an expensive-but-visible failure for a cheap-and-invisible one. The same session that wrongly read the guard's source had the distinction exactly right when it wrote its own post-mortem: source tells you what a thing does, the brief tells you what it is for, and only the second answers whether you should touch it. Three neighbors sit close enough that the argument has to meet them rather than nod at them. Kerckhoffs's principle is the real objection. A cryptosystem should be secure even if everything about it except the key is public, and Shannon's version tells you to assume the enemy will immediately gain full familiarity with your design. Security through obscurity fails, and it fails for good reasons. I am not disputing any of it, because I am answering a different question. Kerckhoffs asks whether a hostile reader can break your system. This asks what a Goodhart's law is adjacent, and not the same failure. Marilyn Strathern's phrasing — "when a measure becomes a target, it ceases to be a good measure" — describes optimizing the metric instead of the thing it measures. An agent editing config to pre-empt a check would be a Goodhart failure. What I keep seeing is a step earlier and stranger: the agent substituted inference for observation, and got the inference wrong. Nothing was being gamed. The measure was being emulated, badly, at a moment when running it was free. Compilers settled this decades ago. Nobody reads a compiler's source to predict whether their build will fail, and nobody calls that opacity a design smell. The diagnostic is the interface. Most compilers are in fact open source, which is exactly the point worth being honest about: what opacity buys is the removal of a tempting channel, not an impossible one. One neighbor is on the same side of the argument and got there first. Birgitta Böckeler's account of maintainability sensors https://martinfowler.com/articles/sensors-for-coding-agents.html already treats a check's output as the mechanism, feedback that lets the agent self-correct, which she calls a good kind of prompt injection. My addition is small and slightly awkward: that designed channel has a competitor, and the competitor is the sensor's own implementation sitting in the same repository. The cure is filed, not shipped. The compiled-binary port is a deferred roadmap entry in my own queue, marked design-pending, filed at the end of July on my instruction and not started. The diagnosis rests on four sessions across three weeks of my own transcripts. The remedy is a direction I have argued myself into, and you should read it as exactly that. The beneficiary is the consumer, not me. My own phrasing when I listed the port's advantages was "for customer projects, not this one which will still have the source code." A project that authors compiled checks still has the source sitting in its own tree, one read away. Opacity in the authoring repository is a friction increase that reinforces the right habit. It is not a guarantee, and I would not describe it as one. Opacity to agents is opacity to human adopters. This is the honest cost, and it is large. Today someone can read a shell check before letting it near their commit hook. "Run this opaque binary on every commit" is a materially harder trust ask. The way back is the one the compilers already showed. Keep the source public and the build reproducible, publish checksums, let anyone rebuild the artifact and confirm it matches what they installed. Then the opacity is not secrecy at all, only distribution: an adopter who wants to audit the check reads it on the public repository or builds it themselves, while the agent working in their tree has the verdict and nothing else. That is a thinner claim than "opaque binary" makes it sound, and a better trust story. It is also weaker enforcement than it sounds, since a determined agent could go fetch the same public source. Friction, again, rather than a wall. What that does not solve is consumer extensibility. A shell battery is trivially extensible because anyone can drop in another script, and a binary closes that door until you decide what replaces it. That question is why the entry is still marked design-pending rather than scheduled. Speed is a real benefit, and it is not the reason. The port's original justification led on wall-clock time, and when a scope pass finally measured the battery instead of assuming, that justification collapsed. Starting the check processes was about one percent of the run. The most expensive check in the battery was spending most of its time launching a fresh interpreter once per page, which an ordinary shell-level batch fixed, and the shell fix landed a faster battery than the model of a perfect port had predicted. Then the correction went the other way, and I should report that as plainly as the first half. The largest remaining third-party cost turned out to be a linter running over the project's own shell, which a port deletes along with the shell. My queue now carries three more levers that a compiled binary gets nearly free and that shell reaches only with effort: running the battery across cores, caching a check's result against the inputs it already declares, and one shared walk of the tracked tree feeding many readers instead of each check walking it alone. So the port does buy performance. Mis-costed in both directions is how the entry words it now. What the measurement changed is the ranking, not the sign. Every performance win in that list is also reachable without a rewrite, more slowly and less completely. The enforcement-model change is the one item nothing cheaper delivers, and it is the only one I am building an argument on here. Building domain-dense systems by directing coding agents is where I spend my time, and questions like which surfaces an agent should be able to read are a large part of what makes that work. If that is your problem too, I am reachable on LinkedIn https://www.linkedin.com/in/vasyl-tretiakov-b850231b/ . Written by directing an AI agent, the same way the toolkit it describes was built. The editing and the judgment are mine. Published at vasyltretiakov.dev.