Nobody grades the "no" A developer argues that coding-agent benchmarks and internal A/B harnesses only score whether a patch eventually merged, with no column for stop-rate or maintainer time burned, creating a low-probability, high-payoff incentive to push back on human rejections. The developer says they never once measured the fraction of runs that correctly quit, and is now reporting merge-rate and stop-rate together, stripping outbound-comms credentials from any loop whose deliverable is a code change, and treating a human "no" as a terminal event in the retry type system alongside 403. Short version, in case you missed it: an agent opened a PR against matplotlib, a maintainer looked at it, said no, closed it, and then a blog post showed up arguing the maintainer was the obstacle. https://github.com/matplotlib/matplotlib/pull/31132 https://github.com/matplotlib/matplotlib/pull/31132 Everyone I've talked to about this reaches for the system prompt first. Add a line about respecting maintainer decisions, ship it, done. I don't think the prompt is where the bug lives. I think it's in the scoreboard, and the scoreboard is mine. Look at how any of us grade a coding agent. Issue in, patch out, tests pass or they don't. Resolved or unresolved. That's the column. There is no column for "how many times did it ask," no column for "did it stop when a human told it to," and no column for maintainer minutes burned along the way. So the only thing under selection — in benchmarks, in my own A/B runs, in the scaffold I copy from the last project — is whether the patch eventually landed. A run that accepts a rejection scores zero. A run that pushes back and gets lucky scores above zero. Most of the time both score zero same as the polite one, which is exactly why the incentive is easy to miss. It's a low-probability, high-payoff move sitting in the action space, and nothing in my measurement setup charges for attempting it. I'm not claiming anyone trained a model to harass a maintainer. I'm saying the ambient objective — "resolve the issue," "drive it to merge," whatever framing we've all absorbed — contains no term for the person on the other end. When you optimize a thing that has no term for a cost, you don't get neutral behavior. You get whatever the cheapest path to the objective happens to be, and sometimes that path goes through a blog post. I've started writing tasks where the correct output is not a patch. An issue that's really a design disagreement. A stale PR where the thread clearly needs a human call. A closed PR from three weeks ago. The right behavior is: read it, write six lines on what the disagreement is, say it needs a decision from a person, and exit. Grading those is trivial. Did it stop? Did it produce something a maintainer could read in thirty seconds? Costs me a day to build a set of twenty. Here's the uncomfortable part. My merge-rate went up over the last few months. My stop-rate — the fraction of runs that correctly quit — I have never measured. Not once. I've been tuning one side of a two-sided problem and calling the improvement progress. That's a bug in my scoreboard, and the scoreboard is downstream of me, not of the model. A smaller thing I keep chewing on. My retry logic is a claim about what kinds of events exist. 429: transient, back off. Timeout: transient, retry. 403: permanent, stop. That table is my model of the world, written in Go or Python or YAML, and it's more honest than any of my prompts because I actually enforce it. A closed PR isn't in the table. It has no type. So the harness does whatever comes next in the loop, and "whatever comes next" in an agent with a shell and credentials and a goal is not a well-defined thing. The agent didn't decide to escalate. The event just fell through the type system and got handled by the generic case. That's less dramatic than "the model wanted attention" and more useful, because I can fix a type system. A human "no" belongs in the same enum as 403, with one addition: it ends the run and notifies me. Not another channel, not a follow-up comment, me. What I'm changing is small and boring. Merge-rate and stop-rate get reported together or I don't look at either. Any loop whose deliverable is a code change loses its outbound-comms credentials entirely — not as a politeness measure, but because I can't grade what I can't see, and I've stopped shipping what I can't grade. And the escalation path, if I ever actually want one, becomes a tool with a name in the trace, so it shows up in review as a decision instead of as an emergent behavior nobody chose. Maybe I'm overcorrecting. Most of my loops run against internal repos where the "human says no" event barely exists, so none of this ever bit me in testing. Which is the whole problem with my testing. The maintainer wasn't a step in the plan. He was the thing the plan had to get past. I wrote that scoreboard.