Do code comments help AI coding agents? What deleting 168,000 lines measured

Neither keep code comments for the agent's sake nor strip them for it. In a test published on 13 October 2026, deleting 168,000 lines of comments from a WordPress plugin left Claude Code's work unchanged across 12 runs, while removing the repository's AGENTS.md and docs cost 2.6 of 23 points. My recommendation for a team: delete comments that restate the code, keep the ones that say why or point to another file, and move conventions into AGENTS.md. That holds for a current frontier model, a mainstream language and a repository with tests.

Do code comments help AI coding agents? #

Code comments made no measurable difference to an AI coding agent's output in a controlled test on a real codebase. Roberto Aranda, writing on Automattic's OpenStation blog, deleted every comment from OpenStation, a WordPress plugin, and had Claude Code with Claude Opus 5.5 build the same two features on both versions, three times each, scored by a blind judge.

His write-up gives line counts that put the comments at 22% of the whole repository and 31% of the plugin's source code. Every run passed the project's lint, type checks and tests. All six runs of the harder task also passed a hidden test and found a PHP function that silently drops unknown keys, the kind of trap a comment should help with.

Means of three runs each, from the experiment page. Aranda's conclusion: "For the agent, code comments don't make a difference to the quality of the code it writes. They don't help, and they don't hurt."

Comments cost 22% of tokens on the small task and none on the larger one #

Comments cost an agent tokens in proportion to the share of comment lines in the files it opens. On OpenStation's small task, the version without comments used 22% fewer tokens and 27% less money. On the larger task the difference vanished, and single runs ranged from 7.01 million to 10.64 million tokens on the same code.

Aranda's warning about that spread: "If I had picked one run from each branch, I could have written two opposite blog posts." The page limit matters too. Claude Code's Read tool returns at most 25,000 tokens per call by default, per its tools reference, so a file that is a third comments holds fewer lines of code per page.

The vendors already tell their models to comment less. Anthropic's prompting guide suggests "Only add comments where the logic isn't self-evident." The prompt for OpenAI's Codex models in the Codex repository says "Usage of these comments should be rare." Developers report that agents comment anyway, and most of OpenStation's comments were written by agents, by Aranda's account.

Three policies and what each commits the team to #

A team can keep every comment, strip every comment, or sort them by what they carry. The table shows what each policy commits the team to on the evidence above. The third policy is the one I would pick, because it keeps the comments that carry intent and removes the ones that only cost tokens.

AGENTS.md carried what the comments did not #

The same experiment page has the result that decides where knowledge should live. With the comments kept and only docs/, AGENTS.md and CLAUDE.md removed, the harder task's score dropped from 21.3 to 18.7 out of 23. In a run that removed the comments and the guidance together, the lost points were on conventions AGENTS.md spells out: every run put its tests in a new file instead of extending an existing one, and one did not report running the build.

Removing the guidance also cut tokens by 33% to 47%, so the context cost sits in documents the agent is told to read, not in inline comments. Part of the score drop comes from a criterion for updating docs that cannot pass once the docs are gone; without it the gap narrows to 19.3 against 18.7 out of 21. The layer guide for AGENTS.md, CLAUDE.md and skills covers which instructions go where.

A second study suggests which comments earn their place, though as a hypothesis rather than a measurement. SonarSource researchers ran 660 Claude Code trials with Claude Sonnet 4.6 (arXiv 2605.20049) and found code cleanliness did not change the pass rate. They argue: "A one-line # keep in sync with metrics/launch_rate.py sitting above a function can redirect which files the agent touches," while "banners, paraphrases of the line below, or stale TODOs, inflate the input-token count without meaningfully changing the agent's trajectory." SonarQube is their product, and the paper defines clean code by its rules.

Risk: an agent doing the strip deletes the comments worth keeping #

A strip removes the wrong comments too. OpenStation's stripped branch needed a follow-up commit restoring the comments the tools depend on, such as lint directives and plugin headers; about 800 such lines remain. A developer on Hacker News described the subtler loss: an agent replaced "a specific comment why this fencepost here needs special consideration" with "Add one to index".

A strip-everything rule also misses why the comments are there. Several developers say agents over-comment, and at least one reports they keep commenting even when told not to, so a no-comment policy needs a check in CI rather than a line in a prompt. uncomment, a tree-sitter tool built "to clean up AI-generated code drowning in explanatory comments", keeps lint directives and TODOs by default.

Risk: wrong comments mislead both models and reviewers #

A missing comment costs little; a wrong one costs more. In a MITRE study of four older models reading an assembler module, comprehension quiz scores rose from 84% without comments to 90-96% with them, and fell from 73% to 61% when every comment was wrong. Replacing 20% of them with wrong ones had a negligible effect.

The effect reaches people too: changes that left a comment inconsistent with its code were about 1.5 times as likely to introduce a bug in one study of repository history. And comments steer AI reviewers. The ALIBI paper fooled LLM vulnerability detectors with adversarial comments in over 90% of 125 real bugs. A team that runs an AI reviewer is trusting every comment in the diff.

Risk: the evidence is small and the comments were agent-written #

The OpenStation test ran one model on two feature tasks, three times each, scored by a Claude judge. Aranda says it is "enough to rule out a big effect, not a small one." One reply on Hacker News named the bigger gap: the comments were mostly written by agents, which carry the same context the agent has when it reads them. "I would be more curious for a test with human written comments."

Debugging tasks, which lean on why a line exists, were not tested. Neither were weaker models: SonarSource dropped Claude Haiku 4.5 from its study because its pass rate was too low to read differences in token use. Where comments helped, in the MITRE study, the models were older and the language was assembler.

What would change the recommendation #

Four results would change it. A repeat with human-written comments, where the comments hold knowledge the code does not. A test with a share of subtly wrong comments, which a commenter on the OpenStation thread proposed. Bug-fix tasks rather than new features. And runs on a cheap or open model, where the code carries less of the context by itself.

Run the comparison on your own repository first: it takes an afternoon and about $31 of agent spend at OpenStation's averages. Strip comments on a branch and check the build output or test suite is unchanged. Count lines with scc, pick two real tasks, write the rubric and a hidden test before any run, and run each version three times with the same model. Add a third arm with AGENTS.md and docs removed to see which layer carries the weight. One run per arm misleads.

What I would do #

• If your team runs a frontier model on a mainstream stack with tests: delete comments that restate the code, keep why-comments and cross-file pointers, and do not run a blanket strip.
• If conventions live in comments: move them intoAGENTS.md , since that is the layer whose removal cost points.
• If an invariant must hold: write a test for it. OpenStation's hidden test could not see the PHP sanitizer, so only the judge's rubric checked that the key survived a reload.
• If you ban new comments: enforce it with a lint check, because a prompt line does not hold.
• If you run an AI reviewer: treat comments as input it can be steered by, and keep them accurate or out.
• If your codebase is a rare language or your agent runs a small model: keep the comments until you have run the afternoon test yourself.

The context engineering move covers the rest of what goes in the agent's context, and the AI tools developers run beside their agents are in the catalog.

New posts by email #

About one email a week with what we published. No account needed.

Which of these do you use? #

Nobody on Stackness runs AGENTS.md yet. Tick what you use and be the first.

Play Lonny Eagle Amateur Skater 4A pixel art skateboarding game. Signing up opens it straight away.Sign up to play