{"slug": "i-read-every-comment-in-my-codebase-every-wrong-one-had-the-same-shape", "title": "I Read Every Comment in My Codebase. Every Wrong One Had the Same Shape.", "summary": "A developer audited all ~2,000 lines of comments in the Coffer codebase and found 16 wrong comments, each describing something outside its own file. The errors fell into three species: roadmap state that arrived, a second case appearing, and drifted numbers. The developer used an AI coding agent to read comments against code and confirmed every defect before editing.", "body_md": "*I read all ~2,000 lines of comments in a codebase against the code they describe. Sixteen were wrong. Every single one of them was wrong in the same way — and it is not the way I expected.*\n\nI work on [Coffer](https://getcoffer.org), a file store that encrypts everything in the browser before it uploads. It is a heavily commented codebase: roughly 2,000 comment lines against 19,500 lines of C# and TypeScript, about one line in ten, in around 670 distinct comment blocks across 225 files.\n\nThat is a deliberate choice, and it has an obvious failure mode. A comment can be wrong; code can't be wrong in the same way, because code is what runs. A wrong comment is *worse* than no comment, because a reader trusts it. So the standing worry with a codebase like this is that at any moment some unknown fraction of those 2,000 lines is quietly lying, and nobody knows which lines.\n\nSo I measured it. Every comment block, read against the code it claims to describe, in one pass. An AI coding agent did the reading — 2,000 lines of prose against 19,500 lines of code is a tedium problem more than a difficulty problem, and this is exactly the shape of tedium a model is good at. Every defect it flagged was confirmed against the referenced source before anything was edited, and two of them needed work outside the editor: one required computing a probability, one required checking the live production host.\n\nThe result, in six commits:\n\n| Outcome | How many |\n|---|---|\n| Comment blocks read | ~670 |\nCorrected — the comment said something the code does not do |\n16 |\nDeleted — the comment said nothing the code doesn't already say |\n5 |\n| Verified correct, left untouched | everything else |\n\nAbout 2.4% wrong. Lower than I feared. But the interesting result isn't the rate.\n\nHere is the finding, and it held without a single exception across all sixteen:\n\n**Every wrong comment was describing something outside its own file.** Not one comment sitting directly above the code it documents was wrong.\n\nFour of the sixteen, to show the shape:\n\n**A deploy script's header comment** described its own sync step, 240 lines below. It claimed the script deletes removed files with `rsync --delete`\n\n. The body of the script explains at length that `rsync --delete`\n\nis exactly what you must *not* do there — it would delete files that live on the server and not in the repository — and the script actually diffs a per-deploy manifest so it can only ever remove what it previously put there. A reader skimming the header would have come away believing the precise opposite of the rule the body exists to enforce.\n\n**A server-side Plan.cs** described the\n\n**A client-side VaultPage.tsx** described the\n\n**A comment in Program.cs** explained why database migration is wrapped in a retry, by describing a\n\nThree species, and all sixteen fit one of them:\n\n| Species | What it is | Example |\n|---|---|---|\nRoadmap state that arrived |\n\"until X lands\", \"the future billing webhook\", \"currently only the welcome email\" |\n`SetPlan` called the billing webhook \"future\"; it had been live for weeks |\nA second case appeared |\nThe comment names the only case that existed when it was written | Six comments said \"Google\" after sign-in became provider-agnostic |\nA number drifted |\nA measurement, stated once, never re-measured | A build plugin claimed it strips \"roughly 1.2 KB\" of HTML comments; it strips 3,202 bytes |\n\nEvery one of them was **true when written**. Not one was sloppy. This is not a story about people writing bad comments.\n\nAdding a second sign-in provider is a normal feature. It touched the auth controller, the startup configuration, a button component. It did not touch the account page. It did not touch the user manager's logging comment. It did not touch two test files.\n\nAnd yet, the moment it merged, six comments in files it never opened became false — because each of them said \"Google\" where the code underneath had quietly become \"any external provider.\" No diff showed that. No test failed. `git blame`\n\non those comments still points at the commit that wrote them, correctly, months earlier.\n\nThat is the whole mechanism. **A comment about local code is checked constantly** — by everyone who edits the line beneath it, in the diff where the change lives. **A comment about remote code is checked by nobody**, because the change that falsifies it happens in a file where the comment isn't visible. The referent moves; the comment stays.\n\nThe measurement species is the same failure with a longer fuse. Here's one, in full, from a recovery key generator:\n\n```\n// Before:\n// Refill in batches so we don't call getRandomValues once per character; a few draws\n// land in the rejected tail, so oversample slightly to usually finish in one batch.\n\n// After:\n// Refill in batches so we don't call getRandomValues once per character. The batch is\n// exactly `length` bytes — no oversampling — and 8 of the 256 byte values (3.1%) fall in\n// the rejected tail, so one batch yields all 32 characters only about a third of the\n// time. The loop just draws another batch until enough survive; correctness never\n// depends on a batch sufficing.\n```\n\nBoth halves of the original were false. There was no oversampling — the batch is exactly `length`\n\nbytes for `length`\n\ncharacters. And \"usually finish in one batch\" is (248/256)^32 = **36%**, which is not usually. The loop was always correct; it just draws again. But you cannot catch that by reading. You catch it by doing the arithmetic, which nobody does to a comment they're scrolling past.\n\nThe honest accounting, from the same audit.\n\n**The cost is real, and smaller than the argument against comments assumes.** Sixteen defects in ~670 blocks, accumulated over the entire life of the project. Fixing them took one evening. That is the true maintenance bill for a decade-dense commenting habit in a codebase this size, and it is not a large number.\n\n**But the cost is not evenly distributed, and that's what makes it manageable.** It is concentrated almost entirely in comments with a remote referent. If you know that, you know where to spend the evening. An audit that reads every comment equally — which is what I did, and I don't recommend repeating it — spends most of its time confirming that comments beside their own code are fine, which they reliably are.\n\n**The worst case is genuinely bad, though.** The deploy script header wasn't merely stale, it was *inverted*: it advertised the exact practice the rest of the file exists to forbid. A reader who trusted it and \"restored\" `rsync --delete`\n\nwould have wiped server-side files that exist nowhere in the repository — on the next deploy, silently. This is the strongest form of the anti-comment argument and it is correct: a wrong comment is worse than no comment, because it converts your reader's diligence into a weapon. Being 97.6% accurate is not much comfort when the 2.4% is load-bearing. This is also the one claim in the post with evidence behind it beyond my own repository: [a 2024 study](https://arxiv.org/abs/2409.10781) of code-comment inconsistency found that inconsistent changes are around **1.5x more likely** to produce a bug-introducing commit than consistent ones. Wrong comments don't just mislead readers in the abstract; they show up in the defect record.\n\n**And the payoff is concentrated too — in the same place.** Look at what those 654 correct blocks were doing. The most valuable one in the whole codebase is that same deploy script, which documents three specific things that were tried and *broke* — including why `rsync --delete`\n\nis forbidden. That knowledge exists nowhere else. Not in the code (the code is the fix, not the fault), not in the tests, not in a commit message anyone will find in two years. The comments carrying the highest value are almost always the ones explaining a **rejected** alternative or a **non-obvious constraint**, and those are exactly the comments most likely to name something outside their own file — the other layer, the number that was measured, the machine. The maintenance burden and the payoff live at the same address.\n\nYou can see the same distinction in what got deleted. Five blocks went, and all five were the same kind of thing: `// Add services to the container.`\n\nabove the line adding services to the container, `// Configure the HTTP request pipeline.`\n\nabove the pipeline configuration, a scaffolding link to the OpenAPI docs, two commented-out dead declarations. Pure restatement, generated by a project template, surviving purely because nobody ever deleted them.\n\nWhereas these stayed, all of them one-liners:\n\n```\n// For EF Core.\n// Hours; 0 = no expiry.\n// OWASP's 2023 minimum\n```\n\nDelete those and you lose the only record of *why*. \"Comments should explain why, not what\" is advice everybody has heard and it is completely correct; what the audit adds is that the *what* comments are the harmless ones. They're useless, they dilute, they should go — but they are never what lies to you. **The comments carrying real information are the same comments that can rot.** There's no version of this where you get the value without the exposure.\n\nI would not run this audit again wholesale. It cost an evening and 94% of the reading confirmed things that were never in danger. What I'd do instead, and what the finding actually supports:\n\n**1. When writing a comment, notice when it points somewhere else.** If a comment names another file, another layer, another service, a number, or a machine, it has a referent you don't control and a maintenance cost the local ones don't have. That's not a reason to avoid writing it — those are the valuable ones. It's a reason to write it so it fails loudly: name the file, name the symbol, state the number with its units and its date, so the next reader can check it in ten seconds instead of wondering.\n\n**2. Grep for time-bound language, occasionally.** `currently`\n\n, `for now`\n\n, `not yet`\n\n, `until … lands`\n\n, `the future`\n\n, `TODO`\n\n. Every \"roadmap state that arrived\" defect announced itself with one of these words. This is a five-minute sweep with a high hit rate, and it's the highest-value habit in this entire post.\n\n**3. Review comments that make cross-file claims the way you'd review an assertion.** A comment saying \"the sweep runs daily\" is a claim about another file's behaviour. In review, that deserves the same \"is that still true?\" you'd give a magic number. A comment saying \"this is case-insensitive to match the server\" is checkable in ten seconds, by someone who has the diff open.\n\n**4. When a feature generalizes something, grep for the old name.** The single most productive five seconds of the entire audit would have been `grep -ri google`\n\non the day the second provider shipped. Six of sixteen defects, all findable, all in files that feature never touched. Every time you turn one case into two — one provider into any provider, one currency into many, one region into several — the old name is sitting in comments somewhere, and it is now a lie.\n\n**5. Verify numbers by computing them, not by reading them.** The two best finds in the audit were the recovery-key probability and the production host's clock. Neither was catchable by reading. One needed arithmetic, one needed leaving the editor entirely. If a comment states a measurement, it can only be checked by measuring.\n\nOne codebase, one project, 225 files, sixteen defects. That's a small sample from a single habitat, and the habitat matters: this codebase has one primary author, which means no comment ever went stale because two people disagreed about what the code was for. In a larger team I'd expect a whole species this audit couldn't produce — comments that were never true, only believed.\n\n**And none of this is new territory, which I should be explicit about rather than let you discover in the comments.** Comment rot is one of the better-studied things in software engineering research. [Fluri et al.](https://link.springer.com/article/10.1007/s11219-009-9075-x) found that about 90% of comment changes happen in the same revision as the code change — the deferred remainder being exactly where inconsistency comes from. [Wen et al.](https://dl.acm.org/doi/10.1109/ICPC.2019.00019) mined 1.3 billion AST-level changes across 1,500 systems to build a taxonomy of the inconsistencies developers actually fix. There is a whole detection subfield downstream of that.\n\n**My central category already has a name, too.** [Jabrayilzade et al.'s taxonomy of inline comment smells](https://link.springer.com/article/10.1007/s10664-023-10425-5) lists eleven, and one of them is **non-local** — comments referencing parts of the code that are not nearby. That is recognisably the thing this whole post is about, named before I got here.\n\nWhat I could not find is anyone claiming what this post claims: that locality *predicts* drift — that the non-local ones are where essentially all of it lives, and the local ones don't drift at all. In that taxonomy \"non-local\" is a comprehension smell: harder to read, harder to verify. The [follow-up detection work](https://arxiv.org/abs/2504.18956) explicitly does not claim those comments go stale more often. And the large inconsistency studies categorise drift by *what* went stale — application logic, design, maintenance — not by where its referent lived. Two searches came up empty, which is weak evidence of novelty rather than proof.\n\nThe local/remote split is also partly definitional, and I want to be honest that I noticed it *after* the fact rather than predicting it. \"Describes something outside its own file\" is a category I drew around sixteen defects once I had them in front of me. It's a clean line and it held perfectly, but sixteen is a small number to draw a law from. So the honest positioning is narrow: **they named the smell; I watched one codebase for its whole life and found the drift sitting entirely inside it.** A hypothesis with a good first result, not a finding.\n\nAnd a comment sitting beside its own code can absolutely still be wrong — it can misdescribe intent from the day it was written. This audit couldn't have detected that, because the check for a local comment is \"does this match the code beneath it,\" and a comment that was born wrong matches nothing in particular. What the audit shows is that local comments don't *drift*. Not that they're true.\n\nThere's one defect I found and deliberately did not fix. A test is named:\n\n```\nDelete_ShouldDeleteAccount_WhenGoogleOnlyUserTypesTheirOwnEmail\n```\n\nSame stale assumption as the six comments — it is not a Google-only path, it's the path every external provider takes. But it's in the *name*. Renaming it is a code change, not a comment fix, and it didn't belong in a commit series that touched nothing executable.\n\nIt's the better illustration of the whole point, though. Comment drift is at least confined to comments, where the fix is free and the blast radius is a confused reader. The same drift in an identifier is a refactor, and it's the reason people who identify their symbols after a single case end up living with the wrong word forever. Your comments and your names go stale by exactly the same mechanism — a second case appears somewhere else — and only one of them is cheap to correct afterwards.\n\n*Coffer is at getcoffer.org. The comments are, as of this week, accurate — a claim with a known shelf life, which is roughly the point of the article.*", "url": "https://wpnews.pro/news/i-read-every-comment-in-my-codebase-every-wrong-one-had-the-same-shape", "canonical_source": "https://dev.to/coffer/i-read-every-comment-in-my-codebase-every-wrong-one-had-the-same-shape-22gc", "published_at": "2026-08-18 12:00:00+00:00", "updated_at": "2026-08-18 12:14:27.517701+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["Coffer", "AI coding agent"], "alternates": {"html": "https://wpnews.pro/news/i-read-every-comment-in-my-codebase-every-wrong-one-had-the-same-shape", "markdown": "https://wpnews.pro/news/i-read-every-comment-in-my-codebase-every-wrong-one-had-the-same-shape.md", "text": "https://wpnews.pro/news/i-read-every-comment-in-my-codebase-every-wrong-one-had-the-same-shape.txt", "jsonld": "https://wpnews.pro/news/i-read-every-comment-in-my-codebase-every-wrong-one-had-the-same-shape.jsonld"}}