{"slug": "your-tests-share-your-blind-spots-readers-don-t", "title": "Your tests share your blind spots. Readers don't.", "summary": "A developer building Infrawise, an MCP server that gives coding agents context about live infrastructure, disclosed that a name-matching linker shipped with a green test suite for 82 days while carrying a structural blind spot: the ambiguity test only checked whether one deployment matched multiple source functions, never the reverse case where two deployments (checkout-handler-prod and checkout-dev) both normalize to the same source file. Three readers in the post's comment thread found the bug that the author's own tests could not, leading to issue #132 and pull request #133. The author argues that for tools AI agents depend on, external readers catch failure modes that author-written tests inherit blind spots around.", "body_md": "On June 14 I shipped a resolver with a green test suite. On September 1 I published a post about it. On September 4, three days later, I filed an issue crediting three people from the comment thread, each of whom had found something my tests never would.\n\nThat is the whole hot take, so here it is up front: **your tests share your blind spots. Strangers reading your code don't.** Tests encode the questions you already thought to ask. Readers who didn't write the code ask different ones. For tools that AI agents depend on, where a wrong answer looks exactly like a right one, I now think the second group is worth more than the first.\n\nThis post is part dev diary, part argument. Everything in it comes from the public tracker of [Infrawise](https://github.com/Sidd27/infrawise) ([npm](https://www.npmjs.com/package/infrawise)), the MCP server I build that gives coding agents context about live infrastructure. Mainly [#132](https://github.com/Sidd27/infrawise/issues/132), [#133](https://github.com/Sidd27/infrawise/pull/133) and [#134](https://github.com/Sidd27/infrawise/issues/134).\n\nAn agent editing `checkout.ts` wants to know what calls that code in production and what the event looks like. The catch: deployed names almost never match source names. You write `export const handler` in `checkout.ts`; your stack deploys it as `checkout-handler-prod`. Something has to connect the two.\n\nInfrawise has two linkers for that. One reads your Terraform or CDK and links by the declared handler path, labelled `proven`. The fallback, `HeuristicLinker`, normalizes names and compares them:\n\n``` js\nconst STAGE_TOKENS = new Set([\n  'prod', 'production', 'dev', 'development', 'staging', 'stage', 'test', 'qa',\n]);\nconst NOISE_TOKENS = new Set(['handler', 'fn', 'func', 'function', 'lambda']);\n\nexport function normalizeName(raw: string): string {\n  let s = raw.toLowerCase().trim();\n  s = s.split('/').pop() ?? s;\n  s = s.replace(/\\.(ts|js|mjs|cjs)$/, '');\n  let segs = s.split(/[-_.\\s]+/).filter(Boolean);\n  if (segs.length > 1 && STAGE_TOKENS.has(segs[segs.length - 1] ?? '')) segs.pop();\n  segs = segs.filter((seg) => !NOISE_TOKENS.has(seg));\n  return segs.join('');\n}\n```\n\n`checkout-handler-prod` becomes `checkout`. The file `checkout.ts` becomes `checkout`. Match, link, label it `inferred`.\n\nThe linker loop at the time looked like this (trimmed, not current code):\n\n``` js\nfor (const lam of lambdaNodes(graph)) {\n  const target = normalizeName(lam.name);\n  if (!target) continue;\n  const matches = fns.filter(/* normalized name or file equals target */);\n  if (matches.length !== 1) continue;\n  links.push({ lambdaId: lam.id, functionId: matches[0].id, confidence: 'inferred' });\n}\n```\n\nAnd the tests shipped alongside it had one I was quietly proud of:\n\n``` js\nit('does not link when multiple functions match (ambiguous)', () => { ... });\n```\n\nSeveral source functions match one deployed function? Refuse. I had thought about ambiguity. Tested it. Green.\n\nHere is what I had not thought about, and so had not tested: the same question in the other direction.\n\nTwo deployed functions: `checkout-handler-prod` and `checkout-dev`. Both normalize to `checkout`. Each one, on its own, finds exactly one matching source function, `checkout.ts`, so each one links. The loop asks \"does this deployment match exactly one function?\" and never asks \"does any *other* deployment match the same function?\"\n\nMy test for ambiguity was the mirror image of my code. The code checked one direction; the test checked the same direction. They agreed with each other perfectly, and both were blind to the same case. That is not bad luck. It's structural: when the same person writes the code and the tests, the tests inherit the author's model of the problem, gaps included. The suite stayed green for 82 days.\n\nIn the comment thread on [Your agent trusts the first match. Should it?](https://dev.to/siddharth_pandey_27/your-agent-trusts-the-first-match-should-it-350k), [@anp2network](https://dev.to/anp2network) read the published package, traced that both linkers returned only successes with no diagnostics channel, and made the point I built the fix around: whether a match was *authored* by the resolver or *observed* in the data has to be recorded at the moment the grouping is built. Normalization deletes `handler`, `prod` and `dev`, the exact tokens that might have told the two deployments apart. After that, nothing downstream can get them back.\n\nThe subtle part: the link might even have been correct. `checkout-handler-prod` and `checkout-dev` could be the same code in two stages. Or two unrelated handlers that share a prefix. The resolver couldn't know, and it didn't say it couldn't know.\n\n[@prpatel05](https://dev.to/prpatel05) argued that `ambiguous: true` is a measurement, not a decision, and a response should be unusable until something narrows it to one match. [@kenwalger](https://dev.to/kenwalger) separated `proven` and `inferred` as answers to two different questions, and argued that a resolver needs permission to say \"undetermined\", or ranking forces it to manufacture certainty.\n\nNone of those three points were in my test file. None of them could have been. I would have had to already know them.\n\nThis is the part I didn't expect. The thread didn't stop at comments.\n\n| When (UTC) | What | \n|---|---|\n| Sep 1, 09:09 | Post published | \n| Sep 4, 05:49 | I file #132: four points, all three commenters credited | \n| Sep 4, 17:57 | Ken's PR #133 lands: an optional `file` input for`analyze_function` (point 3) | \n| Sep 4, 18:14 | Ken files #134, found *while implementing* #133 | \n| Sep 4, 19:19 | v0.28.0 released with the `file` input | \n| Sep 4, 19:41 | #134 closed by the linker fix (points 1 and 2) | \n| Sep 5, 12:11 | v0.28.1 released with the linker fix | \n\n`analyze_function` already refused to pick one and named them as `candidateLambdas`. But the same response also sent this, unconditionally:\n\n```\n\"triggers\": []\n```\n\nKen's issue put it exactly: that is \"a claim about the world rather than a statement about what the tool could determine.\" An agent reading `triggers: []` concludes the function isn't event-driven, and writes a handler with no idea what the event looks like. My tests checked that `candidateLambdas` appeared. None of them asked what `triggers` meant next to it.\n\nKen also argued against their own proposed fix, which is rare and useful:\n\nAny caller doing `result.triggers.map(...)` breaks when the key disappears.\n\nWe shipped it anyway, with a reason code alongside so the key doesn't just vanish. A caller that crashes on a missing key is annoying. A caller that confidently writes the wrong handler because an empty array lied to it costs you an incident.\n\nThe rule now sits in the linker as a comment:\n\n```\n// A refusal is recorded, never encoded as absence: a Lambda with five matches\n// and a Lambda with none must not produce the same output.\nexport interface LinkResult {\n  links: LambdaCodeLink[];\n  unresolved: UnresolvedLambdaLink[];\n}\n```\n\nEvery refusal carries a reason: `no_match`, `multiple_functions` (candidate functions listed), or `multiple_lambdas` (the other colliding deployment names listed). The collision check lives where the grouping is built, because that's the only place the deleted tokens still exist:\n\n```\n// normalizeName deletes the tokens that tell `checkout-handler-prod` from\n// `checkout-dev`, so the collision has to be caught here, where the class\n// is built: nothing downstream can recover the deleted tokens.\nfor (const [key, lams] of byKey) {\n  if (lams.length > 1) {\n    for (const lam of lams) {\n      out.unresolved.push({\n        lambdaId: lam.id,\n        reason: 'multiple_lambdas',\n        candidates: lams.filter((l) => l !== lam).map((l) => l.name),\n      });\n    }\n    continue;\n  }\n  // exactly one deployment for this key: match as before\n}\n```\n\nAsking about `handler` in `checkout.ts` now returns (trimmed):\n\n```\n{\n  \"function\": \"handler\",\n  \"found\": true,\n  \"matches\": [{ \"file\": \"/repo/src/checkout.ts\", \"accesses\": [ ... ] }],\n  \"unresolvedLambdas\": [\n    { \"lambda\": \"checkout-handler-prod\", \"reason\": \"multiple_lambdas\", \"candidates\": [\"checkout-dev\"] },\n    { \"lambda\": \"checkout-dev\", \"reason\": \"multiple_lambdas\", \"candidates\": [\"checkout-handler-prod\"] }\n  ],\n  \"triggers\": []\n}\n```\n\n`triggers` is still empty here, because nothing linked, and now the reason sits next to it. The agent can name both deployments and ask which one you mean. When several deployments *do* link to one shared function, `triggers` is left out of the response entirely.\n\nAnd the test file finally has the case it was missing: two deployments normalizing to one key, zero links, two refusals naming each other.\n\nI went back through the tracker to check whether I was generalizing from one lucky thread. In June, one commenter on [Why Infrawise Uses Deterministic Analysis Instead of an LLM](https://dev.to/siddharth_pandey_27/why-infrawise-uses-deterministic-analysis-instead-of-an-llm-15fk), [@ggle_in](https://dev.to/ggle_in), raised three separate ideas that became issues #55, #56 and #57 on the same day. #55, surfacing how old the analysis is, shipped two weeks later and grew into the freshness block every tool response now carries.\n\nTwo posts, two threads, and between them most of the design changes I'm proudest of this year. In both cases the comments beat my tests, because the commenters had no stake in my model of the problem being right.\n\nI'm not arguing tests are useless. The new collision test will stop this exact bug from coming back, and nothing a commenter says does that. Tests are how you keep a lesson. They're just a poor way to learn one.\n\nIt's also not free. Publishing internals with file paths and line numbers means people will find your mistakes in public. The heuristic now links less than it did: on an account where `orders-prod` and `orders-dev` really are the same code, the old version was right by luck and the new one refuses. And two points from #132 are still open. [#140](https://github.com/Sidd27/infrawise/issues/140): the `proven` label comes from the handler *declared* in IaC, while the *deployed* handler is fetched and then thrown away, so where they disagree, `proven` sits on the stale one. [#131](https://github.com/Sidd27/infrawise/issues/131): when several deployments link to one scanning function, a high-severity finding names only one of them, picked by iteration order.\n\nBoth came straight out of that thread (#131 filed alongside #132, #140 split out of it later), and neither had a failing test.\n\nSo here's the question I actually want answered: where did the last bug your tests couldn't have caught come from, a user, a reviewer, or a stranger reading your code?", "url": "https://wpnews.pro/news/your-tests-share-your-blind-spots-readers-don-t", "canonical_source": "https://dev.to/siddharth_pandey_27/your-tests-share-your-blind-spots-readers-dont-524j", "published_at": "2026-09-24 05:52:04+00:00", "updated_at": "2026-09-24 06:00:18.619420+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols", "mlops"], "entities": ["Infrawise", "Sidd27", "HeuristicLinker", "GitHub", "npm", "Model Context Protocol", "dev.to"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-tests-share-your-blind-spots-readers-don-t", "markdown": "https://wpnews.pro/news/your-tests-share-your-blind-spots-readers-don-t.md", "text": "https://wpnews.pro/news/your-tests-share-your-blind-spots-readers-don-t.txt", "jsonld": "https://wpnews.pro/news/your-tests-share-your-blind-spots-readers-don-t.jsonld"}}