Your tests share your blind spots. Readers don't.

On June 14 I shipped a resolver with a green test suite. On September 1 I published a post about it. On September 4, three days later, I filed an issue crediting three people from the comment thread, each of whom had found something my tests never would.

That is the whole hot take, so here it is up front: your tests share your blind spots. Strangers reading your code don't. Tests encode the questions you already thought to ask. Readers who didn't write the code ask different ones. For tools that AI agents depend on, where a wrong answer looks exactly like a right one, I now think the second group is worth more than the first.

This post is part dev diary, part argument. Everything in it comes from the public tracker of Infrawise (npm), the MCP server I build that gives coding agents context about live infrastructure. Mainly #132, #133 and #134.

An agent editing checkout.ts wants to know what calls that code in production and what the event looks like. The catch: deployed names almost never match source names. You write export const handler in checkout.ts; your stack deploys it as checkout-handler-prod. Something has to connect the two.

Infrawise has two linkers for that. One reads your Terraform or CDK and links by the declared handler path, labelled proven. The fallback, HeuristicLinker, normalizes names and compares them:

const STAGE_TOKENS = new Set([
  'prod', 'production', 'dev', 'development', 'staging', 'stage', 'test', 'qa',
]);
const NOISE_TOKENS = new Set(['handler', 'fn', 'func', 'function', 'lambda']);

export function normalizeName(raw: string): string {
  let s = raw.toLowerCase().trim();
  s = s.split('/').pop() ?? s;
  s = s.replace(/\.(ts|js|mjs|cjs)$/, '');
  let segs = s.split(/[-_.\s]+/).filter(Boolean);
  if (segs.length > 1 && STAGE_TOKENS.has(segs[segs.length - 1] ?? '')) segs.pop();
  segs = segs.filter((seg) => !NOISE_TOKENS.has(seg));
  return segs.join('');
}

checkout-handler-prod becomes checkout. The file checkout.ts becomes checkout. Match, link, label it inferred.

The linker loop at the time looked like this (trimmed, not current code):

for (const lam of lambdaNodes(graph)) {
  const target = normalizeName(lam.name);
  if (!target) continue;
  const matches = fns.filter(/* normalized name or file equals target */);
  if (matches.length !== 1) continue;
  links.push({ lambdaId: lam.id, functionId: matches[0].id, confidence: 'inferred' });
}

And the tests shipped alongside it had one I was quietly proud of:

it('does not link when multiple functions match (ambiguous)', () => { ... });

Several source functions match one deployed function? Refuse. I had thought about ambiguity. Tested it. Green.

Here is what I had not thought about, and so had not tested: the same question in the other direction.

Two deployed functions: checkout-handler-prod and checkout-dev. Both normalize to checkout. Each one, on its own, finds exactly one matching source function, checkout.ts, so each one links. The loop asks "does this deployment match exactly one function?" and never asks "does any other deployment match the same function?"

My test for ambiguity was the mirror image of my code. The code checked one direction; the test checked the same direction. They agreed with each other perfectly, and both were blind to the same case. That is not bad luck. It's structural: when the same person writes the code and the tests, the tests inherit the author's model of the problem, gaps included. The suite stayed green for 82 days.

In the comment thread on Your agent trusts the first match. Should it?, @anp2network read the published package, traced that both linkers returned only successes with no diagnostics channel, and made the point I built the fix around: whether a match was authored by the resolver or observed in the data has to be recorded at the moment the grouping is built. Normalization deletes handler, prod and dev, the exact tokens that might have told the two deployments apart. After that, nothing downstream can get them back.

The subtle part: the link might even have been correct. checkout-handler-prod and checkout-dev could be the same code in two stages. Or two unrelated handlers that share a prefix. The resolver couldn't know, and it didn't say it couldn't know.

@prpatel05 argued that ambiguous: true is a measurement, not a decision, and a response should be unusable until something narrows it to one match. @kenwalger separated proven and inferred as answers to two different questions, and argued that a resolver needs permission to say "undetermined", or ranking forces it to manufacture certainty.

None of those three points were in my test file. None of them could have been. I would have had to already know them.

This is the part I didn't expect. The thread didn't stop at comments.

analyze_function already refused to pick one and named them as candidateLambdas. But the same response also sent this, unconditionally:

"triggers": []

Ken's issue put it exactly: that is "a claim about the world rather than a statement about what the tool could determine." An agent reading triggers: [] concludes the function isn't event-driven, and writes a handler with no idea what the event looks like. My tests checked that candidateLambdas appeared. None of them asked what triggers meant next to it.

Ken also argued against their own proposed fix, which is rare and useful:

Any caller doing result.triggers.map(...) breaks when the key disappears.

We shipped it anyway, with a reason code alongside so the key doesn't just vanish. A caller that crashes on a missing key is annoying. A caller that confidently writes the wrong handler because an empty array lied to it costs you an incident.

The rule now sits in the linker as a comment:

// A refusal is recorded, never encoded as absence: a Lambda with five matches
// and a Lambda with none must not produce the same output.
export interface LinkResult {
  links: LambdaCodeLink[];
  unresolved: UnresolvedLambdaLink[];
}

Every refusal carries a reason: no_match, multiple_functions (candidate functions listed), or multiple_lambdas (the other colliding deployment names listed). The collision check lives where the grouping is built, because that's the only place the deleted tokens still exist:

// normalizeName deletes the tokens that tell `checkout-handler-prod` from
// `checkout-dev`, so the collision has to be caught here, where the class
// is built: nothing downstream can recover the deleted tokens.
for (const [key, lams] of byKey) {
  if (lams.length > 1) {
    for (const lam of lams) {
      out.unresolved.push({
        lambdaId: lam.id,
        reason: 'multiple_lambdas',
        candidates: lams.filter((l) => l !== lam).map((l) => l.name),
      });
    }
    continue;
  }
  // exactly one deployment for this key: match as before
}

Asking about handler in checkout.ts now returns (trimmed):

{
  "function": "handler",
  "found": true,
  "matches": [{ "file": "/repo/src/checkout.ts", "accesses": [ ... ] }],
  "unresolvedLambdas": [
    { "lambda": "checkout-handler-prod", "reason": "multiple_lambdas", "candidates": ["checkout-dev"] },
    { "lambda": "checkout-dev", "reason": "multiple_lambdas", "candidates": ["checkout-handler-prod"] }
  ],
  "triggers": []
}

triggers is still empty here, because nothing linked, and now the reason sits next to it. The agent can name both deployments and ask which one you mean. When several deployments do link to one shared function, triggers is left out of the response entirely.

And the test file finally has the case it was missing: two deployments normalizing to one key, zero links, two refusals naming each other.

I went back through the tracker to check whether I was generalizing from one lucky thread. In June, one commenter on Why Infrawise Uses Deterministic Analysis Instead of an LLM, @ggle_in, raised three separate ideas that became issues #55, #56 and #57 on the same day. #55, surfacing how old the analysis is, shipped two weeks later and grew into the freshness block every tool response now carries.

Two posts, two threads, and between them most of the design changes I'm proudest of this year. In both cases the comments beat my tests, because the commenters had no stake in my model of the problem being right.

I'm not arguing tests are useless. The new collision test will stop this exact bug from coming back, and nothing a commenter says does that. Tests are how you keep a lesson. They're just a poor way to learn one.

It's also not free. Publishing internals with file paths and line numbers means people will find your mistakes in public. The heuristic now links less than it did: on an account where orders-prod and orders-dev really are the same code, the old version was right by luck and the new one refuses. And two points from #132 are still open. #140: the proven label comes from the handler declared in IaC, while the deployed handler is fetched and then thrown away, so where they disagree, proven sits on the stale one. #131: when several deployments link to one scanning function, a high-severity finding names only one of them, picked by iteration order.

Both came straight out of that thread (#131 filed alongside #132, #140 split out of it later), and neither had a failing test.

So here's the question I actually want answered: where did the last bug your tests couldn't have caught come from, a user, a reviewer, or a stranger reading your code?