# Your next model upgrade won't close this gap > Source: > Published: 2026-07-09 06:31:34+00:00 There's a comfortable thing people say when they see an AI agent query a code map. "Nice crutch. For now." The logic underneath it is reasonable. Coding agents are young. Context windows are small and getting bigger. Models are dumb today and will be smart tomorrow. So a structural index, the thing that hands the agent a dependency graph it would otherwise have to reconstruct, looks like a patch over a temporary weakness. Wait two releases. The model will just hold the whole repo in its head and the map becomes a quaint workaround, like a spellchecker for someone who learned to spell. I build one of those maps [Sense](https://github.com/luuuc/sense). I went looking for the data that would kill it. I didn't find it. I found the opposite. What a map hands an agent is a *computed* fact. What a better model hands you is a more confident *guess*. No amount of model progress turns the second into the first, because the difference between them isn't a quality gap that closes with scale. It's a difference of kind. The rest of this piece is the two findings that forced me there. The claim at full strength, because a weak version is easy to knock over. A code map exists to compensate for what the model can't do yet. Today's agent greps, samples, and guesses at structure because it can't read the whole codebase at once. Tomorrow's agent reads all of it, reasons over all of it, and the guessing stops. Bigger windows plus better weights equal no more blind spots. The map is scaffolding you'll tear down once the building stands. If that's true, the right move is to skip the tool and wait. Both findings, in order. The benchmark ran the same task on thirteen real Ruby repos. Pick the hub model of an app, the `Inbox` , the `MergeRequest` , the `Spree::Order` , and ask the agent to find every place that depends on it before a teardown change. The non-obvious dependents, the ones scattered through concerns and workers and config-string registries, were pinned by hand against the source and kept hidden. The agent gets graded against that key, every citation checked to a real `file:line` . The headline arm ran on Claude Code with Opus 4.8. Not a budget model. The best coding model money could buy when this ran. Cold, on Chatwoot, it found 2 of the 11 scattered dependents. On a rerun of the identical prompt, 0. It missed nine that a maintainer has to check before touching that model. On GitLab, the largest open-source Rails monolith there is, it ground for five minutes per run and surfaced 2 of 16. The audit it wrote each time read as finished. It wasn't. It was two-sixteenths of the truth in a tidy, confident report. Handed the map, the same model, same prompt, same pinned commit, caught 11 of 11 on Chatwoot and 13 of 16 on GitLab. You can wave that off as one model having a bad day. The next part you can't. The same lift showed up on GPT-5.5, from a different lab, on different weights. Chatwoot baseline 0.47, with the map 0.82. Across the whole bench, every agent gained: Opus +0.26, Devstral +0.24, Qwen +0.18, Kimi +0.13, GPT-5.5 +0.13. Five models, three model families, US-frontier to China-open to EU-open. Same direction every time. This isn't a weak-model artifact. It isn't a single-lab artifact. The frontier needed the map too. A better model can't fix this one. When the map answers "what depends on `MergeRequest` ," it isn't being clever. It walked the call-and-dependency graph and returned the resolved set. 932 symbols, the same 932 every time you ask. It's a computation over a closed world. The answer is deterministic, and dead-code reasoning works because the world is closed. If it isn't in the graph, it isn't a caller. When the LLM answers the same question, it's inferring. It reads some files, pattern-matches against every Rails app in its training set, and produces its best guess. Run it twice on Chatwoot and you get 2, then 0. Same prompt. Different answer. That's not a bug in the model. That's what inference is. Now turn a better model loose on each side. A better model computes nothing new, because it was never computing. It infers. And a stronger inference engine infers more *confidently*, not more *correctly*. Watch where that goes wrong. On Discourse, the plain agent found a dependent of `Upload` and described its role backwards, named the class that listens for an upload as the one that renders it. On Solidus it called a promotion calculator the rule that decides eligibility. Both audits read beautifully. Both would send a maintainer to change the wrong thing. A smarter model states that kind of mistake with more authority, not less. The confident-wrong answer is the dangerous one, because you only catch it after you've trusted it. Scaling up the inference makes the prose better and the confidence higher. It does not turn a guess into a computation. That gap, between a thing computed and a thing inferred, does not shrink as inference improves. It's a category difference, not a quality difference. You don't close it by buying more inference. The moment it landed: the best model I could pay for, pointed at a famous open-source Rails app, writing a clean audit of a model's teardown, and quietly blind to nine of the things that would break. No flailing. No "I'm not sure." Just a composed, wrong-by-omission report, the kind you'd ship. The model wasn't failing at reasoning. It was reasoning perfectly over the half of the structure it could see. The other half never lived in the files it read. It lived in the edges between them, and you can't reason your way to an edge you never loaded. Whether this is worth adopting turns on one thing. If a code map were just a stopgap for weak models, you'd be betting on a losing horse and you should wait. It isn't, for three reasons that have nothing to do with which LLM is winning this quarter. → **It's repo-specific.** The map is built from *your* code at its current commit. No model memorized your private monolith. It can recite the public Rails API from training, sure, but the win on Rails sat entirely on the non-memorized query-compiler internals, the part that never made it into the weights. Your code is all non-memorized. The map is the only thing that has read it. → **It's always current.** It re-indexes as files change, as you switch branches, as a `git pull` lands. A model's knowledge is frozen at its training cutoff. The map's knowledge is frozen at your last save. One of those is months stale by definition. The other never is. → **It's LLM-agnostic.** It serves the graph over MCP, the wire your editor already uses. One index, every agent. Switch from Opus to GPT-5.5 to whatever ships next quarter and the map doesn't care. You are never stranded by a model swap, because the thing that knows your codebase isn't the model. A bet rides on one outcome. Infrastructure outlasts the outcomes. This is the second kind. The honest version of all this: the map's value isn't fixed. On a small, readable, colocated gem, a strong model reads the whole dependent set in one pass and the map adds nothing. One repo in the bench was a flat tie for exactly that reason. The boundary is real, and the small-repo half of the board is a tie because models keep getting better at reading. But the apps you actually ship aren't gems. They grow. And they grow faster than context windows do. The dependents that matter spread further apart as the codebase gets larger, which is precisely when the model can no longer hold them all. The boundary moves, every model release nudges it. It never closes, because the codebase is always on the other side of it, getting bigger. That's why "wait for the next model" isn't a plan. The next model moves the line a little. Your codebase moved it back further while you were reading this. Don't take the bench's word, or mine. The cheapest possible check is one you run yourself, and it takes a few minutes. Pick the model in your own app you'd schedule a careful afternoon for. Ask your agent cold, *"before I change how this model is torn down, find every place that depends on it."* Watch it grep and guess. Note the count. Then give it the map. → `curl -fsSL https://luuuc.github.io/sense/install.sh | sh` → `sense scan` in the repo you know cold → `sense setup` to connect your agent Ask again. Diff the two answers. If your repo is small and clean, you'll watch a tie, and that's a real result you can trust. Point it at the biggest app in your org and watch the gap open instead. The gap is the part of your own codebase your model can't currently see, and no upgrade on the calendar closes it for you. [The benchmark, the methodology, the raw data for all thirteen.](https://github.com/luuuc/sense/blob/main/bench/verticals/ruby-rails/results/report.md) I build Sense. That's exactly why I went hunting for the data that would sink it, and why all of it is open, the code, the harness, the pinned commits, the transcripts, the judge prompts. Disbelieve this and rerun it. *PS. If you think the next model release closes this, name the release and the repo and I'll run the teardown audit on it, cold then mapped, in public. I want to be wrong about this more than you want me to be. So far the data hasn't let me.*