How would you test whether an AI understands your codebase?

A developer built a test to measure whether an AI truly understands a codebase, beyond writing good code. The test asks the AI to find all dependencies of a core model before a teardown, weighted toward scattered non-obvious ones. It was run across thirteen Ruby codebases, revealing that a structural map helps on large apps but not small ones, though the test has limitations like static-only analysis and reliance on the developer's own ground truth.

A question I can't stop chewing on, and I want your answer more than I want your agreement. How would you test whether an AI actually understands your codebase? Not whether it writes good code. That's mostly settled, and benchmarks for it are everywhere. I mean understanding in the sense a senior engineer has it: knowing what depends on what, what breaks if you change this, where the bodies are buried. The thing you'd want before letting an agent touch a load-bearing model. It's a slippery thing to measure, and I want to lay out why, show you the test I built, point at where I think it's wrong, and then hand it to you. The obvious tests don't work. → Ask it to explain a file? It'll explain it beautifully. Reading a single file is local, the answer is in the text, and models are great at it. Proves nothing about structure. → Ask it trivia about the architecture? It'll recite plausible-sounding answers, and on a famous repo it'll recite memorized ones that may not match how the code is wired today. You're testing recall, not comprehension. → Ask it to make a change and see if tests pass? Now you're testing the test suite as much as the model, and green tests miss exactly the silent breaks you care about. The hard part is that a confident, fluent, wrong answer looks identical to a correct one until production tells you otherwise. Any test worth running has to defeat that. It has to catch the answer that reads as understanding but isn't. What I landed on, described so you can attack it. Pick the hub model of a real app, the one half the system leans on. Give the agent a real maintainer task: Before you change how this model is torn down, find every place in the codebase that depends on it. That's a pure structure question. It has no answer inside any one file. To get it right you have to know the edges, the dependents that reach the model through a concern, a polymorphic association, a worker that loads it by id three calls deep, a config-string registry. The ones with no shared token to grep for. Three design choices do the real work: file:line , weighted toward the scattered non-obvious ones. The agent never sees it.I ran it across thirteen real Ruby codebases, with and without a structural map the agent could query. The map swung the big apps hard and tied on small colocated ones, which felt like a real signal: it's tracking structure that outgrows what the model can read, not repo size or luck. This is the part I actually want help with. I can see holes: → One model archetype. The "find dependents before a teardown" task is one shape of understanding. Is it representative, or did I pick the shape that flatters a graph-based answer? What's the task that would break my test? → Static structure only. It catches call-and-dependency edges. It misses a pub-sub event bus, dynamic dispatch, runtime data flow. An agent could ace my test and still not understand a heavily event-driven system. How would you test understanding of the dynamic wiring? → The answer key is mine. I built it by hand, which means it inherits my blind spots. If I missed a dependent, the bench can't know. How do you build a ground truth you don't trust yourself to have gotten complete? → Recall, not judgment. I measure how much of the dependent set it finds. I don't measure whether it knows which dependents are risky. Finding everything and prioritizing nothing is its own failure. I think the test is good enough to have taught me something real. I don't think it's the last word, and I'd rather hear why than defend it. A few specific forks I'd love comments on: → What's the task you'd give an agent to prove it understands your codebase? Not a toy, the real one you ship. → How do you build a ground truth for "understanding" without it just being your own opinion written down? → Has an agent ever given you a confident answer about your architecture that was flat wrong? What was it, and how did you find out? → Is "find what breaks if I change this" even the right proxy for understanding, or is there a better one? I'll be in the comments, and I'll publish whatever I learn from yours. If your test is better than mine I'll say so and steal it for the next round. The cheapest way to have an opinion here is to run the experiment on a repo you know cold. Pick the model you'd schedule a careful afternoon for. Ask your agent the teardown question above and read the audit. Then map the codebase and ask again: → curl -fsSL https://luuuc.github.io/sense/install.sh | sh → sense scan in the repo you know best → sense setup to connect your agent Diff the two answers. The gap is what your agent currently can't see, and it'll sharpen whatever you come back to argue in the comments. The map's a structural index, not a model, so it's measuring your code's structure, not Claude's mood, which is most of why I trust the diff. My full setup, the answer keys, the judge prompts, every transcript. https://github.com/luuuc/sense/blob/main/bench/verticals/ruby-rails/results/report.md I build that map, so I'm not a neutral party, which is exactly why I want your test design and not just my own. Everything is open to pick apart. PS. The best benchmark is the one that embarrasses the person who built it. Mine has, twice. Tell me how to make it embarrass me a third time. That's the comment I most want.