I benchmarked a code graph against grep across 37 repositories

A developer benchmarked Kivgraph, a local open-source code graph for coding agents, against grep across 37 repositories. The graph matched grep's accuracy (28/29 exact answers) while reducing token usage by ~5.95x, though grep still won on 5 questions when identifiers were rare and known. The developer also introduced 'find_by_intent' for concept-based search, which improved correctness from 7/24 (grep) to 17/24 when combined with model knowledge and repository context.

I've been working on Kivgraph, a local open-source code graph for coding agents. There are already plenty of code search and code graph tools around, so I wasn't very interested in proving that "graphs are better than grep". grep is extremely good when you know what you're looking for. What I wanted to test was something narrower: Can a resolved code graph answer structural questions with the same accuracy while making an agent read significantly less code? I put together 29 questions across 37 repositories written in Go, TypeScript, Rust, Python and Dart. The questions had hand-written ground truth and covered things like: The baseline was intentionally boring: grep for the relevant identifiers, then read the matching files. The results: | Kivgraph | grep + reading | | |---|---|---| | Exact answers | 28/29 | 28/29 | | Returned tokens | 35,961 | 267,980 | | Median context reduction | ~5.95x | — | The accuracy was basically identical. The context usage wasn't. Kivgraph returned around 36k tokens in total versus roughly 268k for grep + reading. But grep still won on 5 of the 29 questions, usually when the identifier was rare and already known. That result actually changed how I think about the tool. A graph shouldn't replace grep. If you already know the name of the thing you're looking for, grep is often exactly the right tool. The graph becomes useful when the question is structural. One of the things I wanted to avoid was building relationships by matching identifiers. Two unrelated methods called Handle shouldn't become connected just because they happen to share a name. For Go, TypeScript and Rust, Kivgraph resolves relationships using: go/types rust-analyzer Dart uses the Dart Analysis Server. Python is intentionally more conservative. When Kivgraph can't prove a relationship using a semantic analyzer, the built-in fallback can report it as CANDIDATE rather than presenting it as an EXACT relationship. That distinction matters when an agent is asking questions such as: "Who actually calls this?" or: "What could break if I change this symbol?" Text search can show occurrences. A resolved graph can tell you which occurrences represent actual relationships. While testing this, I kept running into another problem. Sometimes the coding agent knows exactly what it's looking for conceptually, but has no idea what the codebase calls it. For example: "Where is the code that decides whether a failed request should be retried?" The implementation might use retry . Or it might call the same concept requeue , backoff , reschedule , or something project-specific. That's what find by intent is for. The agent asks the graph what the code does, Kivgraph ranks the symbols and files that are likely to implement it, and once it has an entry point the agent can switch to the resolved graph for callers, references, dependencies or blast radius. There are no embeddings and no model calls inside Kivgraph. The coding agent itself already has a pretty good idea of which implementation words might represent a concept, so it can optionally pass those as hints. I ran a separate 24-question benchmark where the question deliberately didn't contain an identifier from the answer file: | Approach | Correct | |---|---| | grep | 7/24 | | intent only | 6/24 | | intent + likely code vocabulary | 11/24 | | intent + vocabulary + repository | 17/24 | The interesting part for me wasn't that intent search "beat grep". Plain intent didn't. What mattered was combining what the model knows about the problem with what Kivgraph knows about the codebase. The workflow I've ended up with is roughly: natural-language question → find a likely entry point → traverse resolved relationships Kivgraph indexes multiple repositories into the same graph. Every result includes the repository, path, qualified name and line range, so an agent can use the output of one query directly in another. That means questions like: "Which other repository consumes this?" don't require manually searching every repository in the workspace. This was one of the main reasons I started building it in the first place. It's not a replacement for grep. It's not a replacement for the language server. It's not another model sitting between your coding agent and the codebase. And it doesn't turn uncertain relationships into confident ones just to make the graph look more complete. The goal is mostly to give coding agents a smaller, structured view of the codebase when reading a pile of files would be wasteful. Everything runs locally and Kivgraph itself doesn't require an API key or external model. Current language support is Go, TypeScript, Rust, Python and Dart. I'm currently adding Java and C . The benchmark harness, ground truth and captured responses are public. The corpus itself isn't. So the exact benchmark can't be independently reproduced without running the harness against another corpus. That's the biggest limitation of the numbers above, and one I'd rather state explicitly than hide behind the benchmark. If anyone wants to run it against another large codebase, I'd be very interested in seeing where the results fall apart. Kivgraph is open source: https://github.com/Luqueee/kivgraph https://github.com/Luqueee/kivgraph Docs and full benchmark: I'm also curious how other people are handling this with coding agents: where do you draw the line between grep/file reading and a persistent structural index?