{"slug": "i-benchmarked-a-code-graph-against-grep-across-37-repositories", "title": "I benchmarked a code graph against grep across 37 repositories", "summary": "A developer benchmarked Kivgraph, a local open-source code graph for coding agents, against grep across 37 repositories. The graph matched grep's accuracy (28/29 exact answers) while reducing token usage by ~5.95x, though grep still won on 5 questions when identifiers were rare and known. The developer also introduced 'find_by_intent' for concept-based search, which improved correctness from 7/24 (grep) to 17/24 when combined with model knowledge and repository context.", "body_md": "I've been working on Kivgraph, a local open-source code graph for coding agents.\n\nThere are already plenty of code search and code graph tools around, so I wasn't very interested in proving that \"graphs are better than grep\".\n\ngrep is extremely good when you know what you're looking for.\n\nWhat I wanted to test was something narrower:\n\nCan a resolved code graph answer structural questions with the same accuracy while making an agent read significantly less code?\n\nI put together 29 questions across 37 repositories written in Go, TypeScript, Rust, Python and Dart.\n\nThe questions had hand-written ground truth and covered things like:\n\nThe baseline was intentionally boring: grep for the relevant identifiers, then read the matching files.\n\nThe results:\n\n| Kivgraph | grep + reading | |\n|---|---|---|\n| Exact answers | 28/29 | 28/29 |\n| Returned tokens | 35,961 | 267,980 |\n| Median context reduction | ~5.95x | — |\n\nThe accuracy was basically identical.\n\nThe context usage wasn't.\n\nKivgraph returned around 36k tokens in total versus roughly 268k for grep + reading.\n\nBut grep still won on 5 of the 29 questions, usually when the identifier was rare and already known.\n\nThat result actually changed how I think about the tool.\n\nA graph shouldn't replace grep.\n\nIf you already know the name of the thing you're looking for, grep is often exactly the right tool.\n\nThe graph becomes useful when the question is structural.\n\nOne of the things I wanted to avoid was building relationships by matching identifiers.\n\nTwo unrelated methods called `Handle`\n\nshouldn't become connected just because they happen to share a name.\n\nFor Go, TypeScript and Rust, Kivgraph resolves relationships using:\n\n`go/types`\n\n`rust-analyzer`\n\nDart uses the Dart Analysis Server.\n\nPython is intentionally more conservative. When Kivgraph can't prove a relationship using a semantic analyzer, the built-in fallback can report it as `CANDIDATE`\n\nrather than presenting it as an `EXACT`\n\nrelationship.\n\nThat distinction matters when an agent is asking questions such as:\n\n\"Who actually calls this?\"\n\nor:\n\n\"What could break if I change this symbol?\"\n\nText search can show occurrences. A resolved graph can tell you which occurrences represent actual relationships.\n\nWhile testing this, I kept running into another problem.\n\nSometimes the coding agent knows exactly what it's looking for conceptually, but has no idea what the codebase calls it.\n\nFor example:\n\n\"Where is the code that decides whether a failed request should be retried?\"\n\nThe implementation might use `retry`\n\n.\n\nOr it might call the same concept `requeue`\n\n, `backoff`\n\n, `reschedule`\n\n, or something project-specific.\n\nThat's what `find_by_intent`\n\nis for.\n\nThe agent asks the graph what the code does, Kivgraph ranks the symbols and files that are likely to implement it, and once it has an entry point the agent can switch to the resolved graph for callers, references, dependencies or blast radius.\n\nThere are no embeddings and no model calls inside Kivgraph.\n\nThe coding agent itself already has a pretty good idea of which implementation words might represent a concept, so it can optionally pass those as hints.\n\nI ran a separate 24-question benchmark where the question deliberately didn't contain an identifier from the answer file:\n\n| Approach | Correct |\n|---|---|\n| grep | 7/24 |\n| intent only | 6/24 |\n| intent + likely code vocabulary | 11/24 |\n| intent + vocabulary + repository | 17/24 |\n\nThe interesting part for me wasn't that intent search \"beat grep\".\n\nPlain intent didn't.\n\nWhat mattered was combining what the model knows about the problem with what Kivgraph knows about the codebase.\n\nThe workflow I've ended up with is roughly:\n\nnatural-language question → find a likely entry point → traverse resolved relationships\n\nKivgraph indexes multiple repositories into the same graph.\n\nEvery result includes the repository, path, qualified name and line range, so an agent can use the output of one query directly in another.\n\nThat means questions like:\n\n\"Which other repository consumes this?\"\n\ndon't require manually searching every repository in the workspace.\n\nThis was one of the main reasons I started building it in the first place.\n\nIt's not a replacement for grep.\n\nIt's not a replacement for the language server.\n\nIt's not another model sitting between your coding agent and the codebase.\n\nAnd it doesn't turn uncertain relationships into confident ones just to make the graph look more complete.\n\nThe goal is mostly to give coding agents a smaller, structured view of the codebase when reading a pile of files would be wasteful.\n\nEverything runs locally and Kivgraph itself doesn't require an API key or external model.\n\nCurrent language support is Go, TypeScript, Rust, Python and Dart. I'm currently adding Java and C#.\n\nThe benchmark harness, ground truth and captured responses are public.\n\nThe corpus itself isn't.\n\nSo the exact benchmark can't be independently reproduced without running the harness against another corpus.\n\nThat's the biggest limitation of the numbers above, and one I'd rather state explicitly than hide behind the benchmark.\n\nIf anyone wants to run it against another large codebase, I'd be very interested in seeing where the results fall apart.\n\nKivgraph is open source:\n\n[https://github.com/Luqueee/kivgraph](https://github.com/Luqueee/kivgraph)\n\nDocs and full benchmark:\n\nI'm also curious how other people are handling this with coding agents: where do you draw the line between grep/file reading and a persistent structural index?", "url": "https://wpnews.pro/news/i-benchmarked-a-code-graph-against-grep-across-37-repositories", "canonical_source": "https://dev.to/luqueee__/i-benchmarked-a-code-graph-against-grep-across-37-repositories-1n26", "published_at": "2026-08-28 14:43:00+00:00", "updated_at": "2026-08-28 15:19:56.309386+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": ["Kivgraph", "grep", "Go", "TypeScript", "Rust", "Python", "Dart"], "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-a-code-graph-against-grep-across-37-repositories", "markdown": "https://wpnews.pro/news/i-benchmarked-a-code-graph-against-grep-across-37-repositories.md", "text": "https://wpnews.pro/news/i-benchmarked-a-code-graph-against-grep-across-37-repositories.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-a-code-graph-against-grep-across-37-repositories.jsonld"}}