A code graph makes claims grep can't, including wrong ones

A developer audited graphify, a tree-sitter-based code graph tool that trended on GitHub with roughly 131,000 stars, by checking every call edge in a shipped example graph against the source lines it cites. Using a 36-line script on the rsl-siege-manager repo at commit 6085fd6, the check found 184 of 1,142 call edges failed at least one of three validity tests, and 136 of those failures carried the EXTRACTED tag meant to indicate the edge was read directly from source. Most failures traced to nodes that multiple files claim to contain, such as ten identically named test helpers folded into a single node with 124 edges.

The most connected function in a code graph I opened this week has 124 edges. It doesn't exist. It is ten Python test helpers called make siege , one per test file, plus two TypeScript makeSiege helpers and three lines that define something else entirely, all folded into one node. The graph's report ranks it first among "your core abstractions." I found that by checking every call edge in the graph against the line it cites. That check only works because a code graph does something grep and embeddings never do. graphify https://github.com/Graphify-Labs/graphify was on GitHub trending this week, with 1,434 stars in a day and about 131,000 in total. It is a Python CLI plus a /graphify skill for Claude Code, Codex, Cursor and about twenty other assistants. It parses code with tree-sitter, with no LLM and no embeddings for the code part, and writes a graph.json of functions, classes and files joined by edges like calls , imports and inherits . You ask it things with graphify query , explain and path . Every edge is tagged EXTRACTED read straight from the source or INFERRED resolved by graphify . Here is the claim I want to argue. An agent has three ways to find out what a function calls, or what calls it: mark changelog seen calls require member session , at a file and line, with a tag saying how sure it is. Only the graph makes claims. That is the whole value, because a claim is something an agent can act on without reading twenty files. It is also the risk, because a wrong claim comes out in the same format as a right one, tag and all. I didn't install graphify. It is a Python tool, graphify install registers a skill, and graphify claude install adds a hook to the assistant's settings. I don't run those for a post. I didn't need to. The repo ships worked examples with their real output, and one of them is a real public codebase: glitchwerks/rsl-siege-manager https://github.com/glitchwerks/rsl-siege-manager , a FastAPI backend, a React frontend and a Discord bot in one repo. The case study pins commit 6085fd6 and commits the graph.json graphify built from it in May: 1,886 nodes, 3,876 edges. So I cloned the app at that commit and wrote a 36-line Node script. Every calls edge carries the file and line of the call, so for each one it reads that line and asks three questions: js for const e of g.links.filter l = l.relation === 'calls' { const src = node e.source , tgt = node e.target const name = tgt.label.replace /\ \ $/, '' .replace /^\./, '' const text = lineOf e.source file, e.source location const faults = if family src.source file == family tgt.source file faults.push 'wrongLanguage' if src.source file == e.source file faults.push 'callerFromOtherFile' if new RegExp ^| ^\\w ${esc name }\\s < .test text faults.push 'nameNotOnLine' } Is the caller in the same language as the callee? Is the caller defined in the file where the call happens? Does the cited line actually call something by the callee's name? It ran in under half a second, with identical output both times. php edges pass py<- ts caller elsewhere name not on line EXTRACTED 989 853 83 57 84 INFERRED 153 105 34 33 7 958 of the 1,142 call edges pass all three checks. 184 fail at least one, and 136 of those are tagged EXTRACTED , the tag that means "read directly from the source." Most of the failures share one cause: 145 of the 184 touch a node that two or more files claim to contain. Here is one such node with four of its six edges, read from the raw JSON: markChangelogSeen frontend/src/api/changelog.ts L12 <-- contains changelog.ts EXTRACTED <-- contains changelog.py EXTRACTED -- calls require member session EXTRACTED backend/app/api/changelog.py:L89 -- calls ChangelogStatusResponse INFERRED backend/app/api/changelog.py:L95 The real markChangelogSeen is a two-line TypeScript function that sends a POST. The Python endpoint it hits, mark changelog seen , has no node of its own. Two of its calls now belong to the TypeScript function, because the graph merged the two by name. The rest of the failures are name matching gone wide: 22 calls to Python's built-in list point at const list = screen.getByTestId "feature-list" in a TypeScript test. That happened to a lot of code. Of the 769 top-level Python functions in the graphed files, 200 have no node in their own file, and 164 of those share a name, give or take case and underscores, with a node somewhere else. 121 nodes are claimed by two or more files. Six of the committed report's top ten god nodes are helpers merged across files, led by make siege at 124 edges, make member at 92 and make position at 85. The case study's review saw that ranking and concluded that test factories pile up edges because every test uses them. A reasonable reading of the report. One command says otherwise: python $ grep -rln "def make siege " . | wc -l 10 That is the difference between evidence and a claim. grep showed ten functions. The graph had already decided there was one. The merges came from the dedup pass, which matched labels by fuzzy similarity, a sensible thing for doc concepts and a wrong one for code. Issue 1205 https://github.com/Graphify-Labs/graphify/issues/1205 , opened June 9, reported a single --update dropping about 1,300 nodes from a 6,000-node graph. It was fixed the next day: code nodes are now keyed by ID only and never label-merged. Version 0.9.0 put the full path into every node ID. Cross-language INFERRED calls are dropped at build time. The current release is 0.9.93. I read those fixes in the source. I did not run the current version, so I can't say what it builds from this repo today. The committed graph has not been rebuilt since May, and the docs still point at the worked folders as output you can check. What interests me is the changelog itself. It reads like a field guide to wrong edges: short names like log and find collecting hundreds of spurious calls 543 , an npm import landing on an unrelated Python file as an EXTRACTED edge 1638 , three phantom edges that were the only bridge between 2,409 Python nodes and 1,403 TypeScript nodes 1749 . Each got fixed. The class of bug stays, because resolving a call means deciding which of several same-named things it meant, and that is always a judgment. Treat every edge as a citation, not a fact. The current graphify explain prints the call site next to each edge, so checking one is a single read. Do that read before acting on any edge that would change code. For "who calls this" before a rename or a delete, use the graph for the shortlist and grep for the completeness. A graph that misses a caller gives no sign of it. grep's noise is visible: you read it. Watch the install's defaults, too. The always-on hook tells the agent it "MUST run graphify query " before grepping. The same message says to read raw files once the graph has named the lines. Keep that second half, because that read is what makes the first half safe. The limits of my check: it only catches faults that show on the cited line, so a call bound to the wrong same-language twin passes, and 184 is a floor. And it is one repo, graphed five months ago. The project's own code benchmark, 82.0% key-fact coverage against 70.8% for a grep-and-read baseline, rests on six questions. When your agent's tool says "A calls B," what does it check before it believes it?