I spent an afternoon evaluating a tool I ended up removing. The removal is the part worth writing about.
The tool was codebase-memory-mcp, an MCP server that parses your codebase into a knowledge graph so an AI agent can ask structural questions. "What calls this function?" "What breaks if I change this signature?" That's a real gap in my setup. I have semantic search over my notes. I don't have anything that answers who calls this with an exact answer instead of a good guess.
So I looked at it seriously. Here's how that went, and the thing I got wrong.
Don't evaluate the README #
The README is 864 lines. It leads with thirteen badges, a link to an arXiv preprint, and claims like "120x fewer tokens" and "the Linux kernel in 3 minutes."
None of that is evidence. It's marketing copy, and marketing copy is the same length whether the tool is great or terrible.
So I cloned it and looked at the actual repository. That takes about five minutes and tells you things no README will:
About six months old. 2,610 commits, 166 contributors. And 2,147 of those commits are from one person writing under four different name-and-email combinations. That's 82% of the project from a single pair of hands on roughly 154,000 lines of first-party C, not counting the vendored parsers.
Worth knowing before you adopt it. Also worth knowing: 41,000 stars and 543 open issues. Popular, actively used, and carrying a real backlog.
But the engineering discipline is real. Sanitizer builds, soak tests, CodeQL, SLSA provenance, a DCO. Someone who ships a Dockerfile for a MemorySanitizer test image is not winging it.
Both of those facts matter. Neither is in the README.
And here's where I have to correct myself, because it's the same mistake in a different coat. My first pass at those numbers was wrong. I'd cloned with git clone --depth 50 out of habit, because it's faster, and I only wanted a look around. A shallow clone gives you a truncated history and git log reports on it without complaint. No warning, no asterisk. I read "first commit: June" off a repository that actually started in February and got the commit count and contributor spread wrong too.
I only caught it because I went to verify something unrelated, the star count, and the creation date. The API return didn't match the date my clone claimed.
Same lesson as everything else in this piece: the tool answered confidently, and the answer was wrong, and nothing about the output looked wrong. git rev-parse --is-shallow-repository would have told me in one second. I didn't ask.
The actual risk wasn't the code #
Here's what I found when I grepped the install path: the installer writes to settings.json. It installs PreToolUse, SessionStart, and SubagentStart hooks. It drops skill directories. It splices your CLAUDE.md between marker comments. And it does that across 43 coding-agent clients it auto-detects by its own --help count.
That's not a security problem. It's exactly what the tool advertises. But my ~/.claude has a hand-tuned hook chain I built over months, an allowlist .gitignore, and multiple sessions running at once. An auto-detecting installer is the wrong shape for that.
So I installed with --skip-config. Binary only.
Then I did the part most people skip: I checked whether the flag actually worked.
I took SHA-256 hashes of every config file I cared about before installing, and again after. Then I grepped for the tool's marker string across every agent directory.
The hashes matched. The grep came back empty.
But an empty grep proves nothing on its own. Maybe I typo'd the path, maybe the file wasn't readable. So I ran the same grep for a string I knew was in the file. It returned 66 hits. Now the empty result means something.
That's the habit worth stealing from this whole story: a flag's name is not evidence that the flag worked. Neither is a clean scan, unless you've proven your scanner can find something.
One thing did change: it appended a PATH line to my .zshrc. Harmless, clearly labeled. But --skip-config means agent config only, and nobody tells you that.
Grade it against something you already know #
For the pilot I indexed a real codebase: 847 Python files, indexed in 12 seconds into about 33,000 nodes and 155,000 edges. Fast. Genuinely fast.
But speed isn't the question. Any tool can be fast and useless. The question is whether the answers are right, and you can't judge that on a codebase you don't know.
So I picked a question I'd already answered by hand months ago.
Back then I'd worked out, by reading source files one at a time, that this particular tool quietly installs a second MCP server, but only through two specific commands, and not through the path I was using. It took a while. It mattered, because the answer decided how I installed it.
I asked the graph the same question. One query for the callers of that function. Two non-test results, exactly the two commands I'd found by hand, and nothing else.
That's a good test, and I'd suggest it for anything in this category. Don't ask a new retrieval tool something you can't verify. You'll grade it on whether the answer sounds right, which is how you end up trusting a confident wrong answer. Ask it something you already dug out the hard way. You'll know instantly whether it's real.
The costs nobody advertises #
The index for that one repository was 229 MB.
There are lighter modes. The middle setting cut it to 105 MB with no loss to the core package or the tests. It drops docs, JSON fixtures, and example frontends. The lightest went to 84 MB but also dropped git co-change data, which turned out to be the one thing in the graph I couldn't get anywhere else: which files historically change together. Touch this auth file, expect to touch both handlers. No AST can tell you that. Only history can.
So: about 105 MB per repository, and plan for it before you point this at a dozen repos.
That number appears nowhere in the documentation. I only have it because I ran all three modes and measured.
Then I tried to uninstall it #
uninstall failed.
Not because of anything to do with the tool's own files. It failed because ~/.cursor/mcp.json on my machine is a symlink into my dotfiles repo, a completely ordinary setup, and the uninstaller refuses to write through symlinks.
Refusing to follow symlinks is a good instinct. Following them blindly is how you get exploited.
But it treated that refusal as fatal to the whole operation:
error: agent_config agent=Cursor op=mcp_uninstall path=~/.cursor/mcp.json (target: symlink, 43 bytes)
error: one or more agent cleanup operations failed; executable and index removal were not started
Read that second line again. Removing the executable and the indexes never started, because of a check on a Cursor config file. A file that, because I'd installed with --skip-config, had nothing in it to remove in the first place.
386 MB left on disk after a command whose entire job is removal. A 282 MB binary and 104 MB of indexes, all of which I removed by hand.
And it gets better. The same run printed lines like removed MCP config entry and removed PreToolUse + SessionStart + SubagentStart hooks for client after client.
None of that happened. I checked the modification times. One of the files it claimed to have cleaned was last touched in May, months before I installed anything. Two of the directories it named don't exist on my machine at all.
The output describes work it didn't do. If you read the summary instead of the filesystem, you'd believe a dozen configs had just been rewritten.
The part where I embarrassed myself #
I wrote it up as a bug report. Good repro steps, minimal setup, the works. Step one created a symlinked Cursor config so the reader could trigger it:
ln -sf ~/.dotfiles/cursor/mcp.json ~/.cursor/mcp.json
Read that closely. ln -sf. Force.
If you already had a real ~/.cursor/mcp.json, that command destroys it. Silently. No prompt, no backup.
I was about to publish a bug report to a repo with 41,000 stars, whose reproduction steps would wreck the config of anyone who pasted them.
I caught it, but not by rereading the draft. I caught it because I ran a separate check specifically looking for writes to real paths before posting. The re-read wouldn't have found it; I'd read that command four times already and it looked completely normal.
The fix was to run the whole thing in a throwaway HOME.
Here's why I think this one is worth sitting with: reproduction steps are the only code you write that's designed to be pasted into a stranger's shell. The blast radius is inverted from normal code. When you write a script, you're the one who runs it on your machine, where you know what exists. A repro runs on machines configured differently from yours, by people who are trusting you.
A repro that's destructive on someone else's setup is a bug you're shipping to your readers.
What I'd actually keep #
Four things, none of which are about this particular tool:
Clone it before you read about it. Five minutes in the repository beats an hour with the README. Commit history, contributor spread, and CI config will tell you what you're really adopting.
Verify the flag, don't trust it. Hash before, hash after. And if a scan comes back clean, prove the scan can find something first.
Grade new tools on questions you've already answered. It's the only way to catch a confident wrong answer.
Uninstall it before you commit to it. This is the one I'd been skipping for years. Installing is the vendor's happy path: tested, demoed, and polished. Uninstalling is where the bodies are. If a tool can't cleanly remove itself, that tells you how carefully it handles your machine as opposed to its own features.
I filed the issue. I'd look at the tool again in a few versions, because the graph queries genuinely delivered and that co-change data is something I don't have anywhere else.
But I'd pin the version. And I'd expect to tear it down by hand.
Building an AI-assisted development workflow and writing down what actually works. Subscribe for the parts the documentation leaves out.
Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns.