{"slug": "the-uninstall-is-part-of-the-evaluation", "title": "The Uninstall Is Part of the Evaluation", "summary": "A hands-on evaluation of the codebase-memory-mcp MCP server found that the tool's installer writes to settings.json and installs PreToolUse, SessionStart and SubagentStart hooks across 43 auto-detected coding-agent clients, prompting the reviewer to install with --skip-config and verify the flag with SHA-256 hashes and a control grep. The reviewer also found the repository is about six months old with 2,610 commits and 166 contributors, but 2,147 commits (82%) come from one person under four name-and-email combinations across roughly 154,000 lines of first-party C, alongside 41,000 stars and 543 open issues. The reviewer's initial commit and contributor figures were wrong because a `git clone --depth 50` shallow clone truncated history, a discrepancy caught only by cross-checking the repository creation date against the API.", "body_md": "I spent an afternoon evaluating a tool I ended up removing. The removal is the part worth writing about.\n\nThe tool was codebase-memory-mcp, an MCP server that parses your codebase into a knowledge graph so an AI agent can ask structural questions. \"What calls this function?\" \"What breaks if I change this signature?\" That's a real gap in my setup. I have semantic search over my notes. I don't have anything that answers *who calls this* with an exact answer instead of a good guess.\n\nSo I looked at it seriously. Here's how that went, and the thing I got wrong.\n\n## Don't evaluate the README\n\nThe README is 864 lines. It leads with thirteen badges, a link to an arXiv preprint, and claims like \"120x fewer tokens\" and \"the Linux kernel in 3 minutes.\"\n\nNone of that is evidence. It's marketing copy, and marketing copy is the same length whether the tool is great or terrible.\n\nSo I cloned it and looked at the actual repository. That takes about five minutes and tells you things no README will:\n\nAbout six months old. 2,610 commits, 166 contributors. And 2,147 of those commits are from one person writing under four different name-and-email combinations. That's 82% of the project from a single pair of hands on roughly 154,000 lines of first-party C, not counting the vendored parsers.\n\nWorth knowing before you adopt it. Also worth knowing: 41,000 stars and 543 open issues. Popular, actively used, and carrying a real backlog.\n\nBut the engineering discipline is real. Sanitizer builds, soak tests, CodeQL, SLSA provenance, a DCO. Someone who ships a Dockerfile for a MemorySanitizer test image is not winging it.\n\nBoth of those facts matter. Neither is in the README.\n\nAnd here's where I have to correct myself, because it's the same mistake in a different coat. My first pass at those numbers was wrong. I'd cloned with `git clone --depth 50` out of habit, because it's faster, and I only wanted a look around. A shallow clone gives you a *truncated* history and `git log` reports on it without complaint. No warning, no asterisk. I read \"first commit: June\" off a repository that actually started in February and got the commit count and contributor spread wrong too.\n\nI only caught it because I went to verify something unrelated, the star count, and the creation date. The API return didn't match the date my clone claimed.\n\nSame lesson as everything else in this piece: the tool answered confidently, and the answer was wrong, and nothing about the output looked wrong. `git rev-parse --is-shallow-repository` would have told me in one second. I didn't ask.\n\n## The actual risk wasn't the code\n\nHere's what I found when I grepped the install path: the installer writes to `settings.json`. It installs PreToolUse, SessionStart, and SubagentStart hooks. It drops skill directories. It splices your `CLAUDE.md` between marker comments. And it does that across 43 coding-agent clients it auto-detects by its own `--help` count.\n\nThat's not a security problem. It's exactly what the tool advertises. But my `~/.claude` has a hand-tuned hook chain I built over months, an allowlist `.gitignore`, and multiple sessions running at once. An auto-detecting installer is the wrong shape for that.\n\nSo I installed with `--skip-config`. Binary only.\n\nThen I did the part most people skip: I checked whether the flag actually worked.\n\nI took SHA-256 hashes of every config file I cared about before installing, and again after. Then I grepped for the tool's marker string across every agent directory.\n\nThe hashes matched. The grep came back empty.\n\nBut an empty grep proves nothing on its own. Maybe I typo'd the path, maybe the file wasn't readable. So I ran the same grep for a string I *knew* was in the file. It returned 66 hits. Now the empty result means something.\n\nThat's the habit worth stealing from this whole story: **a flag's name is not evidence that the flag worked.** Neither is a clean scan, unless you've proven your scanner can find something.\n\nOne thing did change: it appended a PATH line to my `.zshrc`. Harmless, clearly labeled. But `--skip-config` means *agent* config only, and nobody tells you that.\n\n## Grade it against something you already know\n\nFor the pilot I indexed a real codebase: 847 Python files, indexed in 12 seconds into about 33,000 nodes and 155,000 edges. Fast. Genuinely fast.\n\nBut speed isn't the question. Any tool can be fast and useless. The question is whether the answers are *right*, and you can't judge that on a codebase you don't know.\n\nSo I picked a question I'd already answered by hand months ago.\n\nBack then I'd worked out, by reading source files one at a time, that this particular tool quietly installs a second MCP server, but only through two specific commands, and not through the path I was using. It took a while. It mattered, because the answer decided how I installed it.\n\nI asked the graph the same question. One query for the callers of that function. Two non-test results, exactly the two commands I'd found by hand, and nothing else.\n\nThat's a good test, and I'd suggest it for anything in this category. Don't ask a new retrieval tool something you can't verify. You'll grade it on whether the answer *sounds* right, which is how you end up trusting a confident wrong answer. Ask it something you already dug out the hard way. You'll know instantly whether it's real.\n\n## The costs nobody advertises\n\nThe index for that one repository was 229 MB.\n\nThere are lighter modes. The middle setting cut it to 105 MB with no loss to the core package or the tests. It drops docs, JSON fixtures, and example frontends. The lightest went to 84 MB but also dropped git co-change data, which turned out to be the one thing in the graph I couldn't get anywhere else: which files historically change *together*. Touch this auth file, expect to touch both handlers. No AST can tell you that. Only history can.\n\nSo: about 105 MB per repository, and plan for it before you point this at a dozen repos.\n\nThat number appears nowhere in the documentation. I only have it because I ran all three modes and measured.\n\n## Then I tried to uninstall it\n\n`uninstall` failed.\n\nNot because of anything to do with the tool's own files. It failed because `~/.cursor/mcp.json` on my machine is a symlink into my dotfiles repo, a completely ordinary setup, and the uninstaller refuses to write through symlinks.\n\nRefusing to follow symlinks is a *good* instinct. Following them blindly is how you get exploited.\n\nBut it treated that refusal as fatal to the whole operation:\n\n```\nerror: agent_config agent=Cursor op=mcp_uninstall path=~/.cursor/mcp.json (target: symlink, 43 bytes)\nerror: one or more agent cleanup operations failed; executable and index removal were not started\n```\n\nRead that second line again. Removing the *executable* and the *indexes* never started, because of a check on a Cursor config file. A file that, because I'd installed with `--skip-config`, had nothing in it to remove in the first place.\n\n386 MB left on disk after a command whose entire job is removal. A 282 MB binary and 104 MB of indexes, all of which I removed by hand.\n\nAnd it gets better. The same run printed lines like `removed MCP config entry` and `removed PreToolUse + SessionStart + SubagentStart hooks` for client after client.\n\nNone of that happened. I checked the modification times. One of the files it claimed to have cleaned was last touched in May, months before I installed anything. Two of the directories it named don't exist on my machine at all.\n\nThe output describes work it didn't do. If you read the summary instead of the filesystem, you'd believe a dozen configs had just been rewritten.\n\n## The part where I embarrassed myself\n\nI wrote it up as a bug report. Good repro steps, minimal setup, the works. Step one created a symlinked Cursor config so the reader could trigger it:\n\n```\nln -sf ~/.dotfiles/cursor/mcp.json ~/.cursor/mcp.json\n```\n\nRead that closely. `ln -sf`. Force.\n\nIf you already had a real `~/.cursor/mcp.json`, that command destroys it. Silently. No prompt, no backup.\n\nI was about to publish a bug report to a repo with 41,000 stars, whose reproduction steps would wreck the config of anyone who pasted them.\n\nI caught it, but not by rereading the draft. I caught it because I ran a separate check specifically looking for writes to real paths before posting. The re-read wouldn't have found it; I'd read that command four times already and it looked completely normal.\n\nThe fix was to run the whole thing in a throwaway `HOME`.\n\nHere's why I think this one is worth sitting with: **reproduction steps are the only code you write that's designed to be pasted into a stranger's shell.** The blast radius is inverted from normal code. When you write a script, you're the one who runs it on your machine, where you know what exists. A repro runs on machines configured differently from yours, by people who are trusting you.\n\nA repro that's destructive on someone else's setup is a bug you're shipping to your readers.\n\n## What I'd actually keep\n\nFour things, none of which are about this particular tool:\n\n**Clone it before you read about it.** Five minutes in the repository beats an hour with the README. Commit history, contributor spread, and CI config will tell you what you're really adopting.\n\n**Verify the flag, don't trust it.** Hash before, hash after. And if a scan comes back clean, prove the scan can find something first.\n\n**Grade new tools on questions you've already answered.** It's the only way to catch a confident wrong answer.\n\n**Uninstall it before you commit to it.** This is the one I'd been skipping for years. Installing is the vendor's happy path: tested, demoed, and polished. Uninstalling is where the bodies are. If a tool can't cleanly remove itself, that tells you how carefully it handles *your* machine as opposed to its own features.\n\nI filed the issue. I'd look at the tool again in a few versions, because the graph queries genuinely delivered and that co-change data is something I don't have anywhere else.\n\nBut I'd pin the version. And I'd expect to tear it down by hand.\n\n*Building an AI-assisted development workflow and writing down what actually works. Subscribe for the parts the documentation leaves out.*\n\n*Found this useful? I share practical lessons from my systems engineering journey at* [As The Geek Learns](https://astgl.substack.com).", "url": "https://wpnews.pro/news/the-uninstall-is-part-of-the-evaluation", "canonical_source": "https://astgl.com/p/uninstall-is-part-of-the-evaluation", "published_at": "2026-10-26 12:03:56+00:00", "updated_at": "2026-10-26 12:13:02.910929+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools"], "entities": ["codebase-memory-mcp", "Claude Code", "Model Context Protocol", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-uninstall-is-part-of-the-evaluation", "markdown": "https://wpnews.pro/news/the-uninstall-is-part-of-the-evaluation.md", "text": "https://wpnews.pro/news/the-uninstall-is-part-of-the-evaluation.txt", "jsonld": "https://wpnews.pro/news/the-uninstall-is-part-of-the-evaluation.jsonld"}}