{"slug": "my-context-selector-beat-grep-an-agent-with-grep-beat-it", "title": "My context selector beat grep. An agent with grep beat it.", "summary": "A developer's context-selection tool, cognitive-cache, initially appeared no better than grep in a small benchmark, but a larger SWE-bench evaluation showed it significantly outperformed grep and TF-IDF. The tool uses six weighted signals and a submodular selector to pick files for an LLM context window, and the developer emphasizes that the initial negative result was a small-sample artifact.", "body_md": "[cognitive-cache](https://github.com/Anyesh/cognitive-cache) started from an observation I still think is correct: every coding tool decides what goes into the context window, and almost none of them treat that decision as an explicit selection problem. Cursor greps, RAG systems embed and cosine-search, and plenty of harnesses just stuff files until the budget runs out. The OS analogy is hard to unsee once you have it. The context window is RAM, eviction and retrieval are virtual memory, and token budget allocation is an allocator that nobody has written.\n\nSo I wrote one. Six weighted signals score every file in a repo against a task description, and a two-phase greedy submodular selector picks the highest-value set that fits a token budget. Nine languages, no LLM calls, no API keys. It shipped to PyPI and I used it through MCP for a while.\n\nThen I built a benchmark harness, and the harness spent the next two months dismantling most of what I believed about the tool. This is that sequence, in order, because the order is the interesting part.\n\nThe original benchmark was bad in the way most self-benchmarks are bad. It coupled retrieval to LLM patch generation, reported rank-blind full-set recall, had no significance test, and I almost never ran it.\n\nThe replacement is LLM-free and deterministic. For each issue it clones the repo at the pre-fix commit, produces a full file ranking per strategy, and scores that ranking against the files the gold patch actually modified, using recall@k and MRR. Every head-to-head comparison carries a paired bootstrap 95% confidence interval, so a difference that excludes zero is real and a difference that straddles zero is not a result.\n\nOn the first dataset, 23 hand-curated issues across 8 repos:\n\n| strategy | R@1 | R@5 | R@10 | MRR |\n|---|---|---|---|---|\n| cognitive-cache | 0.170 | 0.428 | 0.511 | 0.540 |\n| grep (keyword count) | 0.170 | 0.409 | 0.559 | 0.533 |\n| lexical (TF-IDF) | 0.159 | 0.362 | 0.496 | 0.487 |\n\nThe recall@5 difference over grep was +0.020 with a 95% CI of [-0.067, +0.109]. Grep won at [recall@10](mailto:recall@10). Whatever the tool was doing, at that sample size it was indistinguishable from counting keywords.\n\nTwo other things fell out of the same run. The per-signal ablation suggested `graph_distance`\n\n, carrying 0.20 of the weight budget, was net negative, and `file_role_prior`\n\nwas doing nothing at 0.07. And the budget selector was leaking badly: ranking recall@5 was 0.428 while end-to-end recall under a 12k budget was 0.282, so the selector was discarding relevant files its own ranking had already surfaced.\n\nI nearly acted on the ablation immediately. A weight sweep stopped me, because every candidate reweighting had a bootstrap lower bound of +0.000. The \"graph_distance is net negative\" reading was itself noise at n=23. The real blocker was not the weights, it was that 23 narrow issues cannot detect or drive an improvement in either direction.\n\nI moved to [SWE-bench](https://www.swebench.com/), which solves the curation problem by construction: real GitHub issues paired with the gold patch that fixed them, pulled over the HuggingFace datasets-server with no auth. Ground truth is the set of source files the patch modified, and because patches only touch files present at the base commit, every ground-truth file is retrievable.\n\nOn a balanced 78-instance subset of SWE-bench Lite across 12 repos:\n\n| strategy | R@1 | R@5 | R@10 | MRR |\n|---|---|---|---|---|\n| cognitive-cache | 0.231 | 0.538 | 0.654 | 0.375 |\n| grep | 0.103 | 0.359 | 0.487 | 0.236 |\n| lexical (TF-IDF) | 0.141 | 0.410 | 0.462 | 0.253 |\n\nRecall@5 over grep: +0.179, CI [+0.064, +0.295]. Over TF-IDF: +0.128, CI [+0.013, +0.244]. Both exclude zero. On a harder 91-issue subset restricted to fixes touching two or more files, the edge held at 0.365 vs 0.252, CI [+0.056, +0.174].\n\nThe earlier negative was a small-sample artifact. That cuts both ways, and it is worth sitting with: if I had published the n=23 result I would have been wrong, and if I had published the n=78 result as the final word I would also have been wrong, for the reason in the next section.\n\n\"Beats grep\" means beats a naive keyword count. It says nothing about the thing my tool would actually be competing with, which is a coding agent running its own search loop.\n\nSo I ran that. Five single-file SWE-bench issues, one each from flask, requests, xarray, pylint, and pytest. Fresh Claude subagents, cold in the repo at the base commit, `grep`\n\n, `glob`\n\n, and `read`\n\nonly, cognitive-cache explicitly forbidden.\n\nThe agent got 5 of 5, averaging about 3.4 tool calls and 19k tokens, and in each case it pinpointed the mechanism rather than just the file. cognitive-cache got 4 of 5, missing pylint #5859, which the agent found in two calls. Naive keyword grep got 3 of 5.\n\nThe token argument inverted at the same time. The agent read a handful of targeted files. A cognitive-cache call returns file contents up to the full token budget, so on these tasks it front-loads more context than a surgical agent consumes. It was costing context, not saving it.\n\nThat is n=5 on hand-picked single-file issues, so it is directional rather than definitive. But it moved the honest positioning from \"more accurate than the agent\" to \"fewer roundtrips on a cold start in unfamiliar code\", which is a much smaller claim.\n\n`graph_distance`\n\nmeasures import-graph distance from the files the task names. It carried 0.20 weight from the day I wrote it, on the reasoning that a fix often lives one hop away from the file the issue mentions.\n\nAcross both datasets it earned nothing. On the 91-issue multi-file set, zeroing it changed recall@5 by +0.000 and MRR by -0.006. That was the pre-registered kill criterion, so I was ready to retire it.\n\nThen I pushed back on my own experiment, which turned out to be the useful move. Multi-file is not the same as multi-hop, a regex-based import graph is not a reference graph, and testing a signal as one term in a drowned weighted sum is not a fair test of the underlying idea. All three critiques were fair, so instead of assuming, I wrote a diagnostic over the 91-issue set to check whether the ground truth was even reachable.\n\nThe answer killed the revival rather than supporting it. 93% of issues have all their ground-truth files reachable from the entry points. But **87% of fix files are themselves entry points, at graph distance 0**, with only about 10% sitting at one to three hops and none further out. The mechanism was never starvation, it was redundancy: real issue text names the files it touches, so `symbol_overlap`\n\nhas already ranked those files highly and import distance adds nothing on top.\n\nThat is not a bug in the experiment. It is a fact about the task distribution, and it explains the agent result too. On issue-shaped work the target is usually named in the prompt, which is exactly the regime where an agentic grep loop is excellent and a static ranker has little room to add value. Claude Code shipping no code index looks less like an omission and more like the same finding, reached earlier.\n\nOne thread survived long enough to be worth reporting separately. I built a jedi-backed neighborhood tool (definition, callers, in-repo callees, with real scope-aware resolution rather than name matching) and tested it two ways.\n\nAgainst agents, on two hard cross-file issues where naive grep recall was 0, I ran a 2x2: agents with grep and read, versus agents offered the neighborhood tool as well. Every arm scored 0.5, finding the file tied to the named concept and missing the cross-file one. **The agents that had the tool invoked it zero times out of two**, despite being told to. Checking the tool's output directly against the symbols those same agents had reasoned about, it surfaced both missed files. The data would have lifted recall from 0.5 to 1.0 if it had been used.\n\nAt scale the value is real but modest: across all 91 multi-file issues, neighborhood-of-seeds alone scores 0.164 (worse than grep's 0.252 standalone), while grep-or-neighborhood combined reaches 0.330, recovering 20 of grep's 257 missed fix files. That is +7.8 absolute points, in line with RepoGraph's own reported 9-12%.\n\nPut the two halves together and the conclusion is about delivery, not capability. A passive tool that competes with the grep reflex for invocation loses the invocation. Value like that only lands if it is injected into results the agent is already reading, not offered as one more thing it could call.\n\nI set `graph_distance`\n\nto weight 0, rewrote the README around the scoreboard instead of the pitch, and stopped building. The remaining pitch would have required a forced-injection layer to capture a modest gain, in a niche that already has direct clones, running against the current of agents that search well on their own.\n\nWhat survives is the measurement rig: the SWE-bench importer that needs no token, the retrieval evaluator, the weight tuner, the multi-hop diagnostic, the neighbor-recovery harness, and the bootstrap CIs that make each of those trustworthy. In a category that ships on anecdotes, that is the part I would build again first.\n\nThe generalizable lesson is not \"context selection is a dead end\". It is that the baseline you skip is the one that decides whether your work matters. I measured against grep for two months because grep was the baseline that let the tool win. The agent was two hours of work and it reframed the entire project.", "url": "https://wpnews.pro/news/my-context-selector-beat-grep-an-agent-with-grep-beat-it", "canonical_source": "https://dev.to/anyesh/my-context-selector-beat-grep-an-agent-with-grep-beat-it-5", "published_at": "2026-07-31 14:00:23+00:00", "updated_at": "2026-07-31 14:35:59.183762+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["cognitive-cache", "SWE-bench", "PyPI", "MCP", "HuggingFace"], "alternates": {"html": "https://wpnews.pro/news/my-context-selector-beat-grep-an-agent-with-grep-beat-it", "markdown": "https://wpnews.pro/news/my-context-selector-beat-grep-an-agent-with-grep-beat-it.md", "text": "https://wpnews.pro/news/my-context-selector-beat-grep-an-agent-with-grep-beat-it.txt", "jsonld": "https://wpnews.pro/news/my-context-selector-beat-grep-an-agent-with-grep-beat-it.jsonld"}}