I compared grep with LSP-backed semantic navigation across code-finding and editing tasks. The results show why a tool's LLM-friendliness may matter as much as the capability behind it.
Why would a coding agent ignore a retrieval interface that returns more precise results?
I explored this question in a small study comparing lexical search with grep
against LSP-backed semantic navigation. I expected semantic navigation to reduce noise and save tokens. Instead, agents often stayed with grep
. When I forced them to use the semantic path first, task success sometimes fell.
This is a question of LLM-friendliness. A tool is not friendly to a model merely because its results are precise. It must return enough context for the next step and present that context in an interface and output shape the model can use directly. Familiarity may also matter: the model may have learned similar action paths during training. The interface properties can be evaluated directly. Training support is a hypothesis consistent with these results, not something this study proves.
The result is not a general argument against LSP. The protocol includes capabilities far beyond code navigation, and this study tested only a small subset. Instead, the results point to a broader engineering problem: a model does not use tools in isolation. It uses them through a harness that defines the available actions, their names, their inputs, and the context returned to the model.
In this post, I describe how code retrieval affected both code-finding and editing tasks, why grep
had an advantage in some conditions, and what this means for agent platforms.
I compared two ways for an agent to retrieve code context. grep
performs lexical search: it finds matching text. The tested LSP-backed tools perform semantic navigation through references, definitions, and document symbols, allowing them to distinguish a real function call from the same word in a comment.
The pilot covered three Claude models, several Python and TypeScript repositories, and multiple task types. I measured token use only when both approaches completed the task successfully. This controls for a common evaluation error: a failed run can appear efficient simply because it stopped early.
On simple code-location tasks, all three models chose the semantic tool only 0% to 6% of the time when both tools were available. Forcing a semantic-first path reduced success from 100% to 89% in that arm.
Reference-completeness tasks produced a different result. When asked to find every caller, the models chose semantic navigation 45% to 57% of the time. The LSP-backed path reached 1.00 precision, compared with 0.76 for grep
, by removing false matches. However, recall stayed near 0.66 in both arms. Semantic navigation did not find more true calls. The remaining limit came from how thoroughly the agent worked, not from retrieval precision. For the stronger models, the precision gain also came with higher token use rather than a saving.
The model doesn't blindly prefer grep — it routes by task
Share of semantic (LSP) tool calls when both grep and LSP are available and the agent chooses freely.
Legend: Opus 4.8 (blue), Sonnet 4.6 (magenta), Haiku 4.5 (green).
| Task | Opus 4.8 | Sonnet 4.6 | Haiku 4.5 |
|---|---|---|---|
| Localization | 0% | 4% | 6% |
| Reference-completeness | 45% | 50% | 57% |
| Multi-file rename | 3% | — | — |
The codebase was also important. On a clean TypeScript repository, LSP-backed navigation produced no F1 gain and used 16% more tokens. On a noisy TypeScript repository, it improved F1 by 0.246 and used 12% fewer tokens. The useful predictor was lexical noise, not whether the language had strong static types.
Codebase noise determines the value of semantic navigation
Accuracy gain from semantic retrieval on reference-completeness (ΔF1 = LSP − grep). Bar colour encodes how noisy grep is on that repo; prec = grep’s precision there.
Legend: blue means grep is clean here; magenta means grep is noisy here.
| Repo | Language | grep precision | ΔF1 (LSP − grep) | Token cost |
|---|---|---|---|---|
| remeda | TypeScript | 1.00 | +0.000 | +16% |
| hono | TypeScript | 0.51 | +0.246 | −12% |
| requests | Python | 0.76 | +0.072 | +19% |
These results are conditional rather than categorical. The agents did not simply “always use grep.” Their routing changed with the task, and the value of LSP-backed navigation changed with the repository.
The tested LSP-backed tools initially returned only a location: a file path, line, and column. The agent then had to open the file to inspect the code. grep
, by contrast, usually returned the matching line immediately: src/auth.ts:42: return validateToken(token)
.
I changed the semantic-navigation response to include source text in a similar shape. The semantic backend and the set of references stayed the same; only the information returned to the model changed. Pass@1 on the rename tasks rose from 0.67 to 0.83, while follow-up file reads fell from 15.2 to 3.2 per episode.
Returning source context improves semantic navigation
Multi-file rename, Opus 4.8, pyright with a pre-warmed index. Same semantic backend in both LSP arms — only the output shape differs.
Legend: grep (blue), LSP — locations only (magenta), LSP + inline context (green).
| Arm | pass@1 | Site recall | Tokens | Follow-up reads |
|---|---|---|---|---|
| grep | 1.00 | 1.000 | 2,451 | 4.3 |
| LSP — locations only | 0.67 | 0.930 | 4,131 | 15.2 |
| LSP + inline context | 0.83 | 0.958 | 3,336 | 3.2 |
This result illustrates a principle that Anthropic also emphasizes in Writing effective tools for agents: tools are interfaces for non-deterministic agents, so the context they return is part of the design. A semantically correct tool can still create a poor agent workflow if each result requires several extra actions to interpret.
The output change does not prove that post-training data caused the improvement. It may also have helped simply because each response contained more useful information. However, the result is consistent with a broader hypothesis: models learn concrete action patterns, not “tool use” in the abstract. A familiar loop—prompt, tool call, readable result, next action—can be part of the capability observed in practice.
Interface familiarity is only part of the explanation. Lexical search also had a real structural advantage for some tasks.
A semantic reference is only one kind of text match. A rename may also need to update comments, docstrings, configuration, or strings. find_references
will not return those by design, while grep
can.
semantic references ⊂ textual occurrences
For text-wide edits, grep
can be the better retrieval tool even for a model with perfect training on LSP-backed navigation.
This gives us two explanations for the observed behavior:
The first explanation follows directly from what the tools retrieve. The second is a hypothesis consistent with the routing and output-format results, but this study did not manipulate training data and therefore cannot prove it.
Here, I use harness to mean the runtime around a model: the instructions placed in context, the tools made available, their input schemas, the shape of their results and errors, and the loop that decides what the model sees next.
This surrounding system can materially change behavior. Anthropic’s work on effective harnesses for long-running agents shows the same idea at a longer time scale: the model alone is not enough to make reliable progress across sessions. Environment setup, progress artifacts, and verification routines affect what the agent can accomplish.
The same principle applies within a single tool loop. When post-training includes agent trajectories, the harness defines the prompts, tool calls, results, and recovery paths in those examples. A model trained through repeated use of read
, grep
, edit
, and bash
may learn policies that depend on those interfaces. Moving the same model into a different tool layer can therefore change its effective capability.
agent capability = model × harness
This is why benchmark results for a model do not always transfer unchanged to a different runtime. Supporting the same model is not necessarily the same as reproducing the same agent. Tool selection, signatures, output formats, and error behavior can all affect the policy the model follows.
This is one reason my team and I built AgentConnect around native coding-agent runtimes. AgentConnect does not place Claude or Codex models inside a shared, generic tool loop. It runs runtimes such as Claude Code and Codex on the user’s own machine, where each runtime keeps its native tools and normal prompt-to-tool workflow.
AgentConnect communicates with these runtimes through the open Agent Client Protocol (ACP). ACP standardizes the boundary between a client and a coding agent, including session setup, prompts, streaming updates, tool-call updates, cancellation, and permission requests. It does not require every runtime to expose the same internal tools.
This separates two concerns. At the outside boundary, a common protocol lets AgentConnect provide team channels, triggers, schedules, session history, collaboration, and control across multiple agents. Inside that boundary, Claude Code can continue to work like Claude Code, and Codex can continue to work like Codex.
The goal is not vendor lock-in. ACP provides an open boundary across runtimes. Our design principle is to preserve the tool surface each model already uses well, then add coordination around it.
These findings do not mean that teams should avoid LSP, MCP, or new agent skills. The study found a clear precision gain from LSP-backed navigation in noisy code, and a small response-format change removed most follow-up reads. The practical lesson is to evaluate a new retrieval interface as part of the full agent loop.
My recommendation is to start with the native tool surface, then apply the following checks when adding a new capability:
path:line:content
may work better than a bare location object.grep
.As Anthropic notes in Building effective agents, successful agent systems often rely on simple, composable patterns. More tools do not automatically produce a more capable agent; tools must be distinct, understandable, and useful within the model’s workflow.
The study shows why “better retrieval” cannot be evaluated outside the full agent system. An interface can be more precise and still use more tokens. It can return correct locations and still create unnecessary reads. A small change in output shape can make the same semantic result much easier for the model to use.
For teams building agent platforms, the implication is straightforward: evaluate the model and harness together. Preserve the interfaces that already support reliable behavior, and test changes against real tasks before assuming that a more sophisticated abstraction will help.
For the full experimental setup, task definitions, and results, see [Does a Language Server Save Tokens for Coding Agents?](https://github.com/agentconnect-md/lsp-vs-grep-token-study).
This is the product principle behind AgentConnect: use an open protocol to connect agents, while keeping each model together with its native runtime and tool loop.
textDocument/rename
, diagnostics, or code actions. A rename-capable LSP might perform differently on the refactoring tasks where grep
did best. The edit tasks were local and are not standard SWE-bench scores. These findings are useful signals, not a final verdict across all models, tools, and codebases.