# AgentConnect study finds coding agents often choose grep over LSP > Source: > Published: 2026-09-04 05:23:15+00:00 # AgentConnect study finds coding agents often choose grep over LSP **Pengcheng Xu's preliminary study found that tool choice followed the task, while location-only LSP results increased follow-up reads and token use.** By [RuntimeWire Staff](/author/runtimewire-staff) ยท Published Primary source: [AgentConnect](https://www.agentconnect.md/blog/grep-beat-lsp-harness/) ## Why it matters Coding-agent performance depends on the complete tool loop. Replacing a native search path with a cleaner abstraction can increase token use and lower success if the output creates extra work. [Pengcheng Xu](https://poytr1.github.io/?ref=runtimewire), author of [AgentConnect's coding-tool study](https://www.agentconnect.md/blog/grep-beat-lsp-harness/?ref=runtimewire), found that coding agents defaulted to grep on symbol-named localization tasks even when semantic navigation was available. Their choices changed with the task, and location-only results from the Language Server Protocol, or LSP, often required extra file reads. AgentConnect published a post about the [preliminary study](https://arxiv.org/abs/2608.13568?ref=runtimewire), an engineering comparison of ordinary text search with LSP-backed semantic navigation. The AgentConnect post is dated August 12, 2026, so this is a recent study rather than a same-day release. Xu tested three Claude models across Python and TypeScript repositories, comparing ordinary text search with LSP-backed semantic navigation. The results were conditional. LSP improved precision where a codebase contained many misleading text matches. It frequently added cost where grep already returned clean results. The largest improvements came from changing the information returned to the model, rather than changing the retrieval engine underneath it. That distinction supports the product architecture Xu and his team chose for AgentConnect. The open-source platform coordinates coding agents across work channels while allowing runtimes such as Claude Code and Codex to retain their native tools and prompt-to-tool loops. Xu's experiment supplies a technical argument for preserving those loops: replacing a familiar interface with a nominally better abstraction can change how the model works, even when the underlying model stays the same. ### The agent usually reached for grep first For simple localization tasks, where the agent had to identify files associated with a named symbol, [Claude Opus 4.8](/models/anthropic/claude-opus-4.8), Sonnet 4.6 and Haiku 4.5 used semantic navigation in only 0% to 6% of free-choice runs. Forcing the agent to start with semantic navigation lowered success from 100% to 89%. The experiment did not show a fixed aversion to LSP. When the task required finding every caller of a function, the models chose semantic navigation 45% to 57% of the time. LSP reached 1.00 precision, compared with 0.76 for grep, because it removed comments and other false matches. Recall remained around 0.66 under both approaches. The language server gave the agent a cleaner set of results, but the agent still stopped before finding every relevant call. In those runs, retrieval precision was no longer the main limit. The model's own thoroughness was. *AgentConnect found that semantic-tool use rose sharply when agents were explicitly asked to find every reference.* Xu also found that language alone was a poor guide for choosing a retrieval strategy. In the clean TypeScript repository remeda, LSP produced no F1 improvement and used 16% more tokens. In the noisier TypeScript repository hono, it improved F1 by about 0.246 while using 12% fewer tokens. On the Python requests repository, it improved F1 by 0.072 but carried a 19% token premium. The important variable was lexical noise. If a grep query produced many unrelated matches, semantic navigation earned its setup and interaction costs. If text search already isolated the right code, LSP became another stop in the agent's route. ### Better output beat a better backend The study's most useful result came from an experiment that left the language server and its reference set unchanged. Xu altered only the response format. A location-only LSP tool returned file names and line numbers, forcing the agent to open each file before deciding what to edit. On a multi-file rename task, that configuration reached 0.67 pass@1 and triggered an average of 15.2 follow-up file reads per episode. When the same tool included the matching source line and two nearby lines in its response, pass@1 rose to 0.83. Follow-up reads fell to 3.2. Grep still led the task with 1.00 pass@1, partly because a complete rename may need to change comments, documentation and strings that semantic reference tools deliberately exclude. The response-format improvement exposes a common mistake in agent tooling. Developers tend to evaluate tools by the correctness of the backend operation. An agent must also decide what to do after receiving the result. A list of technically accurate locations can be less useful than a noisier answer containing enough code to support the next action. Anthropic makes a similar point in its [guidance for building agent tools](https://www.anthropic.com/engineering/writing-tools-for-agents?ref=runtimewire), recommending that developers prototype tools, evaluate agent behavior and revise the interface through repeated testing. Xu's experiment puts numbers behind that advice: cutting a read-modify loop can matter as much as improving retrieval precision. ### AgentConnect is selling the harness argument Xu summarizes the thesis as ["agent capability = model x harness"](https://www.agentconnect.md/blog/grep-beat-lsp-harness/?ref=runtimewire). The harness determines the tools a model sees, their names and arguments, what each call returns, and how failures are presented. Two products can expose the same model and still produce different results because they wrap it in different action loops. AgentConnect has built its platform around that premise. According to its [architecture documentation](https://docs.agentconnect.md/docs/how-it-works?utm_source=openai&ref=runtimewire), agent execution remains on daemons operated by the customer, while a control plane manages permissions, schedules, placement and integrations. AgentConnect communicates with coding runtimes through the open [Agent Client Protocol](https://agentclientprotocol.com/get-started/architecture?ref=runtimewire), preserving each runtime's internal tool surface rather than forcing every model into one generic loop. The study therefore serves two purposes. It is an empirical test with [published code and run data](https://github.com/agentconnect-md/lsp-vs-grep-token-study?ref=runtimewire), and it supports AgentConnect's decision to coordinate agents around their native runtimes. That alignment calls for a narrow reading of the results. The pilot covered three Claude models, a few repositories, one harness and a limited subset of LSP capabilities. Given that scope, the results support a practical rule for builders: semantic retrieval should be routed according to the task and the codebase, and its output must be designed for the model consuming it. Adding a sophisticated tool without measuring the full agent loop can raise token use, add tool calls and reduce success. Xu's work suggests that the next gains in coding agents may come from better plumbing around the model, including the supposedly boring details of what one tool call returns.