Agents Don't Paginate: First-Chunk Selection for LLM Tool Responses

A new arXiv study (2608.26130v1) finds that LLM coding agents never request a second page of tool responses, and that improving the precision-at-1 rate (p1) of first-chunk selection does not systematically improve downstream accuracy. Across 500 SWE-bench Verified tasks and 4,800 LLM calls, a parameter-free keyword scorer raised p1 from 24.2% to 35.0% (+10.8 p.p., p=3.9×10^-8), but per-model accuracy deltas stayed under 3 percentage points and were not consistently signed. Adding four file-metadata signals to the keyword scorer hurt p1 by 4.8 p.p. (p=0.001).

arXiv:2608.26130v1 Announce Type: new Abstract: Coding agents built on large language models LLMs , such as Claude Code, Cursor, OpenAI Codex, GitHub Copilot, and Aider, receive tool responses that routinely exceed the agent's per-turn token budget. The standard remedy, pagination, is available in every protocol that produced these responses; yet across the corpus of session logs from a public Model Context Protocol middleware we observed no agent-initiated requests for a second chunk. The first chunk is what the agent reads, so we ask how often the gold item the one the agent needs is placed first in it: the precision-at-1 rate $p 1$. In a controlled offline benchmark we treat first-chunk selection as a 0/1 knapsack and compare six value functions on 500 SWE-bench Verified tasks, then test whether $p 1$ matters with a single-turn file-localisation probe on five language models 4,800 LLM calls; not an end-to-end resolve-rate test . Two pre-registered hypotheses did not hold and are our main findings. The central one is negative: raising $p 1$ does not systematically raise downstream accuracy. Per-model deltas stay under three percentage points p.p. , are not consistently signed, and no model is significant; the agent recovers the gold from anywhere in the chunk, so what reaches its answer is first-chunk inclusion, not the gold's rank within it. The second: adding four file-metadata signals to a keyword scorer hurts $p 1$ by 4.8 p.p. paired significance test, $p = 0.001$ . A parameter-free keyword scorer does raise $p 1$, from a 24.2% baseline to 35.0% +10.8 p.p., far beyond chance; $p = 3.9 \times 10^{-8}$ , and to 35.8% with a fallback to the tool's native ordering when no keyword matches. But by our central finding this is a rank-1 gain, and rank-1 is the part that does not reach the agent's answer: downstream accuracy does not move.