Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

A study using the geochemical modeling tool PHREEQC finds that LLM agents which compute the correct answer can still answer wrongly when long tool outputs are kept out of the context window and retrieved on demand from saved files, a pattern the authors frame as "Same Simulation, Different Evidence." The research examines where this retrieval-based harness design costs accuracy for agents running a real scientific tool.

LLM agents that compute the right answer can still answer wrongly. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geoc