Improving Agents by Bootstrapping Experientially-Learned Environmental Knowledge

Researchers Shashank Kirtania and co-authors introduced BREW (Bootstrapping expeRientially-learned Environmental knoWledge), a framework that distills an LLM agent's past interaction trajectories into a structured, retrievable knowledge base of natural-language recipes, according to an arXiv paper submitted 25 Nov 2025 and revised 10 Jul 2026. BREW uses Expand-and-Gather Monte Carlo Tree Search (EG-MCTS) to jointly optimize recipe accuracy and retrievability, and adapts hindsight relabeling to convert near-miss trajectories into positive demonstrations. On OSWorld, tau^2-Bench, and SpreadSheetBench, BREW achieved 10-20% gains in task success and 10-15% fewer execution steps over base agents, outperforming existing memory-augmented baselines that can degrade below memoryless performance.

Computer Science Artificial Intelligence Submitted on 25 Nov 2025 v1 https://arxiv.org/abs/2511.20297v1 , last revised 10 Jul 2026 this version, v2 Title:Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledge View PDF https://arxiv.org/pdf/2511.20297 HTML experimental https://arxiv.org/html/2511.20297v2 Abstract:Large Language Model LLM -based agents are increasingly capable of complex, multi-step tasks such as GUI automation, tool use, and data manipulation, yet they cannot learn from experience: each new session rediscovers solutions from scratch. We introduce BREW Bootstrapping expeRientially-learned Environmental knoWledge , a framework that distills an agent's past interaction trajectories into a structured, retrievable knowledge base KB of natural-language recipes, concept-level procedural documents that capture what to do, when it applies, and what to watch out for. Drawing on the principle of library learning from program synthesis, BREW decomposes agent memory into modular, concept-localized documents and formalizes KB construction as a state-space search problem. To navigate this space, we introduce Expand-and-Gather Monte Carlo Tree Search EG-MCTS , a reward-guided algorithm that jointly optimizes recipe accuracy and retrievability across parallel, per-concept search trees. We further adapt hindsight relabeling to convert near-miss trajectories into positive demonstrations, surfacing latent agent competencies as reusable knowledge. On three domain-grounded benchmarks, OSWorld, tau^2-Bench, and SpreadSheetBench, BREW achieves 10-20% gains in task success and 10-15% fewer execution steps over base agents, while consistently outperforming existing memory-augmented baselines that can degrade below memoryless performance. The resulting KB is inspectable, modular, and extensible, providing a transparent and controllable substrate for agent optimization. Submission history From: Shashank Kirtania view email https://arxiv.org/show-email/31753917/2511.20297 Tue, 25 Nov 2025 13:34:54 UTC 437 KB \ v1\ https://arxiv.org/abs/2511.20297v1 v2 Fri, 10 Jul 2026 08:15:53 UTC 552 KB References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .