{"slug": "evolving-self-tuning-benchmarks-for-long-horizon-tool-use", "title": "Evolving Self-tuning Benchmarks for Long-Horizon Tool Use", "summary": "A new arXiv paper (2610.27307v1) proposes TURTLE, an Unsupervised Environment Design approach that evolves tool-use benchmarks using proposer, solver and feedback-generator LLM agents, rewarding the proposer for tasks that are difficult yet solvable based on the solver's empirical performance. On AppWorld, τ-Bench and AutomationBench with Gemma 4 31B-it, the evolved tasks produced a 165% increase in the percentage of hard tasks on AppWorld, increased task diversity in reasoning and simultaneous tool interactions, and elicited a broader range of failure modes than the original tasks. The authors position TURTLE as a scalable alternative to tool-use benchmarks that rely on human supervision or are too domain-specific.", "body_md": "arXiv:2610.27307v1 Announce Type: new \nAbstract: For AI assistants to move beyond simple chat interfaces and reliably assist users with applications like Gmail, Venmo, and Slack, we need to improve their tool-use capabilities. However, existing tool use benchmarks are still limited and difficult to scale because they rely on human supervision or are too domain-specific. To make data generation scalable, we propose, TURTLE (**T** ool **U** se **R** elated **T** asks with **L** ong-Horizon **E** volution), an Unsupervised Environment Design (UED) approach to evolve existing tasks using a proposer, a solver, and a feedback generator large language model agents. The proposer is rewarded for difficult yet solvable tasks based on the solver's empirical performance, and the feedback generator summarizes the solver's execution traces for the proposer to exploit. TURTLE produces evolving, self-tuning test suites adaptable to different environment requirements, from supporting user simulators and dense document retrieval to interacting across multiple apps. Empirical results on AppWorld, $\\tau$-Bench, and AutomationBench using Gemma 4 31B-it show that the evolved tasks target the desired difficulty range (e.g., a 165\\% increase in the percentage of hard tasks on AppWorld), increase task diversity (in terms of reasoning and simultaneous tool interactions), and elicit a broader range of failure modes than the original tasks.", "url": "https://wpnews.pro/news/evolving-self-tuning-benchmarks-for-long-horizon-tool-use", "canonical_source": "https://www.machinebrief.com/news/evolving-self-tuning-benchmarks-for-long-horizon-tool-use-4y80", "published_at": "2026-10-21 04:00:00+00:00", "updated_at": "2026-10-21 04:50:26.846497+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "ai-tools", "machine-learning"], "entities": ["TURTLE", "AppWorld", "τ-Bench", "AutomationBench", "Gemma 4 31B-it", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/evolving-self-tuning-benchmarks-for-long-horizon-tool-use", "markdown": "https://wpnews.pro/news/evolving-self-tuning-benchmarks-for-long-horizon-tool-use.md", "text": "https://wpnews.pro/news/evolving-self-tuning-benchmarks-for-long-horizon-tool-use.txt", "jsonld": "https://wpnews.pro/news/evolving-self-tuning-benchmarks-for-long-horizon-tool-use.jsonld"}}