# Stanford and Google DeepMind build TURTLE to make AI benchmarks evolve > Source: > Published: 2026-11-08 23:42:42+00:00 # Stanford and Google DeepMind build TURTLE to make AI benchmarks evolve Photo: Steve A Johnson / Pexels The system uses AlphaEvolve to make tool-use tests harder while keeping them solvable, so benchmarks can keep pace with the models they measure AI models keep acing the tests built to measure them. Researchers at Stanford University and [Google](https://cryptobriefing.com/markets/alphabet/) DeepMind think the fix is to let the tests fight back. Their new system, TURTLE, takes existing tool-use benchmarks and evolves them into harder versions. The key constraint is that every task must stay solvable, so the result is a tougher exam rather than a broken one. ## How TURTLE turns up the difficulty TURTLE stands for Tool Use Related Tasks with Long-Horizon Evolution. The work is described in an arXiv paper dated October 19, 2026, listed as arXiv:2610.27307. It focuses on tool use, meaning how well an AI agent can operate software to finish real tasks across many steps. Under the hood, TURTLE relies on three cooperating agents. A proposer, built on DeepMind’s AlphaEvolve evolutionary coding agent, suggests changes to a task. A solver model then attempts the modified version. A feedback generator reviews what happened and steers the next round of edits. The mutations take several forms. TURTLE can add new filters, insert extra steps, or bury hidden components inside a task, all while checking that a solution still exists. The solvability check has a concrete bar. A task counted as a success when the solver cleared at least a 50% success rate across a minimum of 10 attempts. ## The numbers from AppWorld and beyond The team tested TURTLE on several established benchmarks, including AppWorld, τ-Bench, and AutomationBench. The solver model throughout was Gemma 4 31B-it. ### AI, tech, and the markets they move—in one daily briefing. Daily. Free. Join 34,000+ readers across crypto, finance, and policy. The supported environments span AppWorld, τ-telecom, τ-banking, and AutomationBench. Collectively they cover everyday digital chores like email, payments, and collaboration. AppWorld produced the headline result. The share of hard tasks jumped 165% in the evolved set, up from 19.8% among the original tasks. Nearly all of the 735 evolved AppWorld tasks landed with solver success rates between 40% and 60%. Compare that with the original benchmarks, where only 69% of tasks were solved consistently. ## Why static benchmarks keep breaking A static benchmark can become ineffective quickly as models improve, and building replacements is slow and labor-intensive. Writing good multi-step tool-use tasks by hand, then verifying each one works, eats up researcher time that could go elsewhere. TURTLE attacks both problems at once. It automates the grunt work and keeps the difficulty calibrated as capabilities shift. The proposer engine has its own pedigree. AlphaEvolve was introduced in 2025 and is powered by DeepMind’s Gemini models, and TURTLE puts it to work on a new problem: designing tests rather than algorithms. ## What this means for AI evaluation When most tasks sit in the 40% to 60% range, small improvements in a model show up as measurable score changes instead of disappearing into a ceiling or a floor. The experiments used a single solver, Gemma 4 31B-it, so tasks tuned against one model may not land in the same difficulty band for another. If a benchmark is evolved against a particular model, it could end up measuring how different a new model is from that reference, rather than raw capability. A method that works across AppWorld, τ-telecom, τ-banking, and AutomationBench suggests it is not tied to one environment, which improves the odds of broader use. **Disclosure:** This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our [Editorial Policy](https://cryptobriefing.com/editorial-policy/).