{"slug": "token-efficient-pi-through-auto-research", "title": "Token efficient Pi through Auto-Research", "summary": "A research project called SoL-Pi has built a scalable recursive self-improvement pipeline that searches for token-efficient coding-agent harness mechanisms, saving a professional researcher $8.75–$13.50 per hour versus native Codex and Claude Code harnesses and $4.36–$5.71 per hour versus Pi. The pipeline fans out 152 proposed directions, screens them via a pre-rollout Oracle Analysis stage, and validates surviving mechanisms on EdgeBench, a 51-task suite of long-horizon executable agent work reserved for held-out evaluation. The work frames recursive self-improvement's lasting value as a search process that scales across public environments rather than any single artifact.", "body_md": "Research project\n\n# SoL-Pi:Scaling Auto-Research Loops for Efficient Agent Harnesses\n\nSpend less without getting less done.\n\n## Introduction\n\nThe arc of coding agents has been short and steep. They moved from [completing\nlines](https://arxiv.org/abs/2107.03374) to [resolving issues across\nrepositories](https://arxiv.org/abs/2405.15793), acting on feedback and even\nmodifying the tools around them. With modern harnesses, [thousands of agents can\nnow work together continuously for a week without human\nintervention](https://cursor.com/blog/self-driving-codebases). This raises a\nbasic question: over an unsupervised run of hours or days, does every token\ncarry the work forward, or does redundancy grow with trajectory length?\n\nThe question deepens with [recursive\nself-improvement](https://www.anthropic.com/institute/recursive-self-improvement)\n(RSI). Once AI can work on complex software systems, it can also work on [the\nsystems that produce AI](https://openai.com/index/introducing-gpt-5-3-codex/).\nRSI may therefore accelerate progress toward more general intelligence. Yet RSI\nis itself token-intensive: every attempt to produce a better system spends\ntokens, whether or not it succeeds. Before scaling RSI, should we first ask AI\nto make AI more efficient?\n\nTo address this problem, we built a scalable RSI pipeline. Agents construct\nexecutable environments for harness rollouts from **public data** and\n**publicly available software environments**. Other agents observe how models\nexplore within the resulting trajectories, then turn those observations into\nauto-research loops that improve the harness under an efficiency objective. The\nmechanisms that survive these searches are combined into SoL-Pi, an agent\nharness focused on token efficiency. For a professional researcher working on a\nsingle problem, SoL-Pi saves **$8.75–$13.50 per hour** vs. native Codex and\nClaude Code harnesses, and **$4.36–$5.71 per hour** vs. Pi.*\n\nLast but not least, SoL-Pi points to a broader view of RSI: its lasting value may lie not in any single artifact, but in a search process that can scale across public environments to discover reusable improvements.\n\nPi is the lightweight, extensible coding-agent harness that serves as our\nresearch substrate. [EdgeBench](https://edge-bench.org/) is a 51-task suite of\nlong-horizon executable agent work; we reserve its tasks, verifiers, and\nfeedback for final held-out evaluation. * The savings range reflects the model\nbackend and is calculated using official API-equivalent pricing.\n\n## Method\n\nWe treat harness improvement as an open-ended RSI problem. We search across different environments for changes that can transfer to unseen settings. The auto-research process generates mechanism ideas in batches, sends each selected idea into an independent loop, and carries forward candidates that survive validation.\n\nThe target is constrained efficiency: we search for reductions in cost or token use subject to a predeclared capability-preservation criterion. The research procedure remains fixed. Surviving mechanisms define the next research frontier.\n\nAuto-research proposal pool\n\n### Parallel auto-research loops\n\nAt the outer level, SoL-Pi fans out a broad idea pool into independent mechanism lineages. The first pass starts with 152 proposed directions. Oracle Analysis, a pre-rollout stage that estimates opportunity from existing trajectories, screens them before the system spends rollout budget. Each selected direction makes a concrete claim about avoidable work in the harness, so the system can reject a weak lineage without discarding progress elsewhere.\n\nThe six proposal families record where a hypothesis begins, not where a surviving mechanism is eventually implemented. ObservationPack, for example, starts in the Context family as C23 and C24, then becomes a mechanism at the observation boundary.\n\nWithin each lineage, we follow Karpathy's conventional\n[**autoresearch experiment loop**](https://github.com/karpathy/autoresearch/blob/master/program.md#the-experiment-loop):\npropose a change, implement it, run the experiment, read the result, then keep\nor discard the change and repeat. We make three adaptations for harness\nresearch.\n\nFirst, implementation runs as a\n[Ralph Loop](https://github.com/anthropics/claude-code/blob/main/plugins/ralph-wiggum/README.md),\nan iterative implementation loop that continues until an explicit exit\ncondition passes. A separate reviewer checks the implementation and its\nbehavioral contract. Failed reviews return the proposal for revision.\n\nSecond, map-reduce analysis handles multiple exploration trajectories. Independent analyzers inspect each trajectory, and a reducer merges their evidence before the next mechanism proposal.\n\nThird, validation uses two isolated splits. We develop and screen candidates on\nthe training set. Once we freeze the mechanism and acceptance rule, the system\nevaluates them on a held-out test set. **Held-out trajectories never enter\nsubsequent analysis, and no agent inside the auto-research loop sees the\nheld-out results.** A failed held-out evaluation rejects the frozen candidate\nwithout becoming feedback for repair or another search episode.\n\n### From compiled workflows to disposable skill loops\n\nOrchestrating auto-research across hundreds of ideas and hundreds of research iterations is not straightforward. Our workflow went through three designs, each differing in where the orchestration lives, how long its code survives, and what breaks first at scale.\n\n**Compiled workflow.** Each agent flow was described in YAML and compiled into\nan executable workflow, so the orchestration lived in a configuration file and\nwas fixed once compiled. The explicit graph made collaboration and handoff\nstraightforward. At scale the fixed graph was the limit: a precompiled workflow\ncould not cover every edge case, runs repeatedly stopped for human repair, and\noperators could not reconstruct the context of hundreds of concurrent attempts.\n\n**Code orchestration.** A lead agent wrote the coordination code, opened\nsessions, passed messages among them, and assembled each workflow at runtime, so\nthe orchestration lived in code that persisted across experiments. This removed\nthe fixed graph. At scale the coordinator was the limit: keeping every loop\ncompatible meant adding branches, tests, and verification scripts to a\nlong-lived and unbounded codebase, and launching a new experiment could require\nmore than ten hours of changes to it.\n\n**Disposable skill loop.** We maintain one minimal loop template together with\nthe instructions for using it. Each experiment instantiates a fresh copy, sets\nits parameters, runs it, and discards its modified orchestration code when the\nloop ends, so orchestration code survives only for the length of one run.\nScaling becomes repeated template instantiation rather than continued expansion\nof one coordinator. Nothing here is free: the template itself has to be kept\ncorrect, since every experiment inherits it. That single shared dependency has\nbeen easier to maintain than either a fixed graph or a growing coordinator.\n\n### Two environment families separate search from evaluation\n\nWe build 535 executable training environments in two families. The first contains 495 tasks with ground-truth trajectories mined from GitHub issue-PR pairs. The second contains 40 synthetic tasks defined by executable verifiers.\n\nFor the first family, the issue specifies the task and the linked PR provides the accepted patch and change history, which serve as the ground-truth trajectory. We restore the repository to the commit immediately before the fix, install its dependencies in an offline image, and withhold the PR and the maintainer's regression test from the agent. We retain the environment only if the test fails before the PR and passes after it, keeping the task, trajectory, and verifier consistent.\n\nThe second family is verifier-driven. We first generate an executable verifier that defines success, then construct an environment around it for open-ended exploration without a reference trajectory. These 40 environments mainly use Terminal-Bench-style verifiers. To keep EdgeBench a valid held-out benchmark, we do not use EdgeBench data or verifiers as sources or templates for environment synthesis. EdgeBench tasks and feedback remain outside harness search; we use them only for held-out validation. This separation makes its score a stronger measure of generalization to unseen task and verifier designs.\n\n### Capability floors constrain efficiency gains\n\nWhat the search learns is a reusable harness mechanism. The capability floor decides which candidates survive. A cheaper candidate fails if it saves by stopping early, skipping necessary verification, or removing evidence required to finish the task.\n\nEach loop applies two acceptance gates. First, every capability metric must stay within its predeclared tolerance. Second, at least one efficiency metric must improve. Among candidates that pass the capability floor, the loop retains nondominated results. The gate applies to one mechanism at a time, so the small losses it permits can accumulate once mechanisms combine: the assembled harness retains roughly 94% of Pi's average score. What the gate rules out is savings that come from getting less done.\n\n## What auto-research loops discovered\n\nAuto-research produced four narrow mechanisms. They map onto tools, context management, observation compression, and multi-agent delegation.\n\n### ToolsAction Fusionone intent, one turn\n\nBase Pi rollouts exposed a recurring sequence: after editing a file, the coding\nagent often issued a command to test, build, or run the change.\n[Action Fusion](https://github.com/NVlabs/SoL-Pi/tree/main/src/sol-pi/extensions/action-fusion)\nturns that sequence into one tool call. The harness applies the edit and runs\nthe command locally, then returns one combined observation without another\nmodel round trip.\n\n### ContextOnline Context Compactcompact at subtask boundaries\n\nKV-cache reuse usually pushes compaction late into a run. Online Context Compact\nuses a different clock: it decomposes a task into subtasks and reconsiders\ncompaction whenever one finishes.\n[Online Context Compact](https://github.com/NVlabs/SoL-Pi/tree/main/src/sol-pi/extensions/online-context-compact)\nturns semantic completion into a compaction trigger, but acts only when expected\nfuture savings can repay the rewrite.\n\n### ObservationsObservationPackpreserve access, remove repetition\n\nIn base Pi, a large file or tool result reappeared in every later request and\noccupied both context and cache. [ObservationPack](https://github.com/NVlabs/SoL-Pi/tree/main/src/sol-pi/extensions/observation-pack)\nchanges that lifecycle by archiving the payload locally. It leaves a handle and\nshort excerpt in context, then recalls exact pages only when needed. The harness\nkeeps the payload accessible without replaying it.\n\n### DelegationEvidence-Preserving Reducerdelegate reading, verify evidence\n\nIn build and test trajectories, often only a few lines of a long log changed\nthe next decision. [Evidence-Preserving Reducer](https://github.com/NVlabs/SoL-Pi/tree/main/src/sol-pi/extensions/evidence-preserving-reducer)\nuses that boundary to delegate the first reading to a cheaper agent. It binds\nthe resulting receipt to the archived log and verifies every quoted line before\nthe frontier agent sees it. Delegation no longer requires trust in a fluent\nsummary.\n\n## Results\n\nIn this work, we used frontier AI models to scale auto-research loops and build\nSoL-Pi, a new agent harness. All comparisons ran each model backend at `xhigh`,\nthe highest reasoning-effort setting in this evaluation. Across EdgeBench\ntasks, SoL-Pi retains roughly 94% of Pi's average score on both model backends,\nand with GPT-5.6 Sol it exceeds the model's native Codex harness. The efficiency\ngains are larger: compared with Pi, it uses 45-49% fewer tokens and costs about\none-third less; compared with the model-native harnesses, it uses 35-64% fewer\ntokens at list-price API cost 50-54% lower.\n\nWe also evaluated Codex, Pi, and SoL-Pi on 63 Terminal-Bench 4 tasks. The results are summarized below.\n\n| Terminal-Bench 4 · 63 tasks Task success and model cost Reported API-equivalent estimates · USD |  |  |  |  | \n|---|---|---|---|---|\n| Harness | Solved | Sol main calls | Total cost | Per solved task | \n|---|---|---|---|---|\n| Codex | 18/63 | $272.35 | $272.35 | $15.13 | \n| Pi | 18/63 | $286.45 | $286.45 | $15.91 | \n| SoL-Pi | 15/63 | $209.90 | $211.12 | $14.07 | \n\n**Task success and model cost** Reported API-equivalent estimates · USD\n\nPer solved task = total model cost divided by the number of solved tasks. Values are rounded to two decimal places.\n\n**Evaluation scope: 63 CPU-only tasks.** GPU-dependent tasks were excluded because the NVIDIA internal evaluation infrastructure available for this study did not support their required long-running executions.\n\n## Efficient Agent Swarm via SoL-Pi\n\nCan an efficient harness also make a collective search more economical? We tested SoL-Pi on [Anthropic’s original performance take-home](https://github.com/anthropics/original_performance_takehome), a kernel-optimization task scored in simulated machine cycles. The question is not just how many agents can search at once, but how much useful, verifiable progress their shared budget buys.\n\nOne **GPT-5.6 Sol coordinator** running in Codex directs **20 GPT-5.6 Luna workers**, all at `xhigh`. The workers form five groups of four, each with independent workspaces and a local evidence board. SoL-Pi runs inside the Pi workers; the coordinator remains a Codex agent.\n\nThe architecture combines independent exploration with selective exchange. Workers ask group peers to reproduce or combine promising findings; the coordinator passes compact insights between groups and keeps some workers exploring alternatives. A shared best result advances only when the coordinator requests acceptance and an independent check confirms a strict improvement.\n\n### A lower-cost verified frontier\n\nWe ran three independent two-hour trials from the same frozen starter: single-agent Sol, Sol with 20 stock-Pi workers, and Sol with 20 SoL-Pi workers. The two control trials ran sequentially, single-agent first. Each run used fresh sessions without previous runs’ solutions or notes.\n\n| Final independently verified results |  |  |  | \n|---|---|---|---|\n| Configuration | Cycles ↓ | Model cost ↓ | Speed thresholds | \n|---|---|---|---|\n| Sol + 20 SoL-Pi | 1,127 | $60.11 | 8/8 | \n| Single Sol | 1,333 | $39.20 | 8/8 | \n| Sol + 20 stock Pi | 1,366 | $82.12 | 7/8 | \n\nIn this single-run comparison, the SoL-Pi swarm reached **1,127 cycles** at **$60.11** in API-equivalent model cost: **17.5% fewer cycles and 26.8% lower cost** than the stock-Pi swarm. Single-agent Sol remained the least expensive condition at $39.20, but ended at 1,333 cycles. More agents alone did not produce a better result: the stock-Pi swarm cost more than single Sol and finished at 1,366 cycles.\n\n## Experimental protocol, cost accounting, and limits\n\nAll final candidates passed the official correctness check, with the tests and simulator held fixed. The 8/8 and 7/8 figures are speed thresholds, not correctness pass rates. Stock Pi’s 1,366-cycle result missed the strict final threshold of fewer than 1,363 cycles. The 147,734-cycle baseline is the repository’s original starter, not the faster starting point used in some historical take-home comparisons.\n\nThe formal score history contains 35 accepted improvements for SoL-Pi, 29 for single Sol, and 44 for stock Pi. Submitted candidate counts are 494, 29, and 460 respectively; these are not all local experiments attempted by the agents. Raw candidates are verified separately; promotion revalidates an immutable snapshot before recording an accepted commit and JSON score point.\n\nCosts reprice recorded usage at each model’s own Standard API rates, including cached input, native compaction and SoL-Pi reducer calls. Sol and Luna are not billed at the same rate; reasoning tokens are already included in output. These are model-use estimates, not subscription invoices. Preparation, supervisory work and requests without returned usage are outside the plotted totals.\n\nThis is one nonrandomized trial per condition. The earlier SoL-Pi trial paused three times for network and authorization repairs: stopped repair time was excluded from its cumulative two-hour budget, but degraded execution intervals remained counted. The controls began with the repaired configuration and explicitly disabled native Codex memories, plugins and extra subagents. These differences and the lack of replication prevent a causal estimate of SoL-Pi’s effect.\n\n## Discussion\n\n### Token efficiency favors reusable improvements\n\nOptimizing a harness on a closed task set creates an easy path to benchmark overfitting. A search loop can move task-related information into the harness or add special rules for patterns that recur in the training tasks. The score may improve even when the change stops working outside that set.\n\nToken efficiency directs the search toward work that recurs across tasks. Removing repeated context, combining actions that require no intermediate decision, and avoiding repeated transmission of large tool outputs all target waste in the interaction loop. They do not depend on knowing the answer to a particular task, so they have a better chance of transferring to new tasks and models.\n\nAn agent can also reduce token use simply by doing less. SoL-Pi therefore applies a predeclared capability floor: token savings count as an efficiency improvement only when task quality remains within the allowed range.\n\n### EdgeBench makes long-horizon efficiency measurable\n\nWe chose EdgeBench because it exercises agents over trajectories that run for several hours. Among the public benchmarks we could find at the time, it was the only one that consistently supported roughly two to twelve hours of continuous reasoning and tool use. Common alternatives such as Terminal-Bench 2.1 and SWE-bench usually finish within an hour.\n\nThat longer time horizon is essential for measuring token efficiency. On shorter tasks, context replay, large tool outputs, cache writes, and extra model turns have little time to accumulate. Small per-turn savings can be hidden by startup costs and variation between tasks. Over several hours, the same sources of waste recur many times, making harness-level efficiency differences easier to measure.\n\nShorter benchmarks remain useful for many questions, but they do not match our research question. SoL-Pi asks whether a harness can reduce repeated overhead throughout a long trajectory while preserving task quality. EdgeBench provides the time horizon needed for that difference to become measurable.\n\n### Breadth escapes local search basins\n\nWe scale auto-research along two axes. A depth-first strategy (DFS) keeps one lineage alive and repeatedly refines its latest candidate. A breadth-first strategy (BFS) expands many independent ideas, then gives each promising branch an auto-research loop to implement, test, and harden it.\n\nIn our runs, we saw a qualitative pattern: after roughly five to ten iterations,\neven GPT-5.6 Sol at `xhigh` could get stuck in a local basin, making smaller\nadjustments to the same design instead of changing direction. Breadth-first\nsearch behaved differently. Most ideas failed validation, but increasing the\ndiversity of starting ideas occasionally exposed an unusual candidate that a\nlater loop could implement and harden. Those jumps produced some of the most\nuseful harness changes in this project. The conversion rate remained low: only\nabout one out of every forty starting ideas survived validation. Breadth made\nthose rare candidates discoverable.\n\n### Breadth and isolation support generalization\n\nIn our experience, two conditions appear especially important. The first is a diverse pool of ideas created through broad search. The second is robust validation: evaluation environments should be fully isolated from search or training and should reflect the variation and failure modes of real-world deployment. Without both, auto-research can improve performance in one environment without improving the system beyond it.\n\n### Humans set priors and inspect survivors\n\nAs AI systems begin to improve the machinery around themselves, the human role in recursive self-improvement remains unsettled. SoL-Pi is best understood as a hybrid auto-research loop. Humans supplied early priors about harness principles and mechanisms. Before scaling the idea pool, they filtered out directions with limited exploratory value for this project, such as searches over base-harness hyperparameters. Once an idea entered an auto-research loop, the loop ran without human intervention through research and validation. When a candidate survived, humans returned to understand the mechanism the agent had discovered and refactor its code into a clean, maintainable implementation.\n\nWhether fully autonomous RSI is ultimately the right end state remains an open question. In an open-ended loop, drift in goals, evidence, or implementation can compound before humans notice it, and the consequences may be difficult to reverse. The relationship between human judgment and machine-scale research therefore needs sustained study. Where should people provide priors, gate the search budget, review accepted changes, or clean up implementations? Where should they step back?\n\n### Task diversity may scale harness quality\n\nSoL-Pi also ran a preliminary closed loop: agents gathered or synthesized tasks, constructed executable environments, collected trajectories, validated candidates, and updated their own harness.\n\nWe call this direction pretraining the harness: before deployment, harness search sees an expanding stream of self-generated tasks and environments and retains mechanisms that survive across that distribution. This line of research is still at an early stage. We expect to see a scaling law for harnesses analogous to those for models: as compute and the diversity of rollout environments grow, harnesses should become more capable, robust, and efficient.\n\n## Efficiency for efficiency\n\nEfficiency may itself become recursive: a more efficient harness could lower the cost of the auto-research used to build its successor. We plan to use SoL-Pi as the starting point for the next research cycle, where lower per-run costs could let a fixed budget cover more executable environments, trajectories, and research ideas. In this view, efficiency is both an outcome of harness research and a resource for expanding the search that follows, so a more efficient harness may help discover an even more efficient one. We call this possibility efficiency for efficiency. It is a long-term research vision rather than a compounding effect demonstrated by the present study.", "url": "https://wpnews.pro/news/token-efficient-pi-through-auto-research", "canonical_source": "https://nvlabs.github.io/SoL-Pi/", "published_at": "2026-09-13 17:27:24+00:00", "updated_at": "2026-09-13 17:46:45.783857+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-tools", "developer-tools", "ai-safety"], "entities": ["SoL-Pi", "Pi", "Codex", "Claude Code", "EdgeBench", "Oracle Analysis", "ObservationPack", "Karpathy"], "alternates": {"html": "https://wpnews.pro/news/token-efficient-pi-through-auto-research", "markdown": "https://wpnews.pro/news/token-efficient-pi-through-auto-research.md", "text": "https://wpnews.pro/news/token-efficient-pi-through-auto-research.txt", "jsonld": "https://wpnews.pro/news/token-efficient-pi-through-auto-research.jsonld"}}