Token efficient Pi through Auto-Research

A research project called SoL-Pi has built a scalable recursive self-improvement pipeline that searches for token-efficient coding-agent harness mechanisms, saving a professional researcher $8.75–$13.50 per hour versus native Codex and Claude Code harnesses and $4.36–$5.71 per hour versus Pi. The pipeline fans out 152 proposed directions, screens them via a pre-rollout Oracle Analysis stage, and validates surviving mechanisms on EdgeBench, a 51-task suite of long-horizon executable agent work reserved for held-out evaluation. The work frames recursive self-improvement's lasting value as a search process that scales across public environments rather than any single artifact.

Research project SoL-Pi:Scaling Auto-Research Loops for Efficient Agent Harnesses Spend less without getting less done. Introduction The arc of coding agents has been short and steep. They moved from completing lines https://arxiv.org/abs/2107.03374 to resolving issues across repositories https://arxiv.org/abs/2405.15793 , acting on feedback and even modifying the tools around them. With modern harnesses, thousands of agents can now work together continuously for a week without human intervention https://cursor.com/blog/self-driving-codebases . This raises a basic question: over an unsupervised run of hours or days, does every token carry the work forward, or does redundancy grow with trajectory length? The question deepens with recursive self-improvement https://www.anthropic.com/institute/recursive-self-improvement RSI . Once AI can work on complex software systems, it can also work on the systems that produce AI https://openai.com/index/introducing-gpt-5-3-codex/ . RSI may therefore accelerate progress toward more general intelligence. Yet RSI is itself token-intensive: every attempt to produce a better system spends tokens, whether or not it succeeds. Before scaling RSI, should we first ask AI to make AI more efficient? To address this problem, we built a scalable RSI pipeline. Agents construct executable environments for harness rollouts from public data and publicly available software environments . Other agents observe how models explore within the resulting trajectories, then turn those observations into auto-research loops that improve the harness under an efficiency objective. The mechanisms that survive these searches are combined into SoL-Pi, an agent harness focused on token efficiency. For a professional researcher working on a single problem, SoL-Pi saves $8.75–$13.50 per hour vs. native Codex and Claude Code harnesses, and $4.36–$5.71 per hour vs. Pi. Last but not least, SoL-Pi points to a broader view of RSI: its lasting value may lie not in any single artifact, but in a search process that can scale across public environments to discover reusable improvements. Pi is the lightweight, extensible coding-agent harness that serves as our research substrate. EdgeBench https://edge-bench.org/ is a 51-task suite of long-horizon executable agent work; we reserve its tasks, verifiers, and feedback for final held-out evaluation. The savings range reflects the model backend and is calculated using official API-equivalent pricing. Method We treat harness improvement as an open-ended RSI problem. We search across different environments for changes that can transfer to unseen settings. The auto-research process generates mechanism ideas in batches, sends each selected idea into an independent loop, and carries forward candidates that survive validation. The target is constrained efficiency: we search for reductions in cost or token use subject to a predeclared capability-preservation criterion. The research procedure remains fixed. Surviving mechanisms define the next research frontier. Auto-research proposal pool Parallel auto-research loops At the outer level, SoL-Pi fans out a broad idea pool into independent mechanism lineages. The first pass starts with 152 proposed directions. Oracle Analysis, a pre-rollout stage that estimates opportunity from existing trajectories, screens them before the system spends rollout budget. Each selected direction makes a concrete claim about avoidable work in the harness, so the system can reject a weak lineage without discarding progress elsewhere. The six proposal families record where a hypothesis begins, not where a surviving mechanism is eventually implemented. ObservationPack, for example, starts in the Context family as C23 and C24, then becomes a mechanism at the observation boundary. Within each lineage, we follow Karpathy's conventional autoresearch experiment loop https://github.com/karpathy/autoresearch/blob/master/program.md the-experiment-loop : propose a change, implement it, run the experiment, read the result, then keep or discard the change and repeat. We make three adaptations for harness research. First, implementation runs as a Ralph Loop https://github.com/anthropics/claude-code/blob/main/plugins/ralph-wiggum/README.md , an iterative implementation loop that continues until an explicit exit condition passes. A separate reviewer checks the implementation and its behavioral contract. Failed reviews return the proposal for revision. Second, map-reduce analysis handles multiple exploration trajectories. Independent analyzers inspect each trajectory, and a reducer merges their evidence before the next mechanism proposal. Third, validation uses two isolated splits. We develop and screen candidates on the training set. Once we freeze the mechanism and acceptance rule, the system evaluates them on a held-out test set. Held-out trajectories never enter subsequent analysis, and no agent inside the auto-research loop sees the held-out results. A failed held-out evaluation rejects the frozen candidate without becoming feedback for repair or another search episode. From compiled workflows to disposable skill loops Orchestrating auto-research across hundreds of ideas and hundreds of research iterations is not straightforward. Our workflow went through three designs, each differing in where the orchestration lives, how long its code survives, and what breaks first at scale. Compiled workflow. Each agent flow was described in YAML and compiled into an executable workflow, so the orchestration lived in a configuration file and was fixed once compiled. The explicit graph made collaboration and handoff straightforward. At scale the fixed graph was the limit: a precompiled workflow could not cover every edge case, runs repeatedly stopped for human repair, and operators could not reconstruct the context of hundreds of concurrent attempts. Code orchestration. A lead agent wrote the coordination code, opened sessions, passed messages among them, and assembled each workflow at runtime, so the orchestration lived in code that persisted across experiments. This removed the fixed graph. At scale the coordinator was the limit: keeping every loop compatible meant adding branches, tests, and verification scripts to a long-lived and unbounded codebase, and launching a new experiment could require more than ten hours of changes to it. Disposable skill loop. We maintain one minimal loop template together with the instructions for using it. Each experiment instantiates a fresh copy, sets its parameters, runs it, and discards its modified orchestration code when the loop ends, so orchestration code survives only for the length of one run. Scaling becomes repeated template instantiation rather than continued expansion of one coordinator. Nothing here is free: the template itself has to be kept correct, since every experiment inherits it. That single shared dependency has been easier to maintain than either a fixed graph or a growing coordinator. Two environment families separate search from evaluation We build 535 executable training environments in two families. The first contains 495 tasks with ground-truth trajectories mined from GitHub issue-PR pairs. The second contains 40 synthetic tasks defined by executable verifiers. For the first family, the issue specifies the task and the linked PR provides the accepted patch and change history, which serve as the ground-truth trajectory. We restore the repository to the commit immediately before the fix, install its dependencies in an offline image, and withhold the PR and the maintainer's regression test from the agent. We retain the environment only if the test fails before the PR and passes after it, keeping the task, trajectory, and verifier consistent. The second family is verifier-driven. We first generate an executable verifier that defines success, then construct an environment around it for open-ended exploration without a reference trajectory. These 40 environments mainly use Terminal-Bench-style verifiers. To keep EdgeBench a valid held-out benchmark, we do not use EdgeBench data or verifiers as sources or templates for environment synthesis. EdgeBench tasks and feedback remain outside harness search; we use them only for held-out validation. This separation makes its score a stronger measure of generalization to unseen task and verifier designs. Capability floors constrain efficiency gains What the search learns is a reusable harness mechanism. The capability floor decides which candidates survive. A cheaper candidate fails if it saves by stopping early, skipping necessary verification, or removing evidence required to finish the task. Each loop applies two acceptance gates. First, every capability metric must stay within its predeclared tolerance. Second, at least one efficiency metric must improve. Among candidates that pass the capability floor, the loop retains nondominated results. The gate applies to one mechanism at a time, so the small losses it permits can accumulate once mechanisms combine: the assembled harness retains roughly 94% of Pi's average score. What the gate rules out is savings that come from getting less done. What auto-research loops discovered Auto-research produced four narrow mechanisms. They map onto tools, context management, observation compression, and multi-agent delegation. ToolsAction Fusionone intent, one turn Base Pi rollouts exposed a recurring sequence: after editing a file, the coding agent often issued a command to test, build, or run the change. Action Fusion https://github.com/NVlabs/SoL-Pi/tree/main/src/sol-pi/extensions/action-fusion turns that sequence into one tool call. The harness applies the edit and runs the command locally, then returns one combined observation without another model round trip. ContextOnline Context Compactcompact at subtask boundaries KV-cache reuse usually pushes compaction late into a run. Online Context Compact uses a different clock: it decomposes a task into subtasks and reconsiders compaction whenever one finishes. Online Context Compact https://github.com/NVlabs/SoL-Pi/tree/main/src/sol-pi/extensions/online-context-compact turns semantic completion into a compaction trigger, but acts only when expected future savings can repay the rewrite. ObservationsObservationPackpreserve access, remove repetition In base Pi, a large file or tool result reappeared in every later request and occupied both context and cache. ObservationPack https://github.com/NVlabs/SoL-Pi/tree/main/src/sol-pi/extensions/observation-pack changes that lifecycle by archiving the payload locally. It leaves a handle and short excerpt in context, then recalls exact pages only when needed. The harness keeps the payload accessible without replaying it. DelegationEvidence-Preserving Reducerdelegate reading, verify evidence In build and test trajectories, often only a few lines of a long log changed the next decision. Evidence-Preserving Reducer https://github.com/NVlabs/SoL-Pi/tree/main/src/sol-pi/extensions/evidence-preserving-reducer uses that boundary to delegate the first reading to a cheaper agent. It binds the resulting receipt to the archived log and verifies every quoted line before the frontier agent sees it. Delegation no longer requires trust in a fluent summary. Results In this work, we used frontier AI models to scale auto-research loops and build SoL-Pi, a new agent harness. All comparisons ran each model backend at xhigh , the highest reasoning-effort setting in this evaluation. Across EdgeBench tasks, SoL-Pi retains roughly 94% of Pi's average score on both model backends, and with GPT-5.6 Sol it exceeds the model's native Codex harness. The efficiency gains are larger: compared with Pi, it uses 45-49% fewer tokens and costs about one-third less; compared with the model-native harnesses, it uses 35-64% fewer tokens at list-price API cost 50-54% lower. We also evaluated Codex, Pi, and SoL-Pi on 63 Terminal-Bench 4 tasks. The results are summarized below. | Terminal-Bench 4 · 63 tasks Task success and model cost Reported API-equivalent estimates · USD | | | | | |---|---|---|---|---| | Harness | Solved | Sol main calls | Total cost | Per solved task | |---|---|---|---|---| | Codex | 18/63 | $272.35 | $272.35 | $15.13 | | Pi | 18/63 | $286.45 | $286.45 | $15.91 | | SoL-Pi | 15/63 | $209.90 | $211.12 | $14.07 | Task success and model cost Reported API-equivalent estimates · USD Per solved task = total model cost divided by the number of solved tasks. Values are rounded to two decimal places. Evaluation scope: 63 CPU-only tasks. GPU-dependent tasks were excluded because the NVIDIA internal evaluation infrastructure available for this study did not support their required long-running executions. Efficient Agent Swarm via SoL-Pi Can an efficient harness also make a collective search more economical? We tested SoL-Pi on Anthropic’s original performance take-home https://github.com/anthropics/original performance takehome , a kernel-optimization task scored in simulated machine cycles. The question is not just how many agents can search at once, but how much useful, verifiable progress their shared budget buys. One GPT-5.6 Sol coordinator running in Codex directs 20 GPT-5.6 Luna workers , all at xhigh . The workers form five groups of four, each with independent workspaces and a local evidence board. SoL-Pi runs inside the Pi workers; the coordinator remains a Codex agent. The architecture combines independent exploration with selective exchange. Workers ask group peers to reproduce or combine promising findings; the coordinator passes compact insights between groups and keeps some workers exploring alternatives. A shared best result advances only when the coordinator requests acceptance and an independent check confirms a strict improvement. A lower-cost verified frontier We ran three independent two-hour trials from the same frozen starter: single-agent Sol, Sol with 20 stock-Pi workers, and Sol with 20 SoL-Pi workers. The two control trials ran sequentially, single-agent first. Each run used fresh sessions without previous runs’ solutions or notes. | Final independently verified results | | | | |---|---|---|---| | Configuration | Cycles ↓ | Model cost ↓ | Speed thresholds | |---|---|---|---| | Sol + 20 SoL-Pi | 1,127 | $60.11 | 8/8 | | Single Sol | 1,333 | $39.20 | 8/8 | | Sol + 20 stock Pi | 1,366 | $82.12 | 7/8 | In this single-run comparison, the SoL-Pi swarm reached 1,127 cycles at $60.11 in API-equivalent model cost: 17.5% fewer cycles and 26.8% lower cost than the stock-Pi swarm. Single-agent Sol remained the least expensive condition at $39.20, but ended at 1,333 cycles. More agents alone did not produce a better result: the stock-Pi swarm cost more than single Sol and finished at 1,366 cycles. Experimental protocol, cost accounting, and limits All final candidates passed the official correctness check, with the tests and simulator held fixed. The 8/8 and 7/8 figures are speed thresholds, not correctness pass rates. Stock Pi’s 1,366-cycle result missed the strict final threshold of fewer than 1,363 cycles. The 147,734-cycle baseline is the repository’s original starter, not the faster starting point used in some historical take-home comparisons. The formal score history contains 35 accepted improvements for SoL-Pi, 29 for single Sol, and 44 for stock Pi. Submitted candidate counts are 494, 29, and 460 respectively; these are not all local experiments attempted by the agents. Raw candidates are verified separately; promotion revalidates an immutable snapshot before recording an accepted commit and JSON score point. Costs reprice recorded usage at each model’s own Standard API rates, including cached input, native compaction and SoL-Pi reducer calls. Sol and Luna are not billed at the same rate; reasoning tokens are already included in output. These are model-use estimates, not subscription invoices. Preparation, supervisory work and requests without returned usage are outside the plotted totals. This is one nonrandomized trial per condition. The earlier SoL-Pi trial paused three times for network and authorization repairs: stopped repair time was excluded from its cumulative two-hour budget, but degraded execution intervals remained counted. The controls began with the repaired configuration and explicitly disabled native Codex memories, plugins and extra subagents. These differences and the lack of replication prevent a causal estimate of SoL-Pi’s effect. Discussion Token efficiency favors reusable improvements Optimizing a harness on a closed task set creates an easy path to benchmark overfitting. A search loop can move task-related information into the harness or add special rules for patterns that recur in the training tasks. The score may improve even when the change stops working outside that set. Token efficiency directs the search toward work that recurs across tasks. Removing repeated context, combining actions that require no intermediate decision, and avoiding repeated transmission of large tool outputs all target waste in the interaction loop. They do not depend on knowing the answer to a particular task, so they have a better chance of transferring to new tasks and models. An agent can also reduce token use simply by doing less. SoL-Pi therefore applies a predeclared capability floor: token savings count as an efficiency improvement only when task quality remains within the allowed range. EdgeBench makes long-horizon efficiency measurable We chose EdgeBench because it exercises agents over trajectories that run for several hours. Among the public benchmarks we could find at the time, it was the only one that consistently supported roughly two to twelve hours of continuous reasoning and tool use. Common alternatives such as Terminal-Bench 2.1 and SWE-bench usually finish within an hour. That longer time horizon is essential for measuring token efficiency. On shorter tasks, context replay, large tool outputs, cache writes, and extra model turns have little time to accumulate. Small per-turn savings can be hidden by startup costs and variation between tasks. Over several hours, the same sources of waste recur many times, making harness-level efficiency differences easier to measure. Shorter benchmarks remain useful for many questions, but they do not match our research question. SoL-Pi asks whether a harness can reduce repeated overhead throughout a long trajectory while preserving task quality. EdgeBench provides the time horizon needed for that difference to become measurable. Breadth escapes local search basins We scale auto-research along two axes. A depth-first strategy DFS keeps one lineage alive and repeatedly refines its latest candidate. A breadth-first strategy BFS expands many independent ideas, then gives each promising branch an auto-research loop to implement, test, and harden it. In our runs, we saw a qualitative pattern: after roughly five to ten iterations, even GPT-5.6 Sol at xhigh could get stuck in a local basin, making smaller adjustments to the same design instead of changing direction. Breadth-first search behaved differently. Most ideas failed validation, but increasing the diversity of starting ideas occasionally exposed an unusual candidate that a later loop could implement and harden. Those jumps produced some of the most useful harness changes in this project. The conversion rate remained low: only about one out of every forty starting ideas survived validation. Breadth made those rare candidates discoverable. Breadth and isolation support generalization In our experience, two conditions appear especially important. The first is a diverse pool of ideas created through broad search. The second is robust validation: evaluation environments should be fully isolated from search or training and should reflect the variation and failure modes of real-world deployment. Without both, auto-research can improve performance in one environment without improving the system beyond it. Humans set priors and inspect survivors As AI systems begin to improve the machinery around themselves, the human role in recursive self-improvement remains unsettled. SoL-Pi is best understood as a hybrid auto-research loop. Humans supplied early priors about harness principles and mechanisms. Before scaling the idea pool, they filtered out directions with limited exploratory value for this project, such as searches over base-harness hyperparameters. Once an idea entered an auto-research loop, the loop ran without human intervention through research and validation. When a candidate survived, humans returned to understand the mechanism the agent had discovered and refactor its code into a clean, maintainable implementation. Whether fully autonomous RSI is ultimately the right end state remains an open question. In an open-ended loop, drift in goals, evidence, or implementation can compound before humans notice it, and the consequences may be difficult to reverse. The relationship between human judgment and machine-scale research therefore needs sustained study. Where should people provide priors, gate the search budget, review accepted changes, or clean up implementations? Where should they step back? Task diversity may scale harness quality SoL-Pi also ran a preliminary closed loop: agents gathered or synthesized tasks, constructed executable environments, collected trajectories, validated candidates, and updated their own harness. We call this direction pretraining the harness: before deployment, harness search sees an expanding stream of self-generated tasks and environments and retains mechanisms that survive across that distribution. This line of research is still at an early stage. We expect to see a scaling law for harnesses analogous to those for models: as compute and the diversity of rollout environments grow, harnesses should become more capable, robust, and efficient. Efficiency for efficiency Efficiency may itself become recursive: a more efficient harness could lower the cost of the auto-research used to build its successor. We plan to use SoL-Pi as the starting point for the next research cycle, where lower per-run costs could let a fixed budget cover more executable environments, trajectories, and research ideas. In this view, efficiency is both an outcome of harness research and a resource for expanding the search that follows, so a more efficient harness may help discover an even more efficient one. We call this possibility efficiency for efficiency. It is a long-term research vision rather than a compounding effect demonstrated by the present study.