Dream-RSI optimizes AI agent search strategies offline by replaying past discovery trees as exact simulators, reducing execution costs across discovery tasks.
Optimizing a long-horizon exploration policy gives slow, expensive feedback, because each policy trial means a full search run of live model calls and code executions. Dream-RSI removes that cost from the inner loop by treating completed search history as a replay environment.
Dream-RSI improves an AI agent's search strategy by replaying the agent's own past runs as exact simulators, scoring thousands of candidate strategies offline at zero execution cost before redeploying the best one.
Across eight discovery tasks in three domains it matches or beats state-of-the-art baselines while using up to 162× fewer agent calls than the SimpleTES baseline on the Lasso regularization path, 1.79× to 2.43× fewer generations on GPU kernel engineering, and over 50× less budget on mathematical optimization.
The method comes from a 2026 technical report by Google, Google DeepMind, the University of Maryland, and the University of Virginia.
Each cycle has three stages:
Every node in a recorded tree already holds its code output, evaluation diagnostics, and score. A candidate policy walks the recorded tree in a different order, batch size, or stopping point and reads those cached outcomes from disk. No model generation runs and no code executes, so a full policy evaluation reduces to memory lookups. Replay returns only recorded children, so a candidate cannot reach branches that were never realized during the original run.
The exploration policy is executable code that decides where to branch, which attempts run in parallel up to the worker budget, how deep to push each branch, and when to stop.
Each candidate policy $m$ is scored on tree $i$ by:
$$V_i^m = \max_{v \in T_i^{m,k_i^{m,\star}}} s_v ;-; \beta_1 N_i^m ;+; \beta_2 \frac{N_i^m}{\max{1,, k_i^{m,\star}}}$$ The first term is the best solution quality $s_v$ reached along the replayed subtree. $N_i^m$ is the number of revealed non-root nodes, which counts the generation-evaluation requests the policy would have spent. $k_i^{m,\star}$ is the number of completed decision rounds. $\beta_1$ penalizes total probe count and $\beta_2$ rewards parallelism per round, so the score trades discovery quality against execution cost and sequential delay.
The next policy is the revision with the highest average replay score across all historical trees. The currently deployed policy is included in the candidate set as the base version, so the winner scores at least as high as the incumbent on that history. Replay score therefore never decreases from one cycle to the next on the recorded traces.
Within a single episode the policy uses a fixed scalar $\beta$ (distinct from $\beta_1$ and $\beta_2$) routed through a schedule function that sets exploration width, pruning aggressiveness, and search patience. Its value for the next cycle is chosen by four deterministic rules that read live performance and offline sweep curves:
This is not Bayesian optimization. There are no priors, likelihoods, or posteriors over hyperparameters. The framework treats the whole policy as arbitrary Python rewritten by an LLM rather than a fixed parameter vector being tuned.
Injecting high-level directional summaries into the agent's prompts scored lower than unguided replay under equal budgets. The directional priors over-constrain the search space and reduce exploration diversity across parallel threads. The ConvDiv ablation shows this gap.
Non-degradation holds on the historical traces, not on unseen tasks; a policy that over-indexes on past trajectories can still drop on held-out work through distribution shift. Dream-RSI limits this three ways: the simulator pool grows every round to reduce variance, policy code runs under strict prefix-only observability with anti-leakage rules that forbid absolute score targets or hardcoded winning node IDs, and replay stays inside the realized search space.
A learned world model predicts transitions with an approximate neural network. A recorded discovery tree returns the exact outcomes that were realized, so replay is an exact simulator over the search space that actually happened rather than a prediction of it.
The framework was tested on eight tasks across three domains:
In each domain Dream-RSI matched or exceeded the strongest baseline, including the Recursive Fixed Exploration baseline.
Dream-RSI, 2026 technical report by Google, Google DeepMind, the University of Maryland, and the University of Virginia. Project page: dream-rsi.com.