Frontier agents may not need a human-written self-improvement recipe. That is the question behind Do LLM Agents Still Need to Be Told How to Self-Improve? from Shobhita Sundaram, Cansu Sancaktar, and Phillip Isola at MIT CSAIL, uploaded November 17, 2026.
Instead of shipping a GRPO script or a reflection template, the team gives a frozen teacher model a sandbox, a two-part budget, and four tools β then measures whether the player context it edits helps on held-out games.
The setup in one paragraph #
A player instance plays turn-based games with hidden rules. A teacher instance of the same weights may edit text artifacts β prompts and skill files β but cannot fine-tune weights. The teacher gets Read, Write, Bash, and Gameplay (run a PuzzleScript file and watch the player try it).
There is no prescribed loop like "reflect, critique, update." The teacher must invent when to generate adversarial puzzles, when to run A/B tests, and when to stop spending gameplay budget. Full harness details and prompts are on the project page linked from arXiv.
Why gameplay instead of coding benchmarks? #
Coding leaderboards dominate November headlines β see SWE-CoWork and the merge-rate natural experiment. Gameplay isolates exploration, planning, and world modeling without GitHub confounders.
Each episode is a partially observable MDP: the player sees observations and actions, not transition rules. Winning requires discovering mechanics within a turn horizon and token cap β closer to how ARC-style evals stress abstraction than to patching a known repo.
Budgets and what "self-organized" means #
Algorithm 1 in the paper is intentionally minimal:
- Update budget: number ofWrite edits to player context.
- Gameplay budget: number ofGameplay tool calls the teacher may observe.
- Session ends when either budget hits zero; authors nudge the teacher to continue if it stalls early.
Main experiments use 20 updates and 150 gameplays per teacher session, three seeds per model, with Claude Opus 4.6/5, GPT-5.6-Sol / GPT-6 Astra, and GLM-4.7 / GLM-5.3 at low reasoning effort.
Finding 1: Capability threshold is real #
PuzzleScript-t20 gains for newer Claude generations in the self-org study
Image: MIT CSAIL Newer generations self-improve; older ones in the same families often do not. Figure 4 in the HTML report shows gains on PuzzleScript-t20 for Opus 5 and peers, while earlier generations flatline.
That matters for product teams debating whether to ship auto-curriculum features only on frontier tiers: the behavior is not uniform across model versions you might still host for cost reasons.
Finding 2: Teachers rediscover familiar pedagogy #
Successful sessions score high on an LLM-judged alignment to Zimmerman-style self-regulated learning subprocesses β planning, calibration, monitoring. Observable tactics include:
- Adversarial game authoring after seeing failure modes
- A/B runs with and without optimized skill files
- Validation suites the teacher writes to estimate progress
- More replay on medium win-rate games , echoing learnability objectives from asymmetric self-play literature
These are the same patterns human ML engineers encode explicitly in agent harness guides β here the model assembles them from primitives.
Finding 3: Text artifacts transfer #
Transfer gains on PuzzleScript, ARC-AGI-3, and MazeBench after teacher optimization
Image: MIT CSAIL Evaluation moves beyond self-generated puzzles:
| Benchmark | What changes vs training |
|---|---|
| PuzzleScript-t30 | Longer horizons from the same engine family |
| ARC-AGI-3 L1 | Different 2D engine and representation |
| MazeBench | 3D navigation with isometric ASCII, up to 100 turns |
Improvement is measured as J(c_final) β J(c_0) > 0 on held-out game sets with fixed horizons and token caps β see equation (1) in the paper. The authors also report that Claude Opus 5 self-organized learning can match GEPA-style optimization that uses 20 real in-distribution games β a striking data-efficiency claim worth replication before you rip out human curricula.
Self-organized learning versus prescribed GEPA on PuzzleScript-t20
Image: MIT CSAIL
Safety angle builders should not skip #
The paper frames the same experiment as a capability result and a safety probe: removing a prescribed loop does not prevent capable models from improving themselves in text space.
That connects to governance threads on OpenAI self-improvement tiers and recursive improvement forecasts. If your agent platform lets models write persistent skill files, this paper is evidence those files can compound β for good or for misaligned shortcuts.
How to try the ideas without the full sandbox #
You may not reproduce PuzzleScript tooling this week. You can still steal the evaluation design:
- Split teacher and player sessions with separate budgets.
- Ban external datasets during teacher time; allow only tool-mediated experience.
- Measure only on held-out suites with stricter horizons than training games.
- Log teacher tool traces and score them against self-regulated learning rubrics β cheaper than end-to-end mystery loops.
Pair that with organizational metrics from Together agent evals so you do not confuse game score gains with production merge rates.
PuzzleScript as a deliberate constraint #
PuzzleScript games compile from text rules on a 2D grid. The Gameplay tool rejects broken games and unreachable win states. That constraint keeps teacher creativity inside a verifiable win predicate β closer to unit tests than open-ended web browsing.
When teachers graduate to the Python Gameplay ablation, the paper reports shifted generalization patterns. Tool designers should note: looser sandboxes increase capability but also surface area for reward hacking unless win conditions stay machine-checkable.
Claude Code and Pi harness choices #
Teachers run in Claude Code and Pi sessions with default system prompts disabled, no web search, no subagents, and no automatic memory. That is intentionally austere: the authors wanted to observe native tool composition, not vendor scaffolding.
If you ship coding agents with rich default prompts, you may already be prescribing a self-improve loop the MIT team removed. A/B your defaults against a minimal tool shell when debugging runaway self-edit behavior.
Evaluation-setting awareness and overfitting #
Section 5 (summarized in the abstract) studies what happens when teachers know evaluation horizons via Gameplay tool parameters or hard-coded constraints. Awareness can steer curricula toward transfer tasks β but also encourage overfitting to the announced horizon.
Product teams mirroring this should rotate hidden holdout suites and periodically change horizon caps so skill files do not memorize a single benchmark profile.
Model family snapshot #
The paper batches six models across three families with matched token budgets. Reported winners cluster on newest generations named in the HTML figures (Opus 5, GPT-6 Astra, GLM-5.3). Treat exact score tables as paper-primary; reproduce from PDF before executive slides.
Implications for RSI discourse #
Recursive self-improvement debates often assume weight updates. This work shows context-space improvement already exhibits curriculum discovery and cross-benchmark transfer without gradient steps β relevant to OpenAI self-improvement safety tiers and Jack Clark's RSI timeline.
Guardrails belong on Write targets, budget caps, and human review of skill files β not only on training cluster access.
Transfer benchmarks in more detail #
PuzzleScript-t20 draws twenty human-authored levels with optimal solutions twelve turns or shorter, evaluated at a twenty-turn cap β the same regime teachers see in the canonical tool. PuzzleScript-t30 lengthens horizons and token budgets, testing whether skill files overfit short puzzles.
ARC-AGI-3 Level 1 swaps engines entirely: 2D rules still, but not PuzzleScript syntax. MazeBench adds 3D navigation with isometric ASCII and up to one hundred turns, stressing worlds unlike teacher-generated grids.
Authors estimate inner expectations with Monte Carlo rollouts (Table 2 in the paper). When you port the idea to production agents, keep rollout counts and variance in the appendix β mean reward alone hides brittle policies.
Comparison to prescribed self-improve pipelines #
Many products ship fixed reflect-and-edit loops with human-designed critique prompts. This paper asks what remains when you delete the loop and keep tools. Strong teachers approximate GEPA on real games while generating synthetic curricula β suggesting prescribed loops may be training wheels that strongest models outgrow.
That does not mean delete your harness tomorrow. It means measure marginal value of each scaffold line on held-out business tasks, not only on teacher-game scores.
Reproducibility checklist #
- Fix teacher and player to the same checkpoint revision.
- Log every Gameplay invocation with game hash and outcome.
- Store diffs on skill files per update budget unit.
- Hold out benchmark suites the teacher never sees in tool arguments.
Ship those logs to security review if agents can Write outside game folders β the MIT sandbox is constrained, your repo may not be.
For coding-specific self-improve fights, pair this gameplay evidence with Spillway prompt-injection defenses and MCP misbinding risks: text artifacts that help on puzzles can still be poisoned channels when Write scopes are too broad.
Related reading #
License note: the HTML preprint is CC BY-NC-SA 4.0 β commercial reuse of figures requires checking terms before slides in paid workshops.
Contact for correspondence is listed as shobhita@mit.edu on the arXiv HTML page if you plan independent replication and need benchmark artifacts beyond the PDF.
Appendix C in the paper documents disabled harness features; reproduce those ablations before claiming your product's default agent is "fully autonomous." Small prompt leaks can reintroduce the loops authors removed.
Reading gets you the idea. Building it with feedback gets you the skill.