Sokoban Speedrun for RL

Fastest recipes to RL models to solve Sokoban to a held-out target on one node:

: RL-fine-tuneLLM TrackQwen3-4B-Instruct-2507from 57% to**>80% held-out pass@1** on one8xH100.: train aNon-LLM Track** from-scratchagent on a single H100**.

Play Sokoban if the task is unfamiliar.

llm/records/2026-06-29_02llm/records/2026-07-02_01llm/records/2026-07-02_02llm/records/2026-07-02_03llm/records/2026-07-14_01llm/records/2026-07-15_01Fastest wall-clock run wins: one run on one 8xH100 node, from training step 1 through the final training update.

Score: the lower 95% bootstrap CI of pass@1 onllm/datasets/sokoban_eval.jsonl— a record must score >0.80.** Eval:8 completions/puzzle, 12,288 tokens, temperature 0.8, top-p 0.95, seed 12345. Fixed:**model,train set, eval set, reward function, hardware.Open: RL algorithm, loss, schedules, engine, parallelism, domain-agnostic rewards, prompt.Not allowed: Sokoban-specific hints, heuristics, or few-shot examples.Verification: Rerun with a second seed; both runs must score above the target. The score column reports the worse of the two runs.

cd llm
uv sync
NODE_GPUS=8 uv run torchrun --standalone --nproc_per_node=3 -m speedrun
uv run python -m eval_speedrun --eval-checkpoint outputs/<run>/step_000051

This track uses PufferLib's Boxoban environment; the initial PPO implementation was forked from pufferlib/torch_pufferl.py.

non_llm/records/2026-06-29_01non_llm/records/2026-06-30_01non_llm/records/2026-07-02_01sgpm2

), 950-step annealnon_llm/records/2026-07-02_02Fastest wall-clock run wins: one run on a single H100, from training step 1 through the final training update.

Score: the lower 95% CI of the held-out solve rate — a record must score >0.70.** Eval:**officialDeepMind Boxobantest splitunfiltered/test

.Open: policy architecture, RL algorithm, optimizer, schedules, implementation.Verification: Rerun with a second seed; both runs must score above the target. The score column reports the worse of the two runs.

cd non_llm
uv sync
uv run python speedrun.py

Each track's assemble_record.sh

( llm/,

) turns a finished run into a record dir: it collects the log, eval JSON, and source snapshot, builds the report, pins the top-level

non_llm/

speedrun.py

, runs verify_record.py

, and adds or refreshes the record's row + redraws the leaderboard and rolling recent-training figures. Configure record runs by editing RECIPE

in speedrun.py

and launch them flag-free — the pinned speedrun.py

then isthe recipe (assembly rejects flag-configured runs). It reads a local

outputs/<RUN>/

by default; pass SOURCE=modal

to pull off the volume.- Train + eval with your track'sRunningcommands. - Assemble the record:

cd llm        # or: cd non_llm
RUN=<RUN> DEST=records/<date>_01_<name> ./assemble_record.sh

The record's

README.md

is scaffolded with a placeholdersection. Review the pinned## Idea

speedrun.py

diff, then fill in by hand: the record's## Idea

, and the new leaderboard row'sDescription+** Contributors**in this top-levelREADME.md

. - Open a PR with the record dir + new row.

Optional: verify it yourself with a second seed (otherwise the maintainers do; either way both seeds must clear the target), assembled into the record's verification/

subdir:

RUN=<VRUN> VERIFY_OF=records/<date>_01_<name> ./assemble_record.sh

The top-level speedrun.py

files always hold the current record's recipe.

@joshua-a-harris's nanoRL speedrun, nanochat, modded-nanoGPT, ScaleRL, ReasoningGym for the LLM-track Sokoban env, DeepMind for Boxoban and PufferLib for the efficient boxoban implementation.