Sokoban Speedrun for RL

A new benchmark competition, Sokoban Speedrun for RL, challenges researchers to train reinforcement learning agents to solve Sokoban puzzles as fast as possible on a single node. The LLM track achieved a record of 48 minutes and 53 seconds using GRPO on Qwen3-4B-Instruct-2507, while the non-LLM track reached 22 minutes and 24 seconds with a CNN-based agent. The competition aims to accelerate RL training recipes and is open to novel algorithms and optimizations.

Fastest recipes to RL models to solve Sokoban to a held-out target on one node: : RL-fine-tune LLM Track llm-track Qwen3-4B-Instruct-2507 https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507 from 57% to 80% held-out pass@1 on one 8xH100 .: train a Non-LLM Track non-llm-track from-scratch agent on a single H100 . Play Sokoban https://www.jeankaddour.com/sokoban if the task is unfamiliar. | | Record time mm:ss | Description | Date | Log | score | Contributors | |---|---|---|---|---|---|---| | 1 | 48:53 | GRPO, LR 1.6e-6 annealed, 75 steps | 2026-06-29 | | llm/records/2026-06-29 02 /JeanKaddour/sokoban speedrun/blob/main/llm/records/2026-06-29 02 grpo 60step llm/records/2026-07-02 01 /JeanKaddour/sokoban speedrun/blob/main/llm/records/2026-07-02 01 grpo 54step llm/records/2026-07-02 02 /JeanKaddour/sokoban speedrun/blob/main/llm/records/2026-07-02 02 weco strategy 52step llm/records/2026-07-02 03 /JeanKaddour/sokoban speedrun/blob/main/llm/records/2026-07-02 03 grpo 48step llm/records/2026-07-14 01 /JeanKaddour/sokoban speedrun/blob/main/llm/records/2026-07-14 01 weco cispo 48step llm/records/2026-07-15 01 /JeanKaddour/sokoban speedrun/blob/main/llm/records/2026-07-15 01 weco cispo 35step Fastest wall-clock run wins: one run on one 8xH100 node, from training step 1 through the final training update. Score: the lower 95% bootstrap CI of pass@1 on llm/datasets/sokoban eval.jsonl /JeanKaddour/sokoban speedrun/blob/main/llm/datasets/sokoban eval.jsonl — a record must score 0.80 . Eval: 8 completions/puzzle, 12,288 tokens, temperature 0.8, top-p 0.95, seed 12345. Fixed: model, train set /JeanKaddour/sokoban speedrun/blob/main/llm/datasets/sokoban train.jsonl , eval set, reward function, hardware. Open: RL algorithm, loss, schedules, engine, parallelism, domain-agnostic rewards, prompt. Not allowed: Sokoban-specific hints, heuristics, or few-shot examples. Verification: Rerun with a second seed; both runs must score above the target. The score column reports the worse of the two runs. cd llm uv sync NODE GPUS=8 uv run torchrun --standalone --nproc per node=3 -m speedrun uv run python -m eval speedrun --eval-checkpoint outputs/<run /step 000051 This track uses PufferLib's https://github.com/pufferai/pufferlib Boxoban environment; the initial PPO implementation was forked from pufferlib/torch pufferl.py https://github.com/PufferAI/PufferLib/blob/4.0/pufferlib/torch pufferl.py . | | Record time mm:ss | Description | Date | Log | score | Contributors | |---|---|---|---|---|---|---| | 1 | 22:24 | cnn-mingru h256 | 2026-06-21 | | non llm/records/2026-06-29 01 /JeanKaddour/sokoban speedrun/blob/main/non llm/records/2026-06-29 01 non llm non llm/records/2026-06-30 01 /JeanKaddour/sokoban speedrun/blob/main/non llm/records/2026-06-30 01 non llm non llm/records/2026-07-02 01 /JeanKaddour/sokoban speedrun/blob/main/non llm/records/2026-07-02 01 non llm sgpm2 , 950-step anneal non llm/records/2026-07-02 02 /JeanKaddour/sokoban speedrun/blob/main/non llm/records/2026-07-02 02 non llm Fastest wall-clock run wins: one run on a single H100, from training step 1 through the final training update. Score: the lower 95% CI of the held-out solve rate — a record must score 0.70 . Eval: official DeepMind Boxoban https://github.com/google-deepmind/boxoban-levels test split unfiltered/test . Open: policy architecture, RL algorithm, optimizer, schedules, implementation. Verification: Rerun with a second seed; both runs must score above the target. The score column reports the worse of the two runs. cd non llm uv sync uv run python speedrun.py Each track's assemble record.sh llm/ /JeanKaddour/sokoban speedrun/blob/main/llm/assemble record.sh , turns a finished run into a record dir: it collects the log, eval JSON, and source snapshot, builds the report, pins the top-level /JeanKaddour/sokoban speedrun/blob/main/non llm/assemble record.sh non llm/ speedrun.py , runs verify record.py , and adds or refreshes the record's row + redraws the leaderboard and rolling recent-training figures. Configure record runs by editing RECIPE in speedrun.py and launch them flag-free — the pinned speedrun.py then is the recipe assembly rejects flag-configured runs . It reads a local outputs/<RUN / by default; pass SOURCE=modal to pull off the volume.- Train + eval with your track's Running running commands. - Assemble the record: cd llm or: cd non llm RUN=<RUN DEST=records/<date 01 <name ./assemble record.sh The record's README.md is scaffolded with a placeholdersection. Review the pinned Idea speedrun.py diff, then fill in by hand: the record's Idea , and the new leaderboard row's Description + Contributors in this top-level README.md . - Open a PR with the record dir + new row. Optional: verify it yourself with a second seed otherwise the maintainers do; either way both seeds must clear the target , assembled into the record's verification/ subdir: RUN=<VRUN VERIFY OF=records/<date 01 <name ./assemble record.sh The top-level speedrun.py files always hold the current record's recipe. @joshua-a-harris https://github.com/joshua-a-harris 's nanoRL speedrun https://joshuaharrissite.substack.com/p/nanorl , nanochat https://github.com/karpathy/nanochat , modded-nanoGPT https://github.com/KellerJordan/modded-nanogpt , ScaleRL https://arxiv.org/abs/2510.13786 , ReasoningGym for the LLM-track Sokoban env https://github.com/open-thought/reasoning-gym , DeepMind for Boxoban https://github.com/google-deepmind/boxoban-levels and PufferLib for the efficient boxoban implementation https://github.com/PufferAI/PufferLib .