{"slug": "sokoban-speedrun-for-rl", "title": "Sokoban Speedrun for RL", "summary": "A new benchmark competition, Sokoban Speedrun for RL, challenges researchers to train reinforcement learning agents to solve Sokoban puzzles as fast as possible on a single node. The LLM track achieved a record of 48 minutes and 53 seconds using GRPO on Qwen3-4B-Instruct-2507, while the non-LLM track reached 22 minutes and 24 seconds with a CNN-based agent. The competition aims to accelerate RL training recipes and is open to novel algorithms and optimizations.", "body_md": "Fastest recipes to RL models to solve Sokoban to a held-out target on one node:\n\n: RL-fine-tune[LLM Track](#llm-track)[Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)from 57% to**>80% held-out pass@1** on one**8xH100**.: train a[Non-LLM Track](#non-llm-track)** from-scratch**agent on a** single H100**.\n\n[Play Sokoban](https://www.jeankaddour.com/sokoban) if the task is unfamiliar.\n\n| # | Record time (mm:ss) | Description | Date | Log | score | Contributors |\n|---|---|---|---|---|---|---|\n| 1 | 48:53 | GRPO, LR 1.6e-6 annealed, 75 steps | 2026-06-29 |\n|\n\n[llm/records/2026-06-29_02](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-06-29_02_grpo_60step)[llm/records/2026-07-02_01](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-02_01_grpo_54step)[llm/records/2026-07-02_02](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-02_02_weco_strategy_52step)[llm/records/2026-07-02_03](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-02_03_grpo_48step)[llm/records/2026-07-14_01](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-14_01_weco_cispo_48step)[llm/records/2026-07-15_01](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-15_01_weco_cispo_35step)Fastest wall-clock run wins: one run on one 8xH100 node, from training step 1 through the final training update.\n\n**Score:** the lower 95% bootstrap CI of pass@1 on[llm/datasets/sokoban_eval.jsonl](/JeanKaddour/sokoban_speedrun/blob/main/llm/datasets/sokoban_eval.jsonl)— a record must score >**0.80**.** Eval:**8 completions/puzzle, 12,288 tokens, temperature 0.8, top-p 0.95, seed 12345.** Fixed:**model,[train set](/JeanKaddour/sokoban_speedrun/blob/main/llm/datasets/sokoban_train.jsonl), eval set, reward function, hardware.**Open:** RL algorithm, loss, schedules, engine, parallelism, domain-agnostic rewards, prompt.**Not allowed:** Sokoban-specific hints, heuristics, or few-shot examples.**Verification:** Rerun with a second seed; both runs must score above the target. The score column reports the worse of the two runs.\n\n```\ncd llm\nuv sync\nNODE_GPUS=8 uv run torchrun --standalone --nproc_per_node=3 -m speedrun\nuv run python -m eval_speedrun --eval-checkpoint outputs/<run>/step_000051\n```\n\nThis track uses [PufferLib's](https://github.com/pufferai/pufferlib) Boxoban environment; the initial PPO implementation was forked from [pufferlib/torch_pufferl.py](https://github.com/PufferAI/PufferLib/blob/4.0/pufferlib/torch_pufferl.py).\n\n| # | Record time (mm:ss) | Description | Date | Log | score | Contributors |\n|---|---|---|---|---|---|---|\n| 1 | 22:24 | cnn-mingru h256 | 2026-06-21 |\n|\n\n[non_llm/records/2026-06-29_01](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/records/2026-06-29_01_non_llm)[non_llm/records/2026-06-30_01](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/records/2026-06-30_01_non_llm)[non_llm/records/2026-07-02_01](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/records/2026-07-02_01_non_llm)`sgpm2`\n\n), 950-step anneal[non_llm/records/2026-07-02_02](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/records/2026-07-02_02_non_llm)Fastest wall-clock run wins: one run on a single H100, from training step 1 through the final training update.\n\n**Score:** the lower 95% CI of the held-out solve rate — a record must score >**0.70**.** Eval:**official[DeepMind Boxoban](https://github.com/google-deepmind/boxoban-levels)test split`unfiltered/test`\n\n.**Open:** policy architecture, RL algorithm, optimizer, schedules, implementation.**Verification:** Rerun with a second seed; both runs must score above the target. The score column reports the worse of the two runs.\n\n```\ncd non_llm\nuv sync\nuv run python speedrun.py\n```\n\nEach track's `assemble_record.sh`\n\n([ llm/](/JeanKaddour/sokoban_speedrun/blob/main/llm/assemble_record.sh),\n\n[) turns a finished run into a record dir: it collects the log, eval JSON, and source snapshot, builds the report, pins the top-level](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/assemble_record.sh)\n\n`non_llm/`\n\n`speedrun.py`\n\n, runs `verify_record.py`\n\n, and adds or refreshes the record's row + redraws the leaderboard and rolling recent-training figures. Configure record runs by editing `RECIPE`\n\nin `speedrun.py`\n\nand launch them flag-free — the pinned `speedrun.py`\n\nthen *is*the recipe (assembly rejects flag-configured runs). It reads a local\n\n`outputs/<RUN>/`\n\nby default; pass `SOURCE=modal`\n\nto pull off the volume.-\n**Train + eval** with your track's[Running](#running)commands. -\n**Assemble** the record:\n\n```\ncd llm        # or: cd non_llm\nRUN=<RUN> DEST=records/<date>_01_<name> ./assemble_record.sh\n```\n\nThe record's\n\n`README.md`\n\nis scaffolded with a placeholdersection. Review the pinned`## Idea`\n\n`speedrun.py`\n\ndiff, then fill in by hand: the record's`## Idea`\n\n, and the new leaderboard row's**Description**+** Contributors**in this top-level`README.md`\n\n. -\n**Open a PR** with the record dir + new row.\n\n*Optional:* verify it yourself with a second seed (otherwise the maintainers do; either way both seeds must clear the target), assembled into the record's `verification/`\n\nsubdir:\n\n```\nRUN=<VRUN> VERIFY_OF=records/<date>_01_<name> ./assemble_record.sh\n```\n\nThe top-level `speedrun.py`\n\nfiles always hold the current record's recipe.\n\n[@joshua-a-harris](https://github.com/joshua-a-harris)'s [nanoRL speedrun](https://joshuaharrissite.substack.com/p/nanorl), [nanochat](https://github.com/karpathy/nanochat), [modded-nanoGPT](https://github.com/KellerJordan/modded-nanogpt), [ScaleRL](https://arxiv.org/abs/2510.13786), [ReasoningGym for the LLM-track Sokoban env](https://github.com/open-thought/reasoning-gym), [DeepMind for Boxoban](https://github.com/google-deepmind/boxoban-levels) and [PufferLib for the efficient boxoban implementation](https://github.com/PufferAI/PufferLib).", "url": "https://wpnews.pro/news/sokoban-speedrun-for-rl", "canonical_source": "https://github.com/JeanKaddour/sokoban_speedrun", "published_at": "2026-07-15 21:51:03+00:00", "updated_at": "2026-07-15 22:25:33.979625+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-research"], "entities": ["Qwen", "PufferLib", "DeepMind", "Hugging Face", "Jean Kaddour"], "alternates": {"html": "https://wpnews.pro/news/sokoban-speedrun-for-rl", "markdown": "https://wpnews.pro/news/sokoban-speedrun-for-rl.md", "text": "https://wpnews.pro/news/sokoban-speedrun-for-rl.txt", "jsonld": "https://wpnews.pro/news/sokoban-speedrun-for-rl.jsonld"}}