# Sokoban Speedrun for RL

> Source: <https://github.com/JeanKaddour/sokoban_speedrun>
> Published: 2026-07-15 21:51:03+00:00

Fastest recipes to RL models to solve Sokoban to a held-out target on one node:

: RL-fine-tune[LLM Track](#llm-track)[Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)from 57% to**>80% held-out pass@1** on one**8xH100**.: train a[Non-LLM Track](#non-llm-track)** from-scratch**agent on a** single H100**.

[Play Sokoban](https://www.jeankaddour.com/sokoban) if the task is unfamiliar.

| # | Record time (mm:ss) | Description | Date | Log | score | Contributors |
|---|---|---|---|---|---|---|
| 1 | 48:53 | GRPO, LR 1.6e-6 annealed, 75 steps | 2026-06-29 |
|

[llm/records/2026-06-29_02](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-06-29_02_grpo_60step)[llm/records/2026-07-02_01](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-02_01_grpo_54step)[llm/records/2026-07-02_02](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-02_02_weco_strategy_52step)[llm/records/2026-07-02_03](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-02_03_grpo_48step)[llm/records/2026-07-14_01](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-14_01_weco_cispo_48step)[llm/records/2026-07-15_01](/JeanKaddour/sokoban_speedrun/blob/main/llm/records/2026-07-15_01_weco_cispo_35step)Fastest wall-clock run wins: one run on one 8xH100 node, from training step 1 through the final training update.

**Score:** the lower 95% bootstrap CI of pass@1 on[llm/datasets/sokoban_eval.jsonl](/JeanKaddour/sokoban_speedrun/blob/main/llm/datasets/sokoban_eval.jsonl)— a record must score >**0.80**.** Eval:**8 completions/puzzle, 12,288 tokens, temperature 0.8, top-p 0.95, seed 12345.** Fixed:**model,[train set](/JeanKaddour/sokoban_speedrun/blob/main/llm/datasets/sokoban_train.jsonl), eval set, reward function, hardware.**Open:** RL algorithm, loss, schedules, engine, parallelism, domain-agnostic rewards, prompt.**Not allowed:** Sokoban-specific hints, heuristics, or few-shot examples.**Verification:** Rerun with a second seed; both runs must score above the target. The score column reports the worse of the two runs.

```
cd llm
uv sync
NODE_GPUS=8 uv run torchrun --standalone --nproc_per_node=3 -m speedrun
uv run python -m eval_speedrun --eval-checkpoint outputs/<run>/step_000051
```

This track uses [PufferLib's](https://github.com/pufferai/pufferlib) Boxoban environment; the initial PPO implementation was forked from [pufferlib/torch_pufferl.py](https://github.com/PufferAI/PufferLib/blob/4.0/pufferlib/torch_pufferl.py).

| # | Record time (mm:ss) | Description | Date | Log | score | Contributors |
|---|---|---|---|---|---|---|
| 1 | 22:24 | cnn-mingru h256 | 2026-06-21 |
|

[non_llm/records/2026-06-29_01](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/records/2026-06-29_01_non_llm)[non_llm/records/2026-06-30_01](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/records/2026-06-30_01_non_llm)[non_llm/records/2026-07-02_01](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/records/2026-07-02_01_non_llm)`sgpm2`

), 950-step anneal[non_llm/records/2026-07-02_02](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/records/2026-07-02_02_non_llm)Fastest wall-clock run wins: one run on a single H100, from training step 1 through the final training update.

**Score:** the lower 95% CI of the held-out solve rate — a record must score >**0.70**.** Eval:**official[DeepMind Boxoban](https://github.com/google-deepmind/boxoban-levels)test split`unfiltered/test`

.**Open:** policy architecture, RL algorithm, optimizer, schedules, implementation.**Verification:** Rerun with a second seed; both runs must score above the target. The score column reports the worse of the two runs.

```
cd non_llm
uv sync
uv run python speedrun.py
```

Each track's `assemble_record.sh`

([ llm/](/JeanKaddour/sokoban_speedrun/blob/main/llm/assemble_record.sh),

[) turns a finished run into a record dir: it collects the log, eval JSON, and source snapshot, builds the report, pins the top-level](/JeanKaddour/sokoban_speedrun/blob/main/non_llm/assemble_record.sh)

`non_llm/`

`speedrun.py`

, runs `verify_record.py`

, and adds or refreshes the record's row + redraws the leaderboard and rolling recent-training figures. Configure record runs by editing `RECIPE`

in `speedrun.py`

and launch them flag-free — the pinned `speedrun.py`

then *is*the recipe (assembly rejects flag-configured runs). It reads a local

`outputs/<RUN>/`

by default; pass `SOURCE=modal`

to pull off the volume.-
**Train + eval** with your track's[Running](#running)commands. -
**Assemble** the record:

```
cd llm        # or: cd non_llm
RUN=<RUN> DEST=records/<date>_01_<name> ./assemble_record.sh
```

The record's

`README.md`

is scaffolded with a placeholdersection. Review the pinned`## Idea`

`speedrun.py`

diff, then fill in by hand: the record's`## Idea`

, and the new leaderboard row's**Description**+** Contributors**in this top-level`README.md`

. -
**Open a PR** with the record dir + new row.

*Optional:* verify it yourself with a second seed (otherwise the maintainers do; either way both seeds must clear the target), assembled into the record's `verification/`

subdir:

```
RUN=<VRUN> VERIFY_OF=records/<date>_01_<name> ./assemble_record.sh
```

The top-level `speedrun.py`

files always hold the current record's recipe.

[@joshua-a-harris](https://github.com/joshua-a-harris)'s [nanoRL speedrun](https://joshuaharrissite.substack.com/p/nanorl), [nanochat](https://github.com/karpathy/nanochat), [modded-nanoGPT](https://github.com/KellerJordan/modded-nanogpt), [ScaleRL](https://arxiv.org/abs/2510.13786), [ReasoningGym for the LLM-track Sokoban env](https://github.com/open-thought/reasoning-gym), [DeepMind for Boxoban](https://github.com/google-deepmind/boxoban-levels) and [PufferLib for the efficient boxoban implementation](https://github.com/PufferAI/PufferLib).