# Coop: A small language model pretrained by volunteers > Source: > Published: 2026-09-30 19:51:50+00:00 A small language model pretrained by volunteers. No server, no funding, no daemon — the whole training loop runs on donated consumer hardware plus the free tiers of Hugging Face and GitHub Actions. **Stage 2 is live**: a ~145M-param model pretraining from scratch on FineWeb-Edu. Stage 1 (15M on TinyStories) completed past its Chinchilla-optimal budget — proof that the whole mechanism works. Current numbers and sample output: [leaderboard](https://github.com/commonsense-ai/coop/blob/ledger/LEADERBOARD.md). The full loop is production-proven, not just designed: multiple volunteers on different machines (Apple Silicon and plain CPU) have trained the same outer step and been averaged into one update — the actual data parallelism. A submission that raced a tick was accepted one step later at reduced staleness weight; repeat rounds from one user merged into a single vote (no token farming); `coop stop` flushed a half-finished round instead of discarding it; and the inbox drains to zero every tick. DiLoCo-style low-communication data parallelism: 1. **Workers** (you) download the current checkpoint from the[HF model repo](https://huggingface.co/commonsense-ai/fineweb-150m) , run`H` local AdamW steps on a personal data shard, and compute a**pseudo-gradient** :`delta = theta_outer - theta_local` . 2. **Submission** is a pull request against a public[HF dataset repo](https://huggingface.co/datasets/commonsense-ai/fineweb-150m-inbox) (the "gradient inbox"), opened with`create_commit(create_pr=True)` . Any free HF account with a write token can submit; the maintainer grants no permissions. 3. **The aggregator** is a GitHub Actions cron job (scheduled every 5 min; GitHub's shared scheduler actually fires anywhere from minutes to a few hours apart — the protocol tolerates any cadence). Each tick is stateless: it reads the checkpoint and the open inbox PRs, drops over-stale submissions, clips and cosine-gates the rest, robust-aggregates them (trimmed mean / geometric median), takes one Nesterov outer step, uploads the new checkpoint, credits contributors in the[ledger](https://github.com/commonsense-ai/coop/tree/ledger/ledger) , regenerates the[leaderboard](https://github.com/commonsense-ai/coop/blob/ledger/LEADERBOARD.md) , and closes the processed PRs. Ledger state lives on the`ledger` branch;`main` only changes through approved pull requests. Weights and optimizer state live **only** on Hugging Face (safetensors). Git holds code, config, and the contributor ledger. Each tick evaluates the new checkpoint on a fixed held-out slice with a fixed seed — the same sequences every time, so a change between steps is the model moving and not the eval sampling something else — and appends one point to [`ledger/history.jsonl`](https://github.com/commonsense-ai/coop/blob/ledger/ledger/history.jsonl). A single val loss says nothing: outer steps move it up as often as down. The direction is a property of the series, so the leaderboard and `coop status` fit a slope over it (against tokens, not outer steps — a step is however much work happened to show up that tick) and report it with its standard error, how many steps improved it, and how long it has been since the best one. "Going down" means the slope clears two standard errors; anything less says so instead of pretending. ~145M parameter decoder-only transformer (12 layers, 14 heads, d=896, 1024 context, 32k byte-level BPE vocab, tied embeddings) on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) — real educational web text. GPUs and Apple Silicon pull their weight here; plain CPUs are better suited to CPU-tier work (see [CONTRIBUTING.md](https://github.com/commonsense-ai/coop/blob/main/CONTRIBUTING.md)). The proof run: a 15M-param model pretrained past its Chinchilla-optimal budget on TinyStories by volunteers in six days, val loss 9.01 → 2.8. It stays usable forever — [commonsense-ai/tinystories-15m](https://huggingface.co/commonsense-ai/tinystories-15m) has the weights, the model card, and a working load-and-generate snippet. The final stage-1 leaderboard is archived as `LEADERBOARD-stage1.md` on the ledger branch. Talk to whatever the volunteers have trained so far — no account, no clone, no training: ``` npx coop-ai run latest # bun: bunx coop-ai run latest ``` It downloads the current checkpoint (cached after the first time), then takes prompts and writes what comes next. `run tinystories` plays the finished stage-1 model instead, which is far more coherent than a run still in progress. One-off and pipeable: ``` coop run latest --prompt "The best way to learn mathematics is" echo "Once upon a time" | coop run tinystories ``` `--tokens`, `--temperature`, `--top-k`, and `--device` are there when you want them; `--revision` pins a specific checkpoint. One command if you have Node or [Bun](https://bun.sh) (it uses [uv](https://docs.astral.sh/uv/) under the hood and tells you how to get it): ``` npx coop-ai start # bun: bunx coop-ai start ``` Or install with uv directly: ``` uv tool install git+https://github.com/commonsense-ai/coop coop start ``` (`uv tool install coop-ai` / `pipx install coop-ai` once the PyPI package clears review.) The first run asks you to paste a Hugging Face [write token](https://huggingface.co/settings/tokens) (free account) — after that it's zero-setup. The worker runs in the background: it builds you a personal data shard (a slice derived from your username so volunteers don't overlap), then trains and submits rounds until you say otherwise. `coop start` then opens the live progress screen, so you can watch the one-time prep and the first round go by: ``` coop · fineweb-150m · naloxene · Apple GPU this round ████████████░░░░░░░░░░░░ 51.2% inner step 256/500 · loss 3.21 · ~2m 02s left the model ███░░░░░░░░░░░░░░░░░░░░░ 12.4% 37.2M of ~300.0M tokens your share ██░░░░░░░░░░░░░░░░░░░░░░ 6.6% rank 2 of 14 · 2,457,600 tokens 3 rounds · 2,457,600 tokens this session — each one submitted for you > keep training (leave this screen) stop contributing ↑↓ move · enter choose · ←→ advanced view · q leave (training keeps going) ``` Left/right swaps the simple view for the advanced one (every field `coop status` prints). Up/down and enter stop the worker without leaving the screen. `q` leaves the screen and keeps training — closing the view never stops anything. ``` coop progress # reopen it any time (`--advanced` starts on the detail view) coop progress --once # one snapshot, no screen; good for pipes coop progress --auto off # `coop start` goes back to printing a summary coop status # the same facts as one printout coop logs -f # watch it work coop stop # stop contributing; `coop start` resumes any time ``` `coop start --rounds 3` contributes a fixed number of rounds and stops by itself, and `coop start --no-progress` skips the screen just this once. Every release publishes `release.json` to the `ledger` branch, and that is the only thing a volunteer's machine polls. `coop status` says when a newer version is out, and: ``` coop update # get it now (works out how you installed coop) coop update --check # what's new, install nothing coop update --auto on # keep it current by itself ``` With `--auto on`, a running worker adopts the new version **between rounds** — never mid-round, so a restart can't cost you trained work — and picks its training back up where it left off. Off by default: nothing on your machine changes unless you ask. A clone is left alone either way; there, `git pull` is the update. One exception, and it only fires when your worker is already broken. If it cannot finish a single round — several failures in a row, a restart, still nothing — it will take a newer version if one exists, even with auto-update off. You asked it to contribute, it isn't contributing, and a fix on the channel is the only thing left that can change that. `coop status` says so, and `coop logs` shows the attempt. Self-updating only works from 0.3.0 on, so an install older than that can't reach it: ``` coop: error: argument cmd: invalid choice: 'update' ``` That means the `coop` on your PATH predates the command. Reinstall it once — `uv tool install --force git+https://github.com/commonsense-ai/coop`, or just use `npx coop-ai`, which always resolves the current code — and it keeps itself current from then on. `uv tool list` is worth a look if you installed early: the package was once named `coop` rather than `coop-ai`, and a leftover of the old name claims the same `coop` executable. Prefer a foreground one-off? `uvx --from git+https://github.com/commonsense-ai/coop coop-join --hf-token hf_...` runs rounds until ctrl-c (`--once` for a single round, `--device cuda|mps|tpu|cpu` to override). coop trains on one when `torch_xla` is importable and it reports a real TPU — detection is automatic, and rounds land on the leaderboard at `tpu` tier. The catch is the environment: `torch_xla` is version-locked to torch, so it has to be installed into the *same* env as coop. From a clone (`uv pip install torch_xla` matched to your torch, then `uv run python -m coop.trainer --data ... --loop`) you control both, which is why that's the route we'd suggest on a TPU VM. `uvx --with torch_xla --from git+https://github.com/commonsense-ai/coop coop-join` is the one-liner shape if you'd rather; whether it resolves depends on your torch. Two things to know before you spend a TPU on this. A single worker process uses one chip — on a v5e-8 that's an eighth of the board, since coop doesn't spawn per-core replicas. And at 15M parameters the steps are small enough that XLA compilation and host overhead eat much of the advantage, so a mid-range GPU is likely to out-earn a TPU here. Playing with the model (`coop run`) deliberately stays on the CPU: XLA recompiles for every new sequence length, which makes generation slower on a TPU than off it. From a clone, the equivalent is: ``` uv sync # NVIDIA box? add `uv pip install --torch-backend auto torch`: # the lockfile ships CPU wheels so aggregator ticks stay fast export HF_TOKEN=hf_... uv run python -m coop.data --skip 0 --docs 20000 uv run python -m coop.trainer --data data/shard_0_20000.bin --loop ``` Run it as often as you like. Accepted submissions earn tokens on the leaderboard; CPU-only machines can also contribute tokenization, dedup, filtering, and eval runs (see [CONTRIBUTING.md](https://github.com/commonsense-ai/coop/blob/main/CONTRIBUTING.md)). ``` export HF_TOKEN=hf_... # write access to both HF repos uv run python -m coop.data --train-tokenizer --skip 0 --docs 50000 uv run python -m coop.aggregate --init # genesis checkpoint at step 0 ``` Then set the `HF_TOKEN` repo secret on GitHub so `.github/workflows/aggregate.yml` can run the tick. ``` uv run pytest -q uv run ruff check . ``` Everything is configured in [config/run.yaml](https://github.com/commonsense-ai/coop/blob/main/config/run.yaml). Architecture rules live in [AGENTS.md](https://github.com/commonsense-ai/coop/blob/main/AGENTS.md). License: undecided — all rights reserved for now, see [LICENSE.md](https://github.com/commonsense-ai/coop/blob/main/LICENSE.md).