Strangers pretrained a language model with HF PRs and a cron job

A group of researchers pretrained a 15M-parameter language model from scratch using a GitHub Actions cron job as the training coordinator and pull requests as gradients, with no dedicated cluster or budget. The project, hosted on GitHub under commonsense-ai/coop, attracted a stranger who trained 23M tokens of the model, and applications are open until July 27.

| |||||||||||| 1 point by | Last week a few of us pretrained a 15M-parameter language model from scratch — past its Chinchilla-optimal token budget — with no cluster, no server, and no budget. The training coordinator is a GitHub Actions cron job. The gradients are pull requests. By the end, someone we'd never met had joined and trained 23M tokens of it. github.com/commonsense-ai/coop — Model + card · huggingface.co/commonsense-ai/tinystories-15m — Gradient inbox — Leaderboard | ||||||||||| Applications are open till July 27. |