BITNET – an autoregressive language model on the 6502 processor

Matt Beton trained a tiny Mamba-based autoregressive language model and built an inference engine to run it on the MOS 6502, an 8-bit processor from 1975 with 32KB RAM, which powered the BBC Micro and Apple II. The project demonstrates that modern language models can operate on extremely constrained hardware.

Matt Beton trained a tiny Mamba-based autoregressive language model and wrote an inference engine to run it on the 8-bit 6502 processor from 1975, with 32KB RAM . The MOS 6502 is an 8-bit microprocessor released in 1975, powering the BBC Micro and the Apple II. I am lucky enough to have access to my dad’s …