Running a Language Model on a $35 FPGA

A developer built a 1.07M-parameter Transformer language model that runs entirely on a $35 iCESugar board's Lattice iCE40UP5K FPGA without a softcore CPU, generating mostly coherent output at roughly 10 tokens per second. The design, trained with quantization-aware training for Int4 on the TinyChat dataset with a 2,048-token BPE vocabulary, fits within the chip's 5,280 logic cells, 128 KB SPRAM, 15 KB block RAM, 8 DSPs and 12 MHz clock, and was built with open-source tooling; project files are on GitHub and the checkpoint on Hugging Face.

Running a Language Model on a $35 FPGA I recently took on a project to implement a language model on an FPGA. In simple terms, a Field Programmable Gate Array FPGA is an IC that can be reconfigured to implement hardware circuit designs written in a Hardware Description Language HDL such as SystemVerilog. They are particularly interesting because you don't run software on them the way you would using microcontrollers. Instead, you reconfigure the chip into a hardware design. This gives you things that you couldn't get in software such as true parallelism in combinational logic circuits. This is how projects like MiSTer https://en.wikipedia.org/wiki/MiSTer are able to end up with implementations of consoles that run extremely close to the original hardware. My goal was to: 1. Implement a hardware design capable of running a modern Transformer based language model without a softcore CPU implemented in the design that can run software compiled with toolchains like GCC . 2. Design and train a model that can generate coherent output but is small enough to fit on an FPGA platform - I chose the iCESugar https://github.com/wuxx/icesugar/blob/master/README en.md which has an iCE40UP5K https://www.latticesemi.com/Products/FPGAandCPLD/iCE40UltraPlus 3. Do it all with Opensource tooling I eventually ended up with a 1.07M parameter model trained using the TinyChat https://huggingface.co/datasets/starhopp3r/TinyChat dataset. It's able to generate mostly coherent output at ~10 tokens per second. Here's an interactive port of the model that runs in the browser you can play with to get a feel for the model's behavior. Reply Next-token candidates Experts: which of the 18 the router picked, once per loop Context: attention sees the last 32 tokens, the delta state remembers the rest You can find the project files on GitHub https://github.com/nishtahir/ice40GPT and model checkpoint on Hugging Face https://huggingface.co/nishtahir/ice40GPT . Hardware Constraints The FPGA platform I was working with is the iCE40UP5K https://www.latticesemi.com/Products/FPGAandCPLD/iCE40UltraPlus on an iCESugar https://github.com/wuxx/icesugar/blob/master/README en.md . This is a heavily resource constrained platform. Looking at the specs, the most important features are: - 5,280 logic cells a 4-input LUT and a flip-flop each - This is where the control logic of my design would go. - 128 KB of SPRAM and 15 KB of block RAM - Memory to hold activations, kv cache etc - 8 DSPs that each do a 16x16 multiply - Integer only arithmetic - 8 MB of SPI flash on the board - Slow memory to hold model weights - a 12 MHz clock - Clock to power the design I effectively have 144KB and no native floating point to work with. This meant designing my model from the onset to be quantized for fixed-point math. V1 - 238K Params I started very simple, with a single layer transformer with self attention, quantization aware trained for Int4. The tokenizer is BPE trained on the TinyChat for a Vocab size of 2048. I took some inspiration from tinyworldformer https://github.com/SheepSeb/tinyworldformer to try a looped transformer https://sebastianraschka.com/llm-architecture-gallery/looped-depth-sharing/ . The idea is to feed the outputs back through some or all layers. This gives you a higher effective depth per token without increasing the number of layers in the model. Regular Looped Testing showed that I got the best performance in my model with 6 loops however this is only on the dataset I was working with. The final architecture looks like Ice40GPT tok : Embedding 2048, 64 pos : Embedding 32, 96 up : QLinear in features=64, out features=96, bias=False blocks : ModuleList 0 : Block norm1 : ModuleList 0-5 : 6 x RMSNorm norm2 : ModuleList 0-5 : 6 x RMSNorm qkv : QLinear in features=96, out features=144, bias=False proj : QLinear in features=96, out features=96, bias=False up : QLinear in features=96, out features=384, bias=False down : QLinear in features=384, out features=96, bias=False norm : RMSNorm The design is a state machine that processes each token. It follows the expected process of embed → norm → attention → feed-forward × 6 loops → logits . When the board boots up, the model weights are copied into SPRAM while Block RAM holds the activations and KV cache. Communication with the device is done over UART at 115200 baud. To save on resources: 1. I precompute lookup tables for exp 1/x and 1/sqrt x . 2. Tokenizer encoding/decoding is done off of the board Scaling to 1M Params Now that I had a working design, I wanted to scale the model to 1M params which was ~5x the size of my current model. This meant looking for opportunities to grow the model while respecting my hardware constraints and maintaining performance. So in no particular order, here are improvements I implemented: Mixture of Experts MoE architecture Moving from a dense model allowed me to increase the size of the feed forward network without significantly increasing the size of the matmuls. I also moved the model weights to flash memory and dynamically copy experts that are needed for each token. This allowed me to increase the size of the model to 1M params but maintain only ~200k active params per token 1M-A200k . RoPE to replace positional embeddings Keeping a second embedding table for token positions was costly. ~1.5 KB. Replacing it with Rotary Positional Embedding https://towardsdatascience.com/rope-clearly-explained/ RoPE allowed me to replace the table with a few more dynamic calculations. 1. Split each head's 24 channels into 12 adjacent pairs. 2. Rotate pair i by an angle that grows with the position. The first pair rotates more than the last pair. 3. Do the rotation with a lookup table of cosines and sines Hybrid Gated delta nets for longer context I wanted the model to get better at longer context responses but couldn't scale the context window because of limited RAM, so I opted for Gated delta nets https://sebastianraschka.com/llms-from-scratch/ch04/08 deltanet/ which maintain a fixed memory budget. In my design, I found that it acted a bit more like long term memory, so the self attention attends to tokens present in the context window where delta net output residuals maintain a summary of tokens that have been pushed out. This is what my model architecture looked like after these changes. Ice40GPT tok : Embedding 2048, 64 up : QLinear in features=64, out features=96, bias=False blocks : ModuleList 0 : Block norm1 : ModuleList 0-5 : 6 x RMSNorm norm2 : ModuleList 0-5 : 6 x RMSNorm qkv : QLinear in features=96, out features=144, bias=False proj : QLinear in features=96, out features=96, bias=False delta : DeltaMixer qkv : QLinear in features=96, out features=96, bias=False gate : QLinear in features=96, out features=4, bias=False proj : QLinear in features=32, out features=96, bias=False moe : Experts router : ModuleList 0-5 : 6 x QLinear in features=96, out features=18, bias=False norm : RMSNorm And this is the current utilization report. Info: Device utilisation: Info: ICESTORM LC: 5215/ 5280 98% Info: ICESTORM RAM: 29/ 30 96% Info: SB IO: 12/ 39 30% Info: SB GB: 6/ 8 75% Info: ICESTORM PLL: 0/ 1 0% Info: SB WARMBOOT: 0/ 1 0% Info: ICESTORM DSP: 8/ 8 100% Info: ICESTORM HFOSC: 0/ 1 0% Info: ICESTORM LFOSC: 0/ 1 0% Info: SB I2C: 0/ 2 0% Info: SB SPI: 0/ 2 0% Info: IO I3C: 0/ 2 0% Info: SB LEDDA IP: 0/ 1 0% Info: SB RGBA DRV: 0/ 1 0% Info: ICESTORM SPRAM: 4/ 4 100% Info: SB FILTER 50NS: 0/ 2 0% Info: Placed 12 cells based on constraints. Info: Max frequency for clock 'clk$SB IO IN $glb clk': 22.50 MHz PASS at 12.00 MHz Info: Max frequency for clock 'clk$SB IO IN $glb clk': 21.81 MHz PASS at 12.00 MHz Conclusion Overall, this was a lot of fun. Working on this project has given me a much better understanding of what goes on under the hood. Learning to build with constraints is always instructive. Looking forward, I think there's still a lot of room to optimize in every part of the system. Things I'm thinking about include: 1. Exploring much more aggressive quants 1 bit and ternary https://en.wikipedia.org/wiki/1.58-bit large language model quants for example . 2. There's also an opportunity to make the datasets much more information dense. It needs to have a limited vocabulary and domain for the size, but there's room to explore how much it's possible to pack in such a small model.