Show HN: WiFi-LLM A developer has created WiFi-LLM, a system that runs a Llama-2-architecture model on an ESP32 microcontroller by streaming weights from a PC over WiFi, keeping the ESP32's RAM usage flat at around 110 KiB regardless of model size. The system uses techniques like chunked matrix projections, a sliding-window KV cache, and streamed logits to fit the model on the device, achieving generation speeds bounded by network bandwidth. The project is open-source and supports ESP32-S3 and classic ESP32 boards with ESP-IDF ≥ 5.x. Run a real Llama-2-architecture model on an ESP32 that could never hold it: the weights live on your PC and stream over WiFi, the entire model crossing the air for every single token . The ESP32 keeps only a small working buffer; RAM usage stays flat no matter the model size, and generation is bounded by network bandwidth, not flash or RAM. heap model ready : free 110 KiB, largest block 68 KiB token 9038 'Once' 7.13 s, free heap 110 KiB, worst-ever heap 45 KiB net 5.38 s 57 reqs, 94.4 ms/req, 3103 KB/s matmul 1.58 s other 0.16 s ======= Once ======= token 2501 ' upon' 6.97 s, free heap 110 KiB, worst-ever heap 45 KiB net 5.22 s 57 reqs, 91.6 ms/req, 3200 KB/s matmul 1.58 s other 0.17 s ======= Once upon ======= token 263 ' a' 6.97 s, free heap 110 KiB, worst-ever heap 44 KiB net 5.24 s 57 reqs, 91.8 ms/req, 3190 KB/s matmul 1.57 s other 0.17 s ======= Once upon a ======= token 931 ' time' 6.96 s, free heap 110 KiB, worst-ever heap 44 KiB net 5.20 s 57 reqs, 91.3 ms/req, 3209 KB/s matmul 1.60 s other 0.16 s ======= Once upon a time ======= example output; numbers depend on your WiFi; replace with a capture from your run Per token, the ESP32 sends the whole forward pass's request recipe in one batched write : embedding row, then per layer rms att → wq → wk → wv → wo → rms ffn → w1 → w3 → w2, then the final norm and a classifier sweep, and the compute path drains the responses in the same order: The tricks that make it fit: No framing, order is the contract : requests name a tensor plus up to two slices ; both sides derive response sizes from shapes, so the wire carries pure payload. LLAMA2 Layers.h ChunkedProject : a layer matrix larger than the buffer is consumed in bites of whole rows, each matmul'd into its slice of the output. Chunking never changes the math. Sliding-window KV cache : k/v stored in q8, pre-RoPE, in a sliding window with BOS pinned. Each step feeds one token. Streamed logits : all 32000 classifier rows flow through the buffer; only the running argmax or Gumbel-max sample survives. Full logits never materialize. Weights are q8 32 int8 quants + f32 scale = 36 B per 32 elements ; activations, norms and scores stay f32. You need: an ESP32-S3 or classic ESP32/WROOM on 2.4 GHz WiFi, ESP-IDF ≥ 5.x, a C++17 compiler on the PC, Python 3 with numpy. cd models python download.py stories15M.bin + tokenizer.bin Karpathy's llama2.c artifacts python quantize.py - stories15M-q8.bin, 3.6x smaller Put your WiFi credentials in main/wifi config.h , then: idf.py set-target esp32s3 or esp32 for a WROOM board idf.py build flash monitor The monitor prints the board's IP and listening on port 9000 . cmake -S host -B host/build cmake --build host/build --config Release ./host/build/Release/ModelHost