{"slug": "show-hn-wifi-llm", "title": "Show HN: WiFi-LLM", "summary": "A developer has created WiFi-LLM, a system that runs a Llama-2-architecture model on an ESP32 microcontroller by streaming weights from a PC over WiFi, keeping the ESP32's RAM usage flat at around 110 KiB regardless of model size. The system uses techniques like chunked matrix projections, a sliding-window KV cache, and streamed logits to fit the model on the device, achieving generation speeds bounded by network bandwidth. The project is open-source and supports ESP32-S3 and classic ESP32 boards with ESP-IDF ≥ 5.x.", "body_md": "Run a real Llama-2-architecture model on an ESP32 that could never hold it: the\nweights live on your PC and stream over WiFi, **the entire model crossing the\nair for every single token**. The ESP32 keeps only a small working buffer;\nRAM usage stays flat no matter the model size, and generation is bounded by\nnetwork bandwidth, not flash or RAM.\n\n```\nheap [model ready]: free 110 KiB, largest block 68 KiB\ntoken   9038 'Once'  (7.13 s, free heap 110 KiB, worst-ever heap 45 KiB)\n        net 5.38 s (57 reqs, 94.4 ms/req, 3103 KB/s) matmul 1.58 s other 0.16 s\n=======\nOnce\n=======\ntoken   2501 ' upon'  (6.97 s, free heap 110 KiB, worst-ever heap 45 KiB)\n        net 5.22 s (57 reqs, 91.6 ms/req, 3200 KB/s) matmul 1.58 s other 0.17 s\n=======\nOnce upon\n=======\ntoken    263 ' a'  (6.97 s, free heap 110 KiB, worst-ever heap 44 KiB)\n        net 5.24 s (57 reqs, 91.8 ms/req, 3190 KB/s) matmul 1.57 s other 0.17 s\n=======\nOnce upon a\n=======\ntoken    931 ' time'  (6.96 s, free heap 110 KiB, worst-ever heap 44 KiB)\n        net 5.20 s (57 reqs, 91.3 ms/req, 3209 KB/s) matmul 1.60 s other 0.16 s\n=======\nOnce upon a time\n=======\n```\n\n*(example output; numbers depend on your WiFi; replace with a capture from your run)*\n\nPer token, the ESP32 sends the **whole forward pass's request recipe in one\nbatched write**: embedding row, then per layer rms_att → wq → wk → wv → wo →\nrms_ffn → w1 → w3 → w2, then the final norm and a classifier sweep, and the\ncompute path drains the responses in the same order:\n\nThe tricks that make it fit:\n\n**No framing, order is the contract**: requests name a tensor plus up to two slices (); both sides derive response sizes from shapes, so the wire carries pure payload.`LLAMA2_Layers.h`\n\n**ChunkedProject**: a layer matrix larger than the buffer is consumed in bites of whole rows, each matmul'd into its slice of the output. Chunking never changes the math.**Sliding-window KV cache**: k/v stored in q8, pre-RoPE, in a sliding window with BOS pinned. Each step feeds one token.** Streamed logits**: all 32000 classifier rows flow through the buffer; only the running argmax (or Gumbel-max sample) survives. Full logits never materialize.**Weights are q8**(32 int8 quants + f32 scale = 36 B per 32 elements); activations, norms and scores stay f32.\n\nYou need: an ESP32-S3 (or classic ESP32/WROOM) on 2.4 GHz WiFi, ESP-IDF ≥ 5.x, a C++17 compiler on the PC, Python 3 with numpy.\n\n```\ncd models\npython download.py    # stories15M.bin + tokenizer.bin (Karpathy's llama2.c artifacts)\npython quantize.py    # -> stories15M-q8.bin, 3.6x smaller\n```\n\nPut your WiFi credentials in `main/wifi_config.h`\n\n, then:\n\n```\nidf.py set-target esp32s3     # or esp32 for a WROOM board\nidf.py build flash monitor\n```\n\nThe monitor prints the board's IP and `listening on port 9000`\n\n.\n\n```\ncmake -S host -B host/build\ncmake --build host/build --config Release\n./host/build/Release/ModelHost <esp32-ip>     # run from the repo root\n```\n\nModelHost connects, greets with the model config, and the ESP32 starts generating a TinyStories tale token by token, one connection per story; it listens again after EOS.\n\n| target | buffer | tok/s | notes |\n|---|---|---|---|\n| ESP32-S3 | 96 KiB | fill in |\n~17 MB downloaded per token |\n| ESP32 (WROOM) | 48 KiB | fill in |\nsmaller TCP window, less DRAM |\n\nThe per-token log splits time into `net`\n\n(wire), `matmul`\n\n, and `other`\n\n, so you\ncan see exactly what you're bound by. Tuning lives in\n[ sdkconfig.defaults](/bertaye/wifi-llm/blob/main/sdkconfig.defaults) (TCP window, WiFi RX buffers) with\nWROOM overrides in\n\n`sdkconfig.defaults.esp32`\n\n.All matmuls are plain scalar loops, no SIMD yet. The ESP32-S3 has 128-bit\nvector instructions (PIE) that fit the q8 int8 dot products well, so there is\nknown headroom on the `matmul`\n\nshare whenever the network stops being the\nbottleneck.\n\n```\nmain/            ESP32 app: main.cpp, WifiManager, Esp32MemoryManager\nmain/tensor/     Tensor.h (q8/f32), TensorOps.h, StreamingTensor.h\nmain/llama2/     StreamingModel.h (ESP32), Model.h (PC reference), wire protocol\nhost/            ModelHost.cpp: loads the model, serves weight slices\nmodels/          download.py, quantize.py, model binaries (not in git)\n```\n\n`Model.h`\n\nruns the same model standalone on the PC (with and without KV cache)\nand doubles as the reference implementation the streaming side is checked\nagainst.\n\nModel, tokenizer and checkpoint format from Andrej Karpathy's\n[llama2.c](https://github.com/karpathy/llama2.c) (stories15M trained on\n[TinyStories](https://arxiv.org/abs/2305.07759)). PC sockets via\n[kissnet](https://github.com/Ybalrid/kissnet).", "url": "https://wpnews.pro/news/show-hn-wifi-llm", "canonical_source": "https://github.com/bertaye/wifi-llm", "published_at": "2026-07-21 13:32:07+00:00", "updated_at": "2026-07-21 13:42:41.469984+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["WiFi-LLM", "Llama-2", "ESP32", "ESP32-S3", "ESP-IDF", "TinyStories", "Karpathy", "ModelHost"], "alternates": {"html": "https://wpnews.pro/news/show-hn-wifi-llm", "markdown": "https://wpnews.pro/news/show-hn-wifi-llm.md", "text": "https://wpnews.pro/news/show-hn-wifi-llm.txt", "jsonld": "https://wpnews.pro/news/show-hn-wifi-llm.jsonld"}}