{"slug": "hn-i-ran-a-56m-parameter-llm-across-three-esp32-s3-boards-via-esp-now", "title": "HN: I ran a 56M parameter LLM across three ESP32-S3 boards via ESP-NOW", "summary": "A developer ran a 56M-parameter language model across three ESP32-S3 N16R8 microcontrollers using ESP-NOW wireless communication, achieving distributed inference of a 50.3M-parameter PLE-based transformer trained on WikiText-103. The system splits the model across boards with a web interface for real-time text generation, demonstrating feasibility of running large language models on low-power embedded devices.", "body_md": "**Distributed micro-LLM inference across three ESP32-S3 N16R8 boards with ESP-NOW communication.**\n\nThis project implements a **distributed AI system** that runs a **56M-parameter language model** across three ESP32-S3 microcontrollers. Inspired by [slvDev/esp32-ai](https://github.com/slvDev/esp32-ai), which demonstrated running TinyStories on a single board, this project extends the architecture to a multi-board distributed system with web-based interaction.\n\nThe model is trained on **WikiText-103** (Wikipedia corpus) using **Per-Layer Embeddings (PLE)** from Google's Gemma architecture, quantized to 4-bit, and split across three boards that communicate via ESP-NOW wireless protocol. The 50.3M-parameter PLE table is split across Board A and Board B (**Split-PLE**) to fit within the 16MB flash per board. Board B maintains a **KV cache** in PSRAM (1.5 MB for 256 positions), enabling the transformer to attend to the full generated sequence instead of operating token-by-token.\n\n```\n┌───────────────────┐      ESP-NOW      ┌───────────────────┐      ESP-NOW     ┌───────────────────┐\n│    Board A        │ ◄──────────────► │    Board B        │ ◄──────────────► │    Board C        │\n│   Embeddings +    │                  │   Core + KV Cache │                  │   PLE_A + Decoder │\n│   PLE_Proj + Head │                  │                   │                  │   + WiFi          │\n│                   │                  │                   │                  │                   │\n│ • BPE Tokenizer   │                  │ • 6 Attn layers   │                  │ • PLE table A     │\n│ • tok_emb 8-bit   │                  │ • 6 FFN layers    │                  │   (12.5 MB)       │\n│ • ple_model_proj  │                  │ • KV Cache (128)  │                  │ • Sampling        │\n│ • out_norm + head │                  │ • PLE_B (half)    │                  │ • Web Server      │\n│                   │                  │ • PLE Gating      │                  │ • Space heuristic │\n│ Flash: 4.31 MB    │                  │ Flash: 13.71 MB   │                  │ Flash: 13.37 MB   │\n└───────────────────┘                  └───────────────────┘                  └───────────────────┘\nBrowser ──WiFi──► Board C ──ESP-NOW──► Board A ──ESP-NOW──► Board B\n                        ◄─────────────── ◄──────────────\n```\n\n**User** types a prompt in the browser (connected to Board C's WiFi AP) and clicks \"Generate\"**Board C** receives the prompt via HTTP POST → forwards it to Board A via ESP-NOW (`MSG_EMBED_REQUEST`\n\n)**Board A** tokenizes the prompt text → for each token: a. Looks up token embedding`x[D]`\n\n(8-bit tok_emb) b. Computes local PLE projection:`tmpP = ple_model_proj_a @ x`\n\n, RMS-norms it c. Requests PLE table row from**Board C** via ESP-NOW: sends`token_id`\n\n→ C looks up`ple_table_a[token]`\n\n→ sends row back d. Combines:`ple = (tmpP + trow * sqrt(Ph)) / sqrt(2)`\n\n→ sends`token_id + x[D] + ple[L×Ph]`\n\nto**Board B** e.**Board B** computes PLE_B[L×Ph] from its local table → combines PLE_A + PLE_B into full PLE[L×P] → runs 6 transformer layers (attention + FFN + PLE gating) → sends hidden state`x[D]`\n\nback to Board A f.**Board A** applies`out_norm`\n\n→ computes output head:`logits[V] = tok_emb^T · rmsnorm(x)`\n\n→ sends top-40 token IDs to Board C g.**Board C** samples the next token → decodes via BPE vocab → appends to generated output**Board C** streams the generated text to the browser via SSE (`/stream`\n\nendpoint)**User** sees the text appear in real-time on the web page\n\nThe 50.3M-parameter PLE table (vocab × layers × 256) is split into two 128-dimension halves:\n\n**PLE_A**(Board C, 12.5 MB, 4-bit group=64): first 128 dims per layer — reloaded from Board C at runtime via ESP-NOW (MSG_PLE_REQUEST / MSG_PLE_RESULT)**PLE_B**(Board B, 12.6 MB, 4-bit group=128): last 128 dims per layer — local on Board B\n\nBoard A keeps the small `ple_model_proj`\n\n(~50 KB, 4-bit) and `ple_proj_norm`\n\n(~0.5 KB, fp32) for local projection. The 12.5 MB PLE table was moved from Board A to Board C to fix partition overflow on Board A and allow upgrading `tok_emb`\n\nto 8-bit for better inference quality.\n\nEach token in the autoregressive loop previously ran through Board B's transformer with `seq_len=1`\n\n— attention only saw the current token, not the history. This crippled coherence because the model was designed for `seq_len=128`\n\ncontext.\n\nBoard B now maintains a **KV cache** in PSRAM (1.5 MB for 256 positions):\n\n- For each new token, Q is computed normally, while K and V are appended to per-layer caches\n- Attention scores are computed against\n**all cached positions**(not just the current token) - RoPE uses the actual position index instead of\n`pos=0`\n\n— for the first time the transformer sees proper position-aware attention - Cache is reset on\n`MSG_SEQ_START`\n\n(new prompt) and caps at`seq_len=256`\n\n- No changes to A, C, or the wireless protocol\n\nThis is the single largest quality improvement: the transformer finally works as designed, attending to the full generated sequence instead of operating token-by-token in isolation.\n\n| Board | MAC Address | Serial Port |\n|---|---|---|\n| A (Embeddings + PLE_Proj + Head) | `14:c1:9f:2a:ac:c8` |\n`/dev/cu.usbmodem5C372059631` |\n| B (Core + KV Cache) | `14:c1:9f:2c:91:10` |\n`/dev/cu.usbmodem5C372065471` |\n| C (PLE_A + Decoder + WiFi AP) | `28:84:85:51:dc:10` |\n`/dev/cu.usbmodem5C4D0363671` |\n\n**ESP-NOW** wireless peer-to-peer (no router needed)- Packets ≤ 250 bytes with automatic fragmentation\n- Custom protocol with sequence numbers and acknowledgments\n- Estimated latency: 1–5ms per round-trip\n\n```\nesp32s3/\n├── README.md                       # This file (English)\n├── README.es.md                    # Spanish version\n├── pyproject.toml                  # Python dependencies\n│\n├── src/                            # Training pipeline (runs on PC/Mac)\n│   ├── model.py                    # PLE TinyLM architecture (PyTorch)\n│   ├── dataset.py                  # TinyStories/WikiText-103 data loading\n│   ├── train.py                    # Training loop\n│   ├── quantize.py                 # 4-bit group-wise quantization\n│   ├── export.py                   # Export to 3 board binaries\n│   └── gen_assets.py               # Generate vocab.h for firmware\n│\n├── firmware/                       # ESP32-S3 firmware (Arduino IDE)\n│   ├── common/\n│   │   ├── llm.h                   # Distributed C inference runtime\n│   │   └── espnow_protocol.h       # ESP-NOW message protocol\n│   │\n│   ├── board_a_embeddings/         # Board A firmware\n│   │   ├── board_a_embeddings.ino  # Main sketch\n│   │   ├── partitions.csv          # Flash partition table\n│   │   └── vocab.h                 # Generated tokenizer vocabulary\n│   │\n│   ├── board_b_core/               # Board B firmware\n│   │   ├── board_b_core.ino        # Main sketch\n│   │   └── partitions.csv\n│   │\n│   ├── board_c_decoder/            # Board C firmware\n│   │   ├── board_c_decoder.ino     # Main sketch (WiFi + Web UI)\n│   │   ├── partitions.csv\n│   │   └── vocab.h                 # Generated tokenizer vocabulary\n│   │\n│   └── model/                      # Exported model binaries (gitignored)\n│       ├── board_a.bin             # 4.31 MB (tok_emb 8-bit + ple_proj + out_norm)\n│       ├── board_b.bin             # 13.71 MB (transformer layers + PLE_B)\n│       ├── board_c.bin             # 13.37 MB (PLE table A, 4-bit group=64)\n│       ├── golden.npz              # Reference logits for verification\n│       └── golden.txt              # Text-format golden reference\n│\n├── tools/                          # Utility scripts\n│   ├── flash_all.sh                # Flash all 3 boards\n│   ├── verify_models.py            # Verify exported binaries\n│   └── setup_env.sh                # Install Python dependencies\n│\n├── data/                           # Dataset (gitignored)\n│   ├── tinystories/                # Raw TinyStories text\n│   ├── wikitext103/                # Raw WikiText-103 text\n│   ├── tokenizer/                  # Trained BPE tokenizer\n│   ├── train.bin                   # Tokenized training data (112M tokens)\n│   └── val.bin                     # Tokenized validation data\n│\n└── runs/                           # Training checkpoints (gitignored)\n    ├── ple-wiki60m-s0.pt            # 56M model checkpoint\n    ├── ple-wiki60m-s0.json          # Training history\n    └── train.log                   # Training output log\n```\n\n| Parameter | Value | |---|---|---| | Architecture | Tiny decoder-only transformer with Split-PLE | | Total Parameters | 56.0M stored | | Core (dense, SRAM) | 1.5M | | PLE Table (flash, split) | 50.3M (25.2M per board) | | Output Head (tied) | 4.2M | | Vocabulary Size | 32,768 (BPE, WikiText-103) | | d_model | 128 | | n_layers | 6 | | n_heads | 4 | | ffn_hidden | 223 | | ple_dim | 256 (128 per board) | | Quantization | tok_emb: 8-bit group=128, PLE table A: 4-bit group=64, rest: 4-bit group=128 | | Model Binary Size | 31.39 MB total (A: 4.31, B: 13.71, C: 13.37) | | Dataset | WikiText-103 (Wikipedia) | | Training Steps | 12,000 | | Batch Size | 8 | | Sequence Length | 128 |\n\n| Metric | Value |\n|---|---|\n| Final Validation Loss (fp32) | 5.26 |\n| Perplexity (fp32) | 192 |\n| 4-bit Quantization Degradation | +0.62 nats |\n| 4-bit Perplexity | 358 |\n| Training Tokens | 12.3M |\n| Training Time | ~6.9 hours (Mac i7 CPU) |\n\n- Python 3.11+\n[uv](https://docs.astral.sh/uv/)package manager- Arduino IDE with ESP32 board package\n- 3x ESP32-S3 N16R8 boards\n\n```\nsource .venv/bin/activate\npython src/dataset.py --dataset wikitext103\npython src/train.py --arm ple --d-model 128 --n-layers 6 --ple-dim 256 \\\n  --target-core 1500000 --batch-size 8 --seq-len 128 --steps 12000 --tag wiki60m\npython src/quantize.py --tag wiki60m\npython src/export.py ple-wiki60m-s0\npython src/gen_assets.py\n# Connect each board one at a time\n./tools/flash_all.sh /dev/cu.usbmodemXXXX\n```\n\nNote:All 3 boards are already flashed. If you need to reflash, see`tools/flash_all.sh`\n\n.\n\n-\nPower on all three boards\n\n-\nConnect your phone/laptop to WiFi:\n\n**ESP32-DIST-AI**(password:`ai123456`\n\n) -\nOpen\n\n`http://192.168.4.1`\n\nin your browser -\nType a prompt and click \"Generate\"\n\n## Architecture — PLE TinyLM v2, Split-PLE, 56M params\n\n- PLE TinyLM v2 with d_model=128, n_layers=6, ple_dim=256\n- Split-PLE: PLE table divided across Board A + Board C (128+128 per layer)\n- PLE table A moved from Board A to Board C (fixed partition overflow on A)\n- Per-token A↔C PLE request/response via ESP-NOW\n- KV cache on Board B: full multi-head causal attention with K/V caching in PSRAM (1.5 MB for 256 positions), RoPE uses actual position indices\n- MAX_SEQ_LEN=256, gen loop=128, supporting 30+ word sequences\n- srand(esp_random()) for non-deterministic sampling\n\n## Training — WikiText-103, PPL 192, 12K steps, 12.3M tokens\n\n- WikiText-103 dataset download and BPE tokenization (112M tokens)\n- Model training: 12,000 steps, batch_size=8, seg_len=128, 12.3M tokens\n- Final fp32 loss: 5.26, Perplexity: 192\n- Training time: ~6.9 hours (Mac i7 CPU)\n\n## Quantization & Export — 4-bit, 31.39 MB total\n\n- 4-bit group-wise quantization (+0.62 nats degradation from fp32, 4-bit PPL 358)\n- tok_emb: 8-bit group=128, PLE table A: 4-bit group=64, rest: 4-bit group=128\n- Export to 3 board binaries: A: 4.31 MB, B: 13.71 MB, C: 13.37 MB\n- Reference golden logits for verification\n\n## Firmware — 3 boards, ESP-NOW, Web UI, autoregressive loop\n\n- Board A: tokenizer + 8-bit tok_emb + PLE projection + output head\n- Board B: 6 transformer layers + PLE_B + PLE gating + KV cache\n- Board C: PLE table A (4-bit group=64) + decoder + WiFi AP + Web UI\n- ESP-NOW protocol with fragmentation, sequence numbers, and acks\n- Web UI for prompt input, streaming output, and SSE endpoint\n- Autoregressive generation loop (up to 128 tokens) across all 3 boards\n- Space-insertion heuristic on Board C\n- Arduino core 3.3.11 compatibility, partition table 1.25 MB\n\n## Flash & Deploy — 3 boards flashed with firmware + model\n\n- Board A: MAC\n`14:c1:9f:2a:ac:c8`\n\n/ port`5C372059631`\n\n- Board B: MAC\n`14:c1:9f:2c:91:10`\n\n/ port`5C372065471`\n\n- Board C: MAC\n`28:84:85:51:dc:10`\n\n/ port`5C4D0363671`\n\n- Flash and verification scripts\n\n## Testing — Local inference confirmed coherent output (~30 words)\n\n- Local inference test confirmed ~30 coherent words\n- Confirmed 4-bit quantization is the quality bottleneck (not protocol)\n- Comparison between full fp32, quantized, and on-device inference\n\n-\n**BPE tokenizer in C**— Replace brute-force vocab search with proper BPE merge implementation -\n**Static ESP-NOW MACs**— Configure known MACs instead of broadcast -\n**Web UI improvements**— Streaming token display, temperature/top-k controls, status indicators -\n**Error handling**— ESP-NOW retransmission and timeout on packet loss -\n**Power optimization**— Deep sleep between inference requests -\n**Audio output**— I2S speaker integration (future)\n\n| Component | Quantity | Notes |\n|---|---|---|\n| ESP32-S3 N16R8 | 3 | 512KB SRAM, 8MB PSRAM, 16MB flash |\n| USB-C cables | 3 | For flashing firmware |\n| Computer | 1 | Mac/Linux/Windows with Arduino IDE |\n\nThis project extends the work of:\n\n— Running 28.9M parameter LLM on a single ESP32-S3 using Per-Layer Embeddings[slvDev/esp32-ai](https://github.com/slvDev/esp32-ai)**Google Gemma**— Per-Layer Embeddings architecture** WikiText-103**— Dataset by Salesforce Research ([arXiv:1609.07843](https://arxiv.org/abs/1609.07843))** TinyStories**— Dataset by Eldan & Li (Microsoft Research,[arXiv:2305.07759](https://arxiv.org/abs/2305.07759))— Inspiration for running tiny LMs in plain C[karpathy/llama2.c](https://github.com/karpathy/llama2.c)\n\nMIT", "url": "https://wpnews.pro/news/hn-i-ran-a-56m-parameter-llm-across-three-esp32-s3-boards-via-esp-now", "canonical_source": "https://github.com/wladimiravila/esp32s3-distributed-ai", "published_at": "2026-07-30 15:07:32+00:00", "updated_at": "2026-07-30 15:22:38.976750+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["ESP32-S3", "ESP-NOW", "WikiText-103", "Gemma", "TinyStories", "slvDev/esp32-ai"], "alternates": {"html": "https://wpnews.pro/news/hn-i-ran-a-56m-parameter-llm-across-three-esp32-s3-boards-via-esp-now", "markdown": "https://wpnews.pro/news/hn-i-ran-a-56m-parameter-llm-across-three-esp32-s3-boards-via-esp-now.md", "text": "https://wpnews.pro/news/hn-i-ran-a-56m-parameter-llm-across-three-esp32-s3-boards-via-esp-now.txt", "jsonld": "https://wpnews.pro/news/hn-i-ran-a-56m-parameter-llm-across-three-esp32-s3-boards-via-esp-now.jsonld"}}