{"slug": "running-a-28-9m-parameter-llm-on-an-8-microcontroller", "title": "Running a 28.9M parameter LLM on an $8 microcontroller", "summary": "A developer known as slvDev has run a 28.9 million parameter language model on an ESP32-S3 microcontroller that costs about $8, generating text at roughly 9.5 tokens per second entirely on-device. The model uses Google's Per-Layer Embeddings technique from Gemma 3n and Gemma 4 to store most parameters in flash memory, overcoming the chip's 512KB SRAM limit that previously restricted such models to 260 thousand parameters. The model was trained on the TinyStories dataset and writes short, coherent stories but cannot answer questions or follow instructions.", "body_md": "Open to Work ·\n[𝕏 slvDev](https://x.com/slvDev) ·\n[LinkedIn](https://www.linkedin.com/in/slvdev/)\n\nThis is a 28.9 million parameter language model that generates text on an ESP32-S3, a microcontroller that costs about $8. It runs on the chip itself, with nothing sent to a server, and it writes each word to a small screen wired to the chip at roughly 9 tokens per second. The last language model people ran on a chip like this had 260 thousand parameters, so this one holds about a hundred times more. It fits because most of the model lives in flash instead of RAM, using an idea from Google's Gemma models called Per-Layer Embeddings.\n\n| Parameters | 28.9M stored (25M of them in a flash lookup table) |\n| Chip | ESP32-S3, about $8, with 512KB SRAM, 8MB PSRAM and 16MB flash |\n| Speed | about 9.5 tok/s end to end (9.7 tok/s of pure compute) |\n| Connectivity | none, everything runs on the device |\n| Model size | 14.9MB at 4-bit |\n\nA microcontroller has very little fast memory. The ESP32-S3 gives you 512KB of SRAM. Normally the whole model has to be reachable from there, which keeps you stuck with tiny models, and that is why the previous model on a chip like this had only 260 thousand parameters.\n\nThe way around it is to stop putting the model in fast memory at all. Most of a language model's parameters sit in an embedding table, which the model reads from rather than computes on. So you can leave that 25 million row table in slow flash and pull only the few rows each token needs, about 450 bytes, while the small part that does the actual work stays in fast memory. The large model then costs almost nothing to run, because you never load most of it. It just sits in flash and gets sampled a little at a time.\n\nThat idea is Google's Per-Layer Embeddings, from Gemma 3n and Gemma 4. Here it runs on the memory layout of a microcontroller instead of a phone or a GPU. As far as I can tell, nobody had tried it on a chip this small.\n\n```\n  SRAM  (fast, tiny)   the \"thinking\" core, used on every token\n  PSRAM (medium)       the output head and working memory\n  FLASH (huge, slow)   the 25M-param table, about 6 rows read per token (~450 B)\n```\n\nThe model was trained on TinyStories, so it writes short, simple stories and mostly keeps them coherent. It will not answer questions, follow instructions, write code, or know facts. That limit comes from the small part of the model that does the reasoning, and the memory trick does not change it. What is interesting here is the architecture, fitting a large model onto a tiny chip, rather than what a 28.9 million parameter model can say.\n\nThe firmware, the wiring, and the flashing steps live in\n[ firmware/esp32_llm/README.md](/slvDev/esp32-ai/blob/main/firmware/esp32_llm/README.md). The training,\nablation, and quantization code is in\n\n`src/`\n\nand `experiments/`\n\n. The full method,\nthe ablations, and the on-chip measurements are written up in\n[.](/slvDev/esp32-ai/blob/main/RESULTS.md)\n\n`RESULTS.md`\n\nTinyStories is the dataset this trains on: short synthetic stories simple enough\nthat a small model can still learn to write coherently (Ronen Eldan and Yuanzhi Li,\nMicrosoft Research, [arXiv:2305.07759](https://arxiv.org/abs/2305.07759)). The other\nhalf is Per-Layer Embeddings, Google's design from the Gemma models, which is what\nlets a big model fit on a small chip.\n\nAndrej Karpathy's [llama2.c](https://github.com/karpathy/llama2.c) is why a lot of\npeople, me included, believe you can train a tiny language model and run it in plain\nC at all. This grew out of that.\n\nI left the messy history in the repo on purpose. That includes a bug I found in my\nown parameter accounting, which had inflated an early number, and the corrected\nresult that followed once I fixed it. The commit history and `RESULTS.md`\n\nshow where\nthe numbers moved and why.", "url": "https://wpnews.pro/news/running-a-28-9m-parameter-llm-on-an-8-microcontroller", "canonical_source": "https://github.com/slvDev/esp32-ai", "published_at": "2026-07-25 18:59:50+00:00", "updated_at": "2026-07-25 19:22:15.829644+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["slvDev", "ESP32-S3", "Google", "Gemma 3n", "Gemma 4", "TinyStories", "Microsoft Research", "Andrej Karpathy"], "alternates": {"html": "https://wpnews.pro/news/running-a-28-9m-parameter-llm-on-an-8-microcontroller", "markdown": "https://wpnews.pro/news/running-a-28-9m-parameter-llm-on-an-8-microcontroller.md", "text": "https://wpnews.pro/news/running-a-28-9m-parameter-llm-on-an-8-microcontroller.txt", "jsonld": "https://wpnews.pro/news/running-a-28-9m-parameter-llm-on-an-8-microcontroller.jsonld"}}