An LLM on an $8 Chip Is Mostly a Memory Trick

A developer known as slvDev has run a 28.9-million-parameter language model on an ESP32-S3 microcontroller, an $8 chip with 512KB of SRAM, achieving 9.5 tokens per second for offline text generation. The trick is that 25 million of those parameters are embedding tables stored in flash memory, leaving only a 3.9-million-parameter dense core in fast RAM, demonstrating that parameter count and DRAM requirement are different quantities. The project, called esp32-ai, maps the model onto the chip's memory hierarchy using hot/cold data separation, a technique also seen in Google's Gemma 3n and Apple's 2023 'LLM in a flash' paper.

AI https://sourcefeed.dev/c/ai Article An LLM on an $8 Chip Is Mostly a Memory Trick The ESP32-S3 demo matters less for what it writes than for how it splits hot and cold parameters. Priya Nair https://sourcefeed.dev/u/priya nair A developer going by slvDev just got a 28.9-million-parameter language model generating text on an ESP32-S3 https://www.espressif.com/en/products/socs/esp32-s3 — a microcontroller you can buy on a dev board for about $8, with 512KB of SRAM, no MMU, and no operating system. It runs at roughly 9.5 tokens per second, fully offline, writing short stories to a little display. The model is trained on TinyStories, so what it produces is charming and useless: it can't answer questions, follow instructions, or state a fact. And yet this is one of the more instructive embedded-AI demos in a while — not because of what the model says, but because of how it fits. The trick: most parameters are cold The headline number is misleading in the best way. Of those 28.9M parameters, about 25M are the embedding tables. Embeddings aren't compute — they're a lookup. Each generated token touches a handful of rows, roughly 450 bytes of data. The dense transformer core that actually does the work is only about 3.9M parameters. So the esp32-ai https://github.com/slvDev/esp32-ai project maps the model onto the ESP32-S3's memory hierarchy the way an embedded engineer would map any hot/cold dataset: a reasoning core of roughly 560K parameters lives in fast SRAM, working buffers and the output head go to 8MB of PSRAM, and the 25M-parameter embedding table sits in 16MB of flash, memory-mapped via XIP so rows are read on demand. Quantized to 4 bits, the whole model is 14.9MB. Flash reads cost latency but no wear — endurance only matters for writes — and the end-to-end penalty is tiny: 9.5 tok/s versus 9.7 tok/s for pure compute. That's the real insight, and it isn't a hack invented for a demo. It's the same idea behind Per-Layer Embeddings in Google's Gemma 3n https://ai.google.dev/gemma/docs/gemma-3n , which is exactly why a "5B" Gemma variant runs in the memory footprint of a 2B model on phones. Apple's "LLM in a flash" paper made the general case back in 2023: parameter count and DRAM requirement are different quantities, and the gap between them is an engineering opportunity. What slvDev did is carry that logic all the way down to a chip with half a megabyte of fast RAM — about a hundred times more parameters than the previous MCU-class record, a 260K TinyStories model in the llama2.c https://github.com/karpathy/llama2.c lineage that Dave Bennett ran on the same chip in 2024. Ignore the viral math The version of this story going around social media compares 28.9M parameters to "the first ChatGPT's 117M" and concludes the ESP32 has a quarter of its smarts. Both halves of that are wrong. 117M was GPT-1, a 2018 research model; ChatGPT launched on GPT-3.5, a 175-billion-parameter-class system. And parameters don't divide into "smarts" anyway — least of all here, where 86% of them are a lookup table. The honest capability statement is the one in the repo itself: a dense core of under 4M parameters writes simple, mostly-coherent children's stories and nothing else. Nobody is replacing a cloud API with this. Framings like "this breaks the cloud model for IoT" are hype; the demo is a proof of memory architecture, not a product. What embedded developers should actually take from it The lesson worth internalizing: when sizing a model for constrained hardware, count hot parameters, not total parameters. Your RAM budget constrains the dense core and activations; flash constrains everything else, and flash is cheap. A 16MB flash part costs almost nothing. That reframing changes what's plausible on hardware most people had written off for generative workloads. Where it plausibly pays off near-term: Offline text-to-speech. Commenters on Hacker News pointed at 20–30M-parameter TTS models — the same weight class, and a genuinely useful capability for a device with no network. TinyML on ESP32-class chips has meant keyword spotting and small CNNs under TensorFlow Lite Micro; on-device generation is a new category. Narrow-domain generation. TinyStories the Eldan and Li paper behind this model showed tiny transformers get coherent when the domain is radically constrained. The same holds for structured status messages, canned-but-varied NPC dialogue in toys, or template-free log summaries — anywhere the output space is small and a round-trip to a server is a dealbreaker. Privacy- and cost-hard deployments. No API key, no per-token bill, no data leaving the device. For a fleet of thousands of $8 sensors, that arithmetic is the entire business case. Adoption is refreshingly concrete: it's plain C on ESP-IDF https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/ , descended from llama2.c, on commodity hardware. If you've flashed an ESP32 before, you can reproduce it this weekend. The trade-offs are equally concrete: 9.5 tok/s is around human reading speed, fine for a display and marginal for interactive use; 4-bit quantization of an already-tiny model leaves no quality headroom; and there's no fine-tuning on device — you train on a real GPU and ship weights. The boundary line matters too. If your product needs instruction following or open-ended chat, the floor is still a Raspberry Pi-class board running llama.cpp with a 0.5B–1B model — a different power budget and a different BOM. The ESP32 tier is for generation-as-a-peripheral, not assistants. Where this goes My read: this demo is a preview of a default, not an outlier. The hot/cold parameter split is already shipping at the phone tier via Gemma 3n's PLE, and there's no reason edge inference runtimes — TFLite Micro, Espressif's own NN tooling, the growing pile of MCU llama2.c forks — won't absorb flash-resident embeddings as a standard technique. Model architects are meeting them halfway by designing for it: making more of the parameter budget cold on purpose. The story a $8 chip tells about a dragon and a little girl is a toy. The memory map underneath it is production thinking, and it's the part worth stealing. Sources & further reading - esp32-ai: Running a 28.9M parameter LLM on an $8 microcontroller https://github.com/slvDev/esp32-ai — github.com - Running a 28.9M parameter LLM on an $8 microcontroller https://news.ycombinator.com/item?id=49050512 — news.ycombinator.com - Someone squeezed a 28.9M LLM onto an ESP32-S3, and so can you https://www.xda-developers.com/someone-squeezed-a-289m-llm-onto-an-esp32-s3-and-so-can-you/ — xda-developers.com - esp32-llm: Running a LLM on the ESP32 https://github.com/DaveBben/esp32-llm — github.com - TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759 — arxiv.org Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.