AIArticle The ESP32-S3 demo matters less for what it writes than for how it splits hot and cold parameters.
Priya Nair A developer going by slvDev just got a 28.9-million-parameter language model generating text on an ESP32-S3 — a microcontroller you can buy on a dev board for about $8, with 512KB of SRAM, no MMU, and no operating system. It runs at roughly 9.5 tokens per second, fully offline, writing short stories to a little display. The model is trained on TinyStories, so what it produces is charming and useless: it can't answer questions, follow instructions, or state a fact.
And yet this is one of the more instructive embedded-AI demos in a while — not because of what the model says, but because of how it fits.
The trick: most parameters are cold #
The headline number is misleading in the best way. Of those 28.9M parameters, about 25M are the embedding tables. Embeddings aren't compute — they're a lookup. Each generated token touches a handful of rows, roughly 450 bytes of data. The dense transformer core that actually does the work is only about 3.9M parameters.
So the esp32-ai project maps the model onto the ESP32-S3's memory hierarchy the way an embedded engineer would map any hot/cold dataset: a reasoning core of roughly 560K parameters lives in fast SRAM, working buffers and the output head go to 8MB of PSRAM, and the 25M-parameter embedding table sits in 16MB of flash, memory-mapped via XIP so rows are read on demand. Quantized to 4 bits, the whole model is 14.9MB. Flash reads cost latency but no wear — endurance only matters for writes — and the end-to-end penalty is tiny: 9.5 tok/s versus 9.7 tok/s for pure compute.
That's the real insight, and it isn't a hack invented for a demo. It's the same idea behind Per-Layer Embeddings in Google's Gemma 3n, which is exactly why a "5B" Gemma variant runs in the memory footprint of a 2B model on phones. Apple's "LLM in a flash" paper made the general case back in 2023: parameter count and DRAM requirement are different quantities, and the gap between them is an engineering opportunity. What slvDev did is carry that logic all the way down to a chip with half a megabyte of fast RAM — about a hundred times more parameters than the previous MCU-class record, a 260K TinyStories model in the llama2.c lineage that Dave Bennett ran on the same chip in 2024.
Ignore the viral math #
The version of this story going around social media compares 28.9M parameters to "the first ChatGPT's 117M" and concludes the ESP32 has a quarter of its smarts. Both halves of that are wrong. 117M was GPT-1, a 2018 research model; ChatGPT launched on GPT-3.5, a 175-billion-parameter-class system. And parameters don't divide into "smarts" anyway — least of all here, where 86% of them are a lookup table.
The honest capability statement is the one in the repo itself: a dense core of under 4M parameters writes simple, mostly-coherent children's stories and nothing else. Nobody is replacing a cloud API with this. Framings like "this breaks the cloud model for IoT" are hype; the demo is a proof of memory architecture, not a product.
What embedded developers should actually take from it #
The lesson worth internalizing: when sizing a model for constrained hardware, count hot parameters, not total parameters. Your RAM budget constrains the dense core and activations; flash constrains everything else, and flash is cheap. A 16MB flash part costs almost nothing. That reframing changes what's plausible on hardware most people had written off for generative workloads.
Where it plausibly pays off near-term: Offline text-to-speech. Commenters on Hacker News pointed at 20–30M-parameter TTS models — the same weight class, and a genuinely useful capability for a device with no network. TinyML on ESP32-class chips has meant keyword spotting and small CNNs under TensorFlow Lite Micro; on-device generation is a new category.Narrow-domain generation. TinyStories (the Eldan and Li paper behind this model) showed tiny transformers get coherent when the domain is radically constrained. The same holds for structured status messages, canned-but-varied NPC dialogue in toys, or template-free log summaries — anywhere the output space is small and a round-trip to a server is a dealbreaker.Privacy- and cost-hard deployments. No API key, no per-token bill, no data leaving the device. For a fleet of thousands of $8 sensors, that arithmetic is the entire business case.
Adoption is refreshingly concrete: it's plain C on ESP-IDF, descended from llama2.c, on commodity hardware. If you've flashed an ESP32 before, you can reproduce it this weekend. The trade-offs are equally concrete: 9.5 tok/s is around human reading speed, fine for a display and marginal for interactive use; 4-bit quantization of an already-tiny model leaves no quality headroom; and there's no fine-tuning on device — you train on a real GPU and ship weights.
The boundary line matters too. If your product needs instruction following or open-ended chat, the floor is still a Raspberry Pi-class board running llama.cpp with a 0.5B–1B model — a different power budget and a different BOM. The ESP32 tier is for generation-as-a-peripheral, not assistants.
Where this goes #
My read: this demo is a preview of a default, not an outlier. The hot/cold parameter split is already shipping at the phone tier via Gemma 3n's PLE, and there's no reason edge inference runtimes — TFLite Micro, Espressif's own NN tooling, the growing pile of MCU llama2.c forks — won't absorb flash-resident embeddings as a standard technique. Model architects are meeting them halfway by designing for it: making more of the parameter budget cold on purpose.
The story a $8 chip tells about a dragon and a little girl is a toy. The memory map underneath it is production thinking, and it's the part worth stealing.
Sources & further reading #
[esp32-ai: Running a 28.9M parameter LLM on an $8 microcontroller](https://github.com/slvDev/esp32-ai)— github.com -
[Running a 28.9M parameter LLM on an $8 microcontroller](https://news.ycombinator.com/item?id=49050512)— news.ycombinator.com -
[Someone squeezed a 28.9M LLM onto an ESP32-S3, and so can you](https://www.xda-developers.com/someone-squeezed-a-289m-llm-onto-an-esp32-s3-and-so-can-you/)— xda-developers.com -
[esp32-llm: Running a LLM on the ESP32](https://github.com/DaveBben/esp32-llm)— github.com -
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?— arxiv.org
Priya Nair· AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.