# An LLM on an $8 Chip Is Mostly a Memory Trick > Source: > Published: 2026-07-26 01:08:35+00:00 [AI](https://sourcefeed.dev/c/ai)Article # An LLM on an $8 Chip Is Mostly a Memory Trick The ESP32-S3 demo matters less for what it writes than for how it splits hot and cold parameters. [Priya Nair](https://sourcefeed.dev/u/priya_nair) A developer going by slvDev just got a 28.9-million-parameter language model generating text on an [ESP32-S3](https://www.espressif.com/en/products/socs/esp32-s3) — a microcontroller you can buy on a dev board for about $8, with 512KB of SRAM, no MMU, and no operating system. It runs at roughly 9.5 tokens per second, fully offline, writing short stories to a little display. The model is trained on TinyStories, so what it produces is charming and useless: it can't answer questions, follow instructions, or state a fact. And yet this is one of the more instructive embedded-AI demos in a while — not because of what the model says, but because of how it fits. ## The trick: most parameters are cold The headline number is misleading in the best way. Of those 28.9M parameters, about 25M are the embedding tables. Embeddings aren't compute — they're a lookup. Each generated token touches a handful of rows, roughly 450 bytes of data. The dense transformer core that actually does the work is only about 3.9M parameters. So the [esp32-ai](https://github.com/slvDev/esp32-ai) project maps the model onto the ESP32-S3's memory hierarchy the way an embedded engineer would map any hot/cold dataset: a reasoning core of roughly 560K parameters lives in fast SRAM, working buffers and the output head go to 8MB of PSRAM, and the 25M-parameter embedding table sits in 16MB of flash, memory-mapped via XIP so rows are read on demand. Quantized to 4 bits, the whole model is 14.9MB. Flash reads cost latency but no wear — endurance only matters for writes — and the end-to-end penalty is tiny: 9.5 tok/s versus 9.7 tok/s for pure compute. That's the real insight, and it isn't a hack invented for a demo. It's the same idea behind Per-Layer Embeddings in Google's [Gemma 3n](https://ai.google.dev/gemma/docs/gemma-3n), which is exactly why a "5B" Gemma variant runs in the memory footprint of a 2B model on phones. Apple's "LLM in a flash" paper made the general case back in 2023: parameter count and DRAM requirement are different quantities, and the gap between them is an engineering opportunity. What slvDev did is carry that logic all the way down to a chip with half a megabyte of fast RAM — about a hundred times more parameters than the previous MCU-class record, a 260K TinyStories model in the [llama2.c](https://github.com/karpathy/llama2.c) lineage that Dave Bennett ran on the same chip in 2024. ## Ignore the viral math The version of this story going around social media compares 28.9M parameters to "the first ChatGPT's 117M" and concludes the ESP32 has a quarter of its smarts. Both halves of that are wrong. 117M was GPT-1, a 2018 research model; ChatGPT launched on GPT-3.5, a 175-billion-parameter-class system. And parameters don't divide into "smarts" anyway — least of all here, where 86% of them are a lookup table. The honest capability statement is the one in the repo itself: a dense core of under 4M parameters writes simple, mostly-coherent children's stories and nothing else. Nobody is replacing a cloud API with this. Framings like "this breaks the cloud model for IoT" are hype; the demo is a proof of memory architecture, not a product. ## What embedded developers should actually take from it The lesson worth internalizing: **when sizing a model for constrained hardware, count hot parameters, not total parameters.** Your RAM budget constrains the dense core and activations; flash constrains everything else, and flash is cheap. A 16MB flash part costs almost nothing. That reframing changes what's plausible on hardware most people had written off for generative workloads. Where it plausibly pays off near-term: **Offline text-to-speech.** Commenters on Hacker News pointed at 20–30M-parameter TTS models — the same weight class, and a genuinely useful capability for a device with no network. TinyML on ESP32-class chips has meant keyword spotting and small CNNs under TensorFlow Lite Micro; on-device generation is a new category.**Narrow-domain generation.** TinyStories (the Eldan and Li paper behind this model) showed tiny transformers get coherent when the domain is radically constrained. The same holds for structured status messages, canned-but-varied NPC dialogue in toys, or template-free log summaries — anywhere the output space is small and a round-trip to a server is a dealbreaker.**Privacy- and cost-hard deployments.** No API key, no per-token bill, no data leaving the device. For a fleet of thousands of $8 sensors, that arithmetic is the entire business case. Adoption is refreshingly concrete: it's plain C on [ESP-IDF](https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/), descended from llama2.c, on commodity hardware. If you've flashed an ESP32 before, you can reproduce it this weekend. The trade-offs are equally concrete: 9.5 tok/s is around human reading speed, fine for a display and marginal for interactive use; 4-bit quantization of an already-tiny model leaves no quality headroom; and there's no fine-tuning on device — you train on a real GPU and ship weights. The boundary line matters too. If your product needs instruction following or open-ended chat, the floor is still a Raspberry Pi-class board running llama.cpp with a 0.5B–1B model — a different power budget and a different BOM. The ESP32 tier is for generation-as-a-peripheral, not assistants. ## Where this goes My read: this demo is a preview of a default, not an outlier. The hot/cold parameter split is already shipping at the phone tier via Gemma 3n's PLE, and there's no reason edge inference runtimes — TFLite Micro, Espressif's own NN tooling, the growing pile of MCU llama2.c forks — won't absorb flash-resident embeddings as a standard technique. Model architects are meeting them halfway by designing for it: making more of the parameter budget cold on purpose. The story a $8 chip tells about a dragon and a little girl is a toy. The memory map underneath it is production thinking, and it's the part worth stealing. ## Sources & further reading - [esp32-ai: Running a 28.9M parameter LLM on an $8 microcontroller](https://github.com/slvDev/esp32-ai)— github.com - [Running a 28.9M parameter LLM on an $8 microcontroller](https://news.ycombinator.com/item?id=49050512)— news.ycombinator.com - [Someone squeezed a 28.9M LLM onto an ESP32-S3, and so can you](https://www.xda-developers.com/someone-squeezed-a-289m-llm-onto-an-esp32-s3-and-so-can-you/)— xda-developers.com - [esp32-llm: Running a LLM on the ESP32](https://github.com/DaveBben/esp32-llm)— github.com - [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://arxiv.org/abs/2305.07759)— arxiv.org [Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. ## Discussion 0 No comments yet Be the first to weigh in.