Developer slvDev_ has run a 28.9-million-parameter TinyStories language model entirely on an Espressif ESP32-S3 microcontroller — an $8 chip with 512KB SRAM — generating text at 9.5 tokens per second with no cloud connection. Using Google's Per-Layer Embeddings scheme, the model keeps 25M parameters in flash and only ~560K in fast RAM. The project is fully open source on GitHub and trending on Hacker News. It challenges the assumption that useful language inference requires cloud infrastructure for IoT and embedded devices.
28.9 million parameters, 9.5 tokens per second, zero cloud. The ESP32-S3 demo nobody saw coming. #
A developer known as slvDev_ has run a 28.9-million-parameter language model entirely on an Espressif ESP32-S3 microcontroller — an approximately $8 chip with 512KB of SRAM and 8MB of PSRAM — generating coherent text at roughly 9.5 tokens per second with no server, no API, and no internet connection. The project, released as open source on GitHub under the repository "esp32-ai," has been trending on Hacker News since July 25.
This is not a Jetson Nano running a quantized Llama. This is a microcontroller that costs less than a sandwich running a transformer model and generating sentences. The implications for edge AI, IoT, and the cloud inference business model are significant.
How It Works: Per-Layer Embeddings #
The key technique is Google's Per-Layer Embeddings scheme, a memory management strategy that keeps the 25-million-parameter embedding table in flash memory accessed via memory-mapped execute-in-place (XIP), while only the approximately 560K-parameter dense core needs fast RAM. Per token generated, only about six embedding rows — roughly 450 bytes — are read from flash.
The model uses the TinyStories architecture, which is designed from the ground up for extremely constrained environments. TinyStories was originally a research project to study how small language models acquire linguistic competency. slvDev_ demonstrated it is small enough to run on hardware that was never designed for inference workloads.
The ESP32-S3 hardware is absurdly modest by AI standards: a dual-core Xtensa LX7 processor at 240 MHz, half a megabyte of on-chip SRAM, and 8MB of external PSRAM. For comparison, even a quantized Llama-3-8B requires roughly 5GB of RAM — more than 600 times the PSRAM available on the ESP32.
The firmware and training code are fully open source. Any developer with a $10 ESP32-S3 development board, a USB cable, and the GitHub repository can flash the firmware and run an LLM locally. No registration. No API key. No monthly subscription.
What It Means for Edge AI #
The dominant model for AI inference in IoT and embedded devices is cloud-dependent: the device captures data, sends it to a cloud endpoint, receives a response from a large model, and acts on it. This model assumes local hardware is too weak to run useful inference. It also creates latency, bandwidth, privacy, and cost constraints.
The ESP32-32 demo breaks that assumption at its cheapest point. If a 28.9M-parameter model can run on an $8 chip, what can run on a $30 chip? A $50 chip? As microcontrollers add more RAM and flash, the performance curve accelerates without requiring a jump to application processors.
The use cases that become practical:
Voice interfaces that work offline. An ESP32 with a small LLM can handle natural language commands for smart home devices without sending audio to a cloud service. Privacy-sensitive applications — medical devices, industrial sensors, security systems — where data must stay local. Low-connectivity environments — agriculture, maritime, remote infrastructure — where cloud inference is impossible because there is no connection.
The Economics of Cloud vs. Edge Inference #
Cloud inference has a per-token cost. Even cheap models — GPT-4o-mini, Claude Haiku, Gemini Flash — cost fractions of a cent per thousand tokens. That cost scales with usage. An IoT device generating thousands of tokens per day, multiplied across millions of devices, produces a recurring revenue stream for cloud providers.
Edge inference on an ESP32 has zero per-token cost after the initial hardware purchase. The electricity to run a 240 MHz microcontroller for inference is negligible — fractions of a watt. The model does not get software updates, improvements, or access to tools and retrieval, but for a class of applications where the task is simple and static — "parse this voice command," "summarize this sensor reading," "generate a status message" — edge inference is economically superior.
This mirrors what happened with edge image recognition. Five years ago, running object detection on a microcontroller was novel. Today, TensorFlow Lite Micro ships in millions of devices. The pattern repeats for language: novel demonstration, open-source release, optimization, then commodity deployment.
The Catch: This Is Not GPT at Home #
The 28.9M-parameter TinyStories model is not a general-purpose reasoning engine. It generates simple, child-like text. It cannot write code, answer complex questions, or follow multi-step instructions. It is a proof of concept that language generation is possible on microcontroller-class hardware — not a replacement for cloud inference.
Scaling from 28.9M to a useful edge model is the next challenge. A 100M-parameter model that fits in 64MB of flash could produce substantially better output while still running on affordable embedded hardware. The Per-Layer Embeddings technique scales; the question is whether the output quality curve justifies the hardware cost curve.
The other limitation is power. The ESP32-S3 draws roughly 200-300 milliwatts during inference. That is low enough for battery-powered devices, but continuous generation would drain a CR2032 coin cell in minutes. Edge LLMs are practical for intermittent inference — wake, generate, sleep — not continuous streaming.
The Open-Source Pattern Starts Again #
slvDev's project follows a now-familiar pattern in AI: a developer demonstrates something "impossible" on commodity hardware, releases the code, and the community iterates until the impossible becomes routine. Stable Diffusion on a laptop. Llama on a phone. Whisper on a Raspberry Pi. And now, an LLM on an $8 microcontroller.
Each iteration chips away at the assumption that AI requires massive cloud infrastructure. The ESP32-S3 demo chips away at the cheapest end of the hardware spectrum. The next developer will take the same technique, a slightly better chip, and a slightly larger model — and the floor for what "possible" looks like drops again.
The cloud AI business model depends on the assumption that inference requires cloud infrastructure. Every edge demonstration that works challenges that assumption. The ESP32-S3 demo is a small crack. Enough small cracks add up.
Get AI news in your inbox
Daily digest of what matters in AI.
Key Terms Explained #
Claude Anthropic's family of AI assistants, including Claude Haiku, Sonnet, and Opus.
Edge AI Running AI models directly on local devices (phones, laptops, IoT devices) instead of in the cloud.
Embedding A dense numerical representation of data (words, images, etc.
Gemini Google's flagship multimodal AI model family, developed by Google DeepMind.