28.9M-parameter LLM runs locally on ESP32-S3 at 9 tokens/s

Slava S. (slvDev) has optimized a 28.9M-parameter LLM to run locally on an ESP32-S3 development board at about 9 tokens/s, generating short stories on an I2C display. The project, called 'esp32-ai', requires an ESP32-S3 with at least 512KB SRAM+8MB SPRAM and 16MB flash to store the 14.9MB 4-bit model file, and was tested on an ~$8 board. This differs from previous ESP32-based LLM projects that used the board as a gateway to online LLMs.

Slava S. slvDev has optimized a 28.9M-parameter LLM running locally on an ESP32-S3 development board at around 9 tokens/s while generating text, or more exactly, telling short stories. We’ve written about LLMs and ESP32 in the past, but usually the boards are used as low-power gateways to interact with online LLMs hosted on powerful servers in data centers. Such products or projects include HiWonderLLM “smart chat module”, the ESP-Claw framework, or the ESP32 Agent Dev Kit. Slava’s “esp32-ai” project is quite different, as everything runs on the ESP32-S3 microcontroller. This won’t work on all ESP32-S3 as you’ll need enough memory and storage to store the 14.9MB 4-bit model file, and Slava tested his demo on an ~$8 ESP32-S3 board with 512KB SRAM+8MB SPRAM and 16MB flash connected to a small I2C display to display short stories. Dave Bennett already did something similar with the ESP32-LLM project, but it only relied ... The post 28.9M-parameter LLM runs locally on ESP32-S3 at 9 tokens/s https://www.cnx-software.com/2026/08/03/28-9m-parameter-llm-runs-locally-on-esp32-s3-at-9-tokens-s/ appeared first on CNX Software - Embedded Systems News https://www.cnx-software.com .