Wikipedia Offline Reader for ESP32 CYD

Alun Morris and Claude Code released an offline Wikipedia reader for the ESP32-2432S028 (Cheap Yellow Display), a 320×240 ILI9341 touchscreen module, storing articles, images, and a search index on a microSD card with no internet connection. The device requires at least an 8 GB microSD card, with the recommended Simple English Wikipedia ZIM file (wikipedia_en_simple_all_maxi_YYYY-MM.zim) being a ~3.3 GB download containing ~285,000 articles and images, processed to ~10 GB. The project supports two hardware configurations: the CYD board and a bare JC2432S024 display module wired to an ESP32-C3 dev board, with PlatformIO environments 'cyd' and 'c3' respectively.

An offline Wikipedia reader for the ESP32-2432S028 "Cheap Yellow Display" — a 320×240 ILI9341 touchscreen module. Articles, images, and a full search index are stored on a microSD card and read on-device with no internet connection. Written by Alun Morris and Claude Code. https://private-user-images.githubusercontent.com/4630866/604418311-b713e92a-6ba9-4e7e-b10c-57586139830f.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3ODgwNDA0MjMsIm5iZiI6MTc4ODA0MDEyMywicGF0aCI6Ii80NjMwODY2LzYwNDQxODMxMS1iNzEzZTkyYS02YmE5LTRlN2UtYjEwYy01NzU4NjEzOTgzMGYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI2MDgyOSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNjA4MjlUMjE0ODQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTM3OWIxZDMwZDlhMzIyN2FjNjM2OTJjYmE3ODJlMzU5YjA2NmUxYjMwZjQwZWIwNWYyZDM5YWJmMDI2MjljOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmcmVzcG9uc2UtY29udGVudC10eXBlPWltYWdlJTJGcG5nIn0. MmzB rALOVaEQFlj8gN ud-IB2n1N ufdDJWaLx29o | Component | Detail | |---|---| | Board | ESP32-2432S028 CYD | | Display | 320×240 ILI9341 HSPI bus | | Touch | XPT2046 resistive shared HSPI | | SD card | VSPI bus — CS=5, MOSI=23, MISO=19, SCK=18 | | Backlight | GPIO 21 | Use PlatformIO environment cyd pio run -e cyd -t upload . The JC2432S024 is a bare 2.4″ 320×240 ILI9341 display module with XPT2046 resistive touch and an SD card slot. It has no processor onboard — wire it to an ESP32-C3 dev board SuperMini or DevKitM-1 . The ESP32-C3 has a single SPI peripheral, so display, SD card and touch controller all share one SPI bus with separate chip-select lines. | Module pin | ESP32-C3 GPIO | Notes | |---|---|---| | SCK | 4 | Shared by TFT + Touch + SD | | MOSI / SDI / T DIN / SD MOSI | 6 | Shared | | MISO / SDO / T DO / SD MISO | 5 | Shared | | CS TFT | 7 | | | DC / RS | 1 | | | RST | 3V3 | Tie high — not driven by firmware | | SD CS | 10 | | | T CS | 3 | | | T IRQ | 8 | PENIRQ — idles HIGH, safe on boot strapping pin | | LED / BL backlight | 0 | Or tie directly to 3V3 for always-on backlight | | VCC | 3V3 | | | GND | GND | Avoid GPIO 2 and GPIO 9 for external loads — they are ESP32-C3 boot-strapping pins. Use PlatformIO environment c3 pio run -e c3 -t upload . First boot: touch calibration runs automatically. Tap the two red crosshairs when prompted. Calibration is saved to flash and skipped on subsequent boots. A microSD card of at least 8 GB is required Simple English Wikipedia uses ~7 GB on card . The database is built from a Kiwix https://www.kiwix.org/ ZIM file. Download one from Wikimedia dumps https://dumps.wikimedia.org/kiwix/zim/wikipedia/ . Recommended: Simple English Wikipedia — wikipedia en simple all maxi YYYY-MM.zim - ~3.3 GB download, ~285 000 articles - Shorter articles and simpler language — well-suited to a small screen - Includes images maxi variant - processed size is ~10GB Other ZIM files will work but larger editions eg full English, ~90 GB may exceed the SD card size the CYD is known to handle 32GB OK, 64GB may work . Choose the maxi variant includes images . The mini variant omits images and has only the top 50-100k articles. There is a small demo Wiki about knots in the preprocessor folder The preprocessor converts a ZIM file into the binary database format read by the firmware. The ESP32 cannot read a ZIM file directly. Several hard constraints make a purpose-built binary format necessary: - ZIM uses zstd cluster compression. Decompressing a zstd cluster requires holding the entire cluster in RAM. ZIM clusters are typically 1–4 MB, which exceeds the ESP32's ~300 KB of usable heap. The preprocessor re-compresses each article individually with LZ4, which decompresses in a few KB of working memory. - ZIM's index structure is too complex for embedded use. ZIM uses a URL-sorted B-tree-style index with variable-length entries. The binary index produced here is fixed-width 80 bytes/record , sorted by normalised title, and paired with a tiny sparse index one entry per 64 articles that fits entirely in RAM ~7 KB . Together they allow title lookup with zero SD seeks to find the scan start. - Images must be in formats the ESP32 can decode. The firmware decodes JPEG via the hardware-accelerated TJpgDec library and QOI via a lightweight software decoder. ZIM stores images as WebP internally with the original format preserved in the file path , which the ESP32 cannot guarantee to decode in available RAM. The preprocessor converts everything to JPEG photos or QOI diagrams/SVGs . - ZIM HTML needs cleaning. Wikipedia ZIM files inject boilerplate footers, navigation chrome, and complex class structures into every article. The preprocessor strips these with BeautifulSoup so the firmware's minimal HTML renderer only has to handle the subset of tags that actually appear in article bodies. The processed folder is bigger than the ZIM because: - ZIM uses WebP images, which is extremely efficient. The preprocessor decodes WebP then re-encodes as JPEG photos or QOI diagrams . QOI is lossless — it faithfully preserves every pixel, which is great for quality but much larger than WebP. JPEG at quality 90 is also larger than WebP at equivalent visual quality. WebP is simply a better codec. - ZIM uses cross-article zstd compression. Articles are packed into large clusters 1–4 MB and compressed together — repeated phrases and boilerplate across articles compress away. The preprocessor re-compresses each article individually with LZ4, which loses that cross-article redundancy. LZ4 is chosen for decompression speed on the ESP32, not compression ratio. - Python 3.9+ - Dependencies listed in preprocessor/requirements.txt : pip install libzim lz4 beautifulsoup4 lxml Pillow cairosvg qoi cairosvg also requires the system cairo library: - Ubuntu/Debian: sudo apt install libcairo2 - macOS: brew install cairo cd preprocessor python3 build wiki db.py <input.zim <output dir Example: python3 build wiki db.py --thumb-size 240x159 wikipedia en simple all maxi 2026-05.zim output en simple all maxi This runs two passes: Pass 1 — indexes all article titles, builds index.bin , sparse index.bin , id index.bin Pass 2 — decompresses, cleans strips ZIM boilerplate , and re-compresses articles in parallel, producing articles NNNN.dat chunks Images — renders and encodes thumbnails into img NNNN.dat chunks Word index — builds word index.bin + title index.bin for full-text "contains" search Build time is roughly 30–60 minutes on a modern PC uses up to 4 CPU cores by default . | Flag | Default | Description | |---|---|---| --limit N | 0 all | Process only the first N articles useful for testing | --workers N | auto | Number of parallel compression workers threads | --thumb-size WxH | 320x212 | Max thumbnail dimensions in pixels | --jpeg-quality Q | 90 | JPEG quality for photo thumbnails, 1–95. Does not affect size much. | --no-images | off | Skip image processing | --images-only | off | Rebuild image database only articles already built | --word-index-only | off | Rebuild word/title index only | --verbose | off | Extra progress output | All files go into <output dir / and must be copied to a wiki/ folder on the SD card root. | File | Description | |---|---| index.bin | Fixed-width title index binary searchable, sorted | sparse index.bin | Every 64th title key — loaded into ESP32 RAM for fast search | id index.bin | Maps article ID → chunk + offset + length | index meta.txt | Metadata: article count, chunk size, database name | articles NNNN.dat | LZ4-compressed article HTML, split into 32 MB chunks | img index.bin | Maps image ID → chunk + offset + length | img NNNN.dat | Encoded image thumbnails JPEG or QOI , split into 4 MB chunks | word index.bin | Word → article ID list for "contains" search | title index.bin | Article title index for "contains" search | - Format the card as FAT32. A 64KB allocation unit is best. - Create a wiki/ folder at the root. - Copy all files from <output dir / into /wiki/ . The card should look like: /wiki/ index.bin sparse index.bin id index.bin index meta.txt articles 0000.dat articles 0001.dat ... img index.bin img 0000.dat ... word index.bin title index.bin The firmware is a PlatformIO project. cd firmware pio run -t upload Monitor serial output at 115200 baud: pio device monitor On first boot the sparse index is cached to LittleFS so subsequent boots are faster. Search : tap the search box to show the keyboard. Type a query and press GO .- Prefix-matching results appear first; a "contains:" divider separates full-text matches. Navigate : tap a blue underlined link to follow it. Tap < BACK to return. Scroll : swipe up/down, or use the arrow buttons in the nav bar. Images : tap an image thumbnail to view it full-screen. firmware/ PlatformIO ESP32 firmware src/ main.cpp Boot, splash screen ui.cpp Search, article, and image views wiki db.cpp SD database access search, load, images html render.cpp HTML → TFT renderer display.cpp TFT helpers, UTF-8 transliteration keyboard.cpp On-screen QWERTY keyboard touch.cpp XPT2046 touch driver config.h Hardware pins, file paths, constants preprocessor/ PC-side database builder build wiki db.py Main build script ZIM → binary DB debug server.py Local HTTP server for browser-based preview output en knots maxi/ small example wiki. Has smaller images to reduce size