{"slug": "wikipedia-offline-reader-for-esp32-cyd", "title": "Wikipedia Offline Reader for ESP32 CYD", "summary": "Alun Morris and Claude Code released an offline Wikipedia reader for the ESP32-2432S028 (Cheap Yellow Display), a 320×240 ILI9341 touchscreen module, storing articles, images, and a search index on a microSD card with no internet connection. The device requires at least an 8 GB microSD card, with the recommended Simple English Wikipedia ZIM file (wikipedia_en_simple_all_maxi_YYYY-MM.zim) being a ~3.3 GB download containing ~285,000 articles and images, processed to ~10 GB. The project supports two hardware configurations: the CYD board and a bare JC2432S024 display module wired to an ESP32-C3 dev board, with PlatformIO environments 'cyd' and 'c3' respectively.", "body_md": "An offline Wikipedia reader for the ESP32-2432S028 (\"Cheap Yellow Display\") — a 320×240 ILI9341 touchscreen module. Articles, images, and a full search index are stored on a microSD card and read on-device with no internet connection.\n\n[Written by Alun Morris and Claude Code.](https://private-user-images.githubusercontent.com/4630866/604418311-b713e92a-6ba9-4e7e-b10c-57586139830f.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3ODgwNDA0MjMsIm5iZiI6MTc4ODA0MDEyMywicGF0aCI6Ii80NjMwODY2LzYwNDQxODMxMS1iNzEzZTkyYS02YmE5LTRlN2UtYjEwYy01NzU4NjEzOTgzMGYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI2MDgyOSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNjA4MjlUMjE0ODQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTM3OWIxZDMwZDlhMzIyN2FjNjM2OTJjYmE3ODJlMzU5YjA2NmUxYjMwZjQwZWIwNWYyZDM5YWJmMDI2MjljOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmcmVzcG9uc2UtY29udGVudC10eXBlPWltYWdlJTJGcG5nIn0._MmzB_rALOVaEQFlj8gN_ud-IB2n1N_ufdDJWaLx29o)\n\n| Component | Detail |\n|---|---|\n| Board | ESP32-2432S028 (CYD) |\n| Display | 320×240 ILI9341 (HSPI bus) |\n| Touch | XPT2046 resistive (shared HSPI) |\n| SD card | VSPI bus — CS=5, MOSI=23, MISO=19, SCK=18 |\n| Backlight | GPIO 21 |\n\nUse PlatformIO environment `cyd`\n\n(`pio run -e cyd -t upload`\n\n).\n\nThe **JC2432S024** is a bare 2.4″ 320×240 ILI9341 display module with XPT2046 resistive touch and an SD card slot. It has **no processor onboard** — wire it to an ESP32-C3 dev board (SuperMini or DevKitM-1).\n\nThe ESP32-C3 has a single SPI peripheral, so display, SD card and touch controller all share one SPI bus with separate chip-select lines.\n\n| Module pin | ESP32-C3 GPIO | Notes |\n|---|---|---|\n| SCK | 4 | Shared by TFT + Touch + SD |\n| MOSI / SDI / T_DIN / SD_MOSI | 6 | Shared |\n| MISO / SDO / T_DO / SD_MISO | 5 | Shared |\n| CS (TFT) | 7 | |\n| DC / RS | 1 | |\n| RST | 3V3 | Tie high — not driven by firmware |\n| SD_CS | 10 | |\n| T_CS | 3 | |\n| T_IRQ | 8 | PENIRQ — idles HIGH, safe on boot strapping pin |\n| LED / BL (backlight) | 0 | Or tie directly to 3V3 for always-on backlight |\n| VCC | 3V3 | |\n| GND | GND |\n\nAvoid GPIO 2 and GPIO 9 for external loads — they are ESP32-C3 boot-strapping pins.\n\nUse PlatformIO environment `c3`\n\n(`pio run -e c3 -t upload`\n\n).\n\n**First boot:** touch calibration runs automatically. Tap the two red crosshairs when prompted. Calibration is saved to flash and skipped on subsequent boots.\n\nA microSD card of **at least 8 GB** is required (Simple English Wikipedia uses ~7 GB on card).\n\nThe database is built from a [Kiwix](https://www.kiwix.org/) ZIM file. Download one from ([Wikimedia dumps](https://dumps.wikimedia.org/kiwix/zim/wikipedia/)).\n\n**Recommended: Simple English Wikipedia — wikipedia_en_simple_all_maxi_YYYY-MM.zim**\n\n- ~3.3 GB download, ~285 000 articles\n- Shorter articles and simpler language — well-suited to a small screen\n- Includes images (\n`_maxi`\n\nvariant) - processed size is ~10GB\n\nOther ZIM files will work but larger editions (eg full English, ~90 GB) may exceed the SD card size the CYD is known to handle (32GB OK, 64GB may work).\n\nChoose the\n\n`_maxi`\n\nvariant (includes images). The`_mini`\n\nvariant omits images and has only the top 50-100k articles.\n\n**There is a small demo Wiki about knots in the preprocessor folder**\n\nThe preprocessor converts a ZIM file into the binary database format read by the firmware.\n\nThe ESP32 cannot read a ZIM file directly. Several hard constraints make a purpose-built binary format necessary:\n\n-\n**ZIM uses zstd cluster compression.** Decompressing a zstd cluster requires holding the entire cluster in RAM. ZIM clusters are typically 1–4 MB, which exceeds the ESP32's ~300 KB of usable heap. The preprocessor re-compresses each article individually with LZ4, which decompresses in a few KB of working memory. -\n**ZIM's index structure is too complex for embedded use.** ZIM uses a URL-sorted B-tree-style index with variable-length entries. The binary index produced here is fixed-width (80 bytes/record), sorted by normalised title, and paired with a tiny sparse index (one entry per 64 articles) that fits entirely in RAM (~7 KB). Together they allow title lookup with zero SD seeks to find the scan start. -\n**Images must be in formats the ESP32 can decode.** The firmware decodes JPEG via the hardware-accelerated TJpgDec library and QOI via a lightweight software decoder. ZIM stores images as WebP internally (with the original format preserved in the file path), which the ESP32 cannot guarantee to decode in available RAM. The preprocessor converts everything to JPEG (photos) or QOI (diagrams/SVGs). -\n**ZIM HTML needs cleaning.** Wikipedia ZIM files inject boilerplate footers, navigation chrome, and complex class structures into every article. The preprocessor strips these with BeautifulSoup so the firmware's minimal HTML renderer only has to handle the subset of tags that actually appear in article bodies.\n\nThe processed folder is bigger than the ZIM because:\n\n-\nZIM uses WebP images, which is extremely efficient. The preprocessor decodes WebP then re-encodes as JPEG (photos) or QOI (diagrams). QOI is lossless — it faithfully preserves every pixel, which is great for quality but much larger than WebP. JPEG at quality 90 is also larger than WebP at equivalent visual quality. WebP is simply a better codec.\n\n-\nZIM uses cross-article zstd compression. Articles are packed into large clusters (1–4 MB) and compressed together — repeated phrases and boilerplate across articles compress away. The preprocessor re-compresses each article individually with LZ4, which loses that cross-article redundancy. LZ4 is chosen for decompression speed on the ESP32, not compression ratio.\n\n- Python 3.9+\n- Dependencies listed in\n`preprocessor/requirements.txt`\n\n:\n\n```\npip install libzim lz4 beautifulsoup4 lxml Pillow cairosvg qoi\n```\n\n`cairosvg`\n\nalso requires the system `cairo`\n\nlibrary:\n\n- Ubuntu/Debian:\n`sudo apt install libcairo2`\n\n- macOS:\n`brew install cairo`\n\n```\ncd preprocessor\npython3 build_wiki_db.py <input.zim> <output_dir>\n```\n\nExample:\n\n```\npython3 build_wiki_db.py --thumb-size 240x159 wikipedia_en_simple_all_maxi_2026-05.zim output_en_simple_all_maxi\n```\n\nThis runs two passes:\n\n**Pass 1**— indexes all article titles, builds`index.bin`\n\n,`sparse_index.bin`\n\n,`id_index.bin`\n\n**Pass 2**— decompresses, cleans (strips ZIM boilerplate), and re-compresses articles in parallel, producing`articles_NNNN.dat`\n\nchunks**Images**— renders and encodes thumbnails into`img_NNNN.dat`\n\nchunks**Word index**— builds`word_index.bin`\n\n+`title_index.bin`\n\nfor full-text \"contains\" search\n\nBuild time is roughly 30–60 minutes on a modern PC (uses up to 4 CPU cores by default).\n\n| Flag | Default | Description |\n|---|---|---|\n`--limit N` |\n0 (all) | Process only the first N articles (useful for testing) |\n`--workers N` |\nauto | Number of parallel compression workers (threads) |\n`--thumb-size WxH` |\n320x212 | Max thumbnail dimensions in pixels |\n`--jpeg-quality Q` |\n90 | JPEG quality for photo thumbnails, 1–95. Does not affect size much. |\n`--no-images` |\noff | Skip image processing |\n`--images-only` |\noff | Rebuild image database only (articles already built) |\n`--word-index-only` |\noff | Rebuild word/title index only |\n`--verbose` |\noff | Extra progress output |\n\nAll files go into `<output_dir>/`\n\nand must be copied to a `wiki/`\n\nfolder on the SD card root.\n\n| File | Description |\n|---|---|\n`index.bin` |\nFixed-width title index (binary searchable, sorted) |\n`sparse_index.bin` |\nEvery 64th title key — loaded into ESP32 RAM for fast search |\n`id_index.bin` |\nMaps article ID → chunk + offset + length |\n`index_meta.txt` |\nMetadata: article count, chunk size, database name |\n`articles_NNNN.dat` |\nLZ4-compressed article HTML, split into 32 MB chunks |\n`img_index.bin` |\nMaps image ID → chunk + offset + length |\n`img_NNNN.dat` |\nEncoded image thumbnails (JPEG or QOI), split into 4 MB chunks |\n`word_index.bin` |\nWord → article ID list for \"contains\" search |\n`title_index.bin` |\nArticle title index for \"contains\" search |\n\n- Format the card as FAT32. A 64KB allocation unit is best.\n- Create a\n`wiki/`\n\nfolder at the root. - Copy all files from\n`<output_dir>/`\n\ninto`/wiki/`\n\n.\n\nThe card should look like:\n\n```\n/wiki/\n  index.bin\n  sparse_index.bin\n  id_index.bin\n  index_meta.txt\n  articles_0000.dat\n  articles_0001.dat\n  ...\n  img_index.bin\n  img_0000.dat\n  ...\n  word_index.bin\n  title_index.bin\n```\n\nThe firmware is a PlatformIO project.\n\n```\ncd firmware\npio run -t upload\n```\n\nMonitor serial output at 115200 baud:\n\n```\npio device monitor\n```\n\nOn first boot the sparse index is cached to LittleFS so subsequent boots are faster.\n\n**Search**: tap the search box to show the keyboard. Type a query and press** GO**.- Prefix-matching results appear first; a \"contains:\" divider separates full-text matches.\n\n**Navigate**: tap a blue underlined link to follow it. Tap**< BACK** to return.**Scroll**: swipe up/down, or use the arrow buttons in the nav bar.** Images**: tap an image thumbnail to view it full-screen.\n\n```\nfirmware/        PlatformIO ESP32 firmware\n  src/\n    main.cpp     Boot, splash screen\n    ui.cpp       Search, article, and image views\n    wiki_db.cpp  SD database access (search, load, images)\n    html_render.cpp  HTML → TFT renderer\n    display.cpp  TFT helpers, UTF-8 transliteration\n    keyboard.cpp On-screen QWERTY keyboard\n    touch.cpp    XPT2046 touch driver\n    config.h     Hardware pins, file paths, constants\n\npreprocessor/    PC-side database builder\n  build_wiki_db.py   Main build script (ZIM → binary DB)\n  debug_server.py    Local HTTP server for browser-based preview\n  output_en_knots_maxi/ small example wiki. Has smaller images to reduce size\n```\n\n", "url": "https://wpnews.pro/news/wikipedia-offline-reader-for-esp32-cyd", "canonical_source": "https://github.com/alunmorris/Offline-Wikipedia-ESP32/tree/master", "published_at": "2026-08-29 21:22:02+00:00", "updated_at": "2026-08-29 21:48:45.891459+00:00", "lang": "en", "topics": ["ai-products", "ai-tools"], "entities": ["Alun Morris", "Claude Code", "ESP32-2432S028", "JC2432S024", "ESP32-C3", "Kiwix", "Wikimedia", "Simple English Wikipedia"], "alternates": {"html": "https://wpnews.pro/news/wikipedia-offline-reader-for-esp32-cyd", "markdown": "https://wpnews.pro/news/wikipedia-offline-reader-for-esp32-cyd.md", "text": "https://wpnews.pro/news/wikipedia-offline-reader-for-esp32-cyd.txt", "jsonld": "https://wpnews.pro/news/wikipedia-offline-reader-for-esp32-cyd.jsonld"}}