{"slug": "fts5-icu-tokenizer-for-sqlite", "title": "FTS5 ICU Tokenizer for SQLite", "summary": "A developer has rewritten the FTS5 ICU Tokenizer for SQLite in Zig 0.16.0, replacing the original C implementation to simplify UTF-8 to UTF-16 conversions for ICU and enable full-text search on non-space-tokenized languages like Thai. The rewrite was assisted by various AI models and is detailed in the project's README.", "body_md": "This project, rewritten in Zig from C, tokenizes text input before indexing it with SQLite’s FTS5. This enables full-text search on non-space-tokenized languages like Thai.\n\nZig 0.16.0\n\nI’ve rewritten much of my previous C code to Zig using various AI models and iterations. The reason for this rewrite is detailed in the README.md but in short, converting UTF-8 to UTF-16 for ICU and then convert it back to UTF-8 for SQLite in C is quite cumbersome.", "url": "https://wpnews.pro/news/fts5-icu-tokenizer-for-sqlite", "canonical_source": "https://ziggit.dev/t/fts5-icu-tokenizer-for-sqlite/17228#post_1", "published_at": "2026-08-14 15:16:45+00:00", "updated_at": "2026-08-14 15:24:55.676775+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["SQLite", "FTS5", "ICU", "Zig"], "alternates": {"html": "https://wpnews.pro/news/fts5-icu-tokenizer-for-sqlite", "markdown": "https://wpnews.pro/news/fts5-icu-tokenizer-for-sqlite.md", "text": "https://wpnews.pro/news/fts5-icu-tokenizer-for-sqlite.txt", "jsonld": "https://wpnews.pro/news/fts5-icu-tokenizer-for-sqlite.jsonld"}}