FTS5 ICU Tokenizer for SQLite

A developer has rewritten the FTS5 ICU Tokenizer for SQLite in Zig 0.16.0, replacing the original C implementation to simplify UTF-8 to UTF-16 conversions for ICU and enable full-text search on non-space-tokenized languages like Thai. The rewrite was assisted by various AI models and is detailed in the project's README.

This project, rewritten in Zig from C, tokenizes text input before indexing it with SQLite’s FTS5. This enables full-text search on non-space-tokenized languages like Thai. Zig 0.16.0 I’ve rewritten much of my previous C code to Zig using various AI models and iterations. The reason for this rewrite is detailed in the README.md but in short, converting UTF-8 to UTF-16 for ICU and then convert it back to UTF-8 for SQLite in C is quite cumbersome.