# FTS5 ICU Tokenizer for SQLite

> Source: <https://ziggit.dev/t/fts5-icu-tokenizer-for-sqlite/17228#post_1>
> Published: 2026-08-14 15:16:45+00:00

This project, rewritten in Zig from C, tokenizes text input before indexing it with SQLite’s FTS5. This enables full-text search on non-space-tokenized languages like Thai.

Zig 0.16.0

I’ve rewritten much of my previous C code to Zig using various AI models and iterations. The reason for this rewrite is detailed in the README.md but in short, converting UTF-8 to UTF-16 for ICU and then convert it back to UTF-8 for SQLite in C is quite cumbersome.