# What Breaks When Your App Has to Reason About Chinese Characters > Source: > Published: 2026-07-25 08:10:47+00:00 Most internationalization advice assumes text is a bag of words separated by spaces. Chinese breaks that assumption at every layer of the stack, and if your product's core value depends on reasoning about individual characters — not just displaying them — you end up rewriting infrastructure you assumed was solved. I spend most of my time on a system that analyzes Chinese characters for naming: stroke counts, five-element (五行) classification, tone patterns, and sourcing candidate characters from classical poetry corpora. Here's what actually bit us. The first thing that breaks is `len()` . ``` name = "𠮷田" # note: the rare variant of 吉, U+20BB7 print(len(name)) # Python 3: 2 -- correct ``` Python 3 gets this right because strings are sequences of code points. JavaScript does not: ``` js const name = "𠮷田"; name.length // 3 -- UTF-16 surrogate pair counted as two [...name].length // 2 -- correct ``` Any CJK-aware code path in JS must use `Array.from()` / spread / `for...of` , never `.length` or `charAt` . This matters immediately for us because a name's stroke-count math is per-character. Counting one character as two silently produces a wrong numerological result — and it never throws, it just returns a plausible wrong answer. Those are the worst bugs. The second thing that breaks is your database column. Extension B/C/D characters (U+20000 and up) are 4-byte UTF-8. MySQL's `utf8` charset is a 3-byte subset and will truncate or reject them: ``` ALTER TABLE characters CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; ``` Note `utf8mb4_bin` , not a general collation. Unicode collations for CJK do fuzzy things you never want in a lookup table — you want exact code point identity. Everyone reaches for `NFKC` because it "cleans up" input. For CJK it is destructive. Unicode has a compatibility block of CJK ideographs (U+F900–U+FAFF) that exists purely for round-tripping legacy encodings. NFC and NFKC both map most of them onto the unified codepoint: ``` python import unicodedata s = "例" # CJK COMPATIBILITY IDEOGRAPH-F9B5 unicodedata.normalize("NFC", s) # -> the unified ideograph, different codepoint ``` That's usually fine. What is not fine is NFKC on mixed content: it converts fullwidth forms to ASCII (`Ａ` → `A` ), collapses fullwidth punctuation, and rewrites halfwidth kana. In a naming corpus, fullwidth vs halfwidth is real signal about the source document. Our rule ended up being: **NFC for storage, never NFKC, and normalize once at the ingest boundary.** Anything that normalizes on read will eventually disagree with an index built before the rule changed. Then there's variation selectors. The Ideographic Variation Sequences at `U+E0100` + attach to a base ideograph to select a specific glyph, and they survive NFC. Dedupe candidates with a plain hash set and `葛` and `葛` + `U+E0100` are two entries. We strip variation selectors for the *index key* but preserve them on the display string — two fields, not one. This is where most implementations get sloppy. The mapping is not a bijection. `发` is both 發 (to emit) and 髮 (hair). `后` covers 後 (after) and 后 (empress).For anything user-facing, use OpenCC rather than a homemade table — it does phrase-level conversion, which resolves a good chunk of the ambiguity: ``` php from opencc import OpenCC cc = OpenCC('s2twp') # simplified -> traditional, Taiwan idiom cc.convert("头发") # 頭髮 -- phrase context picks 髮, not 發 ``` But for a *naming* system, phrase context often doesn't exist — you're evaluating a single character in isolation. Our approach: store the character in both scripts as separate rows with an explicit `variant_group_id` , and never auto-convert at query time. Let the user pick the script; don't guess. Stroke count has the same shape of problem. It's not one number per character — it depends on the script (简 vs 繁), and traditional counts differ between the Kangxi dictionary convention and modern standards for radicals like 艹 (3 strokes in modern counting, 6 under Kangxi where it is treated as 艸). Naming numerology systems typically demand Kangxi counts specifically. If you pull stroke data from a general Unihan field like `kTotalStrokes` , you get the modern count, and your results will be off for a large fraction of characters. Unihan is the right source, but you have to read the right property. Getting that right — Kangxi strokes, five-element classification, and tone-pattern checking against 诗经/楚辞 source lines — is the core of what we ship at [BabyNameAI](https://babynameai.org), and the stroke-count semantics were more work than the entire model layer sitting on top. Classical Chinese has no word delimiters and no reliable segmenter — tools like jieba are trained on modern Mandarin and will happily mis-segment a Tang line. So don't segment. Index character n-grams instead: ``` -- Postgres: trigram index sidesteps a CJK-hostile default text parser CREATE EXTENSION pg_trgm; CREATE INDEX ON poem_lines USING gin (line gin_trgm_ops); ``` For Elasticsearch, the built-in `cjk` analyzer (which produces bigrams) beats a standard analyzer, and beats `ik_smart` for classical text since IK's dictionary is modern vocabulary. Bigrams over a few hundred thousand lines of 诗经/楚辞/唐诗 is a small index and it recalls correctly on the queries that matter: "find every line containing this character in second position of a two-character compound." One last trap: tone data. Pinyin tone marks are combining diacritics under NFD (`zhōng` decomposes to `o` + `U+0304` ) but precomposed under NFC, so comparing differently-normalized pinyin silently fails. And classical tone categories (平上去入) do not map onto modern Mandarin tones — the entering tone 入声 disappeared in Mandarin but survives in Cantonese and Min. Checking a name against classical prosody rules needs a 中古音 rime table, not modern pinyin. None of this is exotic once you've hit it. Every "just use Unicode" tutorial simply stops right before the interesting part.