{"slug": "what-breaks-when-your-app-has-to-reason-about-chinese-characters", "title": "What Breaks When Your App Has to Reason About Chinese Characters", "summary": "A developer building a Chinese character analysis system for naming explains how internationalization assumptions break with CJK text. Python 3 handles Unicode correctly but JavaScript's UTF-16 surrogate pairs cause length errors, MySQL's utf8 charset truncates 4-byte characters, and NFKC normalization destroys meaningful distinctions in naming corpora. The system stores NFC-normalized text with separate index and display fields, uses OpenCC for phrase-level conversion, and maintains explicit variant groups for simplified vs traditional characters.", "body_md": "Most internationalization advice assumes text is a bag of words separated by spaces. Chinese breaks that assumption at every layer of the stack, and if your product's core value depends on reasoning about individual characters — not just displaying them — you end up rewriting infrastructure you assumed was solved.\n\nI spend most of my time on a system that analyzes Chinese characters for naming: stroke counts, five-element (五行) classification, tone patterns, and sourcing candidate characters from classical poetry corpora. Here's what actually bit us.\n\nThe first thing that breaks is `len()`\n\n.\n\n```\nname = \"𠮷田\"          # note: the rare variant of 吉, U+20BB7\nprint(len(name))       # Python 3: 2 -- correct\n```\n\nPython 3 gets this right because strings are sequences of code points. JavaScript does not:\n\n``` js\nconst name = \"𠮷田\";\nname.length            // 3  -- UTF-16 surrogate pair counted as two\n[...name].length       // 2  -- correct\n```\n\nAny CJK-aware code path in JS must use `Array.from()`\n\n/ spread / `for...of`\n\n, never `.length`\n\nor `charAt`\n\n. This matters immediately for us because a name's stroke-count math is per-character. Counting one character as two silently produces a wrong numerological result — and it never throws, it just returns a plausible wrong answer. Those are the worst bugs.\n\nThe second thing that breaks is your database column. Extension B/C/D characters (U+20000 and up) are 4-byte UTF-8. MySQL's `utf8`\n\ncharset is a 3-byte subset and will truncate or reject them:\n\n```\nALTER TABLE characters\n  CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;\n```\n\nNote `utf8mb4_bin`\n\n, not a general collation. Unicode collations for CJK do fuzzy things you never want in a lookup table — you want exact code point identity.\n\nEveryone reaches for `NFKC`\n\nbecause it \"cleans up\" input. For CJK it is destructive.\n\nUnicode has a compatibility block of CJK ideographs (U+F900–U+FAFF) that exists purely for round-tripping legacy encodings. NFC and NFKC both map most of them onto the unified codepoint:\n\n``` python\nimport unicodedata\ns = \"例\"                        # CJK COMPATIBILITY IDEOGRAPH-F9B5\nunicodedata.normalize(\"NFC\", s)     # -> the unified ideograph, different codepoint\n```\n\nThat's usually fine. What is not fine is NFKC on mixed content: it converts fullwidth forms to ASCII (`Ａ`\n\n→ `A`\n\n), collapses fullwidth punctuation, and rewrites halfwidth kana. In a naming corpus, fullwidth vs halfwidth is real signal about the source document.\n\nOur rule ended up being: **NFC for storage, never NFKC, and normalize once at the ingest boundary.** Anything that normalizes on read will eventually disagree with an index built before the rule changed.\n\nThen there's variation selectors. The Ideographic Variation Sequences at `U+E0100`\n\n+ attach to a base ideograph to select a specific glyph, and they survive NFC. Dedupe candidates with a plain hash set and `葛`\n\nand `葛`\n\n+ `U+E0100`\n\nare two entries. We strip variation selectors for the *index key* but preserve them on the display string — two fields, not one.\n\nThis is where most implementations get sloppy. The mapping is not a bijection.\n\n`发`\n\nis both 發 (to emit) and 髮 (hair). `后`\n\ncovers 後 (after) and 后 (empress).For anything user-facing, use OpenCC rather than a homemade table — it does phrase-level conversion, which resolves a good chunk of the ambiguity:\n\n``` php\nfrom opencc import OpenCC\ncc = OpenCC('s2twp')   # simplified -> traditional, Taiwan idiom\ncc.convert(\"头发\")      # 頭髮 -- phrase context picks 髮, not 發\n```\n\nBut for a *naming* system, phrase context often doesn't exist — you're evaluating a single character in isolation. Our approach: store the character in both scripts as separate rows with an explicit `variant_group_id`\n\n, and never auto-convert at query time. Let the user pick the script; don't guess.\n\nStroke count has the same shape of problem. It's not one number per character — it depends on the script (简 vs 繁), and traditional counts differ between the Kangxi dictionary convention and modern standards for radicals like 艹 (3 strokes in modern counting, 6 under Kangxi where it is treated as 艸). Naming numerology systems typically demand Kangxi counts specifically. If you pull stroke data from a general Unihan field like `kTotalStrokes`\n\n, you get the modern count, and your results will be off for a large fraction of characters. Unihan is the right source, but you have to read the right property. Getting that right — Kangxi strokes, five-element classification, and tone-pattern checking against 诗经/楚辞 source lines — is the core of what we ship at [BabyNameAI](https://babynameai.org), and the stroke-count semantics were more work than the entire model layer sitting on top.\n\nClassical Chinese has no word delimiters and no reliable segmenter — tools like jieba are trained on modern Mandarin and will happily mis-segment a Tang line. So don't segment.\n\nIndex character n-grams instead:\n\n```\n-- Postgres: trigram index sidesteps a CJK-hostile default text parser\nCREATE EXTENSION pg_trgm;\nCREATE INDEX ON poem_lines USING gin (line gin_trgm_ops);\n```\n\nFor Elasticsearch, the built-in `cjk`\n\nanalyzer (which produces bigrams) beats a standard analyzer, and beats `ik_smart`\n\nfor classical text since IK's dictionary is modern vocabulary. Bigrams over a few hundred thousand lines of 诗经/楚辞/唐诗 is a small index and it recalls correctly on the queries that matter: \"find every line containing this character in second position of a two-character compound.\"\n\nOne last trap: tone data. Pinyin tone marks are combining diacritics under NFD (`zhōng`\n\ndecomposes to `o`\n\n+ `U+0304`\n\n) but precomposed under NFC, so comparing differently-normalized pinyin silently fails. And classical tone categories (平上去入) do not map onto modern Mandarin tones — the entering tone 入声 disappeared in Mandarin but survives in Cantonese and Min. Checking a name against classical prosody rules needs a 中古音 rime table, not modern pinyin.\n\nNone of this is exotic once you've hit it. Every \"just use Unicode\" tutorial simply stops right before the interesting part.", "url": "https://wpnews.pro/news/what-breaks-when-your-app-has-to-reason-about-chinese-characters", "canonical_source": "https://dev.to/yuanjzhou/what-breaks-when-your-app-has-to-reason-about-chinese-characters-1hk1", "published_at": "2026-07-25 08:10:47+00:00", "updated_at": "2026-07-25 08:33:53.009449+00:00", "lang": "en", "topics": ["natural-language-processing", "developer-tools"], "entities": ["Python", "JavaScript", "MySQL", "Unicode", "OpenCC"], "alternates": {"html": "https://wpnews.pro/news/what-breaks-when-your-app-has-to-reason-about-chinese-characters", "markdown": "https://wpnews.pro/news/what-breaks-when-your-app-has-to-reason-about-chinese-characters.md", "text": "https://wpnews.pro/news/what-breaks-when-your-app-has-to-reason-about-chinese-characters.txt", "jsonld": "https://wpnews.pro/news/what-breaks-when-your-app-has-to-reason-about-chinese-characters.jsonld"}}