{"slug": "cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs", "title": "Cross-Lingual Entity Resolution: Why Your Knowledge Graph Has Four Samsungs", "summary": "A developer building a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data found that cross-lingual entity resolution breaks knowledge graphs when the same entity appears under different scripts. The solution involves transliteration to a common script, blocking to reduce candidate pairs, and language-pair-specific similarity thresholds to accurately merge entities like Samsung Electronics across languages.", "body_md": "Cross-lingual entity resolution is where knowledge graphs quietly break down.\n\nI ran into this building [er-api](https://er-api.hannune.ai), a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data.\n\nThe problem is deceptively simple: Samsung Electronics appears as four different strings across source documents.\n\nEach string hashes to a different node in the graph. A query for Samsung Electronics misses three-quarters of the data unless you happen to use the exact string form the graph was built with. One real company becomes four phantom entities.\n\nThe instinct is to use fuzzy string matching — Levenshtein distance, cosine similarity on character n-grams. This works for typos and abbreviations within the same script. It fails completely across scripts.\n\n\"Samsung Electronics\" and \"サムスン電子\" share no character sequences. No string distance metric will recognize them as the same entity without preprocessing.\n\nThe first step is converting all strings to a common script before comparison.\n\n``` python\nimport jaconv  # Japanese: katakana/hiragana to romaji\nfrom hanja import translate as hanja_to_hangul\nfrom hangul_romanize import Transliterate\nfrom hangul_romanize.rule import RevisedRomanization\n\ntransliterator = Transliterate(RevisedRomanization())\n\ndef canonical_form(text: str, source_lang: str) -> str:\n    \"\"\"Convert a name to a canonical Latin-ish form for comparison.\"\"\"\n    if source_lang == \"ja\":\n        text = jaconv.kata2hepburn(jaconv.hira2kata(text))\n    elif source_lang == \"zh\":\n        text = hanja_to_hangul(text)\n        source_lang = \"ko\"\n\n    if source_lang == \"ko\":\n        text = transliterator.translit(text)\n\n    return text.lower().strip()\n```\n\nAfter transliteration:\n\nNot identical, but close enough that a similarity metric can catch them. Transliteration alone collapses a large fraction of duplicates before any similarity computation runs.\n\nComparing all n-squared candidate pairs across four languages at scale is not viable. For a corpus of 100k entities across four languages, that is 10 billion pairs.\n\nBlocking reduces candidates to a manageable set by only comparing pairs that share a blocking key — a coarse-grained signature that likely-duplicate entities share.\n\n``` php\nimport re\n\ndef blocking_key(name: str, lang: str) -> str:\n    \"\"\"\n    Generate a blocking key for grouping candidate pairs.\n    Strips legal suffixes, normalizes, takes first 4 chars.\n    \"\"\"\n    canonical = canonical_form(name, lang)\n\n    legal_suffixes = [\n        r\"\\b(co|corp|inc|ltd|llc|plc|gmbh|ag|sa|bv)\\b\\.?\",\n        r\"\\b(주식회사|kabushiki kaisha|kk|株式会社|有限公司)\\b\",\n    ]\n    for pattern in legal_suffixes:\n        canonical = re.sub(pattern, \"\", canonical, flags=re.IGNORECASE)\n\n    canonical = re.sub(r\"[^a-z0-9]\", \"\", canonical)\n    return canonical[:4]\n```\n\nAll entities that share the same blocking key enter the same candidate pool for pairwise comparison. Entities in different blocks never get compared.\n\nOnce you have candidate pairs within a block, you need a similarity score. Splink's EM estimator works well here — it learns match probabilities from the data without requiring fully labeled examples.\n\nBut the threshold for what counts as a match is not the same across language pairs.\n\nKorean-English pairs tolerate more abbreviation variance. \"Samsung C&T\" vs. \"Samsung C&T Corporation\" — same entity, different abbreviation patterns. A Korean-English threshold of 0.75 captures this without over-merging.\n\nJapanese-Chinese pairs are tighter. The transliteration paths for Japanese and Chinese overlap less cleanly than Korean-Latin paths. A threshold of 0.85+ is safer here.\n\n```\nLANG_PAIR_THRESHOLDS = {\n    (\"ko\", \"en\"): 0.75,\n    (\"ko\", \"ja\"): 0.80,\n    (\"ko\", \"zh\"): 0.82,\n    (\"en\", \"ja\"): 0.78,\n    (\"en\", \"zh\"): 0.80,\n    (\"ja\", \"zh\"): 0.85,\n}\n\ndef threshold_for_pair(lang_a: str, lang_b: str) -> float:\n    key = tuple(sorted([lang_a, lang_b]))\n    return LANG_PAIR_THRESHOLDS.get(key, 0.80)\n```\n\nSplink's EM estimator can run unsupervised, but performance is better with labeled examples.\n\nFor Korean-English, there is labeled data available from Korean financial filings — companies are required to report both the Korean and English names in disclosure documents. A financial data scraper gives you positive pairs cheaply.\n\nFor Korean-Chinese and Japanese-Chinese, there is almost no open labeled data. What actually worked:\n\n**Generating negative pairs** by sampling clearly distinct companies from different industry sectors — any two companies with no overlapping industry codes and no name similarity are almost certainly different entities.\n\n**Bootstrapping positive pairs** from a subset of companies with known cross-lingual identifiers (Dun & Bradstreet DUNS numbers, Bloomberg tickers) where the same company appears in all four language corpora.\n\n**Manual annotation** for the ambiguous middle tier — about 2,000 pairs reviewed by a bilingual Korean-Chinese speaker.\n\nIt is not elegant. But the EM estimator is sensitive enough to training data quality that this effort pays back in precision.\n\nAfter running cross-lingual ER on the initial corpus:\n\nThe graph shrinks after you apply this. Retrieval quality improves because queries no longer split across phantom duplicates.\n\n*Building er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at hannune.ai.*\n\n*Cover: Alina Grubnyak on Unsplash*", "url": "https://wpnews.pro/news/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs", "canonical_source": "https://dev.to/hannune/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs-2h0b", "published_at": "2026-07-19 02:38:38+00:00", "updated_at": "2026-07-19 02:59:33.522537+00:00", "lang": "en", "topics": ["natural-language-processing", "developer-tools", "machine-learning"], "entities": ["Samsung Electronics", "Samsung C&T", "Splink"], "alternates": {"html": "https://wpnews.pro/news/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs", "markdown": "https://wpnews.pro/news/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs.md", "text": "https://wpnews.pro/news/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs.txt", "jsonld": "https://wpnews.pro/news/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs.jsonld"}}