Cross-Lingual Entity Resolution: Why Your Knowledge Graph Has Four Samsungs

Cross-lingual entity resolution is where knowledge graphs quietly break down.

I ran into this building er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data.

The problem is deceptively simple: Samsung Electronics appears as four different strings across source documents.

Each string hashes to a different node in the graph. A query for Samsung Electronics misses three-quarters of the data unless you happen to use the exact string form the graph was built with. One real company becomes four phantom entities.

The instinct is to use fuzzy string matching — Levenshtein distance, cosine similarity on character n-grams. This works for typos and abbreviations within the same script. It fails completely across scripts.

"Samsung Electronics" and "サムスン電子" share no character sequences. No string distance metric will recognize them as the same entity without preprocessing.

The first step is converting all strings to a common script before comparison.

import jaconv  # Japanese: katakana/hiragana to romaji
from hanja import translate as hanja_to_hangul
from hangul_romanize import Transliterate
from hangul_romanize.rule import RevisedRomanization

transliterator = Transliterate(RevisedRomanization())

def canonical_form(text: str, source_lang: str) -> str:
    """Convert a name to a canonical Latin-ish form for comparison."""
    if source_lang == "ja":
        text = jaconv.kata2hepburn(jaconv.hira2kata(text))
    elif source_lang == "zh":
        text = hanja_to_hangul(text)
        source_lang = "ko"

    if source_lang == "ko":
        text = transliterator.translit(text)

    return text.lower().strip()

After transliteration:

Not identical, but close enough that a similarity metric can catch them. Transliteration alone collapses a large fraction of duplicates before any similarity computation runs.

Comparing all n-squared candidate pairs across four languages at scale is not viable. For a corpus of 100k entities across four languages, that is 10 billion pairs.

Blocking reduces candidates to a manageable set by only comparing pairs that share a blocking key — a coarse-grained signature that likely-duplicate entities share.

import re

def blocking_key(name: str, lang: str) -> str:
    """
    Generate a blocking key for grouping candidate pairs.
    Strips legal suffixes, normalizes, takes first 4 chars.
    """
    canonical = canonical_form(name, lang)

    legal_suffixes = [
        r"\b(co|corp|inc|ltd|llc|plc|gmbh|ag|sa|bv)\b\.?",
        r"\b(주식회사|kabushiki kaisha|kk|株式会社|有限公司)\b",
    ]
    for pattern in legal_suffixes:
        canonical = re.sub(pattern, "", canonical, flags=re.IGNORECASE)

    canonical = re.sub(r"[^a-z0-9]", "", canonical)
    return canonical[:4]

All entities that share the same blocking key enter the same candidate pool for pairwise comparison. Entities in different blocks never get compared.

Once you have candidate pairs within a block, you need a similarity score. Splink's EM estimator works well here — it learns match probabilities from the data without requiring fully labeled examples.

But the threshold for what counts as a match is not the same across language pairs.

Korean-English pairs tolerate more abbreviation variance. "Samsung C&T" vs. "Samsung C&T Corporation" — same entity, different abbreviation patterns. A Korean-English threshold of 0.75 captures this without over-merging.

Japanese-Chinese pairs are tighter. The transliteration paths for Japanese and Chinese overlap less cleanly than Korean-Latin paths. A threshold of 0.85+ is safer here.

LANG_PAIR_THRESHOLDS = {
    ("ko", "en"): 0.75,
    ("ko", "ja"): 0.80,
    ("ko", "zh"): 0.82,
    ("en", "ja"): 0.78,
    ("en", "zh"): 0.80,
    ("ja", "zh"): 0.85,
}

def threshold_for_pair(lang_a: str, lang_b: str) -> float:
    key = tuple(sorted([lang_a, lang_b]))
    return LANG_PAIR_THRESHOLDS.get(key, 0.80)

Splink's EM estimator can run unsupervised, but performance is better with labeled examples.

For Korean-English, there is labeled data available from Korean financial filings — companies are required to report both the Korean and English names in disclosure documents. A financial data scraper gives you positive pairs cheaply.

For Korean-Chinese and Japanese-Chinese, there is almost no open labeled data. What actually worked:

Generating negative pairs by sampling clearly distinct companies from different industry sectors — any two companies with no overlapping industry codes and no name similarity are almost certainly different entities.

Bootstrapping positive pairs from a subset of companies with known cross-lingual identifiers (Dun & Bradstreet DUNS numbers, Bloomberg tickers) where the same company appears in all four language corpora.

Manual annotation for the ambiguous middle tier — about 2,000 pairs reviewed by a bilingual Korean-Chinese speaker.

It is not elegant. But the EM estimator is sensitive enough to training data quality that this effort pays back in precision.

After running cross-lingual ER on the initial corpus:

The graph shrinks after you apply this. Retrieval quality improves because queries no longer split across phantom duplicates.

Building er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at hannune.ai.

Cover: Alina Grubnyak on Unsplash