{"slug": "model-genome-fingerprinting-whether-an-llm-was-trained-from-scratch-or-derived", "title": "Model Genome: Fingerprinting Whether an LLM Was Trained from Scratch or Derived", "summary": "A new open-source tool called Model Genome Korea can determine whether a Korean large language model (LLM) or vision-language model (VLM) was trained from scratch or derived from an open-weight base model, by analyzing the architecture fingerprint from config.json and tokenizer overlap. The tool, applied to nine Korean organizations' public foundation models, found exact architecture matches with Qwen, Llama, and DeepSeek, and identified cases like a model with Qwen2.5-7B architecture but only 0.38 tokenizer overlap, indicating a newly trained Korean tokenizer. The debate intensified after LG released K-EXAONE 2.0 (750B) in late July 2026, with a Zhihu thread exceeding 2.7 million views.", "body_md": "🧬 33\n\n#### Model Genome Korea\n\nDNA lineage test of Korean LLM & VLM foundation models\n\n`config.json`\n\n), `config`\n\n+ tokenizer remain the primary evidence. We applied the exact same yardstick to the public foundation models of nine Korean organizations. Try it live: Building a large language model on top of an open-weight base (Qwen, Llama, DeepSeek, Mistral) is a legitimate, industry-standard practice. But it is *different* from training a foundation model from scratch — and vendors do not always make the distinction explicit. When several labs released DeepSeek-rivaling \"self-developed\" models in late July 2026 (e.g. LG K-EXAONE 2.0, 750B), the debate spilled into Chinese tech communities as well — a Zhihu thread ([→ link](https://www.zhihu.com/question/2067512422555029717)) crossed 2.7M views. The natural question followed: **from scratch, or derived?**\n\nThis is answerable, objectively, from public files. Here is how.\n\n`config.json`\n\n)\nEvery `transformers`\n\ncheckpoint ships a `config.json`\n\n. A handful of fields form a surprisingly discriminative signature:\n\n`model_type`\n\n`vocab_size`\n\n`hidden_size`\n\n`intermediate_size`\n\n`num_hidden_layers`\n\n`num_attention_heads`\n\n/ `num_key_value_heads`\n\n``` python\nimport requests\n\ndef arch_fingerprint(repo):\n    c = requests.get(f\"https://huggingface.co/{repo}/resolve/main/config.json\",\n                     headers={\"User-Agent\": \"genome/1.0\"}).json()\n    return {k: c.get(k) for k in\n            (\"model_type\", \"vocab_size\", \"hidden_size\",\n             \"intermediate_size\", \"num_hidden_layers\",\n             \"num_attention_heads\", \"num_key_value_heads\")}\n```\n\nThe shape tuple `(hidden_size, intermediate_size, num_hidden_layers, heads, kv)`\n\nis effectively a fingerprint of the reference architecture. When a model's tuple **matches a foreign open-weight exactly**, that is strong evidence the architecture was adopted rather than designed independently. Examples we measured:\n\n| Model | shape (h · i · L · heads · kv) | Exact match |\n|---|---|---|\n| a 7B commercial model | 3584 · 18944 · 28 · 28 · 4 | Qwen2.5-7B |\n| a 72B commercial model | 8192 · 29568 · 80 · 64 · 8 | Qwen2.5-72B |\n| a 14B VLM | 5120 · 17408 · 40 · 40 · 8 | Qwen3-14B |\n| an 8B model | 4096 · 14336 · 32 · 32 · 8 | Llama-3.1-8B |\n| a MoE model | 7168 · 18432 · 61 · (moe 2048) | DeepSeek-V3 |\n\nA single coincidental field means nothing; five simultaneously is a fingerprint.\n\nArchitecture alone can mislead. A model can copy a foreign *architecture* but train a genuinely new tokenizer, or vice-versa. The tokenizer is measured directly from `tokenizer.json`\n\n, comparing the vocabulary sets with a min-overlap ratio:\n\n``` python\ndef vocab_set(repo):\n    j = requests.get(f\"https://huggingface.co/{repo}/resolve/main/tokenizer.json\").json()\n    v = j[\"model\"][\"vocab\"]                      # BPE: {token: id}\n    return set(v.keys())\n\ndef tok_overlap(a, b):\n    A, B = vocab_set(a), vocab_set(b)\n    return len(A & B) / min(len(A), len(B))       # 1.0 == subset\n```\n\nThis immediately surfaces things `config`\n\nhides. One model matched **Qwen2.5-7B's architecture exactly**, yet its tokenizer overlapped Qwen by only ~0.38 — a **\"foreign brain, own language\"** case: the architecture was adopted, but a new Korean tokenizer was trained. Conversely, some VLMs reused a base tokenizer verbatim (overlap = 1.000), confirming a straight fine-tune.\n\nA practical trap:\n\n`min(|A|,|B|)`\n\nin the denominator (not the union) is what makes areducedvocabulary that is a strict subset of a larger one score ~1.0 — the correct signal for \"carved out of the base.\"\n\nThe gold-standard question is: **were the weights trained from scratch, or continued-pretrained on a foreign base?** This is where two instructive traps live.\n\nThe naive idea: load `embed_tokens.weight`\n\nfrom both models, and for shared tokens, average the row-wise cosine similarity. If they share lineage, embeddings should be similar.\n\nThey are not — *even when they obviously share lineage*. We measured near-zero mean cosine for **both** a known from-scratch model **and** a known Llama-derivative. The reason is **rotational invariance**: a Transformer's hidden space has no privileged basis, so two models can encode identical information under an arbitrary orthogonal rotation. Row-wise cosine sees rotation as dissimilarity. It cannot distinguish lineage.\n\n**Linear CKA (Centered Kernel Alignment)** is rotation- and isotropic-scale-invariant, so it is the right tool for comparing representations:\n\n``` python\nimport torch\n\ndef linear_cka(X, Y):\n    # X: (n, d1), Y: (n, d2) — SAME token order (shared vocab)\n    X = X - X.mean(0, keepdim=True)\n    Y = Y - Y.mean(0, keepdim=True)\n    num = (X.T @ Y).norm() ** 2\n    den = (X.T @ X).norm() * (Y.T @ Y).norm()\n    return (num / den).item()\n```\n\nA from-scratch model scored **near-zero CKA** against its candidate base — clean evidence of independent pretraining. But a continued-pretrained derivative scored only modestly higher (≈0.25) — barely above the baseline between two *unrelated* models of the same family (≈0.21). Large-scale training reshapes embeddings enough that CKA loses discriminative power on the *derivative* side.\n\n**Conclusion, stated honestly:** the weights axis reliably confirms *from-scratch* (near-zero), but it is **not** a strong detector of *derivation*. For that, `config`\n\n+ tokenizer fingerprints remain primary. We report the weights axis as supporting evidence, not as a verdict on its own.\n\nMost models declare a single attention mechanism. A few mix several. The count of distinct mechanisms in `config.json`\n\nis a cheap proxy for architectural originality:\n\n```\nKEYS = (\"layer_types\", \"linear_attn_config\", \"sliding_window\",\n        \"mamba2_d_state\", \"hyena_filter_order\", \"mla_kv_lora_rank\",\n        \"attention_cls\")\n\ndef attention_diversity(cfg):\n    hits = [k for k in KEYS if k in cfg]\n    # e.g. layer_types = [full×16, sliding×48] -> hybrid (2)\n    return hits\n```\n\nIn our sweep, most Korean models used a single grouped-query or multi-head-latent attention; a couple used a **hybrid** (`layer_types = [full_attention×16, sliding_attention×48]`\n\n); and the most diverse combined mamba2, hyena, MLA, linear attention, gated-delta-net, native-sparse-attention and sliding-window in one stack.\n\nWe collapse the two primary axes (architecture × weights) into one label:\n\n| Genotype | Architecture | Weights |\n|---|---|---|\n🟢 Native |\nself | from-scratch |\n🔵 Adapted |\nmostly self | one axis borrowed |\n🟡 Mixed |\npartial | partial inheritance |\n🔴 Ported |\nforeign (exact match) | inherited |\n\nThe tokenizer overlap and attention diversity are shown alongside, not folded into the verdict, so readers can see the raw evidence.\n\nApplying the identical pipeline to the public foundation models of **nine Korean organizations** (large enterprises, telcos, mid-size firms, and startups), the picture is **not uniform**: some models match a foreign architecture *and* tokenizer exactly (Ported); others use self-built architectures and weights with no foreign match (Native); many sit in between. The per-model breakdown — with a 3D lineage graph, search, and light/dark mode — is in the [Space](https://huggingface.co/spaces/mayafree/Model-Genome-Korea).\n\nThe three functions above are the whole method. Point them at any two repos on the Hub:\n\n```\nprint(arch_fingerprint(\"some/model\"))\nprint(tok_overlap(\"some/model\", \"Qwen/Qwen3-14B\"))\n# weights: load embed_tokens.weight for a shared-vocab pair, then linear_cka\n```\n\nLive demo, full dataset, and 3-language UI: ** Model Genome Korea**.\n\n*Model names, companies, and licenses are the property of their respective owners.*\n\nDNA lineage test of Korean LLM & VLM foundation models", "url": "https://wpnews.pro/news/model-genome-fingerprinting-whether-an-llm-was-trained-from-scratch-or-derived", "canonical_source": "https://huggingface.co/blog/mayafree/model-dna", "published_at": "2026-08-22 15:20:59+00:00", "updated_at": "2026-08-22 15:43:50.975783+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools"], "entities": ["Model Genome Korea", "LG K-EXAONE 2.0", "Qwen2.5-7B", "Qwen2.5-72B", "Qwen3-14B", "Llama-3.1-8B", "DeepSeek-V3", "Zhihu"], "alternates": {"html": "https://wpnews.pro/news/model-genome-fingerprinting-whether-an-llm-was-trained-from-scratch-or-derived", "markdown": "https://wpnews.pro/news/model-genome-fingerprinting-whether-an-llm-was-trained-from-scratch-or-derived.md", "text": "https://wpnews.pro/news/model-genome-fingerprinting-whether-an-llm-was-trained-from-scratch-or-derived.txt", "jsonld": "https://wpnews.pro/news/model-genome-fingerprinting-whether-an-llm-was-trained-from-scratch-or-derived.jsonld"}}