{"slug": "llmpedia-browsing-verifying-and-comparing-the-parametric-encyclopedic-knowledge", "title": "LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs", "summary": "Muhammed Saeed and Simon Razniewski's LLMPEDIA, presented at AACL-IJCNLP 2026, found that only 68.4% of atomic claims drawn from roughly 1.3M articles materialized from the parametric memory of GPT-5-mini, DeepSeek-V3.2 and Llama-3.3-70B were supported when audited against Wikipedia and a curated web stack — more than 21 percentage points below MMLU scores above 90% — with 30.5% of claims insufficient. The live open encyclopedia at llmpedia.net lets visitors inspect each claim, page and verdict at a stable URL through five one-click views covering link traversal, claim-level factuality, cross-model and political-persona comparison, and guided topic drill-down.", "body_md": "##### Abstract\n\nFlagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the availability bias of fixed question sets. LLMPEDIA makes this bias measurable and browsable. We recursively materialized ∼1.3M articles from three model families’ parametric memory (GPT-5-mini, DeepSeek-V3.2, Llama-3.3-70B) without retrieval, then audited a stratified sample of atomic claims against Wikipedia and a curated web stack, coloring every claim supported, refuted, or insufficient (Saeed and Razniewski, 2026). On a uniform random sample the true rate is 68.4% - more than 21 pp below MMLU - with 30.5% of claims insufficient: assertions no benchmark probes and the world’s largest encyclopedia cannot adjudicate - long-tail knowledge or plausible hallucination, the evidence cannot tell - extending to free text the coverage gap GPTKB established for triples (Hu et al., 2025). The resulting live, open encyclopedia lets visitors inspect this frontier one claim at a time through five one-click views - link-traversal exploration, claim-level factuality, cross-model and political-persona comparison, and a guided topic drill-down - each page, claim, and verdict at a stable URL. LLMPEDIA is live at https://llmpedia.net; see the demonstration video.\n- Anthology ID:\n- 2026.aacl-demo.6\n- Volume:\n- [Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations](https://aclanthology.org/volumes/2026.aacl-demo/)\n- Month:\n- November\n- Year:\n- 2026\n- Address:\n- Hengqin, China\n- Editors:\n- [Ningyu Zhang](https://aclanthology.org/people/ningyu-zhang/) ,[Derry Wijaya](https://aclanthology.org/people/derry-tanti-wijaya/)\n- Venues:\n- [AACL](https://aclanthology.org/venues/aacl/) |[IJCNLP](https://aclanthology.org/venues/ijcnlp/)\n- SIG:\n- Publisher:\n- Association for Computational Linguistics\n- Note:\n- Pages:\n- 42–49\n- Language:\n- URL:\n- [https://aclanthology.org/2026.aacl-demo.6/](https://aclanthology.org/2026.aacl-demo.6/)\n- DOI:\n- Cite (ACL):\n- Muhammed Saeed and Simon Razniewski. 2026. [LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs](https://aclanthology.org/2026.aacl-demo.6/) . In*Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations* , pages 42–49, Hengqin, China. Association for Computational Linguistics.\n- Cite (Informal):\n- [LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs](https://aclanthology.org/2026.aacl-demo.6/) (Saeed & Razniewski, AACL-IJCNLP 2026)\n- PDF:\n- [https://aclanthology.org/2026.aacl-demo.6.pdf](https://aclanthology.org/2026.aacl-demo.6.pdf)", "url": "https://wpnews.pro/news/llmpedia-browsing-verifying-and-comparing-the-parametric-encyclopedic-knowledge", "canonical_source": "https://aclanthology.org/2026.aacl-demo.6/", "published_at": "2026-11-02 00:00:00+00:00", "updated_at": "2026-11-05 02:19:50.478301+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-safety", "generative-ai"], "entities": ["LLMPEDIA", "Muhammed Saeed", "Simon Razniewski", "GPT-5-mini", "DeepSeek-V3.2", "Llama-3.3-70B", "Wikipedia", "AACL-IJCNLP 2026"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llmpedia-browsing-verifying-and-comparing-the-parametric-encyclopedic-knowledge", "markdown": "https://wpnews.pro/news/llmpedia-browsing-verifying-and-comparing-the-parametric-encyclopedic-knowledge.md", "text": "https://wpnews.pro/news/llmpedia-browsing-verifying-and-comparing-the-parametric-encyclopedic-knowledge.txt", "jsonld": "https://wpnews.pro/news/llmpedia-browsing-verifying-and-comparing-the-parametric-encyclopedic-knowledge.jsonld"}}