LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs

Muhammed Saeed and Simon Razniewski's LLMPEDIA, presented at AACL-IJCNLP 2026, found that only 68.4% of atomic claims drawn from roughly 1.3M articles materialized from the parametric memory of GPT-5-mini, DeepSeek-V3.2 and Llama-3.3-70B were supported when audited against Wikipedia and a curated web stack — more than 21 percentage points below MMLU scores above 90% — with 30.5% of claims insufficient. The live open encyclopedia at llmpedia.net lets visitors inspect each claim, page and verdict at a stable URL through five one-click views covering link traversal, claim-level factuality, cross-model and political-persona comparison, and guided topic drill-down.

Abstract Flagship language models appear saturated on benchmarks like MMLU Hendrycks et al., 2021 , scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the availability bias of fixed question sets. LLMPEDIA makes this bias measurable and browsable. We recursively materialized ∼1.3M articles from three model families’ parametric memory GPT-5-mini, DeepSeek-V3.2, Llama-3.3-70B without retrieval, then audited a stratified sample of atomic claims against Wikipedia and a curated web stack, coloring every claim supported, refuted, or insufficient Saeed and Razniewski, 2026 . On a uniform random sample the true rate is 68.4% - more than 21 pp below MMLU - with 30.5% of claims insufficient: assertions no benchmark probes and the world’s largest encyclopedia cannot adjudicate - long-tail knowledge or plausible hallucination, the evidence cannot tell - extending to free text the coverage gap GPTKB established for triples Hu et al., 2025 . The resulting live, open encyclopedia lets visitors inspect this frontier one claim at a time through five one-click views - link-traversal exploration, claim-level factuality, cross-model and political-persona comparison, and a guided topic drill-down - each page, claim, and verdict at a stable URL. LLMPEDIA is live at https://llmpedia.net; see the demonstration video. - Anthology ID: - 2026.aacl-demo.6 - Volume: - Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations https://aclanthology.org/volumes/2026.aacl-demo/ - Month: - November - Year: - 2026 - Address: - Hengqin, China - Editors: - Ningyu Zhang https://aclanthology.org/people/ningyu-zhang/ , Derry Wijaya https://aclanthology.org/people/derry-tanti-wijaya/ - Venues: - AACL https://aclanthology.org/venues/aacl/ | IJCNLP https://aclanthology.org/venues/ijcnlp/ - SIG: - Publisher: - Association for Computational Linguistics - Note: - Pages: - 42–49 - Language: - URL: - https://aclanthology.org/2026.aacl-demo.6/ https://aclanthology.org/2026.aacl-demo.6/ - DOI: - Cite ACL : - Muhammed Saeed and Simon Razniewski. 2026. LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs https://aclanthology.org/2026.aacl-demo.6/ . In Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations , pages 42–49, Hengqin, China. Association for Computational Linguistics. - Cite Informal : - LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs https://aclanthology.org/2026.aacl-demo.6/ Saeed & Razniewski, AACL-IJCNLP 2026 - PDF: - https://aclanthology.org/2026.aacl-demo.6.pdf https://aclanthology.org/2026.aacl-demo.6.pdf