Abstract
Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the availability bias of fixed question sets. LLMPEDIA makes this bias measurable and browsable. We recursively materialized ∼1.3M articles from three model families’ parametric memory (GPT-5-mini, DeepSeek-V3.2, Llama-3.3-70B) without retrieval, then audited a stratified sample of atomic claims against Wikipedia and a curated web stack, coloring every claim supported, refuted, or insufficient (Saeed and Razniewski, 2026). On a uniform random sample the true rate is 68.4% - more than 21 pp below MMLU - with 30.5% of claims insufficient: assertions no benchmark probes and the world’s largest encyclopedia cannot adjudicate - long-tail knowledge or plausible hallucination, the evidence cannot tell - extending to free text the coverage gap GPTKB established for triples (Hu et al., 2025). The resulting live, open encyclopedia lets visitors inspect this frontier one claim at a time through five one-click views - link-traversal exploration, claim-level factuality, cross-model and political-persona comparison, and a guided topic drill-down - each page, claim, and verdict at a stable URL. LLMPEDIA is live at https://llmpedia.net; see the demonstration video.
- Anthology ID:
- 2026.aacl-demo.6
- Volume:
- Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations
- Month:
- November
- Year:
- 2026
- Address:
- Hengqin, China
- Editors:
- [Ningyu Zhang](https://aclanthology.org/people/ningyu-zhang/) ,[Derry Wijaya](https://aclanthology.org/people/derry-tanti-wijaya/)
- Venues:
- [AACL](https://aclanthology.org/venues/aacl/) |[IJCNLP](https://aclanthology.org/venues/ijcnlp/)
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 42–49
- Language:
- URL:
- [https://aclanthology.org/2026.aacl-demo.6/](https://aclanthology.org/2026.aacl-demo.6/)
- DOI:
- Cite (ACL):
- Muhammed Saeed and Simon Razniewski. 2026. LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs . InProceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations , pages 42–49, Hengqin, China. Association for Computational Linguistics.
- Cite (Informal):
- [LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs](https://aclanthology.org/2026.aacl-demo.6/) (Saeed & Razniewski, AACL-IJCNLP 2026)
- PDF:
- [https://aclanthology.org/2026.aacl-demo.6.pdf](https://aclanthology.org/2026.aacl-demo.6.pdf)