Rethinking the Evaluation of LLM Fact-Checkers

A new Japanese benchmark, JSFactCheckBench, specifies all three design decisions that prior fact-checking benchmarks leave implicit, using 1,913 claims drawn from LLM answers to questions about misleading information circulating on Japanese social media, each annotated for factuality at a reference time and for importance to the question against preselected Web sources. Across six LLMs, the researchers found every model verifies claims that directly answer the question less accurately than supplementary claims, a gap aggregate scores obscure, and scores also fall when models must locate relevant content in longer or multiple sources and, for all four open models, when the surrounding question and response are included. The authors argue a benchmark that leaves these conditions implicit cannot reveal such weaknesses.

arXiv:2610.30504v1 Announce Type: new Abstract: Large language models LLMs are widely used to answer factual questions, but their answers can contain errors and must be fact-checked claim by claim. Yet the resulting score depends on three design decisions: which questions the claims answer, under what conditions each claim is judged, and which sources count as evidence. No existing benchmark specifies all three. We introduce JSFactCheckBench, a Japanese benchmark that specifies all three, with 1,913 claims from LLM answers to questions about misleading information that circulated on Japanese social media, each annotated for factuality at a reference time and for importance to the question, using evidence from preselected Web sources. Across six LLMs, all verify claims directly answering the question less accurately than supplementary claims, a gap that aggregate scores obscure. Scores also drop when models must locate the relevant content in longer or multiple sources and, for all four open models, when the surrounding question and response are added. These weaknesses are visible only because the benchmark records the conditions behind each label. A benchmark that leaves them implicit cannot reveal them.