NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

A new benchmark called NovGauge, built from 619 paper pairs and 50 multi-paper sets drawn from ICLR reviewer overlap claims and survey co-citations, finds that large language models remain unreliable at assessing scientific novelty. Evaluating 18 LLMs across task, problem, and method dimensions, the benchmark's cascading diagnostic pipeline recorded hallucination rates from 0% to 39%, and among non-hallucinated correct-positive judgments over 70% cited evidence that failed to logically support the stated reason. GPT-5.5 was the best performer at 43-72% Verified F1 across dimensions, while most models retained less than half of their raw F1 after faithfulness verification.

arXiv:2609.11234v1 Announce Type: new Abstract: Large language models LLMs are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.