NoveltyScope: A Multi-Agent Research Assistant for Evidence-Grounded Novelty Assessment of Scientific Papers

A prototype called NoveltyScope, described in arXiv paper 2610.22703v1, assesses the novelty of scientific papers by comparing them against retrieved prior work and assigning a five-level novelty score. On 50 class-balanced RINoBench samples, the multi-agent scoring configuration achieved a mean absolute error of 1.26 with Gemini and 1.16 with Qwen3, versus 1.48 and 1.22 for single-LLM scoring and 1.68 for a vector-similarity baseline. The system uses BM25 and dense retrieval with reciprocal rank fusion over a closed local corpus, with four scoring roles — Prosecutor, Defender, Judge and Critic — and at most one revision of the Judge's score.

arXiv:2610.22703v1 Announce Type: cross Abstract: NoveltyScope is a prototype that compares a scientific paper with retrieved prior work and produces a structured report and a five-level novelty score. The system extracts the core task and at most two author-claimed contributions, then searches a closed local corpus using BM25 and dense retrieval with reciprocal rank fusion. Contribution comparisons request paired passages for potential refutations, and quotations are checked against cached source text. Four scoring roles Prosecutor, Defender, Judge, and Critic use the resulting evidence, with at most one revision of the Judge's score. Evaluation uses 50 class-balanced RINoBench samples whose idea-level labels are retained for full-paper input. Mean absolute error is 1.48 and 1.26 for single-LLM and multi-agent scoring with Gemini, respectively, and 1.22 and 1.16 with Qwen3. The vector-similarity baseline has an error of 1.68. A HiCLIP case study illustrates the contribution comparisons and paired evidence passages. Overall, the results demonstrate the feasibility of evidence-grounded novelty assessment and show that the multi-agent configurations achieve lower scoring errors than their single-LLM counterparts across both evaluated models.