Can LLMs Judge Research Novelty? Ai2 Finds Prompt Wording Flips the Verdict

If you use an LLM to judge whether a research idea is new, how much should you trust the answer? Ai2's new study says: less than you think. In a post on the Hugging Face blog on October 27, 2026, the team reports that small changes to the evaluation setup can produce very different verdicts on the same ideas, including a drop from 93.5 percent to 40.9 percent accuracy for GPT-5.4 from one prompt change. The work accompanies the paper Old Ideas, Novel Problems (arXiv 2610.02022), with data and code released.

TL;DR #

How the benchmark was built #

Instead of using paper acceptance as a stand-in for novelty, Ai2 mined reviewers' explicit statements about originality in ICLR 2026 reviews. Two pools resulted:

• High-novelty: 154 human ideas from accepted papers in the top 10 percent of their area, where reviewers praised novelty and none disputed it.
• Lower-novelty: 145 ideas from rejected papers in the bottom 10 percent, where reviewers raised originality concerns and none praised novelty.

From these the team made two settings. Human-Only pairs human ideas with stronger versus weaker novelty evidence. Human+Generated pairs the 154 validated human ideas with 154 ideas from a simple LLM generator with no search, tools, iteration or feedback. The generated ideas are treated as less novel, which Ai2 acknowledges is a weak label. Validation checks: an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases; an expert reviewed 30 Human+Generated pairs, weighted toward cases the strongest judge got wrong, preferred the human idea in all 30 and judged none of the generated ideas novel; and ideas were rewritten into a common structure to test for writing-style effects.

What they found #

Prompt wording dominates

For GPT-5.4 on Human+Generated pairs, accuracy fell from 93.5 percent to 40.9 percent, below chance, when the prompt dropped the note that reviewers found exactly one idea novel. The model changed more than half its verdicts. That is not a subtle effect: a sentence of context decides whether the judge is excellent or worse than a coin flip. On Human-Only, prompt changes shifted accuracy about 7 points for GPT-5.4 and 21 points for Claude Opus 4.6. Effects were larger on AI-generated ideas. There was no universal best prompt: removing an instruction against treating incremental work as novel helped some judges and hurt others.

AI-versus-AI comparisons are unstable

Comparing Claude Opus 4.5 with Sonnet 4.5, and GPT-5.4 with GPT-5.1, roughly a quarter to a third of verdicts changed across prompt variants. If you use an LLM judge to pick the better of two model-written ideas, the ranking may depend on how you phrased the instruction.

More compute did not reliably help

An agentic GPT-5.6 Sol judge with extra-high reasoning and live arXiv search scored 0.78 macro-F1, against 0.89 for Claude Opus 4.6 with standard retrieval, and cost roughly ten times more per decision. In the authors' words, "more search or compute does not necessarily produce more consistent judgments."

Purpose-built evaluators underperformed

| System (Human+Generated) | Macro-F1 | Total cost | 
|---|---|---|

| Claude Opus 4.6, low reasoning, majority vote of 3 | 0.854 | $2.59 | | Sakana AI Scientist evaluator (Claude Opus 4.6) | 0.772 | $82.05 | | Idea Novelty Checker (Claude Opus 4.6) | 0.642 | $30.66 |

A simple, cheap majority vote beat two heavier systems here. That does not mean those systems are bad at everything; it means that on this benchmark, complexity and cost did not buy accuracy.

Why this matters #

Novelty judging is moving into real workflows: screening submissions, ranking ideas from "AI scientist" systems, prioritising research directions, and filtering grant or paper pipelines. Each of those uses an LLM as a gate. If the gate's verdict depends on a sentence of prompt wording, then results and rankings can be an artifact of the prompt.

This sits within a broader pattern of evaluation fragility that we have covered, including our post on how a Claude-found enzyme claim fared under scrutiny, where "novel" was the contested word, and on Anthropic's auditing agents, where a cheap model matched a larger one on a judgment task. The common lesson is to measure your judge before you rely on it.

Practical takeaways for builders #

1. Treat LLM judges as instruments that need calibration. Build a small labelled set from your own domain and report the judge's accuracy on it before using it at scale.
2. Test prompt sensitivity. Run several semantically equivalent prompts. If verdicts move a lot, the judge is not measuring what you think.
3. Aggregate cheaply. A majority vote of three low-reasoning runs outperformed an expensive agentic system in this study, at a fraction of the cost.
4. Do not over-trust high accuracy on human-written data. Judges that look reliable on human ideas may behave differently on AI-generated ones.
5. Log prompts with results. A verdict without the exact prompt is not reproducible.

If you build evaluation pipelines for agents, see also our look at how a cheap model can match a larger one on audit tasks.

A worked example of how a judge can drift #

Imagine you ask a model to compare two research ideas and say which is more novel. In the first prompt you add a note that exactly one of the ideas was found novel by human reviewers. The model now has a strong cue: its job is to pick the one that looks more original, so it focuses on differences in framing and ambition. In the second prompt you drop that note. The model is no longer told that a difference exists, so it may call both ideas incremental, or treat both as novel, and its choice between them becomes close to arbitrary.

That is a plausible mechanism for a collapse from 93.5 percent to 40.9 percent, though the post reports the numbers, not a proven cause. The point for practitioners is that the instruction carries a lot of the information the judge uses. If your real-world prompt lacks that cue, accuracy measured with the cue will overstate performance.

What the results do not say #

• They do not say LLMs cannot judge novelty at all. The best settings reached macro-F1 around 0.85 to 0.89, which is useful if you can control the prompt.
• They do not say human reviewers are consistent. Peer review has well-documented noise, and the benchmark inherits labels from reviewers' comments.
• They do not say expensive agentic systems are always worse, only that on this benchmark the added search and compute did not reliably improve consistency.
• They do not evaluate novelty in the sense of historical priority across all literature; the ground truth is what ICLR reviewers wrote.

Questions for follow-up work #

1. Would a calibration step, such as running paraphrased prompts and abstaining when they disagree, recover reliability?
2. How do stronger idea generators, with search and iteration, change the Human+Generated results?
3. Do specialised fields with smaller literatures behave differently from machine-learning venues?
4. Can judges be trained on the released data, and does that generalise or just memorise reviewer phrasing?

The released dataset and code make these questions testable by anyone. For teams running automated research pipelines, the sensible near-term move is conservative: use the judge to rank candidates for human review rather than to reject ideas outright, and keep a record of which prompt produced which decision so that later audits are possible.

Limits of the study #

• Labels for AI-generated ideas are weak assumptions, not verified judgments for each idea.
• The expert check covered only 30 pairs, chosen to be difficult.
• The review-mining pipeline was checked on only 50 cases.
• The data cover ICLR 2026 papers and one simple generator, so results may not transfer to other venues, fields or stronger idea-generation agents.
• The numbers are Ai2's; independent replication has not been reported.

How to try it #

Ai2 links the dataset at allenai/novelty-judge-bench and code at github.com/allenai/novelty-eval. A reasonable first step is to run your current judge prompt and two paraphrases against the Human-Only split and compare accuracy. If the spread is large, fix the prompt and aggregation before using the judge for decisions.

Bottom line #

LLMs can sound confident about whether an idea is new, but this study shows the confidence is fragile. A single sentence can swing a strong model from near-perfect to worse than chance. Until judges are calibrated and tested against prompt changes, treat novelty verdicts as a signal to check, not a ruling.

Related reading #

Figures are from Ai2's October 27, 2026 post and paper and are accurate as of that date.