Ai2 releases a benchmark showing research novelty judges swing with prompts

A single prompt change moved GPT-5.4's accuracy at picking the more novel research idea from 93.5% to 40.9%, according to a new benchmark released by Ai2 researcher Noy Sternlicht and five co-authors. The Ai2 results show that small prompt variations can reverse language-model novelty judgments, meaning novelty scores need testing across setups before they guide AI-assisted research.

Ai2 releases a benchmark showing research novelty judges swing with prompts Researcher Noy Sternlicht and co-authors found one prompt change pushed GPT-5.4 from 93.5% accuracy to 40.9% on AI-generated idea comparisons. By Ryan Merket https://runtimewire.com/author/ryan-merket · Published Primary source: Hugging Face Newsroom https://huggingface.co/blog/allenai/novelty-bench Why it matters AI-for-science systems may use language models to rank ideas as original. Ai2's results show that small prompt changes can reverse those judgments, so novelty scores need testing across setups before they guide research. A small prompt change sent GPT-5.4 https://runtimewire.com/models/openai/gpt-5.4 's accuracy at choosing the more novel research idea from 93.5% to 40.9%, according to a new study https://huggingface.co/blog/allenai/novelty-bench?ref=runtimewire by Ai2 https://allenai.org/?ref=runtimewire researcher Noy Sternlicht https://noy-sternlicht.github.io/?ref=runtimewire and five co-authors. The result puts a weak link in AI-assisted science under scrutiny: systems asked to generate research ideas may also rely on another language model to certify that those ideas are original.… When you purchase through links in our articles, we may earn a small commission https://runtimewire.com/affiliate-monetization-standards . This doesn’t affect our editorial independence.