{"slug": "ai2-releases-a-benchmark-showing-research-novelty-judges-swing-with-prompts", "title": "Ai2 releases a benchmark showing research novelty judges swing with prompts", "summary": "A single prompt change moved GPT-5.4's accuracy at picking the more novel research idea from 93.5% to 40.9%, according to a new benchmark released by Ai2 researcher Noy Sternlicht and five co-authors. The Ai2 results show that small prompt variations can reverse language-model novelty judgments, meaning novelty scores need testing across setups before they guide AI-assisted research.", "body_md": "# Ai2 releases a benchmark showing research novelty judges swing with prompts\n\n**Researcher Noy Sternlicht and co-authors found one prompt change pushed GPT-5.4 from 93.5% accuracy to 40.9% on AI-generated idea comparisons.**\n\n        By [Ryan Merket](https://runtimewire.com/author/ryan-merket)\n        · Published \n\nPrimary source: [Hugging Face Newsroom](https://huggingface.co/blog/allenai/novelty-bench)\n\n## Why it matters\n\nAI-for-science systems may use language models to rank ideas as original. Ai2's results show that small prompt changes can reverse those judgments, so novelty scores need testing across setups before they guide research.\n\nA small prompt change sent [GPT-5.4](https://runtimewire.com/models/openai/gpt-5.4)'s accuracy at choosing the more novel research idea from 93.5% to 40.9%, according to a [new study](https://huggingface.co/blog/allenai/novelty-bench?ref=runtimewire) by [Ai2](https://allenai.org/?ref=runtimewire) researcher [Noy Sternlicht](https://noy-sternlicht.github.io/?ref=runtimewire) and five co-authors. The result puts a weak link in AI-assisted science under scrutiny: systems asked to generate research ideas may also rely on another language model to certify that those ideas are original.…\n\nWhen you purchase through links in our articles, [we may earn a small commission](https://runtimewire.com/affiliate-monetization-standards). This doesn’t affect our editorial independence.", "url": "https://wpnews.pro/news/ai2-releases-a-benchmark-showing-research-novelty-judges-swing-with-prompts", "canonical_source": "https://runtimewire.com/article/ai2-novelty-judge-benchmark-prompt-instability", "published_at": "2026-10-27 17:48:53+00:00", "updated_at": "2026-10-27 21:12:58.532370+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-safety", "artificial-intelligence"], "entities": ["Ai2", "Noy Sternlicht", "GPT-5.4", "OpenAI", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai2-releases-a-benchmark-showing-research-novelty-judges-swing-with-prompts", "markdown": "https://wpnews.pro/news/ai2-releases-a-benchmark-showing-research-novelty-judges-swing-with-prompts.md", "text": "https://wpnews.pro/news/ai2-releases-a-benchmark-showing-research-novelty-judges-swing-with-prompts.txt", "jsonld": "https://wpnews.pro/news/ai2-releases-a-benchmark-showing-research-novelty-judges-swing-with-prompts.jsonld"}}