# ChatGPT Scores and Novelty Words: Do Abstracts Game Evaluation? > Source: > Published: 2026-10-25 19:12:58+00:00 ## The Short Answer Explicit novelty claims in research abstracts are examined for an association with higher ChatGPT evaluation scores, because the model is scoring based on titles and abstracts. The key concern is whether wording nudges scoring independent of the underlying research quality. For practitioners, this means LLM-based evaluation of paper quality can be biased by presentation choices in abstracts, so novelty language may change rankings when models are used as first-pass filters. The caveat is that this focuses on evaluation from text (not verification of methods or results), so any “score advantage” from novelty phrasing should be interpreted as a text-signal effect, not proof of genuine novelty. ChatGPT Scores and Novelty Words: Do Abstracts Game Evaluation? ## Introduction If you’ve ever written (or skimmed) a research abstract, you know the genre: you summarise your work, claim what’s new, and try to sound confident without sounding careless. Now here’s the uncomfortable question raised by new research: **are explicit novelty claims in research abstracts associated with higher ChatGPT scores?** This post is based on a new study by Kayvan Kousha and Mike Thelwall, published on arXiv ([original paper](https://arxiv.org/abs/2610.26947)). The authors look at whether ChatGPT’s scoring of research quality from **titles and abstracts** reacts to “this is new” style wording—not just to what the research actually did. The bigger context is research assessment. Large language models like ChatGPT are increasingly used (or at least explored) as fast, scalable tools for evaluating large numbers of papers. But if a model is swayed by wording choices, that could distort what “quality” even means in automated scoring. ## Why This Matters This matters *right now* because research evaluation is under pressure to handle more outputs with fewer expert hours. And titles/abstracts are exactly what most AI systems ingest—partly because full texts can be hard to access and partly because abstracts are compact signals. So if the abstract’s phrasing can nudge AI scores, evaluators could end up rewarding “novelty language” rather than novelty itself. A real-world scenario: imagine a funding body or university committee that uses an LLM as a first-pass filter. If abstracts that use explicit novelty phrases systematically score higher, those papers could rise to the top earlier in the pipeline—meaning humans review fewer papers, but the “top” list is partly shaped by rhetoric. That doesn’t mean novelty doesn’t exist in those papers; it means the *selection pressure* shifts. This study also builds on earlier AI-for-evaluation findings. Prior work has shown that LLM review scores can be sensitive to **presentation-only changes** (including reframing novelty) and that abstracts can be strong enough to drive scores without the model “verifying” anything against the full article. Kousha and Thelwall take that concern and scale it up: instead of small experiments or conference papers, they examine **99,245** UK REF 2021 journal articles across the broad spectrum of disciplines. ## How the Study Tested “Novelty Claims” in Abstracts The core idea is simple: compare ChatGPT and evaluation proxies for papers whose abstracts contain explicit novelty-claim phrases versus papers whose abstracts don’t. ### What data did they use? They used journal articles submitted to the UK’s **REF 2021** (Research Excellence Framework). The dataset contains **99,245 unique journal articles**. Importantly, the same article can contribute to multiple groups because the analysis is structured by REF panels and **Units of Assessment (UoAs)**. Because scoring availability varies, sample sizes differ slightly by analysis: - **ChatGPT overall-quality indicator:** available for **98,820** records - **ChatGPT-`-5-mini` structured scores (originality, significance, rigour):** **99,249** article–panel records - **Rank-gain analysis vs REF departmental proxy:** **106,051** article–UoA records (That “article counted separately per panel/UoA” detail matters for how to interpret totals.) ### What were the “novelty claims”? The authors didn’t just look for the word “novel.” Instead, they built a set of candidate phrases indicating explicit novelty, focusing on **three-word sequences** (3-grams) from abstracts. Because there are *many* possible 3-grams, they: 1. Extracted frequent 3-grams across abstracts (with heuristically chosen cutoffs), 2. Used an LLM (`Gemma 3 12B`) to decide whether each phrase is a likely novelty indicator, 3. Manually checked the `Gemma` results by both authors, 4. Kept only phrases classified as clear novelty indicators by both authors. That final list contained **282 accepted novelty phrases**. So in this study, a “novelty claim” means: **the abstract contains an explicit linguistic cue associated with novelty claims**, not necessarily that the research is genuinely novel. ### What did ChatGPT score? ChatGPT scoring was done from **titles and abstracts only**, using zero-shot prompts aligned to REF 2021 criteria. They used two kinds of outputs: - **ChatGPT overall-quality scores**, averaged across three models: `ChatGPT-4o`, `ChatGPT-4o mini`, and `ChatGPT-5 mini` - **ChatGPT structured scores** from `ChatGPT-5 mini`: **originality**, **significance**, **rigour**, plus overall quality ### What did they use as an evaluation baseline? REF individual article scores are not available publicly, so they used a proxy: - **REF departmental average quality score** for the journal outputs submitted by the relevant department(s) So the baseline is **department-level** rather than true article-level expert judgments. This limitation is crucial: the study can detect association patterns with scoring proxies, but it can’t definitively prove what happens to *individual* articles in human peer review. ## What the Researchers Actually Found About ChatGPT Scores The headline result is straightforward: **abstracts containing explicit novelty-claim phrases tended to score higher in ChatGPT**. ### Overall quality: higher mean scores when novelty phrases appear Out of the **98,820** records with an overall-quality indicator: - **17,257 (17.5%)** abstracts contained at least one novelty phrase - **81,563 (82.5%)** abstracts did not Across **all four REF main panels**, abstracts with novelty phrases received higher **mean ChatGPT overall-quality scores**. The differences were statistically significant and ranged from: - **0.080 points** in Panel A - up to **0.106 points** in Panel C ### Which quality dimension benefits most: originality Here’s where the story becomes more interesting. They compared differences in the `ChatGPT-5 mini` structured scores for: - **originality** - **significance** - **rigour** Across all four REF panels, the **originality score difference** was larger than the significance and rigour differences. To make that concrete, their mean differences (novelty-phrase abstracts minus non-novelty-phrase abstracts) included values like: - **Panel A:** originality **0.116**, significance **0.023**, rigour **0.002** - **Panel B:** originality **0.103**, significance **0.027**, rigour **−0.009** - **Panel C:** originality **0.133**, and both significance and rigour also positive but smaller - **Panel D:** originality **0.074**, significance **0.053**, rigour **0.037** This pattern suggests that the novelty phrases are most strongly associated with higher **originality** scoring, not with a uniform lift across every dimension. ### Discipline-level check: the effect shows up in most UoAs The authors also drilled down across **34 UoAs**. In **30 out of 34** broadly field-based UoAs, the originality difference (novelty phrase vs none) was larger than the corresponding differences for significance and rigour. A few examples the paper calls out: - UoA 3 (Allied Health Professions, Dentistry, Nursing and Pharmacy): originality **0.186** vs significance **0.080** and rigour **0.049** - UoA 12 (Engineering): originality **0.183** vs significance **0.082** and rigour **0.037** - UoA 24 (Sport and Exercise Sciences, Leisure and Tourism): originality **0.227** vs significance **0.130** and rigour **0.053** The exceptions were concentrated in Panel D (arts and humanities), where the novelty-phrase counts were often smaller—making uncertainty larger. ## ChatGPT vs REF Proxy: Do Novelty Phrases Improve Relative Ranking? Association with absolute scores is one thing. Ranking relative to an evaluation baseline is another. ### How ranking was computed They converted both: - ChatGPT score (for the article/panel) - REF departmental proxy score into **percentile ranks within each UoA**. Then they computed: **ChatGPT rank gain = ChatGPT percentile rank − REF proxy percentile rank** So a positive number means: ChatGPT rated that abstract/paper relatively better than the department-average proxy. ### What happened? Across **34 UoAs**, **ChatGPT rank gain** was higher for abstracts with novelty phrases in **33 out of 34** UoAs, and the differences were statistically significant in **26** of them. In one specific UoA example from the paper: - Clinical Medicine: novelty-phrase abstracts (**1,582**) had mean rank **5.160** non-novelty-phrase abstracts (**8,020**) had mean rank **4.731** mean rank difference **0.429**, **p < .001** ### Why this still doesn’t prove “gaming” Even with this relative ranking advantage, the study still can’t prove causation. Why? Because novelty phrases might correlate with real originality. Also, because the baseline is departmental (not individual article) scoring, the relationship could reflect department-level abstract-writing styles or selection patterns. Still, the mechanism the authors worry about remains plausible: **ChatGPT only sees titles and abstracts**, and it likely can’t verify novelty claims against the full research. ## A Deeper Look: More Novelty “Families” Means Higher Originality Scores The paper also tested whether abstracts that contain multiple kinds of novelty language score even higher. They grouped the novelty phrases into **families**, like: - “firstness” (e.g., “for the first”) - “novel/innovation” wording - “new contribution” wording - other novelty-related phrasing Across the **17,347** abstracts containing at least one accepted novelty phrase: - **1,381 (8.0%)** contained phrases from **two or more** novelty families The results: - abstracts with **one** family scored higher than abstracts with **none** - abstracts with **two or more** families scored **highest of all** in every REF panel Again, this could mean either: 1. ChatGPT is sensitive to the rhetoric (language signal effect), or 2. multiple novelty claims reflect richer or more genuinely original research (real novelty effect) And because the study uses text-only scoring, it can’t separate those possibilities cleanly. ## Phrase-Level Effects: Some Wording Hits Harder Than Others To see whether certain phrases correlate with especially large originality-score boosts, the authors compared, for each frequent phrase, the mean originality score for abstracts that include that phrase versus abstracts that include *other* accepted novelty phrases. They looked at phrases appearing in at least **100** abstracts and displayed the **20 largest positive differences**. Some examples with notably higher mean originality-score differences included phrases like: - “report the first” (+**0.348**) - “present the first” (+**0.280**) - “a novel mechanism” (+**0.279**) - “provide the first” (+**0.267**) Interestingly, the most frequent phrases—like: - “the first time” (**N=2,153**) (+** 0.106**) - “for the first” (**N=2,119**) (+** 0.094**) were associated with smaller originality-score differences than the top-boost phrases. The authors interpret this carefully: it could be actual differences in research topics and wording context, not only the phrases themselves. ## What This Means for Using LLMs in Research Assessment If you’re using LLMs (or thinking about it), the main implication is: **explicit novelty language can be a scoring shortcut**—whether or not it matches underlying contribution strength. This is a practical policy question. For example: - Should novelty-related wording be downweighted or controlled for? - Should evaluation prompts explicitly instruct the model to discount unsupported rhetorical cues? - Should abstract-only scoring be treated as a *screening heuristic*, not a proxy for originality? Also, the study strengthens the argument that evaluation pipelines relying on abstracts should incorporate robustness checks. If the scoring is sensitive to linguistic packaging, evaluators may need to measure how stable scores are under paraphrasing or reframing. For readers interested in the larger context of how ChatGPT evaluates research quality, the study discusses and aligns with related findings from this line of work, including how abstract stylistic features can “fool” AI scoring systems (see the broader discussion linked from [the original paper](https://arxiv.org/abs/2610.26947)). ## Key Takeaways - **Abstracts containing explicit novelty-claim phrases scored higher in ChatGPT** across all four REF main panels (overall-quality differences ranged from**0.080 to 0.106** points). - The association was **strongest for ChatGPT’s originality dimension** , not for significance or rigour (e.g., Panel A originality difference**0.116** vs significance**0.023** vs rigour**0.002** ). - In **33 of 34 UoAs** , novelty-phrase abstracts received**higher ChatGPT rank gain** relative to the REF departmental proxy (significant in**26 UoAs** ). - **More types of novelty wording in the same abstract** (two or more novelty “families”) were associated with**even higher originality scores** . - The study **does not prove causation** : novelty phrases might reflect real novelty, and the REF baseline used here is a**departmental average proxy** , not individual article expert scores. - For real-world assessment systems: treat abstract-based LLM scoring as **potentially language-sensitive** , and consider calibration/controls if originality is a major decision criterion. If you want, I can also rewrite this into a “policy checklist” style guide for universities or funding agencies deciding whether (and how) to use LLM scores in evaluation. ## Sources Used This article is a plain-English breakdown of the following peer-reviewed preprint. Read the original for full methodology and results: - [Are explicit novelty claims in research abstracts associated with higher ChatGPT scores?](https://arxiv.org/abs/2610.26947) — arXiv - Authors: Authors: Kayvan Kousha, Mike Thelwall