ChatGPT Scores and Novelty Words: Do Abstracts Game Evaluation?

The Short Answer #

Explicit novelty claims in research abstracts are examined for an association with higher ChatGPT evaluation scores, because the model is scoring based on titles and abstracts. The key concern is whether wording nudges scoring independent of the underlying research quality.

For practitioners, this means LLM-based evaluation of paper quality can be biased by presentation choices in abstracts, so novelty language may change rankings when models are used as first-pass filters. The caveat is that this focuses on evaluation from text (not verification of methods or results), so any “score advantage” from novelty phrasing should be interpreted as a text-signal effect, not proof of genuine novelty.

ChatGPT Scores and Novelty Words: Do Abstracts Game Evaluation?

Introduction #

If you’ve ever written (or skimmed) a research abstract, you know the genre: you summarise your work, claim what’s new, and try to sound confident without sounding careless. Now here’s the uncomfortable question raised by new research: are explicit novelty claims in research abstracts associated with higher ChatGPT scores? This post is based on a new study by Kayvan Kousha and Mike Thelwall, published on arXiv (original paper). The authors look at whether ChatGPT’s scoring of research quality from titles and abstracts reacts to “this is new” style wording—not just to what the research actually did.

The bigger context is research assessment. Large language models like ChatGPT are increasingly used (or at least explored) as fast, scalable tools for evaluating large numbers of papers. But if a model is swayed by wording choices, that could distort what “quality” even means in automated scoring.

Why This Matters #

This matters right now because research evaluation is under pressure to handle more outputs with fewer expert hours. And titles/abstracts are exactly what most AI systems ingest—partly because full texts can be hard to access and partly because abstracts are compact signals. So if the abstract’s phrasing can nudge AI scores, evaluators could end up rewarding “novelty language” rather than novelty itself.

A real-world scenario: imagine a funding body or university committee that uses an LLM as a first-pass filter. If abstracts that use explicit novelty phrases systematically score higher, those papers could rise to the top earlier in the pipeline—meaning humans review fewer papers, but the “top” list is partly shaped by rhetoric. That doesn’t mean novelty doesn’t exist in those papers; it means the selection pressure shifts.

This study also builds on earlier AI-for-evaluation findings. Prior work has shown that LLM review scores can be sensitive to presentation-only changes (including reframing novelty) and that abstracts can be strong enough to drive scores without the model “verifying” anything against the full article. Kousha and Thelwall take that concern and scale it up: instead of small experiments or conference papers, they examine 99,245 UK REF 2021 journal articles across the broad spectrum of disciplines.

How the Study Tested “Novelty Claims” in Abstracts #

The core idea is simple: compare ChatGPT and evaluation proxies for papers whose abstracts contain explicit novelty-claim phrases versus papers whose abstracts don’t.

What data did they use?

They used journal articles submitted to the UK’s REF 2021 (Research Excellence Framework). The dataset contains 99,245 unique journal articles. Importantly, the same article can contribute to multiple groups because the analysis is structured by REF panels and Units of Assessment (UoAs).

Because scoring availability varies, sample sizes differ slightly by analysis:

• ChatGPT overall-quality indicator: available for 98,820 records

• ChatGPT--5-mini structured scores (originality, significance, rigour): 99,249 article–panel records

• Rank-gain analysis vs REF departmental proxy: 106,051 article–UoA records

(That “article counted separately per panel/UoA” detail matters for how to interpret totals.)

What were the “novelty claims”?

The authors didn’t just look for the word “novel.” Instead, they built a set of candidate phrases indicating explicit novelty, focusing on three-word sequences (3-grams) from abstracts.

Because there are many possible 3-grams, they:

1. Extracted frequent 3-grams across abstracts (with heuristically chosen cutoffs),

2. Used an LLM (Gemma 3 12B) to decide whether each phrase is a likely novelty indicator,

3. Manually checked the Gemma results by both authors,

4. Kept only phrases classified as clear novelty indicators by both authors.

That final list contained 282 accepted novelty phrases.

So in this study, a “novelty claim” means: the abstract contains an explicit linguistic cue associated with novelty claims, not necessarily that the research is genuinely novel.

What did ChatGPT score?

ChatGPT scoring was done from titles and abstracts only, using zero-shot prompts aligned to REF 2021 criteria.

They used two kinds of outputs:

• ChatGPT overall-quality scores, averaged across three models: ChatGPT-4o, ChatGPT-4o mini, and ChatGPT-5 mini

• ChatGPT structured scores from ChatGPT-5 mini: originality, significance, rigour, plus overall quality

What did they use as an evaluation baseline?

REF individual article scores are not available publicly, so they used a proxy:

• REF departmental average quality score for the journal outputs submitted by the relevant department(s)

So the baseline is department-level rather than true article-level expert judgments.

This limitation is crucial: the study can detect association patterns with scoring proxies, but it can’t definitively prove what happens to individual articles in human peer review.

What the Researchers Actually Found About ChatGPT Scores #

The headline result is straightforward: abstracts containing explicit novelty-claim phrases tended to score higher in ChatGPT.

Overall quality: higher mean scores when novelty phrases appear

Out of the 98,820 records with an overall-quality indicator:

• 17,257 (17.5%) abstracts contained at least one novelty phrase

• 81,563 (82.5%) abstracts did not Across all four REF main panels, abstracts with novelty phrases received higher mean ChatGPT overall-quality scores. The differences were statistically significant and ranged from:

• 0.080 points in Panel A

• up to 0.106 points in Panel C

Which quality dimension benefits most: originality

Here’s where the story becomes more interesting. They compared differences in the ChatGPT-5 mini structured scores for:

• originality

• significance

• rigour

Across all four REF panels, the originality score difference was larger than the significance and rigour differences.

To make that concrete, their mean differences (novelty-phrase abstracts minus non-novelty-phrase abstracts) included values like:

• Panel A: originality 0.116, significance 0.023, rigour 0.002

• Panel B: originality 0.103, significance 0.027, rigour −0.009

• Panel C: originality 0.133, and both significance and rigour also positive but smaller

• Panel D: originality 0.074, significance 0.053, rigour 0.037

This pattern suggests that the novelty phrases are most strongly associated with higher originality scoring, not with a uniform lift across every dimension.

Discipline-level check: the effect shows up in most UoAs

The authors also drilled down across 34 UoAs.

In 30 out of 34 broadly field-based UoAs, the originality difference (novelty phrase vs none) was larger than the corresponding differences for significance and rigour.

A few examples the paper calls out:

• UoA 3 (Allied Health Professions, Dentistry, Nursing and Pharmacy): originality 0.186 vs significance 0.080 and rigour 0.049

• UoA 12 (Engineering): originality 0.183 vs significance 0.082 and rigour 0.037

• UoA 24 (Sport and Exercise Sciences, Leisure and Tourism): originality 0.227 vs significance 0.130 and rigour 0.053

The exceptions were concentrated in Panel D (arts and humanities), where the novelty-phrase counts were often smaller—making uncertainty larger.

ChatGPT vs REF Proxy: Do Novelty Phrases Improve Relative Ranking? #

Association with absolute scores is one thing. Ranking relative to an evaluation baseline is another.

How ranking was computed

They converted both:

• ChatGPT score (for the article/panel)
• REF departmental proxy score

into percentile ranks within each UoA. Then they computed:

ChatGPT rank gain = ChatGPT percentile rank − REF proxy percentile rank

So a positive number means: ChatGPT rated that abstract/paper relatively better than the department-average proxy.

What happened?

Across 34 UoAs, ChatGPT rank gain was higher for abstracts with novelty phrases in 33 out of 34 UoAs, and the differences were statistically significant in 26 of them.

In one specific UoA example from the paper:

• Clinical Medicine: novelty-phrase abstracts (1,582) had mean rank 5.160

  non-novelty-phrase abstracts (8,020) had mean rank 4.731 mean rank difference 0.429, p < .001

Why this still doesn’t prove “gaming”

Even with this relative ranking advantage, the study still can’t prove causation. Why? Because novelty phrases might correlate with real originality. Also, because the baseline is departmental (not individual article) scoring, the relationship could reflect department-level abstract-writing styles or selection patterns.

Still, the mechanism the authors worry about remains plausible: ChatGPT only sees titles and abstracts, and it likely can’t verify novelty claims against the full research.

A Deeper Look: More Novelty “Families” Means Higher Originality Scores #

The paper also tested whether abstracts that contain multiple kinds of novelty language score even higher.

They grouped the novelty phrases into families, like:

• “firstness” (e.g., “for the first”)

• “novel/innovation” wording

• “new contribution” wording

• other novelty-related phrasing Across the 17,347 abstracts containing at least one accepted novelty phrase:

• 1,381 (8.0%) contained phrases from two or more novelty families

The results:

• abstracts with one family scored higher than abstracts with none

• abstracts with two or more families scored highest of all in every REF panel

Again, this could mean either:

1. ChatGPT is sensitive to the rhetoric (language signal effect), or

2. multiple novelty claims reflect richer or more genuinely original research (real novelty effect)

And because the study uses text-only scoring, it can’t separate those possibilities cleanly.

Phrase-Level Effects: Some Wording Hits Harder Than Others #

To see whether certain phrases correlate with especially large originality-score boosts, the authors compared, for each frequent phrase, the mean originality score for abstracts that include that phrase versus abstracts that include other accepted novelty phrases.

They looked at phrases appearing in at least 100 abstracts and displayed the 20 largest positive differences. Some examples with notably higher mean originality-score differences included phrases like:

- “report the first” (+**0.348**)

- “present the first” (+**0.280**)

- “a novel mechanism” (+**0.279**)

- “provide the first” (+**0.267**)

Interestingly, the most frequent phrases—like:

- “the first time” (**N=2,153**) (+** 0.106**)

- “for the first” (**N=2,119**) (+** 0.094**)

were associated with smaller originality-score differences than the top-boost phrases.

The authors interpret this carefully: it could be actual differences in research topics and wording context, not only the phrases themselves.

What This Means for Using LLMs in Research Assessment #

If you’re using LLMs (or thinking about it), the main implication is: explicit novelty language can be a scoring shortcut—whether or not it matches underlying contribution strength. This is a practical policy question. For example:

• Should novelty-related wording be downweighted or controlled for?

• Should evaluation prompts explicitly instruct the model to discount unsupported rhetorical cues?

• Should abstract-only scoring be treated as a screening heuristic, not a proxy for originality?

Also, the study strengthens the argument that evaluation pipelines relying on abstracts should incorporate robustness checks. If the scoring is sensitive to linguistic packaging, evaluators may need to measure how stable scores are under paraphrasing or reframing.

For readers interested in the larger context of how ChatGPT evaluates research quality, the study discusses and aligns with related findings from this line of work, including how abstract stylistic features can “fool” AI scoring systems (see the broader discussion linked from the original paper).

Key Takeaways #

• Abstracts containing explicit novelty-claim phrases scored higher in ChatGPT across all four REF main panels (overall-quality differences ranged from0.080 to 0.106 points).
• The association was strongest for ChatGPT’s originality dimension , not for significance or rigour (e.g., Panel A originality difference0.116 vs significance0.023 vs rigour0.002 ).
• In 33 of 34 UoAs , novelty-phrase abstracts receivedhigher ChatGPT rank gain relative to the REF departmental proxy (significant in26 UoAs ).
• More types of novelty wording in the same abstract (two or more novelty “families”) were associated witheven higher originality scores .
• The study does not prove causation : novelty phrases might reflect real novelty, and the REF baseline used here is adepartmental average proxy , not individual article expert scores.
• For real-world assessment systems: treat abstract-based LLM scoring as potentially language-sensitive , and consider calibration/controls if originality is a major decision criterion.

If you want, I can also rewrite this into a “policy checklist” style guide for universities or funding agencies deciding whether (and how) to use LLM scores in evaluation.

Sources Used #

This article is a plain-English breakdown of the following peer-reviewed preprint. Read the original for full methodology and results:

• Are explicit novelty claims in research abstracts associated with higher ChatGPT scores? — arXiv
• Authors: Authors: Kayvan Kousha, Mike Thelwall

source & further reading

thepromptindex.com — original article Governance-First AI GPT for EU AI Act SME Compliance (TESSA) Memory-Aware Rewarding: Train LLMs to Use Your ChatGPT Memories Tool Selection in Tool-Heavy Agents: Train Tiny LLMs to Pick the Right API