{"slug": "ai-citation-test-finds-source-order-matters-less-than-it-looks", "title": "AI Citation Test Finds Source Order Matters Less Than It Looks", "summary": "A September 14 arXiv preprint by Sriram Selvam and Anneswa Ghosh found that the raw citation advantage of the top search result over the fifth — 85.1% versus 42.8%, a 42.3 percentage-point gap — largely disappears when matched sources are swapped, with a follow-up set of 56 pairs estimating a 0.0-point effect (95% CI -5.4 to +5.4). The non-peer-reviewed study replayed 129 saved conversations from one GPT-5.4 search agent using Exa across 113 matched page pairs, and found that rewriting a page with headings and lists added an average of 0.50 more citation markers per answer (95% CI 0.20 to 0.84) and raised the likelihood of being cited by 4.5 percentage points (95% CI -1.4 to +10.4). The authors conclude that raw position averages are unreliable and that structured formatting concentrates citation credit on the rewritten page without increasing total citations per answer.", "body_md": "A [new preprint](https://arxiv.org/abs/2609.15164) tested whether tweaking just one part of a source changes AI search citations, with everything else kept the same. In the raw numbers, the top result got cited about twice as often as the fifth. When researchers Sriram Selvam and Anneswa Ghosh reversed the order of matched sources, the impact was much smaller and measured zero in a follow-up test.\n\nPosted to arXiv on September 14, the paper isn’t peer-reviewed. The study covers one GPT-5.4 search agent that uses Exa as its search provider, featuring offline replayed conversations and no live webpage edits.\n\n## How The Test Worked\n\nThe researchers prompted the GPT-5.4 agent to answer 130 common questions by having it perform independent web searches. They recorded every message and search result from the 129 questions it addressed. From these transcripts, they chose pairs of pages that appeared in the same search outcomes and were both screened as supporting the same fact. This screening aimed to find situations where either page could be fairly cited. When true matches were identified, any credit differences were due to how the model apportioned recognition between the two sources, both confirming the same fact.\n\nThat left 113 pairs. A later blinded human check confirmed 103 of them as genuine matches. The researchers replayed each saved conversation four ways, placing one page above or below the other and showing its text either as plain paragraphs or rewritten with headings and lists or a table. Only the final answer was generated again.\n\nBoth versions of the text were generated by AI rewrites of the original page. Grok 4.3 created nearly all of them, with GPT-5.4 used as a fallback for one pair, and a separate Grok review checked that the facts matched. The wording varies between the two versions, so the authors note that the test compares two rewrites but does not specifically isolate formatting differences.\n\n## Raw Position Gap Was Larger Than Swap Effects\n\nIn the initial position of a search call, pages were cited 85.1% of the time in saved transcripts, compared to 42.8% for pages in the fifth position. This creates a difference of 42.3 percentage points.\n\nHere, ‘position’ simply means the order of the five Exa results returned in a single search, not where a page ranks on Google or its position on the live web.\n\nThe study highlights that search providers usually put more relevant pages at the top, so the raw difference reflects both the position and the quality of the pages. When the researchers moved the same page higher within its pair, the chance it was cited at all went up by 7.9 percentage points. However, this finding wasn’t considered statistically significant after accounting for multiple tests.\n\nAnother testing set with 56 pairs, where only the order was switched, showed an estimate of 0.0 points, with a 95% confidence interval from -5.4 to +5.4.\n\nThe [study](https://arxiv.org/html/2609.15164v1) explains that the raw gap and the swap results measure different aspects. Overall, it suggests that position did influence citations in some cases, but averages from raw position data aren’t reliable.\n\n## Structured Rewrites Got More Credit, Not Clearer Entry\n\nPages that were rewritten with headings and lists received an average of 0.50 more citation markers per answer compared to the same pages written as plain paragraphs, with a 95% confidence interval from 0.20 to 0.84. The answers in the test were heavily cited, with a median of 29 markers across six documents.\n\nThe total number of citations per answer didn’t rise, and the number on the other page barely changed. The authors see this as credit being focused more on the rewritten page.\n\nThe main test the researchers conducted, which they planned before starting the experiment, was to see if the page got cited at all. They found that using structured text increased that likelihood by 4.5 percentage points, with a 95% interval from -1.4 to +10.4. The paper points out that this result isn’t conclusive and mentions that the study could reliably detect only effects of about 8.5 points or more.\n\nA more strict comparison, where every word stayed the same but the layout was adjusted to one sentence per list row, boosted citation rates across all 113 pairs. When they repeated the test with a subset, the effect reversed.\n\nIn the discussion section of the [paper](https://arxiv.org/html/2609.15164v1), the authors shared these insights:\n\n“This is an attribution-sensitivity warning, not an optimization tactic.”\n\n## Reruns Changed Citation Outcomes\n\nThe researchers tested 120 responses again using the same inputs, and found that the decision to cite or not for the target page changed in 15% of these cases, roughly one in seven.\n\nThe average count effect remained consistent across these reruns. They estimate that about 45% of the variation in a single run’s effect is due to model randomness.\n\nThe authors recommend rerunning citation tests multiple times and sharing how consistent the results are across those runs.\n\nAdditionally, SparkToro [reported in January](https://www.searchenginejournal.com/ai-recommendations-change-with-nearly-every-query-sparktoro/566242/) that ChatGPT and Google’s AI Overviews each produced the same brand list less than 1% of the time when given the same prompt repeatedly.\n\n## Why This Matters\n\nThe raw position gap in this test was much larger than the average effect observed when researchers swapped source order. An [Ahrefs report](https://www.searchenginejournal.com/schema-markup-didnt-move-ai-citations-in-ahrefs-test/574568/) from May showed pages cited by AI were about three times more likely to include JSON-LD schema, but adding schema didn’t clearly increase citations.\n\nThis raises questions about whether a correlation in a vendor report or your tracking was ever tested by changing the variable, and a single answer is a weak basis for labeling a citation as won or lost.\n\nThe study can’t confirm if reformatting a live page boosts citations, since rewrites only applied to text already retrieved, excluding crawling, retrieval, and ranking processes.\n\n## Looking Ahead\n\nThe researchers re-ran the saved searches on Grok 4.3, discovering that the structured rewrites leaned the same way. However, less than half of Grok’s first replies followed the correct citation format.\n\nThe authors recommend more research to test each scenario a few times, explore different search providers and models, and pay attention to both how often citations occur and if a page is cited at all.\n\n*Featured Image: Accogliente Design/Shutterstock*", "url": "https://wpnews.pro/news/ai-citation-test-finds-source-order-matters-less-than-it-looks", "canonical_source": "https://www.searchenginejournal.com/ai-citation-test-finds-source-order-matters-less-than-it-looks/589806/", "published_at": "2026-09-17 17:02:58+00:00", "updated_at": "2026-09-17 17:25:37.282450+00:00", "lang": "en", "topics": ["generative-engine-optimization", "ai-search", "ai-agents", "ai-research", "structured-data"], "entities": ["Sriram Selvam", "Anneswa Ghosh", "arXiv", "GPT-5.4", "Exa", "Grok 4.3"], "alternates": {"html": "https://wpnews.pro/news/ai-citation-test-finds-source-order-matters-less-than-it-looks", "markdown": "https://wpnews.pro/news/ai-citation-test-finds-source-order-matters-less-than-it-looks.md", "text": "https://wpnews.pro/news/ai-citation-test-finds-source-order-matters-less-than-it-looks.txt", "jsonld": "https://wpnews.pro/news/ai-citation-test-finds-source-order-matters-less-than-it-looks.jsonld"}}