An arXiv paper titled "How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation" (submitted 15 Jun 2026) introduces SearchGEO, a controlled evaluation framework for measuring endorsement corruption in LLM-based web-search agents, per the paper. The authors, led by Yimeng Chen and five co-authors, evaluate 13 LLM backends on 308 cases each. The paper reports overall attack success rate (ASR) varying across backends, from 0.0% on Claude-Sonnet-4.6 to 31.4% on Gemini-3-Flash. The authors describe a five-mode attack taxonomy, a web-evidence manipulation pipeline, multiple output-level metrics, and an auxiliary agent-skill probe that, the paper reports, shows a sharp split: Claude "over-rejects" while GPT "over-trusts." The paper argues recommendation reliability under adversarial search content should be a first-class safety evaluation dimension.
What happened
The arXiv paper "How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation" was submitted on 15 Jun 2026, by Yimeng Chen and five co-authors, per the arXiv entry. The paper introduces SearchGEO, a controlled evaluation framework combining a web-evidence manipulation pipeline, a five-mode attack taxonomy, and multiple output-level metrics, according to the abstract. Per the paper, the authors evaluate 13 LLM backends on 308 cases each and report attack success rate (ASR) variation across backends, with ASR from 0.0% on Claude-Sonnet-4.6 to 31.4% on Gemini-3-Flash.
Technical details (reported)
According to the paper's abstract, SearchGEO operationalizes endorsement corruption by manipulating web evidence visible to a retrieval step and measuring when agent outputs convert attacker-published pages into endorsed claims. The authors report a five-mode attack taxonomy and multiple output-level metrics; they also run an auxiliary "agent-skill probe" where endorsement is framed as an install command. The paper reports that the probe reveals a sharp split among backends, characterizing some backends as "over-rejecting" (Claude) and others as "over-trusting" (GPT), per the abstract.
Technical context
Systematic, pipeline-level evaluations like SearchGEO provide a reproducible way to quantify how retrieval plus LLM synthesis can amplify maliciously crafted web content. Comparable efforts in adversarial retrieval and prompt-injection research typically find model-dependent sensitivity; the reported ASR spread across 13 backends aligns with that pattern. For practitioners, measuring endorsement at the output level complements token-level robustness tests and is especially relevant for systems that synthesize multi-source evidence into recommendations.
Context and significance
The paper places "recommendation reliability under adversarial search content" as a measurable safety dimension. For safety teams and product engineers, the reported per-backend differences and the auxiliary probe suggest that evaluations which combine manipulated retrieval results with LLM response scoring can reveal practical failure modes not visible in isolated model tests. The presence of a clear taxonomy and metrics in SearchGEO could enable standardized comparisons across research and vendor backends.
What to watch
Reproducibility of the reported ASR numbers across independent implementations, the paper's released code and datasets, and follow-up evaluations that include more diverse retrieval stacks and real-world web dynamics. Observers should also watch for vendor responses, external replication studies, and incorporation of endorsement-corruption metrics into third-party benchmarks.
Scoring Rationale #
SearchGEO presents a reproducible, multi-backend measurement framework for a real-world safety concern - endorsement corruption in retrieval-augmented LLM agents. The 13-backend evaluation and clear taxonomy make this notable for security and safety practitioners, though results are from a single preprint with no independent replication yet.
Practice with real Ad Tech data
90 SQL & Python problems · 15 industry datasets
[Active Search Campaigns by BudgetEasy](/problems/sql/active-search-campaigns-by-budget)
[High CPC Clicks & Poor Landing PagesMedium](/problems/sql/high-cpc-clicks-poor-landing-page)
[Campaign ROAS by Attribution ModelHard](/problems/sql/campaign-roas-by-attribution-model)
250 free problems · No credit card