Full Fact Tests 5 AI Chatbots, Logs 7,625 Answers [2026]

A 70-day test of five major AI chatbots found that models from OpenAI, Anthropic, Google, DeepSeek, and xAI routinely handed out outdated, poorly sourced, or flat-out wrong answers, often in the same confident tone they use when they are right. The study, published by the UK fact-checking charity Full Fact and reported by The Independent, asked ChatGPT, Claude, Gemini, DeepSeek, and Grok the same 22 questions every day for ten weeks, generating more than 7,600 individual responses to pick apart.

The findings land at an awkward moment for an industry that has spent 2026 pitching chatbots and AI agents as reliable research assistants, customer-service reps, and even verification layers for other software. Full Fact’s researchers did not produce a single headline error rate. What they did produce is a pattern: models citing numbers that expired months earlier, naming the wrong head of government, and answering ambiguous questions as if only one interpretation existed, all delivered with the same even, authoritative tone.

What Full Fact Actually Tested #

Full Fact is not a university lab or a Silicon Valley red-teaming shop. It is a London-based charity best known for debunking political claims during UK election cycles. For this project, researchers built a repeatable routine: the same 22 questions, submitted to the same five chatbots, every single day, for 70 days starting in mid-July 2026. That design matters. A one-off accuracy test can be skewed by a lucky or unlucky day. A ten-week drumbeat of identical questions shows whether a model’s errors are random noise or a recurring habit.

The questions covered current events, UK government policy figures, and factual claims that change over time, the kind of queries a journalist, student, or customer-support bot might realistically face. Full Fact’s team then extracted what they call atomic facts from each response, breaking long chatbot answers into individually checkable claims, before scoring each one for accuracy, consistency, and timeliness. That process pulled 74,538 discrete, checkable facts out of the 7,625 total responses.

One number from the report has already caused confusion online: Full Fact says 97.8 percent of its own information-extraction decisions were correct. That figure describes how reliably Full Fact’s team coded and categorized the chatbot answers, not how often the chatbots themselves were right. Conflating the two would understate the problem the report actually documents. Full Fact’s published findings do not reduce down to one tidy chatbot accuracy percentage, and the organization is explicit that no single number captures what went wrong.

The Specific Errors That Surfaced #

Rather than a scoreboard, Full Fact’s report reads like a catalog of failure modes, and several examples named specific models. DeepSeek stated on September 15, 2026 that Keir Starmer was still UK prime minister, even though Andy Burnham had taken over in July. That is not a subtle factual nuance. It is the kind of error a human assistant would be fired for making twice.

Gemini had its own stumble: on August 4, 2026, it gave an incorrect figure of 12,210 migrant arrivals for the date in question. Multiple models, including Gemini and Claude, also repeated an outdated weekly asylum-support figure of £8.86, a number that had actually risen to £9.95 back in June 2025. More than a year after that rate changed, chatbots were still citing the old one as current fact.

Full Fact grouped the recurring problems into four categories: outdated information presented as current, failure to account for regional differences within the UK, incorrect or unsupported calculations, and overly confident answers to questions that actually have more than one reasonable interpretation. None of these are exotic hallucinations about invented facts. They are the mundane, predictable failures of a system that does not reliably know when its training data has gone stale.

Confidence Without Accuracy: The Core Warning #

The most quoted line from the research is not about any single error. It is about tone. Chris Morris, chief executive of Full Fact, told reporters that the chatbots respond with the same assured delivery whether their answer is accurate, outdated, or describing something that has not happened yet, as detailed in reporting from The Independent. “Our findings show the answer comes back in the same confident tone whether it is correct, out of date, or describes something that has not yet happened,” Morris said.

Full Fact’s own report distills that same point into a blunter line, available on the organization’s benchmark report page: “The common thread is that a model’s confidence is no guide to its reliability.” That is the sentence most worth sitting with. Users have learned to treat hedging language, phrases like “I’m not sure” or “this may be outdated”, as a cue to double-check. Chatbots rarely hedge. They answer a question about a government minister with the same cadence they use to explain the water cycle, leaving users with no built-in signal for when to be skeptical.

This isn’t a new theoretical risk. Shattered.io has previously covered how a fabricated Claude AI tip about a murder sat unflagged for 72 days before anyone caught the error, and how Google’s Gemini Argon model was caught fabricating emails while still ranking third on a separate agent benchmark. The pattern Full Fact documents, polished delivery regardless of underlying accuracy, is the same mechanism behind both incidents.

Model-by-Model: What We Know and What We Don’t #

Here is where the study gets frustrating for anyone hoping for a league table. Full Fact’s publicly available material does not provide a complete, comparable error-rate percentage for each of the five chatbots, and no outlet covering the research, including the Ukrainian tech outlet dev.ua and the Greek tech site Techblog.gr, has published one either. What exists instead is a set of named, dated examples tied to specific models.

The absence of a named error for ChatGPT and Grok in the available coverage does not mean those two models performed flawlessly. It most likely means reporters and the available excerpts of the report highlighted the most vivid examples rather than listing every slip for every model. Readers should treat the table above as a record of what has been documented publicly, not a verdict that two of the five chatbots passed with a clean sheet.

Why This Study Lands Differently Than Past Chatbot Criticism #

Complaints about chatbot hallucinations are old news by late 2026. What sets Full Fact’s approach apart is the methodology: a fixed set of questions asked daily, for ten straight weeks, across five competing models simultaneously. Most accuracy critiques circulating online are built from scattered, one-off screenshots. Full Fact instead built something closer to a longitudinal clinical trial, which is exactly why the lack of a single clean percentage is more credible, not less. Real-world chatbot failure is messy and context-dependent, and a study that tried to compress it into one number would likely be oversimplifying in the other direction.

It also arrives alongside a broader string of 2026 stories about AI reliability gaps. The UK’s AI Security Institute recently tested more than 55 AI models and flagged breach-related risks across the board, while separate reporting found AI watermarking schemes actually weakened safety guardrails in six of seven models tested. Full Fact’s work adds a third data point to a pattern: independent testers keep finding gaps between how AI vendors describe their models’ reliability and what repeated, structured testing actually shows.

Market Impact: Why Developers and Enterprises Should Care #

For software engineers building on top of these models, the Full Fact findings are less a surprise than a quantified confirmation of what retrieval-augmented generation and grounding pipelines already try to fix. If a flagship model can state a factually wrong figure for migrant arrivals or misname a sitting prime minister weeks after a known change, any production system that pipes raw chatbot output to end users without a verification layer is exposed to the same failure mode. That is a direct argument for RAG architectures, citation-forcing prompts, and live-data grounding rather than relying on a model’s internal training cutoff for anything time-sensitive. There is also a trust dimension that goes beyond engineering. OpenAI and Meta have spent much of 2026 pushing AI agents into consumer and enterprise workflows even as surveys show an 85.5 percent trust gap between how capable agents are and how much users actually trust them to act autonomously. A study showing that the underlying chatbots can’t reliably track a change of prime minister two months after it happened does not help close that gap. Enterprises evaluating AI deployments for anything touching regulatory figures, pricing, or compliance thresholds now have a named, citable study to point to when arguing for human review checkpoints.

The timing also overlaps with a wave of chatbot impersonation and phishing activity. Security researchers have already documented fake ad portals impersonating ChatGPT, Gemini, and Claude to steal MFA codes from users who assume anything that looks and sounds like a major chatbot must be legitimate. Full Fact’s research reinforces the same underlying lesson from a different angle: the smooth, confident voice of a chatbot is not evidence of anything, whether the content came from the real model or a scam clone of it.

Historical Context: A Recurring Industry Problem #

Chatbot factual accuracy has been a known weak point since the first wave of consumer-facing LLMs shipped in 2022 and 2023. Early complaints focused on outright hallucination, models inventing citations, case law, or historical events that never happened. By 2025 and into 2026, as retrieval tools and web-browsing plugins became standard, the conversation shifted toward a subtler failure: models that can access current information but still default to stale training data, or that answer ambiguous questions as though there were only one correct reading.

Full Fact’s test captures that second-generation problem precisely. None of its documented examples, the outdated benefit figure, the wrong migrant-arrivals count, the stale prime minister, involve a model inventing something from nothing. Each is a case of a system failing to update, failing to flag uncertainty, or failing to recognize that a question had more than one valid answer. That is arguably a harder problem to engineer around than hallucination, because it requires models to reason about their own knowledge gaps rather than simply avoid fabrication.

Competitive Comparison: How the Five Chatbots Stack Up #

Because Full Fact did not publish a complete per-model scorecard, a true head-to-head ranking isn’t possible from the public material alone. What can be compared is each company’s broader 2026 positioning on accuracy and safety claims, set against what the study actually documented.

The irony in that table is hard to miss. Gemini and Grok both lean on live-data access as a selling point, yet Gemini is the model with a specifically dated, specifically wrong statistic in Full Fact’s findings. Grounding in live search does not automatically mean a model synthesizes that data correctly, and a web-connected chatbot that misreads its own sources can still produce a confidently wrong answer.

What Full Fact Recommends Developers Do Next #

Beyond cataloging errors, Full Fact’s broader message to AI developers is about verification against live, authoritative sources rather than relying on a model’s static training data for facts that change. That’s a familiar engineering recommendation, but the study gives it fresh weight by showing the gap persists across five different vendors with five different architectures and five different claimed approaches to grounding.

For teams building chatbot-adjacent products, the practical takeaway is to treat any time-sensitive factual claim, a benefit rate, a government minister’s name, a statistic tied to a specific date, as a candidate for mandatory retrieval rather than model memory. That applies whether you’re building on ChatGPT, Claude, Gemini, DeepSeek, or Grok, since the study found comparable classes of error across the set rather than isolating the problem to one vendor.

Predictions: Where This Goes From Here #

• Expect at least one of the five companies named in the study to publicly respond with its own accuracy benchmark data by early November 2026, since silence on a named, dated error (like DeepSeek’s prime-minister mistake) tends to invite follow-up questions from reporters.
• Full Fact is likely to publish a second report in this series given the “report-one” naming convention on its benchmark page, probably expanding the question set or adding a direct per-model comparison that this first report avoided.
• Enterprise AI procurement teams will increasingly request documentation of grounding and freshness guarantees as a contract condition, following the same trajectory as security questionnaires a decade ago.
• Regulatory attention on AI-generated misinformation, already building in the UK and EU, will cite studies like this one as evidence that voluntary self-reporting on accuracy isn’t sufficient, strengthening the case for mandated third-party testing.
• More fact-checking and watchdog organizations outside the UK will likely replicate Full Fact’s daily-question methodology for their own national contexts, since the approach is cheap to run and produces continuously refreshable headlines.

The Bigger Picture for AI Trust #

None of this means chatbots are useless for factual questions. Full Fact’s own framing stops well short of that conclusion, and the four failure categories it identifies, staleness, regional blindness, calculation errors, and overconfident single-answer responses to ambiguous questions, are all addressable engineering problems rather than fundamental limits of the technology. But the study is a useful corrective to a year of AI companies marketing their chatbots as increasingly trustworthy general-purpose assistants. A model that can’t track a change in national leadership two months after the fact, while answering with total confidence, is not ready to be anyone’s unsupervised source of truth.

The takeaway: confidence and correctness are two different things, and right now, five of the industry’s biggest chatbots are better at projecting the first than guaranteeing the second.

Frequently Asked Questions #

What is the Full Fact AI chatbot study?

It’s a research project by the UK fact-checking charity Full Fact that tested ChatGPT, Claude, Gemini, DeepSeek, and Grok by asking the same 22 questions daily for 70 days, starting in mid-July 2026, producing more than 7,600 responses.

Did Full Fact publish an overall accuracy percentage for the chatbots?

No. Full Fact reported that 97.8 percent of its own fact-extraction decisions were correct, but that figure measures the research team’s accuracy in coding responses, not how often the chatbots themselves were right. No overall chatbot error rate was published.

Which chatbot performed worst in the study?

The publicly available report does not rank the five models against each other. It does name DeepSeek as incorrectly stating Keir Starmer was still prime minister weeks after Andy Burnham took over, and names Gemini and Claude for repeating an outdated weekly asylum-support figure.

What types of errors did Full Fact find most often?

Four recurring categories: outdated information presented as current, failure to account for regional differences within the UK, incorrect or unsupported calculations, and overly confident answers to questions with more than one reasonable interpretation.

Why does chatbot “confidence” matter if the answer is wrong?

Full Fact’s chief executive, Chris Morris, said the models respond in the same assured tone regardless of whether an answer is accurate, outdated, or describes something that hasn’t happened. Without a built-in uncertainty signal, users have no easy cue to double-check a wrong answer.

Does this mean AI chatbots shouldn’t be used for factual research?

Not necessarily. It means time-sensitive facts, like government figures, benefit rates, or anything tied to a specific date, should be verified against live, authoritative sources rather than taken at face value from a chatbot’s training data.

How is this different from past hallucination complaints about AI?

Earlier criticism focused on chatbots inventing facts outright. Full Fact’s examples are mostly about models failing to update stale information or recognize ambiguity, a subtler and arguably harder problem than fabrication from nothing.

Will Full Fact test more chatbots or publish follow-up reports?

The report is labeled “report one” on Full Fact’s benchmark page, suggesting additional reports in the series are planned, though no specific publication date has been confirmed.