{"slug": "i-ran-my-own-analysis-past-a-second-model-five-times-i-retracted-twice", "title": "I Ran My Own Analysis Past a Second Model Five Times. I Retracted Twice.", "summary": "A data analyst who ran their own analysis past a second AI model five times retracted twice, with the second model conceding once, and a question that survived all five rounds was settled in 90 seconds by a chart. The method required each side to name what would change its mind and treated messages as documents, not chat, to ensure both models argued about the same object.", "body_md": "# I Ran My Own Analysis Past a Second Model Five Times. I Retracted Twice.\n\nA working method for adversarial review between two AI models on your own data, and what it cost me in retractions. The cheapest instrument settled a question five rounds of argument could not.\n\n## Why this matters\n\nI gave one AI model my analytics analysis and asked a second model to attack it. We went five rounds. I retracted twice, the second model conceded once, and the question that survived all of it was settled in ninety seconds by the cheapest chart on the page. The value was not the second opinion. It was being forced to write down what would prove me wrong before I looked.\n\nI had a piece of analysis I liked. It explained a strange result in my own analytics, it was internally consistent, and it made my work look good. So I handed it to a different AI model and asked that model to take it apart.\n\nWe went five rounds. I retracted twice. The second model conceded once. And the question that survived all five rounds got answered in about ninety seconds by a chart I had been scrolling past the entire time.\n\nThis post is about the method, because the method transfers. The specific finding is a separate post.\n\n## The setup\n\nOne model builds the analysis. A different model, in a separate session, receives the analysis as a document and is instructed to falsify it. I carry the messages between them and run any test either one proposes.\n\nTwo rules make this work rather than turning into two chatbots agreeing with each other.\n\n**Each side must name what would change its mind.** Not “I disagree,” but “the following measurement would make me wrong.” A round that produces no falsifiable statement is a wasted round.\n\n**The messages are documents, not chat.** Every number carries its instrument, its window, and the date it was read. Both models then argue about the same object rather than about their memories of it.\n\n## What actually happened, round by round\n\n**Round one.** I sent my analysis with first-party data attached. Three of the other model’s premises were falsified by that data. It accepted all three without hedging. That was the first sign this was worth continuing, because a model that just agrees is useless and a model that just defends is worse.\n\n**Round two.** It came back with a real objection: I had treated a small exported dataset as a complete census, and the vendor’s own documentation describes that export as a sample. It was right. I had built an argument on top of “there are only 26 of these” when the documentation said the number was a sample of an unknown larger population. Retraction one.\n\n**Round three.** I queried the underlying report endpoint directly and it returned a total record count matching the export. So the small number was not a display truncation, and my retraction had been slightly too generous. Correcting your own correction is unglamorous and it is part of the method.\n\n**Round four.** I found a new field in the data that appeared to explain everything at once, and I wrote it up as a mechanism. The other model refused it. Not the observation, the causal direction. The vendor documents that field as a way of grouping results after the fact, and warns that it can be imprecise for small or new categories. So my data was consistent with two very different worlds, and I could not tell you which one I was in. Retraction two.\n\n**Round five.** I read the daily chart. The metric had been near zero for twelve consecutive days, jumped to a large number overnight, then held flat for eleven days with no weekly variation at all. Human behaviour does not have that shape. Something switched on. Every explanation both models had built across four rounds was an explanation of a rate, and the thing in front of us was not a rate.\n\n## The lesson I did not expect\n\nFour rounds of sophisticated argument, two retractions, one direct API query, and a search through vendor documentation. The thing that reframed all of it was the default chart on the report’s landing page, which I had looked at and not read.\n\nI had been treating a total as the finding and the shape as decoration. That is exactly backwards. A total tells you how much. A shape tells you what kind of process produced it, and what kind of process it is determines whether the total means anything at all.\n\nCheap instruments first, then. Before you commission the study, look at what is already on the screen. This is the same discipline I apply to generated code, where a build exit code outranks a confident summary: see [how I verify AI-generated code before shipping](/blog/ai-code-verification-evidence-based).\n\n## Where two models beat one\n\nBeing wrong out loud, repeatedly, in front of something that would not let it go.\n\nWorking alone, I would have shipped round one. It was coherent, it was supported by real data, and it was flattering. The second model did not know more than I did about my own site. What it had was a job: find the load-bearing assumption and push on it. That is a role, not an intelligence advantage, and roles are cheap to assign. Which model fills the role matters less than the role itself, though it is not nothing: I wrote about [how I route work between Fable 5 and Opus 5](/blog/claude-fable-5-vs-opus-5) elsewhere.\n\nThere is one failure mode worth naming. Two models will happily converge on a fluent, agreeable, wrong answer, because both are optimised to sound reasonable. The counter is the falsification rule. If neither side can state the measurement that would end the disagreement, the conversation has stopped being an investigation and has become prose.\n\n## Running it yourself\n\n- Write the analysis as a document, with every number carrying its instrument, its time window, and the date you read it.\n- Hand it to a second model with one instruction: find the load-bearing assumption and try to falsify it.\n- Carry the reply back yourself. Do not let either side summarise the other.\n- Every round, both sides state what evidence would change their position. No falsifiable statement means the round produced nothing.\n- When a proposed test is cheap, run it before arguing about it.\n- Keep the retractions in the document, dated. They are the part with the highest information content.\n\n## What I am left holding\n\nLess than I started with, stated more precisely, plus a scheduled recheck two days out that will decide between three remaining explanations.\n\nThat is a worse story and a better position. The version I would have published after round one was more impressive and would have been wrong in a way that took months to surface, probably in front of someone who had paid me.\n\n· Sources & further reading\n\n## Sources & Further Reading\n\n### Sources\n\n[Introducing AI Performance in Bing Webmaster Tools (Public Preview) blogs.bing.com](https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview)States that the reported grounding-query data represents a sample of overall citation activity, which falsified one of my load-bearing assumptions.[New AI Visibility Insights in Bing Webmaster Tools: Intents, Topics, Citation Share, Compare blogs.bing.com](https://blogs.bing.com/search/June-2026/New-AI-Visibility-Insights-in-Bing-Webmaster-Tools-Intents-Topics-Citation-Share-Compare)Defines Topics as post-hoc thematic grouping, which is why I could not claim the causal direction I wanted.\n\n### Further reading\n\n[Search Console's Generative AI Report: 3 Things It Hides /blog/search-console-generative-ai-report](/blog/search-console-generative-ai-report)Google's Generative AI report went live for every property on August 11, 2026. No queries, no clicks, no API access. Here is my own 19.7K-impression report.[Why Isn't ChatGPT Citing Your Website? I Tested 5 Axes on a DR 25 Site and Got 1,500 Citations /blog/why-ai-isnt-citing-your-website](/blog/why-ai-isnt-citing-your-website)1,500 AI citations on a DR 25 site in 90 days. The 5 checks (schema, llms.txt, OpenGraph, semantic HTML, robots.txt) that decide whether ChatGPT cites you.[How to Audit Your Website for AI Citation Readiness (7-Site Study) /blog/ai-citability-audit-what-predicts-citations](/blog/ai-citability-audit-what-predicts-citations)How to audit website content for AI citation readiness: a 7-site study of what actually predicts citations. Ahrefs (DA 92) gets cited 5% of the time; a DA-under-10 site hit 15%. Run the same checks on your site, free, before spending another dollar on backlinks.[Why Domain Authority Is Irrelevant for AI Search (And What to Build Instead) /blog/domain-authority-irrelevant-ai-search](/blog/domain-authority-irrelevant-ai-search)You're building backlinks for a metric that doesn't matter to ChatGPT or Perplexity. Data from 7 site audits answers the real question, does site authority matter in AI citation rankings, and shows what to build instead: a DA-92 site got cited 5% of the time, a DA-under-10 site got cited 15%.[Claude Code Trading Bot: 36,000 Lines, 1-in-40 Error Rate /blog/claude-code-production-trading-bot](/blog/claude-code-production-trading-bot)Unverified Claude Code output loses real money once it touches live capital. This claude code trading bot case study shows the two-gate system that cut errors 84% and caught 8 bugs before they cost a dollar.\n\n## What do you think?\n\nI post about this stuff on LinkedIn every day and the conversations there are great. If this post sparked a thought, I'd love to hear it.\n\n[Discuss on LinkedIn](https://www.linkedin.com/in/chudi-nnorukam)", "url": "https://wpnews.pro/news/i-ran-my-own-analysis-past-a-second-model-five-times-i-retracted-twice", "canonical_source": "https://chudi.dev/blog/adversarial-review-of-my-own-analysis", "published_at": "2026-08-19 00:00:00+00:00", "updated_at": "2026-08-20 00:13:40.920412+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/i-ran-my-own-analysis-past-a-second-model-five-times-i-retracted-twice", "markdown": "https://wpnews.pro/news/i-ran-my-own-analysis-past-a-second-model-five-times-i-retracted-twice.md", "text": "https://wpnews.pro/news/i-ran-my-own-analysis-past-a-second-model-five-times-i-retracted-twice.txt", "jsonld": "https://wpnews.pro/news/i-ran-my-own-analysis-past-a-second-model-five-times-i-retracted-twice.jsonld"}}