20 Percent of All ‘Ok’ Web Responses AI Gets Are Not Okay Roughly 20.9% of web pages that return an HTTP 200 status lack the main content a client expects, according to experiments in AI-assisted web scraping reported by Unite.AI's Anderson's Angle column. Treating every 200 response as a success inflated apparent citation drift from 5.8% to 21.9%, while cheap heuristics on page length and redirects flagged around 86% of unusable responses. The findings matter for AI crawlers and scraping pipelines that rely on status codes to confirm a page was retrieved. Anderson's Angle https://www.unite.ai/series/andersons-angle/ 20 Percent of All ‘Ok’ Web Responses AI Gets Are Not Okay Add Unite.AI to your preferred sources on Google https://www.google.com/preferences/source?q=unite.ai Key PointsRoughly 20.9% of pages returning status 200 lack the main content a client expects. View more - Roughly 20.9% of pages returning status 200 lack the main content a client expects. - Accepting every 200 response as success inflated apparent citation drift from 5.8% to 21.9%. - Cheap heuristics on page length and redirects can flag around 86% of unusable responses. My recent experiments in AI-assisted web-scraping have brought home to me the growing extent to which the web is becoming ‘locked-down’. The motivating event for this retrenchment, of course, is the shocking growth https://www.unite.ai/wikimedia-foundation-finds-rogue-openai-agent-activity-on-its-projects/ :~:text=reported%20its%20bandwidth%20usage%20had%20increased%20by%2050 in AI web-scraping, whose uncompromising demand https://www.unite.ai/the-canary-that-reveals-ai-traffic/ :~:text=they%20really%2C%20really%20want%20to%20scrape%20your%20website for new data can end up taking domains offline. This is a good excuse for domains to do what many of them have wanted to do for a long time: make their content accessible only when the user is logged in though naturally, Googlebot typically gets a free pass https://developers.google.com/search/docs/appearance/structured-data/paywalled-content . Facebook completed its gradual adoption of this policy in 2023 https://news.ycombinator.com/item?id=37432651 , and Twitter/X did likewise the same year https://www.investing.com/news/stock-market-news/twitter-now-needs-users-to-sign-in-to-view-tweets-3118107 . Reddit will adopt it later this month https://www.unite.ai/reddit-announces-shift-to-primarily-logged-in-experience/ . In this way, compliance with any country-specific age-limitation enforcement https://www.unite.ai/ai-tool-strips-away-makeup-to-stop-minors-bypassing-age-checks/ becomes a little easier for the domain, since the user is already registered. More importantly, the user’s data can be saved, exploited for local analytics, and – local legislation permitting – sold to data brokers and AI companies. Elsewhere, measures squarely aimed at defending against bots rather than enrolling users have risen more than 58% year-on-year, according to Cloudflare https://blog.cloudflare.com/the-most-seen-ui-on-the-internet-redesigning-turnstile-and-challenge-pages/ :~:text=That%27s%20a%2058%2E1%25%20average%20increase%20in%20security%20checks ; and all that escalating bot activity naturally encourages websites to scrutinize https://repositories.cdlib.org/uc/item/94r9h469 VPN and proxy addresses https://www.unite.ai/residential-proxies-as-the-next-front-in-the-ai-wars/ , potentially denying legitimate users access, together with any ‘malicious’ traffic. The anecdatal aspect of this is that I seem to come up against ‘blocking’ responses or challenges more and more, to the point where I may even abandon a site that begins loading them; and since CAPTCHA -walls have been found to deter users and potential customers https://www.fastly.com/blog/captchas-are-costing-retailers-customers , one can imagine the pressure that causes domain owners to adopt them in spite of that downside. All this new and increased friction is partly logistical because not all domains can withstand the onslaught but also often a kind of negotiating tactic to see if AI companies will offer a single or collective form of remuneration for all that data they so desperately crave, and which has been free for so long, until now. Since AI crawlers have to work at scale, they tend to use ‘headless’ instances of Chrome under the Playwright https://playwright.dev/ framework. These days, for an increasing number of domains, that will get you a 403 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/403 ‘forbidden’ error i.e., host knows you’re automated traffic, now scram : What’s wanted is the ‘200’ https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/200 ‘OK’ client response, even if one has to hand over a pile of money to a residential proxy farm https://www.unite.ai/residential-proxies-as-the-next-front-in-the-ai-wars/ , in concert with other diverse non-human sneaky tricks, in order to obtain it. There’s nothing better than the 200 response; which is to say, there is no client response code with a higher level of positive signal. Neither is it an ambiguous response; it’s never ‘nearly okay’ , ‘going to be okay later’ , or ‘getting more okay’ ; it’s ‘okay ’ . If it’s not okay, it’s not a 200 response. Hello Go Away Except that this lack of granularity, according to a new paper, could be significantly skewing AI datasets, statistical estimations, and reporting ‘good’ status when that is not the case. The study https://arxiv.org/pdf/2611.00884 estimates that around a fifth of 200-category responses contain no usable page content at all , instead delivering empty website shells, CAPTCHA challenges, access restrictions, or substitute pages. The paper states: ‘20.9% of the pages that a static client receives with status 200 do not contain the page’s main content: app shells, bot challenges, block pages, error and placeholder pages, and partial content.’ In this regard, impatient automated systems are tripped up by their own sparseness, since the ‘200’ response is allowed to be equated with successful fetches, even when the fetch just produces a CAPTCHA-wall, a paywall, or some other obstacle. The paper outlines the seven types of content found to frequently complicate a 200 response: For the study, over three days, 10,307 web addresses were tested using both a basic web-fetching program and a browser capable of loading JavaScript, with responses classified according to the content retrieved , rather than their HTTP status codes. The largest categories of ‘unusable’ 202 responses were empty website shells and error or substitute pages, together accounting for more than two-thirds of the failures. Regarding the empty site shells, a website might display its menus and navigation while leaving the requested article or product listing blank; in the latter case, an unrelated page or homepage might appear instead of the requested content – but both could receive the same HTTP 200 status as a successful retrieval offering usable and useful content. By way of a remedy, the paper recommends that web-fetching tools report what was actually retrieved , rather than relying on HTTP status codes, since simple checks for unusually short pages, suspicious redirects and content resembling nonexistent pages could identify most failures. More resource-intensive browser tools could then be used only when a page appears to have failed to load properly. Additionally, AI evaluations could check whether cited sources actually contain the expected information , rather than assuming that an HTTP 200 response means the content was successfully retrieved. The author of the new work also notes that ‘cheap heuristics’ can trivially identify around 86% of unusable responses, using basic checks on page length, redirects and similarities to nonexistent pages: Why So Many Errant 200 Responses? There has been some argument https://stackoverflow.com/questions/9794696/which-http-status-code-means-not-ready-yet-try-again-later over the years, in developer communities, that the 202 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/202 ‘Accepted’ error ‘Requested accepted but not yet completed’ could disambiguate these ‘failed’ 200-status requests; but really, 202 cannot accommodate all the cases outlined in the new paper, and there is no existing code that can. Though the paper does not deal with the issue in depth, the reason why domain owners permit 200 errors might be worth further investigation in future related works. One reason could be that anything that is not a 200 is essentially some kind of error page or obstruction, including the most common, 404 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/404 ‘not found’ errors – where the target content is simply missing. Some users believe https://www.reddit.com/r/SEO/comments/1f1boxk/effect of 404 errors on site ranking/ that an excess of such errors will lead to penalties in SEO and, presumably, GEO https://www.unite.ai/geo-strategy-ai-search-brand-visibility/ , though Google denies this https://www.seroundtable.com/google-404-errors-42405.html :~:text=404%20errors%20do%20not%20cause%20Google%20penalties . In fact, erroneous 200 errors are considered as ‘soft’ 404s https://developers.google.com/search/blog/2008/08/farewell-to-soft-404s by Google, and clearly likely to be treated as more of a problem than plain 404 errors, because these are essentially ‘deceptive’ outcomes, intentional or not. Citation Drift If all this seems like pedantry, one has to consider the petabyte-level scale of the data that is routinely gathered or attempted to be gathered for the purposes of AI training. If a full fifth of that data is unreliable or outright deceptive, the downstream consequences are difficult to shrug off as some kind of ‘collateral inaccuracy’; rather, such false responses could significantly and adversely affect the distribution in datasets and the outcomes of training such data into AI models. For instance, one notable issue around erroneous 200 responses concerns ‘citation drift’ https://aclanthology.org/2025.wasp-main.20.pdf , where an AI-generated answer is no longer supported by the web page that was originally cited as evidence, i.e., the page may have changed, the relevant text may have been removed, or the link may now lead somewhere else. The new study found that apparent citation drift rose from 5.8% to 21.9% when all HTTP 200 responses were accepted as successful retrievals, regardless of what they contained . This means that a source could be wrongly judged to have changed, disappeared or ceased to support an AI-generated claim, simply because a CAPTCHA, empty page or other obstruction had been retrieved in place of the original material. Just Say It’s So The problem was also demonstrated by tests involving deliberately nonexistent web addresses. When random, invented paths were requested from websites, 13.1% of those that responded returned a 200 ‘OK’ status, even though the requested pages did not exist