Stop serving blanks to bots

An analysis of the HTTP Archive's 1 August 2026 mobile crawl, covering 17.2 million pages, found that about 32% of pages serve incomplete content to non-rendering AI crawlers, with roughly 9% delivering under half their content and 21% delivering under 80%. The study attributes the gap to client-side JavaScript rendering, which AI systems such as those feeding ChatGPT, Claude, Perplexity and Google's AI answers ignore, taking only the initial server response. The author concludes that sites should ensure the representation given to AI systems is "complete, traversable and economical to consume" before worrying about ranking or citations.

It may change in the future but nearly all sites today are built for people - you click a link, a browser does a bunch of work and a page appears. This has been the way for thirty plus years but times are rapidly changing while you yearn for a revival of Geocities… A large and growing share of the requests hitting our sites now come from AI systems: the crawlers that feed ChatGPT, Claude, Perplexity and Google’s AI answers along with the assistants that fetch a page on the spot because someone asked them a question. These aren’t browsers and they don’t behave like them, for most sites what these bots come away with isn’t what you’d actually want them to have probably . I went down a bit of a rabbit hole this weekend and I decided to try to measure what agents might be getting using publicly available data via the HTTP Archive https://httparchive.org/ project for some initial data and I also looked at a few external studies as well for good measure. I wanted to see what happens for an agent/bot when a page arrives in the state that the data says it does and how widespread that is or isn’t. TL/DR: Before worrying about whether an AI system ranks or cites you, make sure the representation you’re giving it is complete, traversable and economical to consume. I used data from the HTTP Archive https://httparchive.org/ August 2026 crawl httparchive.crawl.pages and httparchive.crawl.requests , used under CC BY 4.0. I looked at the 1 August 2026 crawl as a snapshot for mobile and clean test runs only. This gave me content for 17.2 million pages and links on 21 million. I then excluded any failed test runs 21.8% , pages rendering under 100 words and pages where the raw HTML held more than the rendered page. My filters deliberately narrow this to content bearing pages where the rendered version adds text and so the results shouldn’t be read as a measurement of every type of web page. This isn’t a new problem and I’m building on work others have already done. HTTP Archive’s Web Almanac has looked at raw vs rendered content for years, SearchVIU did a rendered vs source study back in 2020 and Vercel/MERJ and others have done a lot of the groundwork on rendering and crawler behaviour. I’m joining some of those pieces together out of curiosity to see how widespread they might be. I’m also only measuring structure and not quality so the actual content on a site may be utter rubbish. For example, things like word count will tell me how much text differs but not whether the missing text is important. A lot of modern websites don’t arrive as finished pages, they instead start as an empty frame with a set of instructions for the visitor’s browser to follow to build the page - this is client side rendering typically using JavaScript in whatever flavour of framework you want. It’s what most site platforms do today by default and for a human with a browser it works just fine. AI readers don’t follow those instructions, instead they ask for the page, take whatever comes back in that first response and then leave assuming they got everything. They more or less ignore the JavaScript so your site may be super quick for us humans but you’re serving blanks to these crawlers. HTTP Archive crawls millions of sites a month and gives me both the HTML the server sent and the page after the JavaScript has run. This allows us to see the difference for what a non-rendering bot may see vs a human and how widespread it is. All told, about 32% of all pages should probably have a look taken at them by varying degrees of urgency. There’s definitely also a grey area here with cloaking since you can be showing bots one thing and a human another. Most of the web is fine with about two thirds of pages serving 90% of the content to bots. But, ~9% of the pages serve under half their content and 21% serve under 80% which is to say any optimisation being done is not being seen. For the bot a partial page doesn’t look partial even if there’s a heading and nothing beneath it or a sentence that stops halfway, they can’t really tell if anything is missing. There’s no error or warning so the crawler has no reason to think it received anything but the whole thing. Vercel and MERJ went through over 500 million requests https://vercel.com/blog/the-rise-of-the-ai-crawler from OpenAI’s crawler and found no evidence that it ever ran the JavaScript. Anthropic’s, Perplexity’s and Meta’s crawlers behave in the same way. The only real exceptions are Apple who use a browser based crawler and Google who’ve spent a decade and a great deal of money building the ability to do this and even then it takes a second pass and a delay. So if the content, product descriptions, prices, etc only appear after the browser has done its work then there’s a good chance that the AI systems that everyone is now optimising for are looking at a blank frame where your content should be. Instead, have the server send a finished page also known as server side rendering . It depends on how this is implemented and it may not cost much in visitor experience but it’s worth doing whether or not you care about anything else in this article. The issue with this is speed or more precisely, the perception of speed, as rendering from the server can feel slower than if it happened on the client e.g. Time to First Byte . Crawlers find pages by following links so if a page’s links are built in the browser, a crawler that doesn’t run JavaScript may miss parts of the site. This can also show up in an agents query fan out process or a RAG ingestion pipeline where a non-rendering crawler can’t traverse onward from this HTML response using links contained in that response. Around 39% of pages are probably worth checking for how their internal linking is or isn’t being seen and some pages more so than others. Additionally, in this view, one in fifteen pages is a dead end and 1.36 million pages have no crawlable links in their HTML with 169,377 of those serving their text perfectly well. For an agent this may be the one that does the most damage as it can’t distinguish a genuine dead end from a page whose navigation didn’t show up. It stops as there’s nothing about the page to suggest that anything went wrong and then depending on the page, it then may not discover large amounts of your site. The obvious objection to this is that the missing links are just navigation, repeated everywhere and reachable from another page anyway. When I restricted to internal links then the figure only dropped from 6.5% to 5.9%, meaning that they’re likely genuinely missing but I would need full crawls of each site to validate this. Quantity doesn’t equal importance, losing 10% of the links may remove the only route into an entire section of the site or irrelevant boilerplate and that can’t be stated from this data. Let’s say we’ve solved that rendering issue and the AI system can now read the page. Great What it receives though is the article or product description wrapped in an enormous amount of scaffolding: navigation, menus, footers, cookie banners, tracking pixels, styling and so on. None of it’s of any interest though to a machine that has to answer my inane late night questions. AI systems have a budget for how much they can take in at once and that scaffolding gobbles it all up. Cloudflare’s figures https://blog.cloudflare.com/markdown-for-agents/ suggest a blog post that costs around 16,180 tokens as a normal web page instead costs about 3,150 tokens as plain text which is roughly an 80% saving for the identical information based on that data. A sophisticated crawler can already strip HTML, boilerplate and scripts by itself so this isn’t necessarily an 80% context window saving at inference time. The inefficiency is that every consumer has to download and perform that extraction independently. But what about semantic markup? Yes it’s good to use <main , <article etc. to identify the content in the page and we should all do this. It does make it simpler but it only provides the boundary but it’s a more of a hint and it looks like a large percent of sites aren’t using it: Each one of these things is also an accessibility fix. I did see 48% of pages carry JSON-LD in their HTML which sounds encouraging and great for legacy search engines but JSON-LD can’t be relied upon to survive the HTML to markdown conversion because typical converters strip script elements. The facts still sit in the visible text where they have to be inferred but they can be easily gotten wrong. Schema is also usually placed late in the document so on a big page it can fall past the first 2MB, which is all Googlebot fetches for Search. Additionally only 32% of pages using JSON-LD use @id at all which limits explicit entity identity and linkage between structured objects. The data also tells us the platform and technology a site uses, so is one better than another? I’m not recommending to do platform or technology migrations from one to another but it is striking as to the differences and the potential for improvement. Here I’m looking at the % of pages served that are classified as good, 90% of what’s rendered is in the initial HTML, so machines are better able to get the content. Or bluntly, which platforms are doing the best job of getting their rendered text into the initial HTML? On this measure, Wix and Webflow as platforms look much stronger for being machine readable than others. When looking at the libraries being used, I wasn’t able to see every technology that each platform uses but it was interesting that when implemented by a platform the same libraries appear to perform quite differently. Couple caveats to this data, the technology detection assigns multiple labels per page which means a page can appear in several rows so this isn’t a partition of the web. This is also correlation rather than causation as the kinds of sites built on each platform and how heavily they’re customised may account for some of the differences. The web has always had a way of handling this problem: When something requests a page, it can say what format it would like it in and the server can answer accordingly. This is how your browser gets the right image format without you thinking about it and has been part of how the web works since the nineties. In February this year Cloudflare turned this on https://blog.cloudflare.com/markdown-for-agents/ for plain text / markdown so if the thing asking says it would prefer that format then Cloudflare strips the page down and sends just that instead. Fastly published its own version in June https://www.fastly.com/blog/give-ai-agents-markdown-they-actually-want . Vercel documented it https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation . There’s plugins for CMS’s like WordPress and so on. For most sites this is now becoming a setting https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ but it’s a setting hardly anyone is using yet. Every fetch means downloading the full HTML and converting it locally every time. Even when a site has the plumbing in place to hand over the markdown version, there is an additional snag, the HTML to markdown conversion happens at the CDN in most of these products which means it converts whatever the origin sent. For a client rendered site it faithfully converts an empty frame into empty markdown producing a well formed rendering of nothing. The moment you can serve one version to people and a different version to machines, you’ve recreated one of the conditions that makes cloaking possible. How do you know the two representations still say the same thing? No surprise then that Google and Bing both reacted to the Cloudflare launch with some suspicion https://searchengineland.com/cloudflare-markdown-for-agents-469246 . Bing’s position was that they’ll simply fetch both versions and check that they match, which works but also destroys the whole point. If verification means fetching everything twice then much of the network efficiency benefit disappears. Within a day of the launch, someone also demonstrated https://queryburst.com/blog/cloudflare-markdown-for-agents/ that a site can detect these requests and deliberately send machines a different, tampered version. There’s published security research https://arxiv.org/abs/2509.00124 on the same trick used as an attack: show people a normal page and show AI assistants a version containing hidden instructions designed to manipulate them. Just like that we’re back to cloaking and spam again, only with agents acting out autonomously the impact could be so much worse. So the obvious solution to serve markdown for agents, which could improve efficiency and gives machines a cleaner representation to work from, also comes with a massive cost and needing to rework how the web and crawlers work. That’s not our issue, if our customers are using agents then we should build for what our customer wants. You might’ve heard of llms.txt. The idea was that you publish a file at a standard location listing your best content in a clean format and AI systems then read it, akin to having a robots.txt file, only that the robots.txt file instead is a set of agreed upon deterministic directives. One analysis of over 500 million crawler visits https://limy.ai/blog/llms-txt-in-2026-the-full-guide found only a few hundred requests for that llms.txt file in total. Google has said it won’t support it after publishing and then retracting documentation for it in the same day, though interestingly they do reference it in their documentation in lighthouse https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring … so maybe… who knows. The latest is the guidance not to do anything specific to appear in Google Search https://developers.google.com/search/docs/fundamentals/ai-optimization-guide such as llms.txt, send markdown files etc. to aid discovery which makes sense given the huge amount of infrastructure and investment built around parsing HTML that they have today. I don’t think the llms.txt idea is bad as such but I think the delivery wasn’t the right way to solve this problem. That approach asks every AI system to go and check for a file that, on nine sites out of ten https://seranking.com/blog/llms-txt/ , won’t be there. This is then a wasted request, repeated billions of times on the off chance that it may exist and then the content within it may also vary between publishers and lets not think about anything nefarious you could do within that setup… One could argue to just use WebMCP https://webmachinelearning.github.io/webmcp/ which might make the whole thing about serving markdown files for an agent somewhat moot. Perhaps, but, WebMCP is about providing a set of actions that an agent can take. From the agent perspective, WebMCP looks like a better use of time than llms.txt but it doesn’t necessarily solve the bloat in processing the surrounding page just to get the useful content. The other issue is that not everyone will be able to implement WebMCP. So there could be a combination where something like WebMCP would be available to provide the actions for an agent to take on a site but the data would be available via .md files for example so there’d be no need to traverse the DOM etc. It’d be quicker, more accurate and friendlier to your agents context window I think if it could use a markdown version, but, to prove markdown is providing the equivalent content as the human experience would mean a massive reworking of the way the web and crawlers work today as now you have to verify two formats are the same thing. OK, so what do we do about this? First, perhaps take a look at your platform and libraries being used which may be prohibitive as to what you can do. Try to get your server to send finished pages. The obvious temptation is to keep the fast JavaScript experience for people and serve bots a cached version instead which can be a workaround https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering but be aware that it can end up being treated as cloaking if the variants aren’t the same. In an ideal world I’d want to render or pre-render the HTML for everybody and then use the accept header for providing the desired format. If nothing else, this then at least makes all your content visible in the initial HTML to non-rendering bots and probably better performance overall since most pages can be cached for a long time which might even save you a few $ on your hosting bills. Here’s the list of key fixes I would consider looking at if you want to boost machine understanding of your site which reads like an SEO list… hmmmm… Then turn on plain text answering for requests that ask for it, put it on every response so systems can find it without hunting and then make sure both versions come from the one same source. I don’t think there’s any harm in doing this, it’s a relatively quick change too and it’s on the crawlers then to figure out their efficiency problems and eliminating the bad actors… Additionally I would then take a look at integrating WebMCP and preserving structured entity data alongside the markdown representation. The problem with this list of fixes is that I can’t definitively say whether it will improve an outcome by any exact measure because there are several probabilistic systems involved, in theory we can… maybe… If you have to prioritise with limited resources I would still favour what improves the customer experience over most of these fixes because we already know that improving the actual customer experience has value. I will try to follow up though with a methodology for how to estimate the benefits of these changes. Every AI bot combined sends under 1% of referral traffic but to do that they’re operating at a very disproportionate share of crawl. According to Cloudflare, Anthropic’s crawler takes 11,000 pages for every visitor it sends back and OpenAI’s takes 800 for each referral. In contrast. For reference Google crawls about 5 pages for each visitor they send . Separately, it’s something insane like 29% share of the total googlebot volume on Vercels network to get that tiny amount of traffic. And the insane volume of crawl behind it: Those data sets aren’t directly comparable and crawl per referral isn’t a measure of crawler efficiency. I’m putting them together only to illustrate the scale mismatch between machine consumption and the little traffic returned today by those systems. So it all seems that whatever value that these crawlers produce, the resource asymmetry is striking and anything to help improve that can’t be a bad thing. It’s just there’s currently no real business incentive for anyone to do so unless you own the model crawling thousands of pages and need to reduce costs. With machines burning enormous crawl/processing resources while also having to adapt browser representations it wouldn’t be that far fetched, instead of the current paradigm, for those sites who are more efficient to be rewarded the same as for those with original content. If anyone wants my SQL for pulling this data, let me know when I get a moment I’ll post it up somewhere . Thanks again to HTTP Archive https://httparchive.org/ for making this data available Stop serving blanks to bots https://pub.towardsai.net/stop-serving-blanks-to-bots-a476477c9ccd was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.