{"slug": "the-website-is-not-the-product", "title": "The Website Is Not the Product", "summary": "A developer argues that AI systems are extracting value from the web without returning traffic, citing data showing Anthropic's crawler retrieves 60,000 pages for every one visitor sent back to a publisher—a tenfold increase from six months ago. The developer proposes that publishers must shift from static `llms.txt` files to dynamic MCP endpoints that serve structured knowledge directly to AI agents, as the website is becoming a fallback UI rather than the primary distribution channel. Over 1,000 sites now use `llms.txt`, but the developer notes that no major publication has yet built an MCP endpoint, leaving a conceptual gap where most publishers still treat the website as the product instead of the context it delivers.", "body_md": "`llms.txt`\n\n(1,000+ sites) is the right model but static — it describes content; it doesn't serve it.Anthropic's crawler retrieves 60,000 pages for every one visitor it sends back to a publisher. Six months ago that ratio was 6,000:1. OpenAI's crawler is at 1,500:1. Google, which used to crawl 2 pages per visitor sent, is now at 18:1 and climbing.\n\nThe website as distribution channel is not declining. It is being extracted.\n\nAI systems are consuming the web at scale without returning the traffic that made web publishing economically viable. Cloudflare's CEO confirmed this publicly in January: search referrals have \"plummeted\" because users trust AI summaries and don't follow the footnotes.\n\nThis is not a content quality problem. It's a distribution layer problem.\n\nThe web was built for a world where humans navigated to URLs. That world is not coming back. What's replacing it is a world where AI agents retrieve context on behalf of humans — and the agents don't care about your navigation, your layout, your calls to action, or your SEO metadata. They care about whether your content is parseable, structured, and context-dense.\n\nThe website is becoming the fallback UI. The primary interface is everything else.\n\nThe wrong lesson from this is \"optimize for AI crawlers.\" That's SEO-era thinking applied to a post-SEO world.\n\nThe right lesson is: distribution is now a systems design problem, not a marketing problem.\n\nIf you publish knowledge, you now have three consumer types with incompatible requirements:\n\nMost content strategies optimize for the first. Some are starting to acknowledge the second. Nobody is building for the third.\n\n`llms.txt`\n\nis the first concrete mechanism with traction. Over 1,000 sites now publish one — a curated, structured entry point that tells AI systems what the site contains and where to find it. The model is correct: LLM context windows can't process full websites, so a structured index is practical infrastructure.\n\nBut `llms.txt`\n\nis still a static file. It describes content. It doesn't serve it.\n\nThe next layer is dynamic — an MCP endpoint that serves knowledge on demand, at the right granularity, with the right context pre-loaded. Not a website. Not a search index. A knowledge interface built for agents.\n\nThe thesis: structured, machine-readable content delivered through MCP endpoints is the emerging distribution layer for knowledge. Humans subscribe through whatever surface they prefer. AI agents retrieve through the endpoint. The monetization layer is access to context — not page views.\n\nNobody is building this. Substack has no `llms.txt`\n\n. No major publication has an MCP endpoint. The entire publishing industry is optimizing for a consumption model that is already being displaced.\n\nThe gap is not technical — the tools exist. `llms.txt`\n\nis simple to implement. MCP servers take days to build. The gap is conceptual: most publishers still think the website is the product.\n\nThe website is the UI. Context is the product.\n\nSo what: if you publish knowledge, the question is not whether to maintain a website. It's whether you're building the layer underneath it — the one that serves agents, not browsers. That's where the next distribution advantage is. And right now, almost nobody is there.", "url": "https://wpnews.pro/news/the-website-is-not-the-product", "canonical_source": "https://dev.to/amitrix/the-website-is-not-the-product-2pdf", "published_at": "2026-06-06 07:16:34+00:00", "updated_at": "2026-06-06 07:42:00.644853+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "generative-ai", "large-language-models"], "entities": ["Anthropic", "OpenAI", "Google", "Cloudflare"], "alternates": {"html": "https://wpnews.pro/news/the-website-is-not-the-product", "markdown": "https://wpnews.pro/news/the-website-is-not-the-product.md", "text": "https://wpnews.pro/news/the-website-is-not-the-product.txt", "jsonld": "https://wpnews.pro/news/the-website-is-not-the-product.jsonld"}}