The Web Is Growing A Second Layer – Almost A Third Head

Google has introduced the Open Knowledge Format (OKF) and Agentic Resource Discovery (ARD) specification, adding new layers to the web's machine-readable infrastructure. These developments, alongside existing tools like LLMs.txt and MCP/WebMCP, aim to help AI agents navigate and interact with websites more effectively. SEO professionals must understand each layer's distinct purpose to make informed decisions about AI readiness.

The last few weeks have been noisy. Google shipped something called the Open Knowledge Format https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing . Then Google Developers announced the Agentic Resource Discovery ARD specification https://developers.googleblog.com/announcing-the-agentic-resource-discovery-specification/ . Meanwhile, every SEO LinkedIn feed is lit up with someone either declaring markdown the future of the web or explaining why you should ignore all of it. The truth, as per usual, sits somewhere more interesting than either camp. The web is developing a parallel machine-readable infrastructure https://www.searchenginejournal.com/machine-first-architecture-how-to-build-websites-machines-can-identify-read-cite-use/574431/ MCP/WebMCP, OKF, ARD, LLMs.txt… and SEOs who understand what each layer actually does, rather than treating it all as “AI SEO” or a silver bullet, will make better decisions about where to spend their time. First: The Layer Cake There are at least six distinct things being discussed under the umbrella of “making your site AI-ready.” They sit at different layers and serve different purposes: Crawlable HTML Pages: Still the foundation. Nothing has changed here. Everything else sits on top. Schema.org/Structured Data: Semantic hints baked into HTML that tell machines explicitly what a page is about. It is, in essence, a vocabulary. LLMs.txt: Essentially a navigation file. Its purpose is to essentially tell an AI agent that’s already on your site which pages matter. But as John Mueller puts it on the Search Off the Record podcast https://www.searchenginejournal.com/google-exposes-llms-txt-flaw/579814/ : “If someone is already on your website, maybe some kind of automated system is helpful. Where if it goes, I want to go to Martin’s Splitt and buy a photograph, then the LLM system can go to your website and can look around, like, how do you buy a photograph? Maybe he has some guidelines for me as an agent for buying photographs. That kind of makes sense.” Before ARD came into play, we were presented with another solution for the challenge of interoperability. An MCP, in its simplest explanation, is a standardized way for an AI to connect to your services to extract knowledge or take action. WebMCP, as the name itself suggests, gives websites a way to engage with agents directly. WebMCP is for live browser interactions on a webpage; MCP is for tools and services beyond the page. MCP https://www.searchenginejournal.com/mcp-a2a-nlweb-and-agents-md-the-standards-powering-the-agentic-web/570092/ /WebMCP: Open Knowledge Format OKF : A bundle of markdown files with YAML frontmatter. Agentic Resource Discovery ARD : A new open spec for how agents find and verify tools, skills, and other agents across the web. Here, the focus is not your content; it’s your capabilities. For ecommerce, there’s another layer worth naming separately – the product feed https://www.searchenginejournal.com/why-product-feeds-shouldnt-be-the-most-ignored-seo-system-in-ecommerce/569211/ – quite possibly the future of retail discovery https://www.searchenginejournal.com/googles-product-feed-strategy-points-to-the-future-of-retail-discovery/572291/ . Each layer does something different. I could keep adding to this list; there’s a new layer popping up every five minutes. I’m stopping here. It’s ballooning. What OKF Actually Is And Isn’t Google published the OKF spec https://www.searchenginejournal.com/google-cloud-announces-the-open-knowledge-format/579253/ quietly, bolted to a rebrand of Dataplex into Knowledge Catalog. The format itself is almost disarmingly simple: a directory of markdown files, each with a small YAML header declaring a type, title, description, resource, and some tags. The files link to each other like any markdown document would. That’s it. As Google’s own blog puts it, OKF is “just markdown, just files, just YAML frontmatter.” SEO Suganthan Mohanadasan has a clear breakdown of this https://suganthan.com/blog/open-knowledge-format/ . He describes OKF as one floor in a stack that now includes sitemap.xml which URLs exist , LLMs.txt which pages you most want read , and OKF the library itself . They stack rather than compete. The confusion sets in not when you look at what OKF is, but what it does and in which layer of the agentic and search mayhem it sits. In my mind, OKF is not a retrieval system. It doesn’t replace crawling. And, personally, I do not see a future where AI systems no longer ingest massive amounts of HTML or where search and RAG are not a multistep complex pipeline that consists of self-reported and “unbased” signals. Any self-reported system can and will be gamed. So thinking you can just slam a bunch of markdown files on your site and be THE preferred choice in retrieval and discovery is far-fetched. OKF is a higher-signal source among many. It may reduce parsing cost and improve signal quality, but it doesn’t replace existing pipelines. It’s also worth being honest here: OKF was built for data teams, not marketing sites. It arrived as a way to share internal knowledge, i.e., table schemas, runbooks, metric definitions, between AI agents inside organizations. Pointing it at a public website to me seems a bit like we are yet again repurposing. Francois Vanderseypen makes the most precise point about what OKF actually is and isn’t https://www.linkedin.com/posts/francoisvanderseypen knowledgegraphs-semantics-okf-share-7472881883970768896-5me-/ : a directed graph of markdown files is a web of documents, not a knowledge graph at least not in its purest sense . A real KG has explicit, queryable, typed relations. OKF leaves what a link implies entirely up to the producer, and an LLM still has to infer the semantics every single time it reads it. For me, this points to the crux of how I understand the web and what we do as SEOs. OKF doesn’t change the stack. It adds one more input into it. It’s not a shortcut. There are no shortcuts. The Schema.org Parallel, And Why It Matters One of the patterns to understand here is the one Schema.org already went through. Structured data followed a predictable arc: Adoption – ranking boost – widespread use and gaming – platform learning – reduced dependency as a ranking signal. FAQ schema had a moment in SERPs, then Google discontinued the FAQ rich result https://www.searchenginejournal.com/google-drops-faq-rich-results-from-search/574429/ . The platforms learn from the signals, fold the lessons into the algorithm, and the explicit markup becomes less necessary. OKF and LLMs.txt may follow the same path. They’re most valuable early, as clear signals in a world where AI systems are still learning to parse the web. Over time, if the formats work, the systems learn. Explicit markup becomes redundant or remains a verification layer. For example, in ecommerce, in particular, schema and feed alignment has become more and more important. Another notch in the call for co-ownership of the product feed between SEO and paid teams https://www.searchenginejournal.com/why-your-product-feed-is-an-seo-asset-and-who-should-own-it/575543/ There’s also a subtler point worth making here about the relationship between schema.org and discovery. Jarno van Driel’s deep dive on product variants in Search Engine Journal https://www.searchenginejournal.com/google-search-shopping-product-variants-and-a-gap-to-bridge/500243/ illustrates this well: For years, Google Search and Google Merchant Center had conflicting structured data requirements, forcing publishers to duplicate markup. Schema.org evolves to close gaps, but it’s slow, it’s complex, and implementation is still often a mess. Structured data has never been a plug-and-play ranking lever. OKF won’t be either. Should You Convert Your Site To Markdown? It’s a big fat no from me. That doesn’t mean I won’t test it and apply carefully And John Mueller said it on the Search Off the Record podcast https://developers.google.com/search/podcasts/search-off-the-record : “When it comes to things like a search engine or probably also in generic LLM system, having a website that uses normal HTML for the pages is critical. Because a search engine or crawler can just go to that page. It can recognise all of the other links that are within the website.” The structural information in HTML – nav links, footers, header hierarchies, internal links – is how crawlers understand your site’s shape. Markdown files strip all of that out. You’d be breaking discovery in order to marginally improve machine readability of individual pages. Recently, on LinkedIn, I even saw a piece of research https://www.linkedin.com/feed/update/urn:li:activity:7470379937313759232/ showing how “Your navigation might be eating your LLM it’s ChatGPT Deep Research in fact reading budget.” Interesting findings, but please don’t remove your navigation to “save some tokens” Jono Alderson makes this point brilliantly https://www.jonoalderson.com/conjecture/more-than-words/ : “A page is not just a container for words. It’s an editorial artifact.” Hierarchy, emphasis, placement, what comes first, what’s prominent, what’s tucked in a footnote … these aren’t pretty decorations for humans. “They are signals about meaning.” “When you flatten a page into markdown, you don’t just remove clutter. You remove judgment, and you remove context.” And the moment you publish a machine-only representation, you’ve created a second candidate version of reality. The boring fix still works: Semantic HTML, clear structure, sensible hierarchy, content that exists when the page loads. John Mueller covers the markdown debate extensively in the podcast: The parallel versions problem, the dynamic rendering lessons we already learned the hard way, and why maintaining a shadow version of your site for AI doubles your maintenance burden and creates a debugging nightmare nobody will tell you about. The one exception Mueller carves out is developer documentation: “If you have something like developer documentation, where, again, if the agent or the LLM system already knows about your website and the user says, how do I usethis API? Then if you give the LLM system a Markdown file, it’s a lot easier for it to understand.” Now, I can definitely see a straightforward use case there. What ARD Is Actually Doing The Agentic Resource Discovery specification https://www.searchenginejournal.com/google-microsoft-back-draft-ai-agent-discovery-spec/579894/ , announced by Google on June 17, 2026, is a different beast entirely. It arrived only a couple of days behind OKF, not a coincidence, and is already making huge waves. The problem ARD solves is a coordination one. Right now, an agent has to be wired to each tool, MCP server, or API it uses before it can do anything with it. That works when you’re connecting a handful of known services. It stops scaling the moment the number of available capabilities grows beyond what any team can pre-configure by hand. ARD moves that discovery out of setup and into runtime. The agent finds what it needs when it needs it, rather than only knowing what it was told about in advance. It’s built on two primitives: Catalogs: An ai-catalog.json file hosted on your domain, describing your available capabilities MCP servers, A2A agents, OpenAPI tools . Ownership of the domain acts as the cryptographic foundation for identity and trust. Registries: Search engines for the agentic web. They crawl catalogs, index them, and return matching capabilities with the metadata needed to verify the publisher before connecting. If OKF is about packaging knowledge for consumption, ARD is about advertising capabilities for connection. These are parallel efforts at different layers of the emerging agentic stack. Both shipped within inches of each other and now adopted with the speed of light by some very big players in the game, i.e., Hugging Face and their Discover Tool. It’s possibly a more pragmatic bet than the formal logic layer that came before it and never reached web scale. Time will tell. A Gap Worth Watching Within days of both specs shipping, a contributor opened companion issues on the ARD and OKF repos pointing out something basic was missing: There’s no agreed media type for an OKF bundle https://github.com/ards-project/ard-spec/issues/27 , so a catalog can list one but can’t actually recognize it as OKF without sniffing the contents. In the meantime, publishers are already advertising bundles in production using their own interim types, which, as the issue itself notes, won’t agree with each other. On the face of it, this looks like a small ask, just a request for a shared label. After a bit of a dive into this particular rabbit hole, it turns out that’s quite normal practice. Waiting for full agreement before anyone ships anything is exactly how a spec dies in committee, and shipping fast and patching as real adoption surfaces is an age-old strategy. Application/json itself wasn’t formally registered until 2006 https://en.wikipedia.org/wiki/JSON , roughly five years after JSON was already in wide, informal use. Nobody worried about that, because the cost of the label being unsettled was low: A parser might reject something or fall back ungracefully. But OKF is different, because what happens after the fetch is different. The artifact behind the label is a bundle an autonomous agent is meant to ingest, verify, and potentially act on, inside a discovery system built specifically for agent-to-agent and agent-to-tool connection. Get the type wrong here, or leave an agent to infer it, and the risk isn’t a parse error; it’s a system acting on something it shouldn’t have trusted, with no one checking the result first. I wonder about the risk involved in settling this later rather than sooner in this case. I guess it depends on how fast it gets resolved relative to how fast adoption runs ahead of it. What This Means If You’re An SEO A few honest conclusions and my current thinking: For most marketing and content sites, not much has changed. HTML, well-structured for humans, is still the right foundation. A contact-us form and a clean site architecture will serve you better than any OKF bundle ever will. Discovery still depends on links, authority, user signals … and indexing. LLMs.txt is a signpost, not an SEO tool. It’s useful for helping an agent navigate within your site once it’s already there. It very likely doesn’t make a big difference in how agents find you in the first place. And, probably never will. MCP/WebMCP. Neither is urgent for most marketing sites today, but if you’re building anything with programmatic interfaces or ecommerce flows you want agents to navigate, this is the direction the infrastructure is heading. OKF makes a lot of sense if you’re sitting on structured internal knowledge , i.e., documentation, API references, product specs … and you want to make it easier for agents to consume. The free OKF generator Suganthan built will produce a bundle and give you a graph view of your internal link structure as a side benefit. The structural audit alone seems worth it. But I will be running it on my website first, not on my client’s website. ARD is worth watching if you’re building services with programmatic interfaces. If you have tools, agents, or APIs you want discoverable by other agents, ARD is the emerging standard for how that gets done. Just know the identity layer https://www.searchenginejournal.com/machine-first-architecture-how-to-build-websites-machines-can-identify-read-cite-use/574431/ underneath it, what an agent is actually looking at when it finds your catalog entry, is still being settled in real time, so I’d treat this as infrastructure to watch closely rather than build critical paths on just yet. The schema adoption cycle might repeat. These formats are most valuable now, as early signals. Implement them if you can do it cheaply. Don’t build your strategy around them holding value forever and don’t bank on them as a silver bullet. Ultimately, be aware of the shiny things – if your company has bigger fish to fry, i.e., a terrible website, a brand no one knows or cares for, an audience you don’t understand … then deal with this first before you get caught up in any of these new shiny things. The Underlying Shift What all of this points to is a web that’s genuinely growing a second layer or a third head, one written for machines alongside the one written for browsers and humans. Sitemap.xml told crawlers which URLs existed. Robots.txt told them where not to go. LLMs.txt, OKF, and ARD are similar infrastructure for agentic systems: navigation hints, content packaging, and capability discovery. None of it is mandatory today. None of it replaces solid HTML, authoritative content, sensible structure, or the thing that actually sits underneath all of it: a brand worth finding. But the SEOs who understand what each layer actually does, rather than treating it as a single undifferentiated “AI SEO” category, will make better bets on where to spend their time. My money is on the second layer, a parallel infrastructure written for machines, not a replacement for what already exists. The third head scenario, where agentic systems fully diverge from the human web, would require a different set of bets than any of us are currently making. Big thanks to Jarno van Driel, Jono Alderson, Chris Green, Suganthan Mohanadasan, Kristine Schachinger, Gianluca Fiorelli, Victor Pan, Renee Bigelow and anyone else I’ve missed for some brilliant discussions on this topic over the last few weeks. More Resources: 97% Of LLMs.txt Files Got No Requests, Ahrefs Data Shows https://www.searchenginejournal.com/97-of-llms-txt-files-got-no-requests-ahrefs-data-shows/579478/ WebMCP Can Be Used To Hijack AI Agents, Chrome Warns https://www.searchenginejournal.com/webmcp-can-be-used-to-hijack-ai-agents-chrome-warns/578904/ The Technical SEO Audit Needs A New Layer https://www.searchenginejournal.com/technical-seo-audit-new-layer/571583/ Featured Image: Collagery/Shutterstock