Cloudflare put it plainly this spring: the web learned to speak to browsers, then it learned to speak to search engines, and now it needs to speak to AI agents. Alongside that argument they shipped isitagentready.com, a scanner that grades any site on how well an agent can find it, read it, and use it. The adoption numbers in their launch data are striking for how early we still are: only 78 percent of sites have a robots.txt at all, and some of the newer standards had been implemented by fewer than fifteen sites on the entire internet.
We spent part of this month making lightningjar.com agent-ready, and this post is the field report: what "agent-ready" concretely means, what we shipped, what the scanner says now, and what we learned that the specs do not tell you.
The Four Questions an Agent Asks Your Site #
Strip away the acronyms and the whole topic reduces to four questions, which are also the scanner's four dimensions:
Can I find you? Discoverability: robots.txt, a sitemap, and increasingly, machine-readable pointers to your agent-facing resources.Can I read you? Content accessibility: clean, structured content an agent can consume without fighting your page chrome.Am I allowed? Bot access control: explicit, machine-readable policy about crawling, AI input, and AI training.Can I do anything? Capabilities: actual endpoints (MCP, APIs, in-page tools) that let an agent query and act rather than scrape.
A site that answers all four is not doing anything exotic. It is doing what the web has always done for a new kind of reader: publishing structure and meeting the client where it is.
What We Shipped #
A living site index at /llms.txt. Following the
llms.txt convention, one plain-markdown page now summarizes the whole site: who we are, every blog post, customer story, reading-list entry, research study, and technology page, with an Agents section up top pointing to the machine endpoints. It renders at request time from the same CMS that powers the pages, so it is never stale, and it currently indexes about 150 entries. An agent that fetches one URL gets the entire map.
Markdown for agents, on the same URLs. Every blog post, customer story, and reading-list entry now answers Accept: text/markdown
with its actual markdown source, frontmatter included, plus a token-estimate header so an agent can budget before it reads. The homepage answers the same negotiation with the site index. Humans and agents share one URL; the server simply speaks each visitor's language. This is where the practical win lives: Cloudflare reports that restructuring their own docs this way cut agent token consumption by 31 percent, and our markdown responses are a small fraction of the size of the rendered pages.
Explicit policy in robots.txt. One line of Content Signals states our position: search=yes, ai-input=yes, ai-train=yes
. Everything on this site is published for people and machines alike, and now that is machine-readable policy rather than an assumption. Whatever your position is, the point is to state it where software can read it.
A public, keyless MCP server. The site now runs a read-only Model Context Protocol endpoint at /mcp
: nine tools covering search, blog, customer stories, the reading list, and our Barkup Bench research, over stateless streamable HTTP with no authentication. There is nothing to sign up for because there is nothing private: the tools expose only content that is already public, and the CMS credentials never leave the server. An agent can call read_study
and get a benchmark study's sections directly instead of parsing our dashboard.
Discovery plumbing. The MCP server is advertised in a server card and manifest under /.well-known/
, an RFC 9727 API catalog lists the machine interfaces, HTTP Link headers on the homepage point to all of it (api-catalog
, service-doc
, service-desc
), and an Agent Skills index publishes a SKILL.md that teaches an agent how to query our research data, with a digest computed from the exact bytes the skill route serves so the index can never drift from the artifact.
Tools inside the page itself. Through the emerging WebMCP browser API, the site registers three in-page tools (site search, navigation, and an agent-resources directory) for browser-based agents. And underneath it all, the older hygiene got a refresh too: per-page JSON-LD (BlogPosting, Article, ScholarlyArticle for the research), five Atom feeds rendered at request time, and a feeds index. The full stack, as always, is on our Built With page.
The Score #
Running isitagentready.com against lightningjar.com today returns Level 4 of 5: "Agent-Integrated." Passing checks across all four dimensions: robots.txt, sitemap, Link relations, markdown negotiation, Content Signals, API catalog, MCP server card, Agent Skills, and WebMCP.
What separates us from Level 5 ("Agent-Native") is instructive about where the frontier sits. One item is in flight: DNS-AID, which advertises agent entrypoints in DNS itself, requires record types and DNSSEC that our current DNS host does not support, so that fix is a DNS migration with a runbook rather than a code change. The other two, an auth.md for agent registration and an A2A card for agent-to-agent discovery, describe sites that operate agents and accounts. For a marketing and research site with nothing to log into, we may deliberately stay one notch below the ceiling, and that is a legitimate answer too: agent readiness is about honestly representing what your site is, and ours is a library, a lab, and a storefront window.
What We Learned #
The job is smaller than it looks. Nearly everything above is headers, well-known files, and a handful of routes. There was no rebuild and no new vendor. If your content lives somewhere queryable, the whole kit is days of work.
Serve reality, never a copy. Our llms.txt, feeds, and markdown responses all render at request time from the same source of truth as the pages. The one place we publish a digest, it is computed from the same bytes the route serves. Anything generated as a snapshot will drift, and an agent reading a stale index is worse off than an agent reading none.
Answer probes honestly. Real agent clients probe endpoints with GET, OPTIONS, and DELETE before speaking. Our MCP route answers each verb with something true (a 405 with instructions, a CORS preflight, a stateless acknowledgment) instead of a 404. Several standards in this space are drafts, and the clients are drafts too; being forgiving at the edges is part of the job.
The paint is still wet. The WebMCP spec puts the API on document
, Chrome's preview puts it on navigator
, so we detect both. DNS hosts differ wildly in what record types they support. Caching needs care: every negotiated URL has to vary on the Accept header, or your CDN will happily serve markdown to a human. None of this is hard, but all of it is the kind of thing you only learn by shipping.
Most of all, agent-ready is mostly web-ready. The deepest lesson is how little of this is new. Sitemaps, feeds, structured data, clean text-first pages, honest HTTP: agents reward exactly what search engines rewarded, only more so, and we have argued before that LLMs now read your site as often as humans do. A site built on structured content and typography rather than image-heavy chrome was most of the way here before the acronyms arrived.
Where to Start #
If you run a website, the on-ramp is short and ordered: make sure robots.txt and a sitemap exist and are accurate; add a Content-Signal line stating your AI policy, whatever it is; publish an llms.txt; add structured data and a feed if you have recurring content; consider markdown negotiation if your content lives in a CMS; and stand up a read-only MCP endpoint the day you have data an agent might genuinely query. Then run the scanner and let it tell you what is missing. The web has renegotiated its readership twice before, and the sites that adapted early were the ones search engines learned to trust. The third renegotiation is underway. This time the new reader can talk back and act on the answer, and the sites that meet it halfway will be the ones it recommends.