{"slug": "making-your-docs-site-agent-readable-llms-txt-mcp-and-the-well-known-files-that", "title": "Making your docs site agent-readable: llms.txt, MCP, and the .well-known files that actually matter", "summary": "OrchestKit's documentation site implements a stack of standard files—llms.txt, OpenAPI specs, MCP endpoints, and .well-known identity files—to make its docs machine-readable for AI agents. The project, a free MIT-licensed plugin for Claude Code, publishes these resources at predictable paths so agents can fetch structured data without scraping HTML. The approach includes agent-card.json, a schema.org graph, and explicit robot.txt allowances for AI crawlers.", "body_md": "AI agents increasingly read your docs *instead of* a human. If your documentation site only emits HTML for a browser, an agent has to scrape and guess. There's a better surface — and most of it is a handful of small, standard files. Here's the full stack we ship on the OrchestKit docs site, why each piece exists, and how to verify it.\n\n`llms.txt`\n\n— the agent's table of contents\nA plain-text index at `/llms.txt`\n\n: what the product is, its constraints, and a link map to every machine-readable resource. Keep it under ~30k chars; put the exhaustive page list in `/docs/llms.txt`\n\nand the full corpus in `/llms-full.txt`\n\n. The win: an agent gets oriented in one fetch instead of crawling.\n\nAppend `.md`\n\nto any page URL (or send `Accept: text/markdown`\n\n) and return the raw Markdown. Agents get clean tokens; humans still get the rendered page.\n\nEven a docs site has an API surface (search, page fetch). Publish an OpenAPI document at a predictable path so an agent can call it without reverse-engineering. Pair it with RFC 9727 — a `/.well-known/api-catalog`\n\nlinkset that enumerates every API entry point.\n\nThe Model Context Protocol lets agents call your tools natively. We expose a read-only MCP server over Streamable HTTP at `/api/mcp`\n\nplus a discovery `server-card.json`\n\n. Two tools — search docs, get a doc by id — are enough to be useful.\n\n`.well-known`\n\nidentity files\n`agent-card.json`\n\n(A2A): declares your agent skills.`agent-skills/index.json`\n\n: the Agent Skills Discovery RFC, with a SHA-256 digest per skill so a consumer can verify it.`oauth-protected-resource`\n\n(RFC 9728): if your API is anonymous, `authorization_servers`\n\nis a positive signal, not an omission.Emit a `schema.org`\n\ngraph (`Organization`\n\n, `SoftwareApplication`\n\n, `WebSite`\n\n) linked by `@id`\n\n, with `sameAs`\n\npointing at the registries that already verify you (GitHub, your package registry, Wikidata). One canonical Organization block, reused everywhere, so the graph never sees conflicting identifiers. Never fabricate an `aggregateRating`\n\n— surface real signals (e.g. GitHub stars as an `InteractionCounter`\n\n) instead.\n\n`robots.txt`\n\nExplicitly allow the named AI crawlers you want (GPTBot, ClaudeBot, OAI-SearchBot, Google-Extended…), and emit a `Content-Signal`\n\ndirective. Link your sitemap and a schema-map.\n\n`curl -s https://yoursite/llms.txt`\n\n, fetch each `.well-known`\n\npath, and run your JSON-LD through a structured-data validator. If you build on Claude Code, the open-source **OrchestKit** docs site implements every item above — the source is on GitHub, MIT-licensed, and you can read the route handlers directly.\n\n*I maintain OrchestKit (a free, MIT plugin for Claude Code, 111 skills/37 agents/210 hooks). The agent-discovery surface described here is what its docs site ships today.*", "url": "https://wpnews.pro/news/making-your-docs-site-agent-readable-llms-txt-mcp-and-the-well-known-files-that", "canonical_source": "https://dev.to/yonyonai/making-your-docs-site-agent-readable-llmstxt-mcp-and-the-well-known-files-that-actually-matter-33c6", "published_at": "2026-06-14 17:13:20+00:00", "updated_at": "2026-06-14 17:40:37.115357+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "large-language-models", "ai-infrastructure", "ai-products"], "entities": ["OrchestKit", "Claude Code", "GitHub", "OpenAPI", "Model Context Protocol", "GPTBot", "ClaudeBot", "OAI-SearchBot"], "alternates": {"html": "https://wpnews.pro/news/making-your-docs-site-agent-readable-llms-txt-mcp-and-the-well-known-files-that", "markdown": "https://wpnews.pro/news/making-your-docs-site-agent-readable-llms-txt-mcp-and-the-well-known-files-that.md", "text": "https://wpnews.pro/news/making-your-docs-site-agent-readable-llms-txt-mcp-and-the-well-known-files-that.txt", "jsonld": "https://wpnews.pro/news/making-your-docs-site-agent-readable-llms-txt-mcp-and-the-well-known-files-that.jsonld"}}