{"slug": "the-state-of-agents-md-scoring-the-16-biggest-ai-agent-repos-own-instructions", "title": "The State of AGENTS.md: scoring the 16 biggest AI agent repos' own instructions", "summary": "A systematic evaluation of 16 major AI coding-agent repositories found that none achieved an A grade for their AGENTS.md instruction files, with a mean score of 70.0 and a median grade of C. The openai/codex repo, which popularized the AGENTS.md convention, ranked 13th out of 16 with a score of 66.4, while the best-scoring file belonged to a smaller library with only 170 stars. The scoring rubric, released as open-source software, measures structure, operational coverage, and efficiency, revealing that most repos lack critical operational instructions for agents.", "body_md": "16 repos. About 1.46 million GitHub stars between them. Zero A grades.\n\n`AGENTS.md`\n\nhas quietly become the cross-tool standard for repo-level agent instructions — a single file that Cursor, Codex, Copilot, Claude Code and a growing list of others read on startup. So I went looking for how the AI-agent ecosystem itself uses the convention it created. I swept 36 of the best-known AI coding-agent and agent-framework repositories, found 16 with at least 20k stars shipping a root `AGENTS.md`\n\n, and scored every file with a deterministic engine — no LLM judge, same file same score on every machine, reproducible against the exact commit SHAs at the bottom of this post.\n\nThe result: mean 70.0, median grade C, not a single A. The repo whose team popularized the convention ranks 13th of 16. And the best `AGENTS.md`\n\nI could find anywhere belongs to a library with 170 stars.\n\nThis is the same scorer I used to [grade my own merged MCP contribution](/writing/scoring-my-own-mcp-contribution) and the one behind [the case for static instruction-file security](/writing/instruction-file-security). It is [open source, MIT, stdlib-only](https://github.com/Zandereins/schliff): the rubric is a dict you can read, not a hidden prompt.\n\n## The ranking\n\nThe set is every well-known AI coding-agent or agent-framework project at or above 20k stars that ships a root `AGENTS.md`\n\n, each scored at its HEAD on 2026-07-07 with released `schliff==8.5.0`\n\n. The AGENTS.md rubric is three dimensions: structure (weight 0.4), operational coverage (0.4), and efficiency (0.2).\n\n| # | Repo | Stars | Score | Grade | structure | op. coverage | efficiency |\n|---|---|---|---|---|---|---|---|\n| 1 | block/goose | 51k | 84.2 | B | 95 | 75 | 81 |\n| 2 | langchain-ai/langchain | 141k | 81.0 | B | 85 | 85 | 65 |\n| 3 | ComposioHQ/composio | 29k | 79.6 | B | 85 | 75 | 78 |\n| 4 | sst/opencode | 183k | 77.8 | B | 95 | 65 | 69 |\n| 5 | QwenLM/qwen-code | 26k | 77.8 | B | 85 | 75 | 69 |\n| 6 | n8n-io/n8n | 195k | 77.8 | B | 85 | 75 | 69 |\n| 7 | langgenius/dify | 148k | 75.2 | B | 90 | 55 | 86 |\n| 8 | Significant-Gravitas/AutoGPT | 185k | 74.2 | C | 95 | 45 | 91 |\n| 9 | All-Hands-AI/OpenHands | 80k | 72.4 | C | 80 | 75 | 52 |\n| 10 | Kilo-Org/kilocode | 26k | 71.6 | C | 85 | 75 | 38 |\n| 11 | zed-industries/zed † | 87k | 70.0 | C | 95 | 45 | 70 |\n| 12 | browser-use/browser-use | 103k | 69.4 | C | 75 | 75 | 47 |\n| 13 | openai/codex | 96k | 66.4 | C | 85 | 55 | 52 |\n| 14 | crewAIInc/crewAI | 55k | 55.2 | D | 80 | 15 | 86 |\n| 15 | huggingface/smolagents | 28k | 50.0 | D | 75 | 0 | 100 |\n| 16 | RooCodeInc/Roo-Code | 24k | 37.0 | E | 75 | 0 | 35 |\n\nSeven B, six C, two D, one E. No A, no S. † zed ships `AGENTS.md`\n\nas a symlink to its `.rules`\n\nfile; the score is for the resolved content, and the symlink itself is a finding — a tool that fetches the raw `AGENTS.md`\n\ngets the six-character string `.rules`\n\n, not instructions.\n\nOne repo was excluded as a non-independent data point: OpenInterpreter’s `AGENTS.md`\n\nis byte-identical to openai/codex’s — the same git blob, since the project is a codex derivative for open models. Counting it twice would have been dishonest.\n\n## The standard-bearer paradox\n\n`openai/codex`\n\n— the repo whose team did the most to popularize `AGENTS.md`\n\n— scores 66.4, a C, 13th of 16. The file is not lazy: 22.5 KB across 28 headings, genuinely rich in review conventions and TUI style rules. But the operational-coverage dimension asks a narrow question: does this file equip an agent to *operate* the repo? Test and lint commands are there (`just test`\n\n, `just fix`\n\n). A build command is not. It appears nowhere in the file — for a Rust and Bazel monorepo. There are no commit or PR conventions either. I filed the gaps as [an issue with an offer to fix](https://github.com/openai/codex/issues/31031).\n\nThe pattern generalizes across all sixteen. Structure is uniformly strong — 75 to 95, because headings are easy. What separates the top of the table from the bottom is operational coverage: the real setup, build, and test commands, and the gotchas an agent cannot cheaply rediscover at runtime.\n\n## Famous frameworks, empty operations\n\nThree files earn a D or E, and each survived me reading the raw Markdown by hand — these are not scorer artifacts:\n\n**crewAI (55.2, D):** 1 KB of contribution platitudes — “follow best practices”, DRY, YAGNI. The only concrete commands are for the docs site (`mintlify dev`\n\n). Nothing tells an agent how to set up, build, or test the actual Python framework. Operational coverage: 15 of 100.**smolagents (50.0, D):** the entire file is 157 bytes — four bullet points of generic advice (“Be Pythonic”, “follow OOP principles”). Operational coverage: 0.**Roo-Code (37.0, E):** 445 bytes — one genuinely useful UI-state gotcha and nothing else. No setup, no build, no test. Operational coverage: 0.\n\nA high structure score with zero operational coverage is the signature of a file written for a human skim, not for an agent that has to run the project.\n\n## The best file in the sample has 170 stars\n\nNot one of the sixteen giants reaches an A. The best `AGENTS.md`\n\nin my entire sample — the corpus below plus these sixteen — belongs to [maxcountryman/underway](https://github.com/maxcountryman/underway), a Postgres-backed job queue with 170 stars: **91.0, grade A**. It is 1,728 tokens covering setup, build, test, lint, and project structure as copy-pasteable commands.\n\n`block/goose`\n\n(84.2, B) shows the same shape among the big repos: 4.8 KB, almost five times smaller than codex’s file, covering Setup, Commands (Build / Test / Lint), Structure, and a Development Loop. Operational beats encyclopedic. An A-grade instruction file costs discipline, not headcount.\n\n(For calibration, and fully disclosed: the scorer’s own repo file scores 91.6. That is the ceiling I know how to reach — and a 170-star library reached it independently, which is the more interesting fact.)\n\n## The wild is worse\n\nBefore the sixteen, I scored a 30-file corpus of public `AGENTS.md`\n\nfiles pulled from GitHub code search (collected 2026-04, re-scored on 8.5.0): mean 61.5, median 61.4. Seventeen of the thirty land at D or below. In aggregate, the median public `AGENTS.md`\n\nis a D-plus — prose that reads fine and operates nothing.\n\n## ”Can’t you just game a scorer like this?”\n\nYou could, and an earlier version of this scorer fell for exactly the obvious attack: well-formed headings with junk commands stuffed into fenced code blocks. `echo`\n\n, `ls`\n\n, `pwd`\n\nin a fenced bash block scored 92.5, an A, before the operational-coverage dimension existed — a failure I [wrote into the spec](https://github.com/Zandereins/schliff/blob/main/docs/specs/agents-md-operational-coverage.md) rather than quietly patch. Today that same file scores 56.0, a D, with operational coverage 0: read-only verbs earn nothing, and full credit requires command *diversity*.\n\nOne honest limit remains, documented and pinned by a test: a *plausible fabrication* — invented but realistic-looking commands in well-formed sections — is indistinguishable from a real minimal file by any static scorer. The guarantee is scoped precisely: worthless text cannot outrank operational text. That is a smaller claim than “this measures quality”, and it is deliberately smaller.\n\n## What this measures, and what it does not\n\n**The rubric is opinionated, and its weights are version one.** The 0.4 / 0.4 / 0.2 headline was calibrated against a 30-file corpus without ground-truth labels. The validity evidence today is case-study-level — documented before-and-afters where the score moved with real fixes — not benchmark-level. If you disagree with a weight, the rubric is a readable dict and you can point at the line.**Structure, not truth.** The scorer cannot verify that a documented command actually runs. It reads form, not behavior.**One snapshot.** Every score is for the exact HEAD commit listed below, on engine 8.5.0. Files change; re-run before quoting.**A low score can be a deliberate choice.** A minimal file is a legitimate decision — it simply will not equip an agent, which is the only thing the number claims. In a separate audit, a 14.6k-token skill file kept its efficiency dimension at 47 because inlining was load-bearing. That was an informed decline, not a defect.**The sweep is broad, not exhaustive.** I checked 36 candidates. If I missed a repo above 20k stars with a root`AGENTS.md`\n\n, tell me and I will score it.\n\n## Reproduce it\n\n```\npip install schliff==8.5.0\ncurl -sO https://raw.githubusercontent.com/openai/codex/main/AGENTS.md\nschliff score AGENTS.md\n```\n\nOr paste any file into the [playground](https://schliff-playground.vercel.app), or drop the live badge into your own repo. The whole point of using a deterministic score instead of a judge model is that this table is checkable — every number above comes out of a rule engine you can read, pin, and argue with.\n\n## Appendix: the sweep\n\n**Scored (16):** the table above. **Excluded (1):** OpenInterpreter/open-interpreter — `AGENTS.md`\n\nbyte-identical to openai/codex’s. **Checked, no root AGENTS.md (19):** cline, Aider-AI/aider, continuedev/continue, google-gemini/gemini-cli, anthropics/claude-code, microsoft/vscode-copilot-chat, microsoft/autogen, geekan/MetaGPT, microsoft/semantic-kernel, TabbyML/tabby, AntonOsika/gpt-engineer, SWE-agent/SWE-agent, stackblitz/bolt.new, Pythagora-io/gpt-pilot, reworkd/AgentGPT, TransformerOptimus/SuperAGI, getcursor/cursor, stitionai/devika, plandex-ai/plandex — itself a finding: more than half of the best-known agent repos I checked do not ship the ecosystem’s own standard file at the repo root.\n\nHEAD commits, 2026-07-07: goose `f96f62d9`\n\n· langchain `2d8100c4`\n\n· composio `040ffd49`\n\n· opencode `1c25b2f2`\n\n· qwen-code `40340ef5`\n\n· n8n `66ad8b93`\n\n· dify `6edce14e`\n\n· AutoGPT `e2711b17`\n\n· OpenHands `cc80397e`\n\n· kilocode `b0348cbc`\n\n· zed `fc827a21`\n\n· browser-use `052787f9`\n\n· codex `cca16a10`\n\n· crewAI `799ab0f5`\n\n· smolagents `526069c1`\n\n· Roo-Code `b867ec91`\n\n· underway `89e9bf9f`\n\n.", "url": "https://wpnews.pro/news/the-state-of-agents-md-scoring-the-16-biggest-ai-agent-repos-own-instructions", "canonical_source": "https://fpaul.dev/writing/state-of-agents-md-2026/", "published_at": "2026-07-13 13:16:03+00:00", "updated_at": "2026-07-13 13:35:26.586904+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "ai-research"], "entities": ["openai/codex", "block/goose", "langchain-ai/langchain", "ComposioHQ/composio", "sst/opencode", "QwenLM/qwen-code", "n8n-io/n8n", "langgenius/dify"], "alternates": {"html": "https://wpnews.pro/news/the-state-of-agents-md-scoring-the-16-biggest-ai-agent-repos-own-instructions", "markdown": "https://wpnews.pro/news/the-state-of-agents-md-scoring-the-16-biggest-ai-agent-repos-own-instructions.md", "text": "https://wpnews.pro/news/the-state-of-agents-md-scoring-the-16-biggest-ai-agent-repos-own-instructions.txt", "jsonld": "https://wpnews.pro/news/the-state-of-agents-md-scoring-the-16-biggest-ai-agent-repos-own-instructions.jsonld"}}