{"slug": "i-labeled-558-agents-md-files-here-s-what-they-say-and-what-almost-nobody-writes", "title": "I labeled 558 AGENTS.md files. Here's what they say — and what almost nobody writes down", "summary": "A developer collected 558 public AGENTS.md files and labeled them against a nine-category taxonomy using an auditable rule-based classifier, achieving 92% precision and 70% recall on 55 held-out English files and 88%/73% on 50 Chinese files. The most common categories were prohibitions (85.7%) and build/test commands (82.8%), while gotchas (13.6%) and rules about agent behavior (25.8%) were rarest. The labeled dataset and an accompanying agent-charters tool were released so the numbers can be recomputed.", "body_md": "**TL;DR** — I collected **558 `AGENTS.md` files** from public repos and labeled each one against a 9-category\n\ntaxonomy with a **rule-based** classifier (no LLM in the loop, so it is auditable and recomputable). Then I\n\nblind-labeled held-out samples and compared: **92% precision / 70% recall** on 55 English files,\n\n**88% / 73%** on 50 Chinese files. The most common categories are prohibitions (**85.7%**) and build/test\n\ncommands (**82.8%**). The rarest: **gotchas (13.6%)** and instructions about how the agent itself should behave\n\n(**25.8%**).\n\nAlmost every discussion about `AGENTS.md` is anecdote-led: *my* repo's file works, *my* agent ignores it,\n\na good one is a model upgrade, a bad one is worse than nothing. All of that may be true — but nobody\n\nseems to have the distribution. So I built it: snapshot of 558 files from 558 public repos\n\n(2026-09-10, 5.3 MB, **516 usable for statistics**), labeled, versioned, and published with the tooling.\n\nNine categories: `boundaries`, `build_test`, `workflow`, `structure`, `style`, `environment`, `overview`,\n\n`agent_meta` (rules about the AI itself), `gotchas`. Labeling is done by pattern rules over headings and\n\nbody text — deliberately, because a rule set can be read, argued with, and re-run, and every number below\n\ncan be recomputed from the released dataset. I then measured how well the rules match a human reading:\n\n**100 files in-sample** (upper bound, 90%/75%) and two held-out sets I had never tuned against —\n\n55 English (92%/70%) and 50 Chinese (88%/73%). Held-out numbers use the *conservative* reading\n\n(items I was unsure about count as classifier errors).\n\n| category | share of 516 files | \n|---|---|\n| `boundaries` (what you must never do) | **85.7%** | \n| `build_test` (install/build/test/CI commands) | **82.8%** | \n| `workflow` (branching, commits, review, release) | 67.1% | \n| `structure` | 59.1% | \n| `style` | 54.5% | \n| `environment` | 45.0% | \n| `overview` | 32.2% | \n| `agent_meta` | 25.8% | \n| `gotchas` | 13.6% | \n\nThe 2.9 pp gap between the top two is *smaller* than the known false-positive rate (~3%) of the\n\nprohibition pattern — so the honest statement is **tied for first**, not \"prohibitions beat build commands\".\n\n`gotchas` is dead last at 13.6%. Worse: when people *do* open a \"known issues\" section, a third of it\n\nisn't a gotcha. I hand-read 120 items from those sections:\n\nThat 8% is the part an agent can never derive from the code — and it is exactly the part that is\n\nalmost never written down.\n\n`workflow`\nIn a controlled experiment (11 repos × 3 prompt styles), prompts that listed topics explicitly produced\n\n**9/9 categories**, while prompts that left the slots implicit skipped `workflow` in **11 out of 11** files.\n\nPoint at `workflow` by name and it appears **3/3** times, with real content. The gap is not knowledge,\n\nit is *questions* — which is why I turned the corpus distribution into a checklist tool.\n\nEnglish files fail differently from Chinese ones. English: `gotchas` recall 32–38% — the classifier\n\nmisses casual \"watch out\" prose. Chinese: `agent_meta` recall **26%** — Chinese files express agent rules\n\nin the second person (\"you are the dispatcher, not the executor\"), and the body-pattern rules for that\n\ncategory are entirely English, so the whole style is invisible to them. File-level exact agreement\n\n(9/9 categories identical) is **12%** in both languages.\n\n**49%** point to some other file; **15%** route to a knowledge or rules directory. That's a structural\n\nfact about the format, and it means \"does this repo have an `AGENTS.md`?\" is a much weaker question than\n\n\"what is actually in it\".\n\n```\npip install agent-charters\n\nagent-charters brief      # checklist of the 9 slots + a paste-ready prompt\nagent-charters compare your-AGENTS.md   # your coverage vs the 558-file baseline\nagent-charters refs your-AGENTS.md      # does your file point at paths that exist\n```\n\nHonest note: `compare` is a **checklist, not an oracle**. It warned me that one of the nine categories was\n\nmissing from a file I wrote myself — it was actually present, but the heading used the tool's own slot name\n\ninstead of natural language. That is documented in the repo (along with the exact experiment) rather than\n\nquietly patched, because a tool that tells you \"you're missing X\" should be checked by a human.\n\n`LIMITATIONS.md` in the repo\nI'm looking for **2–3 people who are not me** to run `compare` on an `AGENTS.md` they actually maintain and\n\ntell me where it's wrong — missing a category you clearly have, or claiming one you don't. That is the one\n\npiece of evidence this project doesn't have yet: an external user. Issues and comments are both fine.", "url": "https://wpnews.pro/news/i-labeled-558-agents-md-files-here-s-what-they-say-and-what-almost-nobody-writes", "canonical_source": "https://dev.to/janzong/i-labeled-558-agentsmd-files-heres-what-they-say-and-what-almost-nobody-writes-down-34gb", "published_at": "2026-09-14 12:23:13+00:00", "updated_at": "2026-09-14 12:38:51.975604+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-research"], "entities": ["agent-charters"], "alternates": {"html": "https://wpnews.pro/news/i-labeled-558-agents-md-files-here-s-what-they-say-and-what-almost-nobody-writes", "markdown": "https://wpnews.pro/news/i-labeled-558-agents-md-files-here-s-what-they-say-and-what-almost-nobody-writes.md", "text": "https://wpnews.pro/news/i-labeled-558-agents-md-files-here-s-what-they-say-and-what-almost-nobody-writes.txt", "jsonld": "https://wpnews.pro/news/i-labeled-558-agents-md-files-here-s-what-they-say-and-what-almost-nobody-writes.jsonld"}}