cd /news/developer-tools/i-pulled-480-real-agents-md-claude-m… · home topics developer-tools article
[ARTICLE · art-94205] src=dev.to ↗ pub= topic=developer-tools verified=true sentiment=· neutral

I pulled 480 real AGENTS.md/CLAUDE.md files from 272 repos — and a regex bug taught me not to trust my own heuristics

A developer released a corpus of 480 real AGENTS.md and CLAUDE.md instruction files pulled from 272 production repositories, including React, Grafana, LangChain, and Home Assistant. The project, hosted on GitHub, stores files verbatim under permissive licenses and metadata-only for others, and a regex bug in the analysis pipeline taught the developer not to trust heuristics without large-scale validation.

read2 min views1 publishedAug 12, 2026

A corpus of real AGENTS.md / CLAUDE.md files, the instruction files coding agents like Claude Code, Copilot, and friends read before touching a repo, pulled verbatim from 272 production repos (React, Grafana, LangChain, Home Assistant, and 268 others). Not a curated list of links: the actual files, stored in full wherever the license permits it, greppable in one tree.

Repo: https://github.com/sattva2020/agents-md-in-the-wild One analysis pass tries to detect literal directory-tree diagrams, the kind with box-drawing characters or ASCII branch markers. First version was naive: look for tree-shaped characters on a line, count matches. Ran it across the corpus and got 69% of files "containing a directory map." That felt way too high, a directory tree in an instructions file is a specific, deliberate thing, not something 7 in 10 projects bother writing.

Turned out the regex was matching markdown tables. A row like | src/ | entry point |

has a pipe and something path-shaped on the left, and my "line looks like a tree branch" check didn't care what came after the first slash. Fix: require a run of 3+ consecutive tree-shaped lines, and explicitly reject lines that parse as table rows before counting them. Dropped the number from 69% to 17%, which matches manual spot-checks.

Same story with the "mentions secrets" heuristic. First cut matched bare "token," which turned out to catch files talking about LLM context-window tokens, not API tokens. Had to tighten the pattern to require "api," "access," or similar nearby.

Lesson, twice over: a keyword or shape that looks specific to you is rarely as specific as it looks once you run it against real-world text at scale. 480 files was enough to expose both bugs, I doubt either would've shown up against the 5-10 examples I originally tested with.

Storage is tiered by SPDX license: files under a curated allowlist of about 26 redistributable licenses (MIT, Apache-2.0, BSD variants, GPL/LGPL/AGPL, CC0, etc.) are stored verbatim with provenance. Everything else, including repos with no declared license, gets metadata-only storage: headings and structure, no body text. Default is "all rights reserved" unless the license says otherwise; unlicensed files still get counted in the structural analysis, they just don't get their text redistributed.

It's meant to refresh weekly via a GitHub Action, full disclosure, that's currently stuck behind a billing issue on my GitHub account, so treat "weekly" as aspirational until I sort that out.

Genuinely curious what's missing, or whether there's a pattern signal worth adding. Repo again: https://github.com/sattva2020/agents-md-in-the-wild

── more in #developer-tools 4 stories · sorted by recency
── more on @github 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/i-pulled-480-real-ag…] indexed:0 read:2min 2026-08-12 ·