{"slug": "braindump-extract-coding-rules-from-pr-review-comments", "title": "Braindump: Extract coding rules from PR review comments", "summary": "Pydantic's open-source tool Braindump extracts coding rules from GitHub PR review comments and generates AGENTS.md files, converting 4,668 review comments on pydantic/pydantic-ai between October 2025 and February 2026 into 149 rules at a cost of just over $60. The pipeline, optimized for the pydantic-ai repo, uses Claude for extraction and synthesis, with stages for download, extract, synthesize, dedupe, place, group, and generate, each resumable and configurable via thresholds like min_score (default 0.5).", "body_md": "Extract coding rules from PR review comments and generate `AGENTS.md`\n\nfiles for any GitHub repository.\n\nNote\n\nThis project is 100% vibecoded, and the pipeline and thresholds have been optimized for the [ pydantic-ai](https://github.com/pydantic/pydantic-ai) repo primarily.\n\nIf the pipeline generates unexpected rules for your repo, whether invalid or duplicate or otherwise unhelpful, you're encouraged to tell Claude (or your coding agent of choice) to investigate the issue (by referring to the generated rule IDs) and make changes to the pipeline until it does what you want.\n\nIt's expected that some teams will use their own fork of `braindump`\n\nthat evolves over time to meet their needs: there's no expectation that the version in this repo will work for absolutely everyone, so you don't need to upstream changes unless you believe they are strictly better for every user than what came before.\n\n```\nGitHub PR reviews → download → extract → synthesize → dedupe → place → group → generate → AGENTS.md\n```\n\n**Download**— fetch PR data (reviews, comments, diffs) via`gh`\n\nCLI**Extract**— use Claude to identify actionable changes and generalizable rules from each comment (bot comments are filtered out automatically)**Synthesize**— embed generalizations, cluster by similarity, extract validated rules. Each rule is scored based on the LLM's confidence multiplied by a factor for how many unique PRs the rule's evidence spans (1 PR = 0.6×, 2 = 0.85×, 3 = 0.95×, 4+ = 1.0×), so rules that came up across more reviews score higher.**Dedupe**— three-pass deduplication (embedding clusters + category review + post-consolidation review)** Place**— determine where each rule belongs (root, directory, file, cross-file), filtering by`min_score`\n\nfloor (default 0.5)**Group**— filter by`min_score`\n\nthreshold (default 0.5) and organize by topic for progressive disclosure**Generate**— rephrase rules and write final`AGENTS.md`\n\nfiles\n\nEach stage is resumable — if interrupted, it picks up from where it left off. Pass `--fresh`\n\nto any stage or `run`\n\nto wipe previous outputs and start clean.\n\nAs described in the [\"Fighting Fire With Fire: How We're Scaling Open Source Code Review at Pydantic With AI\"](https://pydantic.dev/articles/scaling-open-source-with-ai) blog post, we used `braindump`\n\nto turn the 4,668 PR review comments @DouweM made on `pydantic/pydantic-ai`\n\nbetween October 2025 and February 2026 into [149 rules](https://github.com/pydantic/pydantic-ai/blob/main/AGENTS.md#coding-guidelines) at a cost of just over $60:\n\n``` bash\n$ uv run braindump --repo pydantic/pydantic-ai run --since 2025-10-01 --authors DouweM --max-rules=150\n\n┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n┃Stage         ┃ Status    ┃ Details                                ┃ Updated   ┃       Cost┃\n┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n│download      │ done      │ 883 PRs | 10,020 review comments, 883  │ 11d ago   │           │\n│              │           │ diffs                                  │           │           │\n│extract       │ done      │ 4,668/10,020 comments → 3,851          │ 10d ago   │     $40.17│\n│              │           │ actionable, 817 rejected → 5,320       │           │           │\n│              │           │ generalizations                        │           │           │\n│synthesize    │ done      │ 5,320 generalizations → 3,004 in 1,054 │ 10d ago   │     $14.22│\n│              │           │ clusters, 2,316 unclustered → 1,238    │           │           │\n│              │           │ rules                                  │           │           │\n│              │           │               (similarity ≥ 0.65, min  │           │           │\n│              │           │ cluster size 2, coherence: 0.87)       │           │           │\n│dedupe        │ done      │ 1,238 → 1,014 rules (224 merged)       │ 4m ago    │      $5.12│\n│place         │ done      │ 1,014 → 197 rules placed (score ≥ 0.8) │ 1m ago    │      $2.14│\n│              │           │ | agents_md_root: 106, agents_md_dir:  │           │           │\n│              │           │ 85, cross_file: 5, file: 1             │           │           │\n│group         │ done      │ 150/197 rules (score ≥ 0.8) → 6        │ 0m ago    │      $0.09│\n│              │           │ locations | 109 inline, 40 in topics   │           │           │\n│generate      │ done      │ 6 AGENTS.md files, 3 topic docs (46    │ 0m ago    │      $0.83│\n│              │           │ KB) | root, docs, pydantic_ai_slim,    │           │           │\n│              │           │ pydantic_ai_slim/pydantic_ai,          │           │           │\n│              │           │ pydantic_ai_slim/pydantic_ai/models,   │           │           │\n│              │           │ tests                                  │           │           │\n└──────────────┴───────────┴────────────────────────────────────────┴───────────┴───────────┘\n\nTotal cost: $62.57\n\nPipeline complete!\n\nGenerated files:\n  data/pydantic/pydantic-ai/7-generate/AGENTS.md\n  data/pydantic/pydantic-ai/7-generate/agent_docs/api-design.md\n  data/pydantic/pydantic-ai/7-generate/agent_docs/code-simplification.md\n  data/pydantic/pydantic-ai/7-generate/agent_docs/documentation.md\n  data/pydantic/pydantic-ai/7-generate/docs/AGENTS.md\n  data/pydantic/pydantic-ai/7-generate/pydantic_ai_slim/AGENTS.md\n  data/pydantic/pydantic-ai/7-generate/pydantic_ai_slim/pydantic_ai/AGENTS.md\n  data/pydantic/pydantic-ai/7-generate/pydantic_ai_slim/pydantic_ai/models/AGENTS.md\n  data/pydantic/pydantic-ai/7-generate/tests/AGENTS.md\n```\n\n[uv](https://docs.astral.sh/uv/)for Python package management[GitHub CLI](https://cli.github.com/)(`gh`\n\n) authenticated for repo access- A\n[Pydantic AI Gateway](https://ai.pydantic.dev/gateway/)API token (or direct provider API keys — see[Model configuration](#model-configuration))\n\nThe `braindump`\n\nCLI is not currently published on PyPI, so the first step is to clone this repo locally. Then run:\n\n```\n# Install dependencies\nuv sync\n\n# Add your Pydantic AI Gateway token to .env\necho \"PYDANTIC_AI_GATEWAY_API_KEY=your-token\" > .env\n\n# Authenticate GitHub CLI (if not already)\ngh auth login\n```\n\nRun the full pipeline:\n\n```\nuv run braindump --repo pydantic/pydantic-ai run --since 2025-10-01\n```\n\nThis will include review comments by all non-bot authors; use `--authors`\n\nto limit this.\n\nThis will write all rules to `AGENTS.md`\n\nthat have a score of at least 0.5, which may end up being too many depending on how many source comments you have. To limit the output to the best rules, you can use the `--min-score`\n\noption. To determine an appropriate value that balances not missing important rules with not overloading the agent's context window, you can run the full pipeline, then use the [ group --preview](#group--organize-by-topic) command to show a table of rule counts and marginal examples at different score thresholds, and then run again from the group stage using\n\n`run --from group --fresh --min-score=<score>`\n\n.All stages support `--concurrency N`\n\nto control parallel LLM/API requests and `--fresh`\n\nto wipe previous outputs before running.\n\n```\nuv run braindump --repo owner/repo run [--from STAGE] [--skip STAGE ...] [--since DATE] [--authors USER] [--min-score 0.5] [--max-rules N] [--fresh]\n```\n\n`--from`\n\n: Start from a specific stage (e.g.`--from synthesize`\n\n)`--skip`\n\n: Skip stages (repeatable, e.g.`--skip download --skip extract`\n\n)`--since`\n\n: Date filter for download (YYYY-MM-DD)`--authors`\n\n: Author filter for extract (default:`all`\n\n)`--min-score`\n\n: Override rule score threshold (default: 0.5)`--max-rules N`\n\n: Cap the number of rules in group stage (top-scored)`--fresh`\n\n: Wipe all stage outputs and start from scratch\n\n```\nuv run braindump --repo owner/repo download [--since YYYY-MM-DD] [--concurrency 5]\nuv run braindump --repo owner/repo extract [--authors USER] [--limit N] [--random] [--prs 1,2,3] [--concurrency 10]\n```\n\n`--prs`\n\n: Filter to specific PR numbers (comma-separated)`--limit N`\n\n: Limit number of comments to process`--random`\n\n: Randomly sample comments (with`--seed`\n\nfor reproducibility)\n\n```\nuv run braindump --repo owner/repo synthesize [--similarity-threshold 0.65] [--min-cluster-size 3] [--concurrency 10]\nuv run braindump --repo owner/repo dedupe [--similarity-threshold 0.75] [--concurrency 10]\nuv run braindump --repo owner/repo place [--min-score 0.5] [--concurrency 10]\nuv run braindump --repo owner/repo group [--min-score 0.5] [--max-rules N] [--preview]\n```\n\n`--min-score`\n\n: Minimum rule score to include (default: 0.5)`--max-rules N`\n\n: Cap the number of rules included (top-scored). Applied after`--min-score`\n\nfiltering.`--preview`\n\n: Show a table of rule counts and marginal examples at different score thresholds, then exit without running the LLM grouping. Useful for picking an appropriate`--min-score`\n\nor`--max-rules`\n\n.\n\nRe-run from group with a different threshold to adjust how many rules end up in the output — no need to re-run place:\n\n```\nuv run braindump --repo owner/repo run --from group --min-score 0.6\nuv run braindump --repo owner/repo run --from group --max-rules 80\nuv run braindump --repo owner/repo generate [--dry-run] [--concurrency 10]\nuv run braindump --repo owner/repo status\n```\n\nShows what data exists per stage, key metrics (including cost), and suggests the next stage to run. Stages whose inputs have changed since they last ran are marked as stale.\n\n```\nuv run braindump --repo owner/repo lookup 42\nuv run braindump --repo owner/repo lookup --search isinstance\nuv run braindump --repo owner/repo lookup --location root\n```\n\nAll per-repo data lives under `data/<owner>/<repo>/`\n\n:\n\n```\ndata/\n  pydantic/\n    pydantic-ai/\n      1-download/         # Raw GitHub API data (PRs, diffs, review comments)\n      2-extract/          # extractions.jsonl, checkpoint.json\n      3-synthesize/       # rules.jsonl, embeddings, clusters\n      4-dedupe/           # rules.jsonl (deduped), embeddings, merge_log\n      5-place/            # placements.jsonl\n      6-group/            # organized_rules.json\n      7-generate/         # Final AGENTS.md files\n      rule_overrides.jsonl\n```\n\nPlace manual rule corrections in `data/<owner>/<repo>/rule_overrides.jsonl`\n\n:\n\n```\n{\"text\": \"Use tuple syntax for isinstance() checks, not | union\"}\n{\"text\": \"Always use explicit re-exports in __init__.py\", \"reason\": \"Enforced by our linter\", \"category\": \"code_style\", \"scope\": \"global\"}\n```\n\nOnly `text`\n\nis required. Optional fields: `reason`\n\n(default: `\"Manual override\"`\n\n), `category`\n\n(default: `\"general\"`\n\n), `scope`\n\n(default: `\"global\"`\n\n), `example_bad`\n\n, `example_good`\n\n.\n\nOverrides cluster with similar extracted rules and replace them during the `dedupe`\n\nstage. After adding or changing overrides, re-run from dedupe:\n\n```\nuv run braindump --repo owner/repo run --from dedupe --fresh\n```\n\nBy default, braindump uses the [Pydantic AI Gateway](https://ai.pydantic.dev/gateway/) with `anthropic:claude-sonnet-4-5`\n\nfor LLM tasks and `openai:text-embedding-3-small`\n\nfor embeddings. Both models are configurable via flags or environment variables:\n\n```\n# Use flags\nuv run braindump --model anthropic:claude-sonnet-4-5 --embedding-model openai:text-embedding-3-small --repo owner/repo run\n\n# Or use environment variables\nexport BRAINDUMP_MODEL=anthropic:claude-sonnet-4-5\nexport BRAINDUMP_EMBEDDING_MODEL=openai:text-embedding-3-small\n```\n\nModel strings follow the [pydantic-ai format](https://ai.pydantic.dev/models/): `provider:model-name`\n\n. Examples:\n\n`gateway/anthropic:claude-sonnet-4-5`\n\n(default LLM, uses Pydantic AI Gateway)`gateway/openai:text-embedding-3-small`\n\n(default embeddings, uses Pydantic AI Gateway)`anthropic:claude-sonnet-4-5`\n\n(direct Anthropic)`openai:text-embedding-3-small`\n\n(direct OpenAI)\n\nWhen using `gateway/...`\n\nmodels, set `PYDANTIC_AI_GATEWAY_API_KEY`\n\n. For direct providers, set the provider-specific key (e.g. `OPENAI_API_KEY`\n\n, `ANTHROPIC_API_KEY`\n\n).\n\nBraindump is automatically instrumented with [Pydantic Logfire](https://pydantic.dev/logfire) for tracing and observability. All LLM calls, HTTP requests, and pipeline stages are traced.\n\nTo enable, authenticate and select a project:\n\n```\nuv run logfire auth\nuv run logfire projects use\n```\n\nTraces are sent only when a Logfire token is present — if you skip this step, everything still works, just without tracing.", "url": "https://wpnews.pro/news/braindump-extract-coding-rules-from-pr-review-comments", "canonical_source": "https://github.com/pydantic/braindump", "published_at": "2026-08-27 16:36:19+00:00", "updated_at": "2026-08-27 16:49:40.372873+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "machine-learning"], "entities": ["Pydantic", "Braindump", "pydantic/pydantic-ai", "Claude", "DouweM", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/braindump-extract-coding-rules-from-pr-review-comments", "markdown": "https://wpnews.pro/news/braindump-extract-coding-rules-from-pr-review-comments.md", "text": "https://wpnews.pro/news/braindump-extract-coding-rules-from-pr-review-comments.txt", "jsonld": "https://wpnews.pro/news/braindump-extract-coding-rules-from-pr-review-comments.jsonld"}}