{"slug": "how-we-validate-12-agent-config-kits-and-what-validation-can-t-do", "title": "How we validate 12 agent config kits (and what validation can't do)", "summary": "A developer published the specification for a validation harness that checks agent configuration kits before release, verifying required files, a sub-300-line CLAUDE.md budget, parseable Cursor frontmatter, baseline sections, absence of placeholder text and unrendered templates, and non-trivial skills. The validator, which exits non-zero to block releases and supports JSON output for CI, caught a real generation bug that left a literal placeholder in every kit's definition of done. The author notes the harness cannot judge whether the advice is good or check semantics, arguing that overclaiming validation is how trust dies.", "body_md": "*Our validator checks every kit before release: files, budgets, frontmatter, sections, placeholders. Here's exactly what it checks,\nwhy those checks exist, and — more interestingly — what it cannot\ncheck, because pretending otherwise is how trust dies.*\n\nAgent config has a trust problem, and it's deserved. The market is\n\nfull of markdown with confident READMEs and no contract. \"Validated\n\nagainst real projects!\" is the new \"military-grade encryption.\" So\n\ninstead of adjectives, this post is the actual spec of our validation\n\nharness — the same script that ships in the pack — including the\n\nfailure it caught, and the honest list of what it does *not* check.\n\nA kit is a set of files with structure. Structure is checkable;\n\ntherefore the contract is structural. Every kit must pass all of:\n\n**1. Required files exist.**\n\n`AGENTS.md`, `CLAUDE.md`, `copilot-instructions.md`, and a\n\n`.cursor/rules/` directory with 3–5 `.mdc` files. Trivial? Every\n\nmissing-file bug in our own builds was \"trivial\" until it shipped.\n\n**2. The CLAUDE.md line budget.**\n\nFewer than 300 lines, enforced, not suggested. The number is a design\n\ndecision (see the compliance-curve argument in our longer piece), but\n\nthe enforcement is the point: budgets without gates become\n\nsuggestions, and suggestions become 900-line files.\n\n**3. Cursor frontmatter parses, with a declared activation mode.**\n\nEach `.mdc` must open with parseable YAML frontmatter containing a\n\nnon-empty `description`, and either `alwaysApply: true|false` or a\n\n`globs` entry. This check exists because of the nastiest silent\n\nfailure in the format: a malformed frontmatter block doesn't error —\n\nCursor treats the file as broken or ignores it, and a rule you\n\nbelieve is protecting your test files simply never loads. The\n\nvalidator makes \"the rule never loads\" a build failure.\n\n**4. The baseline has its sections.**\n\n`AGENTS.md` must contain: Project context, Commands, Code style,\n\nTesting expectations, Commit conventions, Safety rules, Definition of\n\ndone. These aren't bureaucratic — they're the sections agents act on\n\nmost (the command table above all). A kit missing \"Safety rules\" is\n\nmissing the file's most important content, so it fails.\n\n**5. No placeholder text, anywhere.**\n\n`TODO`, `TBD`, `FIXME`, `lorem ipsum`, `XXX` — case-insensitive,\n\nacross every kit file. Config rot starts as a placeholder that\n\noutlived its author.\n\n**6. No unrendered templates.**\n\nThe kits are generated, so generation bugs are a risk class: the\n\nvalidator scans for unrendered variables (`{lint_cmd`, and friends)\n\nin output. This check exists because it caught exactly that bug —\n\na double-formatting trap left the literal text `lint_cmd_placeholder`\n\ninside every kit's definition-of-done. The validator flagged it on\n\nits first real run; the fix took one line; the check stays forever.\n\n**7. Skills exist and are non-trivial.**\n\nAll eight workflow skills must be present and above a minimum size.\n\nA stub skill file is a broken promise.\n\nOutput: human-readable by default, `--json` for CI. Exit code is the\n\nrelease gate: non-zero, no release.\n\nA validator that has never failed is a validator that has never been\n\ntested. Before trusting ours, we negative-tested it — deliberately\n\ncorrupting a kit and watching:\n\n`TODO: fix this` to a kit's AGENTS.md → That loop — break, catch, restore — is the entire basis on which we\n\nask you to trust the green checkmark. The check that caught the\n\nunrendered-template bug was added the same way: a real bug, then a\n\ncheck that makes it permanent history.\n\nNow the honest part, because this is where most tool marketing lies by\n\nimplication.\n\n**It cannot check whether the advice is *good*.** The harness knows\n\n\"Commands\" exists; it cannot know your test command is right for your\n\nrepo. That's why kits mark the command table as the first thing to\n\nedit — and why the validator refuses to pretend otherwise.\n\n**It cannot check semantics.** Whether the baseline contradicts a\n\nCursor rule, whether your anti-pattern list matches your actual codebase\n\n— these need human review or much cleverer tooling. We structure the\n\ngeneration so contradictions are *hard* (tool files are projections of\n\none source), but hard is not impossible, and we won't claim the\n\nvalidator catches what it doesn't.\n\n**It cannot check real-repo firing.** We don't claim globs were\n\n\"tested against N production repos,\" because they weren't. The glob\n\n*syntax and mode declarations* are validated; whether your repo's\n\nlayout matches the glob is checked by you, in your repo, in one\n\nminute, using the harness we ship. Any product claiming automatic\n\nfiring-verification across arbitrary repos is claiming to have run\n\nyour build — ask them what that means, exactly.\n\nBecause everything downstream depends on it. A rule that never loads\n\nis zero percent as good as written, no matter how wise. A budget that\n\nsilently creeps degrades compliance in ways nobody attributes to the\n\ncause. Structural checks are the difference between \"config\" and\n\n\"markdown with aspirations\" — and unlike vibe-based claims, they're\n\nrunnable on your machine, against our kits, *and against your own edits*, before you spend a cent.\n\nThat's the offer, verified: the validator ships in the pack, the\n\nrelease output (12/12 kits, 8/8 skills, green) is in the changelog,\n\nand [the free MIT sample kit](https://piekwerk.gumroad.com/l/free-sample-nextjs) runs through the same harness. Trust the\n\ncheck, not the adjective — and run it yourself.\n\nAgentConfig Studio's twelve kits and the harness they ship with are on\n\n[AgentConfig Studio on Gumroad — all 12 kits](https://piekwerk.gumroad.com/l/agentconfig-studio).", "url": "https://wpnews.pro/news/how-we-validate-12-agent-config-kits-and-what-validation-can-t-do", "canonical_source": "https://dev.to/piekwerk/how-we-validate-12-agent-config-kits-and-what-validation-cant-do-2pop", "published_at": "2026-09-15 11:36:25+00:00", "updated_at": "2026-09-15 12:15:26.028641+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Cursor", "GitHub Copilot"], "alternates": {"html": "https://wpnews.pro/news/how-we-validate-12-agent-config-kits-and-what-validation-can-t-do", "markdown": "https://wpnews.pro/news/how-we-validate-12-agent-config-kits-and-what-validation-can-t-do.md", "text": "https://wpnews.pro/news/how-we-validate-12-agent-config-kits-and-what-validation-can-t-do.txt", "jsonld": "https://wpnews.pro/news/how-we-validate-12-agent-config-kits-and-what-validation-can-t-do.jsonld"}}