How we validate 12 agent config kits (and what validation can't do)

A developer published the specification for a validation harness that checks agent configuration kits before release, verifying required files, a sub-300-line CLAUDE.md budget, parseable Cursor frontmatter, baseline sections, absence of placeholder text and unrendered templates, and non-trivial skills. The validator, which exits non-zero to block releases and supports JSON output for CI, caught a real generation bug that left a literal placeholder in every kit's definition of done. The author notes the harness cannot judge whether the advice is good or check semantics, arguing that overclaiming validation is how trust dies.

Our validator checks every kit before release: files, budgets, frontmatter, sections, placeholders. Here's exactly what it checks, why those checks exist, and — more interestingly — what it cannot check, because pretending otherwise is how trust dies. Agent config has a trust problem, and it's deserved. The market is full of markdown with confident READMEs and no contract. "Validated against real projects " is the new "military-grade encryption." So instead of adjectives, this post is the actual spec of our validation harness — the same script that ships in the pack — including the failure it caught, and the honest list of what it does not check. A kit is a set of files with structure. Structure is checkable; therefore the contract is structural. Every kit must pass all of: 1. Required files exist. AGENTS.md , CLAUDE.md , copilot-instructions.md , and a .cursor/rules/ directory with 3–5 .mdc files. Trivial? Every missing-file bug in our own builds was "trivial" until it shipped. 2. The CLAUDE.md line budget. Fewer than 300 lines, enforced, not suggested. The number is a design decision see the compliance-curve argument in our longer piece , but the enforcement is the point: budgets without gates become suggestions, and suggestions become 900-line files. 3. Cursor frontmatter parses, with a declared activation mode. Each .mdc must open with parseable YAML frontmatter containing a non-empty description , and either alwaysApply: true|false or a globs entry. This check exists because of the nastiest silent failure in the format: a malformed frontmatter block doesn't error — Cursor treats the file as broken or ignores it, and a rule you believe is protecting your test files simply never loads. The validator makes "the rule never loads" a build failure. 4. The baseline has its sections. AGENTS.md must contain: Project context, Commands, Code style, Testing expectations, Commit conventions, Safety rules, Definition of done. These aren't bureaucratic — they're the sections agents act on most the command table above all . A kit missing "Safety rules" is missing the file's most important content, so it fails. 5. No placeholder text, anywhere. TODO , TBD , FIXME , lorem ipsum , XXX — case-insensitive, across every kit file. Config rot starts as a placeholder that outlived its author. 6. No unrendered templates. The kits are generated, so generation bugs are a risk class: the validator scans for unrendered variables {lint cmd , and friends in output. This check exists because it caught exactly that bug — a double-formatting trap left the literal text lint cmd placeholder inside every kit's definition-of-done. The validator flagged it on its first real run; the fix took one line; the check stays forever. 7. Skills exist and are non-trivial. All eight workflow skills must be present and above a minimum size. A stub skill file is a broken promise. Output: human-readable by default, --json for CI. Exit code is the release gate: non-zero, no release. A validator that has never failed is a validator that has never been tested. Before trusting ours, we negative-tested it — deliberately corrupting a kit and watching: TODO: fix this to a kit's AGENTS.md → That loop — break, catch, restore — is the entire basis on which we ask you to trust the green checkmark. The check that caught the unrendered-template bug was added the same way: a real bug, then a check that makes it permanent history. Now the honest part, because this is where most tool marketing lies by implication. It cannot check whether the advice is good . The harness knows "Commands" exists; it cannot know your test command is right for your repo. That's why kits mark the command table as the first thing to edit — and why the validator refuses to pretend otherwise. It cannot check semantics. Whether the baseline contradicts a Cursor rule, whether your anti-pattern list matches your actual codebase — these need human review or much cleverer tooling. We structure the generation so contradictions are hard tool files are projections of one source , but hard is not impossible, and we won't claim the validator catches what it doesn't. It cannot check real-repo firing. We don't claim globs were "tested against N production repos," because they weren't. The glob syntax and mode declarations are validated; whether your repo's layout matches the glob is checked by you, in your repo, in one minute, using the harness we ship. Any product claiming automatic firing-verification across arbitrary repos is claiming to have run your build — ask them what that means, exactly. Because everything downstream depends on it. A rule that never loads is zero percent as good as written, no matter how wise. A budget that silently creeps degrades compliance in ways nobody attributes to the cause. Structural checks are the difference between "config" and "markdown with aspirations" — and unlike vibe-based claims, they're runnable on your machine, against our kits, and against your own edits , before you spend a cent. That's the offer, verified: the validator ships in the pack, the release output 12/12 kits, 8/8 skills, green is in the changelog, and the free MIT sample kit https://piekwerk.gumroad.com/l/free-sample-nextjs runs through the same harness. Trust the check, not the adjective — and run it yourself. AgentConfig Studio's twelve kits and the harness they ship with are on AgentConfig Studio on Gumroad — all 12 kits https://piekwerk.gumroad.com/l/agentconfig-studio .