# How we validate 12 agent config kits (and what validation can't do)

> Source: <https://dev.to/piekwerk/how-we-validate-12-agent-config-kits-and-what-validation-cant-do-2pop>
> Published: 2026-09-15 11:36:25+00:00

*Our validator checks every kit before release: files, budgets, frontmatter, sections, placeholders. Here's exactly what it checks,
why those checks exist, and — more interestingly — what it cannot
check, because pretending otherwise is how trust dies.*

Agent config has a trust problem, and it's deserved. The market is

full of markdown with confident READMEs and no contract. "Validated

against real projects!" is the new "military-grade encryption." So

instead of adjectives, this post is the actual spec of our validation

harness — the same script that ships in the pack — including the

failure it caught, and the honest list of what it does *not* check.

A kit is a set of files with structure. Structure is checkable;

therefore the contract is structural. Every kit must pass all of:

**1. Required files exist.**

`AGENTS.md`, `CLAUDE.md`, `copilot-instructions.md`, and a

`.cursor/rules/` directory with 3–5 `.mdc` files. Trivial? Every

missing-file bug in our own builds was "trivial" until it shipped.

**2. The CLAUDE.md line budget.**

Fewer than 300 lines, enforced, not suggested. The number is a design

decision (see the compliance-curve argument in our longer piece), but

the enforcement is the point: budgets without gates become

suggestions, and suggestions become 900-line files.

**3. Cursor frontmatter parses, with a declared activation mode.**

Each `.mdc` must open with parseable YAML frontmatter containing a

non-empty `description`, and either `alwaysApply: true|false` or a

`globs` entry. This check exists because of the nastiest silent

failure in the format: a malformed frontmatter block doesn't error —

Cursor treats the file as broken or ignores it, and a rule you

believe is protecting your test files simply never loads. The

validator makes "the rule never loads" a build failure.

**4. The baseline has its sections.**

`AGENTS.md` must contain: Project context, Commands, Code style,

Testing expectations, Commit conventions, Safety rules, Definition of

done. These aren't bureaucratic — they're the sections agents act on

most (the command table above all). A kit missing "Safety rules" is

missing the file's most important content, so it fails.

**5. No placeholder text, anywhere.**

`TODO`, `TBD`, `FIXME`, `lorem ipsum`, `XXX` — case-insensitive,

across every kit file. Config rot starts as a placeholder that

outlived its author.

**6. No unrendered templates.**

The kits are generated, so generation bugs are a risk class: the

validator scans for unrendered variables (`{lint_cmd`, and friends)

in output. This check exists because it caught exactly that bug —

a double-formatting trap left the literal text `lint_cmd_placeholder`

inside every kit's definition-of-done. The validator flagged it on

its first real run; the fix took one line; the check stays forever.

**7. Skills exist and are non-trivial.**

All eight workflow skills must be present and above a minimum size.

A stub skill file is a broken promise.

Output: human-readable by default, `--json` for CI. Exit code is the

release gate: non-zero, no release.

A validator that has never failed is a validator that has never been

tested. Before trusting ours, we negative-tested it — deliberately

corrupting a kit and watching:

`TODO: fix this` to a kit's AGENTS.md → That loop — break, catch, restore — is the entire basis on which we

ask you to trust the green checkmark. The check that caught the

unrendered-template bug was added the same way: a real bug, then a

check that makes it permanent history.

Now the honest part, because this is where most tool marketing lies by

implication.

**It cannot check whether the advice is *good*.** The harness knows

"Commands" exists; it cannot know your test command is right for your

repo. That's why kits mark the command table as the first thing to

edit — and why the validator refuses to pretend otherwise.

**It cannot check semantics.** Whether the baseline contradicts a

Cursor rule, whether your anti-pattern list matches your actual codebase

— these need human review or much cleverer tooling. We structure the

generation so contradictions are *hard* (tool files are projections of

one source), but hard is not impossible, and we won't claim the

validator catches what it doesn't.

**It cannot check real-repo firing.** We don't claim globs were

"tested against N production repos," because they weren't. The glob

*syntax and mode declarations* are validated; whether your repo's

layout matches the glob is checked by you, in your repo, in one

minute, using the harness we ship. Any product claiming automatic

firing-verification across arbitrary repos is claiming to have run

your build — ask them what that means, exactly.

Because everything downstream depends on it. A rule that never loads

is zero percent as good as written, no matter how wise. A budget that

silently creeps degrades compliance in ways nobody attributes to the

cause. Structural checks are the difference between "config" and

"markdown with aspirations" — and unlike vibe-based claims, they're

runnable on your machine, against our kits, *and against your own edits*, before you spend a cent.

That's the offer, verified: the validator ships in the pack, the

release output (12/12 kits, 8/8 skills, green) is in the changelog,

and [the free MIT sample kit](https://piekwerk.gumroad.com/l/free-sample-nextjs) runs through the same harness. Trust the

check, not the adjective — and run it yourself.

AgentConfig Studio's twelve kits and the harness they ship with are on

[AgentConfig Studio on Gumroad — all 12 kits](https://piekwerk.gumroad.com/l/agentconfig-studio).