cd /news/ai-agents/i-let-a-script-audit-my-own-product-… · home › topics › ai-agents › article
[ARTICLE · art-173815] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

I let a script audit my own product copy — it caught three lies

A developer running a small digital-kit store for LLM agent builders wrote a roughly 30-line Python script that checks every numeric claim on product pages against the actual zip files buyers download, and it caught three false claims in the catalog, all since fixed. The script parses headings and SKILL.md files to count prompts, skills and runbooks, making marketing copy falsifiable before each catalog change. The developer argues buyers will unzip and count anyway, and pairs the audit with runnable artifacts such as a 50-test eval kit and a 15-payload injection self-test.

recommendation

Before each catalog or copy change, script a check of every numeric claim against the shipped zip contents, and run the artifact's known-answer tests to prove it catches failures.

more in playbook: Verifying model output
by read2 min views6 publishedNov 21, 2026

I run a small store that sells digital kits for people running LLM agents — eval suites, ops runbooks, prompt libraries. Every product page makes claims: "62 prompts," "10 skills," "12 runbooks." Last week I wrote a script that checked every one of those claims against the actual zip files buyers download.

It found three lies. All mine. All fixed, but the exercise taught me more than the fixes did.

The method is embarrassingly simple:

## Q headings, SKILL.md files). What it caught in my catalog:

None of these were malicious. They're what happens when copy is written before the artifact is finished and nobody re-reads it after. Which is exactly why you need the script — human attention doesn't scale, zipfile does.

If you sell anything an agent will read — prompts, skills, runbooks, evals — your buyers' first test is "does the artifact match the promise?" They're going to unzip it and count. A skeptical senior dev (the actual buyer for this stuff) has been burned by "100+ templates" lists that contain 12 entries and filler. The fix isn't better marketing. It's a 30-line Python script that makes your claims falsifiable. I run it before every catalog change now.

This audit only catches numeric claims. It won't tell you the runbook is boring or the prompts are mediocre. For that you need the second check: actually run the artifact. My eval kit ships 50 known-answer tests; I run them against a deliberately dumb agent to prove the harness catches failures. My injection self-test ships 15 payloads and a scorer — the "does it refuse?" question is answered by output, not adjectives.

Verification beats adjectives. That's the whole pitch.

Everything above is checkable: the free eval mini-kit and the injection self-test are downloadable without an email, and the full store (29 kits, BTC checkout, instant zip delivery) is at the link below.

Free samples (no email gate): /free-eval /free-injection

── more in #ai-agents 4 stories · sorted by recency
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/i-let-a-script-audit…] indexed:0 read:2min 2026-11-21 · —