{"slug": "i-let-a-script-audit-my-own-product-copy-it-caught-three-lies", "title": "I let a script audit my own product copy — it caught three lies", "summary": "A developer running a small digital-kit store for LLM agent builders wrote a roughly 30-line Python script that checks every numeric claim on product pages against the actual zip files buyers download, and it caught three false claims in the catalog, all since fixed. The script parses headings and SKILL.md files to count prompts, skills and runbooks, making marketing copy falsifiable before each catalog change. The developer argues buyers will unzip and count anyway, and pairs the audit with runnable artifacts such as a 50-test eval kit and a 15-payload injection self-test.", "body_md": "I run a small store that sells digital kits for people running LLM agents — eval suites, ops runbooks, prompt libraries. Every product page makes claims: \"62 prompts,\" \"10 skills,\" \"12 runbooks.\" Last week I wrote a script that checked every one of those claims against the actual zip files buyers download.\n\nIt found three lies. All mine. All fixed, but the exercise taught me more than the fixes did.\n\nThe method is embarrassingly simple:\n\n`## Q` headings, SKILL.md files).\nWhat it caught in my catalog:\n\nNone of these were malicious. They're what happens when copy is written before the artifact is finished and nobody re-reads it after. Which is exactly why you need the script — human attention doesn't scale, `zipfile` does.\n\nIf you sell anything an agent will read — prompts, skills, runbooks, evals — your buyers' first test is *\"does the artifact match the promise?\"* They're going to unzip it and count. A skeptical senior dev (the actual buyer for this stuff) has been burned by \"100+ templates\" lists that contain 12 entries and filler.\n\nThe fix isn't better marketing. It's a 30-line Python script that makes your claims falsifiable. I run it before every catalog change now.\n\nThis audit only catches *numeric* claims. It won't tell you the runbook is boring or the prompts are mediocre. For that you need the second check: actually run the artifact. My eval kit ships 50 known-answer tests; I run them against a deliberately dumb agent to prove the harness catches failures. My injection self-test ships 15 payloads and a scorer — the \"does it refuse?\" question is answered by output, not adjectives.\n\nVerification beats adjectives. That's the whole pitch.\n\n*Everything above is checkable: the free eval mini-kit and the injection self-test are downloadable without an email, and the full store (29 kits, BTC checkout, instant zip delivery) is at the link below.*\n\n**Free samples (no email gate):**\n\n`/free-eval`\n`/free-injection`", "url": "https://wpnews.pro/news/i-let-a-script-audit-my-own-product-copy-it-caught-three-lies", "canonical_source": "https://dev.to/nexusfoundry/i-let-a-script-audit-my-own-product-copy-it-caught-three-lies-1bbn", "published_at": "2026-11-21 15:29:21+00:00", "updated_at": "2026-11-21 15:42:11.596051+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-products"], "entities": [], "also_reported_by": [], "takeaway": "A ~30-line Python script that parses headings and SKILL.md files and compares counts against product-page claims catches numeric copy errors that manual review misses, though it cannot judge quality, which requires running the artifact against known-answer tests.", "recommendation": {"advice": "Before each catalog or copy change, script a check of every numeric claim against the shipped zip contents, and run the artifact's known-answer tests to prove it catches failures.", "playbook": "output-verification", "playbook_url": "https://wpnews.pro/playbooks/output-verification"}, "alternates": {"html": "https://wpnews.pro/news/i-let-a-script-audit-my-own-product-copy-it-caught-three-lies", "markdown": "https://wpnews.pro/news/i-let-a-script-audit-my-own-product-copy-it-caught-three-lies.md", "text": "https://wpnews.pro/news/i-let-a-script-audit-my-own-product-copy-it-caught-three-lies.txt", "jsonld": "https://wpnews.pro/news/i-let-a-script-audit-my-own-product-copy-it-caught-three-lies.jsonld"}}