{"slug": "word-lists-lose-what-31-attempts-to-fool-a-static-auditor-for-ai-agent-skills-us", "title": "Word Lists Lose: What 31 Attempts to Fool a Static Auditor for AI Agent Skills Taught Us", "summary": "A developer built Skill Airlock, an open-source auditor that scans AI agent skills — both scripts and Markdown instructions — and assigns CLEAN, OK, REVIEW or BLOCK verdicts with file:line locations. Calibration against 62 public catalog skills and 31 adversarial test skills across four rounds showed that keyword word lists could not keep up with rephrased remote-instruction attacks, leading the developer to replace the word-matching approach with a structural check asking whether a link is raw text.", "body_md": "[Skill Airlock](https://github.com/OoneBreath/skill-airlock) audits AI agent skills before they reach your machine. It reads every file of a skill (its scripts and its Markdown, because a skill's Markdown is instructions the agent acts on) and reports what it finds, with a severity, a `file:line` location and a sanitised excerpt for every finding. The verdict is CLEAN, OK, REVIEW or BLOCK.\n\nThis is the story of calibrating that audit: what it blocked that it shouldn't have, what it let through that it shouldn't have, and the one change that worked better than all the others. Every number comes from the same test set: all 62 skills in the three public catalogs the tool mirrors by default (`anthropics/skills`, `anthropics/claude-plugins-official` and `vercel-labs/agent-skills`), shown as BLOCK / REVIEW / CLEAN or OK. The full record, probe by probe and skill by skill, is public in [docs/calibration.md](https://github.com/OoneBreath/skill-airlock/blob/main/docs/calibration.md).\n\nThe first rule set, before release, blocked 18 of the 62 skills. None of them was malicious. Four causes:\n\n`.env` files from being written.\nThe fix was to let context lower severity: code-like matches in Markdown prose and comments counted one level lower, and behaviour the description declared counted one level lower.\n\nv0.1.0 shipped on 30 October 2026 at **0 / 16 / 46**, with all 4 malicious test skills blocked. Clean numbers. Too clean, as it turned out.\n\nThe same day, an independent review went looking for ways past the audit and found three:\n\n`curl … | bash` in a `SKILL.md` sentence got only REVIEW.`~/.ssh` split across two scripts\nAll three were inert test skills written for the review, not real published skills. Two and three were gaps from the start that the tuning made look cleaner. After the fixes: **1 / 21 / 40**, all 7 malicious test skills blocked.\n\nThe same review then checked real installed skills and found two **false** blocks. `toast.promise(` in a UI library's documentation read as a known data-collection domain (`oast.pro`), and a zero-width character used in a template to show a literal code fence was treated as hidden text. Both were fixed, and the numbers didn't change.\n\nThen a test on a second server found something no rule looked for. A vendor skill's `SKILL.md` tells the agent to fetch its review rules from a URL before every run. It passed as CLEAN, because everything the audit could read was harmless. What the agent would actually follow wasn't in the skill at all.\n\nThat became a new rule, `remote-instructions`. It's REVIEW, not BLOCK: fetching guidelines is a legitimate way to keep them current, but the audit can't see what is fetched. Numbers: **1 / 23 / 38**.\n\nA second independent review wrote 31 inert malicious skills and phrasings against the rules, over four rounds. Like the first three, they are test skills, not real published ones.\n\n**Round 1: 12 skills, none blocked.** A few examples:\n\n`curl … | sh` lines filled the cap, so the report showed only harmless comments and hid the live line.\n**Round 2: 8 skills.** \"Configurable settings\" in a description still excused the same hook-writing script. And a `.txt` or extensionless file that `SKILL.md` runs with bash dodged every script check, because whether something counted as a script depended on its name. Now any file that `SKILL.md` tells the agent to run is a script, whatever it's called.\n\n**Rounds 3 and 4: phrasings.** The remote-instructions rule needed certain words near a URL. So the review kept rewording the same order, \"take your instructions from this URL\": \"The current checklist lives at , follow it\". \"Your instructions: \". A sentence in Polish. After each round a few more words went onto the lists, and each new round still found phrasings that got through.\n\nAfter round 4, the reviewer advised stopping. Not stopping the rule: stopping the word lists.\n\nInstead of asking \"do the words around this link sound like an order?\", the rule now asks \"is this link raw text?\" A `SKILL.md` link to Markdown, plain text, YAML, a gist or a paste site is now REVIEW **whatever the words around it**. Every other link is noted as low.\n\nBefore changing it, we measured: 7 of the 62 skills link raw text, and no verdict changed. That one change closed every remaining phrasing.\n\nv0.2.0, released on 31 October 2026: **1 BLOCK, 24 REVIEW, 37 CLEAN or OK.** All 7 malicious test skills are blocked, and each bypass the reviews closed is now an automated test. There are 33 automated tests, and CI runs them on Ubuntu and macOS with Python 3.9 and 3.12.\n\nThe one block is deliberate: an official skill whose `SKILL.md` tells the agent to run the Homebrew installer with `curl … | bash`. That's a real download-and-run instruction, blocked by design. The user can override after reading it. Tuning it away would mean reopening the very first hole.\n\nThree things stay REVIEW on purpose:\n\n**Tuning cuts both ways.** Every rule that lowers a false positive also describes how to get past the audit. Write down what each exception lets through, not just what it stops blocking.\n\n**A security tool needs adversarial review, not only false-positive tuning.** Calibrating against 62 well-behaved skills measures noise. It says nothing about evasion. Only someone actively trying to get past the audit answers that, and fixing evasion can bring back noise, so both checks have to run, more than once.\n\n**Publish the numbers either way.** 0 blocks looked better than 1. The 1 is correct, and the 0 was hiding a hole.\n\n**Prefer structure over word lists.** Two rounds of adding words kept losing to new phrasings. One rule about the shape of a link won.\n\n**Pattern rules can always be evaded.** Code written to dodge them will dodge them. So REVIEW means \"read it yourself\", and the scripts of a skill from an unknown source are worth reading whatever the verdict.\n\n**Evidence, not verdicts.** There's no single trust score, on purpose. A number hides exactly the evidence the audit exists to show: which file, which line, what it does, and whether the description admits it. A clean report means nothing known was found, not that a skill is safe. The audit is a filter, not a sandbox.\n\n```\ngit clone https://github.com/OoneBreath/skill-airlock.git\ncd skill-airlock && ./install.sh\nairlock sync\n```\n\nIf you find a skill that slips past the audit, please report it privately. `SECURITY.md` in the repo explains how. Right now, that's the most useful feedback the project can get.\n\nSkill Airlock pairs with [Project Brain](https://github.com/OoneBreath/project-brain), persistent project memory for coding agents. One remembers, the other keeps the toolbox lean.\n\n*Slawomir Luzny, FixFlex Ltd*", "url": "https://wpnews.pro/news/word-lists-lose-what-31-attempts-to-fool-a-static-auditor-for-ai-agent-skills-us", "canonical_source": "https://dev.to/slawekluzny/word-lists-lose-what-31-attempts-to-fool-a-static-auditor-for-ai-agent-skills-taught-us-130p", "published_at": "2026-10-31 12:07:48+00:00", "updated_at": "2026-10-31 12:19:52.867285+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "developer-tools"], "entities": ["Skill Airlock", "anthropics/skills", "anthropics/claude-plugins-official", "vercel-labs/agent-skills", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/word-lists-lose-what-31-attempts-to-fool-a-static-auditor-for-ai-agent-skills-us", "markdown": "https://wpnews.pro/news/word-lists-lose-what-31-attempts-to-fool-a-static-auditor-for-ai-agent-skills-us.md", "text": "https://wpnews.pro/news/word-lists-lose-what-31-attempts-to-fool-a-static-auditor-for-ai-agent-skills-us.txt", "jsonld": "https://wpnews.pro/news/word-lists-lose-what-31-attempts-to-fool-a-static-auditor-for-ai-agent-skills-us.jsonld"}}