{"slug": "internal-ai-tools-a-test-runner-for-which-claude-skill-loads", "title": "Internal AI tools: a test runner for which Claude skill loads", "summary": "A developer built a small Python test runner that scores which Claude Agent Skill gets loaded for a given request, walking a skills folder, parsing each SKILL.md frontmatter (including folded and literal YAML descriptions), and exposing the library as a single Skill tool. In a test on an invented sales team, seven skills with one-sentence descriptions routed 40 messy requests correctly in 120 of 120 runs, while twenty overlapping skills scored 117 of 120. Rewriting the description of the one skill that missed fixed that request but moved two previously correct requests to the wrong skill in every run, which the developer cites as the reason the regression runner exists.", "body_md": "Internal AI tools built on Claude Agent Skills have a step that nobody tests: the moment Claude reads every skill's name and description and picks one, or none. By the end of this post you will be able to point a small Python runner at your own skills folder and a list of labelled requests, and get a scorecard of which skill Claude loads and every miss. I had Claude build it for a test on an invented sales team; the code below is trimmed from the runner that ships. ![A drawing on an ivory ground: a sales team's skill library as two shelves of open bins with gold dots in them, a speech bubble dropping requests in, a pen rewriting the label of one bin ringed in oxblood, and a red dot pulled across into it from the bin next door]([https://brinvik.com/lab/skill-trigger-test/cover-en-1600x720.png](https://brinvik.com/lab/skill-trigger-test/cover-en-1600x720.png)) ## Why internal AI tools need a regression test for skills A skill is a folder with a SKILL.md. The frontmatter holds a `name\\` and a `description\\`; the body is the procedure. Claude sees only the name and description when it decides what to load, and the body loads after the pick. So the description is the whole interface between a seller's request and the team's recipe. The test that came out of this build: seven sales skills with one-sentence descriptions got 40 messy requests right in 120 of 120 runs. Twenty overlapping skills got 117 of 120. Then Claude rewrote the description of the one skill that missed, which fixed that request and moved two others that had been right into the wrong skill, in every run. That last part is why the runner exists. The full story is in [the write-up on the Claude skill test]([https://brinvik.com/en/resources/internal-ai-tools-claude-skill-test?utm_source=devto&utm_medium=article&utm_campaign=lab-skill-trigger-test&utm_content=devto-en)](<https://brinvik.com/en/resources/internal-ai-tools-claude-skill-test?utm_source=devto&utm_medium=article&utm_campaign=lab-skill-trigger-test&utm_content=devto-en)>). ## Reading the skills folder The runner walks the folder at any depth, reads each SKILL.md's frontmatter, and refuses anything ambiguous. Real skill files use folded YAML descriptions more often than you would think, and a naive `split(\":\")\\` parser reads the description as `>\\`, so the frontmatter parser handles both block styles: `\\`` python def parse\\_frontmatter(block):    \"\"\"name and description from SKILL.md frontmatter, including a folded (>) or literal (|) description    spread over several indented lines, which a plain one-line parser would read as \">\".\"\"\"    meta, lines, i = {}, block.splitlines(), 0    while i < len(lines):        line = lines\\[i\\]        i += 1        if not line or line\\[0\\].isspace() or \":\" not in line:            continue        k, v = line.split(\":\", 1)        k, v = k.strip(), v.strip()        if v\\[:1\\] in (\">\", \"|\"):            parts = \\[\\]            while i < len(lines) and (not lines\\[i\\].strip() or lines\\[i\\]\\[0\\].isspace()):                parts.append(lines\\[i\\].strip())                i += 1            sep = \" \" if v\\[0\\] == \">\" else \"\\\\n\"            v = sep.join(p for p in parts if p)        else:            if len(v) >= 2 and v\\[0\\] == v\\[-1\\] and v\\[0\\] in \"\\\\\"'\":                v = v\\[1:-1\\]            while i < len(lines) and lines\\[i\\]\\[:1\\].isspace() and lines\\[i\\].strip():                v += \" \" + lines\\[i\\].strip()                i += 1        meta\\[k\\] = v    return meta \\``\\` Folders without a SKILL.md, files without a name or description, duplicate names and descriptions over 1,024 characters are reported on stderr rather than silently dropped. A scorecard that quietly leaves out a skill is worse than no scorecard. ## Exposing the library as one tool The selection surface is reconstructed as a single tool called `Skill\\`, whose description lists every skill by name and description. One short system prompt tells Claude to load a listed skill when one covers the request, and to just help when none does: `\\`` python def skill\\_tool(skills):    listing = \"\\\\n\".join(        f\"\\\\n{s\\['name'\\]}\\\\n{s\\['description'\\]}\\\\n\" for s in skills)    return {\"name\": \"Skill\",            \"description\": (\"Load a skill: a set of instructions the team has written for a particular kind of task. \"                            \"Call it with the skill's name when the request is one a listed skill covers; the skill's \"                            \"full instructions then load. Only names from the list below are valid.\\\\n\\\\n\"                            f\"\\\\n{listing}\\\\n\"),            \"input\\_schema\": {\"type\": \"object\", \"properties\": {\"skill\": {\"type\": \"string\"}}, \"required\": \\[\"skill\"\\]}} ERROR = \"\\\\0error\" \\``\\` This is my wording of the surface, not Claude Code's own loader. It reproduces the part a team controls, the descriptions, and it is honest to say a result tells you whether a description steers Claude well rather than what one specific app will do. If you want to repeat the original test word for word, `--team\\` sets the one line of context the job used. ## The pick, and scoring with alternatives Each request is sent once per run with the library tool attached. The first `Skill\\` call is the pick; a reply with no call is \"none\". The reply text is kept too, so a \"none\" that was really a clarifying question can be read later. Labels can accept more than one skill. In the twenty-skill library, a request like \"send a follow up to Maria after today's demo\" is fine in either the call recap or the follow-up email skill, so the label is `call-recap|follow-up-email\\`: `\\`` python def outcome(expected, pick, accepted=None, names=None):    pick = pick or \"none\"    if pick == ERROR:        return \"API error, not scored\"    if pick == expected or pick in (accepted or \\[\\]):        return \"right\"    if expected == \"none\":        return \"loaded a skill when none was needed\"    if pick == \"none\":        return \"loaded nothing\"    if names is not None and pick not in names:        return \"named a skill that does not exist\"    return \"loaded the wrong skill\" \\``\\` A pick that names a skill that does not exist is counted separately from a wrong skill, and an API error is counted but never scored as \"loaded nothing\". ## Saving every run as it finishes The first version wrote its results at the end. A hostile review pointed out that a crash, a Ctrl+C or an exception outside the API call would throw away every paid call. Now every run is appended to a JSONL file and flushed the moment it comes back, and an authentication, permission or not-found error stops the run at once instead of burning through the list. ## Reading the input your team actually has The runner takes a CSV with `text\\` and `expected\\` columns, or JSON. The review also found that Excel's \"CSV UTF-8\" export adds a byte-order mark that turns the header into something that is not `text\\`, and that a header typed as `Text,Expected\\` failed the same way. The reader opens files as `utf-8-sig\\`, lowercases the headers and accepts `none\\` in any case. It also reads the lab build's own request file, with its test and holdout lists, so the published test can be run as shipped. ![Screenshot of the demo: a board with one bin for each of the 20 skills and a hatched bin for no skill, each of the 40 requests sitting in the skill Claude picked, after the pricing skill was rewritten; a Danish request for a tilbud is ringed in the pricing bin and the discount sum in the no-skill bin]([https://brinvik.com/lab/skill-trigger-test/demo-en.png](https://brinvik.com/lab/skill-trigger-test/demo-en.png)) ## The tests There are 8 offline tests, no API key and no network. They cover one-line, folded, literal and wrapped descriptions, CSV with alternatives, Excel's byte-order mark and header case, the lab's own request file and system prompt, nested skill folders, the outcome rules, an API error that is counted and not scored, and a run interrupted halfway that keeps what finished. The fake client is a few lines of `types.SimpleNamespace\\`, which is enough to drive `main()\\` end to end. ## What the numbers looked like With seven skills, every request in every run loaded what it should. With twenty, three runs missed, all on one request. After Claude's rewrite of the pricing skill, the test set was still 117 of 120, but the misses were different ones: the discount sum was fixed, and two Danish requests for a \"tilbud\" (Danish for both a proposal and a quote) moved from the proposal skill to the pricing skill. On 16 requests held back from the rewrite, the score went from 45 to 42 of 48. ![Screenshot of the demo: a board with one bin for each of the 20 skills and a hatched bin for no skill, each of the 40 requests sitting in the skill Claude picked, after the pricing skill was rewritten; a Danish request for a tilbud is ringed in the pricing bin and the discount sum in the no-skill bin]([https://brinvik.com/lab/skill-trigger-test/demo-grown-en.png](https://brinvik.com/lab/skill-trigger-test/demo-grown-en.png)) The pattern a regression list catches and a single fix does not: the fix was right for its request and nobody ran the rest. ## Extending it: a rewrite step that checks everything The lab job behind this post had a second phase the shipped runner leaves out: after the first run, Claude was shown the misses and asked to rewrite the descriptions of the skills involved, and then the whole list ran again. That is how the side effect was found. If you add that step to your own pipeline, two things matter. First, give the rewrite call the whole request list, not only the misses, and ask it to keep every request that was right still right. A brief that says \"fix this miss\" gets a fix for that miss, and the test above is what that looks like. Second, never accept the rewrite on the rewrite's own say-so. Run the full list again, three times per request, and diff the picks against the run before. A request that changes bins is a finding whether it moved to a right skill or a wrong one. Keep the old descriptions in version control beside the new ones, so a bad rewrite is one revert away. A small extra that pays for itself: keep a handful of requests out of every rewrite, the way the lab job held back sixteen. If the test set improves and the held-back set gets worse, the rewrite fitted the words it was shown. ## Running it on your own library `\\`` pip install anthropic python skill\\_trigger\\_test.py --skills ./skills --requests requests.csv --runs 3 \\``\\` Use your own `ANTHROPIC\\_API\\_KEY\\`; the runner never stores it. Fifty-six requests at three runs is 168 calls. Use requests with no customer data in them, because they go to the API. Read every miss before you trust the number; some misses are the label's fault. ## Reading the output The scorecard prints the right count first, then each kind of miss on its own line, then every missed request with the skills it actually got and in how many of the runs. A request that got two different skills across three runs shows both, which is often the most useful line in the output: it tells you exactly which two descriptions overlap. ## Where the runner stops The requests and labels in the shipped kit are invented. One model, Claude Sonnet 5.5, and one rewrite sample. Three runs per request is a small sample. The selection surface is a reconstruction. The runner, the twenty skills and the labelled requests are free from [the skill test runner page]([https://brinvik.com/en/resources/skill-trigger-test?utm_source=devto&utm_medium=article&utm_campaign=lab-skill-trigger-test&utm_content=devto-runner)](<https://brinvik.com/en/resources/skill-trigger-test?utm_source=devto&utm_medium=article&utm_campaign=lab-skill-trigger-test&utm_content=devto-runner)>). The write-up in Danish is [den danske udgave af artiklen]([https://brinvik.com/da/ressourcer/interne-ai-vaerktoejer-test-af-claude-skills?utm_source=devto&utm_medium=article&utm_campaign=lab-skill-trigger-test&utm_content=devto-da)](<https://brinvik.com/da/ressourcer/interne-ai-vaerktoejer-test-af-claude-skills?utm_source=devto&utm_medium=article&utm_campaign=lab-skill-trigger-test&utm_content=devto-da)>). If you would rather have someone write and test a skill library for your sales team, that is [the internal AI tools work I do]([https://brinvik.com/en/services/internal-ai-tools?utm_source=devto&utm_medium=article&utm_campaign=lab-skill-trigger-test&utm_content=devto-service)](<https://brinvik.com/en/services/internal-ai-tools?utm_source=devto&utm_medium=article&utm_campaign=lab-skill-trigger-test&utm_content=devto-service)>).", "url": "https://wpnews.pro/news/internal-ai-tools-a-test-runner-for-which-claude-skill-loads", "canonical_source": "https://dev.to/kimolsen/internal-ai-tools-a-test-runner-for-which-claude-skill-loads-596n", "published_at": "2026-11-20 11:31:11+00:00", "updated_at": "2026-11-20 11:42:33.415573+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "developer-tools"], "entities": ["Claude", "Claude Agent Skills", "Anthropic"], "also_reported_by": [], "takeaway": "Test skill selection as a regression suite: seven one-sentence skill descriptions routed 40 requests correctly in 120 of 120 runs, but rewriting one description to fix a miss broke two previously correct requests every time.", "recommendation": {"advice": "Run a labelled request set against your skills folder and re-run it after any description edit, since fixing one skill's description can silently misroute others.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/internal-ai-tools-a-test-runner-for-which-claude-skill-loads", "markdown": "https://wpnews.pro/news/internal-ai-tools-a-test-runner-for-which-claude-skill-loads.md", "text": "https://wpnews.pro/news/internal-ai-tools-a-test-runner-for-which-claude-skill-loads.txt", "jsonld": "https://wpnews.pro/news/internal-ai-tools-a-test-runner-for-which-claude-skill-loads.jsonld"}}