Will Claude Code call your skill on its own? 38 runs, one skill, seven ways to describe it

A developer ran 38 headless Claude Code 2.1.273 trials against a single commit-message skill, varying only how the skill was described, and found that auto-invocation hinged entirely on whether the one line shown in the skill listing contained the words of the user's request. A vague "Helps with git stuff." description triggered the skill 0 times in 6 runs, while a specific description, the same vague line plus a when_to_use field, and a frontmatter-free file whose first line read "# House commit format" each triggered it 6 times in 6 runs. The author reports that when_to_use offered no advantage over a better description, and that a skill directory named commit-message partially compensated for a vague description on direct requests but not indirect ones.

We gave Claude Code one commit-message skill, changed only how it was described, and ran a small set of requests against it 38 times in headless mode. "Helps with git stuff." was called 0 times in 6 runs; a specific description, the same vague line plus a when to use field, and a file with no frontmatter whose first line was " House commit format" were each called 6 times in 6 runs. The deciding factor was whether the one line Claude sees in the skill listing contained the words of the request, not which field those words came from. The skills docs say Claude "uses this to decide when to apply the skill" about the description field, and the troubleshooting section for a skill that does not trigger starts with "Check the description includes keywords users would naturally say". That is advice, not a number. We wanted to know how sharp the edge is: does a vague description make auto-invocation rarer, or does it switch it off? Does when to use do anything a better description would not? What happens when there is no description at all? We changed one thing at a time and counted. Everything below ran on Claude Code 2.1.273 on 2026-09-16, with the default model the CLI picked for the account, which the transcripts name as claude-opus-5 1m . We ran 38 claude -p calls in total. Each run got a fresh directory from mktemp -d containing exactly one skill at .claude/skills/<name /SKILL.md and a project settings file that turns off the skills that ship with Claude Code: { "disableBundledSkills": true } We also started every run with --setting-sources project , so that nothing from our own ~/.claude user settings, user hooks, personal skills, plugins, our user-level CLAUDE.md was loaded, and with CLAUDE CODE DISABLE AUTO MEMORY=1 , so the runs did not write memory files. The point was a skill listing with one entry in it. The transcripts confirm that: every skill listing attachment in them has "skillCount": 1 . The skill body was the same in every variant apart from one control. It describes a house format for commit messages, and it ends with a marker a model would not produce on its own: House commit format Write the commit message in exactly this shape: - First line: <type <scope : <summary , 60 characters or fewer, imperative mood. - Blank line, then one to three lines that say why the change was needed. - Last line: Refs: none unless the user gave a ticket id. The request always carried the same small diff: a withRetry helper that now rethrows an AuthError immediately and sleeps with exponential backoff between attempts. The wording in front of the diff came in three kinds: The direct request uses the words "commit message". The indirect one asks for the same kind of output without those words. The negative one asks for something the skill is not for. The skill's directory was named kestrel in every variant but one. A name like commit-message is itself a strong hint, and we wanted the description to be the only signal, so we used a word with nothing to do with git and tested the name separately. We counted a run as "invoked" if the transcript under ~/.claude/projects/ contained an assistant tool use block whose name is Skill . As a second check we looked for Refs: none in the final answer. The two signals agreed in all 38 runs: the marker appeared in exactly the 22 runs that called the skill and in none of the 16 that did not. | Variant what the listing said | direct | indirect | other | |---|---|---|---| | description: Helps with git stuff. | 0/3 | 0/3 | "house format" hint: 2/2 | | Specific description below | 3/3 | 3/3 | negative: 0/2 | | Vague description + when to use | 3/3 | 3/3 | negative: 0/2 | | No frontmatter, body starts House commit format | 3/3 | 3/3 | | | No frontmatter, body starts Notes | 0/2 | | | | Vague description, directory named commit-message | 2/2 | 0/2 | | | Specific description + disable-model-invocation: true | 0/2 | | | The specific description was: "Writes git commit messages in this repository's house format type scope : summary line, a short why, a Refs line . Use when the user asks for a commit message or wants a diff summarized for the git log." We ran each main cell three times and the smaller controls twice. With counts this small we do not read the table as rates. What stands out is that no cell was mixed. Every variant either fired on every run of a given request or on none of them. With the vague description, Claude answered all six direct and indirect requests itself, in one turn, without looking at the skill. The answers were reasonable commit messages. They were not in the house format: no type scope : prefix, no Refs: line. A typical first line was "Add exponential backoff to withRetry and fail fast on AuthError". This is the failure that matters in practice. Nothing errors, the answer looks fine, and the only sign the skill was skipped is that the output quietly ignores the convention the skill exists to enforce. If you only check "did I get a commit message", you will not notice. The listing line Claude saw was - kestrel: Helps with git stuff. A request for a commit message is arguably "git stuff". Claude still did not treat that as a reason to open a skill it knew nothing else about. when to use rescued the same vague line The third variant kept description: Helps with git stuff. and added one field: when to use: When the user asks for a commit message, or asks to summarize a diff for the git log. That variant fired 6 times out of 6, the same as the specific description. The docs say when to use is "Appended to description in the skill listing", and the transcript shows exactly what that means: Claude Code joins the two fields with " - " into one line. From the model's side there is no separate field, just a longer description. So in our runs when to use did nothing that putting the same sentence in description would not have done. Where it may still help is in how you maintain the file: the "what it does" part and the "when to use it" part stay apart in the frontmatter. Both fields count toward the same 1,536-character limit on the listing entry, which the docs state in the frontmatter reference. The docs say that if description is omitted, Claude Code "uses the first non-empty line of the markdown content". We tested that with a SKILL.md that had no frontmatter at all. Its first line was the heading House commit format , and the listing read - kestrel: House commit format , with the removed. That skill fired 6 times out of 6, as reliably as the carefully written description. Three words were enough, and the control below points to "commit" as the word that did it. To check that this was about the words and not about the missing frontmatter, we changed only the heading to Notes . The listing became - kestrel: Notes , and the skill fired 0 times in 2 direct runs. This is the result we find most useful. The rule is not "write a long description" or "fill in when to use ". Claude matches the request against one short line, and that line has to contain words that point at the request. A heading can do that by accident, and a description can miss it just as easily. With the directory renamed to commit-message and the vague description unchanged, the listing read - commit-message: Helps with git stuff. The direct request, "Write a commit message for this diff.", invoked it 2 times out of 2. The indirect request, "Give me a one-line summary of this change for the git log.", invoked it 0 times out of 2. The name matched the literal words of one request and not the other. A descriptive name narrows the gap, but it only covers requests that use the name's own words. A description can list the other ways people ask for the same thing, and the specific description and the when to use line both did: they mention "the git log", and both fired on the indirect request every time. We gave the vague kestrel skill one more request: "Write a commit message for this diff in our house format." Nothing in the listing mentions a house format. The skill fired 2 times out of 2. The second of those runs is worth describing. Claude first said it would "check whether the house format is defined anywhere", and tried a Bash command that listed the working directory and read ~/.claude/CLAUDE.md . In -p mode with no allow rules, that command needed approval and was denied; the JSON output records it under permission denials . Claude then called the only skill in the listing. So the vague skill was found, but only after Claude had tried to look for the format elsewhere. The only difference from the six vague runs that never fired was a phrase in the request that suggested a written convention existed somewhere. The specific description and the when to use variant both got the negative request, which asks for an explanation of the diff for a teammate. Neither fired, 0 out of 4 runs in total, and each answer came back in one turn as a short explanation in prose. Making the listing line specific did not make the skill fire on nearby requests that were not about commits. Four runs is a small sample, and one negative wording is only one wording. disable-model-invocation: true : not even listed With the specific description and disable-model-invocation: true , the skill fired 0 times in 2 runs. The transcripts for those two runs had no skill listing attachment at all, while every other run had one. That matches the docs' table for this field: "Description not in context". A manual-only skill is not a skill Claude decides against using. As far as the model can tell, it does not exist until someone types its name. Both answers were ordinary commit messages without the house format. Every run that called the skill reported 3 turns in the JSON output one reported 4: the run with the denied Bash call . Every run that did not call it reported 1. Duration went from 2.5–6.2 seconds without the skill to 6.9–13.6 seconds with it. The total cost usd field, which the output labels as list-price based, went from $0.030–$0.046 to $0.056–$0.077 per run. That range leaves out our very first run $0.22 , which wrote about 20,000 tokens to the prompt cache that later runs mostly read from. All 38 runs together came to $2.21 on that field. In many runs Claude passed an args string when it called the skill, usually a restatement of the request. Our skill body has no $ARGUMENTS placeholder, and the transcripts show Claude Code appending an ARGUMENTS: line to the end of the loaded skill text instead, which is the behaviour the docs describe for that case. The indirect request is where the skill changed the answer the most. The user asked for one line. All 9 indirect runs that called the skill still produced the full house-format message with the reason lines and Refs: none . Five of them gave the one-line summary first and then offered the full message. Four led with the full message and pointed out that its first line was the summary. None of them just returned one line. A skill that says "write the commit message in exactly this shape" does not bend to "just one line" by itself. If a skill should apply to part of a request, its body has to say so. That is a separate lesson from triggering, and a skill that fires more often makes it matter more. The format also pushed something out. Four of the 16 runs that did not call the skill, including both manual-only runs, ended their commit message with a Co-Authored-By trailer. None of the 22 runs that called the skill did. The skill's shape ends with the Refs line, and Claude kept to it. We did not check what decided whether the trailer appeared. The repository that runs our shop has ten project skills, and each description says when the skill should be used; one also lists trigger phrases. This experiment shows why that matters: the listing line is what Claude matches the request against, and in our runs a line either contained the request's words or the skill was ignored. The check we now use on a new skill is the listing line itself, not the SKILL.md. Read the line as - <name : <description - <when to use , and ask whether the words people use for the task appear in it, including the indirect ones. If the skill has no frontmatter, the line is the first line of the body. A heading like "Notes" gives Claude nothing to match, and in our two runs with it, Claude never picked the skill on its own. The runner below writes one variant into a fresh temp directory and runs one request. Swap the frontmatter to test other descriptions. bash /bin/bash usage: ./run.sh <label "<frontmatter lines or empty " "<request " label="$1"; fm="$2"; ask="$3" dir="$ mktemp -d " mkdir -p "$dir/.claude/skills/kestrel" results { -n "$fm" && printf -- '---\n%s\n---\n\n' "$fm" cat body.md } "$dir/.claude/skills/kestrel/SKILL.md" printf '{ "disableBundledSkills": true }\n' "$dir/.claude/settings.json" cd "$dir" && CLAUDE CODE DISABLE AUTO MEMORY=1 claude -p "$ask $ cat "$OLDPWD/diff.txt" " \ --output-format json --max-turns 3 --setting-sources project < /dev/null \ "results/$label.json" ./run.sh vague direct 1 'description: Helps with git stuff.' "Write a commit message for this diff." ./run.sh nodesc direct 1 '' "Write a commit message for this diff." The < /dev/null matters when you start these from a non-interactive shell: our first run, started without it, waited three seconds for stdin and printed a warning. To count, read the transcript for each session id : python import glob, json, os, sys from collections import defaultdict tally = defaultdict lambda: 0, 0 for path in sorted glob.glob os.path.join sys.argv 1 , " .json" : variant, prompt, = os.path.basename path :-5 .split " " session id = json.load open path "session id" transcript = glob.glob os.path.expanduser f"~/.claude/projects/ /{session id}.jsonl" 0 called = False for line in open transcript : record = json.loads line if record.get "type" = "assistant": continue for block in record "message" "content" : if block.get "type" == "tool use" and block "name" == "Skill": called = True tally variant, prompt 0 += called tally variant, prompt 1 += 1 for variant, prompt , hits, runs in sorted tally.items : print f"{variant:14} {prompt:10} {hits}/{runs}" The same transcript has an attachment record of type skill listing whose content is the exact line Claude saw. Reading that line was more useful than rereading our SKILL.md. This is one model, the default one on our account, at whatever effort level the CLI used without our user settings; a smaller model may draw the line somewhere else. Every listing had one skill in it. With dozens of skills competing, or with the bundled skills left on, a vague description may lose to a better-described neighbour instead of simply being ignored, and we did not test that. Descriptions long enough to be cut by the listing budget were out of scope. The counts are small: three runs per main cell, two per control. The pattern held in every cell, but a 3/3 is not a 100% rate, and one wording per request type is not a survey of how people ask for commit messages. All requests were in English. Everything ran in -p mode. We did not test an interactive session, where the Bash lookup in the "house format" run would have asked for approval instead of being denied, and where the conversation before the request might change the decision. We also did not test paths , user-invocable: false , or skills nested below the start directory. Finally, the first request of every run carried between 19,264 and 20,202 input tokens, in two groups about 800 tokens apart that did not line up with the variants. We did not find out what separated them, so we make no claim here about what each description costs in tokens. Rulestack sells skills, rules files, and hooks for Claude Code at rulestack.gumroad.com https://rulestack.gumroad.com?utm source=devto&utm medium=article&utm campaign=will-claude-code-call-your-skill-on-its-own-38-runs-one-skill-seven-ways-to-describe-it . Follow-up measurements go out on Bluesky at @ai-shop.bsky.social https://bsky.app/profile/ai-shop.bsky.social .