{"slug": "skillcheck-update-scorer-fixes-cleaner-failures-honest-token-numbers", "title": "skillcheck Update: Scorer Fixes, Cleaner Failures, Honest Token Numbers", "summary": "The skillcheck static analyzer for SKILL.md files received a hardening update, fixing the description scorer, improving error handling for corrupt files, and correcting token estimate documentation. Median description scores rose from 75 to 90, and corrupt files now fail cleanly with exit code 2 instead of crashing. Token estimates are now benchmarked against tiktoken, showing a consistent over-estimation of 20-30%.", "body_md": "skillcheck is a static analyzer for `SKILL.md`\n\nfiles, the format agents like Claude Code, Copilot, Codex, and Cursor use to load reusable skills. It validates frontmatter, scores description discoverability, checks file references, enforces token budgets, and flags cross-agent compatibility issues. No network calls, no LLM calls, no file mutations. Runs as a CLI, a GitHub Action, or a pre-commit hook.\n\n```\npip install skillcheck\nskillcheck skills/\n```\n\nLatest pass was hardening and accuracy, not features. Here's what changed and why.\n\n**Description scores went up.** Skills that were scoring low because the scorer was broken will now see a jump in scoring. Median across the reference corpus went from 75 to 90. `--explain-score`\n\nalso now tells you which pattern hits or misses instead of just a number. The score exists to predict whether an agent will actually find and trigger your skill, so a scorer that under-credits good descriptions defeats the point. The fix was validated against real-world skills, and the separation held: filler still scores 28-65, well-written descriptions 85-100.\n\n**Corrupt files now fail cleanly instead of crashing.** Before, a bad history ledger or non-UTF-8 skillcheck.toml above the skill dumped a Python traceback. It's now a clear error naming the file and byte offset (exit code 2). Config discovery walks up the directory tree, so one bad file could break every scan under it. Now every untrusted read (ingest, history, config) goes through the same guard before parsing, so they all reject the same way.\n\n**README has been corrected in regards to token estimates.** Without tiktoken, expect roughly 20-30% over-estimation, so install the extra if you're near a budget limit. The offline heuristic feeds the budget checks and its accuracy had never actually been measured, just assumed. It's benchmarked against tiktoken across the full corpus now, and the documented numbers are the measured ones.\n\n```\npip install \"skillcheck[tiktoken]\"\n```\n\nThe rest of the pass is invisible on purpose: flag-conflict logic consolidated to one source of truth, golden-file tests pinning exact diagnostic output, coverage floor raised from 75% to 80% (actual sits at 90%). Diagnostic output across the corpus verified byte-for-byte identical before and after. Nothing changed except what's above.\n\nStatic analyzer for `SKILL.md`\n\nfiles. Validates frontmatter, body sizing, file references, and cross-agent compatibility against the [agentskills.io specification](https://agentskills.io/specification). No network calls. No LLM API calls. No file mutations.\n\n1058 tests cover all rule modules.\n\n```\npip install skillcheck\n```\n\nRequires Python 3.10 or later. For more accurate token estimates, install the optional extra:\n\n```\npip install \"skillcheck[tiktoken]\"\n```\n\nToken counts are estimates, and the sizing rules report them as such. The bands below are measured, not estimated: `scripts/measure_token_error.py`\n\ncompares the offline heuristic against `tiktoken`\n\n`cl100k_base`\n\nacross a corpus of 61 real `SKILL.md`\n\nfiles, for the three spans the rules actually size.\n\n| Span | Rule | Median error | p95 error | Direction |\n|---|---|---|---|---|\n| Whole file | `sizing.total-tokens` |\n23.0% | 30.7% | over-estimates 61/61 |\n| Frontmatter | `disclosure.metadata-budget` |\n25.9% | 35.3% | over-estimates 61/61 |\n| Body | `disclosure.body-budget` |\n22.7% | 30.7% | over-estimates 61/61 |\n\nThe bias is one-directional: the offline heuristic read high on every file in the corpus. For a budget check that…\n\nIf skillcheck flags something in your skills that looks wrong, open an issue. The reference corpus grows from real-world cases and the scorer improves with them.", "url": "https://wpnews.pro/news/skillcheck-update-scorer-fixes-cleaner-failures-honest-token-numbers", "canonical_source": "https://dev.to/moonrunnerkc/skillcheck-update-scorer-fixes-cleaner-failures-honest-token-numbers-2j8h", "published_at": "2026-08-23 03:04:23+00:00", "updated_at": "2026-08-23 03:43:06.884946+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["skillcheck", "Claude Code", "Copilot", "Codex", "Cursor", "agentskills.io", "tiktoken"], "alternates": {"html": "https://wpnews.pro/news/skillcheck-update-scorer-fixes-cleaner-failures-honest-token-numbers", "markdown": "https://wpnews.pro/news/skillcheck-update-scorer-fixes-cleaner-failures-honest-token-numbers.md", "text": "https://wpnews.pro/news/skillcheck-update-scorer-fixes-cleaner-failures-honest-token-numbers.txt", "jsonld": "https://wpnews.pro/news/skillcheck-update-scorer-fixes-cleaner-failures-honest-token-numbers.jsonld"}}