{"slug": "skillgrade-unit-tests-for-your-agent-skills", "title": "Skillgrade: \"Unit tests\" for your agent skills", "summary": "Skillgrade, a new open-source tool, enables developers to create and run unit tests for AI agent skills, ensuring agents correctly discover and use custom skills. The tool supports multiple AI agents including Gemini, Claude, and Codex, and provides features like smoke testing, regression testing, and CI integration.", "body_md": "The easiest way to evaluate your [Agent Skills](https://agentskills.io/home). Tests that AI agents correctly discover and use your skills.\n\nSee [examples/](/mgechev/skillgrade/blob/main/examples) — [superlint](/mgechev/skillgrade/blob/main/examples/superlint) (simple) and [angular-modern](/mgechev/skillgrade/blob/main/examples/angular-modern) (TypeScript grader).\n\n**Prerequisites**: Node.js 20+, Docker\n\n```\nnpm i -g skillgrade\n```\n\n**1. Initialize** — go to your skill directory (must have `SKILL.md`\n\n) and scaffold:\n\n```\ncd my-skill/\nGEMINI_API_KEY=your-key skillgrade init    # or ANTHROPIC_API_KEY / OPENAI_API_KEY\n# Use --force to overwrite an existing eval.yaml\n```\n\nGenerates `eval.yaml`\n\nwith AI-powered tasks and graders. Without an API key, creates a well-commented template.\n\n**2. Edit** — customize `eval.yaml`\n\nfor your skill (see [eval.yaml Reference](#evalyaml-reference)).\n\n**3. Run**:\n\n```\nGEMINI_API_KEY=your-key skillgrade --smoke\n```\n\nThe agent is auto-detected from your API key: `GEMINI_API_KEY`\n\n→ Gemini, `ANTHROPIC_API_KEY`\n\n→ Claude, `OPENAI_API_KEY`\n\n→ Codex. Override with `--agent=claude`\n\n.\n\n**4. Review**:\n\n```\nskillgrade preview          # CLI report\nskillgrade preview browser  # web UI → http://localhost:3847\n```\n\nReports are saved to `$TMPDIR/skillgrade/<skill-name>/results/`\n\n. Override with `--output=DIR`\n\n.\n\n| Flag | Trials | Use Case |\n|---|---|---|\n`--smoke` |\n5 | Quick capability check |\n`--reliable` |\n15 | Reliable pass rate estimate |\n`--regression` |\n30 | High-confidence regression detection |\n\n| Flag | Description |\n|---|---|\n`--eval=NAME[,NAME]` |\nRun specific evals by name (comma-separated) |\n`--grader=TYPE` |\nRun only graders of a type (`deterministic` or `llm_rubric` ) |\n`--trials=N` |\nOverride trial count |\n`--parallel=N` |\nRun trials concurrently |\n`--agent=gemini|claude|codex|acp|opencode|command` |\nOverride agent (default: auto-detect from API key) |\n`--provider=docker|local` |\nOverride provider |\n`--acp-command=CMD` |\nACP agent command (e.g., `gemini --acp` ) |\n`--command=CMD` |\nCommand to run for the `command` agent (e.g., `node mycli.js` ) |\n`--opencode-agent=NAME` |\nOpenCode agent (build|plan|explore) |\n`--opencode-model=MODEL` |\nOpenCode model (provider/model format) |\n`--output=DIR` |\nOutput directory (default: `$TMPDIR/skillgrade` ) |\n`--validate` |\nVerify graders using reference solutions |\n`--ci` |\nCI mode: exit non-zero if below threshold |\n`--threshold=0.8` |\nPass rate threshold for CI mode |\n`--preview` |\nShow CLI results after running |\n\n```\nversion: \"1\"\n\n# Optional: explicit path to skill directory (defaults to auto-detecting SKILL.md)\n# skill: path/to/my-skill\n\ndefaults:\n  agent: gemini          # gemini | claude | codex | acp | opencode | command\n  provider: docker       # docker | local\n  trials: 5\n  timeout: 300           # seconds\n  threshold: 0.8         # for --ci mode\n  grader_model: gemini-3-flash-preview  # default LLM grader model\n  grader_provider: gemini               # default LLM grader provider: gemini | anthropic | openai\n  command: node mycli.js # command to run when agent is 'command' (see Custom Command Agent)\n  acp:                   # ACP agent configuration (optional)\n    command: gemini --acp  # command to start ACP-compatible agent\n    env:                  # optional environment variables\n      DEBUG: \"1\"\n  docker:\n    base: node:20-slim\n    setup: |             # extra commands run during image build\n      apt-get update && apt-get install -y jq\n  environment:           # container resource limits\n    cpus: 2\n    memory_mb: 2048\n\ntasks:\n  - name: fix-linting-errors\n    instruction: |\n      Use the superlint tool to fix coding standard violations in app.js.\n\n    workspace:                           # files copied into the container\n      - src: fixtures/broken-app.js\n        dest: app.js\n      - src: bin/superlint\n        dest: /usr/local/bin/superlint\n        chmod: \"+x\"\n\n    graders:\n      - type: deterministic\n        setup: npm install typescript    # grader-specific deps (optional)\n        run: npx ts-node graders/check.ts\n        weight: 0.7\n      - type: llm_rubric\n        rubric: |\n          Did the agent follow the check → fix → verify workflow?\n        provider: gemini                 # optional: gemini (default) | anthropic | openai\n        model: gemini-2.0-flash          # optional model override\n        weight: 0.3\n\n    # Per-task overrides (optional)\n    agent: claude\n    grader_provider: anthropic   # override default LLM grader provider\n    trials: 10\n    timeout: 600\n```\n\nString values (`instruction`\n\n, `rubric`\n\n, `run`\n\n) support **file references** — if the value is a valid file path, its contents are read automatically:\n\n```\ninstruction: instructions/fix-linting.md\nrubric: rubrics/workflow-quality.md\n```\n\nRuns a command and parses JSON from stdout:\n\n```\n- type: deterministic\n  run: bash graders/check.sh\n  weight: 0.7\n```\n\nOutput format:\n\n```\n{\n  \"score\": 0.67,\n  \"details\": \"2/3 checks passed\",\n  \"checks\": [\n    {\"name\": \"file-created\", \"passed\": true, \"message\": \"Output file exists\"},\n    {\"name\": \"content-correct\", \"passed\": false, \"message\": \"Missing expected output\"}\n  ]\n}\n```\n\n`score`\n\n(0.0–1.0) and `details`\n\nare required. `checks`\n\nis optional.\n\n**Bash example:**\n\n``` bash\n#!/bin/bash\npassed=0; total=2\nc1_pass=false c1_msg=\"File missing\"\nc2_pass=false c2_msg=\"Content wrong\"\n\nif test -f output.txt; then\n  passed=$((passed + 1)); c1_pass=true; c1_msg=\"File exists\"\nfi\nif grep -q \"expected\" output.txt 2>/dev/null; then\n  passed=$((passed + 1)); c2_pass=true; c2_msg=\"Content correct\"\nfi\n\nscore=$(awk \"BEGIN {printf \\\"%.2f\\\", $passed/$total}\")\necho \"{\\\"score\\\":$score,\\\"details\\\":\\\"$passed/$total passed\\\",\\\"checks\\\":[{\\\"name\\\":\\\"file\\\",\\\"passed\\\":$c1_pass,\\\"message\\\":\\\"$c1_msg\\\"},{\\\"name\\\":\\\"content\\\",\\\"passed\\\":$c2_pass,\\\"message\\\":\\\"$c2_msg\\\"}]}\"\n```\n\nUse\n\n`awk`\n\nfor arithmetic —`bc`\n\nis not available in`node:20-slim`\n\n.\n\nEvaluates the agent's session transcript against qualitative criteria:\n\n```\n- type: llm_rubric\n  rubric: |\n    Workflow Compliance (0-0.5):\n    - Did the agent follow the mandatory 3-step workflow?\n\n    Efficiency (0-0.5):\n    - Completed in ≤5 commands?\n  weight: 0.3\n  provider: gemini           # gemini (default) | anthropic | openai\n  model: gemini-2.0-flash    # optional, auto-detected from API key\n```\n\nThe `provider`\n\nfield selects which LLM API to call:\n\n| Provider | API Key Env Var | Base URL Env Var (optional) | Default Model |\n|---|---|---|---|\n`gemini` |\n`GEMINI_API_KEY` |\n- | `gemini-3-flash-preview` |\n`anthropic` |\n`ANTHROPIC_API_KEY` |\n`ANTHROPIC_BASE_URL` |\n`claude-sonnet-4-20250514` |\n`openai` |\n`OPENAI_API_KEY` |\n`OPENAI_BASE_URL` |\n`gpt-4o` |\n\n`ANTHROPIC_BASE_URL`\n\nand `OPENAI_BASE_URL`\n\nenable custom/self-hosted endpoints (Ollama, vLLM, etc.).\n\n```\ngraders:\n  - type: deterministic\n    run: bash graders/check.sh\n    weight: 0.7      # 70% — did it work?\n  - type: llm_rubric\n    rubric: rubrics/quality.md\n    weight: 0.3      # 30% — was the approach good?\n```\n\nFinal reward = `Σ (grader_score × weight) / Σ weight`\n\nUse `--provider=local`\n\nin CI — the runner is already an ephemeral sandbox, so Docker adds overhead without benefit.\n\n```\n# .github/workflows/skillgrade.yml\n- run: |\n    npm i -g skillgrade\n    cd skills/superlint\n    GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }} skillgrade --regression --ci --provider=local\n```\n\nExits with code 1 if pass rate falls below `--threshold`\n\n(default: 0.8).\n\nTip: Use`docker`\n\n(the default) for local development to protect your machine. In CI,`local`\n\nis faster and simpler.\n\n| Variable | Used by |\n|---|---|\n`GEMINI_API_KEY` |\nAgent execution, LLM grading (`provider: gemini` ), `skillgrade init` |\n`ANTHROPIC_API_KEY` |\nAgent execution, LLM grading (`provider: anthropic` ), `skillgrade init` |\n`OPENAI_API_KEY` |\nAgent execution (Codex), LLM grading (`provider: openai` ), `skillgrade init` |\n`ANTHROPIC_BASE_URL` |\nLLM grading (`provider: anthropic` ) — custom Anthropic-compatible endpoint |\n`OPENAI_BASE_URL` |\nLLM grading (`provider: openai` ) — custom OpenAI-compatible endpoint (Ollama, vLLM, etc.) |\n\nVariables are also loaded from `.env`\n\nin the skill directory. Shell values override `.env`\n\n. All values are **redacted** from persisted session logs.\n\nBring your own agent. The built-in adapters (`gemini`\n\n, `claude`\n\n, `codex`\n\n, ...) cover the popular CLIs, but you can point skillgrade at **any command** — a custom script, a [deepagents](https://github.com/langchain-ai/deepagents) loop, or a small orchestrator over the Claude/OpenAI SDKs — without forking the package or implementing an ACP server.\n\n```\nskillgrade --agent=command --command=\"node mycli.js\"\n```\n\nOr in `eval.yaml`\n\n:\n\n```\ndefaults:\n  agent: command\n  command: \"node mycli.js\"\n  provider: local        # run on the host; or use docker + docker.setup to install your CLI\n```\n\n`command`\n\ncan also be set per task to override the default.\n\nThe task instruction is **piped to your command's stdin** (skillgrade writes it to `/tmp/.prompt.md`\n\n, then runs `cat /tmp/.prompt.md | <command>`\n\ninside the workspace directory). If your CLI takes the prompt as an argument instead, wrap it in a one-line script that reads stdin.\n\nYour command runs in the workspace and is free to read/edit files there — graders score the resulting workspace state (and any live checks), not your command's stdout, so any agent slots in cleanly.\n\nis the simplest fit for a custom agent: your command runs on the host with your tools already installed.`provider: local`\n\nstill works — skillgrade does`provider: docker`\n\n**not** auto-install anything for the`command`\n\nagent, so install your CLI and dependencies via`docker.setup`\n\n:\n\n```\ndefaults:\n  agent: command\n  command: \"mycli run\"\n  docker:\n    base: node:20-slim\n    setup: \"npm install -g my-cli-package\"\n```\n\n[OpenCode](https://opencode.ai/) is an AI coding agent that supports multiple AI models and specialized subagents.\n\n```\n# Use OpenCode with default agent and model\nskillgrade --agent=opencode\n\n# Specify OpenCode agent (build|plan|explore)\nskillgrade --agent=opencode --opencode-agent=build\n\n# Specify both agent and model (provider/model format)\nskillgrade --agent=opencode --opencode-agent=build --opencode-model=anthropic/claude-sonnet-4-20250514\n```\n\n| Agent | Description |\n|---|---|\n`build` |\nDefault primary agent with full tool access |\n`plan` |\nRead-only planning/analysis agent |\n`explore` |\nFast codebase exploration agent |\n\nModels are specified in `provider/model`\n\nformat:\n\n| Model | Format |\n|---|---|\n| Claude Sonnet 4 | `anthropic/claude-sonnet-4-20250514` |\n| GPT 5.1 Codex | `opencode/gpt-5.1-codex` |\n\n| Flag | Description |\n|---|---|\n`--agent=opencode` |\nUse OpenCode agent |\n`--opencode-agent=NAME` |\nOpenCode agent (build|plan|explore) |\n`--opencode-model=MODEL` |\nOpenCode model (provider/model format) |\n\n- skillgrade invokes OpenCode CLI with\n`opencode run`\n\n- Passes instruction via temp file to avoid shell escaping issues\n- Supports both agent and model specification\n- Works with\n`--provider=docker`\n\nor`--provider=local`\n\n[Agent Client Protocol (ACP)](https://agentclientprotocol.com/) is an open protocol that standardizes communication between AI coding agents and clients. Using an ACP-compatible agent allows you to evaluate skills without managing API keys directly.\n\n```\n# Use Gemini CLI in ACP mode (requires gemini CLI installed)\nskillgrade --agent=acp --acp-command=\"gemini --acp\"\n\n# Or configure in eval.yaml\ndefaults:\n  agent: acp\n  acp:\n    command: gemini --acp\n```\n\nAny agent that supports the ACP protocol can be used:\n\n| Agent | Command |\n|---|---|\n| Gemini CLI | `gemini --acp` |\n| Other ACP agents | Check agent documentation |\n\n- skillgrade starts the ACP agent as a subprocess\n- Communication happens via JSON-RPC 2.0 over stdio\n- No API key required — authentication is handled by the ACP agent\n- Works best with\n`--provider=local`\n\nsince the ACP agent needs to be available in your environment\n\n| Flag | Description |\n|---|---|\n`--agent=acp` |\nUse ACP-compatible agent |\n`--acp-command=CMD` |\nCommand to start the ACP agent |\n\nThe `--acp-command`\n\ncan also be set in `eval.yaml`\n\nunder `defaults.acp.command`\n\n.\n\n**Grade outcomes, not steps.** Check that the file was fixed, not that the agent ran a specific command.**Instructions must name output files.** If the grader checks for`output.html`\n\n, the instruction must tell the agent to save as`output.html`\n\n.**Validate graders first.** Use`--validate`\n\nwith a reference solution before running real evals.**Start small.** 3–5 well-designed tasks beat 50 noisy ones.\n\nFor a comprehensive guide on writing high-quality skills, check out [skills-best-practices](https://github.com/mgechev/skills-best-practices/). You can also install the skill creator skill to help author skills:\n\n```\nnpx skills add mgechev/skills-best-practices\n```\n\nMIT\n\n*Inspired by SkillsBench and Demystifying Evals for AI Agents.*", "url": "https://wpnews.pro/news/skillgrade-unit-tests-for-your-agent-skills", "canonical_source": "https://github.com/mgechev/skillgrade", "published_at": "2026-07-10 20:25:10+00:00", "updated_at": "2026-07-10 20:35:16.936502+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["Skillgrade", "Gemini", "Claude", "Codex", "Node.js", "Docker"], "alternates": {"html": "https://wpnews.pro/news/skillgrade-unit-tests-for-your-agent-skills", "markdown": "https://wpnews.pro/news/skillgrade-unit-tests-for-your-agent-skills.md", "text": "https://wpnews.pro/news/skillgrade-unit-tests-for-your-agent-skills.txt", "jsonld": "https://wpnews.pro/news/skillgrade-unit-tests-for-your-agent-skills.jsonld"}}