Skillgrade: "Unit tests" for your agent skills

Skillgrade, a new open-source tool, enables developers to create and run unit tests for AI agent skills, ensuring agents correctly discover and use custom skills. The tool supports multiple AI agents including Gemini, Claude, and Codex, and provides features like smoke testing, regression testing, and CI integration.

The easiest way to evaluate your Agent Skills https://agentskills.io/home . Tests that AI agents correctly discover and use your skills. See examples/ /mgechev/skillgrade/blob/main/examples — superlint /mgechev/skillgrade/blob/main/examples/superlint simple and angular-modern /mgechev/skillgrade/blob/main/examples/angular-modern TypeScript grader . Prerequisites : Node.js 20+, Docker npm i -g skillgrade 1. Initialize — go to your skill directory must have SKILL.md and scaffold: cd my-skill/ GEMINI API KEY=your-key skillgrade init or ANTHROPIC API KEY / OPENAI API KEY Use --force to overwrite an existing eval.yaml Generates eval.yaml with AI-powered tasks and graders. Without an API key, creates a well-commented template. 2. Edit — customize eval.yaml for your skill see eval.yaml Reference evalyaml-reference . 3. Run : GEMINI API KEY=your-key skillgrade --smoke The agent is auto-detected from your API key: GEMINI API KEY → Gemini, ANTHROPIC API KEY → Claude, OPENAI API KEY → Codex. Override with --agent=claude . 4. Review : skillgrade preview CLI report skillgrade preview browser web UI → http://localhost:3847 Reports are saved to $TMPDIR/skillgrade/<skill-name /results/ . Override with --output=DIR . | Flag | Trials | Use Case | |---|---|---| --smoke | 5 | Quick capability check | --reliable | 15 | Reliable pass rate estimate | --regression | 30 | High-confidence regression detection | | Flag | Description | |---|---| --eval=NAME ,NAME | Run specific evals by name comma-separated | --grader=TYPE | Run only graders of a type deterministic or llm rubric | --trials=N | Override trial count | --parallel=N | Run trials concurrently | --agent=gemini|claude|codex|acp|opencode|command | Override agent default: auto-detect from API key | --provider=docker|local | Override provider | --acp-command=CMD | ACP agent command e.g., gemini --acp | --command=CMD | Command to run for the command agent e.g., node mycli.js | --opencode-agent=NAME | OpenCode agent build|plan|explore | --opencode-model=MODEL | OpenCode model provider/model format | --output=DIR | Output directory default: $TMPDIR/skillgrade | --validate | Verify graders using reference solutions | --ci | CI mode: exit non-zero if below threshold | --threshold=0.8 | Pass rate threshold for CI mode | --preview | Show CLI results after running | version: "1" Optional: explicit path to skill directory defaults to auto-detecting SKILL.md skill: path/to/my-skill defaults: agent: gemini gemini | claude | codex | acp | opencode | command provider: docker docker | local trials: 5 timeout: 300 seconds threshold: 0.8 for --ci mode grader model: gemini-3-flash-preview default LLM grader model grader provider: gemini default LLM grader provider: gemini | anthropic | openai command: node mycli.js command to run when agent is 'command' see Custom Command Agent acp: ACP agent configuration optional command: gemini --acp command to start ACP-compatible agent env: optional environment variables DEBUG: "1" docker: base: node:20-slim setup: | extra commands run during image build apt-get update && apt-get install -y jq environment: container resource limits cpus: 2 memory mb: 2048 tasks: - name: fix-linting-errors instruction: | Use the superlint tool to fix coding standard violations in app.js. workspace: files copied into the container - src: fixtures/broken-app.js dest: app.js - src: bin/superlint dest: /usr/local/bin/superlint chmod: "+x" graders: - type: deterministic setup: npm install typescript grader-specific deps optional run: npx ts-node graders/check.ts weight: 0.7 - type: llm rubric rubric: | Did the agent follow the check → fix → verify workflow? provider: gemini optional: gemini default | anthropic | openai model: gemini-2.0-flash optional model override weight: 0.3 Per-task overrides optional agent: claude grader provider: anthropic override default LLM grader provider trials: 10 timeout: 600 String values instruction , rubric , run support file references — if the value is a valid file path, its contents are read automatically: instruction: instructions/fix-linting.md rubric: rubrics/workflow-quality.md Runs a command and parses JSON from stdout: - type: deterministic run: bash graders/check.sh weight: 0.7 Output format: { "score": 0.67, "details": "2/3 checks passed", "checks": {"name": "file-created", "passed": true, "message": "Output file exists"}, {"name": "content-correct", "passed": false, "message": "Missing expected output"} } score 0.0–1.0 and details are required. checks is optional. Bash example: bash /bin/bash passed=0; total=2 c1 pass=false c1 msg="File missing" c2 pass=false c2 msg="Content wrong" if test -f output.txt; then passed=$ passed + 1 ; c1 pass=true; c1 msg="File exists" fi if grep -q "expected" output.txt 2 /dev/null; then passed=$ passed + 1 ; c2 pass=true; c2 msg="Content correct" fi score=$ awk "BEGIN {printf \"%.2f\", $passed/$total}" echo "{\"score\":$score,\"details\":\"$passed/$total passed\",\"checks\": {\"name\":\"file\",\"passed\":$c1 pass,\"message\":\"$c1 msg\"},{\"name\":\"content\",\"passed\":$c2 pass,\"message\":\"$c2 msg\"} }" Use awk for arithmetic — bc is not available in node:20-slim . Evaluates the agent's session transcript against qualitative criteria: - type: llm rubric rubric: | Workflow Compliance 0-0.5 : - Did the agent follow the mandatory 3-step workflow? Efficiency 0-0.5 : - Completed in ≤5 commands? weight: 0.3 provider: gemini gemini default | anthropic | openai model: gemini-2.0-flash optional, auto-detected from API key The provider field selects which LLM API to call: | Provider | API Key Env Var | Base URL Env Var optional | Default Model | |---|---|---|---| gemini | GEMINI API KEY | - | gemini-3-flash-preview | anthropic | ANTHROPIC API KEY | ANTHROPIC BASE URL | claude-sonnet-4-20250514 | openai | OPENAI API KEY | OPENAI BASE URL | gpt-4o | ANTHROPIC BASE URL and OPENAI BASE URL enable custom/self-hosted endpoints Ollama, vLLM, etc. . graders: - type: deterministic run: bash graders/check.sh weight: 0.7 70% — did it work? - type: llm rubric rubric: rubrics/quality.md weight: 0.3 30% — was the approach good? Final reward = Σ grader score × weight / Σ weight Use --provider=local in CI — the runner is already an ephemeral sandbox, so Docker adds overhead without benefit. .github/workflows/skillgrade.yml - run: | npm i -g skillgrade cd skills/superlint GEMINI API KEY=${{ secrets.GEMINI API KEY }} skillgrade --regression --ci --provider=local Exits with code 1 if pass rate falls below --threshold default: 0.8 . Tip: Use docker the default for local development to protect your machine. In CI, local is faster and simpler. | Variable | Used by | |---|---| GEMINI API KEY | Agent execution, LLM grading provider: gemini , skillgrade init | ANTHROPIC API KEY | Agent execution, LLM grading provider: anthropic , skillgrade init | OPENAI API KEY | Agent execution Codex , LLM grading provider: openai , skillgrade init | ANTHROPIC BASE URL | LLM grading provider: anthropic — custom Anthropic-compatible endpoint | OPENAI BASE URL | LLM grading provider: openai — custom OpenAI-compatible endpoint Ollama, vLLM, etc. | Variables are also loaded from .env in the skill directory. Shell values override .env . All values are redacted from persisted session logs. Bring your own agent. The built-in adapters gemini , claude , codex , ... cover the popular CLIs, but you can point skillgrade at any command — a custom script, a deepagents https://github.com/langchain-ai/deepagents loop, or a small orchestrator over the Claude/OpenAI SDKs — without forking the package or implementing an ACP server. skillgrade --agent=command --command="node mycli.js" Or in eval.yaml : defaults: agent: command command: "node mycli.js" provider: local run on the host; or use docker + docker.setup to install your CLI command can also be set per task to override the default. The task instruction is piped to your command's stdin skillgrade writes it to /tmp/.prompt.md , then runs cat /tmp/.prompt.md | <command inside the workspace directory . If your CLI takes the prompt as an argument instead, wrap it in a one-line script that reads stdin. Your command runs in the workspace and is free to read/edit files there — graders score the resulting workspace state and any live checks , not your command's stdout, so any agent slots in cleanly. is the simplest fit for a custom agent: your command runs on the host with your tools already installed. provider: local still works — skillgrade does provider: docker not auto-install anything for the command agent, so install your CLI and dependencies via docker.setup : defaults: agent: command command: "mycli run" docker: base: node:20-slim setup: "npm install -g my-cli-package" OpenCode https://opencode.ai/ is an AI coding agent that supports multiple AI models and specialized subagents. Use OpenCode with default agent and model skillgrade --agent=opencode Specify OpenCode agent build|plan|explore skillgrade --agent=opencode --opencode-agent=build Specify both agent and model provider/model format skillgrade --agent=opencode --opencode-agent=build --opencode-model=anthropic/claude-sonnet-4-20250514 | Agent | Description | |---|---| build | Default primary agent with full tool access | plan | Read-only planning/analysis agent | explore | Fast codebase exploration agent | Models are specified in provider/model format: | Model | Format | |---|---| | Claude Sonnet 4 | anthropic/claude-sonnet-4-20250514 | | GPT 5.1 Codex | opencode/gpt-5.1-codex | | Flag | Description | |---|---| --agent=opencode | Use OpenCode agent | --opencode-agent=NAME | OpenCode agent build|plan|explore | --opencode-model=MODEL | OpenCode model provider/model format | - skillgrade invokes OpenCode CLI with opencode run - Passes instruction via temp file to avoid shell escaping issues - Supports both agent and model specification - Works with --provider=docker or --provider=local Agent Client Protocol ACP https://agentclientprotocol.com/ is an open protocol that standardizes communication between AI coding agents and clients. Using an ACP-compatible agent allows you to evaluate skills without managing API keys directly. Use Gemini CLI in ACP mode requires gemini CLI installed skillgrade --agent=acp --acp-command="gemini --acp" Or configure in eval.yaml defaults: agent: acp acp: command: gemini --acp Any agent that supports the ACP protocol can be used: | Agent | Command | |---|---| | Gemini CLI | gemini --acp | | Other ACP agents | Check agent documentation | - skillgrade starts the ACP agent as a subprocess - Communication happens via JSON-RPC 2.0 over stdio - No API key required — authentication is handled by the ACP agent - Works best with --provider=local since the ACP agent needs to be available in your environment | Flag | Description | |---|---| --agent=acp | Use ACP-compatible agent | --acp-command=CMD | Command to start the ACP agent | The --acp-command can also be set in eval.yaml under defaults.acp.command . Grade outcomes, not steps. Check that the file was fixed, not that the agent ran a specific command. Instructions must name output files. If the grader checks for output.html , the instruction must tell the agent to save as output.html . Validate graders first. Use --validate with a reference solution before running real evals. Start small. 3–5 well-designed tasks beat 50 noisy ones. For a comprehensive guide on writing high-quality skills, check out skills-best-practices https://github.com/mgechev/skills-best-practices/ . You can also install the skill creator skill to help author skills: npx skills add mgechev/skills-best-practices MIT Inspired by SkillsBench and Demystifying Evals for AI Agents.