{"slug": "show-hn-skill-up-regression-testing-for-agent-skills", "title": "Show HN: Skill-up – Regression testing for Agent Skills", "summary": "Alibaba released skill-up, an open-source CLI tool for regression testing and iterative improvement of AI Agent Skills, along with a companion agent called skill-upper that automates the eval-to-evolution loop. The tool supports declarative YAML evaluation configs, multiple agent engines including Qoder CLI, Claude Code, and Codex, and produces structured reports in Anthropic-compatible and CI-friendly formats. Skill-up aims to make agent skill quality measurable and repeatable by replacing ad hoc testing with a reusable, multi-engine evaluation framework.", "body_md": "**skill-up** is an evaluation and evolution tool for Agent Skills.\n\n**Evaluation** makes Skill quality measurable and repeatable: declarative YAML cases run across multiple Agent Engines, use rule, script, or Agent judges, and produce structured reports locally or in CI.**Evolution** turns those results into the next improvement: through conversation,**skill-upper** reads failures, automatically repairs or expands the eval suite, reruns skill-up, and keeps iterating with you.\n\n**Eval-to-Evolution Loop with skill-upper**: Create evals through natural conversation, diagnose failures, automatically repair or expand cases, and rerun skill-up until the eval suite evolves.**Declarative Eval Config**: Define evaluation environment, engine, model, and cases through YAML (`eval.yaml`\n\n+`cases/*.yaml`\n\n).**Multi-Engine Support**: Works with Qoder CLI, Claude Code, and Codex as built-in Agent Engines, plus user-defined agents via`engine.custom`\n\n(local transport — see[docs/design/custom-engine.md](/alibaba/skill-up/blob/main/docs/design/custom-engine.md)).**Flexible Judging**: Supports`rule_based`\n\n,`script`\n\n, and`agent_judge`\n\nevaluation strategies.**Structured Reports**: Outputs Anthropic-compatible`grading.json`\n\n,`benchmark.json`\n\n,`benchmark.md`\n\n, plus`result.json`\n\n, JUnit XML, and HTML reports.**Anthropic Compatible**: Import`evals.json`\n\nvia`skill-up import`\n\n, or auto-detect with`--auto`\n\n.**CI-Ready**: Designed for local development and continuous integration pipelines.\n\nThe official [Agent Skills evaluation guide](https://agentskills.io/skill-creation/evaluating-skills) describes the right evaluation loop: write realistic cases, run with and without the Skill, grade outputs, aggregate results, and iterate. `skill-up`\n\nturns that workflow into a reusable CLI:\n\n- Replaces ad hoc run folders with a declarative\n`eval.yaml`\n\n+`cases/*.yaml`\n\nformat. - Closes the improvement loop: skill-upper can interpret failed reports, repair or add eval cases, and drive the next skill-up run through conversation.\n- Automates workspace setup, Skill installation, Agent Engine invocation, judging, and report generation.\n- Supports multiple engines (\n`claude_code`\n\n,`codex`\n\n,`qodercli`\n\n,`qwen_code`\n\n) instead of tying the workflow to one client. - Keeps compatibility with Anthropic-style\n`evals.json`\n\nwhile adding richer judges, CI-friendly commands, and structured reports.\n\nThe recommended way to use skill-up is through **skill-upper**, the Agent Skill\nshipped in this repository. It lets your AI agent create evals, run skill-up,\nunderstand failures, fix the Skill or its evals, add regression coverage, and\nrepeat the loop through conversation.\n\n```\n# Codex, global install\nnpx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a codex -y\n\n# Claude Code, global install\nnpx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a claude-code -y\n```\n\nYou normally do not need to install skill-up first. skill-upper checks for the CLI when it runs and guides the agent through installation if needed.\n\nOpen a project that contains your Skill's `SKILL.md`\n\nin Codex, Claude Code, or\nanother compatible Agent, then ask:\n\n```\nUse skill-upper to evaluate this Skill.\nRead SKILL.md, identify its most important behaviors, create realistic eval\ncases with appropriate judges, validate the configuration, and run skill-up.\nSummarize the results and the highest-impact failures.\n```\n\nskill-upper creates the declarative eval suite and drives the CLI for you:\n\n```\nmy-skill/\n  SKILL.md\n  evals/\n    eval.yaml\n    cases/\n      <case-id>.yaml\nmy-skill-workspace/\n  iteration-1/\n    result.json\n```\n\nContinue in the same conversation:\n\n```\nReview the latest skill-up results. For each failure, determine whether the\nSkill or the eval is wrong. Fix SKILL.md and supporting files, or repair the\neval case and judge as appropriate. Add regression cases for the bugs you\nfound, rerun skill-up, and continue until the important behaviors pass.\n```\n\nThis is the evolution loop: reports become fixes, fixes become regression cases, and every iteration makes the Skill and its eval suite stronger.\n\nYou can still install the CLI directly and hand-write `eval.yaml`\n\nand case\nfiles:\n\n```\ncurl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash\n```\n\nSee the official documentation for\n[Getting Started](https://alibaba.github.io/skill-up/guide/getting-started),\n[Writing Evals](https://alibaba.github.io/skill-up/guide/writing-evals),\n[CLI Reference](https://alibaba.github.io/skill-up/guide/cli-reference), and\n[User Configuration](https://alibaba.github.io/skill-up/guide/user-config).\nWindows-specific setup and limitations are covered in the\n[Windows guide](https://alibaba.github.io/skill-up/guide/windows).\n\nskill-up auto-loads an optional user-level config that supplies default OpenTelemetry env vars and per-environment runtime kwargs. The embedded defaults are empty; downstream consumers maintain their own config file.\n\n```\nembed (empty) < user (~/.config/skill-up/config.yaml) < project ($PWD/.skill-up.yaml) < explicit (--config)\n```\n\n| Source | Path |\n|---|---|\n`embed` |\nempty `Config{}` — no vendor defaults baked in |\n`user` |\n`$SKILL_UP_CONFIG` , else `$XDG_CONFIG_HOME/skill-up/config.yaml` , else `~/.config/skill-up/config.yaml` |\n`project` |\n`$PWD/.skill-up.yaml` |\n`explicit` |\n`--config <path>` (must exist) |\n\nMissing files at the `user`\n\nand `project`\n\nlayers are silently skipped; a missing `--config`\n\npath is a hard error. A corrupt config at any layer also fails the run.\n\n```\nskill-up init                            # writes a template to ~/.config/skill-up/config.yaml (XDG-aware)\nskill-up init --local                    # writes a template to $PWD/.skill-up.yaml\nskill-up init --print                    # prints the template to stdout\nskill-up init --force                    # overwrite an existing file\nskill-up init --config foo.yaml          # reads foo.yaml, writes it to ~/.config/skill-up/config.yaml\nskill-up init --config foo.yaml --local  # reads foo.yaml, writes it to $PWD/.skill-up.yaml\n```\n\nWith `--config <path>`\n\n, `init`\n\nreads that file (validating it as a skill-up\nconfig) and writes its raw bytes to the target — comments and formatting are\npreserved. Without `--config`\n\n, `init`\n\nwrites a commented YAML template.\n\n```\nschema_version: v1alpha1\nkind: SkillUpConfig\n\ntelemetry:\n  service_name: skill-up                              # OTEL_SERVICE_NAME\n  traces_exporter: otlp                                 # OTEL_TRACES_EXPORTER\n  traces:\n    endpoint: http://localhost:4317                     # OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (4317 for grpc, 4318/v1/traces for http/protobuf)\n    protocol: grpc                                      # OTEL_EXPORTER_OTLP_TRACES_PROTOCOL (grpc | http/protobuf); skill-up defaults to grpc\n  resource_attributes:                                  # serialized into OTEL_RESOURCE_ATTRIBUTES\n    deployment.environment: local\n  verbose: false                                        # if true, also enables OTEL_LOG_* payload capture\n\nenv:                                                    # arbitrary defaults, applied only-if-unset\n  OTEL_EXPORTER_OTLP_HEADERS: authorization=${OTLP_TOKEN}\n\nruntime_kwargs:                                         # keyed by environment.type\n  opensandbox:\n    base_url: http://localhost:8080\n    # extensions: '{}'\n```\n\nFor environment variables: any value already set in the process environment wins; the config only fills in missing keys.\n\nFor `runtime_kwargs`\n\n: explicit `--runtime-kwarg`\n\non `run`\n\n> `eval.yaml`\n\n`environment.kwargs`\n\n> user-config `runtime_kwargs[environment.type]`\n\n.\n\nPrefer `${ENV_VAR}`\n\nreferences inside the config file rather than baking secret literals. The redaction mechanism (`userconfig.Redact`\n\n) masks fields tagged `secret:\"true\"`\n\nwhen printing; currently no Config field carries the tag, but the mechanism is in place for future fields.\n\nUse `skill-up import`\n\nto migrate an Anthropic-compatible `evals.json`\n\ninto the YAML layout used by this repo:\n\n```\nskill-up import ./evals/evals.json --output ./evals\n```\n\n| Command | Description |\n|---|---|\n`skill-up run [path]` |\nRun evaluation cases and produce reports |\n`skill-up validate [path]` |\nValidate `eval.yaml` and case files |\n`skill-up list-cases [path]` |\nList all cases referenced by the config |\n`skill-up report <result.json>` |\nGenerate reports from a previous run |\n`skill-up import <evals.json>` |\nImport Anthropic `evals.json` to YAML cases |\n`skill-up debug judge <input.json>` |\nDebug judge module with a JSON input |\n`skill-up debug report <input.json>` |\nDebug report module with a JSON input |\n\nRun your Agent Skill evals in CI on every pull request — and check the same skill\n**across engines** (`claude_code`\n\n/ `codex`\n\n/ `qodercli`\n\n/ `qwen_code`\n\n) in one step. This repo\nships an action at its root ([ action.yml](/alibaba/skill-up/blob/main/action.yml)):\n\n```\n# .github/workflows/skill-eval.yml\nname: Skill Eval\non:\n  pull_request:\n    paths: ['skills/**', 'evals/**', '**/SKILL.md']\njobs:\n  eval:\n    runs-on: ubuntu-latest          # Docker container action — Linux only\n    steps:\n      - uses: actions/checkout@v4\n      - uses: alibaba/skill-up@main  # see \"Versioning\" below\n        with:\n          engine: claude_code        # or codex / qodercli / qwen_code; empty = let eval.yaml decide\n          api-key: ${{ secrets.ANTHROPIC_API_KEY }}\n          base-url: https://api.anthropic.com   # your model endpoint\n          skill-target: evals/eval.yaml\n```\n\nRequirements for the caller: a **Linux** runner (it's a Docker container action),\nand your model credential stored as a repo secret. The runner image is public, so\nno extra registry auth is needed.\n\nKey inputs: `engine`\n\n, `model`\n\n, `provider`\n\n, `api-key`\n\n, `base-url`\n\n, `skill-target`\n\n,\n`parallelism`\n\n. The action prebuilds skill-up + the three engine CLIs into its\nrunner image, so a run is just \"pull image, eval\". See [ action.yml](/alibaba/skill-up/blob/main/action.yml)\nfor the full input/output reference.\n\nThe container image includes a deliberately pinned skill-up CLI version. The\nversion is not resolved from `latest`\n\nwhen a workflow starts, so a given image\ndigest always runs the same CLI.\n\nThe `skill-up-version`\n\ninput is only a fallback for a custom image that does not\nalready contain the `skill-up`\n\nbinary. The official image contains the binary,\nso this input cannot override its bundled version. To find the effective\nversion, check the `skill-up --version`\n\nline in the Action log.\n\nPublishing a new skill-up CLI release does not automatically update the GitHub\nAction image. Maintainers must synchronize the pinned version, publish and test\na new runner image, and update the image digest in `action.yml`\n\n. The complete\nmaintainer procedure is documented in the\n[CI maintenance runbook](/alibaba/skill-up/blob/main/docs/guide/ci-maintenance.md#manual-skill-up-version-synchronization).\n\nThe production Action must use an immutable `sha256:`\n\nimage digest. Do not\nreplace it with `skill-up-runner:latest`\n\n; a mutable tag would allow existing\nAction references to change behavior without a repository commit.\n\n`uses:`\n\npoints at any git ref that contains `action.yml`\n\n. Pin a **release tag**\n(the first release that includes the action onward) or a commit SHA for stability;\n`@main`\n\nalways tracks the latest. Release tags published **before** the action was\nadded do not contain `action.yml`\n\nand cannot be used as the ref.\n\nA CLI release tag captures the `action.yml`\n\nand runner-image digest that existed\nwhen that tag was created. Because the current runner image is refreshed\nmanually after CLI release assets become available, do not assume that a CLI tag\nautomatically contains an Action image with the same CLI version. Until a\nseparate Action release tag process is introduced, use a post-refresh commit SHA\nfor an immutable reference or `@main`\n\nwhen intentionally following Action\nupdates.\n\nApache License 2.0 — see [LICENSE](/alibaba/skill-up/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-skill-up-regression-testing-for-agent-skills", "canonical_source": "https://github.com/alibaba/skill-up", "published_at": "2026-07-30 14:03:43+00:00", "updated_at": "2026-07-30 14:22:37.242651+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "ai-research"], "entities": ["Alibaba", "skill-up", "skill-upper", "Qoder CLI", "Claude Code", "Codex", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/show-hn-skill-up-regression-testing-for-agent-skills", "markdown": "https://wpnews.pro/news/show-hn-skill-up-regression-testing-for-agent-skills.md", "text": "https://wpnews.pro/news/show-hn-skill-up-regression-testing-for-agent-skills.txt", "jsonld": "https://wpnews.pro/news/show-hn-skill-up-regression-testing-for-agent-skills.jsonld"}}