CI/CD for Skills

A new lesson from the Agent Skills Mastery course outlines an eight-stage CI/CD pipeline for agent skills, implemented as GitHub Actions jobs, covering lint, validate, test, evaluate, security scan, publish, registry, and deploy stages. The pipeline automates the skill lifecycle, with lint and validate blocking merges while security scans warn on issues like expanded allowed-tools. The lesson is part of a production-focused curriculum on agent skills engineering.

· Agentic AI · 5 min read 📋 Prerequisites - Skill Lifecycle and Versioning previous lesson 🎯 What You'll Learn - Design an eight-stage CI/CD pipeline for skills, mapped to lifecycle stages - Implement each pipeline stage as a real GitHub Actions job - Explain what should block a merge versus what should only warn Turning the Lifecycle Into Something That Runs The previous lesson defined eight lifecycle stages a skill moves through. Left as a manual process, “review” means someone remembers to check, and “test” means someone remembers to run the eval suite before merging — which, at any real scale, means they sometimes don’t. CI/CD is what turns the lifecycle from a set of good intentions into something that runs the same way, automatically, on every change. The Pipeline Shape Lint → Validate → Test → Evaluate → Security Scan → Publish → Registry → Deploy Each stage maps to something you’ve already learned to do by hand earlier in this course — this lesson is about automating it, in order, with the right stages blocking a merge and the right ones only warning. Stage 1: Lint Basic formatting and structure checks — is the YAML frontmatter well-formed, are required fields present, does the body follow the project’s Markdown conventions. Fast, cheap, and should fail immediately on anything a human reviewer shouldn’t have to catch by eye. Stage 2: Validate The skills-ref validate check from the free course’s SKILL.md Fundamentals /courses/agent-skills-mastery/skill-md-format-specification — naming rules, required-field presence, the format-level checks that determine whether a skill will even load correctly in any client. - name: Validate SKILL.md format run: skills-ref validate ./skills/config-validator Stage 3: Test The trigger-testing practice from the free course’s Testing Skills /courses/agent-skills-mastery/testing-skills — the labeled should-trigger / should-not-trigger query set, run automatically rather than by hand. Stage 4: Evaluate The full evaluation workspace from Testing and Evaluation at Scale /courses/production-agent-skills-engineering/testing-evaluation-at-scale — with-skill vs. without-skill comparison, assertion grading, benchmark collection. This is typically the slowest and most expensive stage, since it involves actually running the skill against real prompts. Stage 5: Security Scan The checklist from Security and Governance /courses/production-agent-skills-engineering/skill-security-governance — automated where it can be: a secrets scanner checking for hardcoded credentials, a check that allowed-tools hasn’t silently expanded beyond what a prior review approved, a description-length or keyword check flagging unusually broad descriptions for extra human review. Stage 6: Publish The skill’s artifacts — SKILL.md and its bundled files — get packaged for distribution, tagged with the version determined by the change per the previous lesson’s semver discipline . Stage 7: Registry The package is pushed to wherever skills are actually discovered from at your organization — covered in full in the next lesson, Skill Registries and Marketplaces /courses/production-agent-skills-engineering/skill-registries-marketplaces . Stage 8: Deploy The new version becomes live for its intended audience — which, depending on the skill’s trust level, might mean immediate availability or a staged rollout to a subset of users first. A Real Workflow name: skill-ci on: pull request: paths: - 'skills/ ' jobs: lint-and-validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Lint SKILL.md frontmatter run: ./scripts/lint-skills.sh - name: Validate against Agent Skills spec run: skills-ref validate ./skills/${{ github.event.pull request.changed skill }} test-triggers: needs: lint-and-validate runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run trigger eval set run: ./scripts/run-trigger-evals.sh skills/${{ github.event.pull request.changed skill }} evaluate-output: needs: test-triggers runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run with-skill / without-skill comparison run: ./scripts/run-output-evals.sh skills/${{ github.event.pull request.changed skill }} - name: Upload benchmark results uses: actions/upload-artifact@v4 with: name: benchmark-results path: workspace/benchmark.json security-scan: needs: lint-and-validate runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Scan for hardcoded secrets run: ./scripts/scan-secrets.sh skills/${{ github.event.pull request.changed skill }} - name: Flag broad descriptions for review run: ./scripts/check-description-scope.sh skills/${{ github.event.pull request.changed skill }} publish-and-deploy: needs: evaluate-output, security-scan if: github.ref == 'refs/heads/main' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Package and tag version run: ./scripts/package-skill.sh - name: Publish to registry run: ./scripts/publish-to-registry.sh This isn’t a complete, ready-to-run pipeline — the referenced scripts are placeholders for logic specific to your organization’s registry and tooling — but the shape is the point: lint and validate run first and fast, testing and evaluation run next and gate on the earlier stages passing, security scanning runs in parallel with evaluation since it doesn’t depend on it, and publish/deploy only happens once everything else has passed, and only on the main branch. What Should Block a Merge vs. What Should Only Warn Not every check deserves the same weight. A reasonable default: Blocking fails the pipeline : frontmatter validation failures, trigger test regressions, security scan hits on hardcoded secrets, benchmark regressions past an agreed threshold say, a major latency or cost increase with no corresponding quality improvement . Warning visible, but doesn’t block : an unusually broad description flagged for human attention, a benchmark change within normal variance, a missing changelog entry on a patch-level change. Treating every warning as a hard block trains people to route around the pipeline rather than fix the underlying issue; treating a real security or correctness regression as a mere warning defeats the purpose of having the pipeline at all. Where that line sits is a judgment call specific to your organization’s risk tolerance — but it should be a deliberate one, not an accident of which checks happened to get implemented as blocking first. Summary - CI/CD turns the eight lifecycle stages from the previous lesson into something that runs the same way automatically, rather than depending on someone remembering to do it manually - The pipeline shape — lint, validate, test, evaluate, security scan, publish, registry, deploy — mirrors practices from earlier in this course and the free companion course, now automated - A real GitHub Actions workflow can implement this directly: fast checks first, expensive evaluation gated behind them, security scanning in parallel, publish/deploy only on a passing main branch - Not every check should block a merge — deciding what blocks versus warns is a deliberate risk-tolerance decision, not an afterthought Next, you’ll build the destination this pipeline’s publish stage actually pushes to — a registry where skills can be found, installed, and trusted at organizational scale.