AI Skills Are Not Just Prompts: A Practical Architecture for Building, Evaluating, Shipping, and Maintaining Agent Skills

A developer outlines a practical architecture for building, evaluating, shipping, and maintaining AI agent skills, arguing that production-grade skills are versioned, testable, routable software components with a defined lifecycle. The post emphasizes progressive context loading and distinguishes skills from other primitives like rules, agents, hooks, and scripts.

The current generation of AI coding agents makes it surprisingly easy to create a "skill." Write a Markdown file. Add instructions. Give it a name. Put it inside .claude/skills/ . Done. Except it isn't. As soon as you build more than a handful of skills, a different set of problems appears: At that point, AI skills stop looking like prompts and start looking like software systems . A production-grade AI skill is not merely a Markdown prompt. It is a versioned, testable, routable, enforceable software component with a defined lifecycle. A useful way to understand an AI skill is to look at its entire lifecycle: Runtime ↓ Scope / Fit ↓ Triggers ↓ Architecture ↓ Anatomy ↓ Content ↓ Enforcement ↓ Measurement ↓ Shipping ↓ Maintenance ↓ Portfolio Each stage answers a different question: | Stage | Core question | |---|---| | Runtime | How does the skill load and execute? | | Fit & scope | Should this be a skill at all? | | Typing & triggers | When should it activate? | | Architecture | How should the workflow operate? | | Anatomy | What files make up the skill? | | Content | How should instructions be written? | | Enforcement | What can be enforced mechanically? | | Measurement | How do we know it works? | | Shipping | How do users receive it? | | Maintenance | How does it survive change? | | Portfolio | How do many skills coexist? | The important insight is that skill engineering covers the entire lifecycle, not just the writing of SKILL.md. Before designing a skill, understand the runtime. A skill may look simple on disk: .claude/ └── skills/ └── code-review/ └── SKILL.md But conceptually the runtime does something like: User request ↓ Skill discovery ↓ Trigger evaluation ↓ Skill activation ↓ Instruction loading ↓ Reference/tool loading ↓ Agent execution ↓ Result The important question is: What does the agent actually see, and when does it see it? This matters because context is limited. A skill might contain: SKILL.md references/ database.md security.md examples.md architecture.md scripts/ validate.js check.sh templates/ report.md You usually don't want every file loaded for every request. Instead: Request ↓ SKILL.md ↓ Determine relevant task ↓ Load relevant reference ↓ Execute required script This is progressive context loading . The main skill file acts as an entry point rather than a giant knowledge dump. One of the biggest mistakes in AI skill design is treating context as free. It isn't. SKILL.md ────────────── 3000 lines 50 examples 20 rules 10 workflows 15 edge cases SKILL.md ↓ Routing ↓ references/ ├── workflow.md ├── security.md └── examples.md The second design gives you more control over what enters the model's context. Design skills around context boundaries, not just file boundaries. The next question is: Should this behavior be implemented as a skill? AI development environments often provide multiple primitives: Skill Rule Agent Hook Script Plugin They are not interchangeable. A skill describes how to perform a class of task . Example: database-migration It might define: 1. Inspect schema 2. Inspect migration history 3. Design migration 4. Implement migration 5. Validate migration 6. Test rollback A rule is a constraint: Never modify production data directly. That is policy, not a workflow. An agent is useful when you need a distinct reasoning or execution role: Main Agent ├── Research Agent ├── Security Agent └── Testing Agent A hook responds to an event: Before tool call ↓ Security check or: After file modification ↓ Formatter A script performs deterministic computation: validate-schema.js The AI may decide when to run it. The script determines how validation works. A good skill should explicitly state what it does not do. For example: This skill handles: - PostgreSQL schema migrations - Migration generation - Migration validation - Rollback testing This skill does not handle: - Application architecture - Production deployment - Database backups - Infrastructure provisioning Why? Because skills tend to grow. A developer starts with: database migration Then adds: database design query optimization backup strategy production deployment Eventually the skill becomes: database-engineering-everything and becomes difficult to reason about. Scope is a defense against skill inflation. Once you know what a skill does, you need to answer: When should it activate? Imagine: frontend-review backend-review security-review performance-review database-review The user says: "Review my authentication API." Multiple skills may be relevant. You now have a routing problem: User request │ ┌─────────────┼─────────────┐ ↓ ↓ ↓ Backend Security Performance If everything activates, the system becomes noisy. If nothing activates, the skill is useless. Think of activation as: User request ↓ Intent classification ↓ Skill candidates ↓ Relevance evaluation ↓ Activation Example: | Request | Expected skill | |---|---| | "Create a PostgreSQL migration" | Database migration | | "Fix this React component" | Frontend | | "Audit authentication" | Security | | "Why is this query slow?" | Database performance | | "Write a technical article" | Documentation | A good skill therefore needs a clear activation signature . "Write a blog about PostgreSQL" → security skill activates "Why is my PostgreSQL migration failing?" → database migration skill does not activate Both are important. Therefore, evaluate activation separately from execution . Once a skill activates, what does it actually do? There are several useful workflow shapes. A route chooses a path: Request ↓ classify / | / | Bug Feature Refactor ↓ ↓ ↓ Debug Implement Refactor A pipeline is sequential: Research ↓ Analyze ↓ Plan ↓ Implement ↓ Test ↓ Review ↓ Report This is particularly useful for engineering skills. A loop supports iteration: Implement ↓ Test ↓ Failed? / Yes No ↓ ↓ Fix Complete ↓ Test again For coding agents, this is natural: Write implementation ↓ Run tests ↓ Read failure ↓ Modify implementation ↓ Run tests again Failure becomes an expected workflow state rather than an exceptional event. A map splits a problem into independent pieces: Research repository │ ┌─────────────┼─────────────┐ ↓ ↓ ↓ Frontend Backend Database ↓ ↓ ↓ Findings Findings Findings └─────────────┼─────────────┘ ↓ Synthesis Real skills often combine these patterns: Route ↓ Pipeline ↓ Map ↓ Loop ↓ Final report This is where AI skills start looking like workflow engines rather than prompts . A mature skill might look like: skills/ └── deep-research/ │ ├── SKILL.md │ ├── references/ │ ├── research-methodology.md │ ├── source-evaluation.md │ └── evidence.md │ ├── scripts/ │ ├── validate-sources.js │ └── generate-report.js │ ├── templates/ │ └── report.md │ └── examples/ ├── example-1.md └── example-2.md Each part has a job. SKILL.md Is the Hub Think of SKILL.md as the router and operating manual. It should explain: What this skill does When it activates What it must accomplish What references to load What workflow to follow What constraints apply How to validate the result It should not necessarily contain every piece of knowledge. SKILL.md / | / | ↓ ↓ ↓ References Scripts Templates The hub provides orchestration. The spokes provide specialized resources. Separate them: SKILL.md Tell the agent what to do. references/ Provide knowledge. scripts/ Perform deterministic operations. This separation makes skills easier to maintain and test. Now we reach the instruction layer. Compare: You might want to check whether tests pass. Run the test suite before declaring the task complete. Do not declare the task complete until the test suite passes. The difference is binding strength . Prefer TypeScript. Use TypeScript for new files. Do not create JavaScript files. New implementation files must use TypeScript. Important rules should be unambiguous. Instead of: The agent should inspect the repository. write: Inspect the repository before making changes. Instead of: The agent will run the tests after implementation. write: Run the tests after implementation. Direct instructions are easier to interpret. Avoid thousands of words explaining philosophy. Prefer operational instructions: 1. Inspect X. 2. Determine Y. 3. Run Z. 4. If Z fails, investigate. 5. Do not proceed until Y is verified. The skill should be operational . This is one of the most important ideas. Don't rely on the model to enforce something that software can enforce. Suppose your skill says: Always run Prettier. The agent might comply. But it might also forget. A stronger design is: AI modifies file ↓ Hook ↓ Prettier ↓ Formatted file Formatting no longer depends entirely on model memory. Consider: Never commit secrets. A prompt can say: RULE: Never commit API keys. But a stronger architecture is: Agent ↓ git commit ↓ secret scanner ↓ Secret found? ├── Yes → Block commit └── No → Continue A useful model is: Human judgment ↓ AI instruction ↓ Automated validation ↓ Mechanical enforcement The further down the stack you go, the less you rely on model compliance. Examples of good candidates for automation: Formatting Linting Type checking Schema validation Tests Secret detection File naming Generated artifacts SQL safety Permission boundaries The AI should focus on tasks requiring judgment. Now ask: How do we know the skill works? A skill should ideally be evaluated, not merely read and trusted. There are two major dimensions. Did the correct skill activate? Example: Prompt: "Create a PostgreSQL migration for the users table." Expected: database-migration → activated You can maintain an evaluation set: ┌────────────────────────────────────┐ │ Activation Evaluation │ ├────────────────────────────────────┤ │ Prompt │ │ Expected skill │ │ Should activate? │ │ Actual skill │ │ Result │ └────────────────────────────────────┘ Once activated, did the skill behave correctly? Suppose the migration skill requires: Inspect schema Check migration history Create migration Validate SQL Test migration Test rollback Behavior evaluation checks those requirements. Skill ↓ ┌────────┴────────┐ ↓ ↓ Activation Behavior ↓ ↓ Correct skill? Correct workflow? A skill could have: 90% activation accuracy 40% behavior accuracy and still be a poor skill. Useful metrics include: correct activations / total activation tests incorrect activations / total tests missed activations / applicable tests successful executions / total executions required behaviors satisfied / required behaviors previously passing cases now failing This turns skill development into an engineering discipline. Another useful activity is comparing your skill against existing approaches. Suppose you create: deep-research Before declaring it complete, ask: What existing research workflows already exist? What evaluation techniques do they use? What source-quality rules are common? What am I missing? Which practices are unnecessarily complicated? This is prior-art auditing . You don't need to reinvent every workflow. A skill isn't useful if nobody can install it. A typical flow is: Development ↓ Repository ↓ Package / Plugin ↓ Distribution ↓ Installation ↓ Skill available A plugin can act as a distribution container: Plugin │ ├── Skills │ ├── research │ ├── testing │ └── code-review │ ├── Hooks ├── Commands └── Configuration This lets users install a cohesive collection rather than manually copying individual files. Once users depend on a skill, versioning matters: research-skill@1.0.0 research-skill@1.1.0 research-skill@2.0.0 Changing: "prefer X" to: "must use X" can change agent behavior significantly. Therefore skill versions should be treated as meaningful behavioral versions. A skill may sit unused for months. Then: Developer ↓ invokes old skill ↓ framework changed ↓ skill behaves differently This is skill dormancy . Skills need lifecycle states and maintenance expectations. AI skills exist inside rapidly changing ecosystems. Things change: AI models Agent runtimes APIs CLI tools Frameworks Repository structures Tool interfaces Best practices Security requirements Therefore: A skill is software, and software drifts. Suppose: Tool v1 ↓ Skill v1 Later: Tool v2 ↓ behavior changed Your skill still assumes the old behavior. That is drift. Suppose your skill contains a copy of external documentation: external methodology ↓ copy ↓ your skill The external source changes. Your copy doesn't. Now: Upstream ↓ Version 3 Your skill ↓ Version 1 You have a maintenance obligation. A good update flow is: Upstream changes ↓ Detect change ↓ Review ↓ Update skill ↓ Run evaluation suite ↓ Check regressions ↓ Release Not: Upstream changed ↓ blindly copy everything One skill is easy. Two are manageable. Ten are interesting. Fifty become an architecture problem. You may eventually have: skills/ ├── research/ ├── frontend/ ├── backend/ ├── database/ ├── security/ ├── testing/ ├── performance/ ├── deployment/ ├── documentation/ ├── architecture/ ├── debugging/ └── code-review/ Now you need portfolio management. Suppose the user says: "Review my API." Potential matches: api-review backend-review security-review performance-review Which one activates? This is a collision . If multiple skills activate, their instructions may conflict. For example: Skill A: "Keep the implementation minimal." Skill B: "Add extensive validation." Skill C: "Refactor the architecture." A skill portfolio therefore needs routing and priority policies. How large should a skill be? Too broad: software-engineering with everything inside it. Too narrow: python read-file write-file check-import check-variable run-test You don't want hundreds of microscopic skills. A better structure might be: Engineering │ ├── Research ├── Implementation ├── Testing ├── Security └── Deployment Each skill owns a meaningful capability. A router can first identify the broad family: User request ↓ Main Router ↓ ┌───────────────┼───────────────┐ ↓ ↓ ↓ Research Engineering Operations ↓ ↓ ↓ Research Backend Deployment skill Security Monitoring Testing Instead of asking: "Which of these 100 skills should run?" you ask: Which family? ↓ Which subcategory? ↓ Which skill? A healthy portfolio needs a retirement policy. A skill may become obsolete because: A lifecycle might be: Experimental ↓ Active ↓ Stable ↓ Deprecated ↓ Retired This prevents the skill directory from becoming a graveyard. A mature skill can be viewed as four major layers: ┌──────────────────────────────────────┐ │ SKILL │ │ │ │ ┌────────────────────────────────┐ │ │ │ Instructions │ │ │ │ What the agent should do │ │ │ └────────────────────────────────┘ │ │ ↓ │ │ ┌────────────────────────────────┐ │ │ │ Workflow │ │ │ │ Route / Pipeline / Loop / Map │ │ │ └────────────────────────────────┘ │ │ ↓ │ │ ┌────────────────────────────────┐ │ │ │ Enforcement │ │ │ │ Hooks / Scripts / Tests │ │ │ └────────────────────────────────┘ │ │ ↓ │ │ ┌────────────────────────────────┐ │ │ │ Evaluation │ │ │ │ Activation / Behavior / QA │ │ │ └────────────────────────────────┘ │ └──────────────────────────────────────┘ This is the important shift in thinking. A skill is not merely: prompt → answer It is closer to: intent ↓ routing ↓ context ↓ workflow ↓ tools ↓ validation ↓ feedback ↓ result Suppose you want: deep-research Its directory could be: deep-research/ ├── SKILL.md ├── references/ │ ├── research-methodology.md │ ├── source-quality.md │ ├── evidence-evaluation.md │ └── synthesis.md ├── scripts/ │ ├── validate-sources.js │ └── generate-report.js ├── templates/ │ └── research-report.md └── evals/ ├── activation.json └── behavior.json This skill performs structured research. It handles: - multi-source research - source evaluation - evidence synthesis - contradiction analysis - structured reporting It does not handle: - software implementation - deployment - generic writing Potential activation prompts: "Research this topic deeply." "Investigate the current state of..." "Compare these technologies using external sources." "Find evidence for and against this claim." Non-activation examples: "Fix this React bug." "Run the tests." "Format this file." Understand question ↓ Decompose question ↓ Identify evidence requirements ↓ Search ↓ Evaluate sources ↓ Extract evidence ↓ Cross-check claims ↓ Synthesize ↓ Write report ↓ Validate citations If two sources disagree: Source A → Claim X Source B → Claim Y then: Conflict detected ↓ Investigate ↓ Find additional sources ↓ Re-evaluate evidence ↓ Resolve / report uncertainty Now the skill combines: Pipeline + Loop Instead of only telling the AI: "Make sure every claim has a citation." build a validator: Report ↓ Citation validator ↓ Missing citation? ├── Yes → fail └── No → pass Example: Prompt: "Do a deep investigation into DuckDB vs PostgreSQL for analytics." Expected: deep-research → YES And: Prompt: "Fix the DuckDB connection bug." Expected: deep-research → NO For a research request: ✓ question decomposition ✓ multiple sources ✓ source quality assessment ✓ evidence extraction ✓ conflicting evidence analysis ✓ synthesis ✓ citations ✓ final report Now regressions can be detected. This model becomes especially interesting when building a larger AI development harness. Imagine: AI HARNESS │ ↓ Intent Router │ ┌───────────────┼───────────────┐ ↓ ↓ ↓ Research Engineering Operations │ │ │ ↓ ↓ ↓ Skills Skills Skills │ │ │ └───────────────┼───────────────┘ ↓ Tools ↓ Enforcement ↓ Evaluation ↓ Reporting This is much more powerful than simply having a folder full of Markdown files. Traditional software: Code ↓ Execution ↓ Result AI software: Intent ↓ Skill ↓ Reasoning ↓ Tools ↓ Result But production AI systems need another layer: Intent ↓ Skill ↓ Reasoning ↓ Tools ↓ Policy ↓ Validation ↓ Result The skill becomes a bridge between natural-language intent and deterministic engineering systems . If there is one idea to take away from all of this, it is: Use AI for judgment. Use software for certainty. Let the model handle: Interpretation Planning Hypothesis generation Trade-offs Synthesis Creative reasoning Let software handle: Formatting Validation Testing Schema checking Permissions Secret detection Deterministic calculations Policy enforcement For example: AI: "These three files probably need to change." Software: "Does the resulting code compile?" AI: "This migration should be safe." Software: "Does the migration actually execute successfully?" AI: "These sources support the conclusion." Software: "Are the required citations present?" That division produces more reliable systems. As AI agents become more capable, the bottleneck increasingly shifts away from: "Can the model write code?" toward: "Can we reliably control how the model works?" That is a different engineering problem. We need to reason about: Activation Context Permissions Workflow Tools Memory Policies Evaluation Regression Versioning Distribution These are systems problems. That is why skill engineering starts resembling: software architecture + prompt engineering + workflow orchestration + testing + policy enforcement + package management SKILL.md concise?Putting everything together: USER INTENT │ ▼ ┌────────────────┐ │ ROUTER │ └───────┬────────┘ │ ▼ ┌──────────────────┐ │ SKILL │ │ │ │ Scope │ │ Instructions │ │ Workflow │ │ References │ └────────┬─────────┘ │ ▼ ┌─────────────────────┐ │ AI REASONING │ └──────────┬──────────┘ │ ┌──────────┴──────────┐ ↓ ↓ Tools / APIs Scripts │ │ └──────────┬──────────┘ ↓ ┌─────────────────┐ │ ENFORCEMENT │ │ │ │ Hooks │ │ Policies │ │ Validators │ └────────┬────────┘ ↓ ┌─────────────────┐ │ EVALUATION │ │ │ │ Activation │ │ Behavior │ │ Regression │ └────────┬────────┘ ↓ RESULT And around the whole system: ┌───────────────────────────────────┐ │ SKILL LIFECYCLE │ │ │ │ Version → Ship → Observe → │ │ Maintain → Update → Deprecate │ │ │ └───────────────────────────────────┘ The simplest way to build an AI skill is: Write SKILL.md The professional way is: Define scope ↓ Define activation ↓ Design workflow ↓ Structure context ↓ Write instructions ↓ Add tools ↓ Mechanically enforce critical rules ↓ Evaluate activation ↓ Evaluate behavior ↓ Version ↓ Ship ↓ Monitor drift ↓ Maintain ↓ Retire when necessary That is the fundamental shift. AI skills should be treated less like prompts and more like software components. A prompt tells an AI what you would like it to do. A well-engineered skill defines: Once you start thinking this way, .claude/skills/ stops being a collection of Markdown files. It becomes an AI-native software architecture layer . The future of agent engineering is not just better prompts — it is better systems around prompts.