{"slug": "i-built-a-homebrew-for-ai-skills-install-flow-and-eval-harness-inside", "title": "I built a Homebrew for AI skills: install flow and eval harness inside", "summary": "A developer built SkillForge, an open-source tool that installs AI coding skills like Homebrew packages, using two pip commands. The tool generates stack-specific skills named after their target technologies, such as 'backend-fastapi-postgres', rather than generic persona-based skills. SkillForge runs locally, binds to 127.0.0.1, ships no telemetry, and supports six LLM provider families.", "body_md": "Last quarter I spent an afternoon writing a SKILL.md for backend FastAPI work by hand. By the time I had a usable prompt set, three templates, and a config file, I realized nobody else on the team would ever do this. Engineering skills are either too rigid, a hand-written prompt for one specific stack, or too generic, a \"full-stack\" skill that tells you to \"use best practices.\" The GPT Store proved the failure mode in public: when quality is not measurable, prompt wrappers win and nobody comes back.\n\nSkillForge is my attempt at a fix. Install is two pip commands and a serve:\n\n```\ncd apps/api && pip install -e \".[dev]\"\ncd ../cli   && pip install -e .\nskillforge serve --build-web    # → http://localhost:8000\n```\n\nThe thing I kept hitting was the category axis. I wanted a skill for a specific stack, FastAPI plus Postgres plus Alembic plus pytest plus Docker, but every skill marketplace wanted me to pick a category first. Categories are the wrong axis. Skills should be named after the stack they target, the way Homebrew formulae are named after the binary they install. A skill called `backend-fastapi-postgres`\n\nis honest about what it does. A skill called \"Senior Backend Engineer\" is vibes.\n\nThe GPT Store angle was the other half. When you cannot measure skill quality, the people who optimize for thumbnails win. The people who optimize for output leave. The marketplace fills with wrappers and stays that way.\n\nFlip the workflow. Describe the need in plain English. The planner picks the tools and explains each pick. The generator produces a focused skill. Skills are named for their stack, not for a persona:\n\n`backend-fastapi-postgres`\n\n, `data-airflow-dbt-bigquery`\n\n, `devops-kubernetes-helm-terraform`\n\n, `ai-rag-langchain-pgvector`\n\n, `observability-opentelemetry-grafana`\n\n, `web-scraping-python-playwright`\n\n.\n\nLocal-first means local-first. SkillForge binds to `127.0.0.1`\n\n, ships no telemetry, and does not auto-execute generated scripts. The only outbound traffic is to the LLM provider you configure. Skills land on disk under `~/.skillforge/skills`\n\n. The filesystem is the source of truth.\n\nThree apps, one service layer. The CLI and the API import the same Python services, so `skillforge plan`\n\non the command line and `POST /api/chat/plan-skill`\n\nfrom the browser run the exact same code path.\n\n```\n┌──────────────────────────────────────────────────────────────────┐\n│  apps/web   (Next.js + TS + Tailwind)   :3000 / :8000 (bundled)  │\n│  ChatPanel → ManifestEditor → SkillPreview → InstallButton       │\n└────────────────────────────┬─────────────────────────────────────┘\n                             │ HTTP\n┌────────────────────────────▼─────────────────────────────────────┐\n│  apps/api   (FastAPI)                       :8000                │\n│  routers/  ──►  services/  ──►  repositories/                     │\n└────────────────────────────┬─────────────────────────────────────┘\n                             │ reuses the same service layer\n┌────────────────────────────▼─────────────────────────────────────┐\n│  apps/cli   (Typer + Rich)                                       │\n│  serve | plan | generate | install | list | validate | remove    │\n└──────────────────────────────────────────────────────────────────┘\n```\n\nSix provider families ship today, all swappable live from the Settings page with no restart: Mock (default, offline, deterministic), OpenAI-compatible (OpenAI, OpenRouter, Groq, Together, Mistral, DeepSeek, xAI, Fireworks, Z.ai), Ollama, Anthropic, Gemini, and Cohere. The Mock provider is the reason the project runs with zero configuration and the reason the test suite passes offline.\n\nA first plan:\n\n```\nskillforge plan \"I need a backend skill for FastAPI, PostgreSQL, Docker, and Pytest\"\n```\n\nThe CLI surface is intentionally small: `serve`\n\n, `plan`\n\n, `generate`\n\n, `install`\n\n, `list`\n\n, `validate`\n\n, `remove`\n\n. The generator is pure with respect to execution. It never calls `subprocess`\n\n. The `scripts/`\n\ndirectory it writes is reference text on disk, runnable when you choose to run it.\n\nA skill lands at `~/.skillforge/skills/<name>/`\n\nwith this shape:\n\n```\n~/.skillforge/skills/backend-fastapi-postgres/\n  SKILL.md\n  README.md\n  config.yaml\n  prompts/\n  templates/\n  scripts/      # real FastAPI server, Alembic, pytest, Dockerfile, MCP server\n  examples/\n```\n\nThe `scripts/`\n\ndirectory is the part that took the longest. Each artifact is a real, runnable file produced by a `ToolArtifactRegistry`\n\nat generation time, not a placeholder. The FastAPI skill ships a working `dev_server.py`\n\n. The data skill ships an Alembic `migrate.sh`\n\n. The devops skill ships a Helm `Chart.yaml`\n\n. The `config.yaml`\n\nis a manifest, not a config dump:\n\n```\nschema_version: \"1.0\"\nskill:\n  name: backend-fastapi-postgres\n  domain: Backend Engineering\ntools:\n  - name: Python\n    category: language\n    reason: Primary language for this backend skill.\nsafety:\n  auto_execute_scripts: false\n  require_user_confirmation_before_install: true\n```\n\nEvery generated skill carries the `safety`\n\nblock. `auto_execute_scripts`\n\nis always false. The installer will not overwrite an existing skill without an explicit flag.\n\nSkillForge is not a new shape. It is the same shape as four package managers you already know.\n\n| SkillForge | Homebrew | npm | VS Code | Helm |\n|---|---|---|---|---|\n`config.yaml` |\nformula (.rb) | package.json | package.json (contributes) | Chart.yaml |\n`SKILL.md` |\ninstall block | main / bin | extension.ts | templates/ |\n`scripts/` |\nresource blocks | bin/ | bundled commands | n/a |\n`~/.skillforge/skills` |\nCellar | node_modules | ~/.vscode/extensions | release namespace |\n`.skillpkg` tarball |\nbottle | pack tarball | .vsix | chart .tgz |\n| marketplace bridge | tap (git repo) | scoped pkg + token | Marketplace publisher | OCI registry |\n`skillforge validate` |\n`brew audit` |\n`npm publish --dry-run` |\nvsce package | `helm lint` |\n\nHomebrew taps federate anyone's git repo of formulae ([https://docs.brew.sh/Formula-Cookbook](https://docs.brew.sh/Formula-Cookbook)). npm scoped packages use auth tokens for verified namespaces. Helm OCI registries host signed charts. The SkillForge marketplace bridge is the same model: anyone can host a marketplace, and the local app pairs with it via a 6-character code.\n\nThe marketplace is not a website. It is an HTTP contract. Anyone can run one. The local app does not care whose marketplace it is paired with. Today the repo ships a `LocalStubAdapter`\n\nthat implements the full publish, search, download, install flow offline, so you can try the entire loop without a cloud backend.\n\nThe pairing flow borrows from VS Code plus GitHub. The local user generates a single-use 6-character code with a 10-minute TTL:\n\n```\n# 1. Local user generates a single-use 6-char code\nPOST /api/marketplace/pair/code\n→ {\"code\": \"AB3X9K\", \"ttl_minutes\": 10}\n\n# 2. User enters the code on the marketplace site\n\n# 3. Marketplace exchanges the code for a 32-byte scoped bearer token\nPOST /api/bridge/pair/complete\nBody: {\"code\": \"AB3X9K\", \"label\": \"skillforge-marketplace\"}\n→ 200: {\"token\": \"<32-byte-urlsafe>\", \"token_id\": \"...\", \"scopes\": [...]}\n```\n\nOnly the SHA-256 hash of the token is stored locally, in a chmod 600 file. Comparison is constant-time via `hmac.compare_digest`\n\n. Pairing endpoints are rate-limited to 10 attempts per minute per client, which makes the 887-million code space provably infeasible to brute force.\n\nDefault scopes are `registry:read`\n\n, `skills:install`\n\n, `skills:publish`\n\n. The dangerous one, `skills:install:unattended`\n\n, is off by default and requires explicit grant. Marketplace-originated installs land in an approval queue. The local user clicks Approve. No silent installs, ever.\n\nThe wire bundle is a gzipped tarball called `.skillpkg`\n\n:\n\n```\nskill-creator.skillpkg\n├── PACKAGING        # JSON: name, version, packaged_at, packaged_by\n├── manifest.json    # canonical SkillManifest\n├── SKILL.md\n├── config.yaml\n├── prompts/\n├── templates/\n├── scripts/\n└── examples/\n```\n\nPath traversal is rejected on unpack. The manifest is the source of truth.\n\nThe vision: anyone designs a skill, anyone hosts a marketplace, anyone installs. The reason the GPT Store filled with prompt wrappers is that there was no quality signal. SkillForge ships with one.\n\nThe eval harness is the quality signal. It lives at `/eval`\n\nin the Web UI.\n\nPick a skill. Pick a prompt suite. For each prompt, the harness runs the skill's SKILL.md as guidance against the configured provider, then asks an LLM-as-judge to score the response 0 through 10 against the skill's own `output_standards`\n\n. Results stream into an expandable table with color-coded scores and reasoning. Runs persist to SQLite so you can track scores across iterations.\n\n```\nPOST /api/eval/run\n{\n  \"skill_name\": \"backend-fastapi-postgres\",\n  \"suite\": \"general\",\n  \"provider\": \"anthropic\",\n  \"judge_provider\": \"openai-compatible\"\n}\n→ 200: {\"run_id\": \"...\", \"results\": [...]}\n```\n\nCompare mode is where the harness earns its keep. Pick two or more skills and a shared suite. You get an aggregate score, a win count, and per-prompt side-by-side cards with the winner highlighted. Manual override is supported. When five people publish a `backend-fastapi-postgres`\n\nskill, you can run them head to head on the same suite and see which one actually wins. Quality becomes measurable.\n\nThe cost guard caps completions per run at `SKILLFORGE_EVAL_MAX_CALLS=50`\n\n. Eval never executes generated scripts. It only calls the chat API.\n\nTwo honest caveats. The judge and the generator can already be configured to use different providers, which kills the worst of the self-grading bias. The default config still uses the same provider for both, and most users will not change it. The roadmap item Tier 2.1 is to make the judge default to a different provider than the generator. The second caveat: per-domain output standards are still generic. Every domain currently judges against one shared standards list. Tier 1.2 on the roadmap specializes them.\n\nThe GPT Store failed partly because there was no quality signal. SkillForge ships with one. It is rough, it has known bias, and it is better than nothing.\n\nSafety first. The local server binds to `127.0.0.1`\n\nby default. A `LocalOriginGuardMiddleware`\n\nrejects browser requests whose `Origin`\n\nor `Referer`\n\nheader names a non-loopback host, which is the CSRF and DNS-rebinding defense Jupyter and VS Code use. Token comparison is constant-time. Install is atomic: write to a staging directory, then `os.replace`\n\n. SQLite runs in WAL mode with `busy_timeout=5000`\n\nand `synchronous=NORMAL`\n\n, so concurrent eval and registry access no longer hits \"database is locked.\" Generated scripts are never executed automatically. The tool executor requires explicit `confirm=True`\n\n, an allowlist match, and a 30-second timeout.\n\nNow the limits. These are the engagement magnet, so I will be direct about them:\n\n`./scripts/build-binary.sh`\n\n.If you want to find a bug, those bullets are where to look first.\n\nThe big vision is a federated marketplace where anyone can design a skill, publish it, and have end users compare skill outputs head to head. Best skills rise. Weak skills get forked and improved.\n\nConcrete items on the roadmap:\n\nOut of scope by design: cloud sync, multi-user workspaces, auth, team permissions, remote deployment, auto-executing generated scripts. If those matter to you, the MIT license and the clean service layer make forking straightforward.\n\n```\ngit clone https://github.com/sulthonzh/skillforge\ncd skillforge\ncd apps/api && pip install -e \".[dev]\"\ncd ../cli   && pip install -e .\nskillforge serve --build-web\n```\n\nI want bug reports and edge cases, not stars. Tell me where it breaks. Specific things I would like feedback on:\n\nThe file `apps/api/skillforge_api/data/tool_catalog.yaml`\n\nis a great first PR. Add a domain. Add a tool. Add a reason. The catalog is the part that gets better with every contributor.", "url": "https://wpnews.pro/news/i-built-a-homebrew-for-ai-skills-install-flow-and-eval-harness-inside", "canonical_source": "https://dev.to/sulthonzh/i-built-a-homebrew-for-ai-skills-install-flow-and-eval-harness-inside-20a4", "published_at": "2026-06-18 08:09:00+00:00", "updated_at": "2026-06-18 08:21:12.835303+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models", "generative-ai", "ai-tools"], "entities": ["SkillForge", "FastAPI", "PostgreSQL", "Docker", "Pytest", "Homebrew", "OpenAI", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-homebrew-for-ai-skills-install-flow-and-eval-harness-inside", "markdown": "https://wpnews.pro/news/i-built-a-homebrew-for-ai-skills-install-flow-and-eval-harness-inside.md", "text": "https://wpnews.pro/news/i-built-a-homebrew-for-ai-skills-install-flow-and-eval-harness-inside.txt", "jsonld": "https://wpnews.pro/news/i-built-a-homebrew-for-ai-skills-install-flow-and-eval-harness-inside.jsonld"}}