# I built a Homebrew for AI skills: install flow and eval harness inside > Source: > Published: 2026-06-18 08:09:00+00:00 Last quarter I spent an afternoon writing a SKILL.md for backend FastAPI work by hand. By the time I had a usable prompt set, three templates, and a config file, I realized nobody else on the team would ever do this. Engineering skills are either too rigid, a hand-written prompt for one specific stack, or too generic, a "full-stack" skill that tells you to "use best practices." The GPT Store proved the failure mode in public: when quality is not measurable, prompt wrappers win and nobody comes back. SkillForge is my attempt at a fix. Install is two pip commands and a serve: ``` cd apps/api && pip install -e ".[dev]" cd ../cli && pip install -e . skillforge serve --build-web # → http://localhost:8000 ``` The thing I kept hitting was the category axis. I wanted a skill for a specific stack, FastAPI plus Postgres plus Alembic plus pytest plus Docker, but every skill marketplace wanted me to pick a category first. Categories are the wrong axis. Skills should be named after the stack they target, the way Homebrew formulae are named after the binary they install. A skill called `backend-fastapi-postgres` is honest about what it does. A skill called "Senior Backend Engineer" is vibes. The GPT Store angle was the other half. When you cannot measure skill quality, the people who optimize for thumbnails win. The people who optimize for output leave. The marketplace fills with wrappers and stays that way. Flip the workflow. Describe the need in plain English. The planner picks the tools and explains each pick. The generator produces a focused skill. Skills are named for their stack, not for a persona: `backend-fastapi-postgres` , `data-airflow-dbt-bigquery` , `devops-kubernetes-helm-terraform` , `ai-rag-langchain-pgvector` , `observability-opentelemetry-grafana` , `web-scraping-python-playwright` . Local-first means local-first. SkillForge binds to `127.0.0.1` , ships no telemetry, and does not auto-execute generated scripts. The only outbound traffic is to the LLM provider you configure. Skills land on disk under `~/.skillforge/skills` . The filesystem is the source of truth. Three apps, one service layer. The CLI and the API import the same Python services, so `skillforge plan` on the command line and `POST /api/chat/plan-skill` from the browser run the exact same code path. ``` ┌──────────────────────────────────────────────────────────────────┐ │ apps/web (Next.js + TS + Tailwind) :3000 / :8000 (bundled) │ │ ChatPanel → ManifestEditor → SkillPreview → InstallButton │ └────────────────────────────┬─────────────────────────────────────┘ │ HTTP ┌────────────────────────────▼─────────────────────────────────────┐ │ apps/api (FastAPI) :8000 │ │ routers/ ──► services/ ──► repositories/ │ └────────────────────────────┬─────────────────────────────────────┘ │ reuses the same service layer ┌────────────────────────────▼─────────────────────────────────────┐ │ apps/cli (Typer + Rich) │ │ serve | plan | generate | install | list | validate | remove │ └──────────────────────────────────────────────────────────────────┘ ``` Six provider families ship today, all swappable live from the Settings page with no restart: Mock (default, offline, deterministic), OpenAI-compatible (OpenAI, OpenRouter, Groq, Together, Mistral, DeepSeek, xAI, Fireworks, Z.ai), Ollama, Anthropic, Gemini, and Cohere. The Mock provider is the reason the project runs with zero configuration and the reason the test suite passes offline. A first plan: ``` skillforge plan "I need a backend skill for FastAPI, PostgreSQL, Docker, and Pytest" ``` The CLI surface is intentionally small: `serve` , `plan` , `generate` , `install` , `list` , `validate` , `remove` . The generator is pure with respect to execution. It never calls `subprocess` . The `scripts/` directory it writes is reference text on disk, runnable when you choose to run it. A skill lands at `~/.skillforge/skills//` with this shape: ``` ~/.skillforge/skills/backend-fastapi-postgres/ SKILL.md README.md config.yaml prompts/ templates/ scripts/ # real FastAPI server, Alembic, pytest, Dockerfile, MCP server examples/ ``` The `scripts/` directory is the part that took the longest. Each artifact is a real, runnable file produced by a `ToolArtifactRegistry` at generation time, not a placeholder. The FastAPI skill ships a working `dev_server.py` . The data skill ships an Alembic `migrate.sh` . The devops skill ships a Helm `Chart.yaml` . The `config.yaml` is a manifest, not a config dump: ``` schema_version: "1.0" skill: name: backend-fastapi-postgres domain: Backend Engineering tools: - name: Python category: language reason: Primary language for this backend skill. safety: auto_execute_scripts: false require_user_confirmation_before_install: true ``` Every generated skill carries the `safety` block. `auto_execute_scripts` is always false. The installer will not overwrite an existing skill without an explicit flag. SkillForge is not a new shape. It is the same shape as four package managers you already know. | SkillForge | Homebrew | npm | VS Code | Helm | |---|---|---|---|---| `config.yaml` | formula (.rb) | package.json | package.json (contributes) | Chart.yaml | `SKILL.md` | install block | main / bin | extension.ts | templates/ | `scripts/` | resource blocks | bin/ | bundled commands | n/a | `~/.skillforge/skills` | Cellar | node_modules | ~/.vscode/extensions | release namespace | `.skillpkg` tarball | bottle | pack tarball | .vsix | chart .tgz | | marketplace bridge | tap (git repo) | scoped pkg + token | Marketplace publisher | OCI registry | `skillforge validate` | `brew audit` | `npm publish --dry-run` | vsce package | `helm lint` | Homebrew taps federate anyone's git repo of formulae ([https://docs.brew.sh/Formula-Cookbook](https://docs.brew.sh/Formula-Cookbook)). npm scoped packages use auth tokens for verified namespaces. Helm OCI registries host signed charts. The SkillForge marketplace bridge is the same model: anyone can host a marketplace, and the local app pairs with it via a 6-character code. The marketplace is not a website. It is an HTTP contract. Anyone can run one. The local app does not care whose marketplace it is paired with. Today the repo ships a `LocalStubAdapter` that implements the full publish, search, download, install flow offline, so you can try the entire loop without a cloud backend. The pairing flow borrows from VS Code plus GitHub. The local user generates a single-use 6-character code with a 10-minute TTL: ``` # 1. Local user generates a single-use 6-char code POST /api/marketplace/pair/code → {"code": "AB3X9K", "ttl_minutes": 10} # 2. User enters the code on the marketplace site # 3. Marketplace exchanges the code for a 32-byte scoped bearer token POST /api/bridge/pair/complete Body: {"code": "AB3X9K", "label": "skillforge-marketplace"} → 200: {"token": "<32-byte-urlsafe>", "token_id": "...", "scopes": [...]} ``` Only the SHA-256 hash of the token is stored locally, in a chmod 600 file. Comparison is constant-time via `hmac.compare_digest` . Pairing endpoints are rate-limited to 10 attempts per minute per client, which makes the 887-million code space provably infeasible to brute force. Default scopes are `registry:read` , `skills:install` , `skills:publish` . The dangerous one, `skills:install:unattended` , is off by default and requires explicit grant. Marketplace-originated installs land in an approval queue. The local user clicks Approve. No silent installs, ever. The wire bundle is a gzipped tarball called `.skillpkg` : ``` skill-creator.skillpkg ├── PACKAGING # JSON: name, version, packaged_at, packaged_by ├── manifest.json # canonical SkillManifest ├── SKILL.md ├── config.yaml ├── prompts/ ├── templates/ ├── scripts/ └── examples/ ``` Path traversal is rejected on unpack. The manifest is the source of truth. The vision: anyone designs a skill, anyone hosts a marketplace, anyone installs. The reason the GPT Store filled with prompt wrappers is that there was no quality signal. SkillForge ships with one. The eval harness is the quality signal. It lives at `/eval` in the Web UI. Pick a skill. Pick a prompt suite. For each prompt, the harness runs the skill's SKILL.md as guidance against the configured provider, then asks an LLM-as-judge to score the response 0 through 10 against the skill's own `output_standards` . Results stream into an expandable table with color-coded scores and reasoning. Runs persist to SQLite so you can track scores across iterations. ``` POST /api/eval/run { "skill_name": "backend-fastapi-postgres", "suite": "general", "provider": "anthropic", "judge_provider": "openai-compatible" } → 200: {"run_id": "...", "results": [...]} ``` Compare mode is where the harness earns its keep. Pick two or more skills and a shared suite. You get an aggregate score, a win count, and per-prompt side-by-side cards with the winner highlighted. Manual override is supported. When five people publish a `backend-fastapi-postgres` skill, you can run them head to head on the same suite and see which one actually wins. Quality becomes measurable. The cost guard caps completions per run at `SKILLFORGE_EVAL_MAX_CALLS=50` . Eval never executes generated scripts. It only calls the chat API. Two honest caveats. The judge and the generator can already be configured to use different providers, which kills the worst of the self-grading bias. The default config still uses the same provider for both, and most users will not change it. The roadmap item Tier 2.1 is to make the judge default to a different provider than the generator. The second caveat: per-domain output standards are still generic. Every domain currently judges against one shared standards list. Tier 1.2 on the roadmap specializes them. The GPT Store failed partly because there was no quality signal. SkillForge ships with one. It is rough, it has known bias, and it is better than nothing. Safety first. The local server binds to `127.0.0.1` by default. A `LocalOriginGuardMiddleware` rejects browser requests whose `Origin` or `Referer` header names a non-loopback host, which is the CSRF and DNS-rebinding defense Jupyter and VS Code use. Token comparison is constant-time. Install is atomic: write to a staging directory, then `os.replace` . SQLite runs in WAL mode with `busy_timeout=5000` and `synchronous=NORMAL` , so concurrent eval and registry access no longer hits "database is locked." Generated scripts are never executed automatically. The tool executor requires explicit `confirm=True` , an allowlist match, and a 30-second timeout. Now the limits. These are the engagement magnet, so I will be direct about them: `./scripts/build-binary.sh` .If you want to find a bug, those bullets are where to look first. The big vision is a federated marketplace where anyone can design a skill, publish it, and have end users compare skill outputs head to head. Best skills rise. Weak skills get forked and improved. Concrete items on the roadmap: Out of scope by design: cloud sync, multi-user workspaces, auth, team permissions, remote deployment, auto-executing generated scripts. If those matter to you, the MIT license and the clean service layer make forking straightforward. ``` git clone https://github.com/sulthonzh/skillforge cd skillforge cd apps/api && pip install -e ".[dev]" cd ../cli && pip install -e . skillforge serve --build-web ``` I want bug reports and edge cases, not stars. Tell me where it breaks. Specific things I would like feedback on: The file `apps/api/skillforge_api/data/tool_catalog.yaml` is a great first PR. Add a domain. Add a tool. Add a reason. The catalog is the part that gets better with every contributor.