Open source · Hosted Cloud, standalone app, or Rails engine
Every change to your AI is a guess until you test it. CompletionKit runs it against real inputs, scores the outputs, and shows you what improved and what broke before it ships.
Free + self-hostable · OpenAI · Anthropic · Ollama · 100+ via OpenRouter
01The problem
You changed the prompt. It feels better. You need to know. #
"Seems better" isn't a metric you can ship behind — but without the right tools, it's usually all you've got.
You're shipping on vibes.
Without numbers, "looks right" is the bar — and "looks right" depends entirely on which inputs you happened to test.
Prompts drift, silently.
A one-line tweak to a string literal doesn't show up in review as "prompt change." Behavior shifts in prod, and nobody can point to when or why.
Every fix risks a quiet regression.
v4 nails the case that prompted the change. The inputs you didn't think to retest? They might be worse — and you'd never know.
02How a run works
Four moves. Then you have evidence. #
A run is the unit of work — reproducible by prompt, dataset, model, metrics.
Write a prompt. Template with
{{vars}}
. Upload a CSV of real inputs. CompletionKit merges them, one prompt per row. -
Run a model. OpenAI, Anthropic, Ollama, or any of 100+ via OpenRouter. Same dataset, any number of models in parallel.
Score against your metrics. An LLM judge scores each output 1–5 on the metrics you define. Empathy, clarity, policy — whatever
goodmeans for you. - Iterate. Re-run. Edit and the prompt forks a new version. Ask CompletionKit to suggest one, grounded in the judge's feedback on your data.
Got outputs already — production logs, another model, hand-curated examples? Skip steps 1 and 2: a judge-only run grades any column in your dataset against your metrics, no generation needed.
03The unlock
Point your coding agent at CompletionKit. Walk away. #
REST, MCP, the standalone Bench — same shape. Plug Claude Code or Cursor into the MCP server and tell it to make a prompt better. It does.
You set the bar.
Define the metrics and the score a prompt has to clear. That judgment is yours; it's the one part of the loop the agent can't do for you.
The agent runs the loop.
Point Claude Code or Cursor at the MCP server. It revises the prompt, runs it on your dataset, and re-scores, pass after pass, with no babysitting.
You approve the result.
It stops at the first version that clears your bar and hands it back. Review the diff and ship it, or send it round again.
04Compared
What you actually get, line by line. #
Every tool here is doing real work. CompletionKit is the shape we kept needing.
| OpenAI Evals | Workbench | Braintrust | Langfuse | Promptfoo | CompletionKit | |
|---|---|---|---|---|---|---|
| Multi-provider | — | — | ||||
| Local models (Ollama) | — | — | — | |||
| Custom scoring metrics | Partial | Partial | ||||
| AI suggestions from your data | — | Generic | — | — | — | |
| Versioned prompts via API | — | — | — | |||
| MCP server | — | — | — | — | — | |
| Free + self-hostable | Partial | Partial | — |
05Three ways to run it
Same product. Three ways to ship it. #
Cloud is the fastest start. The standalone app is for self-hosting on your own infra. The engine is a Ruby gem you mount into an existing Rails app. Same code underneath — pick the deployment that fits.
Cloud
Sign up, paste your provider keys, run your first eval in under five minutes.
- Free tier: 20 runs / month
- Bring your own provider keys
- Team workspaces and roles
Standalone app
Same webapp, your infra. Clone the repo, point at a Postgres, run web + worker.
-
No multi-tenancy, no phone-home
-
Provider keys via env or Settings
-
Source-available · BSL 1.1 Rails engine
Add the gem, mount the engine, run the migrations — shares your app's auth and DB.
- Plays nice with your existing auth
- Active Job (Solid Queue, Sidekiq, …)
- No separate deploy
FAQ #
Which providers does it support?
OpenAI, Anthropic, Ollama (or any OpenAI-compatible local endpoint), and 100+ models via OpenRouter.
Do I have to self-host?
No. CompletionKit Cloud is the hosted version with a free tier — sign up and you're running. If you'd rather host it yourself, you've got two flavors: deploy the bundled standalone app on your own infra, or mount the Rails engine inside an existing Rails app. Same product on every path; you bring the provider keys.
How does my app use a prompt?
Each published prompt has a versioned URL. Your app calls it, gets the template and model, and runs the LLM the way it already does. No SDK required — it's just JSON over HTTP.
What if I already have outputs from another system?
Use a judge-only run. Drop the outputs into a column of your dataset, point the run at that column, define your metrics — the LLM judge scores every row against your rubric. No prompt, no generation step. Same scoring, same per-row review you'd get from a full run. Great for grading production logs, comparing a different model's outputs, or auditing hand-written examples.
Is the engine free?
Yes — free for any use, including in production and inside your own commercial product. Source-available under BSL 1.1; the only carve-out is offering CompletionKit itself to third parties as a hosted or managed service. Auto-converts to GPL-3 after three years. Versions 0.2.x and earlier remain MIT.
Who made it?
Built by Homemade Software.