{"slug": "paretoops-ci-gate-on-cost-per-successful-task-for-llm-evals", "title": "ParetoOps – CI gate on cost per successful task for LLM evals", "summary": "ParetoOps launched a CI gate that checks every pull request against accuracy and cost-per-successful-task limits, reading results already produced by Promptfoo, LangSmith, Braintrust, DeepEval or any CSV/JSON file with one row per task. In the company's sample regression dataset, a prompt change that still cleared a 50% accuracy threshold actually dropped accuracy 15.0 points (95% CI -22.5 to -7.5) and raised cost per success ×1.42 (×1.24–×1.64), a verdict a threshold check alone would have passed. The tool runs offline, never calls a model, requires Node 18 or newer, and is free today, with a 30-day Pro trial that adds comparison against the last good run available via an interest list.", "body_md": "# Did that prompt or model change make your AI agent worse, or more expensive?\n\nParetoOps runs in your CI and reads the results your eval tool already produces. It checks every pull request against your limits, and with Pro against your last good run, so you get a clear verdict before it merges.\n\nWant the regression check on your pull requests? [Join the interest list for a 30-day Pro trial →](https://paretoops.dev/interest/?src=home-hero)\n\nTry it on your own results: `npx pareto-ops analyze results.json --min-acc 0.9`\n\nWorks with Promptfoo, LangSmith, Braintrust and DeepEval · Runs offline, never calls a model · Free to use today\n\nPro A regression, caught on the pull request\n\n**ParetoOps** commented on this pull request❌ FAILED\n\n**WORSE**\n\n| Metric | Baseline | This change | Change (95% CI) | Verdict | \n|---|---|---|---|---|\n| Accuracy | 81.5% | 66.5% | -15.0pp (-22.5pp to -7.5pp) | WORSE | \n| Cost / Success | $0.0155 | $0.0219 | ×1.42 (×1.24–×1.64) | WORSE | \n\nThe change still clears the 50% accuracy threshold, so a threshold check alone would have passed it.\n\n[full report](https://paretoops.dev/sample-report/)). Real output from the sample regression dataset.\n\nFree keys get a pull request comment too: your options ranked by cost per successful task, and whether your accuracy and cost limits passed. Pro adds this comparison with your last good run.\n\n## Three steps, no new infrastructure\n\n### Keep running the evals you have\n\nPromptfoo, LangSmith, Braintrust, DeepEval, or any CSV or JSON with one row per task. ParetoOps doesn't run evals or call a model.\n\n### Add one step to CI\n\n`uses: certiqa/paretoops-action@v1` in GitHub Actions, or one command in GitLab, Jenkins or a cron job. [Set it up →](https://paretoops.dev/docs/getting-started/ci-gate/)\n\n### Get a verdict on every pull request\n\nA comment says pass or fail against your limits, what you're paying for each task that passes, and with Pro, what changed since your last good run.\n\n### See it on sample data in two minutes\n\nDownload the sample eval results ParetoOps ships with and run the free analysis. No sign-up, no key, nothing leaves your machine.\n\nNeeds Node 18 or newer. Prefer Python? `pip install pareto-ops` installs the same command. [Walk through the report →](https://paretoops.dev/docs/getting-started/first-analysis/)\n\n``` bash\n$ curl -O https://paretoops.dev/eval_benchmark.json\n$ npx pareto-ops analyze eval_benchmark.json --min-acc 0.9\n\nGemini 2.5 Flash    67.0%   $0.0022 per success   ✔ OPTIMAL\nClaude Haiku 4.5    92.0%   $0.0035 per success   ✔ OPTIMAL\nClaude Sonnet 5     94.0%   $0.0233 per success   ✔ OPTIMAL\nClaude Opus 4.8     80.0%   $0.0327 per success   ✖ DOMINATED\n\n🌟 Recommended Sweet Spot: Claude Haiku 4.5 at $0.0035/successful\n   task (92.0% accuracy)\n```\n\nCondensed from the real output on the sample dataset. The sweet spot is the best pick: the cheapest option above your accuracy bar.\n\n## Code changes get tests. Prompt and model changes mostly get a glance.\n\nA pass mark like \"accuracy above 50%\" only asks whether a number clears a line. It can't tell you the change made things worse than what you already had. The same sample change, two verdicts:\n\nAccuracy is 66.5%, above the 50% bar you set. Nothing in a threshold check says the change got worse.\n\nOn the 200 tasks both runs share, accuracy fell 15.0 points (95% CI -22.5 to -7.5) and cost per success rose ×1.42 (×1.24–×1.64). The change is worse, not unlucky.\n\n### Small edits, big swings\n\nRewording a prompt moved results 11 to 58 times more than rerunning it ([Chen, Qian et al., 2026](https://paretoops.dev/research/#noise-floor)). Comparing two averages can't catch that.\n\n### The cheaper price list isn't always cheaper\n\nIn 32% of model pairs studied, the model with the lower list price cost more overall ([Chen, Zhang et al., 2026](https://paretoops.dev/research/#price-reversal)).\n\n[Join the interest list to request a 30-day Pro trial →](https://paretoops.dev/interest/?src=home-verdicts)\n\nThe verdicts are real engine output on the sample data shipped with ParetoOps. [See the full report →](https://paretoops.dev/sample-report/)\n\n## Two questions every team with an eval ends up asking\n\n### 1. Is this change safe to merge?\n\nPrompts, retrieval settings and model versions change all the time, and each change can quietly cost accuracy or money. This is the question for any team, even with a single model: ParetoOps checks every pull request against your limits (Free) and against your last good run (Pro).\n\n### 2. Which configuration should we run?\n\nWhen you pick a model, switch to a new release, or route easy requests to a cheaper one, ParetoOps ranks every option by what a successful task actually costs, and recommends the cheapest one that clears your accuracy bar.\n\n### Find the cheapest option that's good enough\n\n**Cost per successful task** is total spend divided by the tasks that passed. A cheap model that fails often can cost more per good answer than a pricier one that doesn't.\n\n- **Flags options you can drop:** beaten on both cost and accuracy.\n- **Recommends a best pick:** the cheapest option above your accuracy bar.\n- **Prices tokens correctly:** prompt caching and reasoning tokens included.\n\n### \"We only evaluate one model.\" Why would anyone run more than one?\n\n- **New releases and price changes:** \"can we switch?\" is answered by re-running the same eval.\n- **Routing:** easy requests to a small model, hard ones to a large one, split by measured results.\n- **Agents:** each step may use a different model.\n- **Prompts count too:** two prompts on one model are two options to compare.\n- **A tested fallback** if your provider has an outage or raises prices.\n\n[Compare with your own numbers →](https://paretoops.dev/calculator/) · [Can't I just always use the frontier model? →](https://paretoops.dev/learn/#always-frontier)\n\n**Offline**, never calls a model, no telemetry · **Deterministic**, arithmetic on your data · **Any CI**, one command that exits 0 or 1 ([GitHub Action](https://github.com/certiqa/paretoops-action), GitLab, Jenkins) · [Closed source, how to verify →](https://paretoops.dev/trust/)\n\nEvery statistic here traces to a cited study or the sample data shipped with the engine ([research](https://paretoops.dev/research/)). [See every command →](https://paretoops.dev/product/)\n\n## Start free. Add the regression gate with Pro.\n\n### Free\n\n- Cost per successful task for every option\n- Which options to drop, and the best pick\n- CI check on accuracy and cost limits, with pull request comments\n- All eval formats, Python SDK\n\n[Start free](https://paretoops.dev/docs/getting-started/install/)\n\n### Pro\n\n**$99** per month, billed yearly · per GitHub organization\n\n- Regression gate against your last good run, task by task, with confidence intervals\n- Baselines stored on a git branch\n- Early stopping for failing evals\n- 3D frontier with P95 latency\n- Includes everything in Free\n\n[Join the interest list](https://paretoops.dev/interest/?src=home-plans)\n\nRequest a 30-day Pro trial key of the regression gate. We'll get in touch.\n\n## Questions\n\n## What is cost per successful task (or cost of pass) for LLMs?\n\nCost per successful task (also called cost-of-pass in academic literature) is the total inference spend on a set of benchmark tasks divided by the number of tasks that actually passed. Because failed model calls still cost money, a model with a low token price that fails often frequently costs significantly more per useful output than a more expensive, reliable model.\n\n## Why does comparing price per token mislead LLM buyers (price reversal)?\n\nPrice per token only reflects nominal provider rates per million tokens. It ignores how many tokens a model consumes to solve a prompt, how many reasoning or \"thinking\" tokens it generates, and how often it fails. In published research, in 32% of model pairs studied, the model with the lower list price cost more in total—in extreme cases up to 28x more—due to verbosity and reasoning token overhead.\n\n## I only evaluate one model. Is ParetoOps still useful?\n\nYes. The main use is checking every prompt, retrieval or model-version change against your last good run, which works with a single model. The model comparison is the second use: when a new model ships, a price changes, or you want to route easy requests to a cheaper option, you run the same eval on the alternatives and see which one costs least per successful task. A configuration is more than a model, so two prompts on the same model count as two options.\n\n## How do I catch prompt and model regressions in CI/CD before merging?\n\nWith ParetoOps CI gating (`pareto-ops gate` or `pareto-ops ci`), every pull request can be evaluated against your minimum accuracy and maximum cost-per-success thresholds. The gate exits with code 0 on pass or 1 on failure, blocking regressions in GitHub Actions, GitLab CI, Jenkins, or any CI runner. A threshold only checks a fixed number, so a change can clear it and still be worse than before. Pro adds a paired, task-by-task comparison against your last good run, with confidence intervals, which catches that.\n\n## Does ParetoOps call model APIs or send my evaluation data to external servers?\n\nNo. ParetoOps is 100% offline and deterministic. It performs local arithmetic on the benchmark results your eval framework generates. It makes no API calls to LLM providers, sends zero telemetry, and verifies license keys completely offline.\n\n## What does ParetoOps Pro add over Free?\n\nPro is for engineering teams shipping LLM changes through pull requests: a paired, task-by-task regression gate against your last good run (with confidence intervals), baselines stored on a git branch, early stopping for failing eval runs, step-level cost attribution for agentic runs, and a 3D frontier with P95 latency. Pull request comments are part of Free with a free key.\n\n## Put a verdict on every prompt and model change\n\nRun the free check on your eval results in a minute. No sign-up.", "url": "https://wpnews.pro/news/paretoops-ci-gate-on-cost-per-successful-task-for-llm-evals", "canonical_source": "https://paretoops.dev/", "published_at": "2026-10-18 21:25:47+00:00", "updated_at": "2026-10-18 21:48:33.742331+00:00", "lang": "en", "topics": ["ai-tools", "mlops", "large-language-models", "ai-agents", "developer-tools"], "entities": ["ParetoOps", "Promptfoo", "LangSmith", "Braintrust", "DeepEval", "GitHub Actions", "GitLab", "Jenkins"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/paretoops-ci-gate-on-cost-per-successful-task-for-llm-evals", "markdown": "https://wpnews.pro/news/paretoops-ci-gate-on-cost-per-successful-task-for-llm-evals.md", "text": "https://wpnews.pro/news/paretoops-ci-gate-on-cost-per-successful-task-for-llm-evals.txt", "jsonld": "https://wpnews.pro/news/paretoops-ci-gate-on-cost-per-successful-task-for-llm-evals.jsonld"}}