{"slug": "openrouter-ori-eval-prove-the-best-model-for-what-you-re-building", "title": "OpenRouter Ori Eval: Prove the Best Model for What You're Building", "summary": "OpenRouter launched Ori Eval, a tool that lets developers evaluate and compare AI models on their own prompts, data, and harness to identify the best model for their specific application. The tool scans a codebase for model calls, writes eval files, runs candidate models in parallel via OpenRouter, and returns a comparison table with metrics like bug catch rate, latency, and cost. Ori Eval pins the harness and model during runs so any result change can be attributed solely to model changes, and it requires no prior eval-writing experience.", "body_md": "# Ori Eval: Prove the Best Model for What You're Building\n\nJacky Liang ·\n\nAs more and more apps add AI functionality, the choice of which model to use for what you’re building remains just as hard, if not harder, with [over 500 models](https://openrouter.ai/models) you can choose from.\n\nIn practice, that choice is often made without a systematic method: a recommendation from social media, a benchmark leaderboard, or a general sense that one model is currently the strongest.\n\nThese resources are genuinely useful, but each has the same limit. A benchmark measures a fixed task set, and a recommendation reflects someone else’s application. Neither can tell you how a model behaves in your app, on your harness, on your data, and on your prompts.\n\nMeanwhile new models ship every week, and the cost of re-evaluating them by hand is high enough that many teams delay the decision, or keep a model that no longer fits.\n\nAt OpenRouter, we know a thing or two about models.\n\nWhat we’ve learned is that there is no definitive best model - only the best model for what you’re building.\n\n[Ori Eval](https://openrouter.ai/ori/eval) helps you find that single model, and prove it to you.\n\nTo get started, tell your agent:\n\n```\nrun curl -fsSL https://openrouter.ai/skills/spawn-ori-eval and follow the instructions in its output to get started\n```\n\nOri Eval will walk you through how to choose the best model for what you’re building like a friendly and experienced engineering friend - no eval experience needed.\n\n## Tl;dr\n\n- Ori Eval runs your agent on your own prompts, asserts on the tools it called, and grades open-ended answers with an LLM judge.\n- Ori pins the harness and the model during a run. The environment stays the same, so if an eval result changed, you know that the change can only be attributed to a change in the model.\n- Ori Eval routes through OpenRouter, so model comparisons cover every model and lab.\n- You don’t need to know how to write an eval. Ori Eval finds every place your code calls a model, asks what matters to you, and writes the eval files.\n- An eval file is code. Run it in CI to block a regression, and re-run it when a new model comes out.\n- To start, tell your coding agent:\n`run curl -fsSL https://openrouter.ai/skills/spawn-ori-eval and follow the instructions in its output to get started`\n\n.\n\n## Ask a question, get an answer (with proof)\n\nTo use Ori Eval, you can simply tell your favorite coding agent:\n\nrun curl -fsSL\n\n[https://openrouter.ai/skills/spawn-ori-eval]and follow the instructions in its output\n\nYour agent hands the request to Ori Eval. Ori Eval explores your codebase, and it comes back with questions. Before it runs any evaluations, it works with you to help figure out what’s important to you — is it cost, performance, latency, speed, tool call accuracy, etc. It then chooses the 5 latest models that fit your requirement, and confirms with you.\n\nOnce all these necessary requirements have been gathered, Ori Eval writes a `review.eval.ts`\n\nfile, runs your agent against the candidate models in parallel, and returns a table:\n\n| model | catch | p50 | $/PR | result |\n|---|---|---|---|---|\n| anthropic/claude-opus-5 | 94% | 38s | $0.041 | pass |\n| openai/gpt-5.6-sol | 92% | 44s | $0.038 | pass |\n| moonshotai/kimi-k3 | 90% | 31s | $0.019 | pass |\n| z-ai/glm-5.2 | 86% | 26s | $0.008 | pass |\n| google/gemini-3-pro | 84% | 52s | $0.062 | fail (cost) |\n\nThe recommendation comes with the reason: let’s say, the top bug catch rate inside your model cost criteria, and the value pick if bug review volume grows.\n\n## Don’t worry if you’ve never written an eval before\n\nWriting good evals is not easy, that’s why we handle the annoying part for you.\n\nOri Eval scans your codebase for every place a model runs, and shows you what it found: the use case surface, the exact file, and the model you use there right now. It then asks where you want the eval to cover, and what matters most to you: accuracy, speed, cost, or something else?\n\nOnce Ori Eval is done interviewing you, it writes the eval file from your answers, and runs it.\n\nIt’s that easy.\n\n## Consistent scores, run after run\n\nBecause Ori Eval is an agent, it can pin the harness, the model, and the effort for the duration of a run. It also comes pre-tuned: we’ve already chosen the harness and the model that work best for eval work, so you don’t have to.\n\n## An eval checks three things\n\nAn eval file is a `*.eval.ts`\n\nfile that runs with `bun test`\n\n. It checks the tools the agent called, the tools it avoided, and the quality of the answer:\n\n``` js\nconst run = await agent.run(\"dinner in Lisbon?\");\nrun.tool(\"search\").toBeCalled();\nrun.tool(\"delete_file\").toNotBeCalled();\nrun.toComplete();\n```\n\nFor open-ended answers, an LLM-as-a-judge grades the output. Ori Eval helps you set the grading criteria and the minimum score, so even tricky open-ended questions can be evaluated.\n\n## Every bug becomes something you can test against\n\nTell Ori Eval about a bug in plain words: let’s say, a support agent is issuing refunds without even checking the order first — a pretty big problem.\n\nOri Eval writes an eval that asserts `lookup_order`\n\ngets called. The eval fails proving that the bug exists. You then fix the agent, and the eval passes. The assertion stays in your test suite, so you’ll always be able to catch it.\n\n## Block regressions, and re-run when the field moves\n\nAdd `ori eval`\n\nto a GitHub Actions workflow. It exits like `bun test`\n\n, so failed evals also fails the build, and a regression never reaches prod.\n\nSince what Ori Eval writes is just code, you can also easily schedule them to run every so often. One of our early beta testers now runs model comparisons monthly. When a new model drops and it performs better than the existing model in your code base, a PR gets opened, and all you have to do is merge it and experience all the benefits without you even thinking about it.\n\n## Get started\n\nTell your coding agent:\n\n```\nrun curl -fsSL https://openrouter.ai/skills/spawn-ori-eval and follow the instructions in its output to get started\n```\n\nThat’s it.\n\nThe skill installs Ori (our pre-tuned coding agent), asks you to sign in, interviews you, and runs the eval. If you use the [OpenRouter MCP server](https://openrouter.ai/docs/guides/overview/mcp-server), run `/spawn-ori-eval`\n\ninstead, and skip the URL.\n\nTo install Ori by hand:\n\n```\ncurl -fsSL https://openrouter.ai/labs/ori/install.sh | bash\n```\n\nThen run `ori login`\n\n. Running evals also needs [Bun](https://bun.sh).\n\nRead more on the [Ori Eval page](https://openrouter.ai/ori/eval) or Ori Eval docs.", "url": "https://wpnews.pro/news/openrouter-ori-eval-prove-the-best-model-for-what-you-re-building", "canonical_source": "https://openrouter.ai/blog/announcements/ori-eval/", "published_at": "2026-08-03 16:37:15+00:00", "updated_at": "2026-08-03 16:52:49.788251+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "ai-infrastructure"], "entities": ["OpenRouter", "Ori Eval", "Jacky Liang"], "alternates": {"html": "https://wpnews.pro/news/openrouter-ori-eval-prove-the-best-model-for-what-you-re-building", "markdown": "https://wpnews.pro/news/openrouter-ori-eval-prove-the-best-model-for-what-you-re-building.md", "text": "https://wpnews.pro/news/openrouter-ori-eval-prove-the-best-model-for-what-you-re-building.txt", "jsonld": "https://wpnews.pro/news/openrouter-ori-eval-prove-the-best-model-for-what-you-re-building.jsonld"}}