{"slug": "reducing-llm-costs-50-using-best-execution-for-intelligence", "title": "Reducing LLM Costs 50% Using Best-Execution for Intelligence", "summary": "Ship, a new inference endpoint from an unnamed provider, claims to reduce large language model costs by 50% while maintaining capability and behavioral equivalence with the original model. The service achieves this through inference-time optimization and offers a quality SLA alongside price and availability commitments. Ship replaces the original model string with \"ship-like/<original>\" to halve costs without requiring prompt changes.", "body_md": "`Ship`\n\nis an endpoint that provides output indistinguishable from the model you're already using, at half the price.\n\nJust replace `model=\"<original>\"`\n\nwith `model=\"ship-like/<original>\"`\n\n, and all requests will be 50% less expensive. We take the risk that any individual request costs us more to execute. This allows you to either (A) reduce costs on existing workflows or (B) increase quality by using more intelligence with the same budget.\n\nWe achieve this by doing inference-time optimization with two guarantees: capability equivalence and behavioral equivalence. Capability equivalence means that any problem solved by your original model will also be solved by `Ship`\n\n. Behavioral equivalence means that you don't need to change the prompt when you start using `Ship`\n\n; you'll get the same behavior (everything outside the capabilities, e.g. the shape of the output, average number of tool calls, instruction following, etc. remains the same).\n\nBy defining and measuring reference-relative quality, we can turn model quality into a contractual service level. Ship is the first endpoint to offer a quality SLA alongside its price and availability commitments.\n\nThis blog post lays out:\n\n`Ship`\n\nmaintains capability equivalence and behavioral equivalence`Ship`\n\n: scaling Inference-Time Optimization`Ship`\n\nWe make two claims, and measure each directly:\n\n`ref`\n\n) solves, `Ship`\n\nalso solves. Formally, `1 - P(Ship solves x | ref solves x) < ε`\n\n.`Ship`\n\n's outputs are drawn from the same distribution of behaviors as the reference model's — same output shapes, tool-use patterns, formatting, refusals, and so on. It should solve the problems equally well. Formally, `distance(P_D(x, behavior(Ship(x))), P_D(x, behavior(ref(x)))) < ε`\n\nfor any given distance metric and behavior.This doesn't mean `Ship`\n\ngives literally identical outputs. Even the same language model isn't identical to itself under that definition: call the reference model twice and you get different tokens, a different number of tool calls, different phrasing. Instead, capability and behavioral equivalence is a claim about *distributions*: conditioned on the same input, `Ship`\n\n's outputs are drawn from a sufficiently similar distribution of behaviors as the reference model's.\n\nTo measure these, we use two sets of evaluations:\n\nThe public benchmarks let anyone verify our results. The private benchmarks show that we haven't overfit, because they come from third parties who run real applications. This means `Ship`\n\ngeneralizes to distributions it was never trained on, constructed from real-world use. In this blog post, we detail the public benchmarks. As our partners run private benchmarks, we’ll report them as well.\n\nCapability equivalence asks a single question: when your reference model can solve a problem, does `Ship`\n\nstill solve it?\n\nAcross a large number of benchmarks, Ship maintains the same performance as the reference model.\n\nThis also holds true for private benchmarks (see [What People Have Built with Ship](https://thesean.webflow.io/blog/introducing-ship#section-5) ).\n\nA key part of capability equivalence is solving the same kinds of problems. Here. we can see that not only is the overall performance of `Ship`\n\ncomparable to the base model, but that `Ship`\n\nsolves the specific problems that the reference model solves.\n\nTasks grouped into bands by GPT-5.6 Sol's solve rate (deep band = base solved it); within each band every column is sorted by its own solve rate. Rows are **not** task-aligned across columns. Under each column: its overall solve rate and correlation *r* with the base model.\n\n[[toggle-end]]\n\nIn addition to solving the same problem, you want the problem to be solved without having to modify your prompts or harness. We call this behavioral equivalence. It matters for two reasons: (A) it guards against overfitting — a system can ace traditional benchmarks while behaving worse in ways the benchmark doesn't capture — and (B) it means you don't have to invest excessive engineering effort to use the new system. `Ship`\n\nwas optimized for behavioral equivalence to maintain a low switching cost for users.\n\nToday, our definition of behavioral equivalence is arguably too strong: we're probably preserving properties that don't matter in production and don't affect the end-user experience. For example, our current measurements include output length. It's not clear this matters for practical use, and in many cases it might be better to generate fewer tokens to save cost. In the future we'll refine the definition to capture just the characteristics that matter; today, we've tried to match as closely as possible every property we measured.\n\nWe also report how specific decision-relevant behavioral axes line up. Below is a scorecard.\n\nReal trace embeddings (OpenAI `text-embedding-3-small`\n\n), task-centered then t-SNE to 2D. Ship's traces land in the base model's cloud; GPT-5.5 stays apart.\n\n[[toggle-end]]\n\nHow do we achieve capability and behavioral equivalence? The naïve approaches (always using one model or adding a simple router) don't work. Different models behave differently when given the same prompt and have different capabilities. Those approaches also miss optimizations required to make cache-aware decisions, deal with agentic tasks in complex environments, and miss other optimizations which can drive capability and behavioral equivalence. Instead, `Ship`\n\noptimizes the execution of each request at inference time.\n\nInference optimization is analogous to just-in-time compilation in compilers. Traditional compilers (like traditional training) optimize for the general case, because they have not observed the specific request at run-time. The additional information from inference-time allows for more effective optimization.\n\nLLMs are typically frozen after training. A frozen model cannot adapt every part of its execution to the specific request in front of it: the compute allocation and available behavior are largely fixed by training, decoding settings, and the prompt.\n\nAn analogy to program compilation is useful. A traditional ahead-of-time compiler must produce a general binary before it sees real runtime inputs. A just-in-time (JIT) compiler can improve execution after observing those inputs, specializing hot paths and spending optimization effort where it pays. The important idea is not compilation itself; it is that runtime information enables optimizations that were unavailable ahead of time.\n\nModel training is analogous to ahead-of-time compilation: it produces a frozen model before that model has seen your request. `Ship`\n\nis analogous to just-in-time compilation. At inference time, `Ship`\n\ndetermines how much compute to spend and which method should handle the request. A method can be a model, a model with a harness and tools, a cascade, an ensemble, a plain program, a model modified in weight-space or activation-space, or a combination of the above. Like a JIT system, `Ship`\n\nbenefits from runtime information unavailable during training: the request itself, together with verified outcomes from similar real traffic.\n\nGiven a set of models and ways of intervening on those models, there is a combinatorial number of methods for fulfilling a request. This makes it a search problem, analogous to Chess or Go; our aim is to build the equivalent of AlphaGo or AlphaZero. This is **best execution:** fulfilling requests in the best possible way given a set of models.\n\nThat's why `Ship`\n\ncan cut cost by ~50% while maintaining capability and behavioral equivalence: it optimizes how each request is executed. We’re building towards a world where everyone gets the best possible intelligence per dollar.\n\nCutting inference cost in half isn't just a line item. It changes what's economically possible to build. Because `Ship`\n\nis a drop-in replacement, teams saw the savings without touching their prompts or their evals. Here's what they did with them.\n\n`Ship`\n\nLowers LLM BillsAri Messer, Head of Partnerships @ Kilo Code\n\nWe’ve run Ship as a stealth model with trillions of tokens processed from Kilo Code users all over the world. Across that traffic, it worked like a charm at half the cost. By making top-tier intelligence dramatically more affordable, Ship quickly became an indispensable daily driver for developers and knowledge workers, powering everything from code generation to complex agentic workflows. Excited to see it move out of stealth!\n\n`Ship`\n\nMakes New Use Cases EconomicalNimrod Kor, CTO @ Baz:\n\nHaving seen millions of PRs, we’ve built a benchmark of the 168 hardest bugs our systems have encountered. This is the benchmark we use internally to improve our harness and to choose what models to put in production.\n\nSwapping in Ship for our reference model was very easy, just changing a base_url and model string. We saw no statistical difference in performance on our benchmark. When we put it in production, user satisfaction and all of the end-user metrics we measure stayed steady.\n\nThe cost savings have let us ship new features that otherwise wouldn’t have been economical. We were already spending millions of dollars on our existing feature set, and our usage has more than doubled. Ship lets us support new features like plan review, security review, and merge review.\n\nModel APIs traditionally guarantee infrastructure: uptime, rate limits, and latency. They do not guarantee that the intelligence delivered remains at a particular level.\n\nShip makes that possible by treating the reference model as a quality grade. When you request `ship-like/<original>`\n\n, you are not asking us to run a particular implementation. You are asking us to deliver the reference model’s capabilities and behavior within defined statistical tolerances.\n\nThat grade has two requirements: Ship must retain the reference model’s capabilities, and its behavior must remain within some bounds (e.g. the reference model’s own run-to-run variation).\n\nOnce quality has a grade, best execution becomes well-defined: find the least expensive execution that satisfies it (or the highest quality). Ship can change how a request is fulfilled, but it cannot silently lower the grade. For our public product, this is an SLO (what we measure and build around).\n\nOur quality SLA turns that measurement into a commitment. We take the execution risk, the cost risk, and the risk that optimization degrades quality. Enterprise customers with an SLA receive a specified grade of intelligence at a specified price.\n\n`Ship`\n\nis our first product for best-execution: an endpoint that delivers the highest intelligence per dollar. By scaling inference-time optimization, it can shift the Pareto curve for intelligence: 50% lower cost regardless of the model you were using before.\n\nWe're excited to release more endpoints like this, and to share more about the underlying research.\n\n**Get started:**\n\n`model=\"<original>\"`\n\nwith `model=\"ship-like/<original>\"`\n\n. Want `Ship`\n\nfor a reference model we don't yet support, or have a workload to test? [Get in touch](mailto:contact@thesean.ai?subject=Contact%20from%20Thesian%20Website).", "url": "https://wpnews.pro/news/reducing-llm-costs-50-using-best-execution-for-intelligence", "canonical_source": "https://www.thesean.ai/blog/introducing-ship", "published_at": "2026-07-21 21:50:13+00:00", "updated_at": "2026-07-21 22:22:33.947284+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-products"], "entities": ["Ship"], "alternates": {"html": "https://wpnews.pro/news/reducing-llm-costs-50-using-best-execution-for-intelligence", "markdown": "https://wpnews.pro/news/reducing-llm-costs-50-using-best-execution-for-intelligence.md", "text": "https://wpnews.pro/news/reducing-llm-costs-50-using-best-execution-for-intelligence.txt", "jsonld": "https://wpnews.pro/news/reducing-llm-costs-50-using-best-execution-for-intelligence.jsonld"}}