Inference engineering for software engineers Inference providers Modal, Fireworks AI and Baseten are positioning themselves as the route for companies moving off frontier-lab APIs onto open-weight models, according to a Pragmatic Engineer survey of the three vendors. The piece cites Uber cutting per-token costs by 50%, Pinterest by 90% and AT&T by 56%, and contrasts multi-tenant API swapping with dedicated GPU deployments — including Revolut running 56 separate B200 units itself — while flagging tail latency, cache hit rates and rate limits as the main engineering hurdles. It lays out a two-stage path: replace the frontier API with a multi-tenant inference endpoint, then move to dedicated GPUs with KV cache sizing, failover and speculative decoding once rate limits and performance consistency become problems. A recent trend we’ve covered is of tech companies moving to open models https://newsletter.pragmaticengineer.com/p/the-pulse-tech-companies-move-to to rein in AI spend: Uber cut its per-token costs by 50%, Pinterest by 90%, and even AT&T has achieved a 56% decrease. But how do you move from a frontier lab’s APIs to using open models? There are two options: - “Bare metal:” With plenty of time and money, you can go “bare metal” and purchase a few dozen NVIDIA GPUs and use those AI supercomputers, like Revolut has done https://ai.engineer/nyc/2026/talks/revolut : running 56 separate B200 units itself to serve its inference needs. This is a lot of work to set it up, but after that, operational costs are as cheap as it gets. - Inference provider: most companies contract with an inference provider that operates large fleets of GPUs, and makes it easy for customers to onboard and start using open models fast. Onboarding to an inference provider to run LLM workloads comes with challenges, such as dealing with tail latencies, worrying about cache hit rates, rate limits, and more. Inference engineering is a set of engineering practices that we believe is increasingly beneficial for software engineers to familiarize themselves with. To go deeper into this area, we’ve talked with three leading inference providers to learn more: Modal https://modal.com/ , Fireworks AI https://fireworks.ai/ , and Baseten https://www.baseten.co/ . Today, we cover: 1. What are inference providers & how do they differ from AI labs? They have GPU infrastructure similar to frontier labs, don’t train their own frontier models and are experts at hosting open models. 2. When to switch from frontier labs? The most common reasons are to reduce costs, have more control over latency, and to fine-tune AI models. 3. Choosing a model, provider, and deployment strategy. Use evals to find the right open model for your use case, evaluate vendors, and decide between multi-tenant API usage and dedicated GPUs. 4. Stage 1: API replacement using a multi-tenant inference setup . The easiest way to onboard to an inference provider is to “swap” the API of a frontier lab to an API of an inference provider. But there may be latency problems and caching issues at high loads. 5. Signs you’ve outgrown multi-tenant inference APIs. Hitting rate limits, performance consistency problems, and overpaying for usage are signs that it might be time to adopt dedicated GPUs. 6. Stage 2: Configuring dedicated GPU nodes. With dedicated GPUs, you’ll need to get into the details: size of the model, interactivity versus throughput, KV cache sizing, failover options, and more. 7. Speculative decoding. If there’s unused compute during the decode phase, you might consider a Speculator to further boost inference performance. As usual, we have no affiliation with vendors mentioned in this article, and have not been paid to mention any company. Special thanks to Richard Gong Inference lead, Modal , Dmytro Dzhulgakov cofounder and CTO, Fireworks , and Philip Kiely Special Projects at Baseten for sharing details about inference engineering challenges. 1. What are inference providers & how do they differ from AI labs? Inference providers are companies that sell access to LLMs which they run on users’ behalf. From the outside, this isn’t so different to what OpenAI and Anthropic offer: an API endpoint, ability to stream tokens, and pay based on token usage. But what makes inference providers different from frontier labs is the models running behind the endpoint: inference providers run open-weight models, whereas frontier labs train their own models. In this field, Modal, Fireworks, and Baseten each operate slightly different strategies: - Modal https://modal.com/ sees itself as a cloud infrastructure provider that focuses on making its offering programmable via its SDKs. It natively supports inference services through its SDKs, but leans heavily on the open source ecosystem for running its inference workloads efficiently. Performance tweaks to its stack are merged upstream. In the words of Richard Gong, Inference Lead at Modal: “when you run code on Modal, you should be able to see all the code that’s running.” - Fireworks https://fireworks.ai/ positions itself as a provider for specialized AI workloads. Most of its business is in dedicated deployments, which often include fine-tuning and provision of latency guarantees. Much fine-tuning and post-training is available for self-service. Fireworks also offers forward-deployed researchers that can help with post-training custom models or components. Dmytro Dzhulgakov, Fireworks’ CTO, says: “our tagline is specialized intelligence, which means we will tune everything from model to infrastructure for your particular use case.” - Baseten https://www.baseten.co/ describes itself as an all-in-one place to build agents. It starts with the basic infrastructure layer with uptime guarantees, flexible commercial terms, without vendor lock-in. Baseten builds additional layers for building agentic workloads on top of its inference platform. These additional layers include sandboxes, governance services, and self-serving, post-training functionality. To customers, inference providers offer LLM APIs that can be used as drop-in replacements for frontier labs’ APIs. However, inference providers are not the same, as suggested above: 1. Inference providers don’t pre-train their own foundational models . A single pre-training run for frontier-level LLMs can cost millions of dollars, and only hyperscalers and specialized frontier labs can afford it. Inference providers run post-training on open-weight models because it’s significantly cheaper. 2. Competitive pricing. The main business of inference providers is offering open-weight models at competitive rates. 3. Dedicated hardware. A big part of the appeal of inference providers is the ability to have dedicated hardware for users to interact with vendors on the infrastructure layer. 4. Efficiency & customization. Inference providers offer similar commoditized open models, and compete heavily on efficiency and customization services. As an example of efficiency, Fireworks announced https://fireworks.ai/blog/ember-1 its Ember-1 model in September 2026, that is post-trained on the Kimi K3 foundation, delivering a similar quality level but with 40% fewer reasoning tokens consumed. Consuming far fewer reasoning tokens gives Fireworks an edge on cost compared to a vanilla Kimi K3 model. In general, frontier labs have poor uptime, and this helps inference vendors win market share. API Uptime provided by frontier labs isn’t at the levels expected of modern cloud providers: currently, the self-declared uptime of Anthropic https://status.claude.com/ is 99.5%, and for OpenAI https://status.openai.com/ it’s 99.97% for API, and 99.2% for ChatGPT. These numbers are well below the four nines 99.99%+ uptime expected of cloud providers – which is something that a vendor like AWS is known to uphold https://health.aws.amazon.com/health/status . Imagine you’re a developer aiming to provide a 99.99% uptime guarantee for your application; you can’t rely on a single frontier LLM provider for the service’s critical path. But in the graphs above, degraded performance slots aren’t counted as downtime. As all these APIs are multi-tenant, it means that performance degradations will naturally hit some tenants – with no way to be aware of the degradation, or do anything about it. Reliability is an ongoing concern with the LLMs of frontier providers and inference providers. Even though many inference vendors offer higher reliability guarantees than frontier models, inference providers can also go down, too. For companies that use inference providers, it’s common to use an LLM gateway with fallback logic to another provider – or to a frontier API. We explored this in a deepdive with Ramp https://newsletter.pragmaticengineer.com/p/why-ramp-built-inspect , whose LLM router project started with this exact problem statement. 2. When to switch from frontier labs? Moving from frontier AI labs to an independent inference provider is often driven by three factors: cost, control, and the need for more customization. Most common reason: cost Many companies start considering inference providers because they have pressing cost concerns. In many cases, a frontier model is more capable than necessary for most requests, meaning a “dumber” model is just as good for simpler tasks – and nobody would notice the difference In such cases, it’s a “no-brainer” for an engineering team to choose to route some requests to a cheaper model hosted by an inference provider. “Smart routing” is an obvious way to cut costs. An LLM router – the “smart router” – is prompt-aware and can route an LLM request to the most cost-efficient provider that can solve the request. This is an approach that fuelled Uber’s ~50% cut in per-token costs this summer https://blog.pragmaticengineer.com/the-pulse-tech-companies-move-to-open-ai-models/ . Providers offer OpenAI/Anthropic-compatible API endpoints, so switching providers is trivial. All that is needed is to point your app at the new endpoint operated by a “smart router.” But how to decide which model to serve? Generic benchmarks are often insufficient because they usually do not fully represent a team’s traffic patterns. Most inference providers ask for prompt samples or eval https://newsletter.pragmaticengineer.com/p/evals sets in order to gauge which model best serves a customer’s use case. The experience of using a new LLM endpoint resembles using APIs provided by frontier labs, while API calls are still charged per token, and response time SLAs generally don’t exist. More control over latency When you use a frontier lab’s endpoint, there is little control over operational issues like response latency and model availability. Opting for a dedicated setup with an inference provider offers full control over both: - Latency : model response times are more predictable because the app has dedicated capacity that can scale by usage. - Model availability : Engineers can run the model of their choosing on their dedicated GPUs and keep operating it as long as needed, avoiding the risk of a frontier lab changing models from one day to the next. Fine-tuning model quality: customization When an inference stack runs on dedicated hardware with open source models, it’s possible to work with inference providers to finetune a model’s quality as required, which optimizes latency or throughput performance. Richard Gong of Modal says that around 10% of their customers go this far, suggesting that customization is for “power users”. Fine-tuning a model’s quality is typical in two scenarios: - Tweaking a smaller model : achieving a similar level of quality with a smaller model, with fewer parameters. This performance gain is achieved via performance tweaks not possible with off-the-shelf models. Inference providers allow you to post-train a smaller model for your specific use case. A post-trained small model can then outperform the original model in those specialized scenarios. - Fine tuning for niche use cases. Improving quality when general purpose models aren’t good enough, often because the data isn’t publicly available. For example, projects in the legal and medical sectors tend to benefit from this, according to the Fireworks team. 3. Choosing a model, provider, and deployment strategy Most inference providers offer the exact same models to run, and the differences lie in model performance and quality. For example, model quality can differ by default context length, and some providers cap this to below what the model supports. There’s also quantization https://developer.nvidia.com/blog/model-quantization-concepts-methods-and-why-it-matters/ to consider, where a model’s weights are compressed to a lower numerical precision, or default temperature https://aiwiki.ai/wiki/temperature aka controlling how random the output is. Here’s a practical approach for selecting a provider: 1. Model : Choose an AI model to switch to 2. Provider : pick an inference provider to host the model, based on model quality, performance, and cost 3. Deployment model: decide whether to use a multi-tenant service, or to move to dedicated GPU nodes Choosing a model The most obvious way to decide which model to use is to run an eval set against the various model options. In the latest episode of The Pragmatic Engineer Podcast, Microsoft’s CEO Satya Nadella argues https://newsletter.pragmaticengineer.com/p/software-engineering-after-code-becomes that private evals shall be a big component of tech companies’ intellectual property IP in the future. He says: “Public benchmarks are great, but your private evals will matter more: you need your own eval for the outcome that matters to you. Then you look at performance across all these models and continuously optimize. Our customers start with a frontier model, then say: ‘I’ve got all those traces. This is a deterministic workflow, so I don’t need the next big model drop. I want to optimize the cost.’ They drop in another model, use the traces to fine-tune it – and there you have it.” An example of public benchmarks is from Artificial Analysis https://artificialanalysis.ai/ that maintains an eval set to compare costs for generic tasks across different models. Here are the average costs per task, broken down by model: Let’s say you build an app on top of Anthropic’s Sonnet 5.5 model, costing $5.46 on average per task in the index. If Kimi K3 is sufficient for this use case, then that model is ~60% cheaper to run at only $2.00 per task. Note that the cost comparison here is per-task, not per-token. This makes sense because different models require different numbers of tokens to complete a task, which makes it harder to compare model costs without your own private evals Having said that, deciding whether a model is good enough requires more than just looking at public benchmark results. Ideally, there’s an eval set https://newsletter.pragmaticengineer.com/p/evals to compare models with. In the absence of that, running an A/B experiment is a good alternative. For example, if you want to switch agentic coding workflows to an inference provider, you could simply let some engineers use the new model and give feedback on the results. Choosing a provider All providers tweak open models to squeeze out more performance without degrading quality too much. Therefore, model outputs and quality vary among providers. A useful tool for comparing providers’ model accuracy is Artificial Analysis’ endpoint accuracy index. This is a score that benchmarks results from various inference providers against its own, self-hosted version. An index value of 100% means the provider’s results are the same as Artificial Analysis’ own hosted model. Endpoint accuracy is broadly similar and the scores of providers range from 93% to 102%. The variance comes from performance and quality tweaks that inference providers make to their versions of open models. Since LLMs are non-deterministic, this measurement has a 95% confidence interval https://en.wikipedia.org/wiki/Confidence interval – which adds another layer of complexity to evaluation. Minor score differences are considered within the same level as Artificial Analysis’ reference model, hence the green coloring of bars that are technically below 100%. Deployment model: multi-tenant or dedicated? All providers’ offerings include a multi-tenant API, as well as dedicated GPUs to run LLMs. This deployment progression is typically a graduation path: most deployments start as drop-in API replacements of frontier labs’ APIs, and as usage grows they start switching to dedicated infrastructure. When switching to an open model hosted by an inference provider, the per-token cost decreases. However, you are still running in a multi-tenant environment, which comes with similar performance bottlenecks, like when using the API endpoint of a frontier model. The difference is that with an inference provider, you can migrate to a dedicated hosting environment and have full control over a deployment’s performance. Having your own deployment controlling the GPUs with the inference provider makes it possible to fine-tune a model’s quality with techniques such as post-training. Typical graduation path from multi-tenant API to dedicated GPUs The most typical path for a startup using LLMs: 1. Use a frontier lab’s API endpoints 2. Move over to using an open model, using the API endpoint of an inference provider 3. Rent dedicated GPUs from the inference provider Moving to an inference provider is usually coupled with achieving some kind of product-market fit. All three inference providers we spoke with told us that by the time a customer signs up by getting to “stage 1” see below , their product has gained traction in the market. Per-token costs only become important with non-trivial levels of user activity. “Stage 2” with dedicated GPUs is a level that implies significant budget commitments, a more mature product, and large-scale usage. Getting inference capacity: this can be tricky, given the global GPU shortage We previously covered how there’s a CPU shortage inside cloud providers https://newsletter.pragmaticengineer.com/p/the-pulse-a-new-trend-of-cpu-shortages – and so there should be no surprise that there’s an even more pressing GPU shortage. Even large inference providers are having a hard time getting all the GPUs they need. The inference providers we talked with all run on top of several cloud providers – plus some also manage their own bare metal – exactly to get as much capacity as they can. Getting access to more GPUs justifies the added operational complexity from running in multiple clouds. Being able to quickly onboard new cloud providers that have spare capacity is becoming a key differentiator in the industry. If you need dedicated GPUs, know that it will probably be expensive. Scarce capacity is one of the reasons why getting a dedicated setup typically requires long-term budget commitments upfront. To put it simply: you’ll need to pay a larger amount, and commit for paying for those GPUs on a longer timeframe. 4. Stage 1: API replacement, using a multi-tenant inference setup For teams with hefty LLM bills, the easiest way to decrease cost is to replace frontier labs’ models in Claude Code, Codex, OpenCode, with a frontier-level open model. Looking at data from OpenCode https://opencode.ai/data/ and OpenRouter https://openrouter.ai/rankings top-models , the current most popular choices of engineering teams for this include DeepSeek 4.1 Flash, GLM 5.3, and others including Kimi K3.1 Pro. The easiest way to carry out a replacement is to point an application to an API provided by the inference provider. In practice, this means using a multi-tenant API where other customers will be hitting the same endpoint, similar to using an API endpoint from a frontier lab. At first, everything will be great, but then you may come across problems of time-to-first-token TTFT latency, caching-related costs, and latency issues. These are also problems of using APIs from frontier labs, the difference being that with inference providers, there’s the option of solving most of them by moving over to a dedicated deployment. Latency problems: TTFT, inference time, user-experienced latency Time-to-first-token is a measure of how long it takes from the moment a request hits the LLM API, to when the client starts receiving the first tokens. The distribution of TTFT latencies has a long tail: A long right tail means that a significant proportion of requests have double the median latency or longer. So, when measuring latencies, the p95/p99 latencies are just as important as p50 latencies. TTFT can be measured in two places: - Inference time: provider-instrumented TTFT shows the pure inference time. Some providers expose this metric to customers on dashboards, while some include it in their responses. Frontier labs typically do neither. - User-experienced latency: client-instrumented TTFT shows the full user-experienced latency, including things like network transport, or the TLS handshake. Server-side inference is just one part of the overall time that an app’s user spends awaiting a response. To achieve truly fast p95 times for user latency, it’s best to consider optimizations like connection pooling https://en.wikipedia.org/wiki/Connection pool on the LLM API consumer side. When serving LLM APIs on shared infrastructure, inference providers run many copies of the LLM inference engine that serves customer requests. A single inference engine can serve multiple requests concurrently. A new replica is created when the number of concurrent requests exceeds the current replica capacity. Scaling of replicas is one reason why inference-only TTFT latencies vary so much. Other reasons why TTFT can vary: 1. ‘Noisy neighbour’ effects : a request is batched with another request that’s more expensive to process, so it takes longer than if the request was in a batch with a similar request. 2. Cold start : happens when a request is waiting in a queue because there is no capacity in existing replicas. The inference provider is creating new replicas but this isn’t instantaneous, so a request has to wait until the new replica becomes available until the new replica starts up, from a “cold”, offline state . 3. Cold cache: when sending a large request for the first time, it may end up on a replica without the relevant KV cache loaded the cache is “cold” – which is the opposite of “hot cache” when the KV cache is already populated . An inference provider will typically try to keep requests sticky to the same replica because that keeps replica’s KV cache hot. Sometimes, this isn’t possible because your cache expired, or it landed on the replica that doesn’t have the relevant KV cache. KV cache then has to either be copied from cold storage if available, or if it’s not, fully processed before the first token can be generated. Prompt cache utilization is observable and actionable by engineers building on LLM APIs. You can get prompt cache utilization metrics from the response of the OpenAI completions API