AI spend has gone from experiment to budget line in about two years, and almost every FinOps team now manages some of it (State of FinOps 2026). The bill arrives from Bedrock, OpenAI, Copilot seats and GPU instances, each priced differently, and the VP of Engineering is usually the one who has to explain it. Most of the decisions below shrink that bill before you reach the bigger question of a self-hosted LLM vs API.
The short version
- Price per token keeps falling, but agents send far more tokens per task, so the bill still grows.
- A committed-spend discount is today’s biggest cut. It lowers the price, not the volume.
- Prompt caching is the cheapest engineering lever, and it saves most on features that send long context for short answers.
- Anything nobody waits on can run through a batch API at half price on Anthropic, Bedrock and OpenAI.
- Most routine work does not need a frontier model. A small model from the same vendor is often the quickest test, before any open-weight move.
- Self-hosting pays off only at high, steady volume.
Where the bill comes from
An enterprise AI bill can come from five kinds of provider, and each charges in a different way. A per-token API alone splits every model into input, output, cache-write and cache-read prices.
That bill also mixes two kinds of spend that grow for different reasons.
Tokens your engineers use to build. Copilot, Claude Code, Cursor and ChatGPT. This grows with headcount and with how much people lean on the tools. Since June 2026, Copilot seats come with AI Credits and usage past them is billed by token (GitHub), so seats now behave a little like an API. You manage this spend like other tooling: plan tier, model access and spend caps.
Tokens your product uses to serve customers. The support agent, the document summariser, the search assistant. This grows with customer usage and comes out of the margin on every customer. The rest of this post is mostly about this kind, because that is where caching, routing and self-hosting change the bill.
Why AI inference costs rise when token prices fall
Five patterns drive the growth.
Agents re-read everything. An agent calls the model in a loop until the task is done, and every call resends the system prompt, the tool definitions and everything that has happened so far. A ten-step task reads 23 times the tokens of a single reply.
The number of steps changes from one task to the next, so two tasks that look alike can cost very different amounts. That makes per-feature budgets hard to set.
Failures cost the same as successes. Retries, timeouts and answers that a guardrail throws away are all billed.
Frontier models do routine work. Teams pick the strongest model while prototyping and never revisit the choice. Classification and extraction then run on a model priced for hard reasoning.
Idle GPUs still bill. If you run your own capacity, your cost per token depends on how busy the GPU is, because the hourly rate does not change.
Nobody owns the total. Each team picks its own models, vendors and fixes, so no one sees the whole bill.
Which layers you control
Every AI workload has three layers: what you send (prompt, context, caching and model choice), where it runs (shared or dedicated hardware) and what runs it (the serving engine and the model weights). With seats, the vendor controls all three. A per-token API hands you the first. Dedicated GPUs add the second, and self-hosting gives you all three, along with the work of running them.
Enterprise AI discounts and their limits
A committed-spend deal with your cloud or model vendor is the quickest saving, and it needs no engineering work. It is the biggest cut most teams get today. It lowers the price of each token but not the number of tokens, so the bill still grows with usage, and the renewal starts from a higher base. The levers below change how many tokens you buy, and they keep paying after the contract is signed.
Lever 1: prompt caching on Bedrock and the Claude API
Most of each request is a prefix that repeats: the system prompt, tool definitions and shared examples. Providers can cache that prefix. Writing it to the cache costs more than normal input, and reading it back costs a tenth of the price or less (Anthropic pricing).
Caching cuts the input side of the bill by about 85%, but it never touches output. So the saving on your total bill depends on how many input tokens you send for each output token. A chat feature with short prompts saves about a quarter. An agent or a retrieval feature that sends long context for a short answer saves far more. The pattern holds on every Claude model we priced. Caching also lowers time to first token, because the model skips work on the cached prefix.
The cache only pays off if the prefix stays the same from call to call. Put static content first and anything that changes on every call after the cache breakpoint. A timestamp or user ID inside the system prompt turns every call into a fresh cache write, and you never get the cheap reads. Bedrock and the Claude API report cache reads and writes on every response, so you can get each feature’s cache-hit rate from the logs you already have.
Lever 2: batch inference for work nobody waits on
Some requests have a person waiting: a chat reply, a code suggestion, the next step of an agent. Many others do not. Nightly document classification, report summaries, data extraction and evaluation runs can finish in an hour and nobody notices.
Those jobs can go through a batch API, where you send many requests together and collect the results later. Anthropic’s Batch API, Bedrock batch inference and OpenAI’s Batch API each charge half the normal rate, and batch stacks with caching. A list of your AI jobs, each marked by whether someone waits on the answer, shows how much spend could move.
Lever 3: a smaller model for routine work
Your teams probably run jobs like these every day. An agent turns meeting notes into Jira tickets. Another summarises long Confluence pages, drafts release notes from merged pull requests, or tags incoming support tickets by product area. None of these needs a frontier model’s reasoning, yet many of them run on one.
The price gap is large even inside one vendor: Claude Haiku costs about a fortieth of Opus per token. Switching models within a vendor changes one line of config and keeps your contract, your security review and your data path as they are.
Open-weight models such as Llama, Qwen, DeepSeek and gpt-oss are the other route. In a September 2026 ETR survey, about a third of enterprise AI tokens already ran on open-weight models, and most teams running one named cost as a reason (Techstrong.ai). They still get a smaller share of spend than closed models, because they carry the cheap, high-volume work.
Whichever route you test, do it on your own data. Take a week of real requests for one job, run them through the cheaper model, check the outputs against what your current model produced, and compare the cost per correct answer.
There are three ways to run an open-weight model, in order of effort:
- A managed open-model API. Groq, Together AI, Fireworks and Baseten serve open models per token. You change an endpoint and a model name. This is the cheapest way to test.
- Dedicated GPUs. You rent GPUs by the hour and the platform runs the serving stack. You pay whether the GPU is busy or idle.
- Your own GPUs and serving stack. You run the engine, the scheduler and the hardware, and you carry the on-call.
Self-hosted LLM vs API: rent, route or own
The last decision is how much of the stack to own. Make it one workload at a time. Published frameworks such as Portkey’s and DField Solutions’ weigh the same four factors: data sensitivity, the capability the task needs, how high and steady the volume is, and whether you have a team to run GPUs.
Sensitive data does not on its own force self-hosting. Bedrock and similar services run the model inside your cloud provider’s boundary under your existing agreement, which many regulated teams accept. Self-hosting is the answer when policy rules out even that.
Rent means paying a provider per token. It suits low or spiky traffic and work that needs a frontier model. Idle time costs nothing.
Route means sending each task to the cheapest model that does it well: frontier models for hard reasoning, small or open models for routine work. A gateway in front of your providers does the routing and shows spend by model, key and team.
Own means paying for GPUs by the hour, so it only wins when the GPU stays busy.
Against a frontier model, one busy GPU can pay for itself. Against a small model it rarely does on price alone. Then the reasons to own are data that cannot leave, a fine-tuned model no provider will serve, or latency you cannot buy. Add the engineers and the on-call rota before you compare, and plan for capacity above your average so peaks do not queue.
Where to start: benchmark one workload
Gateways and provider consoles already show the spend. Switching is harder, because nobody moves a feature to a cheaper model until they trust it on their own work, in front of their own users, every day.
One2N builds that confidence with engineering teams, one workload at a time:
- Benchmark. We run open-weight and frontier models side by side on your real tasks and report quality, latency and cost per task for each.
- Fit. For each workload we recommend a model and a way to buy it: frontier API, inference provider, dedicated GPUs or your own hardware.
- Tune and run. Where an open model comes close but falls short, we fine-tune it, then help you serve and monitor it in production.
Pick the workload you would most like to move off a frontier API and write to partners@one2n.io. We will benchmark it with you.
Sources
Prices change often. Everything here was checked on 4 November 2026.
- FinOps Foundation, State of FinOps 2026 : 98% of FinOps teams now manage AI spend, up from 31% two years earlier.
- Anthropic, Claude pricing : cache writes cost 1.25 times the input price (five-minute cache) or 2 times (one hour); cache reads cost 10%, or 5% on Opus 5.5 and Sonnet 5.5; Haiku 5.5 is $0.10 and $0.50 per million input and output tokens against $4 and $20 for Opus 5.5; batch is half price.
- AWS, [Amazon Bedrock pricing](https://aws.amazon.com/bedrock/pricing/) : batch inference and cache pricing on Bedrock.
- OpenAI, [API pricing](https://openai.com/api/pricing/) : Batch API discount.
- GitHub, [Copilot usage-based billing](https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/) : seat prices and AI Credits.
-
Techstrong.ai, Open-weight AI models expected to capture 41% of enterprise token usage , 22 September 2026: ETR survey of 200 enterprises; open-weight models at 34% of token usage, up from 23%; 42% run one in production; 69% of those named cost.
-
Baseten H100 list price: computeprices.com ; confirm onbaseten.co/pricing . AI spend has gone from experiment to budget line in about two years, and almost every FinOps team now manages some of it (State of FinOps 2026). The bill arrives from Bedrock, OpenAI, Copilot seats and GPU instances, each priced differently, and the VP of Engineering is usually the one who has to explain it. Most of the decisions below shrink that bill before you reach the bigger question of a self-hosted LLM vs API.
The short version
- Price per token keeps falling, but agents send far more tokens per task, so the bill still grows.
- A committed-spend discount is today’s biggest cut. It lowers the price, not the volume.
- Prompt caching is the cheapest engineering lever, and it saves most on features that send long context for short answers.
- Anything nobody waits on can run through a batch API at half price on Anthropic, Bedrock and OpenAI.
- Most routine work does not need a frontier model. A small model from the same vendor is often the quickest test, before any open-weight move.
- Self-hosting pays off only at high, steady volume.
Where the bill comes from
An enterprise AI bill can come from five kinds of provider, and each charges in a different way. A per-token API alone splits every model into input, output, cache-write and cache-read prices.
That bill also mixes two kinds of spend that grow for different reasons.
Tokens your engineers use to build. Copilot, Claude Code, Cursor and ChatGPT. This grows with headcount and with how much people lean on the tools. Since June 2026, Copilot seats come with AI Credits and usage past them is billed by token (GitHub), so seats now behave a little like an API. You manage this spend like other tooling: plan tier, model access and spend caps.
Tokens your product uses to serve customers. The support agent, the document summariser, the search assistant. This grows with customer usage and comes out of the margin on every customer. The rest of this post is mostly about this kind, because that is where caching, routing and self-hosting change the bill.
Why AI inference costs rise when token prices fall
Five patterns drive the growth.
Agents re-read everything. An agent calls the model in a loop until the task is done, and every call resends the system prompt, the tool definitions and everything that has happened so far. A ten-step task reads 23 times the tokens of a single reply.
The number of steps changes from one task to the next, so two tasks that look alike can cost very different amounts. That makes per-feature budgets hard to set.
Failures cost the same as successes. Retries, timeouts and answers that a guardrail throws away are all billed.
Frontier models do routine work. Teams pick the strongest model while prototyping and never revisit the choice. Classification and extraction then run on a model priced for hard reasoning.
Idle GPUs still bill. If you run your own capacity, your cost per token depends on how busy the GPU is, because the hourly rate does not change.
Nobody owns the total. Each team picks its own models, vendors and fixes, so no one sees the whole bill.
Which layers you control
Every AI workload has three layers: what you send (prompt, context, caching and model choice), where it runs (shared or dedicated hardware) and what runs it (the serving engine and the model weights). With seats, the vendor controls all three. A per-token API hands you the first. Dedicated GPUs add the second, and self-hosting gives you all three, along with the work of running them.
Enterprise AI discounts and their limits
A committed-spend deal with your cloud or model vendor is the quickest saving, and it needs no engineering work. It is the biggest cut most teams get today. It lowers the price of each token but not the number of tokens, so the bill still grows with usage, and the renewal starts from a higher base. The levers below change how many tokens you buy, and they keep paying after the contract is signed.
Lever 1: prompt caching on Bedrock and the Claude API
Most of each request is a prefix that repeats: the system prompt, tool definitions and shared examples. Providers can cache that prefix. Writing it to the cache costs more than normal input, and reading it back costs a tenth of the price or less (Anthropic pricing).
Caching cuts the input side of the bill by about 85%, but it never touches output. So the saving on your total bill depends on how many input tokens you send for each output token. A chat feature with short prompts saves about a quarter. An agent or a retrieval feature that sends long context for a short answer saves far more. The pattern holds on every Claude model we priced. Caching also lowers time to first token, because the model skips work on the cached prefix.
The cache only pays off if the prefix stays the same from call to call. Put static content first and anything that changes on every call after the cache breakpoint. A timestamp or user ID inside the system prompt turns every call into a fresh cache write, and you never get the cheap reads. Bedrock and the Claude API report cache reads and writes on every response, so you can get each feature’s cache-hit rate from the logs you already have.
Lever 2: batch inference for work nobody waits on
Some requests have a person waiting: a chat reply, a code suggestion, the next step of an agent. Many others do not. Nightly document classification, report summaries, data extraction and evaluation runs can finish in an hour and nobody notices.
Those jobs can go through a batch API, where you send many requests together and collect the results later. Anthropic’s Batch API, Bedrock batch inference and OpenAI’s Batch API each charge half the normal rate, and batch stacks with caching. A list of your AI jobs, each marked by whether someone waits on the answer, shows how much spend could move.
Lever 3: a smaller model for routine work
Your teams probably run jobs like these every day. An agent turns meeting notes into Jira tickets. Another summarises long Confluence pages, drafts release notes from merged pull requests, or tags incoming support tickets by product area. None of these needs a frontier model’s reasoning, yet many of them run on one.
The price gap is large even inside one vendor: Claude Haiku costs about a fortieth of Opus per token. Switching models within a vendor changes one line of config and keeps your contract, your security review and your data path as they are.
Open-weight models such as Llama, Qwen, DeepSeek and gpt-oss are the other route. In a September 2026 ETR survey, about a third of enterprise AI tokens already ran on open-weight models, and most teams running one named cost as a reason (Techstrong.ai). They still get a smaller share of spend than closed models, because they carry the cheap, high-volume work.
Whichever route you test, do it on your own data. Take a week of real requests for one job, run them through the cheaper model, check the outputs against what your current model produced, and compare the cost per correct answer.
There are three ways to run an open-weight model, in order of effort:
- A managed open-model API. Groq, Together AI, Fireworks and Baseten serve open models per token. You change an endpoint and a model name. This is the cheapest way to test.
- Dedicated GPUs. You rent GPUs by the hour and the platform runs the serving stack. You pay whether the GPU is busy or idle.
- Your own GPUs and serving stack. You run the engine, the scheduler and the hardware, and you carry the on-call.
Self-hosted LLM vs API: rent, route or own
The last decision is how much of the stack to own. Make it one workload at a time. Published frameworks such as Portkey’s and DField Solutions’ weigh the same four factors: data sensitivity, the capability the task needs, how high and steady the volume is, and whether you have a team to run GPUs.
Sensitive data does not on its own force self-hosting. Bedrock and similar services run the model inside your cloud provider’s boundary under your existing agreement, which many regulated teams accept. Self-hosting is the answer when policy rules out even that.
Rent means paying a provider per token. It suits low or spiky traffic and work that needs a frontier model. Idle time costs nothing.
Route means sending each task to the cheapest model that does it well: frontier models for hard reasoning, small or open models for routine work. A gateway in front of your providers does the routing and shows spend by model, key and team.
Own means paying for GPUs by the hour, so it only wins when the GPU stays busy.
Against a frontier model, one busy GPU can pay for itself. Against a small model it rarely does on price alone. Then the reasons to own are data that cannot leave, a fine-tuned model no provider will serve, or latency you cannot buy. Add the engineers and the on-call rota before you compare, and plan for capacity above your average so peaks do not queue.
Where to start: benchmark one workload
Gateways and provider consoles already show the spend. Switching is harder, because nobody moves a feature to a cheaper model until they trust it on their own work, in front of their own users, every day.
One2N builds that confidence with engineering teams, one workload at a time:
- Benchmark. We run open-weight and frontier models side by side on your real tasks and report quality, latency and cost per task for each.
- Fit. For each workload we recommend a model and a way to buy it: frontier API, inference provider, dedicated GPUs or your own hardware.
- Tune and run. Where an open model comes close but falls short, we fine-tune it, then help you serve and monitor it in production.
Pick the workload you would most like to move off a frontier API and write to partners@one2n.io. We will benchmark it with you.
Sources
Prices change often. Everything here was checked on 4 November 2026.
- FinOps Foundation, State of FinOps 2026 : 98% of FinOps teams now manage AI spend, up from 31% two years earlier.
- Anthropic, Claude pricing : cache writes cost 1.25 times the input price (five-minute cache) or 2 times (one hour); cache reads cost 10%, or 5% on Opus 5.5 and Sonnet 5.5; Haiku 5.5 is $0.10 and $0.50 per million input and output tokens against $4 and $20 for Opus 5.5; batch is half price.
- AWS, [Amazon Bedrock pricing](https://aws.amazon.com/bedrock/pricing/) : batch inference and cache pricing on Bedrock.
- OpenAI, [API pricing](https://openai.com/api/pricing/) : Batch API discount.
- GitHub, [Copilot usage-based billing](https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/) : seat prices and AI Credits.
-
Techstrong.ai, Open-weight AI models expected to capture 41% of enterprise token usage , 22 September 2026: ETR survey of 200 enterprises; open-weight models at 34% of token usage, up from 23%; 42% run one in production; 69% of those named cost.
-
Baseten H100 list price: computeprices.com ; confirm onbaseten.co/pricing .
Share
Share
On this page
Section
On this page
In this post
section