AI adoption is being hindered by token costs and there being no up-front pricing for AI requests leading to excessive token spending. AI agents could make this problem many times worse as they are not - yet -constrained by token cost guardrails.
What are tokens? How are they priced and what could be done to prevent token spend disasters.
An unnamed enterprise reportedly racked up a half-billion-dollar bill on Anthropic’s Claude after forgetting to set usage limits for employees, and another report says Uber engineers have already spent their entire AI budget for 2026. Tokens are AI large language models' (LLMs) and chatbots' way of breaking down natural language prompts and generating natural language outputs.
An input prompt, such as “Tell me about sedimentation” will be divided into component parts;
Tell
me
about
sediment
ation
Each of these five components is a token. The division of sedimentation into two tokens enables the LLM to grasp the meaning of sediment being deposited or made (-ation), as an -ation suffix can be attached to many nouns; capitalization, computerization, taxation, etc. It avoids the LLM having to search for every possible word with an -ation suffix in its index. Instead it cuts down the search scope, and hence time, by only searching for “sediment” and “ation.”
Tokens, data objects, are mapped into vector embeddings, mathematically calculated number strings that represent their meaning and are stored in an index. A word, such as cat, indicating the animal, could have multiple vectors representing different aspects (dimensions) of a cat: living being, domestic pet, feline-specific(whiskers agility), size, etc. These separate it from non-living things, dogs, mice, tigers, stiffed toys, etc.
A 5-dimension vector embedding for cat could hypothetically be [1.5, -0.4, 7.2, 19.6, 20.2].
The vectors exist in a space and similar data objects can be found by being close to the search object in that space.
Tokens, once vectorized, are processed by vastly expensive GPUs servers with their high-bandwidth memory. An Nvidia DGX SuperPOD with 32 nodes and 256 GPUs in total could cost $12-20 million. A reference architecture 140DGX H100 node SuperPOD with Quantum-2 InfiniBand networking, storage, networking infrastructure, and supporting systems might cost upwards of $100 million. This is reflected in foundation model token pricing. Chatbot and LLM providers like OpenAI and Anthropic charge their users on a token basis and you can roughly assess how many tokens a prompt could generate. One English token might be around four characters, or about 0.75 words.: 750 words might generate 1,000 tokens. Both the input prompt and the output response generate tokens, so a 750 word input with a 2,000 word output could mean you use 3,667 tokens.
The more tokens you use the longer the AI takes to respond. a single prompt and response session has its tokens stored in a context window, which has a limited capacity. Decagon says a 100,000-token context window holds roughly 75,000 English words — about a 300-page book.
If the context window is too small, then the LLM suffers memory loss and the AI’s response is incomplete or faulty. Context window size has grown as foundation models have developed, with GPT-4o supporting up to 128,000 tokens, Claude 3.5 Sonnet 200,000, and up to 2 million for select Gemini 1.5 Pro customers.
Pricing varies per model provider. [Intuition Labs](https://intuitionlabs.ai/articles/chatgpt-api-pricing-2026-token-costs-limits) provides a ChatGPT pricing chart:
Intuition Labs says: “OpenAI’s ChatGPT API uses a token-based billing system. Each model has a fixed price per input token and per output token. These prices can differ (often output tokens cost more) and also depend on factors like whether the model’s response was cached. Developers incur two charges per API call: one for the prompt tokens, one for the generated tokens . For example, if you send 100 input tokens and receive 400 output tokens using a model priced at $10 per million output tokens, the input cost is $10×(100/1,000,000) and the output cost is $10×(400/1,000,000), and total cost is their sum.”
Enterprise users can have controls imposed to limit their token spend, with per user or seat quotas, usage-based throttling, per-project quotas, model tier limits and so forth. It’s a fast developing area as the model providers compete to provide the best service while optimizing their revenues and competing with open weight models.
AI agents represent a whole new token cost dilemma as they could initiate grandiose token spend unless given guardrails of some kind. Treat them as digital employees and have FinOps people monitor them very, very carefully.