Your API dashboard shows a spike, but the feature "felt small." Look at the tokens: what the model reads, writes, and re-sends on every step. Tokenization is not academic. Context limits, per-token pricing, and latency all use the same unit, and dense code or odd formatting can cost more than the word count suggests.
Language models do not read whole words or raw characters. They read tokens, sub-word chunks from a fixed vocabulary. In English, a common word is often one token; rare words and code can split into several, and punctuation and spaces count too. OpenAI's rule of thumb is about four characters per token, or about three-quarters of a word, so 100 tokens is roughly 75 words.
That ratio drifts. Non-English text, code, and weird formatting usually burn more tokens per visible line. Before you argue about model tier, paste representative prompts into a tokenizer and note input tokens per request. Guessing from word count is how teams under-budget chat history and tool output.
The context window is a token budget for one response, and the latest user message is only one part of it. System instructions, prior turns, tool definitions, and tool results all sit in that budget. Output tokens need room too. In chat, unless your app trims or summarizes, the full history goes back on the next turn, so input tokens climb every step.
Exceeding the limit is not one universal behavior. Depending on the API, the request may be rejected or generation may stop, and many applications truncate or compact the input before sending it. Compaction changes what the model sees next; it does not make the window bigger. Anthropic's documentation also warns that accuracy and recall can degrade as context grows, what it calls context rot. For cost planning, whether a prompt fits in the window and whether it is cheap to keep sending are different questions.
For most teams on hosted models, inference is the bill that repeats: every message and agent step is another run. Training is someone else's capex; your product pays per call.
Providers charge input and output separately. On Anthropic's price list as of October 2026, output costs five times as much as input for each Claude model. Reasoning models can bill hidden reasoning as output, so a short visible answer can still be expensive. Repeated static instructions may qualify for prompt caching: Anthropic charges about a tenth of normal input to read a cached block, with a higher cost the first time you write it.
One successful task is rarely one call. Agents plan, call tools, retry, and often carry a growing context each time. Cost per task is roughly the sum over every call of (input tokens × input price) plus (output tokens × output price), including failures and retries.
Jeetu Patel, Cisco's president and chief product officer, argued in episode 67 that price per token could turn into a vanity metric: the price per token falls while long-running agents use more tokens. What matters to him is the price of the outcome. Your estimator should still start with tokens, because that is what you control in design.
Use your provider's current input and output rates. If you only remember the Anthropic ratio above, set OUTPUT_PRICE = 5 * INPUT_PRICE as a placeholder until you paste real numbers from the price list.
Label this a sketch, not production accounting:
def monthly_cost(tasks_per_month, calls_per_task, avg_input_tokens,
avg_output_tokens, input_price, output_price):
calls = tasks_per_month * calls_per_task
per_call = (avg_input_tokens * input_price
+ avg_output_tokens * output_price) / 1_000_000
return calls * per_call
INPUT_PRICE = 1.0 # placeholder
OUTPUT_PRICE = 5 * INPUT_PRICE # the ratio on Anthropic's list
bill = monthly_cost(50_000, 8, 12_000, 800, INPUT_PRICE, OUTPUT_PRICE)
Run three scenarios on the same workflow: today's averages, a heavy month (longer history, more retries), and a trimmed prompt (fewer input tokens or fewer calls per task). The line that moves the total most is where you spend engineering time.
Set calls_per_task realistically for agents. A single user action might be dozens of inferences. Divide total cost by completed tasks, not by API calls, when you compare features.
I think that today a lot of organizations are not thinking about their token spend regularly and they're getting hit with their bills.
Jiaona Zhang, Laurel
Jiaona Zhang ties that to "token maxxing": mandating AI use without asking whether the task earned the tokens, like using a model to redo slide fonts when one click would do. The estimator is how you catch that before finance does.
If output tokens dominate (long answers, reasoning traces, verbose tool JSON), tighten schemas, cap completion length, or route easy steps to a smaller model. If input tokens dominate (system prompt, full chat log, fat tool results), trim history, summarize older turns, cache stable instructions, and stop pasting whole documents when a chunk will do.
Jeetu Patel describes rationing tokens like headcount: a fixed pool and an expected output. That matches measuring cost per completed task and watching context on every turn, themes that show up across AI infrastructure conversations on the show.
The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.