“Token Limit Exceeded.”
We’ve all encountered this error while working with Large Language Models. Most of us simply increase the limit and continue — but very few stop to ask a more important question: What exactly is a token?
There’s another question that often goes unnoticed:
Does the same sentence consume the same number of tokens across GPT, Claude, Gemini, and other models? The answer is
Tokens are the* hidden currency* behind every AI interaction. They determine our API bill, response latency, context window, and how efficiently our applications scale. Yet despite being one of the most fundamental concepts in Generative AI, they’re still widely misunderstood.
By the end of this article, we’ll understand what tokens are, why different models count them differently, how to measure them in Python, and how to use that knowledge to build more efficient AI applications.
One startup accidentally spent thousands of dollars because it resent entire chat histories on every request. The culprit wasn’t the model. It was tokens.
Think of tokens like the weight of a parcel. The heavier the parcel, the more you pay to ship it. LLM providers work the same way — they charge based on tokens, not characters or words.
The difference is: you choose what goes in the envelope. Understanding tokens gives you control over how heavy that parcel is.
Tokens determine:
A token is a small chunk of text. It could be:
The important thing: the model never reads raw text. It only understands numbers. Every token is mapped to a unique numerical ID using the model’s vocabulary before being processed by the neural network.
Prompt follows this lifecycle before the model can process it.
Step 1 — Encoding: Your text is split into chunks (tokens)Step 2 — Mapping: Each chunk is converted to a unique number IDStep 3 — Processing: The model runs billions of calculations on those numbers**Step 4 **— Decoding: The output numbers are converted back to readable text
So when you send “Hello World”, the model doesn’t see two words. It sees something like: [13225, 5922]
import tiktokenenc = tiktoken.encoding_for_model("gpt-4o")text = "Hello World"tokens = enc.encode(text)print(tokens) # [13225, 5922]print(len(tokens)) # 2
One of the biggest misconceptions is that every LLM counts tokens in the same way.
The answer is no.
Each provider uses its own tokenizer — the algorithm responsible for splitting text into tokens before the model processes it. As a result, the same sentence can produce different token counts across GPT, Claude, Gemini, and other LLMs.
If we measure API usage, the reported input tokens may also include hidden message formatting such as chat roles and system prompts. This is why token counts from different providers aren’t always directly comparable.
Tokenizers are trained differently, and vocabulary size matters enormously.
Take the word “understanding”. Depending on the tokenizer:
Most modern models use subword tokenization — a middle ground that handles common words as single tokens and splits rare words into recognisable pieces. This is why:
import tiktokenenc = tiktoken.encoding_for_model("gpt-4o")words = ["hello", "understanding", "supercalifragilistic", "antidisestablishmentarianism"]for word in words: tokens = enc.encode(word) parts = [enc.decode([t]) for t in tokens] print(f"{word!r:35} → {len(tokens)} token(s): {parts}")# Sample Output:# 'hello' → 1 token(s): ['hello']#'understanding' → 2 token(s): ['under', 'standing']#'supercalifragilistic' → 6 token(s): ['super', 'cal', 'if', 'rag', 'il', 'istic']#'antidisestablishmentarianism' → 6 token(s): ['ant', 'idis', 'est', 'ablishment', 'arian', 'ism']
This is one of the most common misconceptions. Let’s put numbers to it:
import tiktokenenc = tiktoken.encoding_for_model("gpt-4o")text = ( "The wise owl of the moonlight forest, where ancient trees stretch their branches toward the starry sky.")char_count = len(text)token_count = len(enc.encode(text))print(f"Characters : {char_count}") # 103print(f"Tokens : {token_count}") # ~21print(f"Ratio : {char_count / token_count:.1f} chars per token") # ~4.9
A rough rule of thumb for English: 1 token ≈ 4 characters ≈ ¾ of a word.
But this ratio breaks down fast with technical content, code, uncommon languages, or emoji — all of which consume tokens less efficiently.
Every API call has two separate token buckets, priced differently:**Input tokens are the text you send to the model, including prompts, chat history, and retrieved documents. Output tokens **are the words generated by the model in its response.
Here’s the critical part: output tokens cost 3–5× more than input tokens across most providers. Generating text is computationally harder than reading it.
If your app generates long, detailed responses, output cost will dominate your bill — not the size of your prompt. This changes how you should think about optimisation.
The context window is the maximum number of tokens a model can process in a single call. Think of it as the model’s whiteboard — everything must fit on it at once: your prompt, the chat history, and the reply being generated.
Larger windows mean the model can “see” more at once. But there’s a cost-latency tradeoff that developers often miss:
A common mistake: sending a 50,000-token document with every user message, even when the user only asks a simple question. The document gets charged as input tokens every single time.
The hidden cost most developers discover too late: growing chat history
Here’s the trap that catches almost every developer building a chatbot.
Models have no memory between API calls. To maintain a conversation, your app resend the entire conversation history on every new message — including previous user messages and the model’s responses. That means by turn 10, you’re paying to process turns 1 through 9 as input tokens — every time.
INPUT_PRICE = 3.00 / 1_000_000 # Claude Sonnet 4 - per input tokenOUTPUT_PRICE = 15.00 / 1_000_000 # Claude Sonnet 4 - per output tokenSYSTEM_TOKENS = 500 # Fixed system promptTOKENS_PER_TURN = 300 # Average tokens added per exchangeOUTPUT_TOKENS = 400 # Average response lengthtotal_cost = 0for turn in range(1, 11): history_tokens = turn * TOKENS_PER_TURN input_tokens = SYSTEM_TOKENS + history_tokens call_cost = ( (input_tokens * INPUT_PRICE) + (OUTPUT_TOKENS * OUTPUT_PRICE) ) total_cost += call_cost print( f"Turn {turn:2d}: " f"{input_tokens:5d} input tokens | " f"${call_cost:.4f} per call" )print(f"\n10-turn session total: ${total_cost:.4f}")# Sample Output:# Turn 1: 800 input tokens | $0.0084 per call# Turn 2: 1100 input tokens | $0.0093 per call# Turn 3: 1400 input tokens | $0.0102 per call# Turn 4: 1700 input tokens | $0.0111 per call# Turn 5: 2000 input tokens | $0.0120 per call# Turn 6: 2300 input tokens | $0.0129 per call# Turn 7: 2600 input tokens | $0.0138 per call# Turn 8: 2900 input tokens | $0.0147 per call# Turn 9: 3200 input tokens | $0.0156 per call# Turn 10: 3500 input tokens | $0.0165 per call## 10-turn session total: $0.1245
By turn 10, you’re sending 3,500 input tokens just to process one new message. The cost per session nearly doubles from turn 1 to turn 10 — and this compounds fast at scale.
Many chat applications send the entire conversation history with every request.
As conversations become longer, token usage grows rapidly.
Turn 1 → 800 tokens
Turn 5 → 2,000 tokens
Turn 10 → 3,500 tokens
Turn 20 → 6,800 tokens
If you keep resending old messages, costs increase silently and responses become slower.
Tokenizers are trained on real-world data, and popular languages dominate that data. This has a direct consequence: code written in Python or JavaScript is tokenized far more efficiently than equivalent logic in less common languages.
The same principle applies to natural languages. English and Spanish tokenize efficiently. Less common languages can cost 3–5× more tokens to express the same information — a genuine accessibility and cost issue that often goes unacknowledged.
NOTE: A single emoji like🎉 often costs 3 tokens. At millions of requests per month, even emoji can become expensive.
1. Always read usage metadata Most provider returns token usage in the response. Make this a reflex:
usage = response.usageprint(f"Input: {usage.input_tokens} | Output: {usage.output_tokens}")
Once you’re tracking it, you’ll spot inefficiencies immediately.
2. Benchmark token counts across providers The same prompt can cost 3× more on one provider vs another due to tokenizer differences. Always measure before committing to a model in production.
3. Write concise prompts Filler, hedges, and repetition inflate token counts without improving output quality.
4. Watch for rare vocabulary Technical jargon, domain-specific terms, and unusual words get split into many tokens. Where possible, rephrase using common vocabulary, especially in high-frequency prompt templates.
5. Trim or summarise chat history Don’t send the full conversation history forever. Keep the last N turns, or use a cheap model to summarise older turns into a short paragraph before including them in the next call.
Before understanding tokens, you think of prompts as text.
After understanding tokens, you think of them as cost units, latency drivers, and context budgets.
That shift changes how you design prompts, choose models, and build AI applications that scale efficiently.
Whether you’re building a chatbot, an AI agent, or a production-grade LLM application, understanding tokens helps you make better engineering decisions — not just reduce costs.
So the next time you see “Token Limit Exceeded”, you won’t just increase the limit. You’ll understand why it happened and how to optimize it.
Track your token usage. Measure before you optimize. And remember: a context window isn’t free storage — it’s a budget. Spend it wisely.
LLM Tokens Explained: Cost, Memory, Speed and Context Windows was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.