Fix Context length exceeded
by accounting for the total budget, not by randomly deleting half the prompt. Identify the model and measure each request component first. Then remove mechanical duplicates, irrelevant history, and heavy tool results. After resending, verify both that the error is gone and that required facts and answer completeness remain intact.
Context is the working memory of one request, not just the latest user prompt. In simplified form:
system instructions
+ conversation history
+ current message
+ images and documents
+ tool definitions
+ tool results
+ output / thinking budget
= total context usage
Anthropic's context-window documentation explicitly includes the system prompt, all messages, images, documents, tool definitions, tool results, and generated response. Prompt caching changes the cost of reused tokens, but it does not remove them from the window.
Do not hard-code a “universal limit”: the context window and overflow behavior depend on the selected model and API. Check the current model card on the day you configure it.
| Component | What to measure | Safe reduction |
|---|---|---|
| System instructions | Repeated rules and long examples | Merge duplicates and retain mandatory constraints |
| History | Tokens by message and old branches | Remove irrelevant branches or replace them with a verifiable summary |
| Files and RAG fragments | Each document's size, duplicates, and low-relevance chunks | Lower top_k , deduplicate, and pass only the required sections |
| Tool definitions | Unused tools and long descriptions | Pass only the tools needed for the current step |
| Tool results | Complete JSON, logs, HTML, base64, and repeated responses | Keep required fields, links, and identifiers; store large data outside the prompt |
| Output budget | ||
max_tokens and thinking budget |
||
| Reserve a realistic amount or split a large result into stages |
Claude has a separate Token Counting API that accounts for messages and tools before a request is sent. For another provider, use its own counter if available. A local tokenizer is useful for early warnings, but its estimate is not a guaranteed server-side count for a different model.
For a request sent through BetterToken, the Dashboard lets you compare input, output, and cache tokens after a test. The current API documentation helps verify the contract, but the Dashboard does not show the full prompt or replace preflight token counting; it is a post-request budget check.
Look for repeated system rules, the same file in several messages, duplicate RAG chunks, repeated schemas, and complete logs pasted more than once. This is the safest stage because it reduces size without changing the task.
Separate long-lived facts from the temporary flow of the conversation. Preserve goals, accepted decisions, mandatory constraints, and open questions. Remove old reasoning, rejected alternatives, and tool results that have already been processed, or compact them into a structured summary.
A bad summary says “we discussed the integration.” A useful one records the chosen endpoint, schema version, accepted constraints, confirmed facts, and next step.
Pass relevant sections instead of an entire document. In a tool result, keep the fields required by the next step instead of the full HTTP response or log. Do not discard sources or mandatory data merely to make the request fit; split the work into several verifiable stages instead.
Input and output share the total budget. If the request nearly fills the window, the model may not have enough room for a complete answer. Reduce optional input, set a realistic output budget, or split the result into parts. Remove mechanical duplicates before critical facts.
A larger-context model may be the right choice for a document that cannot be split safely. Moving to a larger window without removing duplicates only postpones the next failure and may reduce information density.
components = count_by_section(request)
estimated_input = sum(components)
reserved_output = requested_output_budget
if estimated_input + reserved_output approaches current_model_window:
remove exact duplicates
drop irrelevant history
compact tool results and retrieved chunks
count again
send only after required facts and constraints remain present
approaches
is intentionally not replaced by a fixed percentage. The necessary headroom depends on counter accuracy, the model, thinking, and the specific API's behavior.
Compare the new request against this checklist:
context length exceeded
or prompt is too long
.If the error disappears but the model forgets a key constraint, the fix failed. Restore the mandatory block and free space by removing less relevant history or a heavy tool result. If the answer is cut off, inspect the output budget separately; it is another part of the same total window.
The working sequence is: identify the model → count components → remove exact duplicates → extract irrelevant history → compact files and tool results → leave room for the answer → resend → verify quality. This addresses the cause without turning context into an arbitrarily truncated set of facts.
Originally published on the BetterToken blog.
BetterToken provides pay-as-you-go access to AI model APIs through
OpenAI-compatible and Anthropic-compatible endpoints — useful if you are wiring
Claude Code, Codex, or your own tooling to a custom base URL.
See the docs to get started.