Your LLM provider is probably serving you 32K context no matter what the model card says

A developer running a hosted chat and coding agent on open-weight models reports that most hosted LLM endpoints serve roughly 32K tokens of context regardless of the context window advertised on the model card, with only a handful reaching 256K and none found actually serving the 1M figures from release announcements. Because providers rarely document the effective ceiling and truncation fails silently rather than returning an error, the developer warns that agentic runs, gateway fallbacks and RAG retrieval tuning can break in ways that look like model weakness rather than a serving configuration limit. The developer, who discloses building the Grunz chat and coding agent, suggests probing endpoints with oversized prompts to read the true maximum from error text or watching where truncation begins to bite.

I run a hosted chat and coding agent on open-weight models. This is the single finding that cost me the most time in the last few months, and almost nobody talks about it. The context window advertised on a model card is a property of the weights. The context window you actually receive is a property of whoever is serving them. Across the hosted endpoints I have tested, most serve around 32K regardless of what the card claims. A handful reach 256K. I have not yet found one actually serving the 1M figures that appear in release announcements. This is entirely reasonable from the operator's side. Max context length is a KV-cache budget traded off against concurrency. Serving a million tokens per request to everyone would be ruinous. The problem is not that operators do this. The problem is that it is almost never documented and it fails silently. That is the part that makes it expensive. You do not get a 413. You do not get a warning. The request is served against whichever ceiling is lowest and the front of your context is simply gone. For chat, this is close to invisible. The conversation gets slightly dumber over time and you assume that is just how it goes. For anything agentic it is the binding constraint, and it fails in a way that does not look like a context failure: From the outside that reads as "this model is bad at long tasks." It is the serving config. There is no clean way to do this, which is itself the story. Two methods that work: Send a deliberately oversized prompt and read the error. Providers usually leak the true maximum in the error text even when the docs do not mention it. Push 500K tokens at an endpoint claiming 1M and see what comes back. Watch where truncation starts biting. In a long run, find the point at which the earliest content stops influencing output, and count backwards. Uglier, but it works when the error text is unhelpful. Provider documentation has not been reliable for this in my experience. Neither has the model card. 1. A benchmark number is a score for a model at one served context length, not a score for a model. The same weights at 32K and at 200K are not the same agent. If you are comparing providers on cost per token without pinning the served ceiling, you are comparing two different things and calling the difference price. 2. If you use a gateway with fallbacks, the ceiling can change mid-session. A run that starts on a 200K provider and fails over to a 32K one does not error. It truncates. That is a correctness problem, not a performance one, and nothing in the response tells the caller it happened. 3. In RAG, this silently invalidates your retrieval tuning. Chunk size, top-k and reranking are all tuned against an assumed budget taken from the card. If you tuned for 128K and you are served 32K, you are over-retrieving, your reranked chunks get truncated, and the answer comes back confident and wrong. Then you go tune the embedding model, which was never the problem. Reranking in particular buys you nothing if the top results do not survive into the prompt. A discoverable way to read the effective served context per request. Model metadata, a response header, anything. Right now every client reverse-engineers it from error strings, which is absurd for a number that determines whether your application works. If you know of a provider that publishes this honestly, I would genuinely like to hear about it. Founder disclosure: I build Grunz https://grunzai.com , a chat and coding agent running open-weight models, which is how I ended up learning all of this the expensive way.