I run a content pipeline on a Mac mini (48GB unified memory) that splits long blog drafts into platform-specific short-form pieces. That job — read a 30-page document, hold the whole thing in mind, extract what matters for YouTube Shorts vs TikTok vs Reels — is exactly what long-context LLMs are supposed to be good at.
Mine wasn't. It kept "forgetting" the second half of every document, dropping key details, and producing shallow summaries no matter how I tuned the prompt.
I did what you'd do. Simplified the prompt. Rewrote the template. Swapped models. Re-downloaded them, twice. Spent entire evenings after work on this, convinced the model was the problem — a Q4_K_M quantized 13B–20B model should handle long documents, right? The symptoms said otherwise: solid on the first pages, incoherent by the end.
Classic context-window behavior. I just didn't see it yet.
Then I actually read the LM Studio load log instead of scrolling past it:
context_length: 4096
The model I was running supports 262,144 tokens of context. It was loaded with 4,096.
That's 1.6% of what the model can do. A 48-lane highway restricted to one lane — and every long document I fed it was quietly getting truncated into memory of just the opening section.
LM Studio's just-in-time model picks a conservative default context length on first load. For chat and short Q&A, 4096 is plenty and keeps memory pressure low — a sensible default for most users. For document-scale work, it's a silent killer. Nothing errors out. Nothing warns you. The model just appears to have a bad memory.
Context is the model's working memory. Cap it at 4k tokens and a 30-page brief becomes "read the first two pages, forget the rest."
Two things:
Context Length: 260000
(whatever your model supports — check the model card, not the default), then reload. On 48GB of unified memory the larger KV cache is entirely affordable.Immediate, dramatic improvement. Full-document comprehension, per-platform extraction without drift, details intact end to end. Same model, same hardware, same prompt — one setting was capping ~98% of the model's effective utility for my workload.
Defaults are tuned for the average case, and document-scale synthesis is not the average case. When a local LLM "feels dumb," check what it was actually loaded with before blaming the weights:
context_length
is actually in effect?The most expensive performance bug I've shipped was a single default value.
This post is based on a first-hand work log, written with AI assistance.