Your local RAG isn't slow — it re-reads every document on every question

An engineer found that their local RAG app was slow because it re-read every retrieved document from token zero on every question, with prefill dominating generation by roughly five to one. The fix involved deduplicating on content rather than chunk id, cutting a query from 291 to 150 seconds, and revealing that the prompt cache was never used due to a query-rewriting service call.

A user opens a project with nine files in it, types the most obvious question anyone types at a document app — "what are these documents about?" — and waits. 291 seconds. Then they ask a second question, about one of those documents, and wait again. Minutes, not seconds. At that point the app has told them something about itself, and what it has told them is: this model is slow and probably stupid. The model was neither. It was reading. It read the entire retrieved corpus, from token zero, for question one. Then it read it again for question two. Setup, for symptom matching: an offline desktop app on llama.cpp, M4 Pro with 24 GB, a 14B at Q5, nine files in the project. Two throughput numbers, both from my own logs: generation: ~8 tokens/second <- normal for a 14B at Q5 on this box prefill: ~100 tokens/second <- also normal Neither is a bug. The bug is the ratio, and what got multiplied by it. That one question assembled a 13,773-token prompt , and the turn as a whole pushed 17,373 prompt tokens through the model once you count the service calls it makes on the side. At a hundred tokens a second, reading dominated writing by roughly five to one; the profile came out north of 85% prefill. Divide those numbers yourself and you'll land a few dozen seconds off. Prefill throughput sags as the context grows, and one "question" is more than one model call. The shape is the point, not the arithmetic: the machine spent its evening reading, and the part the user was waiting for — the answer — was cheap. I want to be precise about the thing I had gotten wrong for a long time, because I don't think I'm alone in it. I had indexing. Chunks, embeddings, a vector store, the whole ritual, run once when files are added. What I assumed that ritual bought me was that the documents were, in some sense, already read . It buys nothing of the kind. Retrieval is a table of contents, not a memory. It finds the right pieces quickly; it does not make the model read them faster, and it does not make the model remember having read them. Every question ships fresh text into the context window and the model chews it from the first token, at prefill speed, every time. In classical RAG, indexing time and reading time are separate budgets, and only one of them is ever spent in advance. Three things came out of the logs before I got to the interesting part. I run lexical search by default and vector search behind a flag. On the machines where both are on, both contribute passages to the prompt. They union their results, deduplicated by chunk id. Chunk id. Not content. The two engines index with different chunk boundaries, so the same paragraph arrives as two different ids with substantially the same text, and the union happily keeps both. Roughly 9,000 tokens per question were the same passages, twice. Not once in a while — on every question, for as long as both engines had been on. Nobody caught it by reading the prompt, because nobody reads the prompt; it is a wall of text that scrolls past in a debug log and looks exactly like a wall of text is supposed to look. Deduplicating on normalized content instead of id took the question from 291 to 150 seconds . That is the least interesting bug here and it was llama.cpp keeps a per-slot prompt cache. Send a prompt that shares a prefix with the last one that slot saw, and it skips prefill for the shared part. In a chat app that should be most of the system prompt, most of the time. Mine, trimmed: slot update slots: id 0 | task 412 | n past = 3, cache tokens = 3, n prompt tokens = 13773 Three tokens. Beginning-of-sequence and a bit of chat template. The cache had never once helped, on any turn, since the feature existed. The cause was a helper I'd been pleased with. Before retrieval, a small service call rewrites the user's question into a better search query — resolving "it" and "that contract" against the conversation. It's a good feature. It ran with its own system prompt, and it ran into the same slot as the conversation. Slot caches key on the longest common prefix. Two different system prompts diverge at token three. So the service call evicted the conversation's cached prefix, the conversation's next turn evicted the service call's, and the two took turns doing this forever. The cache was working exactly as designed. It was caching a conversation that alternated, every single turn, with a completely different conversation, in the same chair. Splitting the slots — a pool for conversations, a pool for service calls, never shared — moved the hit rate from 0% to 45-48% . There was already a background "study" pass in the codebase, meant to pre-summarize documents after import. It kept its queue in process memory. Close the app, and every unfinished item is gone. Not retried — gone, with no record that it had been scheduled. And it only ran while the app was open and otherwise idle, which on a desktop app is a narrow and unreliable window. Net effect: summaries were essentially never present when a question arrived, so every question fell back to reading document bodies. A background job that doesn't survive a restart is a background job that never finishes, because the user closes the window constantly and does not consider this an unusual thing to do. Here is the reframe the whole thing turned on. Prefill is not a cost you can optimize away — a 14B reading N tokens has to read N tokens. It is a cost you can move . The question is whether the model reads a document at question time, while a human watches a spinner, or at indexing time, when nobody is waiting. Three pieces, in order of how cheap they are. For each file, a small structured record: document type, parties, dates, amounts, page count, and a table of contents extracted from heading patterns with a regular expression. Not with the model — with a regex, at import, in milliseconds. That's about 200 tokens per document. For a nine-file project, "what are these documents about?" now has a 1,800-token answer surface where it used to have 13,773. The honest limit: heading extraction works on documents that have formatting and fails flat on an unstructured wall of text. Those fall through to the pass below. Per file: chunk it, summarize each chunk, summarize the summaries. This costs exactly the minutes you were paying before — the model still reads the whole document — except it costs them once , on import, and never again. The queue rules matter more than the summarization prompt: The user does pay for this. They pay in fan noise, once per file, at a moment when they are not staring at a progress bar. Routing is the risky part and I'd rather name the risk than sell around it. Misroute a specific question into the digest lane and you answer from a summary that dropped the exact number the user wanted, confidently and wrongly. Two mitigations, both boring: the router is biased to escalate — if a question mentions a term that appears in a passport's table of contents but not in the summary, it goes to chunks — and the app shows which lane answered, so a thin answer has an obvious "go read it properly" next to it. | before | after | || |---|---|---|---| | "what are these documents about?", 9 files | 291 s | 102 s | 2.9x | | prompt for that question | 13,773 tok | 5,709 tok | 2.4x | | the same question asked verbatim again | minutes | ~1 s | exact-match cache | | prefix cache hit rate, specific questions | 0% | 45-48% | slot split | 102 seconds is not a good number. It is a much better number, and it is an honest one: a 14B on a laptop reading five thousand tokens has to read five thousand tokens, and no amount of architecture argues with that. The exact-match cache is the cheapest line in the table and it exists because of a behavior I did not predict. Users re-ask the identical question. They close the app, come back, and type the same words to see whether it's still right. Hashing the normalized question plus the resolved context set and keeping the answer turns that into a second. The reusable part of a prompt is the part that doesn't change. System prompt and passports are stable, so they cache. Retrieved excerpts change with the question — by construction, since changing them is the entire job of retrieval. You could force them to cache. Retrieve once per conversation, freeze the context, and every subsequent turn shares a long identical prefix. I sat with that for a while and turned it down. The second question in a conversation is usually about something the first question didn't retrieve; freezing the context buys cache hits and pays for them in wrong answers. 45-48% is what the stable prefix is genuinely worth in this layout, and I'd rather report that number than a better one I bought with accuracy. One thing that is free: order the prompt by volatility . Stable first system prompt, passports , volatile last excerpts, then the question . Get that backwards and your hit rate is zero no matter how much of the prompt is technically stable. Before claiming any of this was novel I looked at what the neighbors ship: LM Studio, AnythingLLM, Jan, GPT4All, Open WebUI with Ollama. All of them chunk and embed at index time. Not one of them precomputes per-document digests or summaries. The best of them keep a prompt cache, which helps with the system prompt and does nothing for the retrieved half. Which means that on "what are these documents about?" — the single most common opening question a human asks a document app, the one they type before they type anything else — every one of these re-reads the corpus at full prefill cost, every time. I'm supposed to call that a gap in the market. It's really a gap in the default architecture: the reference RAG design does retrieval at question time and nothing at index time except embeddings, everybody copied it faithfully, and the copy is correct. It's just that "correct" and "the user waited five minutes" are compatible states. Measure prefill against generation before you optimize anything. If 85% of your wall clock is reading, a faster sampler and a smaller quant are noise. The engine prints both numbers; find the line. Deduplicate context sources by content, not by id. Two retrievers with different chunk boundaries will hand you the same paragraph twice and neither will look wrong in isolation. This was 9,000 tokens a question in my app. Give service LLM calls their own KV slot. Query rewriting, classification, title generation — anything with its own system prompt sharing a slot with the conversation will zero your prefix cache and the logs will still say the cache is enabled. Precompute passports and summaries at index time. A ~200-token structured passport per document, with headings pulled by regex rather than by the model, answers most broad questions on its own and costs no GPU at all. Put the study queue in the database. In-memory queues on a desktop app do not survive contact with users, who close windows. Checkpoint per chunk, resume on launch, yield to live traffic, stop on battery. Cache the literal repeats. People re-ask the same question verbatim more than you'd think. Hash the normalized question plus the resolved context set. If you want to know whether any of this is worth your afternoon, the arithmetic is short enough to run before you commit to it: file size, prefill speed, number of questions, and it tells you how many seconds of pure reading you have already signed up for. That's how long will my rag wait.py — https://github.com/JackYU96/rag-rereads-every-question https://github.com/JackYU96/rag-rereads-every-question — in the repo next to this post — no dependencies, one file, bring your own numbers. The last time I chased a number like this, the KV cache itself turned out to be eating it: "V cache quantization requires flash attn" — the llama.cpp error that quietly halves your context window https://dev.to/dreamdeck/v-cache-quantization-requires-flashattn-the-llamacpp-error-that-quietly-halves-your-context-1kdb .