{"slug": "your-local-rag-isn-t-slow-it-re-reads-every-document-on-every-question", "title": "Your local RAG isn't slow — it re-reads every document on every question", "summary": "An engineer found that their local RAG app was slow because it re-read every retrieved document from token zero on every question, with prefill dominating generation by roughly five to one. The fix involved deduplicating on content rather than chunk id, cutting a query from 291 to 150 seconds, and revealing that the prompt cache was never used due to a query-rewriting service call.", "body_md": "A user opens a project with nine files in it, types the most obvious question\n\nanyone types at a document app — \"what are these documents about?\" — and waits.\n\n291 seconds.\n\nThen they ask a second question, about one of those documents, and wait again.\n\nMinutes, not seconds. At that point the app has told them something about\n\nitself, and what it has told them is: this model is slow and probably stupid.\n\nThe model was neither. It was reading. It read the entire retrieved corpus,\n\nfrom token zero, for question one. Then it read it again for question two.\n\nSetup, for symptom matching: an offline desktop app on llama.cpp, M4 Pro with\n\n24 GB, a 14B at Q5, nine files in the project.\n\nTwo throughput numbers, both from my own logs:\n\n```\ngeneration:  ~8 tokens/second     <- normal for a 14B at Q5 on this box\nprefill:   ~100 tokens/second     <- also normal\n```\n\nNeither is a bug. The bug is the ratio, and what got multiplied by it. That one\n\nquestion assembled a **13,773-token prompt**, and the turn as a whole pushed\n\n**17,373 prompt tokens** through the model once you count the service calls it\n\nmakes on the side. At a hundred tokens a second, reading dominated writing by\n\nroughly five to one; the profile came out north of 85% prefill.\n\nDivide those numbers yourself and you'll land a few dozen seconds off. Prefill\n\nthroughput sags as the context grows, and one \"question\" is more than one model\n\ncall. The shape is the point, not the arithmetic: the machine spent its evening\n\nreading, and the part the user was waiting for — the answer — was cheap.\n\nI want to be precise about the thing I had gotten wrong for a long time,\n\nbecause I don't think I'm alone in it. I had indexing. Chunks, embeddings, a\n\nvector store, the whole ritual, run once when files are added. What I assumed\n\nthat ritual bought me was that the documents were, in some sense, *already\nread*.\n\nIt buys nothing of the kind. Retrieval is a table of contents, not a memory. It\n\nfinds the right pieces quickly; it does not make the model read them faster,\n\nand it does not make the model remember having read them. Every question ships\n\nfresh text into the context window and the model chews it from the first token,\n\nat prefill speed, every time. In classical RAG, indexing time and reading time\n\nare separate budgets, and only one of them is ever spent in advance.\n\nThree things came out of the logs before I got to the interesting part.\n\nI run lexical search by default and vector search behind a flag. On the\n\nmachines where both are on, both contribute passages to the prompt. They union\n\ntheir results, deduplicated by chunk id.\n\nChunk id. Not content. The two engines index with different chunk boundaries,\n\nso the same paragraph arrives as two different ids with substantially the same\n\ntext, and the union happily keeps both.\n\nRoughly **9,000 tokens per question** were the same passages, twice. Not once\n\nin a while — on every question, for as long as both engines had been on. Nobody\n\ncaught it by reading the prompt, because nobody reads the prompt; it is a wall\n\nof text that scrolls past in a debug log and looks exactly like a wall of text\n\nis supposed to look.\n\nDeduplicating on normalized content instead of id took the question from **291\nto 150 seconds**. That is the least interesting bug here and it was\n\nllama.cpp keeps a per-slot prompt cache. Send a prompt that shares a prefix\n\nwith the last one that slot saw, and it skips prefill for the shared part. In a\n\nchat app that should be most of the system prompt, most of the time.\n\nMine, trimmed:\n\n```\nslot update_slots: id  0 | task 412 | n_past = 3, cache_tokens = 3, n_prompt_tokens = 13773\n```\n\nThree tokens. Beginning-of-sequence and a bit of chat template. The cache had\n\nnever once helped, on any turn, since the feature existed.\n\nThe cause was a helper I'd been pleased with. Before retrieval, a small service\n\ncall rewrites the user's question into a better search query — resolving \"it\"\n\nand \"that contract\" against the conversation. It's a good feature. It ran with\n\nits own system prompt, and it ran **into the same slot** as the conversation.\n\nSlot caches key on the longest common prefix. Two different system prompts\n\ndiverge at token three. So the service call evicted the conversation's cached\n\nprefix, the conversation's next turn evicted the service call's, and the two\n\ntook turns doing this forever. The cache was working exactly as designed. It\n\nwas caching a conversation that alternated, every single turn, with a\n\ncompletely different conversation, in the same chair.\n\nSplitting the slots — a pool for conversations, a pool for service calls, never\n\nshared — moved the hit rate from **0% to 45-48%**.\n\nThere was already a background \"study\" pass in the codebase, meant to\n\npre-summarize documents after import. It kept its queue in process memory.\n\nClose the app, and every unfinished item is gone. Not retried — gone, with no\n\nrecord that it had been scheduled. And it only ran while the app was open and\n\notherwise idle, which on a desktop app is a narrow and unreliable window.\n\nNet effect: summaries were essentially never present when a question arrived,\n\nso every question fell back to reading document bodies. A background job that\n\ndoesn't survive a restart is a background job that never finishes, because the\n\nuser closes the window constantly and does not consider this an unusual thing\n\nto do.\n\nHere is the reframe the whole thing turned on. Prefill is not a cost you can\n\noptimize away — a 14B reading N tokens has to read N tokens. It is a cost you\n\ncan **move**. The question is whether the model reads a document at question\n\ntime, while a human watches a spinner, or at indexing time, when nobody is\n\nwaiting.\n\nThree pieces, in order of how cheap they are.\n\nFor each file, a small structured record: document type, parties, dates,\n\namounts, page count, and a table of contents extracted from heading patterns\n\nwith a regular expression. Not with the model — with a regex, at import, in\n\nmilliseconds.\n\nThat's about 200 tokens per document. For a nine-file project, \"what are these\n\ndocuments about?\" now has a **1,800-token** answer surface where it used to\n\nhave 13,773.\n\nThe honest limit: heading extraction works on documents that have formatting\n\nand fails flat on an unstructured wall of text. Those fall through to the pass\n\nbelow.\n\nPer file: chunk it, summarize each chunk, summarize the summaries. This costs\n\nexactly the minutes you were paying before — the model still reads the whole\n\ndocument — except it costs them **once**, on import, and never again.\n\nThe queue rules matter more than the summarization prompt:\n\nThe user does pay for this. They pay in fan noise, once per file, at a moment\n\nwhen they are not staring at a progress bar.\n\nRouting is the risky part and I'd rather name the risk than sell around it.\n\nMisroute a specific question into the digest lane and you answer from a summary\n\nthat dropped the exact number the user wanted, confidently and wrongly. Two\n\nmitigations, both boring: the router is biased to escalate — if a question\n\nmentions a term that appears in a passport's table of contents but not in the\n\nsummary, it goes to chunks — and the app shows which lane answered, so a thin\n\nanswer has an obvious \"go read it properly\" next to it.\n\n| before | after | ||\n|---|---|---|---|\n| \"what are these documents about?\", 9 files | 291 s | 102 s | 2.9x |\n| prompt for that question | 13,773 tok | 5,709 tok | 2.4x |\n| the same question asked verbatim again | minutes | ~1 s | exact-match cache |\n| prefix cache hit rate, specific questions | 0% | 45-48% | slot split |\n\n102 seconds is not a good number. It is a much better number, and it is an\n\nhonest one: a 14B on a laptop reading five thousand tokens has to read five\n\nthousand tokens, and no amount of architecture argues with that.\n\nThe exact-match cache is the cheapest line in the table and it exists because\n\nof a behavior I did not predict. Users re-ask the identical question. They\n\nclose the app, come back, and type the same words to see whether it's still\n\nright. Hashing the normalized question plus the resolved context set and\n\nkeeping the answer turns that into a second.\n\nThe reusable part of a prompt is the part that doesn't change. System prompt\n\nand passports are stable, so they cache. Retrieved excerpts change with the\n\nquestion — by construction, since changing them is the entire job of retrieval.\n\nYou could force them to cache. Retrieve once per conversation, freeze the\n\ncontext, and every subsequent turn shares a long identical prefix. I sat with\n\nthat for a while and turned it down. The second question in a conversation is\n\nusually about something the first question didn't retrieve; freezing the\n\ncontext buys cache hits and pays for them in wrong answers. 45-48% is what the\n\nstable prefix is genuinely worth in this layout, and I'd rather report that\n\nnumber than a better one I bought with accuracy.\n\nOne thing that is free: **order the prompt by volatility**. Stable first\n\n(system prompt, passports), volatile last (excerpts, then the question). Get\n\nthat backwards and your hit rate is zero no matter how much of the prompt is\n\ntechnically stable.\n\nBefore claiming any of this was novel I looked at what the neighbors ship: LM\n\nStudio, AnythingLLM, Jan, GPT4All, Open WebUI with Ollama.\n\nAll of them chunk and embed at index time. Not one of them precomputes\n\nper-document digests or summaries. The best of them keep a prompt cache, which\n\nhelps with the system prompt and does nothing for the retrieved half.\n\nWhich means that on \"what are these documents about?\" — the single most common\n\nopening question a human asks a document app, the one they type before they\n\ntype anything else — every one of these re-reads the corpus at full prefill\n\ncost, every time.\n\nI'm supposed to call that a gap in the market. It's really a gap in the default\n\narchitecture: the reference RAG design does retrieval at question time and\n\nnothing at index time except embeddings, everybody copied it faithfully, and\n\nthe copy is correct. It's just that \"correct\" and \"the user waited five\n\nminutes\" are compatible states.\n\n**Measure prefill against generation before you optimize anything.** If 85% of\n\nyour wall clock is reading, a faster sampler and a smaller quant are noise. The\n\nengine prints both numbers; find the line.\n\n**Deduplicate context sources by content, not by id.** Two retrievers with\n\ndifferent chunk boundaries will hand you the same paragraph twice and neither\n\nwill look wrong in isolation. This was 9,000 tokens a question in my app.\n\n**Give service LLM calls their own KV slot.** Query rewriting, classification,\n\ntitle generation — anything with its own system prompt sharing a slot with the\n\nconversation will zero your prefix cache and the logs will still say the cache\n\nis enabled.\n\n**Precompute passports and summaries at index time.** A ~200-token structured\n\npassport per document, with headings pulled by regex rather than by the model,\n\nanswers most broad questions on its own and costs no GPU at all.\n\n**Put the study queue in the database.** In-memory queues on a desktop app do\n\nnot survive contact with users, who close windows. Checkpoint per chunk, resume\n\non launch, yield to live traffic, stop on battery.\n\n**Cache the literal repeats.** People re-ask the same question verbatim more\n\nthan you'd think. Hash the normalized question plus the resolved context set.\n\nIf you want to know whether any of this is worth your afternoon, the arithmetic\n\nis short enough to run before you commit to it: file size, prefill speed,\n\nnumber of questions, and it tells you how many seconds of pure reading you have\n\nalready signed up for. That's `how_long_will_my_rag_wait.py`\n\n— [https://github.com/JackYU96/rag-rereads-every-question](https://github.com/JackYU96/rag-rereads-every-question) — in the repo next\n\nto this post — no dependencies, one file, bring your own numbers.\n\nThe last time I chased a number like this, the KV cache itself turned out to be\n\neating it: [\"V cache quantization requires flash_attn\" — the llama.cpp error\nthat quietly halves your context\nwindow](https://dev.to/dreamdeck/v-cache-quantization-requires-flashattn-the-llamacpp-error-that-quietly-halves-your-context-1kdb).", "url": "https://wpnews.pro/news/your-local-rag-isn-t-slow-it-re-reads-every-document-on-every-question", "canonical_source": "https://dev.to/dreamdeck/your-local-rag-isnt-slow-it-re-reads-every-document-on-every-question-18jg", "published_at": "2026-08-26 10:18:51+00:00", "updated_at": "2026-08-26 10:43:52.671926+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": ["llama.cpp", "M4 Pro"], "alternates": {"html": "https://wpnews.pro/news/your-local-rag-isn-t-slow-it-re-reads-every-document-on-every-question", "markdown": "https://wpnews.pro/news/your-local-rag-isn-t-slow-it-re-reads-every-document-on-every-question.md", "text": "https://wpnews.pro/news/your-local-rag-isn-t-slow-it-re-reads-every-document-on-every-question.txt", "jsonld": "https://wpnews.pro/news/your-local-rag-isn-t-slow-it-re-reads-every-document-on-every-question.jsonld"}}