{"slug": "architecting-lean-llm-caching-how-to-drop-a-20m-row-table-without-losing-your-ai", "title": "Architecting lean LLM caching: how to drop a 20M-row table without losing your AI memory", "summary": "A developer architecting a lean LLM caching system for agentic pipelines discovered that storing cache in a small, orthogonal lookup table keyed on a stable identifier allows dropping a 20-million-row table without losing AI memory. The approach reduces a 16 GB table to a few MB, enabling safe data wiping and reloading in storage-constrained environments. The developer also highlights a performance gotcha with indexing and recommends using CTAS over in-place UPDATE to avoid MVCC bloat.", "body_md": "Any team running an agentic pipeline on a periodically-reloaded dataset caches the output of inputs that repeat across periods to save cost. How you store that cache matters. Storing it in the same table as the data is safe, but makes you unable to wipe the data from the previous period before loading the new one. Keeping both on disk to use the cache can work, but if you're in a storage-constrained environment, you can't keep both on disk and have to find a way to remove the old one before loading the new one while still keeping the cache. The fix is to move the cache into a small, orthogonal lookup table keyed on a stable identifier, so it survives the wipe AND takes almost no disk.\n\nThis article walks through that pattern: the discovery that made it possible, the two-table shape, and the one operational gotcha you'll hit if you don't watch for it.\n\nThe move to a lookup table starts with noticing something about your data.\n\nA pipeline can process 20 million rows every period and only pass 60,000 *distinct* inputs to the LLM. The other 19.94 million rows are duplicates of those 60k values, spread across various dates, event types, geographies. Same key, different context.\n\nOnce you see that number, the design writes itself. You don't need to store 20 million enrichment results. You need to store 60,000. The 20 million rows can each *reference* one of those 60k answers.\n\nConcretely:\n\n`data`\n\n)`enrichment_cache`\n\n)The compression ratio is doing the work here. A 16 GB table you can't afford to keep two copies of becomes a few-MB table you can afford to keep *forever*. That's what makes wiping the working table safe: the cache is the durable output of every past LLM call, and it's tiny.\n\nWhen the next data arrives, the pipeline runs this sequence:\n\n`data`\n\ntable via whatever ingest mechanism you use.`data`\n\nagainst `enrichment_cache`\n\non the input key, copying every already-known enrichment onto the new rows. `data`\n\nfor rows still missing enrichment. Those are the genuinely-new inputs. Every LLM response gets written to the row in `data`\n\nAND appended to `enrichment_cache`\n\nso the input never needs another LLM call.The SQL for step 3 is the shortest and most important:\n\n```\nUPDATE data d\nSET    enrichment = c.enrichment\nFROM   enrichment_cache c\nWHERE  d.input_key = c.input_key\n  AND  d.enrichment IS NULL;\n```\n\nWhile this UPDATE syntax works perfectly on smaller datasets, running a bulk in-place update across 20 million rows in production triggers a massive MVCC performance penalty. Postgres creates millions of dead tuples, causing table bloat and heavy disk I/O serialization. To bypass this bottleneck, we eventually migrated this step from an in-place UPDATE to a CTAS (Create Table As Select) and swap architecture during the initial ingestion phase. Part 3 of this series breaks down the mechanics of that zero-bloat swap pattern.\n\n**Why keep the cache in Postgres rather than an in-memory store like Redis?**\n\nAny relational database already gives you durable, transactional, indexed key-value storage for free. Adding a separate in-memory cache is another moving part in the deploy, another failure mode to reason about, and another operational surface (memory limits, eviction, replication). If the cache fits comfortably in a relational table and the lookup is an exact match on a stable key, keep infrastructure count low unless a hard constraint forces you to add pieces.\n\n**The indexing gotcha.**\n\nThe cache table is small, so it's tempting to think the join will be trivially fast. It won't be, not at 20M rows on the working side. Without an index on `enrichment_cache.input_key`\n\n, the join has to hash the entire cache table for each batch. That's fine on 60k rows once, but Postgres's planner may still pick a plan that costs you more than you expect at scale, and the equivalent per-row lookup inside step 4 (`SELECT ... WHERE input_key = %s`\n\n) will be a sequential scan every time.\n\nFix once, forever:\n\n```\nCREATE UNIQUE INDEX enrichment_cache_key_idx ON enrichment_cache (input_key);\n```\n\nA B-tree index on the lookup column turns every cache read into a sub-millisecond index lookup and lets the planner use an index join for the bulk propagate in step 3. This is the one line that separates \"cache works in principle\" from \"cache is fast enough that you notice the pipeline is done.\"\n\n*The cheapest LLM call is the one you don't make. The cheapest byte of disk is the one you don't store.*\n\nMove the durable, expensive knowledge (LLM answers) into a small, orthogonal lookup table. Let the big working table remain a disposable window on the current period. The wipe becomes safe, the cache stays warm, and the bill collapses.", "url": "https://wpnews.pro/news/architecting-lean-llm-caching-how-to-drop-a-20m-row-table-without-losing-your-ai", "canonical_source": "https://dev.to/wondadav/architecting-lean-llm-caching-how-to-drop-a-20m-row-table-without-losing-your-ai-memory-3g2n", "published_at": "2026-07-18 13:28:25+00:00", "updated_at": "2026-07-18 13:58:50.593063+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Postgres", "Redis"], "alternates": {"html": "https://wpnews.pro/news/architecting-lean-llm-caching-how-to-drop-a-20m-row-table-without-losing-your-ai", "markdown": "https://wpnews.pro/news/architecting-lean-llm-caching-how-to-drop-a-20m-row-table-without-losing-your-ai.md", "text": "https://wpnews.pro/news/architecting-lean-llm-caching-how-to-drop-a-20m-row-table-without-losing-your-ai.txt", "jsonld": "https://wpnews.pro/news/architecting-lean-llm-caching-how-to-drop-a-20m-row-table-without-losing-your-ai.jsonld"}}