{"slug": "i-tried-building-llm-memory-the-stupid-way-it-worked-surprisingly-well", "title": "I Tried Building LLM Memory the Stupid Way — It Worked Surprisingly Well", "summary": "A developer experimenting with local RAG systems found that a simple line-length filter—keeping only lines over 10 words—can effectively preserve conversational memory for LLMs. In tests with 13-14 specific questions about past chats, the system recovered almost all details, including exact amounts, dates, and cross-conversation connections, despite retrieving only eight chunks. The approach failed on short code snippets, but the developer considers it a useful context builder rather than a production-grade memory system.", "body_md": "This started as a fun experiment.\n\nI've been working on a local RAG system , and at some point I started wondering about something that sounds simple but is actually quite difficult:\n\nCan an LLM remember old conversations without sending its entire history back into the context window?\n\nI didn't want to build a complicated memory architecture just to answer that question. So I tried something almost ridiculous. I exported my old chats and extracted conversations from Json files.\n\nThe entire memory extractor was basically one rule\n\nIf a line had more than 10 words, keep it.\n\nIf it had 10 or fewer, throw it away.\n\nThat's pretty much it.\n\nNo LLM summarization during ingestion. No expensive memory model. No elaborate memory classification.\n\nThe resulting text was then chunked and fed into my existing RAG. For every question, I allowed retrieval of only 8 chunks.\n\nSo the experiment looked roughly like this:\n\nOld conversations\n\n↓\n\nKeep lines with >10 words\n\n↓\n\nCreate chunks\n\n↓\n\nBM25 + semantic retrieval\n\n↓\n\nTop 8 chunks\n\n↓\n\nFeed back to same LLM\n\nThen I decided to actually try to break it.\n\nI used my own old conversations\n\nInstead of asking generic questions, I went back to conversations from the extracted chats and asked the LLM very specific questions.\n\nThings like old procurement discussions, financial figures, dates, coding decisions, project discussions and relationships between completely separate conversations.\n\nI asked roughly 13–14 questions.\n\nLLM failed.\n\nThen I fed the chunks from my RAG\n\nAnd honestly, I expected it to fail much more often.\n\nIt didn't.\n\nIt recovered almost everything I asked about.\n\nIt could retrieve exact amounts and dates. It could recover old technical discussions. In some cases, it could connect information from separate conversations and reconstruct what had happened.\n\nAnd remember:\n\nThe model only received eight retrieved chunks.\n\nIt wasn't given my entire historical conversation corpus.\n\nThen I found the obvious weakness\n\nThe one clear failure was actually quite revealing.\n\nIt was terrible at remembering code and commands may be since they are often less than 10 words. So my brilliant memory algorithm simply throws them away. But I tried to make it as a context builder rather than a storage. So I am satisfied with results.\n\nThe LLM was restricted to use internet or think only to answer from its memory. I think if that was allowed that single miss would haven’t occurred. Its not a production grade memory management system. But it works gr8.", "url": "https://wpnews.pro/news/i-tried-building-llm-memory-the-stupid-way-it-worked-surprisingly-well", "canonical_source": "https://dev.to/rajesh_kumar_ea50a6f5ab02/i-tried-building-llm-memory-the-stupid-way-it-worked-surprisingly-well-4n8j", "published_at": "2026-08-10 13:56:22+00:00", "updated_at": "2026-08-10 14:16:43.428428+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/i-tried-building-llm-memory-the-stupid-way-it-worked-surprisingly-well", "markdown": "https://wpnews.pro/news/i-tried-building-llm-memory-the-stupid-way-it-worked-surprisingly-well.md", "text": "https://wpnews.pro/news/i-tried-building-llm-memory-the-stupid-way-it-worked-surprisingly-well.txt", "jsonld": "https://wpnews.pro/news/i-tried-building-llm-memory-the-stupid-way-it-worked-surprisingly-well.jsonld"}}