{"slug": "my-local-ai-assistant-got-worse-when-i-remembered-too-much", "title": "My Local AI Assistant Got Worse When I Remembered Too Much", "summary": "A developer found that their local AI assistant running Qwen3-4B-4bit degraded after 196 messages due to raw conversation history replay. The fix was splitting memory into short-term RAM-only session history with a 200-message cap and a small list of up to 30 distilled facts, using a hosted model for fact extraction. The developer also added a tool bias override to force live data lookups.", "body_md": "I moved a personal AI assistant onto a small local model last week and immediately hit a boring problem: the model was fine, but my memory layer was not.\n\nThe old version persisted raw conversation history and replayed it back into the prompt. That worked well enough with hosted models. Then I pointed the same app at a local OpenAI-compatible server running Qwen3-4B-4bit through Swama on the Mac mini.\n\nAfter 196 accumulated messages, the assistant started doing the classic small-model failure mode: parroting its own previous replies, over-weighting stale context, and sounding less useful the more “memory” I gave it.\n\nThe fix was not a vector database. It was deleting most of the memory.\n\nI split memory into two different things:\n\nShort-term history now stays in RAM only. It resets after an idle gap, and it has a hard cap so a marathon session cannot poison every future turn.\n\n```\nself.session_memory: Dict[str, ConversationBufferMemory] = {}\nself._last_activity: Dict[str, float] = {}\n\nidle_limit = int(os.getenv(\"SESSION_IDLE_MINUTES\", \"120\")) * 60\nlast = self._last_activity.get(user_id)\n\nif last is not None and now - last > idle_limit:\n    del self.session_memory[user_id]\n```\n\nLong-term memory is not chat logs. It is a small list of distilled facts: preferences, people, devices, recurring activities, that kind of thing. Maximum 30 facts per user.\n\n```\n_FACT_EXTRACTION_PROMPT = \"\"\"\nUpdate the fact list. Add only stable, personal facts worth remembering across\nconversations: preferences, interests, people, pets, places, devices, recurring\nactivities. Ignore small talk, one-off requests, and anything the assistant said\nabout itself.\n\nReturn ONLY a JSON array of strings.\n\"\"\"\n```\n\nThe fact extraction runs in a background thread after each exchange. The chat path should not wait for memory housekeeping.\n\n```\nthreading.Thread(\n    target=self._extract_facts,\n    args=(user_id, msgs[-2].content, msgs[-1].content),\n    daemon=True,\n).start()\n```\n\nI also deliberately use a hosted model for the distillation step. The local 4B model is good enough for fast interaction, but long-term memory cleanup is one of those places where quality matters more than latency. It is off the response path anyway.\n\nThe other local-model tweak was tool bias. Small models are much more likely to answer from stale weights even when tools exist, especially if the system prompt says anything like “use your knowledge first.” So the Swama handler adds a blunt override for live data:\n\n```\n_TOOL_BIAS = (\n    \" IMPORTANT OVERRIDE: for anything happening NOW - weather, sea or\"\n    \" kitesurfing conditions, device/home status, prices, news, live data\"\n    \" of any kind - you MUST call the matching tool. Never answer those\"\n    \" from memory. /no_think\"\n)\n```\n\nQwen3 also emits `<think>`\n\nblocks even when asked not to, including mid-stream after tool calls, so the streaming handler strips those tags incrementally. Not glamorous, but necessary if you do not want raw reasoning markup leaking into a voice/chat UI.\n\nThe useful lesson was this:\n\nMemory is not “more previous tokens.”\n\nFor a personal assistant, raw transcript replay is the cheapest thing to build and one of the easiest ways to make the system worse. The assistant needs enough recent context to hold the current conversation, plus a tiny set of stable facts that survive across sessions.\n\nEverything else is prompt pollution with a better name.\n\n**Source:** Recent personal assistant backend work: Swama local model support, Qwen3-4B-4bit via an OpenAI-compatible endpoint, RAM-only session history, 2-hour idle reset, 200-message cap, background fact extraction, and 30 persisted user facts.\n\n**Tags:** ai, python, llm, devops\n\n**Status:** published", "url": "https://wpnews.pro/news/my-local-ai-assistant-got-worse-when-i-remembered-too-much", "canonical_source": "https://dev.to/toddsullivan/my-local-ai-assistant-got-worse-when-i-remembered-too-much-3egp", "published_at": "2026-07-21 08:06:35+00:00", "updated_at": "2026-07-21 08:30:43.312728+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["Qwen3", "Swama", "Mac mini"], "alternates": {"html": "https://wpnews.pro/news/my-local-ai-assistant-got-worse-when-i-remembered-too-much", "markdown": "https://wpnews.pro/news/my-local-ai-assistant-got-worse-when-i-remembered-too-much.md", "text": "https://wpnews.pro/news/my-local-ai-assistant-got-worse-when-i-remembered-too-much.txt", "jsonld": "https://wpnews.pro/news/my-local-ai-assistant-got-worse-when-i-remembered-too-much.jsonld"}}