My Local AI Assistant Got Worse When I Remembered Too Much A developer found that their local AI assistant running Qwen3-4B-4bit degraded after 196 messages due to raw conversation history replay. The fix was splitting memory into short-term RAM-only session history with a 200-message cap and a small list of up to 30 distilled facts, using a hosted model for fact extraction. The developer also added a tool bias override to force live data lookups. I moved a personal AI assistant onto a small local model last week and immediately hit a boring problem: the model was fine, but my memory layer was not. The old version persisted raw conversation history and replayed it back into the prompt. That worked well enough with hosted models. Then I pointed the same app at a local OpenAI-compatible server running Qwen3-4B-4bit through Swama on the Mac mini. After 196 accumulated messages, the assistant started doing the classic small-model failure mode: parroting its own previous replies, over-weighting stale context, and sounding less useful the more “memory” I gave it. The fix was not a vector database. It was deleting most of the memory. I split memory into two different things: Short-term history now stays in RAM only. It resets after an idle gap, and it has a hard cap so a marathon session cannot poison every future turn. self.session memory: Dict str, ConversationBufferMemory = {} self. last activity: Dict str, float = {} idle limit = int os.getenv "SESSION IDLE MINUTES", "120" 60 last = self. last activity.get user id if last is not None and now - last idle limit: del self.session memory user id Long-term memory is not chat logs. It is a small list of distilled facts: preferences, people, devices, recurring activities, that kind of thing. Maximum 30 facts per user. FACT EXTRACTION PROMPT = """ Update the fact list. Add only stable, personal facts worth remembering across conversations: preferences, interests, people, pets, places, devices, recurring activities. Ignore small talk, one-off requests, and anything the assistant said about itself. Return ONLY a JSON array of strings. """ The fact extraction runs in a background thread after each exchange. The chat path should not wait for memory housekeeping. threading.Thread target=self. extract facts, args= user id, msgs -2 .content, msgs -1 .content , daemon=True, .start I also deliberately use a hosted model for the distillation step. The local 4B model is good enough for fast interaction, but long-term memory cleanup is one of those places where quality matters more than latency. It is off the response path anyway. The other local-model tweak was tool bias. Small models are much more likely to answer from stale weights even when tools exist, especially if the system prompt says anything like “use your knowledge first.” So the Swama handler adds a blunt override for live data: TOOL BIAS = " IMPORTANT OVERRIDE: for anything happening NOW - weather, sea or" " kitesurfing conditions, device/home status, prices, news, live data" " of any kind - you MUST call the matching tool. Never answer those" " from memory. /no think" Qwen3 also emits