I moved a personal AI assistant onto a small local model last week and immediately hit a boring problem: the model was fine, but my memory layer was not.
The old version persisted raw conversation history and replayed it back into the prompt. That worked well enough with hosted models. Then I pointed the same app at a local OpenAI-compatible server running Qwen3-4B-4bit through Swama on the Mac mini.
After 196 accumulated messages, the assistant started doing the classic small-model failure mode: parroting its own previous replies, over-weighting stale context, and sounding less useful the more “memory” I gave it.
The fix was not a vector database. It was deleting most of the memory.
I split memory into two different things:
Short-term history now stays in RAM only. It resets after an idle gap, and it has a hard cap so a marathon session cannot poison every future turn.
self.session_memory: Dict[str, ConversationBufferMemory] = {}
self._last_activity: Dict[str, float] = {}
idle_limit = int(os.getenv("SESSION_IDLE_MINUTES", "120")) * 60
last = self._last_activity.get(user_id)
if last is not None and now - last > idle_limit:
del self.session_memory[user_id]
Long-term memory is not chat logs. It is a small list of distilled facts: preferences, people, devices, recurring activities, that kind of thing. Maximum 30 facts per user.
_FACT_EXTRACTION_PROMPT = """
Update the fact list. Add only stable, personal facts worth remembering across
conversations: preferences, interests, people, pets, places, devices, recurring
activities. Ignore small talk, one-off requests, and anything the assistant said
about itself.
Return ONLY a JSON array of strings.
"""
The fact extraction runs in a background thread after each exchange. The chat path should not wait for memory housekeeping.
threading.Thread(
target=self._extract_facts,
args=(user_id, msgs[-2].content, msgs[-1].content),
daemon=True,
).start()
I also deliberately use a hosted model for the distillation step. The local 4B model is good enough for fast interaction, but long-term memory cleanup is one of those places where quality matters more than latency. It is off the response path anyway.
The other local-model tweak was tool bias. Small models are much more likely to answer from stale weights even when tools exist, especially if the system prompt says anything like “use your knowledge first.” So the Swama handler adds a blunt override for live data:
_TOOL_BIAS = (
" IMPORTANT OVERRIDE: for anything happening NOW - weather, sea or"
" kitesurfing conditions, device/home status, prices, news, live data"
" of any kind - you MUST call the matching tool. Never answer those"
" from memory. /no_think"
)
Qwen3 also emits <think>
blocks even when asked not to, including mid-stream after tool calls, so the streaming handler strips those tags incrementally. Not glamorous, but necessary if you do not want raw reasoning markup leaking into a voice/chat UI.
The useful lesson was this:
Memory is not “more previous tokens.”
For a personal assistant, raw transcript replay is the cheapest thing to build and one of the easiest ways to make the system worse. The assistant needs enough recent context to hold the current conversation, plus a tiny set of stable facts that survive across sessions.
Everything else is prompt pollution with a better name.
Source: Recent personal assistant backend work: Swama local model support, Qwen3-4B-4bit via an OpenAI-compatible endpoint, RAM-only session history, 2-hour idle reset, 200-message cap, background fact extraction, and 30 persisted user facts.
Tags: ai, python, llm, devops
Status: published