If you’ve been following my recent experiments, you know I’m a huge advocate for running open-weight models locally. But one of the biggest friction points with local LLMs is the context window. Even with a massive 256k context limit, you can't, and shouldn't, dump your entire history into every prompt. It bloats VRAM and destroys inference speeds.
The solution is a Personalized Long-Term Memory RAG (Retrieval-Augmented Generation). Today, I'm going to walk you through exactly how I built an autonomous, privacy-focused memory layer that seamlessly injects context into my chats using Mem0, Qdrant, and Open WebUI, completely bypassing the need for clunky UI document uploads.
The Hardware & OS
My current daily driver for this setup is a modular mini PC from Framework running Fedora 44.
Here are the specs of my machine:
- Ryzen™ AI Max+ 395 - 128GB (I've allocated 118GB of this to VRAM for hosting models)
- Storage: WD_BLACK™ SN7100 NVMe™ - M.2 2280 - 4TB
For local inference, I'm relying entirely on an integrated AMD GPU. Off a heavy model like qwen3-coder-next
pushes the APU to its limits and I regularly see GPU utilization pinned at 100% with VRAM usage hovering right around 40-50GB. Despite being integrated graphics, this configuration handles the RAG pipeline flawlessly without breaking a sweat.
We will be using Podman instead of Docker, keeping things native to the Fedora ecosystem.
Step 1: Spin Up the Vector Database (Qdrant)
Qdrant acts as the limitless hard drive for our AI's memories. It stores the mathematical vectors and text payloads on disk so they persist across reboots.
Run this Podman command to spin up Qdrant using host networking:
Bash
podman run -d \
--name qdrant \
--network=host \
-v ~/.local/share/qdrant_data:/qdrant/storage:z \
--restart always \
qdrant/qdrant:latest
Step 2: Prepare the Ollama Models
We need two models running in Ollama:
The Embedder: Translates text into 768-dimensional vectors.The Chat Model: The brains of the operation.
Bash
ollama pull nomic-embed-text:latest
ollama pull qwen3-coder-next:q4_K_M
(Tip: Make sure the Ollama daemon is running in the background via systemctl status ollama).
Step 3: The Magic Middleware (Mem0 Proxy)
Instead of hacking memory directly into Open WebUI, we are deploying a lightweight FastAPI proxy. This sits between your UI and Ollama. When you send a message, the proxy intercepts it, searches Qdrant for relevant past facts, injects them into the system prompt, and forwards it to Qwen. This is the RAG step in action!
Crucially, it also features a background task that silently extracts new facts from your conversation and saves them to Qdrant without adding any latency to your chat response.
Create a file at ~/memory_proxy.py
(or wherever you wish) and paste this in:
Python
import os
import uvicorn
import asyncio
from fastapi import FastAPI, Request, BackgroundTasks
from fastapi.responses import StreamingResponse
import httpx
from mem0 import Memory
os.environ["MEM0_TELEMETRY"] = "False"
app = FastAPI(title="Mem0 Local Proxy with Auto-Extract")
mem0_config = {
"vector_store": {
"provider": "qdrant",
"config": {
"collection_name": "lead_prompt_memories",
"host": "localhost",
"port": 6333,
"embedding_model_dims": 768,
}
},
"llm": {
"provider": "ollama",
"config": {
"model": "qwen3-coder-next:q4_K_M",
"temperature": 0.1,
"max_tokens": 4096,
"ollama_base_url": "http://127.0.0.1:11434",
}
},
"embedder": {
"provider": "ollama",
"config": {
"model": "nomic-embed-text:latest",
"ollama_base_url": "http://127.0.0.1:11434",
}
}
}
memory = Memory.from_config(mem0_config)
OLLAMA_BASE_URL = "http://127.0.0.1:11434"
def extract_and_save_memory(messages: list, user_id: str):
try:
memory.add(messages, user_id=user_id)
print(f"\n[MEM0 BACKGROUND]: Memory extraction complete for user '{user_id}'.")
except Exception as e:
print(f"\n[MEM0 BACKGROUND ERROR]: Failed to extract memory: {e}")
@app.get("/v1/models")
async def list_models():
async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.get(f"{OLLAMA_BASE_URL}/v1/models")
return resp.json()
@app.post("/v1/chat/completions")
async def chat_completions(request: Request, background_tasks: BackgroundTasks):
body = await request.json()
messages = body.get("messages", [])
user_id = body.get("user", "john")
user_query = ""
for msg in reversed(messages):
if msg.get("role") == "user":
user_query = msg.get("content", "")
break
memories_str = ""
if user_query:
try:
results = memory.search(query=user_query, filters={"user_id": user_id})
mem_list = [r["memory"] for r in results.get("results", [])]
if mem_list:
memories_str = "\n".join(f"- {m}" for m in mem_list)
except Exception as e:
print(f"[Proxy Memory Error]: {e}")
if memories_str:
context_prompt = (
f"\n\n[Relevant User Context & Saved Preferences]:\n{memories_str}\n"
)
if messages and messages[0].get("role") == "system":
messages[0]["content"] += context_prompt
else:
messages.insert(0, {"role": "system", "content": f"You are a helpful assistant.{context_prompt}"})
body["messages"] = messages
if user_query:
background_tasks.add_task(extract_and_save_memory, messages[-2:], user_id)
async def stream_generator():
async with httpx.AsyncClient(timeout=300.0) as client:
async with client.stream(
"POST", f"{OLLAMA_BASE_URL}/v1/chat/completions", json=body, headers={"Content-Type": "application/json"}
) as resp:
async for chunk in resp.aiter_bytes():
yield chunk
if body.get("stream", False):
return StreamingResponse(stream_generator(), media_type="text/event-stream")
else:
async with httpx.AsyncClient(timeout=300.0) as client:
resp = await client.post(f"{OLLAMA_BASE_URL}/v1/chat/completions", json=body)
return resp.json()
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=11435)
Step 4: Daemonize the Proxy with Systemd
We want this running persistently without tying up a terminal window.
Create a systemd user service:
nano ~/.config/systemd/user/memory-proxy.service
Ini, TOML
[Unit]
Description=Mem0 OpenAI Local Proxy Service
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/python3 /home/john/memory_proxy.py
WorkingDirectory=/home/john
Restart=always
RestartSec=3
Environment=MEM0_TELEMETRY=False
[Install]
WantedBy=default.target
Enable and start it:
Bash
systemctl --user daemon-reload
systemctl --user enable --now memory-proxy.service
Step 5: Open WebUI Configuration
First, spin up the WebUI container (again, native host networking is best here):
Bash
podman run -d \
--name open-webui \
--network=host \
-v open-webui:/app/backend/data:z \
--restart always \
ghcr.io/open-webui/open-webui:main
Now, wire the UI to our memory proxy rather than directly to Ollama:
- Navigate to
http://localhost:3000
and go toAdmin Panel -> Settings -> Connections. Disable the standard Ollama API connection.Enable the OpenAI API connection and set the URL to our proxy:http://host.containers.internal:11435/v1
(with any dummy text for the API key).- Save the configuration.
Crucial Final Step: Open WebUI has its own internal RAG and web search mechanisms that will try to hijack queries before they reach our proxy. To prevent this:
- Go to
Workspace -> Models and edit your
Qwen3 Coder
preset. - Scroll down to the capabilities/tools section and manually disable Knowledge/RAG and Web Search.
The Result
You now have a model running locally that remembers. Because the memory proxy is exposed via standard OpenAI API format, you aren't locked into Open WebUI either: you can point Zed, terminal curl scripts, or any other LLM front-end to localhost:11435/v1
and take your personalized vector brain with you. Secondly, you are also not tied into qwen3-coder-next and can use any local model you wish, and even share context between models!