Building an Infinite Memory Local AI Stack on Fedora

A developer built a personalized long-term memory RAG stack on Fedora 44 using Mem0, Qdrant, and Open WebUI, with a FastAPI proxy that injects relevant past facts into chat prompts and auto-extracts new memories in the background. The setup runs on a Framework mini PC with a Ryzen AI Max+ 395 APU, allocating 118GB of 128GB RAM to VRAM, and uses Podman for containerization. The system bypasses manual document uploads, enabling autonomous, privacy-focused memory for local LLMs.

If you’ve been following my recent experiments, you know I’m a huge advocate for running open-weight models locally. But one of the biggest friction points with local LLMs is the context window. Even with a massive 256k context limit, you can't, and shouldn't, dump your entire history into every prompt. It bloats VRAM and destroys inference speeds. The solution is a Personalized Long-Term Memory RAG Retrieval-Augmented Generation . Today, I'm going to walk you through exactly how I built an autonomous, privacy-focused memory layer that seamlessly injects context into my chats using Mem0 , Qdrant , and Open WebUI , completely bypassing the need for clunky UI document uploads. The Hardware & OS My current daily driver for this setup is a modular mini PC from Framework running Fedora 44 . Here are the specs of my machine: - Ryzen™ AI Max+ 395 - 128GB I've allocated 118GB of this to VRAM for hosting models - Storage: WD BLACK™ SN7100 NVMe™ - M.2 2280 - 4TB For local inference, I'm relying entirely on an integrated AMD GPU . Offloading a heavy model like qwen3-coder-next pushes the APU to its limits and I regularly see GPU utilization pinned at 100% with VRAM usage hovering right around 40-50GB . Despite being integrated graphics, this configuration handles the RAG pipeline flawlessly without breaking a sweat. We will be using Podman instead of Docker, keeping things native to the Fedora ecosystem. Step 1: Spin Up the Vector Database Qdrant Qdrant acts as the limitless hard drive for our AI's memories. It stores the mathematical vectors and text payloads on disk so they persist across reboots. Run this Podman command to spin up Qdrant using host networking: Bash podman run -d \ --name qdrant \ --network=host \ -v ~/.local/share/qdrant data:/qdrant/storage:z \ --restart always \ qdrant/qdrant:latest Step 2: Prepare the Ollama Models We need two models running in Ollama: The Embedder: Translates text into 768-dimensional vectors. The Chat Model: The brains of the operation. Bash ollama pull nomic-embed-text:latest ollama pull qwen3-coder-next:q4 K M Tip: Make sure the Ollama daemon is running in the background via systemctl status ollama . Step 3: The Magic Middleware Mem0 Proxy Instead of hacking memory directly into Open WebUI, we are deploying a lightweight FastAPI proxy. This sits between your UI and Ollama. When you send a message, the proxy intercepts it, searches Qdrant for relevant past facts, injects them into the system prompt, and forwards it to Qwen. This is the RAG step in action Crucially, it also features a background task that silently extracts new facts from your conversation and saves them to Qdrant without adding any latency to your chat response. Create a file at ~/memory proxy.py or wherever you wish and paste this in: Python python import os import uvicorn import asyncio from fastapi import FastAPI, Request, BackgroundTasks from fastapi.responses import StreamingResponse import httpx from mem0 import Memory os.environ "MEM0 TELEMETRY" = "False" app = FastAPI title="Mem0 Local Proxy with Auto-Extract" Configure Mem0 mem0 config = { "vector store": { "provider": "qdrant", "config": { "collection name": "lead prompt memories", "host": "localhost", "port": 6333, "embedding model dims": 768, } }, "llm": { "provider": "ollama", "config": { "model": "qwen3-coder-next:q4 K M", "temperature": 0.1, "max tokens": 4096, "ollama base url": "http://127.0.0.1:11434", } }, "embedder": { "provider": "ollama", "config": { "model": "nomic-embed-text:latest", "ollama base url": "http://127.0.0.1:11434", } } } memory = Memory.from config mem0 config OLLAMA BASE URL = "http://127.0.0.1:11434" Background extraction logic def extract and save memory messages: list, user id: str : try: memory.add messages, user id=user id print f"\n MEM0 BACKGROUND : Memory extraction complete for user '{user id}'." except Exception as e: print f"\n MEM0 BACKGROUND ERROR : Failed to extract memory: {e}" @app.get "/v1/models" async def list models : async with httpx.AsyncClient timeout=30.0 as client: resp = await client.get f"{OLLAMA BASE URL}/v1/models" return resp.json @app.post "/v1/chat/completions" async def chat completions request: Request, background tasks: BackgroundTasks : body = await request.json messages = body.get "messages", user id = body.get "user", "john" user query = "" for msg in reversed messages : if msg.get "role" == "user": user query = msg.get "content", "" break 1. READ: Retrieve existing memories memories str = "" if user query: try: results = memory.search query=user query, filters={"user id": user id} mem list = r "memory" for r in results.get "results", if mem list: memories str = "\n".join f"- {m}" for m in mem list except Exception as e: print f" Proxy Memory Error : {e}" 2. INJECT: Add retrieved context to system prompt if memories str: context prompt = f"\n\n Relevant User Context & Saved Preferences :\n{memories str}\n" if messages and messages 0 .get "role" == "system": messages 0 "content" += context prompt else: messages.insert 0, {"role": "system", "content": f"You are a helpful assistant.{context prompt}"} body "messages" = messages 3. WRITE: Schedule background extraction if user query: background tasks.add task extract and save memory, messages -2: , user id 4. STREAM back to client async def stream generator : async with httpx.AsyncClient timeout=300.0 as client: async with client.stream "POST", f"{OLLAMA BASE URL}/v1/chat/completions", json=body, headers={"Content-Type": "application/json"} as resp: async for chunk in resp.aiter bytes : yield chunk if body.get "stream", False : return StreamingResponse stream generator , media type="text/event-stream" else: async with httpx.AsyncClient timeout=300.0 as client: resp = await client.post f"{OLLAMA BASE URL}/v1/chat/completions", json=body return resp.json if name == " main ": uvicorn.run app, host="0.0.0.0", port=11435 Step 4: Daemonize the Proxy with Systemd We want this running persistently without tying up a terminal window. Create a systemd user service: nano ~/.config/systemd/user/memory-proxy.service Ini, TOML Unit Description=Mem0 OpenAI Local Proxy Service After=network.target Service Type=simple ExecStart=/usr/bin/python3 /home/john/memory proxy.py WorkingDirectory=/home/john Restart=always RestartSec=3 Environment=MEM0 TELEMETRY=False Install WantedBy=default.target Enable and start it: Bash systemctl --user daemon-reload systemctl --user enable --now memory-proxy.service Step 5: Open WebUI Configuration First, spin up the WebUI container again, native host networking is best here : Bash podman run -d \ --name open-webui \ --network=host \ -v open-webui:/app/backend/data:z \ --restart always \ ghcr.io/open-webui/open-webui:main Now, wire the UI to our memory proxy rather than directly to Ollama: - Navigate to http://localhost:3000 and go to Admin Panel - Settings - Connections . Disable the standard Ollama API connection. Enable the OpenAI API connection and set the URL to our proxy: http://host.containers.internal:11435/v1 with any dummy text for the API key .- Save the configuration. Crucial Final Step: Open WebUI has its own internal RAG and web search mechanisms that will try to hijack queries before they reach our proxy. To prevent this: - Go to Workspace - Models and edit your Qwen3 Coder preset. - Scroll down to the capabilities/tools section and manually disable Knowledge/RAG and Web Search . The Result You now have a model running locally that remembers . Because the memory proxy is exposed via standard OpenAI API format, you aren't locked into Open WebUI either: you can point Zed, terminal curl scripts, or any other LLM front-end to localhost:11435/v1 and take your personalized vector brain with you. Secondly, you are also not tied into qwen3-coder-next and can use any local model you wish, and even share context between models