{"slug": "building-an-infinite-memory-local-ai-stack-on-fedora", "title": "Building an Infinite Memory Local AI Stack on Fedora", "summary": "A developer built a personalized long-term memory RAG stack on Fedora 44 using Mem0, Qdrant, and Open WebUI, with a FastAPI proxy that injects relevant past facts into chat prompts and auto-extracts new memories in the background. The setup runs on a Framework mini PC with a Ryzen AI Max+ 395 APU, allocating 118GB of 128GB RAM to VRAM, and uses Podman for containerization. The system bypasses manual document uploads, enabling autonomous, privacy-focused memory for local LLMs.", "body_md": "If you’ve been following my recent experiments, you know I’m a huge advocate for running open-weight models locally. But one of the biggest friction points with local LLMs is the context window. Even with a massive 256k context limit, you can't, and shouldn't, dump your entire history into every prompt. It bloats VRAM and destroys inference speeds.\n\nThe solution is a **Personalized Long-Term Memory RAG (Retrieval-Augmented Generation)**. Today, I'm going to walk you through exactly how I built an autonomous, privacy-focused memory layer that seamlessly injects context into my chats using **Mem0**, **Qdrant**, and **Open WebUI**, completely bypassing the need for clunky UI document uploads.\n\n### The Hardware & OS\n\nMy current daily driver for this setup is a modular mini PC from Framework running **Fedora 44**.\n\nHere are the specs of my machine:\n\n- Ryzen™ AI Max+ 395 - 128GB (I've allocated 118GB of this to VRAM for hosting models)\n- Storage: WD_BLACK™ SN7100 NVMe™ - M.2 2280 - 4TB\n\nFor local inference, I'm relying entirely on an **integrated AMD GPU**. Offloading a heavy model like `qwen3-coder-next`\n\npushes the APU to its limits and I regularly see GPU utilization pinned at 100% with VRAM usage hovering right around **40-50GB**. Despite being integrated graphics, this configuration handles the RAG pipeline flawlessly without breaking a sweat.\n\nWe will be using **Podman** instead of Docker, keeping things native to the Fedora ecosystem.\n\n### Step 1: Spin Up the Vector Database (Qdrant)\n\nQdrant acts as the limitless hard drive for our AI's memories. It stores the mathematical vectors and text payloads on disk so they persist across reboots.\n\nRun this Podman command to spin up Qdrant using host networking:\n\nBash\n\n```\npodman run -d \\\n  --name qdrant \\\n  --network=host \\\n  -v ~/.local/share/qdrant_data:/qdrant/storage:z \\\n  --restart always \\\n  qdrant/qdrant:latest\n```\n\n### Step 2: Prepare the Ollama Models\n\nWe need two models running in Ollama:\n\n**The Embedder:** Translates text into 768-dimensional vectors.**The Chat Model:** The brains of the operation.\n\nBash\n\n```\nollama pull nomic-embed-text:latest\nollama pull qwen3-coder-next:q4_K_M\n```\n\n*(Tip: Make sure the Ollama daemon is running in the background via systemctl status ollama)*.\n\n### Step 3: The Magic Middleware (Mem0 Proxy)\n\nInstead of hacking memory directly into Open WebUI, we are deploying a lightweight FastAPI proxy. This sits between your UI and Ollama. When you send a message, the proxy intercepts it, searches Qdrant for relevant past facts, injects them into the system prompt, and forwards it to Qwen. This is the RAG step in action!\n\nCrucially, it also features a **background task** that silently extracts new facts from your conversation and saves them to Qdrant without adding any latency to your chat response.\n\nCreate a file at `~/memory_proxy.py`\n\n(or wherever you wish) and paste this in:\n\nPython\n\n``` python\nimport os\nimport uvicorn\nimport asyncio\nfrom fastapi import FastAPI, Request, BackgroundTasks\nfrom fastapi.responses import StreamingResponse\nimport httpx\nfrom mem0 import Memory\n\nos.environ[\"MEM0_TELEMETRY\"] = \"False\"\n\napp = FastAPI(title=\"Mem0 Local Proxy with Auto-Extract\")\n\n# Configure Mem0\nmem0_config = {\n    \"vector_store\": {\n        \"provider\": \"qdrant\",\n        \"config\": {\n            \"collection_name\": \"lead_prompt_memories\",\n            \"host\": \"localhost\",\n            \"port\": 6333,\n            \"embedding_model_dims\": 768,\n        }\n    },\n    \"llm\": {\n        \"provider\": \"ollama\",\n        \"config\": {\n            \"model\": \"qwen3-coder-next:q4_K_M\",\n            \"temperature\": 0.1,\n            \"max_tokens\": 4096,\n            \"ollama_base_url\": \"http://127.0.0.1:11434\",\n        }\n    },\n    \"embedder\": {\n        \"provider\": \"ollama\",\n        \"config\": {\n            \"model\": \"nomic-embed-text:latest\",\n            \"ollama_base_url\": \"http://127.0.0.1:11434\",\n        }\n    }\n}\n\nmemory = Memory.from_config(mem0_config)\nOLLAMA_BASE_URL = \"http://127.0.0.1:11434\"\n\n# Background extraction logic\ndef extract_and_save_memory(messages: list, user_id: str):\n    try:\n        memory.add(messages, user_id=user_id)\n        print(f\"\\n[MEM0 BACKGROUND]: Memory extraction complete for user '{user_id}'.\")\n    except Exception as e:\n        print(f\"\\n[MEM0 BACKGROUND ERROR]: Failed to extract memory: {e}\")\n\n@app.get(\"/v1/models\")\nasync def list_models():\n    async with httpx.AsyncClient(timeout=30.0) as client:\n        resp = await client.get(f\"{OLLAMA_BASE_URL}/v1/models\")\n        return resp.json()\n\n@app.post(\"/v1/chat/completions\")\nasync def chat_completions(request: Request, background_tasks: BackgroundTasks):\n    body = await request.json()\n    messages = body.get(\"messages\", [])\n    user_id = body.get(\"user\", \"john\")\n\n    user_query = \"\"\n    for msg in reversed(messages):\n        if msg.get(\"role\") == \"user\":\n            user_query = msg.get(\"content\", \"\")\n            break\n\n    # 1. READ: Retrieve existing memories\n    memories_str = \"\"\n    if user_query:\n        try:\n            results = memory.search(query=user_query, filters={\"user_id\": user_id})\n            mem_list = [r[\"memory\"] for r in results.get(\"results\", [])]\n            if mem_list:\n                memories_str = \"\\n\".join(f\"- {m}\" for m in mem_list)\n        except Exception as e:\n            print(f\"[Proxy Memory Error]: {e}\")\n\n    # 2. INJECT: Add retrieved context to system prompt\n    if memories_str:\n        context_prompt = (\n            f\"\\n\\n[Relevant User Context & Saved Preferences]:\\n{memories_str}\\n\"\n        )\n        if messages and messages[0].get(\"role\") == \"system\":\n            messages[0][\"content\"] += context_prompt\n        else:\n            messages.insert(0, {\"role\": \"system\", \"content\": f\"You are a helpful assistant.{context_prompt}\"})\n\n    body[\"messages\"] = messages\n\n    # 3. WRITE: Schedule background extraction\n    if user_query:\n        background_tasks.add_task(extract_and_save_memory, messages[-2:], user_id)\n\n    # 4. STREAM back to client\n    async def stream_generator():\n        async with httpx.AsyncClient(timeout=300.0) as client:\n            async with client.stream(\n                \"POST\", f\"{OLLAMA_BASE_URL}/v1/chat/completions\", json=body, headers={\"Content-Type\": \"application/json\"}\n            ) as resp:\n                async for chunk in resp.aiter_bytes():\n                    yield chunk\n\n    if body.get(\"stream\", False):\n        return StreamingResponse(stream_generator(), media_type=\"text/event-stream\")\n    else:\n        async with httpx.AsyncClient(timeout=300.0) as client:\n            resp = await client.post(f\"{OLLAMA_BASE_URL}/v1/chat/completions\", json=body)\n            return resp.json()\n\nif __name__ == \"__main__\":\n    uvicorn.run(app, host=\"0.0.0.0\", port=11435)\n```\n\n### Step 4: Daemonize the Proxy with Systemd\n\nWe want this running persistently without tying up a terminal window.\n\nCreate a systemd user service:\n\n```\nnano ~/.config/systemd/user/memory-proxy.service\n```\n\nIni, TOML\n\n```\n[Unit]\nDescription=Mem0 OpenAI Local Proxy Service\nAfter=network.target\n\n[Service]\nType=simple\nExecStart=/usr/bin/python3 /home/john/memory_proxy.py\nWorkingDirectory=/home/john\nRestart=always\nRestartSec=3\nEnvironment=MEM0_TELEMETRY=False\n\n[Install]\nWantedBy=default.target\n```\n\nEnable and start it:\n\nBash\n\n```\nsystemctl --user daemon-reload\nsystemctl --user enable --now memory-proxy.service\n```\n\n### Step 5: Open WebUI Configuration\n\nFirst, spin up the WebUI container (again, native host networking is best here):\n\nBash\n\n```\npodman run -d \\\n  --name open-webui \\\n  --network=host \\\n  -v open-webui:/app/backend/data:z \\\n  --restart always \\\n  ghcr.io/open-webui/open-webui:main\n```\n\nNow, wire the UI to our memory proxy rather than directly to Ollama:\n\n- Navigate to\n`http://localhost:3000`\n\nand go to**Admin Panel -> Settings -> Connections**. **Disable** the standard Ollama API connection.**Enable** the OpenAI API connection and set the URL to our proxy:`http://host.containers.internal:11435/v1`\n\n(with any dummy text for the API key).- Save the configuration.\n\n**Crucial Final Step:** Open WebUI has its own internal RAG and web search mechanisms that will try to hijack queries before they reach our proxy. To prevent this:\n\n- Go to\n**Workspace -> Models** and edit your`Qwen3 Coder`\n\npreset. - Scroll down to the capabilities/tools section and manually\n**disable Knowledge/RAG and Web Search**.\n\n### The Result\n\nYou now have a model running locally that *remembers*. Because the memory proxy is exposed via standard OpenAI API format, you aren't locked into Open WebUI either: you can point Zed, terminal curl scripts, or any other LLM front-end to `localhost:11435/v1`\n\nand take your personalized vector brain with you. Secondly, you are also not tied into qwen3-coder-next and can use any local model you wish, and even share context between models!", "url": "https://wpnews.pro/news/building-an-infinite-memory-local-ai-stack-on-fedora", "canonical_source": "https://leadprompt.sh/a/739-Building-an-Infinite-Memory-Local-AI-Stack-on-Fedora", "published_at": "2026-07-31 17:37:35+00:00", "updated_at": "2026-07-31 17:52:21.378356+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Mem0", "Qdrant", "Open WebUI", "Fedora 44", "Framework", "Ryzen AI Max+ 395", "Ollama", "FastAPI"], "alternates": {"html": "https://wpnews.pro/news/building-an-infinite-memory-local-ai-stack-on-fedora", "markdown": "https://wpnews.pro/news/building-an-infinite-memory-local-ai-stack-on-fedora.md", "text": "https://wpnews.pro/news/building-an-infinite-memory-local-ai-stack-on-fedora.txt", "jsonld": "https://wpnews.pro/news/building-an-infinite-memory-local-ai-stack-on-fedora.jsonld"}}