# Building an Infinite Memory Local AI Stack on Fedora

> Source: <https://leadprompt.sh/a/739-Building-an-Infinite-Memory-Local-AI-Stack-on-Fedora>
> Published: 2026-07-31 17:37:35+00:00

If you’ve been following my recent experiments, you know I’m a huge advocate for running open-weight models locally. But one of the biggest friction points with local LLMs is the context window. Even with a massive 256k context limit, you can't, and shouldn't, dump your entire history into every prompt. It bloats VRAM and destroys inference speeds.

The solution is a **Personalized Long-Term Memory RAG (Retrieval-Augmented Generation)**. Today, I'm going to walk you through exactly how I built an autonomous, privacy-focused memory layer that seamlessly injects context into my chats using **Mem0**, **Qdrant**, and **Open WebUI**, completely bypassing the need for clunky UI document uploads.

### The Hardware & OS

My current daily driver for this setup is a modular mini PC from Framework running **Fedora 44**.

Here are the specs of my machine:

- Ryzen™ AI Max+ 395 - 128GB (I've allocated 118GB of this to VRAM for hosting models)
- Storage: WD_BLACK™ SN7100 NVMe™ - M.2 2280 - 4TB

For local inference, I'm relying entirely on an **integrated AMD GPU**. Offloading a heavy model like `qwen3-coder-next`

pushes the APU to its limits and I regularly see GPU utilization pinned at 100% with VRAM usage hovering right around **40-50GB**. Despite being integrated graphics, this configuration handles the RAG pipeline flawlessly without breaking a sweat.

We will be using **Podman** instead of Docker, keeping things native to the Fedora ecosystem.

### Step 1: Spin Up the Vector Database (Qdrant)

Qdrant acts as the limitless hard drive for our AI's memories. It stores the mathematical vectors and text payloads on disk so they persist across reboots.

Run this Podman command to spin up Qdrant using host networking:

Bash

```
podman run -d \
  --name qdrant \
  --network=host \
  -v ~/.local/share/qdrant_data:/qdrant/storage:z \
  --restart always \
  qdrant/qdrant:latest
```

### Step 2: Prepare the Ollama Models

We need two models running in Ollama:

**The Embedder:** Translates text into 768-dimensional vectors.**The Chat Model:** The brains of the operation.

Bash

```
ollama pull nomic-embed-text:latest
ollama pull qwen3-coder-next:q4_K_M
```

*(Tip: Make sure the Ollama daemon is running in the background via systemctl status ollama)*.

### Step 3: The Magic Middleware (Mem0 Proxy)

Instead of hacking memory directly into Open WebUI, we are deploying a lightweight FastAPI proxy. This sits between your UI and Ollama. When you send a message, the proxy intercepts it, searches Qdrant for relevant past facts, injects them into the system prompt, and forwards it to Qwen. This is the RAG step in action!

Crucially, it also features a **background task** that silently extracts new facts from your conversation and saves them to Qdrant without adding any latency to your chat response.

Create a file at `~/memory_proxy.py`

(or wherever you wish) and paste this in:

Python

``` python
import os
import uvicorn
import asyncio
from fastapi import FastAPI, Request, BackgroundTasks
from fastapi.responses import StreamingResponse
import httpx
from mem0 import Memory

os.environ["MEM0_TELEMETRY"] = "False"

app = FastAPI(title="Mem0 Local Proxy with Auto-Extract")

# Configure Mem0
mem0_config = {
    "vector_store": {
        "provider": "qdrant",
        "config": {
            "collection_name": "lead_prompt_memories",
            "host": "localhost",
            "port": 6333,
            "embedding_model_dims": 768,
        }
    },
    "llm": {
        "provider": "ollama",
        "config": {
            "model": "qwen3-coder-next:q4_K_M",
            "temperature": 0.1,
            "max_tokens": 4096,
            "ollama_base_url": "http://127.0.0.1:11434",
        }
    },
    "embedder": {
        "provider": "ollama",
        "config": {
            "model": "nomic-embed-text:latest",
            "ollama_base_url": "http://127.0.0.1:11434",
        }
    }
}

memory = Memory.from_config(mem0_config)
OLLAMA_BASE_URL = "http://127.0.0.1:11434"

# Background extraction logic
def extract_and_save_memory(messages: list, user_id: str):
    try:
        memory.add(messages, user_id=user_id)
        print(f"\n[MEM0 BACKGROUND]: Memory extraction complete for user '{user_id}'.")
    except Exception as e:
        print(f"\n[MEM0 BACKGROUND ERROR]: Failed to extract memory: {e}")

@app.get("/v1/models")
async def list_models():
    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.get(f"{OLLAMA_BASE_URL}/v1/models")
        return resp.json()

@app.post("/v1/chat/completions")
async def chat_completions(request: Request, background_tasks: BackgroundTasks):
    body = await request.json()
    messages = body.get("messages", [])
    user_id = body.get("user", "john")

    user_query = ""
    for msg in reversed(messages):
        if msg.get("role") == "user":
            user_query = msg.get("content", "")
            break

    # 1. READ: Retrieve existing memories
    memories_str = ""
    if user_query:
        try:
            results = memory.search(query=user_query, filters={"user_id": user_id})
            mem_list = [r["memory"] for r in results.get("results", [])]
            if mem_list:
                memories_str = "\n".join(f"- {m}" for m in mem_list)
        except Exception as e:
            print(f"[Proxy Memory Error]: {e}")

    # 2. INJECT: Add retrieved context to system prompt
    if memories_str:
        context_prompt = (
            f"\n\n[Relevant User Context & Saved Preferences]:\n{memories_str}\n"
        )
        if messages and messages[0].get("role") == "system":
            messages[0]["content"] += context_prompt
        else:
            messages.insert(0, {"role": "system", "content": f"You are a helpful assistant.{context_prompt}"})

    body["messages"] = messages

    # 3. WRITE: Schedule background extraction
    if user_query:
        background_tasks.add_task(extract_and_save_memory, messages[-2:], user_id)

    # 4. STREAM back to client
    async def stream_generator():
        async with httpx.AsyncClient(timeout=300.0) as client:
            async with client.stream(
                "POST", f"{OLLAMA_BASE_URL}/v1/chat/completions", json=body, headers={"Content-Type": "application/json"}
            ) as resp:
                async for chunk in resp.aiter_bytes():
                    yield chunk

    if body.get("stream", False):
        return StreamingResponse(stream_generator(), media_type="text/event-stream")
    else:
        async with httpx.AsyncClient(timeout=300.0) as client:
            resp = await client.post(f"{OLLAMA_BASE_URL}/v1/chat/completions", json=body)
            return resp.json()

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=11435)
```

### Step 4: Daemonize the Proxy with Systemd

We want this running persistently without tying up a terminal window.

Create a systemd user service:

```
nano ~/.config/systemd/user/memory-proxy.service
```

Ini, TOML

```
[Unit]
Description=Mem0 OpenAI Local Proxy Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /home/john/memory_proxy.py
WorkingDirectory=/home/john
Restart=always
RestartSec=3
Environment=MEM0_TELEMETRY=False

[Install]
WantedBy=default.target
```

Enable and start it:

Bash

```
systemctl --user daemon-reload
systemctl --user enable --now memory-proxy.service
```

### Step 5: Open WebUI Configuration

First, spin up the WebUI container (again, native host networking is best here):

Bash

```
podman run -d \
  --name open-webui \
  --network=host \
  -v open-webui:/app/backend/data:z \
  --restart always \
  ghcr.io/open-webui/open-webui:main
```

Now, wire the UI to our memory proxy rather than directly to Ollama:

- Navigate to
`http://localhost:3000`

and go to**Admin Panel -> Settings -> Connections**. **Disable** the standard Ollama API connection.**Enable** the OpenAI API connection and set the URL to our proxy:`http://host.containers.internal:11435/v1`

(with any dummy text for the API key).- Save the configuration.

**Crucial Final Step:** Open WebUI has its own internal RAG and web search mechanisms that will try to hijack queries before they reach our proxy. To prevent this:

- Go to
**Workspace -> Models** and edit your`Qwen3 Coder`

preset. - Scroll down to the capabilities/tools section and manually
**disable Knowledge/RAG and Web Search**.

### The Result

You now have a model running locally that *remembers*. Because the memory proxy is exposed via standard OpenAI API format, you aren't locked into Open WebUI either: you can point Zed, terminal curl scripts, or any other LLM front-end to `localhost:11435/v1`

and take your personalized vector brain with you. Secondly, you are also not tied into qwen3-coder-next and can use any local model you wish, and even share context between models!