A few weeks ago I sat in a system-design conversation with a NYC-based startup.
They asked about to build a system design for a memory layer for their sales agent: not just a basic RAG, but how an agent should remember, update, and forget facts about a user and their conversation with the system over time — without blocking the chat reply, without stuffing the entire history into every prompt, and without treating every similar embedding as the same fact.
I knew the buzzwords. I did not have a crisp mental model.
Well I did build a complete RAG with strong redis cache and cache TTL and invalidation as well, unfortunately didn’t clear that round, so I decided to fill my knowledge gap by building one and currently one of the best memory layer architectures for conversational agents out there is owned by mem0. I read how Mem0-style pipelines are described in the wild — hybrid retrieval + LLM judgment — and implemented a small but complete version: CogLayer. Here’s the research paper by them which I referred— https://arxiv.org/pdf/2504.19413
This article is the write-up I wish I’d had before that interview: the why, the architecture, and a path you can click through yourself.
In chatbots, people often mix three different things:
-
Context window — whatever fits in the current prompt
-
Conversation history — raw turns (“User said… Assistant said…”)
-
Durable memory — atomic facts you want to keep across sessions
(“Alex prefers dark roast coffee,” “User adopted a dog named Scout”)
Mem0-style systems focus on (3), while still using (2) as fuel for extraction. The hard part is not storing text. The hard part is:
That last point is why the architecture splits into two paths.
The core idea: hybrid recall + judgment
Vector search is excellent at cheap recall:
“Given this sentence, what are the top-s similar memories for this user?”
It is terrible at semantic judgment on its own.
“Likes X” and “hates X” can sit next to each other in embedding space. Similarity will happily retrieve both. Only a model (or careful rules) can say: update, delete, or noop.
So the hybrid loop is:
Embeddings / Qdrant — Narrow the world to a few relevant memories |
LLM EXTRACT — Pull durable candidate facts from a message pair + recent context
LLM DECIDE — Pick exactly one tool: ADD / UPDATE / DELETE / NOOP
Async worker — Apply the tool to the store so chat stays snappy
Mem0 popularized this style of thinking for production agents. CogLayer is a learning implementation of that loop — not a clone of Mem0 Cloud, and not a claim to replace it. The goal is understanding you can demo and explain.
Two paths: Read (sync) and Write (async) Read path:
Every /chat request:
-
User sends a query
-
Embed the query → Qdrant top-k , filtered by
user_id -
Load recent conversation turns
-
Build a prompt (instructions + memories + history + query)
-
LLM generates a reply
-
Return the reply immediately
-
Fire a background job (Celery / RabbitMQ) — do not block the HTTP response.
Write path:
The worker receives the message pair (user query + assistant reply):
-
Save the pair to the conversation store
-
Build extraction prompt P: rolling summary S + last ~10 turns + new pair
-
LLM #1 — EXTRACT→ candidate facts
-
For each candidate: embed → top-s similar memories for that user
-
LLM #2 — DECIDE → one tool call
-
Apply ops to JSON memory store/or you own storage and Qdrant(re-embed on ADD/UPDATE)
Summary S in CogLayer is computed when the write path runs(on each new pair). Some production designs cache summary with a periodic summarizer to save LLM cost at scale. For learning and for short demos, on-write summarization is the honest default.
Why system design interviews care about this?
If you only say “we’ll use a vector database/RAG,” you miss: Mem0-style design is a clean story for all of the above. Building even a thin version forces you to confront the failure modes: stale vectors after JSON updates, eager .delay() with no worker, EXTRACT inventing facts, DECIDE ADDing near-duplicates.
Those scars are better interview fuel than a perfect diagram.
Learning it by seeing it:
CogLayer includes an animated lab (Excalidraw-ish whiteboard UI) that steps through Read and Write:
Repo — https://github.com/Swaraj07082/CogLayer
I didn’t build CogLayer to compete with Mem0.
I built it because I couldn’t explain a memory layer under pressure — and explanation without implementation is fragile.
If you’re learning agent memory, preparing for AI system design, or teaching the difference between RAG-for-documents and memory-for-users, walk the /learn on my repo once. Then break it on purpose: kill the worker, skip Qdrant sync, force a contradictory fact. Watch what fails. That’s how the diagram becomes intuition.
Further reading:
Peace Out! ✌️
I Failed a Memory-layer System Design Round. So I built one! was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.