How to Build a Semantic Memory System for AI Agents Without Hermes Agent

A guide explains how to build a semantic memory system for AI agents using local vector databases and embedding pipelines, drawing on lessons from the Hermes Agent experiment. The system comprises three layers—storage, embedding pipeline, and retrieval/injection—and can be implemented in environments like Claude Code without vendor lock-in. ChromaDB is recommended as an accessible starting point for local vector storage.

How to Build a Semantic Memory System for AI Agents Without Hermes Agent Hermes Agent proved memory matters, but you can build better recall inside Claude Code. Learn storage, injection, and semantic search using local vector DBs. Why AI Agents Keep Forgetting And How to Fix It Memory is the difference between an AI agent that’s genuinely useful and one you have to re-explain everything to every single session. If you’ve spent any time building agents, you’ve hit this wall. The model reasons well. The tool calls work. But the moment context resets, it’s back to square one. This is exactly what made the Hermes Agent experiment interesting. It showed — clearly, repeatedly — that when you give an AI agent a structured semantic memory system, its performance on complex, multi-step tasks improves substantially. Agents stopped repeating themselves. They built on prior reasoning. They started feeling less like stateless APIs and more like persistent collaborators. The good news: you don’t need Hermes Agent to get that. You can build a semantic memory system for AI agents from scratch using local vector databases, embedding pipelines, and smart injection logic. This guide walks through exactly how to do that inside environments like Claude Code or any custom agent framework. What Hermes Agent Actually Demonstrated Hermes Agent — built on top of Ollama and local models — gained attention for its emphasis on persistent, context-aware memory. The core insight wasn’t revolutionary in theory, but it was compelling in practice: agents that could recall relevant past interactions performed better on reasoning tasks than those operating from scratch. What it specifically demonstrated: Retrieval-augmented context matters more than raw model size in long-horizon tasks- Storing memories as vector embeddings, rather than raw text logs, allows for semantic retrieval — not just keyword matching - Agents with memory made fewer contradictory decisions across sessions - The retrieval step itself could be lightweight enough to run locally without meaningful latency The limitation of Hermes Agent was mostly about portability and flexibility. It worked within its own ecosystem. If you’re building on Claude Code, LangChain, CrewAI, or a custom Python agent, you need the same capability without the lock-in. The Anatomy of a Semantic Memory System Before writing any code, it’s worth understanding the three functional layers of a semantic memory system. Layer 1: Storage Where memories live. In a semantic system, this is a vector database — a store that holds numerical representations of text embeddings alongside the original content and metadata. Layer 2: The Embedding Pipeline The process of converting text agent observations, user inputs, decisions, outcomes into vector embeddings. This is what enables semantic search rather than keyword matching. Two sentences that mean the same thing but use different words will have similar vectors. Layer 3: Retrieval and Injection When the agent needs to act, it queries the memory store using the current context as a search key. The most semantically relevant memories are retrieved and injected into the agent’s prompt or context window. These three layers are independent. You can swap out the vector DB, change embedding models, or redesign the injection logic without touching the others. That modularity is intentional — your memory system should be infrastructure, not a bottleneck. Choosing a Local Vector Database Running memory locally matters for several reasons: latency, privacy, cost, and control. Here are the main options worth considering. ChromaDB ChromaDB https://www.trychroma.com/ is probably the most accessible starting point. It runs in-process no separate server needed for small deployments , has a clean Python API, and integrates natively with LangChain and LlamaIndex. Good for: prototyping, local development, small to medium agent deployments Limitations: not ideal for high-concurrency production workloads python import chromadb client = chromadb.Client collection = client.create collection "agent memory" collection.add documents= "User prefers concise responses", "Project deadline is Friday" , ids= "mem 001", "mem 002" results = collection.query query texts= "What are the user's preferences?" , n results=3 Qdrant Qdrant is a purpose-built vector search engine with a Rust core, which makes it fast. It supports filtering, named vectors, and payload indexing. You can run it locally via Docker with a single command. Good for: production-grade local deployments, agents that need filtered retrieval Limitations: requires Docker; slightly more setup than Chroma FAISS Facebook AI Similarity Search FAISS is a library rather than a database. It’s extremely fast for pure similarity search but doesn’t handle persistence, metadata, or CRUD operations natively. You’d wrap it yourself or use it through a higher-level library. Good for: high-performance similarity search when you’re managing storage separately Limitations: no built-in persistence; you handle serialization LanceDB A newer entrant, LanceDB stores vectors in a columnar format on disk. It’s serverless by design and has good performance for agents that need to scan large memory stores. For most builders starting out, ChromaDB gets you running in under 10 minutes. Move to Qdrant when you need more control. Building Your Embedding Pipeline The embedding pipeline converts raw text into vectors. The model you use determines how well your semantic search performs. Choosing an Embedding Model For local use, these are the practical options: via Ollama — solid general-purpose embeddings, runs locally, 768 dimensions nomic-embed-text via Ollama — higher quality, 1024 dimensions, slightly slower mxbai-embed-large via OpenAI API — excellent quality if you’re okay with API calls, cost is minimal text-embedding-3-small via HuggingFace — fast, lightweight, works entirely offline sentence-transformers/all-MiniLM-L6-v2 Everyone else built a construction worker. We built the contractor. One file at a time. UI, API, database, deploy. For a Claude Code environment where you want full local operation: php import ollama def embed text text: str - list float : response = ollama.embeddings model="nomic-embed-text", prompt=text return response "embedding" What to Embed This is a design decision that significantly affects memory quality. Don’t just dump raw conversation logs. Embed these: - Summarized observations “User confirmed the report format should be CSV, not JSON” - Agent decisions with context “Chose to use the search tool because the query required current data” - Task outcomes “Completed email draft — user approved with minor edits” - User preferences revealed during interaction - Errors and how they were resolved Avoid embedding: - Raw, unprocessed transcripts too noisy, retrieval quality degrades - Ephemeral tool outputs that aren’t meaningful long-term - Duplicate or near-duplicate entries use deduplication logic Memory Extraction Step Before storing, have the agent or a lightweight summarizer extract the memory-worthy content from a conversation turn: php def extract memory conversation turn: str, llm - str: prompt = f"""Extract the key factual information, preferences, or decisions from this interaction that would be useful to remember in future sessions. Be concise — one to three sentences maximum. Interaction: {conversation turn} Memory:""" return llm.complete prompt This step dramatically improves retrieval quality because you’re storing clean, dense signal rather than raw conversation noise. Implementing Semantic Search Once you have stored memories, retrieval needs to be fast and relevant. Basic Retrieval Pattern python def retrieve relevant memories query: str, collection, n results: int = 5, min relevance threshold: float = 0.7 - list str : query embedding = embed text query results = collection.query query embeddings= query embedding , n results=n results, include= "documents", "distances" Filter by relevance threshold memories = for doc, distance in zip results "documents" 0 , results "distances" 0 : ChromaDB returns L2 distance; lower is more similar similarity = 1 - distance if similarity = min relevance threshold: memories.append doc return memories Hybrid Search for Better Recall Pure semantic search sometimes misses exact matches. A hybrid approach combines vector similarity with keyword filtering: - Run semantic search to get top-N candidates - Apply BM25 or simple keyword scoring as a re-ranking step - Return the re-ranked top results For most agent use cases, pure semantic search with a reasonable threshold 0.65–0.75 is sufficient. Add hybrid search when your agents work with lots of proper nouns, codes, or identifiers that semantic models don’t handle well. Temporal Decay Not all memories are equally current. A preference the user stated six months ago may have changed. Add a recency factor to your scoring: python import math from datetime import datetime def score with recency similarity: float, timestamp: datetime, decay rate: float = 0.01 - float: days old = datetime.now - timestamp .days recency score = math.exp -decay rate days old return similarity 0.7 + recency score 0.3 Tune the weights based on your use case. A coding assistant should weight recency lightly older patterns are still valid . A customer support agent should weight it heavily account status changes frequently . Memory Injection Strategies Retrieved memories are only useful if you inject them into the agent’s context at the right time, in the right format. System Prompt Injection The simplest approach: prepend retrieved memories to the system prompt before each agent turn. php def build system prompt base prompt: str, relevant memories: list str - str: if not relevant memories: return base prompt memory block = "\n".join f"- {mem}" for mem in relevant memories return f"""{base prompt} Relevant Memory The following information from previous interactions is relevant to this task: {memory block} Use this context to inform your responses, but don't reference it explicitly unless asked.""" Structured Memory Context For agents with longer tasks, a more structured format helps the model parse and use memories correctly: Long-Term Memory from previous sessions User Preferences - Prefers bullet points over paragraphs in summaries - Works in EST timezone, availability 9am–5pm Project Context - Current project: Q3 financial report automation - Last completed: data ingestion pipeline completed 2024-11-15 - Next milestone: visualization layer Known Issues - API rate limits hit when batch size 100 records Episodic vs. Semantic Memory Borrowing from cognitive science, it helps to separate two memory types: Episodic memory : Specific events “On Tuesday, the user rejected the CSV format proposal” Semantic memory : General knowledge the agent has built up “This user prefers JSON output formats” Store them in separate collections with different retrieval weights. Episodic memories are useful for continuity; semantic memories are useful for preference and behavioral adaptation. Avoiding Context Window Bloat Memory injection can eat your context window fast. Keep these constraints in mind: - Set a hard cap on injected memories 5–8 is usually enough - Compress older memories with periodic summarization runs - Use importance scoring to prioritize high-signal memories over low-signal ones - Consider a two-tier system: hot memory recent, in-context and cold memory older, retrieved on demand Managing Memory Over Time A memory system that only grows will eventually degrade in quality and performance. Deduplication Before storing a new memory, check if a semantically similar one already exists: php def should store memory new memory: str, collection, threshold: float = 0.92 - bool: results = collection.query query embeddings= embed text new memory , n results=1 if not results "distances" 0 : return True similarity = 1 - results "distances" 0 0 return similarity < threshold Periodic Consolidation Run a consolidation job daily or weekly that: - Clusters similar memories together - Generates a summary of each cluster - Replaces the cluster with the summary - Removes memories below an importance threshold This is the rough equivalent of how sleep consolidates human memory — you’re compressing episodes into denser, more useful representations. Memory Importance Scoring Not all memories are worth keeping. Add an importance score when storing: - High: User stated explicit preference, agent made a significant decision, error with resolution - Medium: Useful context, task status - Low: Routine confirmations, obvious inferences Drop low-importance memories after 30 days unless they’ve been accessed recently. How MindStudio Fits Into This Picture Building and maintaining all of this infrastructure — the vector DB, the embedding pipeline, the injection logic, the consolidation jobs — is real work. It’s manageable, but it adds up fast across a multi-agent system. This is where MindStudio’s approach to multi-agent workflows https://app.mindstudio.ai makes a practical difference. Rather than wiring memory infrastructure together yourself for every agent, MindStudio lets you build agents with persistent data access baked into the workflow — using Airtable, Notion, Google Sheets, or custom datastores as structured memory layers, connected through its 1,000+ integrations. Plans first. Then code. Remy writes the spec, manages the build, and ships the app. For teams building more sophisticated setups, the Agent Skills Plugin https://www.mindstudio.ai/blog/introducing-agent-skills @mindstudio-ai/agent lets Claude Code agents and other custom agents call MindStudio’s capabilities as simple method calls. You get managed infrastructure for things like agent.runWorkflow without building the orchestration yourself. That means your agent can offload memory consolidation, summarization, or retrieval-augmented lookup to a MindStudio workflow — while your core agent focuses on reasoning. If you’re earlier in the build process and want to prototype memory-enabled agents without setting up vector DBs locally, MindStudio’s no-code builder https://app.mindstudio.ai lets you do that in under an hour using its visual workflow tools and built-in data connectors. You can try it free at mindstudio.ai https://www.mindstudio.ai . Common Mistakes to Avoid Storing Too Much, Too Literally The most common mistake is treating memory as a log. You don’t want a transcript — you want distilled signal. Always run a summarization/extraction step before storing. Ignoring Retrieval Quality Building storage is the easy part. Retrieval quality determines whether the system actually helps. Test your semantic search with real agent queries. Tune your similarity thresholds. Add hybrid search if pure semantic search underperforms. Single Collection for Everything Separate your memory types. User preferences, project context, error history, and episodic events should live in different collections with different retrieval strategies. No Memory Validation Agents can store incorrect memories. Add a confidence layer: if the memory was derived from an uncertain inference rather than an explicit statement, tag it accordingly and weight it lower during retrieval. Forgetting Write Performance For agents running at high frequency, write performance matters. ChromaDB in-process mode is fine for low-throughput agents. For anything that writes memories at high frequency, benchmark your setup or move to Qdrant with a persistent server. Frequently Asked Questions What is a semantic memory system for AI agents? A semantic memory system stores agent experiences, observations, and context as vector embeddings and retrieves them using semantic similarity search. Unlike keyword search, it finds relevant memories based on meaning — so a query about “user formatting preferences” will surface memories about layout choices even if those exact words weren’t used. How is this different from just using a long context window? Long context windows include everything — relevant or not. A semantic memory system selectively retrieves only the most relevant information for a given query. This means less noise, lower token costs, and the ability to retain knowledge across sessions without hitting context limits. Which vector database should I use for a local agent? For most builders, ChromaDB is the fastest starting point — it runs in-process with no separate server and has a clean Python API. If you need production-grade performance, filtering capabilities, or are deploying at scale, Qdrant running locally via Docker is the better choice. How do I prevent my agent from storing bad memories? Use an extraction step that filters raw interactions down to high-confidence, factual statements before storing. Add a deduplication check to avoid redundant entries. Tag uncertain inferences separately from explicit statements, and weight them lower during retrieval. Can I build a semantic memory system without using OpenAI’s embedding API? Yes. Ollama supports several strong local embedding models like nomic-embed-text and mxbai-embed-large that run entirely on your machine. The HuggingFace sentence-transformers library also provides high-quality embeddings with no API calls required. How many memories should I inject into each agent prompt? Typically 3–8 is the right range. More than that and you’re adding noise that can confuse the model and bloat your context window. Use similarity thresholds to ensure only genuinely relevant memories are included rather than injecting a fixed number regardless of relevance. Key Takeaways - Semantic memory systems have three layers: storage vector DB , an embedding pipeline, and retrieval/injection logic — each independently replaceable. - ChromaDB is the fastest local starting point; Qdrant is better for production workloads. - Always summarize and extract signal before storing — never log raw conversations. - Inject 3–8 relevant memories per agent turn using similarity thresholds, not fixed counts. - Separate episodic memories events from semantic memories preferences and patterns for better retrieval quality. - Add temporal decay, deduplication, and periodic consolidation to keep memory quality high over time. Hermes Agent showed that memory changes what agents can do. Building that capability yourself, with tools you control, puts you in a much better position than depending on a single framework. The infrastructure isn’t complicated — but the design decisions matter. If you want to skip the infrastructure setup and start with a working agent that has persistent memory and data access, MindStudio https://www.mindstudio.ai is worth exploring. The average build takes under an hour, and you can connect it to the same datastores your agents already use.