{"slug": "beyond-the-prompt-how-to-build-an-ai-agent-that-actually-learns-from-its", "title": "Beyond the Prompt: How to Build an AI Agent That Actually Learns From Its Mistakes", "summary": "The article argues that typical AI agents are not truly autonomous because they lack memory and cannot learn from mistakes, functioning instead as stateless systems. It introduces Hermes, an open-source cognitive architecture designed to overcome this by using a self-authored journal for persistent memory and a closed learning loop. Hermes autonomously decides what to remember, storing factual knowledge and user preferences in structured files to enable continuous self-improvement through reflection.", "body_md": "We’ve all seen the standard pattern of AI development over the past couple of years. You write a system prompt, wrap it in an API call, hook up a vector database for Retrieval-Augmented Generation (RAG), and call it an \"autonomous agent.\"\nBut let’s be honest: these systems aren't actually autonomous. They are stateless function evaluators. They receive a prompt, generate a response, and instantly forget everything that just happened. If they make a mistake on step three of a task, they are highly likely to repeat that exact same mistake on step five. They don’t learn, they don’t adapt, and they don’t grow. They are stuck in a digital version of Groundhog Day.\nTo build truly autonomous software, we need a profound architectural shift. We need systems that can observe their own performance, write their own journals, and build their own custom tools.\nEnter Hermes: an open-source, self-learning cognitive architecture.\nIn this deep dive, we will break down the core theoretical architecture of Hermes. We’ll look at real production code to see how it implements persistent memory, executes a closed learning loop, and achieves continuous self-improvement through reflection.\n(The concepts and code demonstrated here are drawn from my ebook Hermes Agent, The Self-Evolving AI Workforce)\nTo understand why Hermes represents a shift in agentic AI, consider the analogy of a master craftsman.\nA novice apprentice reads a manual, builds a single chair, and immediately forgets the nuances of the wood they used. A master craftsman, however, keeps a detailed journal. In this journal, they document every project: what type of joinery worked, how a specific varnish reacted to humidity, the client’s unique aesthetic preferences, and the shortcuts that saved hours of labor.\nBefore starting a new project, the master reviews this journal. After completing it, they write a new entry, updating their techniques. Over decades, this journal becomes an extension of their mind.\nThis is exactly how Hermes operates. Instead of treating memory as a cold database query, Hermes integrates a structured, self-authored \"journal\" directly into an LLM reasoning loop. It is built on three interlocking theoretical pillars:\nLet’s explore how these three pillars are implemented in code.\nMost agent architectures rely on vector databases to handle memory. While vector search is great for retrieving raw documents, it is terrible for maintaining an agent's identity, behavioral guidelines, or deep user relationships. If a user says, \"I prefer concise code reviews with bullet points,\" that shouldn't be a vector chunk that might get retrieved if the cosine similarity is high enough. It should be a core parameter of how the agent behaves.\nHermes solves this by using a structured, LLM-authored plain-text journal split into two files:\nMEMORY.md\n: Factual knowledge about the environment, tasks, and accumulated domain knowledge.USER.md\n: A deep, evolving model of the user's persona, work style, and expectations.This memory is initialized directly when the agent starts:\n# From run_agent.py - Memory initialization\nif not skip_memory:\nmem_config = _agent_cfg.get(\"memory\", {})\nself._memory_enabled = mem_config.get(\"memory_enabled\", False)\nself._user_profile_enabled = mem_config.get(\"user_profile_enabled\", False)\nself._memory_nudge_interval = int(mem_config.get(\"nudge_interval\", 10))\nif self._memory_enabled or self._user_profile_enabled:\nfrom tools.memory_tool import MemoryStore\nself._memory_store = MemoryStore(\nmemory_char_limit=mem_config.get(\"memory_char_limit\", 2200),\nuser_char_limit=mem_config.get(\"user_char_limit\", 1375),\n)\nself._memory_store.load_from_disk()\nInstead of waiting for the user to explicitly say \"remember this,\" Hermes autonomously decides what is worth preserving. It does this using a nudge interval (memory.nudge_interval\n). Every $N$ turns, the agent triggers a background review process guided by the following system prompt:\n_MEMORY_REVIEW_PROMPT = (\n\"Review the conversation above and consider saving to memory if appropriate.\\n\\n\"\n\"Focus on:\\n\"\n\"1. Has the user revealed things about themselves — their persona, desires, \"\n\"preferences, or personal details worth remembering?\\n\"\n\"2. Has the user expressed expectations about how you should behave, their work \"\n\"style, or ways they want you to operate?\\n\\n\"\n\"If something stands out, save it using the memory tool. \"\n\"If nothing is worth saving, just say 'Nothing to save.' and stop.\"\n)\nCrucially, this review does not block the user interface. A self-improving agent must not sacrifice real-time performance for the sake of learning. Hermes spawns this review in a background thread, cloning its state so the user never experiences latency:\ndef _spawn_background_review(self, messages_snapshot, review_memory=False, review_skills=False):\n\"\"\"Spawn a background thread to review the conversation for memory/skill saves.\"\"\"\ndef _run_review():\nreview_agent = AIAgent(\nmodel=self.model,\nmax_iterations=16,\nquiet_mode=True,\nplatform=self.platform,\nprovider=self.provider,\napi_mode=_parent_runtime.get(\"api_mode\"),\nbase_url=_parent_runtime.get(\"base_url\"),\napi_key=_parent_runtime.get(\"api_key\"),\ncredential_pool=getattr(self, \"_credential_pool\", None),\nparent_session_id=self.session_id,\nenabled_toolsets=[\"memory\", \"skills\"],\n)\nreview_agent._memory_store = self._memory_store\nreview_agent.run_conversation(user_message=prompt, conversation_history=messages_snapshot)\n# ... collect actions and print summary\nt = threading.Thread(target=_run_review, daemon=True, name=\"bg-review\")\nt.start()\nBy directly inheriting the parent’s _memory_store\n, the background agent writes directly to disk. On the very next turn, this updated markdown is injected straight into the system prompt:\n# From _build_system_prompt - Memory injection\nif self._memory_store:\nif self._memory_enabled:\nmem_block = self._memory_store.format_for_system_prompt(\"memory\")\nif mem_block:\nprompt_parts.append(mem_block)\nif self._user_profile_enabled:\nuser_block = self._memory_store.format_for_system_prompt(\"user\")\nif user_block:\nprompt_parts.append(user_block)\nThe second pillar of Hermes is the Closed Learning Loop. Traditional reinforcement learning requires complex external reward functions, and supervised learning requires massive labeled datasets. Hermes bypasses both by using its own tool execution outcomes as a real-time feedback signal.\nThe loop consists of four rapid-fire stages:\nThis execution loop is managed inside run_conversation()\n:\nwhile (api_call_count < self.max_iterations and self.iteration_budget.remaining > 0) or self._budget_grace_call:\n# 1. Build API messages from current history\n# 2. Call the model (streaming or non-streaming)\n# 3. Parse tool calls from the response\n# 4. Execute tool calls, append results to messages\n# 5. Continue loop\nBecause the raw tool outcomes—including stack traces and bash errors—are appended directly to the conversation history, the LLM uses its in-context learning capabilities to debug itself on the fly.\nTo prevent the agent from getting stuck in infinite loops (e.g., repeatedly reading the same directory when a search fails), Hermes uses a ToolCallGuardrailController\n. If the model repeats a non-progressing action, the guardrail intervenes and injects a synthetic warning:\n# From run_agent.py - Guardrail observation after tool execution\nfunction_result = self._append_guardrail_observation(\nfunction_name, function_args, function_result, failed=is_error,\n)\nAdditionally, Hermes enforces an IterationBudget\n. This class acts as a form of \"learning pressure,\" forcing the agent to be highly strategic with its tool usage:\nclass IterationBudget:\n\"\"\"Thread-safe iteration counter for an agent.\"\"\"\ndef __init__(self, max_total: int):\nself.max_total = max_total\nself._used = 0\nself._lock = threading.Lock()\ndef consume(self) -> bool:\nwith self._lock:\nif self._used >= self.max_total:\nreturn False\nself._used += 1\nreturn True\ndef refund(self) -> None:\nwith self._lock:\nif self._used > 0:\nself._used -= 1\nIf an action is highly efficient (like programmatically running code), the budget can be refunded. If the agent wastes LLM calls on repetitive tasks, it quickly runs out of budget and is forced to summarize and exit. This design teaches the agent to prioritize high-leverage tools.\nAs the conversation grows, it risks overflowing the LLM's context window. Instead of hard-truncating old messages (which causes catastrophic forgetting), Hermes uses a ContextCompressor\n.\nAn auxiliary LLM reviews the middle turns of the conversation and summarizes them into a highly dense semantic paragraph, discarding raw execution logs but retaining the core decisions, discoveries, and outcomes:\n# From _compress_context\nmessages, active_system_prompt = self._compress_context(\nmessages, system_message, approx_tokens=approx_tokens,\ntask_id=effective_task_id,\n)\nWhile the closed learning loop handles short-term adaptation within a single conversation, reflection is how Hermes achieves long-term, structural self-improvement.\nAfter a conversation ends, Hermes reviews its performance to extract procedural knowledge. It doesn't just ask \"What did I learn about this user?\"—it asks \"What did I learn about how to solve this class of problem?\"\nThese procedural insights are compiled into Skills. Skills are structured directories stored in ~/.hermes/skills/\ncontaining:\nSKILL.md\n: A step-by-step markdown guide explaining how to perform a task.references/\n: Session-specific details or condensed knowledge bases.templates/\n: Starter files and boilerplates.scripts/\n: Statically re-runnable scripts.The background agent is pushed to actively seek out these learning opportunities via the _SKILL_REVIEW_PROMPT\n:\n_SKILL_REVIEW_PROMPT = (\n\"Review the conversation above and update the skill library. Be \"\n\"ACTIVE — most sessions produce at least one skill update, even if \"\n\"small. A pass that does nothing is a missed learning opportunity, \"\n\"not a neutral outcome.\\n\\n\"\n# ... detailed guidance on when to update/create skills\n\"If you notice two existing skills that overlap, note it in your \"\n\"reply — the background curator handles consolidation at scale.\\n\\n\"\n\"'Nothing to save.' is a real option but should NOT be the \"\n\"default. If the session ran smoothly with no corrections and \"\n\"produced no new technique, just say 'Nothing to save.' and stop. \"\n\"Otherwise, act.\"\n)\nThis prompt programs a growth mindset directly into the agent. It treats mistakes, user corrections, and unexpected workarounds not as failures, but as first-class signals to update its procedural skills.\nIf an agent continuously creates skills, its library will eventually suffer from bloat, duplicate skills, and outdated APIs. To solve this, Hermes includes a background system service called the Curator\n(agent/curator.py\n).\nThe Curator runs periodically to analyze the skill library, merge overlapping skills, and archive stale ones:\n# From AGENTS.md - Curator invariants\n# Invariants:\n# - Curator only touches skills with created_by: \"agent\" provenance —\n# bundled + hub-installed skills are off-limits.\n# - Never deletes; max destructive action is archive.\n# - Pinned skills are exempt from every auto-transition and from the\n# LLM review pass.\nBy enforcing these safety invariants, Hermes ensures that self-improvement never results in the accidental deletion of core system tools or user-pinned workflows.\nWhen you put these three pillars together, you get a continuous, self-reinforcing learning loop:\n[1. Interaction] ──> User talks to Agent, executing tools in real-time.\n│\n▼\n[2. Closed-Loop Adaptation] ──> Agent debugs tool errors within the turn.\n│\n▼\n[3. Persistent Memory] ──> Background thread saves facts to MEMORY.md / USER.md.\n│\n▼\n[4. Skill Update] ──> Reflection creates/updates procedural guides in SKILL.md.\n│\n▼\n[5. Curator Maintenance] ──> Background curator merges, prunes, and archives skills.\n│\n▼\n[6. Next Turn] ──> Updated memory and skills are injected into the next prompt.\nThis cycle allows the agent", "url": "https://wpnews.pro/news/beyond-the-prompt-how-to-build-an-ai-agent-that-actually-learns-from-its", "canonical_source": "https://dev.to/programmingcentral/beyond-the-prompt-how-to-build-an-ai-agent-that-actually-learns-from-its-mistakes-4n1j", "published_at": "2026-05-22 20:00:00+00:00", "updated_at": "2026-05-22 20:34:28.633450+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "open-source", "developer-tools"], "entities": ["Hermes", "Hermes Agent", "Groundhog Day", "Retrieval-Augmented Generation", "RAG"], "alternates": {"html": "https://wpnews.pro/news/beyond-the-prompt-how-to-build-an-ai-agent-that-actually-learns-from-its", "markdown": "https://wpnews.pro/news/beyond-the-prompt-how-to-build-an-ai-agent-that-actually-learns-from-its.md", "text": "https://wpnews.pro/news/beyond-the-prompt-how-to-build-an-ai-agent-that-actually-learns-from-its.txt", "jsonld": "https://wpnews.pro/news/beyond-the-prompt-how-to-build-an-ai-agent-that-actually-learns-from-its.jsonld"}}