OnCallMemory: Building a Persistent AI Incident Response Agent with Hindsight

A team at HackwithHyderabad 3.0 built OnCallMemory, an AI incident-response agent that uses Vectorize Hindsight persistent memory to help on-call engineers reuse knowledge from previously resolved production incidents. The system follows a Retain → Recall → Reflect cycle, storing symptoms, root causes, resolution steps and outcomes after each incident and retrieving relevant past incidents when a new alert arrives. A side-by-side Memory OFF vs Memory ON comparison shows the agent moving from generic troubleshooting suggestions to recalling prior root causes and successful fixes, such as connection-pool exhaustion behind a SEV1 checkout API latency spike.

Introduction Modern engineering teams deal with production incidents that often resemble problems they have already solved. However, incident knowledge is frequently scattered across tickets, logs, documentation and team conversations. This inspired us to build OnCallMemory, an AI incident-response agent that uses persistent memory to help on-call engineers leverage previous incident experience. The Problem A typical AI assistant can analyze the current alert, but without persistent memory it has no knowledge of how similar incidents were handled previously. This can result in generic troubleshooting recommendations even when the engineering team has already solved the same problem before. Our Solution OnCallMemory gives the incident-response agent persistent operational memory using Vectorize Hindsight. The system follows three major memory operations: Retain → Recall → Reflect Retain When an incident is resolved, important information such as symptoms, root cause, resolution steps and outcome is stored in Hindsight. Recall When a new incident arrives, OnCallMemory searches its historical memory for relevant previous incidents. Reflect Hindsight Reflect allows the system to reason across accumulated incident memories and identify recurring operational patterns. Memory OFF vs Memory ON One of the key features of our project is a side-by-side comparison. With Memory OFF, the agent analyzes only the current incident and provides generic troubleshooting suggestions. With Memory ON, Hindsight retrieves relevant historical incidents, allowing the agent to use previous root causes and successful resolution approaches. Example Consider a SEV1 checkout API latency incident where the p99 response time reaches 9200ms and application logs indicate connection-pool starvation. Without historical memory, the agent may identify several possible causes. With Hindsight memory, the agent can recall previous checkout API incidents and identify whether connection-pool exhaustion has previously caused similar failures and what resolution worked. Architecture The system consists of: Streamlit interface AI incident-response agent Vectorize Hindsight memory LLM Incident dataset Hindsight Retain, Recall and Reflect operations Why Persistent Memory Matters The key idea behind OnCallMemory is that every resolved incident can become useful knowledge for future incidents. Instead of starting from zero, the agent can build on previous operational experience. Conclusion OnCallMemory demonstrates how persistent memory can make AI agents more useful for real-world engineering workflows. The goal is simple: turn previous incident experience into persistent operational knowledge. Built for HackwithHyderabad 3.0 using Vectorize Hindsight.