Here's what surprised me most about incident response. The fix for many outages is already written down, usually in a postmortem from three weeks ago, and the on-call engineer still spends twenty minutes trying the wrong thing first.
In the synthetic incident history I made for this project, the same payments service fails three times in two months. All three times it looks the same: HTTP 5xx on /v2/charge, right after a deploy. And in two of those three postmortems, someone writes almost the same sentence: scaling the pods did nothing. A copilot without memory will suggest scaling the pods again, because nothing tells it that it was already tried.
So my friend Faraz and I built RecallOps, an incident-response copilot that remembers every past incident. My half of the work was the memory layer: how incidents get stored, how they come back, and how the agent gets better after every resolution. This post is about that part.
Code: https://github.com/Madhan-raj-k/recall-ops
Live site: https://recall-ops-three.vercel.app
An LLM is good at reasoning about an alert. What it can't know is your history:
payments-api 5xx after a deploy has meant a bad SDK timeout, a shrunken DB connection pool, and a stale Vault secret,argocd app rollback payments-api fixed it in minutes,
Stuffing every postmortem into the prompt doesn't scale, and plain vector search over documents loses things like when something happened, which entities connect two incidents, and which fix failed. I wanted real agent memory, not a pile of embeddings. (If the distinction is new to you, Vectorize has a good explainer: What is agent memory?)
I used Hindsight, an open-source agent memory system from Vectorize. Its API is deliberately small: three verbs.
Those three verbs map almost exactly onto an incident lifecycle: an alert comes in (recall), the engineer needs judgement (reflect), the incident is resolved (retain). The Hindsight docs cover the full model. I used the Python SDK, hindsight-client 1.2.0, against Hindsight Cloud.
I kept every Hindsight call in one file, memory.py. The rest of the app only sees an IncidentMemory interface, which made testing much easier (more on that below).
Bank design. There is one memory bank per on-call organisation (recallops-shopnova-oncall), not one per service. That was deliberate: the interesting links cross services. A DNS change breaks inventory, or a Redis job evicts carts. When the bank is first used, I configure it with two missions:
self._client.create_bank(
bank_id=self.bank_id,
retain_mission=RETAIN_MISSION,
reflect_mission=REFLECT_MISSION,
)
The retain mission tells extraction what matters in a postmortem: the service, symptoms, root cause, the fix that worked, the fix that did not work, teams, runbooks and time to resolve. It also says to keep exact technical names. The reflect mission makes reflect behave like a senior SRE: cite incident IDs, prefer fixes that worked, warn about the ones that failed.
What gets stored. Each resolved incident becomes one retained item:
def _item(incident: Incident) -> dict:
return {
"content": incident.to_memory_text(),
"timestamp": incident.occurred_at,
"context": f"postmortem for {incident.service} incident",
"document_id": incident.incident_id,
"metadata": incident.metadata(),
"tags": incident.tags(),
}
Three choices here mattered more than I expected:
timestamp is when the incident happeneddocument_id is the incident IDservice:payments-api
The content itself is a plain narrative ("Root cause: … Fix that worked: … Attempts that did NOT work: …"). I don't do any structuring myself. Extraction is Hindsight's job.
Recall on every alert. When triage runs with memory on:
resp = self._client.recall(
bank_id=self.bank_id,
query=query,
types=["world", "experience", "observation"],
max_tokens=2048,
budget="mid",
)
Recall is cut to a token budget rather than a top-k count, and that turned out to be exactly right for an agent. I know how much prompt space I can spare, and Hindsight fills it. The recalled facts, with dates and source incident IDs, go into the Groq LLM prompt and into a "memories recalled" panel, so the engineer can see why the agent suggests something.
Reflect for judgement. On top of raw recall, I ask the bank one question: what most likely caused this, what fixed it before, and what should on-call avoid? That reflect answer is shown to the engineer and passed to the LLM too.
This is the part I'm proudest of. After the incident, the engineer fills in a short form: root cause, what fixed it, what didn't. agent.record_resolution() validates it and retains it straight into the same bank. No retraining and no re-indexing.
There's a unit test for exactly this loop, and it's the whole idea of the project: the agent gets better because it remembers.
Then I ran it live against Hindsight Cloud. Before the postmortem, the loyalty-points alert only recalled loosely related incidents, and the plan guessed "database contention / connection-pool exhaustion (see INC‑1003)". I retained one postmortem through record_resolution(), and the very next recall, with no waiting, returned it as the top memories. The plan opened with "Postgres lock contention on points_ledger … (see INC‑20261109‑71C2)" and recommended the replica-and-batching fix.
Same alert, same LLM, same prompt. The only difference is memory:
PagerDuty SEV1: payments-api 5xx rate 22% on /v2/charge, started 6 minutes after deploy v3.21.0
These are trimmed quotes from a live run against Hindsight Cloud with openai/gpt-oss-120b on Groq:
Memory OFF: "Regression introduced in v3.21.0 causing unhandled exceptions in the /v2/charge handler." Then grep the logs and diff the release. Confidence: "Low – no prior incident data." It's sensible, but it's generic.
Memory ON: "Configuration regression introduced in v3.21.0 – e.g., Hikari maximumPoolSize reduced (as in INC‑1016)", followed by the Stripe timeout (INC‑1001) and stale Vault secret (INC‑1013) hypotheses. It checks maximumPoolSize, STRIPE_HTTP_TIMEOUT and the Vault secret version before argocd app rollback payments-api, and it adds the DBA team. Then the line that matters most: "Avoid: Scaling the payments‑api pods (worsened DB pool exhaustion in INC‑1016). Blind pod restarts (only gave temporary relief in INC‑1001 & INC‑1016)."
Reflect, on its own, came back with a "What to Avoid (Failed Fixes)" section: "Do NOT scale pods: in both INC-1001 and INC-1016, scaling pods (e.g., from 6 to 12) failed to resolve the issue."
If Hindsight or Groq fails, triage carries on and shows a warning. Groq calls retry with back-off, and there's a deterministic fallback plan. The 24 unit tests run against a fake Hindsight client, so CI needs no keys.
The first live run taught me something I hadn't expected: recall for the payments alert returned 80 small extracted facts, not 3 incidents. That's Hindsight filling the token budget with granular facts and consolidated observations, which is great for coverage but bloats the prompt. The fix was a one-line cap: MAX_PROMPT_MEMORIES = 30 goes to the LLM, and the UI still shows everything.
My first instinct was also to parse "what worked" and "what failed" into neat structured fields myself. I dropped that idea: real postmortems are messy, and hand-written parsing would throw away detail. Retaining the narrative and letting Hindsight extract the facts is the better design. It has a cost, though. Recall returns extracted facts, not my original lines, so my no-LLM fallback (which looks for "Fix that worked:") is weaker on real Hindsight data than on the offline demo store.
The bigger limitation is human. Memory is only as good as what gets retained. A postmortem that says "restarted stuff, it worked" teaches the agent nothing. And similar symptoms can have different causes; our payments data has three. RecallOps ranks hypotheses, it doesn't prove one, so the engineer still has to verify.
pip install -r requirements.txt, python seed.py, streamlit run app.py)
Built with Faraz. The incident data in the repo is synthetic, about a fictional store called ShopNova.