# The Fix Was Already in Our Postmortems. I Built an On-Call Agent That Actually Remembers It.

> Source: <https://dev.to/madhan_rajmadhan_f6030ac/the-fix-was-already-in-our-postmortems-i-built-an-on-call-agent-that-actually-remembers-it-23ag>
> Published: 2026-11-09 17:19:15+00:00

Here's what surprised me most about incident response. The fix for many outages is already written down, usually in a postmortem from three weeks ago, and the on-call engineer still spends twenty minutes trying the wrong thing first.

In the synthetic incident history I made for this project, the same payments service fails three times in two months. All three times it looks the same: HTTP 5xx on `/v2/charge`, right after a deploy. And in two of those three postmortems, someone writes almost the same sentence: *scaling the pods did nothing.* A copilot without memory will suggest scaling the pods again, because nothing tells it that it was already tried.

So my friend Faraz and I built **RecallOps**, an incident-response copilot that remembers every past incident. My half of the work was the memory layer: how incidents get stored, how they come back, and how the agent gets better after every resolution. This post is about that part.

Code: [https://github.com/Madhan-raj-k/recall-ops](https://github.com/Madhan-raj-k/recall-ops)

Live site: [https://recall-ops-three.vercel.app](https://recall-ops-three.vercel.app)

An LLM is good at reasoning about an alert. What it can't know is *your* history:

`payments-api` 5xx after a deploy has meant a bad SDK timeout, a shrunken DB connection pool, and a stale Vault secret,`argocd app rollback payments-api` fixed it in minutes,
Stuffing every postmortem into the prompt doesn't scale, and plain vector search over documents loses things like *when* something happened, *which entities* connect two incidents, and which fix *failed*. I wanted real agent memory, not a pile of embeddings. (If the distinction is new to you, Vectorize has a good explainer: [What is agent memory?](https://vectorize.io/what-is-agent-memory))

I used [Hindsight](https://github.com/vectorize-io/hindsight), an open-source agent memory system from Vectorize. Its API is deliberately small: three verbs.

Those three verbs map almost exactly onto an incident lifecycle: an alert comes in (recall), the engineer needs judgement (reflect), the incident is resolved (retain). The [Hindsight docs](https://hindsight.vectorize.io/) cover the full model. I used the Python SDK, `hindsight-client` 1.2.0, against Hindsight Cloud.

I kept every Hindsight call in one file, `memory.py`. The rest of the app only sees an `IncidentMemory` interface, which made testing much easier (more on that below).

**Bank design.** There is one memory bank per on-call organisation (`recallops-shopnova-oncall`), not one per service. That was deliberate: the interesting links cross services. A DNS change breaks inventory, or a Redis job evicts carts. When the bank is first used, I configure it with two missions:

```
self._client.create_bank(
    bank_id=self.bank_id,
    retain_mission=RETAIN_MISSION,
    reflect_mission=REFLECT_MISSION,
)
```

The retain mission tells extraction what matters in a postmortem: the service, symptoms, root cause, the fix that worked, the fix that *did not* work, teams, runbooks and time to resolve. It also says to keep exact technical names. The reflect mission makes reflect behave like a senior SRE: cite incident IDs, prefer fixes that worked, warn about the ones that failed.

**What gets stored.** Each resolved incident becomes one retained item:

``` php
def _item(incident: Incident) -> dict:
    return {
        "content": incident.to_memory_text(),
        "timestamp": incident.occurred_at,
        "context": f"postmortem for {incident.service} incident",
        "document_id": incident.incident_id,
        "metadata": incident.metadata(),
        "tags": incident.tags(),
    }
```

Three choices here mattered more than I expected:

`timestamp` is when the incident happened`document_id` is the incident ID`service:payments-api`
The content itself is a plain narrative ("Root cause: … Fix that worked: … Attempts that did NOT work: …"). I don't do any structuring myself. Extraction is Hindsight's job.

**Recall on every alert.** When triage runs with memory on:

```
resp = self._client.recall(
    bank_id=self.bank_id,
    query=query,
    types=["world", "experience", "observation"],
    max_tokens=2048,
    budget="mid",
)
```

Recall is cut to a token budget rather than a top-k count, and that turned out to be exactly right for an agent. I know how much prompt space I can spare, and Hindsight fills it. The recalled facts, with dates and source incident IDs, go into the Groq LLM prompt and into a "memories recalled" panel, so the engineer can see *why* the agent suggests something.

**Reflect for judgement.** On top of raw recall, I ask the bank one question: *what most likely caused this, what fixed it before, and what should on-call avoid?* That reflect answer is shown to the engineer and passed to the LLM too.

This is the part I'm proudest of. After the incident, the engineer fills in a short form: root cause, what fixed it, what didn't. `agent.record_resolution()` validates it and retains it straight into the same bank. No retraining and no re-indexing.

There's a unit test for exactly this loop, and it's the whole idea of the project: the agent gets better because it remembers.

Then I ran it live against Hindsight Cloud. Before the postmortem, the `loyalty-points` alert only recalled loosely related incidents, and the plan guessed "database contention / connection-pool exhaustion (see INC‑1003)". I retained one postmortem through `record_resolution()`, and the very next recall, with no waiting, returned it as the top memories. The plan opened with "Postgres lock contention on `points_ledger` … (see INC‑20261109‑71C2)" and recommended the replica-and-batching fix.

Same alert, same LLM, same prompt. The only difference is memory:

`PagerDuty SEV1: payments-api 5xx rate 22% on /v2/charge, started 6 minutes after deploy v3.21.0`

These are trimmed quotes from a live run against Hindsight Cloud with `openai/gpt-oss-120b` on Groq:

**Memory OFF:** "Regression introduced in `v3.21.0` causing unhandled exceptions in the `/v2/charge` handler." Then grep the logs and diff the release. Confidence: "Low – no prior incident data." It's sensible, but it's generic.

**Memory ON:** "Configuration regression introduced in v3.21.0 – e.g., Hikari `maximumPoolSize` reduced (as in INC‑1016)", followed by the Stripe timeout (INC‑1001) and stale Vault secret (INC‑1013) hypotheses. It checks `maximumPoolSize`, `STRIPE_HTTP_TIMEOUT` and the Vault secret version before `argocd app rollback payments-api`, and it adds the DBA team. Then the line that matters most: "Avoid: Scaling the payments‑api pods (worsened DB pool exhaustion in INC‑1016). Blind pod restarts (only gave temporary relief in INC‑1001 & INC‑1016)."

Reflect, on its own, came back with a "What to Avoid (Failed Fixes)" section: "Do NOT scale pods: in both INC-1001 and INC-1016, scaling pods (e.g., from 6 to 12) failed to resolve the issue."

If Hindsight or Groq fails, triage carries on and shows a warning. Groq calls retry with back-off, and there's a deterministic fallback plan. The 24 unit tests run against a fake Hindsight client, so CI needs no keys.

The first live run taught me something I hadn't expected: recall for the payments alert returned 80 small extracted facts, not 3 incidents. That's Hindsight filling the token budget with granular facts and consolidated observations, which is great for coverage but bloats the prompt. The fix was a one-line cap: `MAX_PROMPT_MEMORIES = 30` goes to the LLM, and the UI still shows everything.

My first instinct was also to parse "what worked" and "what failed" into neat structured fields myself. I dropped that idea: real postmortems are messy, and hand-written parsing would throw away detail. Retaining the narrative and letting Hindsight extract the facts is the better design. It has a cost, though. Recall returns *extracted facts*, not my original lines, so my no-LLM fallback (which looks for "Fix that worked:") is weaker on real Hindsight data than on the offline demo store.

The bigger limitation is human. Memory is only as good as what gets retained. A postmortem that says "restarted stuff, it worked" teaches the agent nothing. And similar symptoms can have different causes; our payments data has three. RecallOps ranks hypotheses, it doesn't prove one, so the engineer still has to verify.

`pip install -r requirements.txt`, `python seed.py`, `streamlit run app.py`)
Built with Faraz. The incident data in the repo is synthetic, about a fictional store called ShopNova.