{"slug": "the-fix-was-already-in-our-postmortems-i-built-an-on-call-agent-that-actually-it", "title": "The Fix Was Already in Our Postmortems. I Built an On-Call Agent That Actually Remembers It.", "summary": "A developer and a collaborator built RecallOps, an incident-response copilot that stores every past incident in a persistent memory layer so the agent can recall which fixes worked and which failed. The memory layer is built on Hindsight, Vectorize's open-source agent memory system, using its three-verb recall/reflect/retain API, with one memory bank per on-call organization rather than per service to capture cross-service links. The developer reports that a copilot without memory will re-suggest scaling pods even when prior postmortems record that scaling did nothing.", "body_md": "Here's what surprised me most about incident response. The fix for many outages is already written down, usually in a postmortem from three weeks ago, and the on-call engineer still spends twenty minutes trying the wrong thing first.\n\nIn the synthetic incident history I made for this project, the same payments service fails three times in two months. All three times it looks the same: HTTP 5xx on `/v2/charge`, right after a deploy. And in two of those three postmortems, someone writes almost the same sentence: *scaling the pods did nothing.* A copilot without memory will suggest scaling the pods again, because nothing tells it that it was already tried.\n\nSo my friend Faraz and I built **RecallOps**, an incident-response copilot that remembers every past incident. My half of the work was the memory layer: how incidents get stored, how they come back, and how the agent gets better after every resolution. This post is about that part.\n\nCode: [https://github.com/Madhan-raj-k/recall-ops](https://github.com/Madhan-raj-k/recall-ops)\n\nLive site: [https://recall-ops-three.vercel.app](https://recall-ops-three.vercel.app)\n\nAn LLM is good at reasoning about an alert. What it can't know is *your* history:\n\n`payments-api` 5xx after a deploy has meant a bad SDK timeout, a shrunken DB connection pool, and a stale Vault secret,`argocd app rollback payments-api` fixed it in minutes,\nStuffing every postmortem into the prompt doesn't scale, and plain vector search over documents loses things like *when* something happened, *which entities* connect two incidents, and which fix *failed*. I wanted real agent memory, not a pile of embeddings. (If the distinction is new to you, Vectorize has a good explainer: [What is agent memory?](https://vectorize.io/what-is-agent-memory))\n\nI used [Hindsight](https://github.com/vectorize-io/hindsight), an open-source agent memory system from Vectorize. Its API is deliberately small: three verbs.\n\nThose three verbs map almost exactly onto an incident lifecycle: an alert comes in (recall), the engineer needs judgement (reflect), the incident is resolved (retain). The [Hindsight docs](https://hindsight.vectorize.io/) cover the full model. I used the Python SDK, `hindsight-client` 1.2.0, against Hindsight Cloud.\n\nI kept every Hindsight call in one file, `memory.py`. The rest of the app only sees an `IncidentMemory` interface, which made testing much easier (more on that below).\n\n**Bank design.** There is one memory bank per on-call organisation (`recallops-shopnova-oncall`), not one per service. That was deliberate: the interesting links cross services. A DNS change breaks inventory, or a Redis job evicts carts. When the bank is first used, I configure it with two missions:\n\n```\nself._client.create_bank(\n    bank_id=self.bank_id,\n    retain_mission=RETAIN_MISSION,\n    reflect_mission=REFLECT_MISSION,\n)\n```\n\nThe retain mission tells extraction what matters in a postmortem: the service, symptoms, root cause, the fix that worked, the fix that *did not* work, teams, runbooks and time to resolve. It also says to keep exact technical names. The reflect mission makes reflect behave like a senior SRE: cite incident IDs, prefer fixes that worked, warn about the ones that failed.\n\n**What gets stored.** Each resolved incident becomes one retained item:\n\n``` php\ndef _item(incident: Incident) -> dict:\n    return {\n        \"content\": incident.to_memory_text(),\n        \"timestamp\": incident.occurred_at,\n        \"context\": f\"postmortem for {incident.service} incident\",\n        \"document_id\": incident.incident_id,\n        \"metadata\": incident.metadata(),\n        \"tags\": incident.tags(),\n    }\n```\n\nThree choices here mattered more than I expected:\n\n`timestamp` is when the incident happened`document_id` is the incident ID`service:payments-api`\nThe content itself is a plain narrative (\"Root cause: … Fix that worked: … Attempts that did NOT work: …\"). I don't do any structuring myself. Extraction is Hindsight's job.\n\n**Recall on every alert.** When triage runs with memory on:\n\n```\nresp = self._client.recall(\n    bank_id=self.bank_id,\n    query=query,\n    types=[\"world\", \"experience\", \"observation\"],\n    max_tokens=2048,\n    budget=\"mid\",\n)\n```\n\nRecall is cut to a token budget rather than a top-k count, and that turned out to be exactly right for an agent. I know how much prompt space I can spare, and Hindsight fills it. The recalled facts, with dates and source incident IDs, go into the Groq LLM prompt and into a \"memories recalled\" panel, so the engineer can see *why* the agent suggests something.\n\n**Reflect for judgement.** On top of raw recall, I ask the bank one question: *what most likely caused this, what fixed it before, and what should on-call avoid?* That reflect answer is shown to the engineer and passed to the LLM too.\n\nThis is the part I'm proudest of. After the incident, the engineer fills in a short form: root cause, what fixed it, what didn't. `agent.record_resolution()` validates it and retains it straight into the same bank. No retraining and no re-indexing.\n\nThere's a unit test for exactly this loop, and it's the whole idea of the project: the agent gets better because it remembers.\n\nThen I ran it live against Hindsight Cloud. Before the postmortem, the `loyalty-points` alert only recalled loosely related incidents, and the plan guessed \"database contention / connection-pool exhaustion (see INC‑1003)\". I retained one postmortem through `record_resolution()`, and the very next recall, with no waiting, returned it as the top memories. The plan opened with \"Postgres lock contention on `points_ledger` … (see INC‑20261109‑71C2)\" and recommended the replica-and-batching fix.\n\nSame alert, same LLM, same prompt. The only difference is memory:\n\n`PagerDuty SEV1: payments-api 5xx rate 22% on /v2/charge, started 6 minutes after deploy v3.21.0`\n\nThese are trimmed quotes from a live run against Hindsight Cloud with `openai/gpt-oss-120b` on Groq:\n\n**Memory OFF:** \"Regression introduced in `v3.21.0` causing unhandled exceptions in the `/v2/charge` handler.\" Then grep the logs and diff the release. Confidence: \"Low – no prior incident data.\" It's sensible, but it's generic.\n\n**Memory ON:** \"Configuration regression introduced in v3.21.0 – e.g., Hikari `maximumPoolSize` reduced (as in INC‑1016)\", followed by the Stripe timeout (INC‑1001) and stale Vault secret (INC‑1013) hypotheses. It checks `maximumPoolSize`, `STRIPE_HTTP_TIMEOUT` and the Vault secret version before `argocd app rollback payments-api`, and it adds the DBA team. Then the line that matters most: \"Avoid: Scaling the payments‑api pods (worsened DB pool exhaustion in INC‑1016). Blind pod restarts (only gave temporary relief in INC‑1001 & INC‑1016).\"\n\nReflect, on its own, came back with a \"What to Avoid (Failed Fixes)\" section: \"Do NOT scale pods: in both INC-1001 and INC-1016, scaling pods (e.g., from 6 to 12) failed to resolve the issue.\"\n\nIf Hindsight or Groq fails, triage carries on and shows a warning. Groq calls retry with back-off, and there's a deterministic fallback plan. The 24 unit tests run against a fake Hindsight client, so CI needs no keys.\n\nThe first live run taught me something I hadn't expected: recall for the payments alert returned 80 small extracted facts, not 3 incidents. That's Hindsight filling the token budget with granular facts and consolidated observations, which is great for coverage but bloats the prompt. The fix was a one-line cap: `MAX_PROMPT_MEMORIES = 30` goes to the LLM, and the UI still shows everything.\n\nMy first instinct was also to parse \"what worked\" and \"what failed\" into neat structured fields myself. I dropped that idea: real postmortems are messy, and hand-written parsing would throw away detail. Retaining the narrative and letting Hindsight extract the facts is the better design. It has a cost, though. Recall returns *extracted facts*, not my original lines, so my no-LLM fallback (which looks for \"Fix that worked:\") is weaker on real Hindsight data than on the offline demo store.\n\nThe bigger limitation is human. Memory is only as good as what gets retained. A postmortem that says \"restarted stuff, it worked\" teaches the agent nothing. And similar symptoms can have different causes; our payments data has three. RecallOps ranks hypotheses, it doesn't prove one, so the engineer still has to verify.\n\n`pip install -r requirements.txt`, `python seed.py`, `streamlit run app.py`)\nBuilt with Faraz. The incident data in the repo is synthetic, about a fictional store called ShopNova.", "url": "https://wpnews.pro/news/the-fix-was-already-in-our-postmortems-i-built-an-on-call-agent-that-actually-it", "canonical_source": "https://dev.to/madhan_rajmadhan_f6030ac/the-fix-was-already-in-our-postmortems-i-built-an-on-call-agent-that-actually-remembers-it-23ag", "published_at": "2026-11-09 17:19:15+00:00", "updated_at": "2026-11-09 17:26:18.241468+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "mlops", "artificial-intelligence"], "entities": ["RecallOps", "Hindsight", "Vectorize", "Faraz", "GitHub", "Vercel"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-fix-was-already-in-our-postmortems-i-built-an-on-call-agent-that-actually-it", "markdown": "https://wpnews.pro/news/the-fix-was-already-in-our-postmortems-i-built-an-on-call-agent-that-actually-it.md", "text": "https://wpnews.pro/news/the-fix-was-already-in-our-postmortems-i-built-an-on-call-agent-that-actually-it.txt", "jsonld": "https://wpnews.pro/news/the-fix-was-already-in-our-postmortems-i-built-an-on-call-agent-that-actually-it.jsonld"}}