{"slug": "the-anatomy-of-an-ai-on-call-agent-why-the-next-incident-responder-wont-be-human", "title": "The Anatomy of an AI On-Call Agent: Why the Next Incident Responder Won’t Be Human", "summary": "Fintech company Ramp deployed an autonomous AI on-call agent called OCA that assisted with 575 merged fixes across 1,220 incident pull requests over four weeks, cutting human investigation time by roughly 50% and requiring 37% fewer human engineers per incident. The agent delivers an interim hypothesis to Slack within five minutes of an alert and opens a drafted fix pull request within thirteen minutes. The architecture relies on just-in-time prompting via execution hooks rather than large monolithic system prompts, using a three-loop design to counter LLM premature closure.", "body_md": "Every software engineer who has ever carried a pager knows the particular dread of a 3:15 AM alert.\n\nYour phone violently buzzes on the nightstand. You stumble to your desk in the dark, squinting through bleary eyes at a blinding terminal screen. PagerDuty tells you that `payments-service-prod` is throwing a 500-error spike.\n\nNow the clock is ticking, and the most frustrating part of incident response begins: **the twenty-minute log-hunting tax.**\n\nYou open six browser tabs. You pull up Datadog to check error graphs. You check CloudWatch for CPU throttling. You pull up GitHub to see what got merged into `main` over the last four hours. You query Postgres read-replicas to see if a database migration locked a critical table.\n\nYou haven’t fixed anything yet. You haven’t even written a single line of code. You’ve just burned twenty minutes acting as an over-caffeinated, human data-routing cable.\n\nFor years, the industry assumed that the only answer to this problem was throwing more engineers at the rotation and paying hundreds of thousands of dollars in SRE overtime.\n\nLast week, fintech unicorn **Ramp** revealed how they radically shifted this paradigm.\n\nThey deployed an autonomous AI agent called **OCA** (On-Call Assistant). Across four weeks and 1,220 incident pull requests, OCA assisted with **575 merged fixes**. More importantly, incidents handled with agent assistance required **37% fewer human engineers** and cut total human investigation time by roughly **50%**.\n\nWithin five minutes of an alert firing, the agent delivers an interim hypothesis to Slack. Within thirteen minutes, it opens a drafted pull request with the code fix waiting for human review.\n\nBeneath Ramp’s proprietary stack lies an architectural pattern that every modern engineering team will be building over the next two years.\n\nHere is the universal architecture of an autonomous incident responder, the hidden failure modes corporate blogs don’t talk about, and the blueprint to build your own.\n\n## 1. The Real Breakthrough: Stop Bloating Your System Prompts\n\nWhen teams first try to build an “AI DevOps Assistant,” they almost always make the same fatal design mistake: **they write a massive 20,000-token system prompt.**\n\nThey stuff the prompt with:\n\n- Every runbook in the company wiki.\n- Formatting guidelines for Slack messages.\n- Rules on how to query Datadog.\n- Warnings not to hallucinate database schemas.\n\nAnd every engineer who has deployed an agent with a giant, monolithic prompt knows what happens next: **the model gets confused, ignores half the instructions, and hallucinates confident nonsense.**\n\nWorse, LLMs have an innate cognitive flaw known as **premature closure**. The moment an LLM spots a single plausible explanation, such as a deploy that went out 20 minutes ago, it immediately stops investigating. It constructs a convincing story blaming that deploy, posts it to the channel, and sends human responders down a 45-minute wild goose chase.\n\nRamp’s key insight wasn’t using a bigger model. It was adopting **Just-in-Time (JIT) Prompting via Execution Hooks.**\n\nInstead of telling the agent 50 rules before it starts, the agent harness operates like a proactive flight controller. It stays silent while the agent gathers data. But the second the agent tries to post its final findings to Slack, an execution hook steps in front of the door and issues a **dynamic detour**:\n\n*“STOP. Before you post to Slack, you must run the reasoning-traps checklist: Have you confirmed this error didn’t exist prior to the deploy? Have you cross-referenced your theory against the database read-replica?”*\n\nThis changes everything. You aren’t praying that the model remembers page 14 of your system prompt; you are intercepting the model at the exact millisecond of action.\n\n## 2. The 3-Loop Architecture\n\nTo make an autonomous incident agent work reliably without burning tokens or taking down production, the system must be decoupled into three distinct loops:\n\n### Loop 1: The Parallel Investigation Sandbox\n\nThe agent must never run on raw production infrastructure. It should be spawned inside an ephemeral container containing a clean checkout of your codebase, equipped with read-only API tools:\n\n- **Log Tool:** Queries Elasticsearch, CloudWatch, or Datadog.\n- **Git Tool:** Examines commit histories, PR diffs, and blame files.\n- **Read-Replica DB Tool:** Inspects table schemas and active query locks.\n\n### Loop 2: The JIT Hook Detours (The Safety Valve)\n\nWhen the agent formulates a theory, it tries to call its `post_to_slack` tool. This is where your middleware intercepts execution. In Ramp’s implementation, they created **six sequential detours**:\n\n1. *Time Window Calibration:* Checking if the query lookback window was wide enough.\n2. *Database Grounding:* Cross-referencing findings against Postgres state.\n3. *Reasoning Traps Checklist:* Forcing the model to disprove its own theory.\n4. *Sanity Check:* Verifying metric units (e.g. milliseconds vs. seconds).\n5. *Formatting Review:* Ensuring clean Slack markdown formatting.\n6. *Action Proposal:* Verifying that a reproducible remediation plan exists.\n\nCrucially, these detours must be **one-shot detours**. If you strictly block an agent until it satisfies a rigid validator, the agent gets trapped in an infinite retry loop, burning compute until it hits a timeout. By allowing the agent through after attempting the detours, you get 90% of the safety benefit with zero deadlock risk.\n\n### Loop 3: The Action & Human Approval Gate\n\nThe agent is an investigator, not a dictator. It outputs two artifacts:\n\n1. **The 5-Minute Interim Slack Update:** A plain-English summary:*“Here is what broke, here is the probable root cause, here are the logs supporting this claim, and here is what I am investigating next.”*\n2. **The Draft Pull Request:** A sandboxed branch containing the proposed unit test and hotfix.\n\nA human engineer clicks the final green button. The human remains the pilot; the agent is the automated radar and co-pilot.\n\n## 3. The 3 Production Scars: Where Naive Setups Fail\n\nIf you attempt to deploy an on-call agent using off-the-shelf tutorials, here are the three production traps that will break your system within the first week:\n\n### Scar #1: The 1-Hour Lookback Illusion\n\nMost log-monitoring APIs default to a 60-minute query window. If an incident fires at 2:00 PM, an agent queries for error spikes, sees the earliest timestamp at 1:40 PM, and instinctively deduces: *“The issue started at 1:40 PM.”* It immediately blames a benign deploy pushed at 1:38 PM.\n\nIn reality, the underlying database index corruption had been silently degrading query performance for three days, and 1:40 PM was simply when traffic crested the failure threshold.\n\n- **The Guardrail:** Program your logging tool to automatically run two queries for every error: the short window (last 60 minutes) and a baseline control window (same 60-minute period 7 days ago).\n\n### Scar #2: Read-Replica “Denial of Service”\n\nGiving an autonomous agent a SQL query tool connected to your database is dangerous, not because it will drop a table (you should always use read-only credentials), but because LLMs love writing unindexed `SELECT * FROM events WHERE ...` queries with three wildcards.\n\nDuring a severe P0 incident when database CPU is already spiked, an agent firing six unindexed queries can easily take down the read-replica cluster.\n\n- **The Guardrail:** Force all agent-generated SQL queries through a strict query planner with a mandatory**500-millisecond timeout** and a hard limit on scanned rows.\n\n### Scar #3: The PII Leak in Public Channels\n\nIncident channels often have 30+ stakeholders watching: engineers, product managers, support leads, and executives. When an agent pulls raw exception traces containing customer email addresses, API tokens, or credit card metadata and dumps them into Slack, you have instantly created a compliance violation.\n\n- **The Guardrail:** A deterministic regex/NER redaction layer must sit between the agent’s reasoning engine and the Slack webhook. Zero raw payloads reach public channels.\n\n## 4. How to Prototype This Weekend (The Minimal Blueprint)\n\nYou don’t need a team of 200 engineers or custom infrastructure to build a working prototype. You can build a functioning version using standard, accessible tools:\n\n```\n# The Minimal 3-Node Incident Graph (Conceptual State Machine)\nfrom typing import TypedDict, List\nfrom langgraph.graph import StateGraph, END\n\nclass IncidentState(TypedDict):\n    incident_id: str\n    error_logs: List[str]\n    recent_commits: List[str]\n    hypothesis: str\n    verified: bool\n    draft_pr_url: str\n\ndef ingest_alert(state: IncidentState):\n    # 1. Fetch recent logs from CloudWatch/Datadog\n    # 2. Fetch git commits from GitHub API over the last 3 hours\n    return {\"error_logs\": [\"500 KeyError: 'user_id' in /checkout\"], \"recent_commits\": [\"Fix checkout payload (#412)\"]}\n\ndef investigate_and_hypothesize(state: IncidentState):\n    # LLM correlates git diff with stack trace\n    # Prompt: \"Given this diff and this error, identify the root cause.\"\n    return {\"hypothesis\": \"PR #412 changed 'userId' to 'user_id' breaking legacy clients.\"}\n\ndef jit_verification_hook(state: IncidentState):\n    # Dynamic Detour: Check if the hypothesis is grounded\n    # If not verified, route to self-correction before posting\n    return {\"verified\": True}\n\ndef post_slack_and_open_pr(state: IncidentState):\n    # Post interim hypothesis to Slack & create draft branch\n    return {\"draft_pr_url\": \"https://github.com/org/repo/pull/415\"}\n\n# Wire the graph\nworkflow = StateGraph(IncidentState)\nworkflow.add_node(\"ingest\", ingest_alert)\nworkflow.add_node(\"investigate\", investigate_and_hypothesize)\nworkflow.add_node(\"verify\", jit_verification_hook)\nworkflow.add_node(\"act\", post_slack_and_open_pr)\n\nworkflow.set_entry_point(\"ingest\")\nworkflow.add_edge(\"ingest\", \"investigate\")\nworkflow.add_edge(\"investigate\", \"verify\")\nworkflow.add_edge(\"verify\", \"act\")\nworkflow.add_edge(\"act\", END)\n```\n\nWith just three nodes (**Ingest, Investigate, and Verify**), you have the foundational loop of an on-call agent.\n\n## 5. The Strategic Bottom Line\n\nFor engineering leaders, the ROI of on-call agents isn’t about eliminating headcount.\n\nIt is about **protecting engineering momentum**.\n\nWhen your senior engineers spend 40% of their on-call shift digging through messy JSON logs at 3:00 AM, the true cost isn’t just the downtime. The real cost is the cognitive burnout, the missed sprint goals, and the resignation letters three months later.\n\nBy turning the on-call engineer from a **manual log-sleuth** into an **executive code-reviewer**, you transform the worst part of software engineering into a 5-minute confirmation workflow.\n\nThe future of production reliability isn’t faster humans reading logs. It’s autonomous systems that do the digging before the human even opens their laptop.\n\n### Further Reading & Resources\n\n- **[Ramp Builders: How We Built OCA](https://builders.ramp.com/post/how-we-built-oca-our-ai-on-call-assistant)** : The original engineering post detailing their 4-week incident metrics and on-call assistant architecture.\n- **[Claude Code: Sub-Agents & Tool Hooks](https://code.claude.com/docs/en/hooks)** : Official documentation on implementing Just-in-Time execution hooks and detours.\n- **[LangGraph State Machines](https://langchain-ai.github.io/langgraph/)** : A lightweight framework for implementing human-in-the-loop validation gates and multi-node agent triage workflows.\n- **[Keep: Open-Source AIOps & Alert Orchestration](https://github.com/keephq/keep)** : An open-source alert management platform designed to trigger custom agent workflows and deduplicate incident telemetry.\n\n*If you enjoyed this breakdown, subscribe to **[MLnotes](https://mlnotes.substack.com/)** for weekly, bite-sized systems engineering and AI architecture deep-dives. If your team is rethinking its on-call rotation, share this article with your lead.*", "url": "https://wpnews.pro/news/the-anatomy-of-an-ai-on-call-agent-why-the-next-incident-responder-wont-be-human", "canonical_source": "https://mlnotes.substack.com/p/the-anatomy-of-an-ai-on-call-agent", "published_at": "2026-09-22 15:53:34+00:00", "updated_at": "2026-09-22 16:27:14.595935+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-products", "developer-tools", "mlops"], "entities": ["Ramp", "OCA", "Slack", "Datadog", "CloudWatch", "GitHub", "Postgres", "PagerDuty"], "alternates": {"html": "https://wpnews.pro/news/the-anatomy-of-an-ai-on-call-agent-why-the-next-incident-responder-wont-be-human", "markdown": "https://wpnews.pro/news/the-anatomy-of-an-ai-on-call-agent-why-the-next-incident-responder-wont-be-human.md", "text": "https://wpnews.pro/news/the-anatomy-of-an-ai-on-call-agent-why-the-next-incident-responder-wont-be-human.txt", "jsonld": "https://wpnews.pro/news/the-anatomy-of-an-ai-on-call-agent-why-the-next-incident-responder-wont-be-human.jsonld"}}