cd /news/ai-agents/incident-context-tracing-production-… · home topics ai-agents article
[ARTICLE · art-135223] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

Incident Context: tracing production failures without guessing

A developer built Incident Context, an incident-investigation agent that models operational records — services, deployments, changes, runbooks, and incidents — as connected Sanity documents and traces explicit relationships between them rather than relying on keyword matches. The agent follows a Sanity Context MCP Knowledge Base to separate confirmed evidence from inference and preserves source paths behind every report, refusing to generate an unsupported fallback answer if Sanity Context is unreachable. In a demo comparing two consecutive-day outages, it traced distinct root causes — a DB_POOL_SIZE reduction for INC-142 and a PAYMENT_TIMEOUT_MS reduction for INC-208 — instead of correlating on shared timeout mentions.

by read5 min views1 publishedSep 20, 2026

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.

Incident Context is an incident-investigation agent for questions that cannot afford a confident but unsupported answer.

During an outage, evidence is rarely contained in one document. It is spread across services and dependencies, deployments and released versions, configuration changes, previous incidents, and version-specific runbooks.

Incident Context models those records as connected Sanity documents. The agent follows their relationships through a Sanity Context MCP Knowledge Base, separates confirmed evidence from inference, and preserves the source paths behind every report.

A keyword search can find "timeout" in both INC-142 and INC-208. But correlation is not causation.

Both incidents were production outages on consecutive days. Both involved configuration changes. Both mentioned timeouts. That is not enough to determine whether they share a root cause or require the same remediation.

The useful answer is not a matching paragraph. It is a supported path across an incident, an affected service, a deployment, a configuration change, and an applicable runbook.

The agent models operational records as a connected graph:

Service    → depends on → Service
Deployment → belongs to → Service
Deployment → includes   → Change
Change     → modifies   → Service
Runbook    → applies to → Service
Incident   → affects    → Service
Incident   → relates to → Deployment
Incident   → relates to → Change
Incident   → references → Runbook

When you ask it to compare the two incidents, the agent follows the structured path, not keyword matches:

INC-208
→ affected checkout-api
→ related deployment checkout-api v2.3.0
→ included PAYMENT_TIMEOUT_MS configuration change
→ reduced from 5000ms to 1500ms

This traversal becomes an evidence trail in the interface:

Notice the evidence trail at the top. It shows the relationships the agent followed: INC-142 affected payment-api and was caused by a DB_POOL_SIZE reduction; INC-208 affected checkout-api and was caused by a PAYMENT_TIMEOUT_MS reduction. Two different paths. Two different root causes.

Below that: 4 confirmed evidence items, 0 inferences, 2 sources. The model did not guess or correlate. It followed the graph.

Operational data contains many correlations:

Those relationships are useful evidence. They are not automatically proof of causation.

The agent preserves the distinction. Every investigation returns:

If Sanity Context cannot be reached, the application does not generate an unsupported fallback answer.

Sanity is not being used as a generic document store behind a chat box. Its references form the investigation graph.

The project defines five document types: service, deployment, change, runbook, incident. Their references encode the operational relationships.

During the build, I included the references required for an investigation:

service.dependencies[] → service`` deployment.service → service``deployment.changes[] → change`` change.service → service``runbook.service → service`` incident.affectedServices[] → service``incident.relatedDeployments[] → deployment`` incident.relatedChanges[] → change``incident.relatedRunbook → runbook Sanity Context distilled those connected documents into navigable, source-linked entries. At investigation time, the agent reads the Knowledge Base outline through initial_context, selects relevant entries, retrieves them through Knowledge Base tools, and traces explicit relationships. The system prompt explicitly forbids turning correlation into confirmed causation.

Live application: https://incident-context.vercel.app/

No login is required.

The dataset represents a small production system:

web-app → checkout-api → payment-api → postgres

Try these questions:

What changed before INC-208?
Trace the services affected by INC-208.
Which runbook applies to INC-142?
Compare INC-142 and INC-208.

The comparison question is the strongest demonstration. It asks the agent to distinguish incidents that share related terminology but have different evidence paths.

Repository: https://github.com/anilloutombam/incident-context

The repository contains two standalone applications:

agent/   Next.js investigation interface and API
sanity/  Sanity Studio, schemas, and demo data
php
flowchart LR
    User[Incident question] --> UI[Next.js interface]
    UI -->|POST /api/chat| API[Next.js API]
    API --> Loop[AI SDK agent loop]
    Loop <-->|model requests| Gemini[Gemini 3.5 Flash-Lite]
    Loop <-->|MCP tool calls| MCP[Sanity Context MCP]

    Content[Sanity Content Lake<br/>Services · Deployments · Changes<br/>Runbooks · Incidents]
    Content -->|builds| KB[Sanity Knowledge Base]
    KB -->|serves cited entries| MCP

    Loop --> Report[Zod-validated report]
    Report -->|JSON response| UI
    UI --> Output[Evidence trail · Confirmed evidence<br/>Inferences · Next step · Sources]

The Gemini key and Sanity organization token remain on the server. The browser receives a validated report, not credentials.

The API validates the final response against a Zod schema instead of asking the model to return arbitrary Markdown:

{
  answer,
  evidenceTrail: [
    {
      from,
      relationship,
      to,
      source
    }
  ],
  confirmedEvidence,
  inferences,
  recommendedNextStep,
  sources: [
    {
      label,
      path
    }
  ]
}

This allows the interface to render relationships, facts, inferences, and sources as distinct product elements instead of parsing presentation from a generated string.

The deployed agent includes:

/api/health The project uses the published mcp-failure-lab package to test the dependency the agent cannot operate without.

The live compatibility scenario calls initial_context and verifies that the Sanity Context endpoint responds within ten seconds.

A separate local resilience suite covers bounded delays, hanging requests, malformed MCP responses, and connection loss.

pnpm test:mcp
pnpm test:mcp:live
pnpm test:mcp:faults

The difficult part was not connecting a model to an MCP endpoint. It was deciding what the model was allowed to claim.

I initially let the model connect any dots it found—deployments before incidents, shared keywords, related services. The output looked authoritative. But INC-142 and INC-208 both mentioned timeouts and both affected the payment path. One was a database connection exhaustion issue. One was a configuration change that had been too aggressive. The model could see both, but it could not reliably say which caused which.

I had to split facts from guesses in the output, enforce it in the system prompt, and then show that separation in the UI. Because incident responders need to know what you are certain about.

The useful answer is not a matching paragraph. It is a supported path, with source citations, and an explicit boundary between evidence and inference.

── more in #ai-agents 4 stories · sorted by recency
── more on @sanity 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/incident-context-tra…] indexed:0 read:5min 2026-09-20 ·