{"slug": "how-to-investigate-a-production-incident-using-an-ai-agent-appsignal-mcp", "title": "How to Investigate a Production Incident Using an AI Agent (AppSignal MCP)", "summary": "AppSignal's MCP server gives AI agents direct access to production monitoring data, enabling incident investigations in plain English. In a real scenario, an agent triaged a payments service alert, identified a Stripe::CardError spike with a 5x error rate increase at 2:42 a.m., and surfaced a backtrace pointing to PaymentProcessor#charge, cutting investigation time from 10-15 minutes to seconds.", "body_md": "An incident has hit your product. I've been there: you're context-switching between hosting, CI/CD, codebase, AppSignal for monitoring, and whatever else your product depends on to minimize downtime and potential losses. You're trying to piece everything together, but it takes a lot of time, and that's something you don't have.\n\nAI agents connected to your tooling and your monitoring data via MCP free up that time for you. Instead of juggling dashboards, you can just ask an agent something like, \"Mate, what's happening?\" The agent will then perform queries to AppSignal, correlate the data, and reason about a potential cause.\n\n**TL;DR** [AppSignal's MCP server](https://docs.appsignal.com/mcp-server) gives your AI agents direct access to production monitoring data: errors, performance metrics, anomaly alerts, deploy markers, and much more. This article will guide you through a real incident investigation, from the first alert to a confirmed root cause, using plain English and your favorite AI agent.\n\nMCP and Its Role in Incident Response\n\nI've found MCP impossible to avoid for the past couple of months. You've probably heard of it even if you haven't touched it yet.\n\nMCP is an open standard that makes it possible for AI agents to interact with external tools and data sources, like your monitoring tool.\n\nAppSignal provides an [MCP server](https://docs.appsignal.com/mcp.html) that exposes a bunch of tools across different areas:\n\nYour agent can pull all of that data and combine it with other context, thus creating a more complete image of the situation.\n\nAnd that's where the real difference lies: instead of checking a dashboard and trying to figure everything out on your own, the agent just tells you what's happening.\n\nSetup in Two Minutes\n\nThe complete two-minute instructions are available in the [AppSignal MCP docs](https://docs.appsignal.com/mcp.html). But here's the short version using the public endpoint:\n\nPrerequisites\n\n[AppSignal account](https://appsignal.com/users/sign_up)with at least one app firing data- MCP token from\n[Accounting Settings > MCP Tokens](https://appsignal.com/users/mcp_tokens) - Claude Code, Cursor, VS Code, Zed or Windsurf installed\n\nClaude Code\n\nCursor\n\nEdit `~/cursor/mcp.json`\n\n:\n\nOne-click install\n\nIf you'd rather skip hand-editing config files, use these deep links:\n\nRunning Windsurf, Zed, Gemini CLI, GitHub Copilot CLI, the Claude app, or OpenAI Codex instead? Full setup steps for each are on the [AppSignal MCP docs](https://docs.appsignal.com/mcp-server).\n\nVerify the connection by asking your agent, \"What apps do I have in AppSignal?\" If it returns your app list, you're good to go.\n\nReal Incident Investigation\n\nHere is how things work in practice.\n\n**The scenario**: It's 2 past midnight. An alert fires. The error rate in the payments service has spiked 5 times over the notifying limit. Normally, you would open AppSignal, check error grouping, cross-reference deploy history, pull host metrics, and search logs. About 10-15 mins later you might have a solid hypothesis.\n\nWith AppSignal MCP, the investigation can look like this:\n\n**Step 1: Triage**\n\nWhat's going on with the payments service in the last 30 minutes?\n\nThe agent calls your tools scoped to the last 30 minutes and returns:\n\n- Error rate increased 5x starting at 2:42 a.m.\n- Dominant error:\n`Stripe::CardError`\n\n- Possibly happening in a different time zone\n- No deployments in the last 6 hours\n- P99 latency increased from 180 ms to 2.1 s\n\nYour understanding of the situation has been upgraded from \"something is wrong\" to a structured incident summary in seconds, without you touching the dashboard.\n\n**Step 2: Dig in**\n\nShow me a sample backtrace for the Stripe::CardError.\n\nThe agent now queries the incident tool for the most relevant recent matches and surfaces:\n\n- Full stack trace pointing to\n`PaymentProcessor#charge`\n\n- Specific error messages\n- Amount of users affected over the last 30 mins\n\nNow, you're able to pinpoint the exact piece of code involved and the scale of impact.\n\n**Step 3: Check for correlations**\n\nDid anything change around 2:40 a.m.? Deploys, traffic spikes?\n\nYour agent pulls deploy markers via the `get_app_resources`\n\ntool and cross-references them with the incident timeline:\n\n- No deploys since 9 a.m.\n- Normal traffic volume, no spike\n- External API response times: Stripe latency increased from 50 ms to 1.8 s starting at 2:20 a.m.\n\nYou've got a clear timeline, thanks to the agent.\n\n**Step 4: Confirm the situation**\n\nShow me Stripe API response times over the last two hours.\n\nThe agent calls `get_metrics_timeseries`\n\nfor the external service metric and returns a chart of Stripe response times. The spike lines up exactly with the error increase.\n\n**Conclusion**: This is an external service degradation. Time to check Stripe's status page and notify the responsible people.\n\n**Total investigation time:** < 3 mins.\n\n**Step 5: Document and hand off**\n\nAdd a note to the Stripe::CardError incident summarizing what we've found. Mark it as acknowledged and assign it to me.\n\nThe agent runs `manage_incident_note`\n\nand `update_incidents`\n\n. The incident now has context attached, and whoever picks it up next doesn't have to reconstruct the investigation from scratch.\n\nPrompt Cheat Sheet\n\nIf you've been woken up at 2 a.m. and your brain is still booting up, feel free to take a peek at the prompt cheat sheet, no strings attached.\n\n| Phase | Prompts |\n|---|---|\nTriage | \"What errors have increased in the last hour?\" / \"Is anything broken in production right now?\" |\nScoping | \"Which endpoints are affected?\" / \"How many users are impacted?\" |\nTimeline | \"When did this start?\" / \"What changed around [time]?\" |\nCorrelation | \"Were there any deploys before the spike?\" / \"Is this correlated with CPU or memory?\" |\nDeep dive | \"Show me a sample backtrace for `Stripe::CardError` \" (or `MyApp.Payments.ChargeError` in Elixir, `stripe.error.CardError` in Python) / \"What's the slowest ActiveRecord, Ecto, or Django ORM query right now?\" |\nComparison | \"Compare error rates today vs. yesterday.\" / \"Is this higher than our baseline?\" |\nResolution | \"Has the error rate recovered?\" / \"Show me the incident status across all open issues\" |\nHandoff | \"Summarize what we've found and add it as a note to incident #[number]\" |\n\nWhat AppSignal MCP Can and Cannot Do\n\nMCP is not a one-click solution to all incidents. Therefore, it's worth looking at what it can do well and where its limits are.\n\n**MCP can:**\n\n- Query errors, stack traces, performance data, host metrics, and deploy markers\n- Correlate events with natural language filters\n- Summarize patterns and surface anomalies faster than manual dashboard navigation\n- Update incidents, assign handles, add notes, and manage anomaly triggers\n\n**MCP cannot:**\n\n- Access data AppSignal doesn't have; if you're not catching stuff with AppSignal, the agent simply can't find it\n- Trigger rollbacks, restart services, or take action outside AppSignal (unless you connect a different tool and chain them)\n- Replace human judgement\n- Guarantee perfect accuracy on ambiguous queries\n\nMCP allows your AI agent to be a very fast analyst by giving it near-instant, context-aware data. Still, decisions are made solely by you.\n\nTips for Getting the Most Out of MCP\n\nThe following tips will help you interact more effectively with the MCP agent and get faster, more accurate insights.\n\n**Be specific about time ranges:**\"Last 30 mins\" produces tighter and more useful results than \"recently\". The agent passes these as filters straight to[AppSignal's API](https://docs.appsignal.com/api.html).**Name services and endpoints directly:** The agent has no knowledge of your internal jargon, so giving it precise service and endpoint names is a good practice.**Iterate:** Start broadly with a question like, \"What's happening?\", then narrow it down with, \"Tell me more about the X error\". You should treat incident investigation with agents as a conversation.**Verify before escalation:** I always confirm in the AppSignal dashboard before posting in Slack that it's definitely an external service's (or anyone else's) fault. The agent may be fast, but it's not infallible.**Combine signals:** If you're sending logs to AppSignal, ask the agent to search them alongside error data. Cross-signal correlation is a huge time saver.\n\nNext Steps\n\nAI will not replace monitoring, and it shouldn't. However, with an MCP, you can chain tools together to get a much clearer picture of an ongoing incident.\n\nI think AppSignal's MCP server makes incident resolution faster, especially under pressure. For more on this distinction, see MCP Won't Replace Your Monitoring Tool.\n\nTo get started, you should check out [AppSignal MCP docs](https://docs.appsignal.com/mcp.html) for setup and [MCP Tool Reference](https://docs.appsignal.com/mcp/reference.html) for the full query surface.\n\nNo AppSignal account yet? Start a [free 30-day trial](https://appsignal.com/users/sign_up). No credit card required.\n\nFrequently Asked Questions (FAQ)\n\nDoes this work with any AI agent or only specific ones?\n\nClaude Code, the Claude app, Cursor, Windsurf, Zed, VS Code, GitHub Copilot CLI, Gemini CLI, and OpenAI Codex all have setup instructions. Any MCP-compatible client will work. The ones mentioned here are just the ones with configurations tested in the [docs](https://docs.appsignal.com/mcp.html#configuration).\n\nCan the agent only read data, or can it update incidents too?\n\nBoth. The agent can update incident state and severity, assign handlers, add investigation notes, manage anomaly triggers, and create dashboards. Permissions are configurable per toolset when you generate your MCP token.\n\nWhat if my app is not sending all signals to AppSignal yet?\n\nThe agent can only query the data that AppSignal already has. If you are not sending logs, log queries will return nothing. If you have no anomaly triggers configured, anomaly queries will return nothing, as well. The more signals you share with AppSignal, the more useful the investigation workflow becomes, which is a good reason to instrument fully before you need it.\n\nIs this safe to use during an active incident with real production data?\n\nAbsolutely. MCP token permissions are configurable per area (app, exceptions, performance, metrics, anomalies, dashboards, logging), each set to read, write, or disabled, so read-only tokens are available if you want to limit the agent to querying only. As for write operations, those stay inside AppSignal's own data: incidents, anomaly triggers, log actions, and dashboards. The agent cannot touch your infrastructure via this integration.\n\nHow does AppSignal's MCP server compare to Scout's or Honeybadger's?\n\nScout and Honeybadger both ship official MCP servers too, but neither goes as far as AppSignal's:\n\n**Scout**— read-only across the board. It can query metrics, traces, jobs, errors, and anomalies, but there's no way for an agent to update an incident, manage a trigger, or take any action at all.**Honeybadger**— scoped to error/fault data only. No APM, logs, metrics, or dashboards are exposed, and its write access is limited to project administration rather than managing faults or incidents themselves.**AppSignal**— the only one of the three that lets an agent close the loop: querying errors, performance, metrics, logs, and anomalies, then acting on what it finds by updating incidents, adding notes, managing triggers, and building dashboards, all from the same conversation.\n\nPublished\n\n## Wondering what you can do next?\n\n[Try out AppSignal with a 30-day free trial](https://appsignal.com/users/sign_up).[Reach out to our support team with any feedback or questions](mailto:support@appsignal.com).\n\n- Share this article on social media\n\n## Dejan Lukić\n\nOur guest author Dejan is an electronics and backend engineer, who is pursuing entrepreneurship with SaaS and service-based agencies and is passionate about content creation.\n\n[All articles by Dejan Lukić](/authors/dejan-lukić.html)\n\nBecome our next author!\n\n[Find out more](/write-for-us.html)\n\n## AppSignal monitors your apps\n\nAppSignal provides insights for Ruby, Rails, Elixir, Phoenix, Node.js, Express and many other frameworks and libraries. We are located in beautiful Amsterdam. We love [stroopwafels](https://www.appsignal.com/waffles). If you do too, [let us know](mailto:contact@appsignal.com). We might send you some!\n\n[Discover AppSignal](https://www.appsignal.com)", "url": "https://wpnews.pro/news/how-to-investigate-a-production-incident-using-an-ai-agent-appsignal-mcp", "canonical_source": "https://blog.appsignal.com/2026/08/13/how-to-investigate-a-production-incident-using-an-ai-agent-appsignal-mcp.html", "published_at": "2026-08-13 00:00:00+00:00", "updated_at": "2026-08-13 12:39:03.881669+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "ai-infrastructure"], "entities": ["AppSignal", "MCP", "Claude Code", "Cursor", "VS Code", "Zed", "Windsurf", "Stripe"], "alternates": {"html": "https://wpnews.pro/news/how-to-investigate-a-production-incident-using-an-ai-agent-appsignal-mcp", "markdown": "https://wpnews.pro/news/how-to-investigate-a-production-incident-using-an-ai-agent-appsignal-mcp.md", "text": "https://wpnews.pro/news/how-to-investigate-a-production-incident-using-an-ai-agent-appsignal-mcp.txt", "jsonld": "https://wpnews.pro/news/how-to-investigate-a-production-incident-using-an-ai-agent-appsignal-mcp.jsonld"}}