{"slug": "building-truesre-autonomous-multi-agent-incident-remediation-with-trueforge-and", "title": "Building TrueSRE: Autonomous Multi-Agent Incident Remediation with TrueForge, FastMCP, and Daytona", "summary": "An engineer built TrueSRE, an autonomous multi-agent platform for Site Reliability Engineering that diagnoses and remediates production outages in under 25 seconds. Built on the TrueForge Agent Harness, FastMCP, and Daytona sandboxing, it uses a six-agent swarm and cryptographic human-in-the-loop safety gates to ensure safe rollbacks and incident resolution.", "body_md": "💥 The 3:00 AM On-Call Nightmare\n\nIf you’ve ever carried an on-call pager for a production Kubernetes cluster, you know the dread of a 3:00 AM alert:\n\nYour phone buzzes aggressively. The payment gateway is crashing with CrashLoopBackOff. Users are encountering 500 internal errors, and the business is losing thousands of dollars per minute.\n\nYou sit down half-asleep and start the manual diagnostic grind:\n\nOpen Prometheus and craft PromQL queries to check if the service is down (up == 0).\n\nSift through thousands of Loki log lines searching for JVM or Go panic stack traces.\n\nRun kubectl describe pod to extract container termination exit codes (ExitCode: 137 OOMKilled).\n\nSwitch to GitHub to review recent commits and unified diffs to see who deployed what.\n\nFormulate a hypothesis, draft a rollback command, and pray it doesn't break production further.\n\nBy the time the issue is resolved, 45 minutes have passed.\n\nAccording to Gartner, enterprise IT downtime costs an average of $5,600 per minute ($300,000+ per hour).\n\nWhy Standard AI Chatbots Fail in Production\n\nGeneric LLM wrappers cannot solve this. When given access to a terminal, naive AI agents frequently hallucinate destructive commands (rm -rf /, dropping databases, or deleting the wrong namespaces) with zero security boundaries.\n\nWe asked ourselves: Can we build an autonomous, multi-agent SRE swarm that diagnoses and remediates real infrastructure in under 25 seconds—while guaranteeing 100% safety?\n\nThat is why we built TrueSRE.\n\n🌟 Introducing TrueSRE\n\nTrueSRE is an enterprise-grade autonomous Site Reliability Engineering multi-agent platform. Built on the TrueForge Agent Harness, official Model Context Protocol (FastMCP), Daytona Isolated Sandboxing, and Cryptographic HMAC-SHA256 Human-In-The-Loop (HITL) Safety Gates, TrueSRE takes outages from detection to verified resolution in under 25 seconds.\n\n```\n   🚨 3:00 AM Outage Detected (SEV-1 OOMKilled)\n                       │\n                       ▼\n```\n\n🧠 TrueForge Swarm Dispatched (PromQL, Loki, K8s, Git VCS via FastMCP)\n\n│\n\n▼\n\n🔬 Bayesian Competing Hypotheses (4 Ranked Possibilities — 92% Confidence)\n\n│\n\n▼\n\n📦 Daytona Isolated Sandbox Dry-Run (ExitCode: 0 Syntax Pre-Validation)\n\n│\n\n▼\n\n🔐 Cryptographic HMAC-SHA256 Human-In-The-Loop Approval Gate\n\n│\n\n▼\n\n⚡ FastMCP Mutating Rollout Undo + Closed-Loop Telemetry Verification\n\n│\n\n▼\n\n📝 Auto-Generated Blameless Post-Mortem & Vector RAG Incident Memory\n\n🏛️ System Architecture: The 4-Layer Foundation\n\nTrueSRE is engineered with strict separation of concerns across 4 distinct layers:\n\nPresentation Layer (Cyber-SRE Mission Control)\n\nA high-performance React 18 + Vite + Tailwind CSS dashboard that streams real-time agent thoughts, tool dispatches, live topology maps, and interactive Daytona terminals over WebSockets (ws://localhost:8000/ws/stream).\n\nTrueForge Agentic Runtime (FastAPI Backend)\n\nThe core intelligence engine. TrueForge coordinates turn-based agent execution, dynamic tool selection, and state transitions across a deterministic 15-node state machine.\n\nModel Context Protocol (FastMCP) & Daytona Sandbox Boundary\n\nDecouples AI reasoning from raw infrastructure execution. An official FastMCP JSON-RPC server exposes 18 enterprise tools, while ephemeral Daytona sandboxes test remediation runbooks before touching live workloads.\n\nInfrastructure & Telemetry Stack\n\nDirect integration with real Kubernetes clusters, Prometheus ([http://localhost:9090](http://localhost:9090)), Grafana Loki ([http://localhost:3100](http://localhost:3100)), Jaeger Tracing, and local Git VCS repositories.\n\n🤖 The 6-Agent Specialized Swarm\n\nRather than relying on a single monolithic prompt, TrueSRE delegates responsibilities across 6 specialized agent personas:\n\nAgent Persona Role in TrueForge Harness Key Responsibility\n\n🛡️ Incident Commander Triage & State Orchestration Classifies severity (SEV-1 to SEV-4), coordinates agent lifecycles, and broadcasts state.\n\n📈 Telemetry Investigator Metrics & Log Diagnostics Dispatches PromQL queries (up, rate, P95 latency) and queries Loki LogQL streams.\n\n🔍 RCA Reasoner 5-Whys Root Cause & Git Blame Analyzes unified Git diffs and evaluates 4 competing hypotheses with probabilistic confidence.\n\n🛠️ Remediation Planner Runbook Synthesis & Daytona Dry-run Formulates rollback runbooks and validates YAML schemas inside an isolated Daytona sandbox.\n\n🔒 Safety HITL Gatekeeper Blast Radius & Cryptographic Gate Calculates downstream blast radius and enforces single-use HMAC-SHA256 operator authorization.\n\n📝 Post-Mortem Scribe Blameless RCA & Incident Memory Auto-compiles markdown incident reports and stores vector embeddings in incident memory.\n\n🛡️ Enterprise Safety: Daytona Sandboxing & Cryptographic HITL\n\nSafety was our #1 architectural constraint. We implemented two bulletproof safety mechanisms:\n\nbash\n\nkubectl rollout undo deployment/payment-service --dry-run=client -o yaml\n\nIf an agent or prompt injection attempts a destructive command like rm -rf /, TrueSRE's Security Policy Engine intercepts it instantly:\n\ntext\n\nSECURITY_VIOLATION: Execution blocked by TrueSRE Daytona Policy.\n\nReason: Command matches catastrophic destruction pattern: rm -rf\n\nRisk Level: FORBIDDEN | Exit Code: 126\n\nThe FastMCP server validates this cryptographic token at the tool boundary. Unsigned requests are rejected with MUTATION_BLOCKED, and the token is revoked immediately upon execution.\n\n🔄 End-to-End Walkthrough: Resolving a Real SEV-1 Outage in 20 Seconds\n\nLet’s trace a real out-of-memory crash loop on our payment-service:\n\nDetection: Prometheus fires an alert—payment-service pods are restarting with exit code 137 (OOMKilled).\n\nInvestigation: The Telemetry Investigator dispatches FastMCP tools to query Prometheus availability (up == 0) and extracts JVM heap overflow stack traces from Loki.\n\nRoot Cause Analysis: The RCA Reasoner inspects recent Git commits, finds commit #e89a10f in BufferPool.java, and discovers an unbounded memory cache regression.\n\nHypothesis Evaluation: The Bayesian reasoner scores 4 competing hypotheses, ranking the memory leak code regression at 92% confidence.\n\nDaytona Dry-Run: The Remediation Planner tests k8s_rollout_undo in Daytona and confirms Exit Code: 0.\n\nHuman Approval: The operator clicks [Approve & Execute Remediation] in the Mission Control modal.\n\nClosed-Loop Verification: FastMCP executes the rollout undo. TrueSRE re-queries Prometheus and mathematically proves:\n\nError Rate: $84.5% \\rightarrow 0.04%$ ($\\Delta -84.46%$)\n\nP95 Latency: $1,840\\text{ms} \\rightarrow 92\\text{ms}$ (95% drop)\n\nCluster Health: Restored to nominal (up == 1).\n\nBlameless Post-Mortem: The Post-Mortem Scribe synthesizes a complete Markdown post-mortem report and stores vector embeddings in our Incident Memory store.\n\nTotal MTTR: Under 20 seconds.\n\n💡 What We Learned & Key Takeaways\n\nDecoupling Reasoning from Execution with MCP: The Model Context Protocol (MCP) proved to be an absolute game-changer. By treating infrastructure tools as standardized JSON-RPC endpoints, we could swap diagnostic tools without touching agent logic.\n\nTrueForge Agent Harness Reliability: TrueForge’s turn-based orchestration allowed us to manage complex multi-agent handoffs between Telemetry, RCA, and Remediation planners seamlessly.\n\nSafety Cannot Be an Afterthought: Autonomous agents must be bounded by sandboxed dry-runs and cryptographic authorization tokens. TrueSRE proves that autonomy and enterprise safety can coexist.\n\n🔗 Try TrueSRE Yourself!\n\nGitHub Repository: [https://github.com/VishalRRajput/TrueSRE](https://github.com/VishalRRajput/TrueSRE)\n\nTech Stack: Python 3.11, FastAPI, TrueForge, FastMCP, Daytona, React 18, Vite, Kubernetes, Prometheus, Loki.\n\nBuilt with ❤️ by Vishal Rajput for the TrueForge Agent Harness Hackathon.", "url": "https://wpnews.pro/news/building-truesre-autonomous-multi-agent-incident-remediation-with-trueforge-and", "canonical_source": "https://dev.to/vishal_rajput_0deae83dd7a/building-truesre-autonomous-multi-agent-incident-remediation-with-trueforge-fastmcp-and-daytona-1aai", "published_at": "2026-08-29 21:54:16+00:00", "updated_at": "2026-08-29 22:18:51.981844+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "mlops", "ai-safety"], "entities": ["TrueSRE", "TrueForge", "FastMCP", "Daytona", "Prometheus", "Grafana Loki", "Kubernetes", "React"], "alternates": {"html": "https://wpnews.pro/news/building-truesre-autonomous-multi-agent-incident-remediation-with-trueforge-and", "markdown": "https://wpnews.pro/news/building-truesre-autonomous-multi-agent-incident-remediation-with-trueforge-and.md", "text": "https://wpnews.pro/news/building-truesre-autonomous-multi-agent-incident-remediation-with-trueforge-and.txt", "jsonld": "https://wpnews.pro/news/building-truesre-autonomous-multi-agent-incident-remediation-with-trueforge-and.jsonld"}}