{"slug": "stripe-auto-heals-its-database-fleet-with-dijkstra-not-llms", "title": "Stripe Auto-Heals Its Database Fleet with Dijkstra, Not LLMs", "summary": "Stripe's engineering team built an auto-remediation system for its global MongoDB fleet that uses Dijkstra's algorithm and state-graph search instead of LLMs, cutting pager volume by 30% (about 200 fewer pages per year) and reducing degraded shard time by 12 cumulative days across roughly 2,000 shards. The system, detailed on Stripe's engineering blog, models shard states and atomic operations as a graph, computing recovery paths that respect safety invariants, and switches from breadth-first search to cost-aware Dijkstra to handle unreachable healthy states by finding the least-broken configuration.", "body_md": "[Cloud & Infra](https://sourcefeed.dev/c/cloud)Article\n\n# Stripe Auto-Heals Its Database Fleet with Dijkstra, Not LLMs\n\nA fifty-year-old planning technique cut pager volume 30% across 2,000 MongoDB shards.\n\n[Emeka Okafor](https://sourcefeed.dev/u/emeka_okafor)\n\nWhile half the industry is wiring LLM agents into incident response, [Stripe](https://stripe.com) quietly shipped something more interesting: a remediation system with no model in the loop at all. Its engineers modeled their global [MongoDB](https://www.mongodb.com) fleet — roughly 2,000 shards across more than 40 distinct layouts, underpinning $1.9 trillion in 2025 payment volume — as a state graph, and let Dijkstra's algorithm compute recovery plans. The results, [detailed on Stripe's engineering blog](https://stripe.dev/blog/how-stripe-uses-graph-search-and-state-machines-to-auto-remediate-a-global-database-fleet) and corroborated by InfoQ's coverage: pager volume down 30%, about 200 fewer pages a year, and 12 fewer cumulative days of shards sitting in degraded states.\n\nThat's a genuinely good outcome, but the numbers aren't the story. The story is the design philosophy, because it cuts against where most of the ops tooling market is heading — and it's the more defensible bet.\n\n## Runbooks are imperative. This is a planner.\n\nThe dominant model for auto-remediation, from Rundeck scripts to PagerDuty automation to Meta's venerable FBAR, is imperative: detect a known failure signature, fire the matching pre-written procedure. It works until reality composes failures your runbook authors didn't anticipate. Stripe's baseline illustrates the ceiling — in one six-month window, 124 pages for misconfigured shards and another 32 for single-node-down incidents that came tangled with secondary issues, each blocking critical operations like index builds for about an hour while a human untangled it.\n\nStripe's system inverts the model. Instead of encoding *procedures*, it encodes three things: the current state of a shard (per-node voting rights, priorities, hidden status; shard-level oplog sizes and regional distribution), a goal state (the source-of-truth healthy configuration), and a set of atomic operations that form edges between states — add or remove a vote, hide a node, rebuild it, resize an oplog. Recovery becomes pathfinding: search the graph from the broken state to the healthy one, pruning any intermediate state that violates a safety invariant, like an even number of voting members or a configuration where a minority of nodes hold votes.\n\nIf that sounds familiar, it should. This is classical AI planning — the STRIPS lineage of goal states, operators, and state-space search that's older than most people running on-call rotations. It's also the same declarative instinct behind Kubernetes reconciliation, except [Kubernetes](https://kubernetes.io) controllers converge one resource at a time, while Stripe's planner reasons about the *sequence* — which matters enormously when the intermediate steps can break quorum on a database holding payment data.\n\n## The BFS-to-Dijkstra switch is the real lesson\n\nStripe started with breadth-first search, which finds the shortest operation sequence to full health. BFS has a brutal failure mode, though: when full health is unreachable — say, hardware is dead and a replacement isn't provisioned yet — it returns nothing. The old system would simply give up and page a human, or worse, a cancelled workflow would strand the shard in an unpredictable intermediate state.\n\nThe fix was to make the search cost-aware. Edge weights combine a misconfiguration score (composable rules like `VotingMisconfigRule`\n\nand `AZMisconfigRule`\n\nthat quantify how wrong a state is) with the operation's estimated duration — a config update costs 3 units, an oplog resize 19, a full node rebuild 101. Run Dijkstra over that graph and you get something BFS can't give you: when the perfect state is unreachable, the algorithm returns the path to the *least broken reachable state*. Partial remediation for free, from the same code path. And because cost encodes time, the planner naturally prefers three cheap config changes over one node rebuild that takes 33 times longer.\n\nThe other load-bearing decision is execution. Plans run on [Temporal](https://temporal.io), so a node rebuild that takes hours survives deploys and crashes. But precomputing paths requires evaluating thousands of candidate states, which you can't do against real infrastructure — so Stripe put a `CommonContext`\n\ninterface between the remediation logic and the world. In simulation mode it evaluates thousands of state transitions per second in memory; in production mode the *identical logic* drives real Temporal activities. What you simulate is what you execute. Anyone who's watched a dry-run mode drift out of sync with the real code path knows exactly which failure class that kills.\n\n## Should you build this?\n\nNot reflexively. The pattern has real preconditions, and Stripe's post is honest about the modeling cost. It pays off when you have: a fleet large enough that the same failure classes recur weekly (their layout-count and pager numbers cleared that bar easily); operations that are genuinely atomic and machine-verifiable; and invariants you can state precisely enough for a search algorithm to prune on. If your incidents are mostly novel application-level failures, a planner over infrastructure states buys you little.\n\nBut if you operate any replicated stateful system — Postgres with Patroni, Kafka, Elasticsearch, Vitess — the shape transfers directly, and the ingredients are commodity: Temporal (or another durable execution engine) for orchestration, a few hundred lines of state modeling, a textbook algorithm. The hard work is deciding what \"healthy\" means precisely enough to write it down. That's also where the leverage compounds: Stripe found that once the operations and invariants existed, *new* capabilities fell out of the search. Onboarding a new shard layout required zero new remediation code — the planner just finds paths through the existing operation set — and the same machinery is being extended to layout migrations and batched maintenance windows, replacing bespoke workflows they'd otherwise hand-build.\n\nThat's my takeaway, and my editorial bet. LLM agents will keep winning at incident *diagnosis*, where the input is messy logs and tribal knowledge. But for *execution* against systems where a wrong step loses quorum, determinism wins: a Dijkstra plan is exhaustively simulable, provably invariant-preserving, and produces the same output every time. Stripe just demonstrated that at 2,000-shard scale, and the pattern was sitting in a fifty-year-old textbook the whole time. Before you give an agent the keys to your database fleet, it's worth asking whether what you actually need is a graph, a goal state, and a priority queue.\n\n## Sources & further reading\n\n[Emeka Okafor](https://sourcefeed.dev/u/emeka_okafor)· Security Editor\n\nEmeka has spent over a decade tracking threat actors, vulnerability disclosures, and the evolving landscape of application security, bringing a sharp continent-spanning perspective to his reporting. He's known for translating dense CVE advisories into clear, actionable context that developers and security teams alike actually read.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/stripe-auto-heals-its-database-fleet-with-dijkstra-not-llms", "canonical_source": "https://sourcefeed.dev/a/stripe-auto-heals-its-database-fleet-with-dijkstra-not-llms", "published_at": "2026-08-09 09:08:06+00:00", "updated_at": "2026-08-09 09:56:36.174924+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "developer-tools"], "entities": ["Stripe", "MongoDB", "Dijkstra", "InfoQ", "Kubernetes", "PagerDuty", "Meta"], "alternates": {"html": "https://wpnews.pro/news/stripe-auto-heals-its-database-fleet-with-dijkstra-not-llms", "markdown": "https://wpnews.pro/news/stripe-auto-heals-its-database-fleet-with-dijkstra-not-llms.md", "text": "https://wpnews.pro/news/stripe-auto-heals-its-database-fleet-with-dijkstra-not-llms.txt", "jsonld": "https://wpnews.pro/news/stripe-auto-heals-its-database-fleet-with-dijkstra-not-llms.jsonld"}}