Cloud & InfraArticle A fifty-year-old planning technique cut pager volume 30% across 2,000 MongoDB shards.
[Emeka Okafor](https://sourcefeed.dev/u/emeka_okafor)
While half the industry is wiring LLM agents into incident response, [Stripe](https://stripe.com) quietly shipped something more interesting: a remediation system with no model in the loop at all. Its engineers modeled their global [MongoDB](https://www.mongodb.com) fleet — roughly 2,000 shards across more than 40 distinct layouts, underpinning $1.9 trillion in 2025 payment volume — as a state graph, and let Dijkstra's algorithm compute recovery plans. The results, [detailed on Stripe's engineering blog](https://stripe.dev/blog/how-stripe-uses-graph-search-and-state-machines-to-auto-remediate-a-global-database-fleet) and corroborated by InfoQ's coverage: pager volume down 30%, about 200 fewer pages a year, and 12 fewer cumulative days of shards sitting in degraded states.
That's a genuinely good outcome, but the numbers aren't the story. The story is the design philosophy, because it cuts against where most of the ops tooling market is heading — and it's the more defensible bet.
Runbooks are imperative. This is a planner. #
The dominant model for auto-remediation, from Rundeck scripts to PagerDuty automation to Meta's venerable FBAR, is imperative: detect a known failure signature, fire the matching pre-written procedure. It works until reality composes failures your runbook authors didn't anticipate. Stripe's baseline illustrates the ceiling — in one six-month window, 124 pages for misconfigured shards and another 32 for single-node-down incidents that came tangled with secondary issues, each blocking critical operations like index builds for about an hour while a human untangled it.
Stripe's system inverts the model. Instead of encoding procedures, it encodes three things: the current state of a shard (per-node voting rights, priorities, hidden status; shard-level oplog sizes and regional distribution), a goal state (the source-of-truth healthy configuration), and a set of atomic operations that form edges between states — add or remove a vote, hide a node, rebuild it, resize an oplog. Recovery becomes pathfinding: search the graph from the broken state to the healthy one, pruning any intermediate state that violates a safety invariant, like an even number of voting members or a configuration where a minority of nodes hold votes.
If that sounds familiar, it should. This is classical AI planning — the STRIPS lineage of goal states, operators, and state-space search that's older than most people running on-call rotations. It's also the same declarative instinct behind Kubernetes reconciliation, except Kubernetes controllers converge one resource at a time, while Stripe's planner reasons about the sequence — which matters enormously when the intermediate steps can break quorum on a database holding payment data.
The BFS-to-Dijkstra switch is the real lesson #
Stripe started with breadth-first search, which finds the shortest operation sequence to full health. BFS has a brutal failure mode, though: when full health is unreachable — say, hardware is dead and a replacement isn't provisioned yet — it returns nothing. The old system would simply give up and page a human, or worse, a cancelled workflow would strand the shard in an unpredictable intermediate state.
The fix was to make the search cost-aware. Edge weights combine a misconfiguration score (composable rules like VotingMisconfigRule
and AZMisconfigRule
that quantify how wrong a state is) with the operation's estimated duration — a config update costs 3 units, an oplog resize 19, a full node rebuild 101. Run Dijkstra over that graph and you get something BFS can't give you: when the perfect state is unreachable, the algorithm returns the path to the least broken reachable state. Partial remediation for free, from the same code path. And because cost encodes time, the planner naturally prefers three cheap config changes over one node rebuild that takes 33 times longer.
The other load-bearing decision is execution. Plans run on Temporal, so a node rebuild that takes hours survives deploys and crashes. But precomputing paths requires evaluating thousands of candidate states, which you can't do against real infrastructure — so Stripe put a CommonContext
interface between the remediation logic and the world. In simulation mode it evaluates thousands of state transitions per second in memory; in production mode the identical logic drives real Temporal activities. What you simulate is what you execute. Anyone who's watched a dry-run mode drift out of sync with the real code path knows exactly which failure class that kills.
Should you build this? #
Not reflexively. The pattern has real preconditions, and Stripe's post is honest about the modeling cost. It pays off when you have: a fleet large enough that the same failure classes recur weekly (their layout-count and pager numbers cleared that bar easily); operations that are genuinely atomic and machine-verifiable; and invariants you can state precisely enough for a search algorithm to prune on. If your incidents are mostly novel application-level failures, a planner over infrastructure states buys you little.
But if you operate any replicated stateful system — Postgres with Patroni, Kafka, Elasticsearch, Vitess — the shape transfers directly, and the ingredients are commodity: Temporal (or another durable execution engine) for orchestration, a few hundred lines of state modeling, a textbook algorithm. The hard work is deciding what "healthy" means precisely enough to write it down. That's also where the leverage compounds: Stripe found that once the operations and invariants existed, new capabilities fell out of the search. Onboarding a new shard layout required zero new remediation code — the planner just finds paths through the existing operation set — and the same machinery is being extended to layout migrations and batched maintenance windows, replacing bespoke workflows they'd otherwise hand-build.
That's my takeaway, and my editorial bet. LLM agents will keep winning at incident diagnosis, where the input is messy logs and tribal knowledge. But for execution against systems where a wrong step loses quorum, determinism wins: a Dijkstra plan is exhaustively simulable, provably invariant-preserving, and produces the same output every time. Stripe just demonstrated that at 2,000-shard scale, and the pattern was sitting in a fifty-year-old textbook the whole time. Before you give an agent the keys to your database fleet, it's worth asking whether what you actually need is a graph, a goal state, and a priority queue.
Sources & further reading #
Emeka Okafor· Security Editor Emeka has spent over a decade tracking threat actors, vulnerability disclosures, and the evolving landscape of application security, bringing a sharp continent-spanning perspective to his reporting. He's known for translating dense CVE advisories into clear, actionable context that developers and security teams alike actually read.
Discussion 0 #
No comments yet
Be the first to weigh in.