{"slug": "i-spent-years-balancing-ledgers-now-i-balance-redis-connections", "title": "I Spent Years Balancing Ledgers. Now I Balance Redis Connections.", "summary": "A developer who spent years in accounting and finance has applied ledger-based discipline to production incidents, building AlertEngine as an operational governance system rather than a monitoring tool. The system enforces a strict hierarchy where policy determines incident detection and recovery, AI only diagnoses and recommends, and humans must authorize every action—with all decisions recorded in an immutable, actor-attributed audit trail. The source-available project, built to address the challenge of managing infrastructure in Zimbabwe where engineers may not always have laptop access, allows recovery authorization via a single tap on WhatsApp without requiring SSH or runbooks.", "body_md": "I spent my career in accounting and finance before building infrastructure in Zimbabwe.\n\nIn accounting, every transaction has three properties:\n\nAuthorization — no entry without approval\n\nImmutability — once recorded, never altered\n\nReconciliation — every debit has a corresponding credit, provable by audit\n\nWhen I started building FastAPI AlertEngine, I applied the same discipline to production incidents. The result is not a monitoring tool. It's an operational governance system.\n\nMonitoring tools tell you what broke after it broke. Datadog, Grafana, Sentry — they produce beautiful post-mortems.\n\nGovernance tools enforce that nothing executes without authorization, and they prove it afterward.\n\nMost teams conflate the two. They buy monitoring, assume governance, and get surprised when auditors ask: \"Who approved that deploy?\"\n\nAlertEngine separates them explicitly:\n\nplain\n\nDetection → Policy (deterministic, no AI)\n\nDiagnosis → AI (explains, recommends, does not decide)\n\nAuthorization → Human (engineer taps approve)\n\nExecution → Webhook (your infrastructure, your control)\n\nAudit → Ledger (immutable, replayable, actor-attributed)\n\nThis is not a feature list. It's an architectural hierarchy enforced by code.\n\nEngineers in Zimbabwe aren't always at laptops when things break. WhatsApp is ubiquituous and can be the operational control plane.\n\nThat constraint produces something better than a dashboard: alerts that find you, with a single tap to authorise recovery. No SSH. No runbooks. No \"log into Grafana and interpret the graph.\"\n\nJust: \"Something broke. Here's why. Tap approve. Nothing runs without you.\"\n\n**The Ledger Philosophy**\n\nIn finance, a ledger has two sides: what happened, and who authorized it.\n\nAlertEngine's audit trail has the same structure:\n\nJSON\n\n{\n\n\"timestamp\": 1717344000,\n\n\"incident_id\": \"inc-abc123-1685000000\",\n\n\"stage\": \"AUTHORIZED\",\n\n\"actor\": \"engineer\",\n\n\"decision\": \"approve\",\n\n\"reason\": \"Database connection pool exhausted — restart recommended\",\n\n\"confidence\": 0.87,\n\n\"policy_version\": \"1.0.0\",\n\n\"tenant_id\": \"tenant-xyz789\"\n\n}\n\nEvery entry is append-only. Every entry has an actor. Every entry is replayable.\n\nThis is not logging. Logging tells you what the system did. A ledger tells you who authorized it and why.\n\nPolicy Is the Floor. AI Is the Ceiling.\n\nThe most important architectural decision in AlertEngine is this:\n\nClaude cannot trigger a state transition.\n\nPolicy decides whether an incident exists. Policy decides when a system has recovered. Claude diagnoses and explains — but the state machine doesn't listen to Claude. It listens to incident_policy.py.\n\nWhen health metrics recover, the pipeline doesn't ask Claude what to do. It calls should_recover(score, err) and if the threshold is met, it transitions to RECOVERED with actor=\"policy\". Claude's recommendation is irrelevant.\n\nA confident wrong AI diagnosis cannot cause an incident to escalate\n\nA policy recovery override is logged as actor: \"policy\" — auditors can see exactly when and why\n\nChanging thresholds is a one-line edit in one file, versioned, and logged in every subsequent audit entry\n\nThe audit trail never lies about who made the decision\n\n**Three forces are converging:**\n\nI'm also building a payment orchestration platform for the African \"hustler\" context. Getting infrastructure funding in Zimbabwe is genuinely hard.\n\nSo I packaged the operational governance layer as a standalone product. It solves a real problem — I needed it myself at 2am. It also funds the bigger build.\n\nThat felt worth being honest about.\n\n*The Code*\n\nThe orchestrator is source-available. Every claim in this post is verifiable:\n\norchestrator/pipeline.py — policy hierarchy, actor=\"policy\" on recovery override\n\norchestrator/incident_policy.py — single POLICY dict, versioned, env-configurable\n\norchestrator/audit.py — append-only Redis LIST, full actor attribution, replayable\n\nRead the code. Audit the architecture. Then decide if your infrastructure deserves the same discipline as your accounting.\n\nGitHub: github.com/Tandem-Media/fastapi-alertengine\n\nInstall:\n\nbash\n\npip install fastapi-alertengine\n\nManaged orchestrator: [anchorflowalertengine@outlook.com](mailto:anchorflowalertengine@outlook.com)\n\nBuilt in Harare, Zimbabwe. 🇿🇼", "url": "https://wpnews.pro/news/i-spent-years-balancing-ledgers-now-i-balance-redis-connections", "canonical_source": "https://dev.to/tandemmedia/i-spent-years-balancing-ledgers-now-i-balance-redis-connections-pb", "published_at": "2026-06-03 09:57:54+00:00", "updated_at": "2026-06-03 10:13:03.726598+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "ai-products", "ai-startups", "mlops"], "entities": ["FastAPI", "AlertEngine", "Datadog", "Grafana", "Sentry", "Zimbabwe", "WhatsApp"], "alternates": {"html": "https://wpnews.pro/news/i-spent-years-balancing-ledgers-now-i-balance-redis-connections", "markdown": "https://wpnews.pro/news/i-spent-years-balancing-ledgers-now-i-balance-redis-connections.md", "text": "https://wpnews.pro/news/i-spent-years-balancing-ledgers-now-i-balance-redis-connections.txt", "jsonld": "https://wpnews.pro/news/i-spent-years-balancing-ledgers-now-i-balance-redis-connections.jsonld"}}