{"slug": "designing-idempotent-payment-microservices-guardrailed-langgraph-dispute-agents", "title": "Designing Idempotent Payment Microservices & Guardrailed LangGraph Dispute Agents", "summary": "A developer built SentinelPay, an open-source payment microservice stack that pairs a double-entry ledger with a LangGraph-based dispute agent, keeping the LLM strictly advisory while deterministic code makes refund decisions. The system enforces idempotency keys at both the API and database layers, uses ordered row-level locking to avoid deadlocks and double-spends, and fixes a Redis sliding-window rate-limiter bug where same-millisecond requests overwrote each other, letting excess traffic through. In live testing, a $25 dispute was escalated to ops rather than auto-credited because the LLM's 0.45 risk score exceeded the 0.35 threshold.", "body_md": "*How I built **SentinelPay**—a fault-tolerant double-entry ledger and stateful AI dispute resolver where the LLM scores risk, and plain code decides who gets their money back.*\n\nIn banking software, two failure modes destroy trust faster than anything else: **corrupting balances under concurrent retries**, and **letting a probabilistic AI model make deterministic financial decisions**.\n\nMost AI fintech demos wire an LLM straight to a refund endpoint and call it a day. I built **[SentinelPay](https://github.com/raghuvendra34/sentinelpay)** to do the opposite—enforcing strict distributed-systems invariants at the ledger layer while keeping generative AI strictly inside an auditable, deterministic guardrail boundary.\n\nEvery architectural decision in SentinelPay flows from three invariants:\n\n`Idempotency-Key` enforcement.`tx_dlq`).` JSONB`.\nSentinelPay splits responsibilities across four containerized services (plus PostgreSQL 16 and Redis 7):\n\n`:8001`):`:8002`):` DEBIT` + `CREDIT`) with row-level locks.` DISPUTE_OPENED` events from `tx_events` via a Redis consumer group, runs a 3-node workflow, and persists resolutions.`:3000`):\nWhen two concurrent transfers hit the same pair of accounts in opposite directions (Alice → Ben and Ben → Alice), naive row locking causes deadlocks. And when five simultaneous $400 withdrawals hit a $1,000 balance, naive read-then-write logic allows a double-spend.\n\nSentinelPay prevents both inside a single SQL transaction:\n\n`BIGINT` cents with a schema-level `CHECK (balance_cents >= 0)` constraint—never floating-point numbers.\n\n```\nSELECT account_id, balance_cents FROM accounts\nWHERE account_id = ANY($1::uuid[]) ORDER BY account_id FOR UPDATE;\n```\n\nNetwork timeouts mean clients *will* retry `POST /api/v1/transfers`. Every transfer requires an `Idempotency-Key` header enforced at two layers:\n\n`200 OK` (instead of `201 Created`).` UNIQUE` constraint on `transactions.idempotency_key` as the final database backstop.\nTo stop burst traffic from hammering the ledger, each account has a Redis sorted set (`ZSET`) implementing a 60-second sliding-window log (max 10 requests/minute, returning `429 Too Many Requests` with `Retry-After: 60`).\n\n**The bug:** In version 1, I used the millisecond timestamp as both the sorted-set *score* and *member*. Under concurrent bursts, two requests arriving within the same millisecond overwrote each other's member in the set—making `ZCARD` undercount and silently letting excess traffic slip through.\n\n**The fix:** Appending a short random nonce to each member makes same-millisecond requests distinct while keeping the timestamp as the score:\n\n``` python\nimport uuid\npipe.zremrangebyscore(key, 0, now_ms - 60_000).zadd(key, {f\"{now_ms}:{uuid.uuid4().hex[:8]}\": now_ms}).zcard(key)\n```\n\nA regression unit test using an in-memory fake Redis now proves that **12 requests fired in the exact same millisecond** yield **10 allowed and 2 blocked**, whereas the original implementation allowed all 12.\n\nLLM inference takes seconds; payment APIs must return in milliseconds. Whenever a user raises a dispute (or sends a transfer of $500+, which is auto-flagged), the gateway pushes an event to the Redis Stream `tx_events` and returns immediately.\n\nThe **Agent Worker** reads via a consumer group (`XREADGROUP`):\n\nThe dispute worker runs a 3-node **LangGraph** state machine:\n\n`retrieve_context`:` evaluate_policy`:`{risk_score, category, explanation}`.` guardrails` (pure deterministic code):\n\n``` php\ndef decide_dispute(amount_cents: int, risk_score: float, prior_disputes: int) -> str:\n    return \"AUTO_CREDIT_APPROVED\" if amount_cents <= 10_000 and risk_score <= 0.35 and prior_disputes == 0 else \"ESCALATED_TO_OPS\"\n```\n\nDuring live testing, I sent a $25 transfer seconds after two others—one of which was **$499**—and disputed it with *\"I did not authorise this payment.\"*\n\n`$25.00`, `0` prior disputes, 30-day mean `$262.00`.` 0.45`, category `<= $100`), prior disputes OK (` 0.45 > 0.35` (FAIL)`ESCALATED_TO_OPS` (`DISPUTE_ESCALATED`).\nWhen I inspected the stored JSONB trace, the model's prose explanation falsely described the burst as *\"three transfers of $25\"*—completely hallucinating the $499 transfer. **Yet the final decision was 100% correct, because the rule engine reads structured numbers, never prose.** That single run validated the entire architecture.\n\nAll core guarantees are backed by an **11-test `pytest` suite** (unit + live Docker integration):\n\n| Invariant Tested | Result | \n|---|---|\n| **Concurrency double-spend:** 5 simultaneous $400 transfers on a $1,000 balance | 2 succeeded, 3 `INSUFFICIENT_FUNDS` , final balance`$200.00` | \n| **Idempotent retry:** Same`Idempotency-Key` sent twice | `201` then`200` replayed; 1 transaction, 1`DEBIT` + 1`CREDIT` | \n| **Burst rate limit:** 12 requests from one account within 60s | Requests 1–10 → `201` ; 11–12 →`429` | \n| **Same-millisecond regression:** 12 simultaneous timestamps | 10 allowed, 2 blocked | \n| **Guardrail boundary matrix:** Inclusive limits & confidence bounds | 7/7 parameterized cases passed | \n\nSentinelPay is a portfolio-scale architecture with deliberate simplifications I document upfront:\n\nThe complete Docker Compose setup, SQL schema, LangGraph worker, and `pytest` verification logs are open source on GitHub: [github.com/raghuvendra34/sentinelpay](https://github.com/raghuvendra34/sentinelpay)", "url": "https://wpnews.pro/news/designing-idempotent-payment-microservices-guardrailed-langgraph-dispute-agents", "canonical_source": "https://dev.to/raghuvendra34/designing-idempotent-payment-microservices-guardrailed-langgraph-dispute-agents-48j5", "published_at": "2026-11-20 21:40:22+00:00", "updated_at": "2026-11-20 21:45:00.517233+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "developer-tools", "ai-safety"], "entities": ["SentinelPay", "LangGraph", "PostgreSQL", "Redis", "GitHub"], "also_reported_by": [], "takeaway": "Keep LLM output advisory and gate financial actions with deterministic code—SentinelPay auto-credits only when amount is under $100, risk score is at or below 0.35, and prior disputes are zero, escalating everything else.", "recommendation": {"advice": "Route LLM risk scores through a deterministic policy function with explicit thresholds, and append a random nonce to Redis rate-limiter members so same-millisecond requests aren't overwritten.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/designing-idempotent-payment-microservices-guardrailed-langgraph-dispute-agents", "markdown": "https://wpnews.pro/news/designing-idempotent-payment-microservices-guardrailed-langgraph-dispute-agents.md", "text": "https://wpnews.pro/news/designing-idempotent-payment-microservices-guardrailed-langgraph-dispute-agents.txt", "jsonld": "https://wpnews.pro/news/designing-idempotent-payment-microservices-guardrailed-langgraph-dispute-agents.jsonld"}}