Designing Idempotent Payment Microservices & Guardrailed LangGraph Dispute Agents A developer built SentinelPay, an open-source payment microservice stack that pairs a double-entry ledger with a LangGraph-based dispute agent, keeping the LLM strictly advisory while deterministic code makes refund decisions. The system enforces idempotency keys at both the API and database layers, uses ordered row-level locking to avoid deadlocks and double-spends, and fixes a Redis sliding-window rate-limiter bug where same-millisecond requests overwrote each other, letting excess traffic through. In live testing, a $25 dispute was escalated to ops rather than auto-credited because the LLM's 0.45 risk score exceeded the 0.35 threshold. How I built SentinelPay —a fault-tolerant double-entry ledger and stateful AI dispute resolver where the LLM scores risk, and plain code decides who gets their money back. In banking software, two failure modes destroy trust faster than anything else: corrupting balances under concurrent retries , and letting a probabilistic AI model make deterministic financial decisions . Most AI fintech demos wire an LLM straight to a refund endpoint and call it a day. I built SentinelPay https://github.com/raghuvendra34/sentinelpay to do the opposite—enforcing strict distributed-systems invariants at the ledger layer while keeping generative AI strictly inside an auditable, deterministic guardrail boundary. Every architectural decision in SentinelPay flows from three invariants: Idempotency-Key enforcement. tx dlq . JSONB . SentinelPay splits responsibilities across four containerized services plus PostgreSQL 16 and Redis 7 : :8001 : :8002 : DEBIT + CREDIT with row-level locks. DISPUTE OPENED events from tx events via a Redis consumer group, runs a 3-node workflow, and persists resolutions. :3000 : When two concurrent transfers hit the same pair of accounts in opposite directions Alice → Ben and Ben → Alice , naive row locking causes deadlocks. And when five simultaneous $400 withdrawals hit a $1,000 balance, naive read-then-write logic allows a double-spend. SentinelPay prevents both inside a single SQL transaction: BIGINT cents with a schema-level CHECK balance cents = 0 constraint—never floating-point numbers. SELECT account id, balance cents FROM accounts WHERE account id = ANY $1::uuid ORDER BY account id FOR UPDATE; Network timeouts mean clients will retry POST /api/v1/transfers . Every transfer requires an Idempotency-Key header enforced at two layers: 200 OK instead of 201 Created . UNIQUE constraint on transactions.idempotency key as the final database backstop. To stop burst traffic from hammering the ledger, each account has a Redis sorted set ZSET implementing a 60-second sliding-window log max 10 requests/minute, returning 429 Too Many Requests with Retry-After: 60 . The bug: In version 1, I used the millisecond timestamp as both the sorted-set score and member . Under concurrent bursts, two requests arriving within the same millisecond overwrote each other's member in the set—making ZCARD undercount and silently letting excess traffic slip through. The fix: Appending a short random nonce to each member makes same-millisecond requests distinct while keeping the timestamp as the score: python import uuid pipe.zremrangebyscore key, 0, now ms - 60 000 .zadd key, {f"{now ms}:{uuid.uuid4 .hex :8 }": now ms} .zcard key A regression unit test using an in-memory fake Redis now proves that 12 requests fired in the exact same millisecond yield 10 allowed and 2 blocked , whereas the original implementation allowed all 12. LLM inference takes seconds; payment APIs must return in milliseconds. Whenever a user raises a dispute or sends a transfer of $500+, which is auto-flagged , the gateway pushes an event to the Redis Stream tx events and returns immediately. The Agent Worker reads via a consumer group XREADGROUP : The dispute worker runs a 3-node LangGraph state machine: retrieve context : evaluate policy : {risk score, category, explanation} . guardrails pure deterministic code : php def decide dispute amount cents: int, risk score: float, prior disputes: int - str: return "AUTO CREDIT APPROVED" if amount cents <= 10 000 and risk score <= 0.35 and prior disputes == 0 else "ESCALATED TO OPS" During live testing, I sent a $25 transfer seconds after two others—one of which was $499 —and disputed it with "I did not authorise this payment." $25.00 , 0 prior disputes, 30-day mean $262.00 . 0.45 , category <= $100 , prior disputes OK 0.45 0.35 FAIL ESCALATED TO OPS DISPUTE ESCALATED . When I inspected the stored JSONB trace, the model's prose explanation falsely described the burst as "three transfers of $25" —completely hallucinating the $499 transfer. Yet the final decision was 100% correct, because the rule engine reads structured numbers, never prose. That single run validated the entire architecture. All core guarantees are backed by an 11-test pytest suite unit + live Docker integration : | Invariant Tested | Result | |---|---| | Concurrency double-spend: 5 simultaneous $400 transfers on a $1,000 balance | 2 succeeded, 3 INSUFFICIENT FUNDS , final balance $200.00 | | Idempotent retry: Same Idempotency-Key sent twice | 201 then 200 replayed; 1 transaction, 1 DEBIT + 1 CREDIT | | Burst rate limit: 12 requests from one account within 60s | Requests 1–10 → 201 ; 11–12 → 429 | | Same-millisecond regression: 12 simultaneous timestamps | 10 allowed, 2 blocked | | Guardrail boundary matrix: Inclusive limits & confidence bounds | 7/7 parameterized cases passed | SentinelPay is a portfolio-scale architecture with deliberate simplifications I document upfront: The complete Docker Compose setup, SQL schema, LangGraph worker, and pytest verification logs are open source on GitHub: github.com/raghuvendra34/sentinelpay https://github.com/raghuvendra34/sentinelpay