# Designing Idempotent Payment Microservices & Guardrailed LangGraph Dispute Agents

> Source: <https://dev.to/raghuvendra34/designing-idempotent-payment-microservices-guardrailed-langgraph-dispute-agents-48j5>
> Published: 2026-11-20 21:40:22+00:00

**Recommendation:** Route LLM risk scores through a deterministic policy function with explicit thresholds, and append a random nonce to Redis rate-limiter members so same-millisecond requests aren't overwritten.
More in the playbook "Designing agent systems": <https://wpnews.pro/playbooks/agent-architecture>

*How I built **SentinelPay**—a fault-tolerant double-entry ledger and stateful AI dispute resolver where the LLM scores risk, and plain code decides who gets their money back.*

In banking software, two failure modes destroy trust faster than anything else: **corrupting balances under concurrent retries**, and **letting a probabilistic AI model make deterministic financial decisions**.

Most AI fintech demos wire an LLM straight to a refund endpoint and call it a day. I built **[SentinelPay](https://github.com/raghuvendra34/sentinelpay)** to do the opposite—enforcing strict distributed-systems invariants at the ledger layer while keeping generative AI strictly inside an auditable, deterministic guardrail boundary.

Every architectural decision in SentinelPay flows from three invariants:

`Idempotency-Key` enforcement.`tx_dlq`).` JSONB`.
SentinelPay splits responsibilities across four containerized services (plus PostgreSQL 16 and Redis 7):

`:8001`):`:8002`):` DEBIT` + `CREDIT`) with row-level locks.` DISPUTE_OPENED` events from `tx_events` via a Redis consumer group, runs a 3-node workflow, and persists resolutions.`:3000`):
When two concurrent transfers hit the same pair of accounts in opposite directions (Alice → Ben and Ben → Alice), naive row locking causes deadlocks. And when five simultaneous $400 withdrawals hit a $1,000 balance, naive read-then-write logic allows a double-spend.

SentinelPay prevents both inside a single SQL transaction:

`BIGINT` cents with a schema-level `CHECK (balance_cents >= 0)` constraint—never floating-point numbers.

```
SELECT account_id, balance_cents FROM accounts
WHERE account_id = ANY($1::uuid[]) ORDER BY account_id FOR UPDATE;
```

Network timeouts mean clients *will* retry `POST /api/v1/transfers`. Every transfer requires an `Idempotency-Key` header enforced at two layers:

`200 OK` (instead of `201 Created`).` UNIQUE` constraint on `transactions.idempotency_key` as the final database backstop.
To stop burst traffic from hammering the ledger, each account has a Redis sorted set (`ZSET`) implementing a 60-second sliding-window log (max 10 requests/minute, returning `429 Too Many Requests` with `Retry-After: 60`).

**The bug:** In version 1, I used the millisecond timestamp as both the sorted-set *score* and *member*. Under concurrent bursts, two requests arriving within the same millisecond overwrote each other's member in the set—making `ZCARD` undercount and silently letting excess traffic slip through.

**The fix:** Appending a short random nonce to each member makes same-millisecond requests distinct while keeping the timestamp as the score:

``` python
import uuid
pipe.zremrangebyscore(key, 0, now_ms - 60_000).zadd(key, {f"{now_ms}:{uuid.uuid4().hex[:8]}": now_ms}).zcard(key)
```

A regression unit test using an in-memory fake Redis now proves that **12 requests fired in the exact same millisecond** yield **10 allowed and 2 blocked**, whereas the original implementation allowed all 12.

LLM inference takes seconds; payment APIs must return in milliseconds. Whenever a user raises a dispute (or sends a transfer of $500+, which is auto-flagged), the gateway pushes an event to the Redis Stream `tx_events` and returns immediately.

The **Agent Worker** reads via a consumer group (`XREADGROUP`):

The dispute worker runs a 3-node **LangGraph** state machine:

`retrieve_context`:` evaluate_policy`:`{risk_score, category, explanation}`.` guardrails` (pure deterministic code):

``` php
def decide_dispute(amount_cents: int, risk_score: float, prior_disputes: int) -> str:
    return "AUTO_CREDIT_APPROVED" if amount_cents <= 10_000 and risk_score <= 0.35 and prior_disputes == 0 else "ESCALATED_TO_OPS"
```

During live testing, I sent a $25 transfer seconds after two others—one of which was **$499**—and disputed it with *"I did not authorise this payment."*

`$25.00`, `0` prior disputes, 30-day mean `$262.00`.` 0.45`, category `<= $100`), prior disputes OK (` 0.45 > 0.35` (FAIL)`ESCALATED_TO_OPS` (`DISPUTE_ESCALATED`).
When I inspected the stored JSONB trace, the model's prose explanation falsely described the burst as *"three transfers of $25"*—completely hallucinating the $499 transfer. **Yet the final decision was 100% correct, because the rule engine reads structured numbers, never prose.** That single run validated the entire architecture.

All core guarantees are backed by an **11-test `pytest` suite** (unit + live Docker integration):

| Invariant Tested | Result | 
|---|---|
| **Concurrency double-spend:** 5 simultaneous $400 transfers on a $1,000 balance | 2 succeeded, 3 `INSUFFICIENT_FUNDS` , final balance`$200.00` | 
| **Idempotent retry:** Same`Idempotency-Key` sent twice | `201` then`200` replayed; 1 transaction, 1`DEBIT` + 1`CREDIT` | 
| **Burst rate limit:** 12 requests from one account within 60s | Requests 1–10 → `201` ; 11–12 →`429` | 
| **Same-millisecond regression:** 12 simultaneous timestamps | 10 allowed, 2 blocked | 
| **Guardrail boundary matrix:** Inclusive limits & confidence bounds | 7/7 parameterized cases passed | 

SentinelPay is a portfolio-scale architecture with deliberate simplifications I document upfront:

The complete Docker Compose setup, SQL schema, LangGraph worker, and `pytest` verification logs are open source on GitHub: [github.com/raghuvendra34/sentinelpay](https://github.com/raghuvendra34/sentinelpay)
