cd /news/ai-agents/designing-idempotent-payment-microse… · home › topics › ai-agents › article
[ARTICLE · art-173496] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

Designing Idempotent Payment Microservices & Guardrailed LangGraph Dispute Agents

A developer built SentinelPay, an open-source payment microservice stack that pairs a double-entry ledger with a LangGraph-based dispute agent, keeping the LLM strictly advisory while deterministic code makes refund decisions. The system enforces idempotency keys at both the API and database layers, uses ordered row-level locking to avoid deadlocks and double-spends, and fixes a Redis sliding-window rate-limiter bug where same-millisecond requests overwrote each other, letting excess traffic through. In live testing, a $25 dispute was escalated to ops rather than auto-credited because the LLM's 0.45 risk score exceeded the 0.35 threshold.

recommendation

Route LLM risk scores through a deterministic policy function with explicit thresholds, and append a random nonce to Redis rate-limiter members so same-millisecond requests aren't overwritten.

more in playbook: Designing agent systems
by read4 min views1 publishedNov 20, 2026

How I built SentinelPay—a fault-tolerant double-entry ledger and stateful AI dispute resolver where the LLM scores risk, and plain code decides who gets their money back.

In banking software, two failure modes destroy trust faster than anything else: corrupting balances under concurrent retries, and letting a probabilistic AI model make deterministic financial decisions.

Most AI fintech demos wire an LLM straight to a refund endpoint and call it a day. I built SentinelPay to do the opposite—enforcing strict distributed-systems invariants at the ledger layer while keeping generative AI strictly inside an auditable, deterministic guardrail boundary.

Every architectural decision in SentinelPay flows from three invariants:

Idempotency-Key enforcement.tx_dlq). JSONB. SentinelPay splits responsibilities across four containerized services (plus PostgreSQL 16 and Redis 7):

:8001)::8002): DEBIT + CREDIT) with row-level locks. DISPUTE_OPENED events from tx_events via a Redis consumer group, runs a 3-node workflow, and persists resolutions.:3000): When two concurrent transfers hit the same pair of accounts in opposite directions (Alice → Ben and Ben → Alice), naive row locking causes deadlocks. And when five simultaneous $400 withdrawals hit a $1,000 balance, naive read-then-write logic allows a double-spend.

SentinelPay prevents both inside a single SQL transaction:

BIGINT cents with a schema-level CHECK (balance_cents >= 0) constraint—never floating-point numbers.

SELECT account_id, balance_cents FROM accounts
WHERE account_id = ANY($1::uuid[]) ORDER BY account_id FOR UPDATE;

Network timeouts mean clients will retry POST /api/v1/transfers. Every transfer requires an Idempotency-Key header enforced at two layers:

200 OK (instead of 201 Created). UNIQUE constraint on transactions.idempotency_key as the final database backstop. To stop burst traffic from hammering the ledger, each account has a Redis sorted set (ZSET) implementing a 60-second sliding-window log (max 10 requests/minute, returning 429 Too Many Requests with Retry-After: 60).

The bug: In version 1, I used the millisecond timestamp as both the sorted-set score and member. Under concurrent bursts, two requests arriving within the same millisecond overwrote each other's member in the set—making ZCARD undercount and silently letting excess traffic slip through.

The fix: Appending a short random nonce to each member makes same-millisecond requests distinct while keeping the timestamp as the score:

import uuid
pipe.zremrangebyscore(key, 0, now_ms - 60_000).zadd(key, {f"{now_ms}:{uuid.uuid4().hex[:8]}": now_ms}).zcard(key)

A regression unit test using an in-memory fake Redis now proves that 12 requests fired in the exact same millisecond yield 10 allowed and 2 blocked, whereas the original implementation allowed all 12.

LLM inference takes seconds; payment APIs must return in milliseconds. Whenever a user raises a dispute (or sends a transfer of $500+, which is auto-flagged), the gateway pushes an event to the Redis Stream tx_events and returns immediately.

The Agent Worker reads via a consumer group (XREADGROUP):

The dispute worker runs a 3-node LangGraph state machine:

retrieve_context: evaluate_policy:{risk_score, category, explanation}. guardrails (pure deterministic code):

def decide_dispute(amount_cents: int, risk_score: float, prior_disputes: int) -> str:
    return "AUTO_CREDIT_APPROVED" if amount_cents <= 10_000 and risk_score <= 0.35 and prior_disputes == 0 else "ESCALATED_TO_OPS"

During live testing, I sent a $25 transfer seconds after two others—one of which was $499—and disputed it with "I did not authorise this payment."

$25.00, 0 prior disputes, 30-day mean $262.00. 0.45, category <= $100), prior disputes OK ( 0.45 > 0.35 (FAIL)ESCALATED_TO_OPS (DISPUTE_ESCALATED). When I inspected the stored JSONB trace, the model's prose explanation falsely described the burst as "three transfers of $25"—completely hallucinating the $499 transfer. Yet the final decision was 100% correct, because the rule engine reads structured numbers, never prose. That single run validated the entire architecture.

All core guarantees are backed by an 11-test pytest suite (unit + live Docker integration):

Invariant Tested Result
Concurrency double-spend: 5 simultaneous $400 transfers on a $1,000 balance 2 succeeded, 3 INSUFFICIENT_FUNDS , final balance$200.00
Idempotent retry: SameIdempotency-Key sent twice 201 then200 replayed; 1 transaction, 1DEBIT + 1CREDIT
Burst rate limit: 12 requests from one account within 60s Requests 1–10 → 201 ; 11–12 →429
Same-millisecond regression: 12 simultaneous timestamps 10 allowed, 2 blocked
Guardrail boundary matrix: Inclusive limits & confidence bounds 7/7 parameterized cases passed

SentinelPay is a portfolio-scale architecture with deliberate simplifications I document upfront:

The complete Docker Compose setup, SQL schema, LangGraph worker, and pytest verification logs are open source on GitHub: github.com/raghuvendra34/sentinelpay

── more in #ai-agents 4 stories · sorted by recency
── more on @sentinelpay 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/designing-idempotent…] indexed:0 read:4min 2026-11-20 · —