How I built SentinelPay—a fault-tolerant double-entry ledger and stateful AI dispute resolver where the LLM scores risk, and plain code decides who gets their money back.
In banking software, two failure modes destroy trust faster than anything else: corrupting balances under concurrent retries, and letting a probabilistic AI model make deterministic financial decisions.
Most AI fintech demos wire an LLM straight to a refund endpoint and call it a day. I built SentinelPay to do the opposite—enforcing strict distributed-systems invariants at the ledger layer while keeping generative AI strictly inside an auditable, deterministic guardrail boundary.
Every architectural decision in SentinelPay flows from three invariants:
Idempotency-Key enforcement.tx_dlq). JSONB.
SentinelPay splits responsibilities across four containerized services (plus PostgreSQL 16 and Redis 7):
:8001)::8002): DEBIT + CREDIT) with row-level locks. DISPUTE_OPENED events from tx_events via a Redis consumer group, runs a 3-node workflow, and persists resolutions.:3000):
When two concurrent transfers hit the same pair of accounts in opposite directions (Alice → Ben and Ben → Alice), naive row locking causes deadlocks. And when five simultaneous $400 withdrawals hit a $1,000 balance, naive read-then-write logic allows a double-spend.
SentinelPay prevents both inside a single SQL transaction:
BIGINT cents with a schema-level CHECK (balance_cents >= 0) constraint—never floating-point numbers.
SELECT account_id, balance_cents FROM accounts
WHERE account_id = ANY($1::uuid[]) ORDER BY account_id FOR UPDATE;
Network timeouts mean clients will retry POST /api/v1/transfers. Every transfer requires an Idempotency-Key header enforced at two layers:
200 OK (instead of 201 Created). UNIQUE constraint on transactions.idempotency_key as the final database backstop.
To stop burst traffic from hammering the ledger, each account has a Redis sorted set (ZSET) implementing a 60-second sliding-window log (max 10 requests/minute, returning 429 Too Many Requests with Retry-After: 60).
The bug: In version 1, I used the millisecond timestamp as both the sorted-set score and member. Under concurrent bursts, two requests arriving within the same millisecond overwrote each other's member in the set—making ZCARD undercount and silently letting excess traffic slip through.
The fix: Appending a short random nonce to each member makes same-millisecond requests distinct while keeping the timestamp as the score:
import uuid
pipe.zremrangebyscore(key, 0, now_ms - 60_000).zadd(key, {f"{now_ms}:{uuid.uuid4().hex[:8]}": now_ms}).zcard(key)
A regression unit test using an in-memory fake Redis now proves that 12 requests fired in the exact same millisecond yield 10 allowed and 2 blocked, whereas the original implementation allowed all 12.
LLM inference takes seconds; payment APIs must return in milliseconds. Whenever a user raises a dispute (or sends a transfer of $500+, which is auto-flagged), the gateway pushes an event to the Redis Stream tx_events and returns immediately.
The Agent Worker reads via a consumer group (XREADGROUP):
The dispute worker runs a 3-node LangGraph state machine:
retrieve_context: evaluate_policy:{risk_score, category, explanation}. guardrails (pure deterministic code):
def decide_dispute(amount_cents: int, risk_score: float, prior_disputes: int) -> str:
return "AUTO_CREDIT_APPROVED" if amount_cents <= 10_000 and risk_score <= 0.35 and prior_disputes == 0 else "ESCALATED_TO_OPS"
During live testing, I sent a $25 transfer seconds after two others—one of which was $499—and disputed it with "I did not authorise this payment."
$25.00, 0 prior disputes, 30-day mean $262.00. 0.45, category <= $100), prior disputes OK ( 0.45 > 0.35 (FAIL)ESCALATED_TO_OPS (DISPUTE_ESCALATED).
When I inspected the stored JSONB trace, the model's prose explanation falsely described the burst as "three transfers of $25"—completely hallucinating the $499 transfer. Yet the final decision was 100% correct, because the rule engine reads structured numbers, never prose. That single run validated the entire architecture.
All core guarantees are backed by an 11-test pytest suite (unit + live Docker integration):
| Invariant Tested | Result |
|---|---|
| Concurrency double-spend: 5 simultaneous $400 transfers on a $1,000 balance | 2 succeeded, 3 INSUFFICIENT_FUNDS , final balance$200.00 |
Idempotent retry: SameIdempotency-Key sent twice |
201 then200 replayed; 1 transaction, 1DEBIT + 1CREDIT |
| Burst rate limit: 12 requests from one account within 60s | Requests 1–10 → 201 ; 11–12 →429 |
| Same-millisecond regression: 12 simultaneous timestamps | 10 allowed, 2 blocked |
| Guardrail boundary matrix: Inclusive limits & confidence bounds | 7/7 parameterized cases passed |
SentinelPay is a portfolio-scale architecture with deliberate simplifications I document upfront:
The complete Docker Compose setup, SQL schema, LangGraph worker, and pytest verification logs are open source on GitHub: github.com/raghuvendra34/sentinelpay