Here’s a moment every team shipping an LLM app eventually hits: a request times out, something retries, and now support is fielding a ticket that says the AI charged someone twice.
Nobody wrote a bug. The model didn’t hallucinate. You shipped a retry path with no idempotency behind it — and in an LLM app, that gap is far more dangerous than it is in a normal CRUD backend.
Backend engineers already know idempotency. GET and DELETE are naturally safe, POST isn’t, and an Idempotency-Key plus a lock handles it. That playbook isn’t enough here, for three reasons.
LLMs aren’t deterministic. The same prompt returns a different response every time once temperature goes above 0. You can’t hash the request body to tell a retry apart from a user sending two similar messages.
Agent tool calls are a stateful chain of side effects. Say an agent chains three tools: create_order(), then charge_card(), then send_confirmation_email() — and the network dies right after step two. The framework retries the whole chain. The card gets charged twice. Support calls it an AI bug. It’s a missing idempotency layer.
Streaming blurs what “success” means. A normal API call is either done or it isn’t. In a stream, you might have consumed 500 tokens when the connection drops. Success or failure? Does the retry get billed again?
All three compound under high concurrency, long tasks, and multi-agent setups. Here are the five traps that show up in production, and the fix for each.
Say your RAG app runs 1,000 queries a day at roughly $0.003 each (~1,000 input tokens + 300 output). A 5% timeout rate means 50 retries a day. With no idempotency cache, every one of those retries pays full price for a call you already made.
Run the math:
Five thousand dollars, on requests you already paid for once. And it produces no error log, no failed request, and no alert — the bill is the only place it ever shows up.
It gets worse when the client also retries. A frontend retry library fires again after the server already succeeded, and now you’ve paid for the LLM call twice, written the DB twice, and pushed the notification twice.
// idempotent-llm-call.tsimport { createClient } from "redis";import { createHash } from "crypto"; interface LLMRequest { model: string; messages: Array<{ role: string; content: string }>; temperature?: number;} const redis = createClient({ url: process.env.REDIS_URL }); async function idempotentLLMCall( idempotencyKey: string, request: LLMRequest, ttlSeconds = 3600): Promise<string> { const cacheKey = `llm:idem:${idempotencyKey}`; const lockKey = `llm:lock:${idempotencyKey}`; // 1. Check the cache first const cached = await redis.get(cacheKey); if (cached) { return JSON.parse(cached).response; } // 2. Acquire the lock with SET NX so concurrent retries don't all hit the LLM const lockAcquired = await redis.set(lockKey, "1", { NX: true, EX: 30, // 30s lock timeout — past this, assume the LLM call is stuck }); if (!lockAcquired) { // Someone else holds the lock — poll the cache until they finish return await pollForResult(cacheKey, 30000); } try { // 3. The actual LLM call const response = await callLLMAPI(request); // 4. Atomically cache the result (repeat requests within the TTL return this) await redis.setEx( cacheKey, ttlSeconds, JSON.stringify({ response, cachedAt: Date.now() }) ); return response; } finally { await redis.del(lockKey); }} async function pollForResult(cacheKey: string, timeoutMs: number): Promise<string> { const deadline = Date.now() + timeoutMs; while (Date.now() < deadline) { await new Promise((r) => setTimeout(r, 200)); const result = await redis.get(cacheKey); if (result) return JSON.parse(result).response; } throw new Error("Idempotency lock timeout: upstream may have failed");}
The key move is SET NX — set only if the key doesn’t exist yet. Among concurrent retries, exactly one actually calls the LLM. The rest wait for that result instead of firing their own request.
The right construction depends on where the call originates:
One caveat on that last one: it’s fuzzy by design, and similar prompts can carry different intent. In production, reserve exact idempotency for system-level calls and keep semantic caching as a separate, looser layer for free-form user input. Don’t merge the two.
This is the most dangerous idempotency failure in LLM apps. When an agent calls external tools — charging a card, sending an email, writing to a database — those side effects are usually irreversible.
A postmortem write-up from tianpan.co (April 2026) catalogued real incidents from this exact failure mode:
The root cause is identical every time. The framework retries at the “did I get a model response” layer, not the “did this side effect already happen” layer. LangChain and every major agent SDK have this gap, because their retry logic was built around model output, not tool execution state.
Three design decisions worth calling out:
SSE streaming is standard for LLM apps now. But it blurs exactly where success ends:
client receives chunk 1..200 → connection drops → client retries→ server re-calls the LLM → user sees a duplicated prefix plus the full response→ billed twice
It gets worse if you do side effects mid-stream — saving to the DB every 100 tokens, say. A retry re-runs those side effects too.
// streaming-session-manager.tsinterface StreamingSession { sessionId: string; status: "streaming" | "complete" | "failed"; chunks: string[]; totalTokens: number; completedAt?: number;} class StreamingSessionManager { constructor(private redis: RedisClient) {} async startSession(sessionId: string): Promise<void> { const session: StreamingSession = { sessionId, status: "streaming", chunks: [], totalTokens: 0, }; await this.redis.setEx( `stream:session:${sessionId}`, 3600, JSON.stringify(session) ); } async appendChunk(sessionId: string, chunk: string): Promise<void> { // RPUSH appends atomically without overwriting existing chunks await this.redis.rPush(`stream:chunks:${sessionId}`, chunk); await this.redis.expire(` stream:chunks:${sessionId}`, 3600); } async completeSession(sessionId: string): Promise<void> { await this.redis.hSet(`stream:session:${sessionId}`, { status: "complete", completedAt: Date.now().toString(), }); } async tryResume(sessionId: string): Promise<string[] | null> { const session = await this.redis.get(`stream:session:${sessionId}`); if (!session) return null; const parsed: StreamingSession = JSON.parse(session); if (parsed.status === "complete") { // Already finished — return every chunk, no re-calling the LLM const chunks = await this.redis.lRange( `stream:chunks:${sessionId}`, 0, -1 ); return chunks; } if (parsed.status === "streaming") { // Interrupted — return what we have so the client renders from the break const partialChunks = await this.redis.lRange( `stream:chunks:${sessionId}`, 0, -1 ); return partialChunks; // Caller decides: resume or regenerate } return null; }}
And the client side of the contract:
// The client sends a session_id with every requestconst sessionId = `${userId}-${Date.now()}-${Math.random().toString(36).slice(2)}`; const response = await fetch("/api/chat", { method: "POST", headers: { "Content-Type": "application/json", "X-Stream-Session-ID": sessionId, // Reuse the same sessionId on retry "X-Resume-From": lastReceivedIndex.toString(), // Which chunk to resume from }, body: JSON.stringify({ messages }),});
In a RAG pipeline, the ingestion stage — chunk, embed, upsert — gets re-run more often than you’d think. Users re-upload the same file. A pipeline job re-triggers. A crashed worker restarts and reruns the task.
Without dedup, you end up with three copies of the same document’s embeddings in your vector store. Retrieval returns duplicates, ranking gets polluted, and generation quality drops.
On a personal knowledge base of roughly 2,000 documents, the difference was stark. A pipeline rerun with zero content changes dropped from 1,000 embedding calls to zero. A 10%-changed rerun dropped to just the 100 documents that actually changed.
Duplicate vectors went from 3× (after three reruns) to none — and removing them lifted retrieval accuracy by about 8% on MRR@5, self-measured.
Plenty of LLM apps trigger an agent off an incoming webhook — a new GitHub PR triggers a code review, a new ticket triggers a classifier. Webhook providers guarantee at-least-once delivery: no HTTP 200 back in time, and they redeliver until they get one.
If your handler isn’t idempotent, here’s the sequence:
// webhook-dedup.tsinterface WebhookEvent { id: string; // The provider's unique event ID type: string; payload: unknown; deliveredAt: number;} async function handleWebhookIdempotent( event: WebhookEvent, redis: RedisClient, handler: (event: WebhookEvent) => Promise<void>): Promise<{ status: "processed" | "duplicate" | "processing" }> { const eventKey = `webhook:event:${event.id}`; // Atomic SET NX: only the first request to arrive can set this key const isFirst = await redis.set(eventKey, JSON.stringify({ status: "processing", startedAt: Date.now(), }), { NX: true, EX: 300, // Redeliveries within 5 minutes count as duplicates }); if (!isFirst) { // A record already exists — check its status const existing = await redis.get(eventKey); const record = existing ? JSON.parse(existing) : null; if (record?.status === "done") { return { status: "duplicate" }; // Idempotent — just return 200 } if (record?.status === "processing") { // Still in flight from a concurrent delivery. // Return 200 so the provider stops retrying, but skip the handler. return { status: "processing" }; } } try { await handler(event); await redis.set(eventKey, JSON.stringify({ status: "done", completedAt: Date.now(), }), { EX: 300 }); return { status: "processed" }; } catch (error) { // On failure, delete the key so the next delivery can retry await redis.del(eventKey); throw error; }}
Where each provider puts the event ID:
Each of these five traps maps to one of four protection layers, running from the request’s entry point down to the actual side effect. Every layer needs its own idempotency control — one layer catching the request doesn’t excuse the next from checking.
The pattern repeats at every layer: check first, lock with SET NX, do the work, cache the result, release the lock. What changes layer to layer is only the TTL and the definition of the work — an LLM call, a tool side effect, a document embedding, a webhook handler.
Myth 1 — a zero temperature. Setting temperature to 0 makes output more deterministic. It does nothing to stop a duplicate call. The API still gets hit twice, the bill still doubles, the side effect still runs twice. Idempotency is a request-layer problem, not a model-determinism problem.
Myth 2 — telling the prompt. An instruction like only call charge_card one time can’t constrain a framework’s retry behavior. When the framework retries on timeout, it isn’t re-reading your prompt. Tool-level protection lives in code, not in the prompt.
Myth 3 — assuming idempotent means runs exactly once. The precise definition: the same logical operation produces the same result no matter how many times it runs. A failed operation is allowed to retry — failure doesn’t count as already executed. Your dedup logic should only lock in records that succeeded.
Myth 4 — putting the key in the request body. If the client times out at the HTTP layer while the server is still processing, a retry that regenerates a fresh UUID sends a brand-new key — and your protection never fires. The key has to be stable at the level of business semantics, not generated fresh on every HTTP attempt.
□ LLM API calls: Idempotency-Key + Redis SET NX lock + TTL-cached result□ Agent tool calls: a dedup ledger keyed on run_id + tool + args_hash□ Streaming: a session manager that stores chunks and supports resume□ RAG ingestion: SHA-256 content fingerprint + vector-store upsert semantics□ Webhooks: event ID dedup, SET NX lock, delete the key on failure to allow retry□ Monitoring: track duplicate_skipped / idempotency_hit — a spike means upstream is over-delivering□ TTLs: match each layer's TTL to its real idempotency window, not "cache forever"
Idempotency isn’t a new problem. What’s new is how much harder LLM apps make it — non-deterministic models, stateful tool chains, streams that blur what done means.
Designing for it from your first LLM API call is far cheaper than explaining a double charge after the fact.
If this saved you from explaining a double charge to a very annoyed user, a clap 👏 (or fifty) helps other developers find it. And tell me in the comments which of these five traps bit you first — I read every one.
Your AI Agent Keeps Retrying. It’s Costing You $5,000 a Year. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.