Your AI Agent Keeps Retrying. It’s Costing You $5,000 a Year.

A missing idempotency layer in LLM applications can cost teams $5,000 a year in wasted API calls, according to a technical analysis. The article explains that non-deterministic models, stateful agent tool chains, and streaming responses make standard backend idempotency playbooks insufficient, and provides a Redis-based solution using SET NX locks and caching to prevent duplicate charges and retries.

Here’s a moment every team shipping an LLM app eventually hits: a request times out, something retries, and now support is fielding a ticket that says the AI charged someone twice. Nobody wrote a bug. The model didn’t hallucinate. You shipped a retry path with no idempotency behind it — and in an LLM app, that gap is far more dangerous than it is in a normal CRUD backend. Backend engineers already know idempotency. GET and DELETE are naturally safe, POST isn’t, and an Idempotency-Key plus a lock handles it. That playbook isn’t enough here , for three reasons. LLMs aren’t deterministic. The same prompt returns a different response every time once temperature goes above 0. You can’t hash the request body to tell a retry apart from a user sending two similar messages. Agent tool calls are a stateful chain of side effects. Say an agent chains three tools: create order , then charge card , then send confirmation email — and the network dies right after step two. The framework retries the whole chain . The card gets charged twice. Support calls it an AI bug. It’s a missing idempotency layer. Streaming blurs what “success” means. A normal API call is either done or it isn’t. In a stream, you might have consumed 500 tokens when the connection drops. Success or failure? Does the retry get billed again? All three compound under high concurrency, long tasks, and multi-agent setups. Here are the five traps that show up in production, and the fix for each. Say your RAG app runs 1,000 queries a day at roughly $0.003 each ~1,000 input tokens + 300 output . A 5% timeout rate means 50 retries a day. With no idempotency cache, every one of those retries pays full price for a call you already made. Run the math: Five thousand dollars, on requests you already paid for once. And it produces no error log, no failed request, and no alert — the bill is the only place it ever shows up. It gets worse when the client also retries. A frontend retry library fires again after the server already succeeded, and now you’ve paid for the LLM call twice, written the DB twice, and pushed the notification twice. js // idempotent-llm-call.tsimport { createClient } from "redis";import { createHash } from "crypto"; interface LLMRequest { model: string; messages: Array<{ role: string; content: string } ; temperature?: number;} const redis = createClient { url: process.env.REDIS URL } ; async function idempotentLLMCall idempotencyKey: string, request: LLMRequest, ttlSeconds = 3600 : Promise<string { const cacheKey = llm:idem:${idempotencyKey} ; const lockKey = llm:lock:${idempotencyKey} ; // 1. Check the cache first const cached = await redis.get cacheKey ; if cached { return JSON.parse cached .response; } // 2. Acquire the lock with SET NX so concurrent retries don't all hit the LLM const lockAcquired = await redis.set lockKey, "1", { NX: true, EX: 30, // 30s lock timeout — past this, assume the LLM call is stuck } ; if lockAcquired { // Someone else holds the lock — poll the cache until they finish return await pollForResult cacheKey, 30000 ; } try { // 3. The actual LLM call const response = await callLLMAPI request ; // 4. Atomically cache the result repeat requests within the TTL return this await redis.setEx cacheKey, ttlSeconds, JSON.stringify { response, cachedAt: Date.now } ; return response; } finally { await redis.del lockKey ; }} async function pollForResult cacheKey: string, timeoutMs: number : Promise<string { const deadline = Date.now + timeoutMs; while Date.now < deadline { await new Promise r = setTimeout r, 200 ; const result = await redis.get cacheKey ; if result return JSON.parse result .response; } throw new Error "Idempotency lock timeout: upstream may have failed" ;} The key move is SET NX — set only if the key doesn’t exist yet. Among concurrent retries, exactly one actually calls the LLM. The rest wait for that result instead of firing their own request. The right construction depends on where the call originates: One caveat on that last one: it’s fuzzy by design, and similar prompts can carry different intent. In production, reserve exact idempotency for system-level calls and keep semantic caching as a separate, looser layer for free-form user input. Don’t merge the two. This is the most dangerous idempotency failure in LLM apps. When an agent calls external tools — charging a card, sending an email, writing to a database — those side effects are usually irreversible. A postmortem write-up from tianpan.co April 2026 catalogued real incidents from this exact failure mode: The root cause is identical every time. The framework retries at the “did I get a model response” layer, not the “did this side effect already happen” layer. LangChain and every major agent SDK have this gap, because their retry logic was built around model output, not tool execution state. tool idempotency ledger.pyimport hashlibimport jsonimport timefrom typing import Any, Callable, Optionalfrom dataclasses import dataclassimport redis @dataclassclass ToolCallRecord: tool name: str call id: str args hash: str status: str "pending" | "success" | "failed" result: Optional Any executed at: float completed at: Optional float class ToolIdempotencyLedger: """ Checks the ledger before every tool call. A call that already succeeded returns its cached result instead of re-running a side-effecting tool when the agent retries. """ def init self, redis client: redis.Redis, ttl seconds: int = 86400 : self.redis = redis client self.ttl = ttl seconds def call key self, agent run id: str, tool name: str, args: dict - str: args hash = hashlib.sha256 json.dumps args, sort keys=True .encode .hexdigest :16 return f"tool:idem:{agent run id}:{tool name}:{args hash}" def wrap self, agent run id: str, tool name: str : """Decorator: turns any tool function into an idempotent version.""" def decorator fn: Callable - Callable: def wrapped args, kwargs - Any: cache key = self. call key agent run id, tool name, kwargs Check the ledger existing = self.redis.get cache key if existing: record = json.loads existing if record "status" == "success": print f" Idempotent Returning cached result for {tool name}" return record "result" elif record "status" == "pending": Last call is still in flight — wait it out raise RuntimeError f"Tool {tool name} is already executing pending . " f"Wait before retrying. call key={cache key}" Record pending pending record = { "tool name": tool name, "status": "pending", "executed at": time.time , "result": None, } self.redis.setex cache key, self.ttl, json.dumps pending record try: result = fn args, kwargs Record success success record = { pending record, "status": "success", "result": result, "completed at": time.time , } self.redis.setex cache key, self.ttl, json.dumps success record return result except Exception as e: Record failure retries are allowed failed record = { pending record, "status": "failed", "error": str e , "completed at": time.time , } self.redis.setex cache key, self.ttl, json.dumps failed record raise return wrapped return decorator Usageledger = ToolIdempotencyLedger redis client @ledger.wrap agent run id="run-abc-123", tool name="charge card" def charge card user id: str, amount: float, currency: str = "USD" : """Idempotent now: the same run id + args execute exactly once.""" return payment gateway.charge user id, amount, currency Three design decisions worth calling out: SSE streaming is standard for LLM apps now. But it blurs exactly where success ends: client receives chunk 1..200 → connection drops → client retries→ server re-calls the LLM → user sees a duplicated prefix plus the full response→ billed twice It gets worse if you do side effects mid-stream — saving to the DB every 100 tokens, say. A retry re-runs those side effects too. // streaming-session-manager.tsinterface StreamingSession { sessionId: string; status: "streaming" | "complete" | "failed"; chunks: string ; totalTokens: number; completedAt?: number;} class StreamingSessionManager { constructor private redis: RedisClient {} async startSession sessionId: string : Promise<void { const session: StreamingSession = { sessionId, status: "streaming", chunks: , totalTokens: 0, }; await this.redis.setEx stream:session:${sessionId} , 3600, JSON.stringify session ; } async appendChunk sessionId: string, chunk: string : Promise<void { // RPUSH appends atomically without overwriting existing chunks await this.redis.rPush stream:chunks:${sessionId} , chunk ; await this.redis.expire stream:chunks:${sessionId} , 3600 ; } async completeSession sessionId: string : Promise<void { await this.redis.hSet stream:session:${sessionId} , { status: "complete", completedAt: Date.now .toString , } ; } async tryResume sessionId: string : Promise<string | null { const session = await this.redis.get stream:session:${sessionId} ; if session return null; const parsed: StreamingSession = JSON.parse session ; if parsed.status === "complete" { // Already finished — return every chunk, no re-calling the LLM const chunks = await this.redis.lRange stream:chunks:${sessionId} , 0, -1 ; return chunks; } if parsed.status === "streaming" { // Interrupted — return what we have so the client renders from the break const partialChunks = await this.redis.lRange stream:chunks:${sessionId} , 0, -1 ; return partialChunks; // Caller decides: resume or regenerate } return null; }} And the client side of the contract: // The client sends a session id with every requestconst sessionId = ${userId}-${Date.now }-${Math.random .toString 36 .slice 2 } ; const response = await fetch "/api/chat", { method: "POST", headers: { "Content-Type": "application/json", "X-Stream-Session-ID": sessionId, // Reuse the same sessionId on retry "X-Resume-From": lastReceivedIndex.toString , // Which chunk to resume from }, body: JSON.stringify { messages } ,} ; In a RAG pipeline, the ingestion stage — chunk, embed, upsert — gets re-run more often than you’d think. Users re-upload the same file. A pipeline job re-triggers. A crashed worker restarts and reruns the task. Without dedup, you end up with three copies of the same document’s embeddings in your vector store. Retrieval returns duplicates, ranking gets polluted, and generation quality drops. python rag ingest idempotent.pyimport hashlibimport jsonfrom typing import Optionalimport redisfrom pathlib import Path def compute document fingerprint content: bytes - str: """SHA-256 of the raw bytes — filename and path don't matter, only content.""" return hashlib.sha256 content .hexdigest def idempotent ingest content: bytes, metadata: dict, redis client: redis.Redis, vector store, embedder, ttl days: int = 30, - dict: """ Idempotent ingestion: identical content gets embedded exactly once. Returns {"status": "cached" | "ingested", "doc id": str, "chunks": int} """ fingerprint = compute document fingerprint content dedup key = f"rag:ingest:fingerprint:{fingerprint}" Check the dedup cache existing = redis client.get dedup key if existing: record = json.loads existing print f" RAG Ingest Skipping duplicate: {fingerprint :8 }... doc id={record 'doc id' } " return {"status": "cached", record} Mark pending so two workers can't ingest the same document at once lock key = f"rag:ingest:lock:{fingerprint}" lock acquired = redis client.set lock key, "1", nx=True, ex=120 if not lock acquired: raise RuntimeError f"Document {fingerprint :8 }... is being ingested by another worker" try: The actual ingestion text = content.decode "utf-8", errors="replace" chunks = chunk document text embeddings = embedder.embed batch c.text for c in chunks doc id = f"doc-{fingerprint :16 }" Upsert into the vector store same ID overwrites, in most stores vector store.upsert vectors= { "id": f"{doc id}-chunk-{i}", "values": emb, "metadata": { metadata, "chunk index": i, "text": chunks i .text}, } for i, emb in enumerate embeddings result = {"doc id": doc id, "chunks": len chunks , "fingerprint": fingerprint} Cache the dedup record repeat ingestion within the TTL returns "cached" redis client.setex dedup key, ttl days 86400, json.dumps result return {"status": "ingested", result} finally: redis client.delete lock key On a personal knowledge base of roughly 2,000 documents, the difference was stark. A pipeline rerun with zero content changes dropped from 1,000 embedding calls to zero. A 10%-changed rerun dropped to just the 100 documents that actually changed. Duplicate vectors went from 3× after three reruns to none — and removing them lifted retrieval accuracy by about 8% on MRR@5 , self-measured. Plenty of LLM apps trigger an agent off an incoming webhook — a new GitHub PR triggers a code review, a new ticket triggers a classifier. Webhook providers guarantee at-least-once delivery: no HTTP 200 back in time, and they redeliver until they get one. If your handler isn’t idempotent, here’s the sequence: // webhook-dedup.tsinterface WebhookEvent { id: string; // The provider's unique event ID type: string; payload: unknown; deliveredAt: number;} async function handleWebhookIdempotent event: WebhookEvent, redis: RedisClient, handler: event: WebhookEvent = Promise<void : Promise<{ status: "processed" | "duplicate" | "processing" } { const eventKey = webhook:event:${event.id} ; // Atomic SET NX: only the first request to arrive can set this key const isFirst = await redis.set eventKey, JSON.stringify { status: "processing", startedAt: Date.now , } , { NX: true, EX: 300, // Redeliveries within 5 minutes count as duplicates } ; if isFirst { // A record already exists — check its status const existing = await redis.get eventKey ; const record = existing ? JSON.parse existing : null; if record?.status === "done" { return { status: "duplicate" }; // Idempotent — just return 200 } if record?.status === "processing" { // Still in flight from a concurrent delivery. // Return 200 so the provider stops retrying, but skip the handler. return { status: "processing" }; } } try { await handler event ; await redis.set eventKey, JSON.stringify { status: "done", completedAt: Date.now , } , { EX: 300 } ; return { status: "processed" }; } catch error { // On failure, delete the key so the next delivery can retry await redis.del eventKey ; throw error; }} Where each provider puts the event ID: Each of these five traps maps to one of four protection layers, running from the request’s entry point down to the actual side effect. Every layer needs its own idempotency control — one layer catching the request doesn’t excuse the next from checking. The pattern repeats at every layer: check first, lock with SET NX, do the work, cache the result, release the lock. What changes layer to layer is only the TTL and the definition of the work — an LLM call, a tool side effect, a document embedding, a webhook handler. Myth 1 — a zero temperature. Setting temperature to 0 makes output more deterministic. It does nothing to stop a duplicate call. The API still gets hit twice, the bill still doubles, the side effect still runs twice. Idempotency is a request-layer problem, not a model-determinism problem. Myth 2 — telling the prompt. An instruction like only call charge card one time can’t constrain a framework’s retry behavior. When the framework retries on timeout, it isn’t re-reading your prompt. Tool-level protection lives in code, not in the prompt. Myth 3 — assuming idempotent means runs exactly once. The precise definition: the same logical operation produces the same result no matter how many times it runs. A failed operation is allowed to retry — failure doesn’t count as already executed. Your dedup logic should only lock in records that succeeded . Myth 4 — putting the key in the request body. If the client times out at the HTTP layer while the server is still processing, a retry that regenerates a fresh UUID sends a brand-new key — and your protection never fires. The key has to be stable at the level of business semantics , not generated fresh on every HTTP attempt. □ LLM API calls: Idempotency-Key + Redis SET NX lock + TTL-cached result□ Agent tool calls: a dedup ledger keyed on run id + tool + args hash□ Streaming: a session manager that stores chunks and supports resume□ RAG ingestion: SHA-256 content fingerprint + vector-store upsert semantics□ Webhooks: event ID dedup, SET NX lock, delete the key on failure to allow retry□ Monitoring: track duplicate skipped / idempotency hit — a spike means upstream is over-delivering□ TTLs: match each layer's TTL to its real idempotency window, not "cache forever" Idempotency isn’t a new problem. What’s new is how much harder LLM apps make it — non-deterministic models, stateful tool chains, streams that blur what done means. Designing for it from your first LLM API call is far cheaper than explaining a double charge after the fact. If this saved you from explaining a double charge to a very annoyed user, a clap 👏 or fifty helps other developers find it. And tell me in the comments which of these five traps bit you first — I read every one. Your AI Agent Keeps Retrying. It’s Costing You $5,000 a Year. https://pub.towardsai.net/your-ai-agent-keeps-retrying-its-costing-you-5-000-a-year-cca8251209a1 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.