{"slug": "your-ai-agent-keeps-retrying-its-costing-you-5000-a-year", "title": "Your AI Agent Keeps Retrying. It’s Costing You $5,000 a Year.", "summary": "A missing idempotency layer in LLM applications can cost teams $5,000 a year in wasted API calls, according to a technical analysis. The article explains that non-deterministic models, stateful agent tool chains, and streaming responses make standard backend idempotency playbooks insufficient, and provides a Redis-based solution using SET NX locks and caching to prevent duplicate charges and retries.", "body_md": "Here’s a moment every team shipping an LLM app eventually hits: a request times out, something retries, and now support is fielding a ticket that says the AI charged someone twice.\n\nNobody wrote a bug. The model didn’t hallucinate. You shipped a retry path with no idempotency behind it — and in an LLM app, that gap is far more dangerous than it is in a normal CRUD backend.\n\nBackend engineers already know idempotency. GET and DELETE are naturally safe, POST isn’t, and an Idempotency-Key plus a lock handles it. **That playbook isn’t enough here**, for three reasons.\n\n**LLMs aren’t deterministic.** The same prompt returns a different response every time once temperature goes above 0. You can’t hash the request body to tell a retry apart from a user sending two similar messages.\n\n**Agent tool calls are a stateful chain of side effects.** Say an agent chains three tools: create_order(), then charge_card(), then send_confirmation_email() — and the network dies right after step two. The framework retries the *whole chain*. The card gets charged twice. Support calls it an AI bug. It’s a missing idempotency layer.\n\n**Streaming blurs what “success” means.** A normal API call is either done or it isn’t. In a stream, you might have consumed 500 tokens when the connection drops. Success or failure? Does the retry get billed again?\n\nAll three compound under high concurrency, long tasks, and multi-agent setups. Here are the five traps that show up in production, and the fix for each.\n\nSay your RAG app runs 1,000 queries a day at roughly $0.003 each (~1,000 input tokens + 300 output). A 5% timeout rate means 50 retries a day. With no idempotency cache, every one of those retries pays full price for a call you already made.\n\nRun the math:\n\nFive thousand dollars, on requests you already paid for once. And it produces **no error log, no failed request, and no alert** — the bill is the only place it ever shows up.\n\nIt gets worse when the client also retries. A frontend retry library fires again after the server already succeeded, and now you’ve paid for the LLM call twice, written the DB twice, and pushed the notification twice.\n\n``` js\n// idempotent-llm-call.tsimport { createClient } from \"redis\";import { createHash } from \"crypto\"; interface LLMRequest {  model: string;  messages: Array<{ role: string; content: string }>;  temperature?: number;} const redis = createClient({ url: process.env.REDIS_URL }); async function idempotentLLMCall(  idempotencyKey: string,  request: LLMRequest,  ttlSeconds = 3600): Promise<string> {  const cacheKey = `llm:idem:${idempotencyKey}`;  const lockKey = `llm:lock:${idempotencyKey}`;   // 1. Check the cache first  const cached = await redis.get(cacheKey);  if (cached) {    return JSON.parse(cached).response;  }   // 2. Acquire the lock with SET NX so concurrent retries don't all hit the LLM  const lockAcquired = await redis.set(lockKey, \"1\", {    NX: true,    EX: 30, // 30s lock timeout — past this, assume the LLM call is stuck  });   if (!lockAcquired) {    // Someone else holds the lock — poll the cache until they finish    return await pollForResult(cacheKey, 30000);  }   try {    // 3. The actual LLM call    const response = await callLLMAPI(request);     // 4. Atomically cache the result (repeat requests within the TTL return this)    await redis.setEx(      cacheKey,      ttlSeconds,      JSON.stringify({ response, cachedAt: Date.now() })    );     return response;  } finally {    await redis.del(lockKey);  }} async function pollForResult(cacheKey: string, timeoutMs: number): Promise<string> {  const deadline = Date.now() + timeoutMs;  while (Date.now() < deadline) {    await new Promise((r) => setTimeout(r, 200));    const result = await redis.get(cacheKey);    if (result) return JSON.parse(result).response;  }  throw new Error(\"Idempotency lock timeout: upstream may have failed\");}\n```\n\nThe key move is SET NX — set only if the key doesn’t exist yet. Among concurrent retries, **exactly one actually calls the LLM.** The rest wait for that result instead of firing their own request.\n\nThe right construction depends on where the call originates:\n\nOne caveat on that last one: it’s fuzzy by design, and similar prompts can carry different intent. In production, **reserve exact idempotency for system-level calls** and keep semantic caching as a separate, looser layer for free-form user input. Don’t merge the two.\n\nThis is the most dangerous idempotency failure in LLM apps. When an agent calls external tools — charging a card, sending an email, writing to a database — those side effects are usually irreversible.\n\nA postmortem write-up from tianpan.co (April 2026) catalogued real incidents from this exact failure mode:\n\nThe root cause is identical every time. **The framework retries at the “did I get a model response” layer, not the “did this side effect already happen” layer.** LangChain and every major agent SDK have this gap, because their retry logic was built around model output, not tool execution state.\n\n```\n# tool_idempotency_ledger.pyimport hashlibimport jsonimport timefrom typing import Any, Callable, Optionalfrom dataclasses import dataclassimport redis @dataclassclass ToolCallRecord:    tool_name: str    call_id: str    args_hash: str    status: str  # \"pending\" | \"success\" | \"failed\"    result: Optional[Any]    executed_at: float    completed_at: Optional[float] class ToolIdempotencyLedger:    \"\"\"    Checks the ledger before every tool call. A call that already    succeeded returns its cached result instead of re-running a    side-effecting tool when the agent retries.    \"\"\"     def __init__(self, redis_client: redis.Redis, ttl_seconds: int = 86400):        self.redis = redis_client        self.ttl = ttl_seconds     def _call_key(self, agent_run_id: str, tool_name: str, args: dict) -> str:        args_hash = hashlib.sha256(            json.dumps(args, sort_keys=True).encode()        ).hexdigest()[:16]        return f\"tool:idem:{agent_run_id}:{tool_name}:{args_hash}\"     def wrap(self, agent_run_id: str, tool_name: str):        \"\"\"Decorator: turns any tool function into an idempotent version.\"\"\"        def decorator(fn: Callable) -> Callable:            def wrapped(*args, **kwargs) -> Any:                cache_key = self._call_key(agent_run_id, tool_name, kwargs)                 # Check the ledger                existing = self.redis.get(cache_key)                if existing:                    record = json.loads(existing)                    if record[\"status\"] == \"success\":                        print(f\"[Idempotent] Returning cached result for {tool_name}\")                        return record[\"result\"]                    elif record[\"status\"] == \"pending\":                        # Last call is still in flight — wait it out                        raise RuntimeError(                            f\"Tool {tool_name} is already executing (pending). \"                            f\"Wait before retrying. call_key={cache_key}\"                        )                 # Record pending                pending_record = {                    \"tool_name\": tool_name,                    \"status\": \"pending\",                    \"executed_at\": time.time(),                    \"result\": None,                }                self.redis.setex(cache_key, self.ttl, json.dumps(pending_record))                 try:                    result = fn(*args, **kwargs)                     # Record success                    success_record = {                        **pending_record,                        \"status\": \"success\",                        \"result\": result,                        \"completed_at\": time.time(),                    }                    self.redis.setex(cache_key, self.ttl, json.dumps(success_record))                    return result                 except Exception as e:                    # Record failure (retries are allowed)                    failed_record = {                        **pending_record,                        \"status\": \"failed\",                        \"error\": str(e),                        \"completed_at\": time.time(),                    }                    self.redis.setex(cache_key, self.ttl, json.dumps(failed_record))                    raise             return wrapped        return decorator # Usageledger = ToolIdempotencyLedger(redis_client) @ledger.wrap(agent_run_id=\"run-abc-123\", tool_name=\"charge_card\")def charge_card(user_id: str, amount: float, currency: str = \"USD\"):    \"\"\"Idempotent now: the same run_id + args execute exactly once.\"\"\"    return payment_gateway.charge(user_id, amount, currency)\n```\n\nThree design decisions worth calling out:\n\nSSE streaming is standard for LLM apps now. But it blurs exactly where success ends:\n\n```\nclient receives chunk 1..200 → connection drops → client retries→ server re-calls the LLM → user sees a duplicated prefix plus the full response→ billed twice\n```\n\nIt gets worse if you do side effects mid-stream — saving to the DB every 100 tokens, say. **A retry re-runs those side effects too.**\n\n```\n// streaming-session-manager.tsinterface StreamingSession {  sessionId: string;  status: \"streaming\" | \"complete\" | \"failed\";  chunks: string[];  totalTokens: number;  completedAt?: number;} class StreamingSessionManager {  constructor(private redis: RedisClient) {}   async startSession(sessionId: string): Promise<void> {    const session: StreamingSession = {      sessionId,      status: \"streaming\",      chunks: [],      totalTokens: 0,    };    await this.redis.setEx(      `stream:session:${sessionId}`,      3600,      JSON.stringify(session)    );  }   async appendChunk(sessionId: string, chunk: string): Promise<void> {    // RPUSH appends atomically without overwriting existing chunks    await this.redis.rPush(`stream:chunks:${sessionId}`, chunk);    await this.redis.expire(` stream:chunks:${sessionId}`, 3600);  }   async completeSession(sessionId: string): Promise<void> {    await this.redis.hSet(`stream:session:${sessionId}`, {      status: \"complete\",      completedAt: Date.now().toString(),    });  }   async tryResume(sessionId: string): Promise<string[] | null> {    const session = await this.redis.get(`stream:session:${sessionId}`);    if (!session) return null;     const parsed: StreamingSession = JSON.parse(session);     if (parsed.status === \"complete\") {      // Already finished — return every chunk, no re-calling the LLM      const chunks = await this.redis.lRange(        `stream:chunks:${sessionId}`,        0,        -1      );      return chunks;    }     if (parsed.status === \"streaming\") {      // Interrupted — return what we have so the client renders from the break      const partialChunks = await this.redis.lRange(        `stream:chunks:${sessionId}`,        0,        -1      );      return partialChunks; // Caller decides: resume or regenerate    }     return null;  }}\n```\n\nAnd the client side of the contract:\n\n```\n// The client sends a session_id with every requestconst sessionId = `${userId}-${Date.now()}-${Math.random().toString(36).slice(2)}`; const response = await fetch(\"/api/chat\", {  method: \"POST\",  headers: {    \"Content-Type\": \"application/json\",    \"X-Stream-Session-ID\": sessionId, // Reuse the same sessionId on retry    \"X-Resume-From\": lastReceivedIndex.toString(), // Which chunk to resume from  },  body: JSON.stringify({ messages }),});\n```\n\nIn a RAG pipeline, the ingestion stage — chunk, embed, upsert — gets re-run more often than you’d think. Users re-upload the same file. A pipeline job re-triggers. A crashed worker restarts and reruns the task.\n\nWithout dedup, you end up with three copies of the same document’s embeddings in your vector store. **Retrieval returns duplicates, ranking gets polluted, and generation quality drops.**\n\n``` python\n# rag_ingest_idempotent.pyimport hashlibimport jsonfrom typing import Optionalimport redisfrom pathlib import Path def compute_document_fingerprint(content: bytes) -> str:    \"\"\"SHA-256 of the raw bytes — filename and path don't matter, only content.\"\"\"    return hashlib.sha256(content).hexdigest() def idempotent_ingest(    content: bytes,    metadata: dict,    redis_client: redis.Redis,    vector_store,    embedder,    ttl_days: int = 30,) -> dict:    \"\"\"    Idempotent ingestion: identical content gets embedded exactly once.    Returns {\"status\": \"cached\" | \"ingested\", \"doc_id\": str, \"chunks\": int}    \"\"\"    fingerprint = compute_document_fingerprint(content)    dedup_key = f\"rag:ingest:fingerprint:{fingerprint}\"     # Check the dedup cache    existing = redis_client.get(dedup_key)    if existing:        record = json.loads(existing)        print(f\"[RAG Ingest] Skipping duplicate: {fingerprint[:8]}... (doc_id={record['doc_id']})\")        return {\"status\": \"cached\", **record}     # Mark pending so two workers can't ingest the same document at once    lock_key = f\"rag:ingest:lock:{fingerprint}\"    lock_acquired = redis_client.set(lock_key, \"1\", nx=True, ex=120)    if not lock_acquired:        raise RuntimeError(f\"Document {fingerprint[:8]}... is being ingested by another worker\")     try:        # The actual ingestion        text = content.decode(\"utf-8\", errors=\"replace\")        chunks = chunk_document(text)        embeddings = embedder.embed_batch([c.text for c in chunks])        doc_id = f\"doc-{fingerprint[:16]}\"         # Upsert into the vector store (same ID overwrites, in most stores)        vector_store.upsert(            vectors=[                {                    \"id\": f\"{doc_id}-chunk-{i}\",                    \"values\": emb,                    \"metadata\": {**metadata, \"chunk_index\": i, \"text\": chunks[i].text},                }                for i, emb in enumerate(embeddings)            ]        )         result = {\"doc_id\": doc_id, \"chunks\": len(chunks), \"fingerprint\": fingerprint}         # Cache the dedup record (repeat ingestion within the TTL returns \"cached\")        redis_client.setex(            dedup_key, ttl_days * 86400, json.dumps(result)        )         return {\"status\": \"ingested\", **result}    finally:        redis_client.delete(lock_key)\n```\n\nOn a personal knowledge base of roughly 2,000 documents, the difference was stark. A pipeline rerun with **zero content changes dropped from 1,000 embedding calls to zero.** A 10%-changed rerun dropped to just the 100 documents that actually changed.\n\nDuplicate vectors went from 3× (after three reruns) to none — and removing them lifted retrieval accuracy by about **8% on MRR@5**, self-measured.\n\nPlenty of LLM apps trigger an agent off an incoming webhook — a new GitHub PR triggers a code review, a new ticket triggers a classifier. Webhook providers guarantee **at-least-once** delivery: no HTTP 200 back in time, and they redeliver until they get one.\n\nIf your handler isn’t idempotent, here’s the sequence:\n\n```\n// webhook-dedup.tsinterface WebhookEvent {  id: string;          // The provider's unique event ID  type: string;  payload: unknown;  deliveredAt: number;} async function handleWebhookIdempotent(  event: WebhookEvent,  redis: RedisClient,  handler: (event: WebhookEvent) => Promise<void>): Promise<{ status: \"processed\" | \"duplicate\" | \"processing\" }> {  const eventKey = `webhook:event:${event.id}`;   // Atomic SET NX: only the first request to arrive can set this key  const isFirst = await redis.set(eventKey, JSON.stringify({    status: \"processing\",    startedAt: Date.now(),  }), {    NX: true,    EX: 300, // Redeliveries within 5 minutes count as duplicates  });   if (!isFirst) {    // A record already exists — check its status    const existing = await redis.get(eventKey);    const record = existing ? JSON.parse(existing) : null;     if (record?.status === \"done\") {      return { status: \"duplicate\" }; // Idempotent — just return 200    }     if (record?.status === \"processing\") {      // Still in flight from a concurrent delivery.      // Return 200 so the provider stops retrying, but skip the handler.      return { status: \"processing\" };    }  }   try {    await handler(event);     await redis.set(eventKey, JSON.stringify({      status: \"done\",      completedAt: Date.now(),    }), { EX: 300 });     return { status: \"processed\" };  } catch (error) {    // On failure, delete the key so the next delivery can retry    await redis.del(eventKey);    throw error;  }}\n```\n\nWhere each provider puts the event ID:\n\nEach of these five traps maps to one of four protection layers, running from the request’s entry point down to the actual side effect. **Every layer needs its own idempotency control** — one layer catching the request doesn’t excuse the next from checking.\n\nThe pattern repeats at every layer: check first, lock with SET NX, do the work, cache the result, release the lock. What changes layer to layer is only the TTL and the definition of the work — an LLM call, a tool side effect, a document embedding, a webhook handler.\n\n**Myth 1 — a zero temperature.** Setting temperature to 0 makes output more deterministic. It does nothing to stop a duplicate call. The API still gets hit twice, the bill still doubles, the side effect still runs twice. Idempotency is a request-layer problem, not a model-determinism problem.\n\n**Myth 2 — telling the prompt.** An instruction like *only call charge_card one time* can’t constrain a framework’s retry behavior. When the framework retries on timeout, it isn’t re-reading your prompt. **Tool-level protection lives in code, not in the prompt.**\n\n**Myth 3 — assuming idempotent means runs exactly once.** The precise definition: *the same logical operation produces the same result no matter how many times it runs.* A failed operation is allowed to retry — failure doesn’t count as already executed. Your dedup logic should only lock in records that **succeeded**.\n\n**Myth 4 — putting the key in the request body.** If the client times out at the HTTP layer while the server is still processing, a retry that regenerates a fresh UUID sends a brand-new key — and your protection never fires. **The key has to be stable at the level of business semantics**, not generated fresh on every HTTP attempt.\n\n```\n□ LLM API calls: Idempotency-Key + Redis SET NX lock + TTL-cached result□ Agent tool calls: a dedup ledger keyed on run_id + tool + args_hash□ Streaming: a session manager that stores chunks and supports resume□ RAG ingestion: SHA-256 content fingerprint + vector-store upsert semantics□ Webhooks: event ID dedup, SET NX lock, delete the key on failure to allow retry□ Monitoring: track duplicate_skipped / idempotency_hit — a spike means upstream is over-delivering□ TTLs: match each layer's TTL to its real idempotency window, not \"cache forever\"\n```\n\nIdempotency isn’t a new problem. What’s new is how much harder LLM apps make it — non-deterministic models, stateful tool chains, streams that blur what done means.\n\n**Designing for it from your first LLM API call is far cheaper than explaining a double charge after the fact.**\n\nIf this saved you from explaining a double charge to a very annoyed user, a clap 👏 (or fifty) helps other developers find it. And tell me in the comments which of these five traps bit you first — I read every one.\n\n[Your AI Agent Keeps Retrying. It’s Costing You $5,000 a Year.](https://pub.towardsai.net/your-ai-agent-keeps-retrying-its-costing-you-5-000-a-year-cca8251209a1) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/your-ai-agent-keeps-retrying-its-costing-you-5000-a-year", "canonical_source": "https://pub.towardsai.net/your-ai-agent-keeps-retrying-its-costing-you-5-000-a-year-cca8251209a1?source=rss----98111c9905da---4", "published_at": "2026-08-01 21:01:01+00:00", "updated_at": "2026-08-01 21:22:09.443854+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Redis", "LLM"], "alternates": {"html": "https://wpnews.pro/news/your-ai-agent-keeps-retrying-its-costing-you-5000-a-year", "markdown": "https://wpnews.pro/news/your-ai-agent-keeps-retrying-its-costing-you-5000-a-year.md", "text": "https://wpnews.pro/news/your-ai-agent-keeps-retrying-its-costing-you-5000-a-year.txt", "jsonld": "https://wpnews.pro/news/your-ai-agent-keeps-retrying-its-costing-you-5000-a-year.jsonld"}}