{"slug": "why-ai-apps-crash-under-real-traffic-and-how-to-fix-it-in-2-lines", "title": "Why AI Apps Crash Under Real Traffic (And How to Fix It in 2 Lines)", "summary": "A developer released @rohan9024/retrykit v2.0.0, a zero-dependency TypeScript reliability library for AI apps that replaces generic retry wrappers such as p-retry with LLM-aware handling of rate limits, multi-key rotation, circuit breaking, stream recovery and context compaction. The package's wrapAI() helper sets maxRetries to 0 on official SDKs to prevent stacked retries from multiplying into 9x-12x traffic storms, quarantines keys on hard billing errors like insufficient_quota instead of sleeping, and preserves tool-call message pairs when trimming context.", "body_md": "**TL;DR:** Generic retry wrappers like `p-retry` or `while` loops weren't built for LLM APIs. They multiply SDK retries into **9x–12x traffic storms**, sleep on hard billing errors (`insufficient_quota`), corrupt streamed UI text on mid-stream disconnects, and break tool-call message histories. [`@rohan9024/retrykit`](https://github.com/rohan9024/retrykit) (` v2.0.0`) is a **zero-dependency TypeScript reliability engine** for AI apps that solves rate limits, multi-key rotation, circuit breaking, stream recovery, and atomic context compaction in one package.\n\nIf you have ever shipped an AI feature with OpenAI, Anthropic, or Gemini, you have probably seen this story play out in production:\n\n`429 Too Many Requests` (or Anthropic returns `529 Overloaded`).` maxRetries: 2` by default). Wrapping your own 3-retry loop around it means `3 × 4`\nThis guide explains **why standard HTTP retry logic fails on LLM APIs** and how to fix it cleanly in under a minute using **RetryKit (**`@rohan9024/retrykit`**)**.\n\nBefore writing code, it helps to understand what actually goes wrong when you use a generic retry helper for LLMs:\n\n`maxRetries` Multiplication)\nMost official AI SDKs already retry failed requests twice internally. When developers add a custom `retry()` wrapper or queue around the SDK without setting `maxRetries: 0`, retries multiply instead of adding up—flooding your own API quota during an outage.\n\n`insufficient_quota`)\nNot every `429` error means *\"slow down and try again.\"* OpenAI returns `429` for `insufficient_quota` or spend limits, and Anthropic returns `enforced_spend_limit_reached`. Waiting 1s → 2s → 4s → 8s on a dead key just freezes your user's request for 15+ seconds for no reason. You should **quarantine that key immediately and rotate to a backup key** with zero sleep.\n\nProviders return rate-limit resets in wildly different formats:\n\n`retry-after-ms` (milliseconds)`retry-after` (seconds or HTTP date strings)`\"464ms\"`, `\"1m30s\"`)` anthropic-ratelimit-requests-reset`)\nGeneric backoff ignores these headers—retrying either too early (hitting another 429) or much too late.\n\nSuppose your LLM streams 40 tokens to the user's chat screen and then stalls on token 41. If a naive wrapper restarts the HTTP call from scratch and yields tokens from the beginning, **your user sees the first paragraph pasted twice**.\n\nSafe streaming recovery must treat **pre-first-token failures** differently from **mid-stream failures** (where resuming requires an explicit checkpoint mechanism like `getCheckpoint`).\n\nWhen an agent hits a context limit and you trim older chat messages blindly, you can accidentally delete an `assistant` message containing `tool_calls` while leaving its `tool` response behind (or vice versa). Both OpenAI and Anthropic reject orphaned tool pairs with a hard `400 Bad Request`.\n\n`@rohan9024/retrykit` (Zero Dependencies, AI-Native)\n[`@rohan9024/retrykit`](https://www.npmjs.com/package/@rohan9024/retrykit) is a lightweight, zero-dependency TypeScript reliability library built specifically for AI apps, Node.js backends, and Edge runtimes (`Node >= 20`, Bun, Deno, Cloudflare Workers).\n\nInstall it with your package manager:\n\n```\nnpm install @rohan9024/retrykit\n```\n\n*(Also works with* `pnpm add @rohan9024/retrykit`*,* `yarn add @rohan9024/retrykit`*, or* `bun add @rohan9024/retrykit`*.)*\n\nThe fastest way to protect an existing codebase is `@rohan9024/retrykit/auto`:\n\n``` python\nimport OpenAI from \"openai\";\nimport { wrapAI } from \"@rohan9024/retrykit/auto\";\n\nconst openai = wrapAI(\n  new OpenAI({ apiKey: process.env.OPENAI_API_KEY }),\n  { provider: \"openai\" }\n);\n\n// Use the SDK exactly as before:\nconst response = await openai.chat.completions.create({\n  model: \"gpt-4o\",\n  messages: [{ role: \"user\", content: \"Explain circuit breakers simply.\" }],\n});\n```\n\n`wrapAI()` handles automatically under the hood:\n`maxRetries: 0` so the SDK and RetryKit never stack retries.`OPENAI_API_KEYS` (comma-separated) or `OPENAI_API_KEY_1`, `OPENAI_API_KEY_2`, etc. (plus `ANTHROPIC_API_KEYS` and `GEMINI_API_KEYS`).` Authorization`, `x-api-key`, `x-goog-api-key`) instead of mutating shared global client state.` credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded`, and `enforced_spend_limit_reached` and rotates immediately instead of wasting sleep time.`AbortController` signals (`attemptTimeoutMs`, `totalDeadlineMs`) so timed-out network calls actually abort instead of leaking sockets.\nAlready have a big TypeScript project? Run the built-in static doctor CLI to find unguarded AI clients and hidden double-retry loops:\n\n```\nnpx @rohan9024/retrykit doctor\n```\n\nIt scans your source files and points to the exact file and line where:\n\n`new OpenAI(...)` or `new Anthropic(...)` is called without protection`maxRetries > 0` could trigger double-retry amplification\nYou can also scaffold a project starter config safely with:\n\n```\nnpx @rohan9024/retrykit init --dry-run\n```\n\nRetryKit v2.0.0 ships clean ESM + CommonJS + TypeScript declarations across focused subpaths with `\"sideEffects\": false`:\n\n| Import Path | What It Solves | \n|---|---|\n| `@rohan9024/retrykit/auto` | Drop-in `wrapAI()` proxy, multi-key discovery (`discoverApiKeys` ), SDK retry neutralization | \n| `@rohan9024/retrykit/core` | Core `retry()` ,`CircuitBreaker` ,`KeyPool` , backoff + jitter, linked timeouts | \n| `@rohan9024/retrykit/fetch` | Composable `createRetryFetch()` with safe body cloning & idempotency checks | \n| `@rohan9024/retrykit/stream` | Streaming watchdog & explicit `getCheckpoint` -based safe recovery | \n| `@rohan9024/retrykit/compaction` | Tool-call-safe atomic context compaction (never orphans `tool_calls` /`tool` results) | \n| `@rohan9024/retrykit/budget` | `RetryBudget` and`TokenBucketRetryBudget` to cap retry amplification | \n| `@rohan9024/retrykit/ai-sdk` | Zero-dependency structural middleware ( `transformParams` ,`wrapGenerate` ,`wrapStream` ) for Vercel AI SDK | \n| `@rohan9024/retrykit/mcp` | Safe Model Context Protocol ( `MCP` ) retry wrapper (read-only allowlists; never retries unsafe`tools/call` by default) | \n| `@rohan9024/retrykit/validate` | Retry-on-validation-failure utilities | \n\n| Capability | `@rohan9024/retrykit` v2.0 | `p-retry` /`async-retry` | `cockatiel` | Raw OpenAI / Anthropic SDK | \n|---|---|---|---|---|\n| **Zero runtime dependencies** | Yes ( `0` ) | No | Yes | Full SDK | \n| **OpenAI / Anthropic reset header parsing** | Yes ( `464ms` ,`1m30s` , RFC 3339,`529` ) | No | No | Basic | \n| **Hard billing error fast-rotation** | Yes | No (sleeps & fails) | No | No | \n| **Per-request multi-key rotation** | Yes | No | No | No | \n| **Circuit breakers + token-bucket retry budgets** | Yes | No | Partial | No | \n| **Checkpoint-safe stream handling** | Yes ( `getCheckpoint` ) | No | No | No | \n| **Atomic tool-call context compaction** | Yes | No | No | No | \n| **Vercel AI SDK & MCP adapters included** | Yes | No | No | No | \n| **AI Coding Agent Skill + CLI**`doctor` | Yes | No | No | No | \n\nBuilding with an AI coding assistant? RetryKit includes an official **Agent Skill** (`skills/retrykit/SKILL.md` following the `agentskills.io` standard), `llms.txt`, `llms-full.txt`, `AGENTS.md`, and `context7.json`.\n\n```\nnpx skills add rohan9024/retrykit --skill retrykit\n```\n\n*\"Inspect my project for AI SDK calls (*`openai`*,* `@anthropic-ai/sdk`*, Vercel AI SDK, or* `fetch` *LLM requests). Run* `npx @rohan9024/retrykit doctor`*, install* `@rohan9024/retrykit` *with my project's package manager, wrap the clients using* `@rohan9024/retrykit/auto`*, ensure* `maxRetries: 0` *is set to prevent double-retry storms, and verify with tests.\"*\n\nYou can also run RetryKit's built-in stdio MCP diagnostic server in any MCP-compatible editor:\n\n```\nnpx @rohan9024/retrykit mcp\n```\n\nUse `@rohan9024/retrykit/auto` (`wrapAI`) or `@rohan9024/retrykit/core` (`retry` + `TokenBucketRetryBudget`). RetryKit automatically sets request-level `maxRetries: 0` on wrapped OpenAI/Anthropic SDK calls so SDK retries and custom retries never multiply, while honoring `retry-after-ms`, `x-ratelimit-reset-*`, and jittered backoff.\n\n`429 insufficient_quota` errors?\nNo. Sleeping and retrying an exhausted key wastes latency. RetryKit detects `insufficient_quota`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded`, and `enforced_spend_limit_reached`, failing fast on that key so a backup key can take over immediately.\n\nIf a stream fails mid-way after chunks have already been emitted, silently replaying from the beginning would duplicate text. RetryKit refuses silent replay after output begins unless you supply an explicit `getCheckpoint` recovery mechanism.\n\nNo. `@rohan9024/retrykit` has `\"dependencies\": {}` (zero runtime dependencies) and exposes tree-shakeable ESM/CJS subpath exports with `\"sideEffects\": false`.\n\n```\nnpm install @rohan9024/retrykit\n```\n\nIf RetryKit saves you from a production `429` incident, leave a star on GitHub or open an issue telling us which AI SDK or framework adapter you want next!", "url": "https://wpnews.pro/news/why-ai-apps-crash-under-real-traffic-and-how-to-fix-it-in-2-lines", "canonical_source": "https://dev.to/rohan9024/why-ai-apps-crash-under-real-traffic-and-how-to-fix-it-in-2-lines-43a1", "published_at": "2026-11-21 21:35:06+00:00", "updated_at": "2026-11-21 21:43:03.553848+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-agents", "mlops"], "entities": ["@rohan9024/retrykit", "OpenAI", "Anthropic", "Gemini", "Cloudflare Workers", "Deno", "Bun"], "also_reported_by": [], "takeaway": "Set maxRetries to 0 on official AI SDKs before adding any retry wrapper, since the SDK's default two retries multiplied by a custom three-retry loop produce up to 12 requests per call during an outage.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/why-ai-apps-crash-under-real-traffic-and-how-to-fix-it-in-2-lines", "markdown": "https://wpnews.pro/news/why-ai-apps-crash-under-real-traffic-and-how-to-fix-it-in-2-lines.md", "text": "https://wpnews.pro/news/why-ai-apps-crash-under-real-traffic-and-how-to-fix-it-in-2-lines.txt", "jsonld": "https://wpnews.pro/news/why-ai-apps-crash-under-real-traffic-and-how-to-fix-it-in-2-lines.jsonld"}}