# Preventing Runaway LLM Bills: Building a Resilient AI Worker Queue with Redis, BullMQ, and Model Fallbacks

> Source: <https://dev.to/sidsri14/preventing-runaway-llm-bills-building-a-resilient-ai-worker-queue-with-redis-bullmq-and-model-23eh>
> Published: 2026-11-05 16:45:27+00:00

Building autonomous agents or AI-driven background tasks is exciting—until a bad retry loop or unexpected 429 rate-limit cascades into a $2,000 API bill overnight.

In production, naive API calls with standard `try/catch` fail for three predictable reasons:

In this tutorial, we will construct a production-ready **AI Worker Queue** in TypeScript using **BullMQ** and **Redis**, featuring:

``` php
flowchart TD
    A[Incoming Task Request] --> B[BullMQ Job Queue]
    B --> C{Daily Budget Pre-Check}
    C -- Exceeds Spend Ceiling --> D[Abort / Kill Switch Triggered]
    C -- Within Budget Limit --> E[Resilient Multi-Provider Router]
    E -- Primary 429/Timeout --> F[Fallback Provider: Groq / Anthropic]
    E -- Primary Success --> G[Worker Output Delivered]
    F --> G
    G --> H[Atomic Redis Token Increment]
```

Before an expensive prompt is dispatched to an LLM provider, we pre-calculate estimated tokens and verify against an atomic Redis counter:

```
export class BudgetManager {
  private redis: Redis;
  private keyPrefix = "ai_budget:daily:";

  public async canExecute(estimatedTokens: number): Promise<{ allowed: boolean; currentUsage: number; reason?: string }> {
    const today = new Date().toISOString().split("T")[0];
    const key = `${this.keyPrefix}${today}`;
    const current = parseInt((await this.redis.get(key)) || "0", 10);

    if (current + estimatedTokens > MAX_DAILY_TOKEN_BUDGET) {
      return {
        allowed: false,
        currentUsage: current,
        reason: `Daily token limit breached: ${current} + ${estimatedTokens} > ${MAX_DAILY_TOKEN_BUDGET}`,
      };
    }

    return { allowed: true, currentUsage: current };
  }

  public async recordUsage(tokens: number): Promise<number> {
    const today = new Date().toISOString().split("T")[0];
    const key = `${this.keyPrefix}${today}`;
    const total = await this.redis.incrby(key, tokens);
    await this.redis.expire(key, 172800); // 48h TTL
    return total;
  }
}
```

If your primary provider returns a `429 Too Many Requests` or `500 Internal Error`, you shouldn't crash the queue. Instead, fall back to secondary high-availability endpoints seamlessly:

```
export class ResilientModelRouter {
  constructor(private providers: IModelProvider[]) {}

  public async executeWithFailover(req: ModelRequest): Promise<ModelResponse> {
    for (const provider of this.providers) {
      try {
        return await provider.execute(req);
      } catch (err: any) {
        console.warn(`[Failover] Provider ${provider.name} failed: ${err.message}. Trying next fallback...`);
      }
    }
    throw new Error(`All fallback providers exhausted for request [${req.id}]`);
  }
}
```

We bind the budget check and failover router directly into a BullMQ worker process with jittered exponential retries:

``` js
const worker = new Worker<ModelRequest, ModelResponse>(
  "resilient-ai-jobs",
  async (job: Job<ModelRequest>) => {
    const estimatedTokens = Math.ceil((job.data.prompt.length / 4)) + 500;

    // 1. Budget kill switch check
    const budgetStatus = await budgetManager.canExecute(estimatedTokens);
    if (!budgetStatus.allowed) {
      throw new Error(`[BUDGET_EXCEEDED] ${budgetStatus.reason}`);
    }

    // 2. Execute with automatic provider failover
    const response = await router.executeWithFailover(job.data);

    // 3. Atomically account for actual token consumption
    await budgetManager.recordUsage(response.tokensUsed);

    return response;
  },
  { connection: redisConnection, concurrency: 5 }
);
```

The complete, working source code is open-sourced on GitHub:

👉 **GitHub: sidsri14/resilient-ai-worker-kit**

For enterprise queue dashboards, distributed rate-limiting middleware, and production Docker configs, grab the **Enterprise License** via [Polar.sh](https://buy.polar.sh/polar_cl_4mrUXvOPjiB4LEIEBsVyJshejMA8oq7XIHPQV163dbE).