# AI Rate Limiting for LLM Calls in TypeScript

> Source: <https://dev.to/lakbud/ai-rate-limiting-for-llm-calls-in-typescript-3mfo>
> Published: 2026-11-20 18:10:06+00:00

**Recommendation:** Queue LLM calls locally with a concurrency and rate limiter before they hit the provider, and count retries against the same budget to avoid 429 storms.
More in the playbook "Controlling LLM cost": <https://wpnews.pro/playbooks/llm-cost>

You fire 50 LLM calls at once. A batch job, a fan out over documents, a busy hour. The provider only lets 10 through at a time, so the rest come back as `429 Too Many Requests`.

The usual fix is to retry 429s with backoff. That helps, but notice what it means: every retry is a request you sent after you already got rate limited. You are reacting to a limit you could have stayed under.

This post shows the difference with real numbers and how to rate limit LLM calls in TypeScript with [VernLLM](https://vernllm.dev), before the provider says no.

I ran a 50 call burst against a scripted provider that accepts 10 requests in flight and rejects the rest with 429. Both runs used `maxRetries: 2`.

|  | Retries only | `rateLimit: { maxConcurrent: 10 }` | 
|---|---|---|
| Calls answered | 20 | 50 | 
| Calls failed | 30 | 0 | 
| Provider 429s | 110 | 0 | 
| Requests sent | 130 | 50 | 

Retries alone sent 130 requests, collected 110 rejections, and still lost 30 calls. With a local limit, every call waited its turn and went out once.

```
npm i vern-llm openai
python
import OpenAI from 'openai';
import { VernLLM } from 'vern-llm';
import { fromOpenAI } from 'vern-llm/adapters';

const llm = new VernLLM({
  client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY })),
  model: 'gpt-6.2-sol',
  rateLimit: {
    requestsPerMinute: 500,
    tokensPerMinute: 200_000,
    maxConcurrent: 20,
  },
});

const results = await Promise.all(tickets.map((t) => llm.call({ userContent: t.text })));
```

`rateLimit` has three buckets, each optional:

A call goes out only when every configured bucket has room. Otherwise it waits.

Calls that cannot go out yet wait in strict first in, first out order. A large call is never starved by smaller ones behind it.

You decide how long a call may wait and how deep the queue may get:

```
rateLimit: {
  requestsPerMinute: 60,
  maxQueueMs: 10_000, // default 30000
  maxQueueSize: 50,   // default unbounded
},
```

When a call gives up waiting, it fails with a `rate_limited` error and a code you can branch on (`rate_limit_queue_timeout` or `rate_limit_queue_full`). These never reached the provider, so VernLLM does not retry them. A call too big to ever fit under `tokensPerMinute` fails right away with `rate_limit_capacity_exceeded` instead of blocking the queue forever.

Capacity is taken per attempt, not per call. A retry is a real request against the same provider limit, so it waits in the same queue as everything else. That is what keeps retries from turning one 429 into a storm of them.

You often do not know the provider's real limit, or it changes with your tier and their load. `aimd` lets the requests ceiling adjust itself: it grows a little on every success and shrinks on a real 429.

```
rateLimit: {
  requestsPerMinute: 500,
  aimd: { increaseBy: 10, decreaseFactor: 0.5, minCapacity: 50, maxCapacity: 1000 },
},
```

It shrinks at most once per 60 seconds, so a burst of 429s from calls already in flight counts as one overload. With `proactiveFloor`, adapters that can read rate limit headers shrink the ceiling when the provider reports low remaining capacity, before a 429 happens.

A 429 that still gets through keeps its own code, `provider_rate_limited`, so you can tell a real provider limit apart from VernLLM's own queue in your logs.

With fallback, each target gets its own limiter. A burst against a struggling primary cannot use up the slots a healthy backup needs:

``` js
const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-6.2-sol',
  rateLimit: { maxConcurrent: 50 },
  fallback: {
    client: fromGemini(gemini),
    model: 'gemini-4-neon',
    rateLimit: { maxConcurrent: 10 },
  },
});
```

The built in limiter lives in one process. If you run several instances, pass your own `RateLimiterAdapter`, backed by Redis or anything else, so every instance shares one budget against the provider.

Retrying 429s is damage control. A local limit keeps you under the provider's ceiling in the first place: queue in order, count every retry, adapt when the provider pushes back.

```
npm i vern-llm
```

Docs: [Rate limiting](https://vernllm.dev/docs/core/rate-limiting)
