cd /news/large-language-models/ai-rate-limiting-for-llm-calls-in-ty… · home › topics › large-language-models › article
[ARTICLE · art-173393] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

AI Rate Limiting for LLM Calls in TypeScript

A developer demonstrated that adding a local concurrency limiter to LLM calls in TypeScript, using the VernLLM library, eliminated provider 429 errors in a 50-call burst test. With retries alone, the test sent 130 requests, collected 110 rate-limit rejections and lost 30 calls; with rateLimit maxConcurrent set to 10, all 50 calls succeeded with zero 429s. The library offers requests-per-minute, tokens-per-minute and concurrency buckets, FIFO queuing with configurable timeouts, AIMD-based adaptive ceilings and per-target limiters for fallbacks.

recommendation

Queue LLM calls locally with a concurrency and rate limiter before they hit the provider, and count retries against the same budget to avoid 429 storms.

more in playbook: Controlling LLM cost
by read3 min views9 publishedNov 20, 2026

You fire 50 LLM calls at once. A batch job, a fan out over documents, a busy hour. The provider only lets 10 through at a time, so the rest come back as 429 Too Many Requests.

The usual fix is to retry 429s with backoff. That helps, but notice what it means: every retry is a request you sent after you already got rate limited. You are reacting to a limit you could have stayed under.

This post shows the difference with real numbers and how to rate limit LLM calls in TypeScript with VernLLM, before the provider says no.

I ran a 50 call burst against a scripted provider that accepts 10 requests in flight and rejects the rest with 429. Both runs used maxRetries: 2.

Retries only rateLimit: { maxConcurrent: 10 }
Calls answered 20 50
Calls failed 30 0
Provider 429s 110 0
Requests sent 130 50

Retries alone sent 130 requests, collected 110 rejections, and still lost 30 calls. With a local limit, every call waited its turn and went out once.

npm i vern-llm openai
python
import OpenAI from 'openai';
import { VernLLM } from 'vern-llm';
import { fromOpenAI } from 'vern-llm/adapters';

const llm = new VernLLM({
  client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY })),
  model: 'gpt-6.2-sol',
  rateLimit: {
    requestsPerMinute: 500,
    tokensPerMinute: 200_000,
    maxConcurrent: 20,
  },
});

const results = await Promise.all(tickets.map((t) => llm.call({ userContent: t.text })));

rateLimit has three buckets, each optional:

A call goes out only when every configured bucket has room. Otherwise it waits.

Calls that cannot go out yet wait in strict first in, first out order. A large call is never starved by smaller ones behind it.

You decide how long a call may wait and how deep the queue may get:

rateLimit: {
  requestsPerMinute: 60,
  maxQueueMs: 10_000, // default 30000
  maxQueueSize: 50,   // default unbounded
},

When a call gives up waiting, it fails with a rate_limited error and a code you can branch on (rate_limit_queue_timeout or rate_limit_queue_full). These never reached the provider, so VernLLM does not retry them. A call too big to ever fit under tokensPerMinute fails right away with rate_limit_capacity_exceeded instead of blocking the queue forever.

Capacity is taken per attempt, not per call. A retry is a real request against the same provider limit, so it waits in the same queue as everything else. That is what keeps retries from turning one 429 into a storm of them.

You often do not know the provider's real limit, or it changes with your tier and their load. aimd lets the requests ceiling adjust itself: it grows a little on every success and shrinks on a real 429.

rateLimit: {
  requestsPerMinute: 500,
  aimd: { increaseBy: 10, decreaseFactor: 0.5, minCapacity: 50, maxCapacity: 1000 },
},

It shrinks at most once per 60 seconds, so a burst of 429s from calls already in flight counts as one overload. With proactiveFloor, adapters that can read rate limit headers shrink the ceiling when the provider reports low remaining capacity, before a 429 happens.

A 429 that still gets through keeps its own code, provider_rate_limited, so you can tell a real provider limit apart from VernLLM's own queue in your logs.

With fallback, each target gets its own limiter. A burst against a struggling primary cannot use up the slots a healthy backup needs:

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-6.2-sol',
  rateLimit: { maxConcurrent: 50 },
  fallback: {
    client: fromGemini(gemini),
    model: 'gemini-4-neon',
    rateLimit: { maxConcurrent: 10 },
  },
});

The built in limiter lives in one process. If you run several instances, pass your own RateLimiterAdapter, backed by Redis or anything else, so every instance shares one budget against the provider.

Retrying 429s is damage control. A local limit keeps you under the provider's ceiling in the first place: queue in order, count every retry, adapt when the provider pushes back.

npm i vern-llm

Docs: Rate limiting

── more in #large-language-models 4 stories · sorted by recency
openalternative.co · · #large-language-models
MemexAI
── more on @vernllm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/ai-rate-limiting-for…] indexed:0 read:3min 2026-11-20 · —