UnlimitedNIM – Proxy that makes Nvidia Nim's 40 RPM feel unlimited

UnlimitedNIM, a streaming reverse proxy for NVIDIA's NIM free tier, eliminates 429 rate-limit errors for coding agents by enforcing a 40-request-per-minute sliding window and priority queue, with tests showing 260+ requests completed with zero leaked 429s. The proxy, compatible with Cline, Roo Code, and OpenCode, queues requests by priority (user messages, tool calls, debug dumps) and streams responses full-duplex, promoting stuck low-priority requests after a TTL to prevent starvation.

Make NVIDIA's 40 RPM free tier feel unlimited. Smart priority queue proxy for coding agents. Works with Cline · Roo Code · OpenCode · Any OpenAI-compatible agent NVIDIA's free NIM tier limits you to 40 requests per minute . Coding agents like Cline, Roo Code, and OpenCode generate bursts of traffic — an agent loop firing tool calls, context reads, and model invocations in parallel will slam straight into that wall. The result: a flood of 429 Rate Limit Exceeded errors, retried work, frustrated agents, and stalls in the middle of a task. You don't get more throughput from NVIDIA — but you can stop wasting the window you have. UnlimitedNIM is a thin, streaming reverse proxy that sits between your coding agent and NVIDIA's API. It enforces NVIDIA's 40 RPM limit for you — in a way that's invisible to your agent: Sliding-window rate limiter — never fires more than 40 requests per minute, so NVIDIA never sees a burst and never returns 429. Priority queue — when the window is full, requests wait in line instead of failing: priority 1 — user messages you typed it, you're waiting priority 3 — agent loop tool calls the bulk of traffic priority 5 — debug dumps and huge stack traces fire when there's room Full-duplex streaming — SSE responses stream straight through, chunk by chunk. Starvation protection — a low-priority request stuck longer than the TTL is promoted so it eventually fires. Backpressure — a bounded queue rejects with 503 + Retry-After instead of leaking memory or flooding NVIDIA. 260+ real NVIDIA requests driven through the proxy — zero leaked 429s, zero rejections. Wave 1 40 concurrent fired instantly. Wave 2 60 more, fired while wave 1 was still in flight queued and streamed in across the next window — every single request completed, and not one rate-limit error reached a client. Coding agent ──POST /v1/chat/completions──▶ UnlimitedNIM ──▶ NVIDIA API 40 RPM cap ▲ │ └──── 200 + SSE stream ◄────┘ or waits in priority queue - A request arrives with an X-Priority header 1/3/5 . - If the current 60s window has room, it fires immediately. - Otherwise it enters the min-heap priority queue — highest priority lowest number leaves first; same priority is FIFO. - As the window rolls over or requests finish , the queue drains up to 40/min, never exceeding the limit. - Streaming responses relay byte-for-byte; client disconnects abort the upstream call and free the slot the window slot is never refunded — NVIDIA already counted it, so we never re-fire . 1. Set your NVIDIA key either export it or copy .env.example → .env and fill it in export NVIDIA API KEY=your-key-here 2. Install pip install -r requirements.txt 3. Run python main.py INFO: Uvicorn running on http://0.0.0.0:8000 4. Point your agent at http://localhost:8000/v1 Verify it's alive: curl http://localhost:8000/status All options come from environment variables or a .env file auto-loaded . | Variable | Default | Description | |---|---|---| NVIDIA API KEY | required | Your NVIDIA API key used upstream | NVIDIA BASE URL | https://integrate.api.nvidia.com/v1 | Upstream NVIDIA base URL | NVIDIA DEFAULT MODEL | meta/llama-3.3-70b-instruct | Model used when a client sends an unknown model | NVIDIA MODELS | built-in list | Comma-separated valid models passed through untouched | REWRITE UNKNOWN MODELS | true | true = rewrite unknown client models to default; false = pass through | MAX RPM | 40 | Max requests per minute matches NVIDIA free tier | WINDOW SIZE | 60 | Rate-limit window in seconds | MAX QUEUE | 500 | Max queue depth before rejecting with 503 + Retry-After | RETRY TTL | 300 | Seconds before a queued low-priority request is promoted to priority 1 | HTTP RETRIES | 3 | Upstream 429 retries with exponential backoff | BACKOFF BASE | 1.0 | Backoff base seconds sleeps 1s, 2s, 4s | CONNECT TIMEOUT | 30 | httpx connect timeout s | READ TIMEOUT | 60 | httpx read timeout s | WRITE TIMEOUT | 30 | httpx write timeout s | POOL TIMEOUT | 30 | httpx pool timeout s | MAX STREAM AGE | 300 | Max age of an in-flight stream before the sweeper force-releases it | SWEEP INTERVAL | 5 | Sweeper interval s | LARGE BODY BYTES | 262144 | Payload bytes above which a request defaults to priority 5 | HOST | 0.0.0.0 | Bind address | PORT | 8000 | Bind port | Add to opencode.json or ~/.config/opencode/opencode.json : { "$schema": "https://opencode.ai/config.json", "provider": { "nvidia": { "npm": "@ai-sdk/openai-compatible", "name": "UnlimitedNIM", "options": { "baseURL": "http://localhost:8000/v1", "apiKey": "anything-works" }, "models": { "nvidia/nemotron-3.5-lightning-30b-a3b": { "name": "Nemotron 3.5 Lightning 30B" } } } } } - Settings → API Provider → OpenAI Compatible - Base URL: http://localhost:8000/v1 - API Key: anything the proxy uses its own NVIDIA key - Model: nvidia/nemotron-3.5-lightning-30b-a3b - Settings → API Configuration → Provider → OpenAI Compatible - Base URL: http://localhost:8000/v1 - API Key: anything - Model: nvidia/nemotron-3.5-lightning-30b-a3b load test.py simulates heavy agentic traffic 100 concurrent requests, mixed priorities, two waves and reports status counts, leaked 429s, and queue wait times: python load test.py full run 40 + 60 waves python load test.py --total 12 --wave1 5 --wave2 7 quick smoke For safe local runs without touching NVIDIA, start the included mock upstream and point the proxy at it: python mock upstream.py terminal 1 NVIDIA BASE URL=http://localhost:9001/v1 python main.py terminal 2 app/ config.py env-driven settings rate limiter.py sliding-window limiter + priority queue + in-flight tracking proxy.py FastAPI app, streaming upstream, 429 retry, validation main.py entry point load test.py agentic traffic simulator mock upstream.py fake NVIDIA upstream for local testing tests/ 43 tests covering all 34 test-case specs The suite implements the full 34-case spec — window resets, previous-minute in-flight tracking, priority ordering, starvation promotion, clock skew, upstream 429/500 handling, client disconnects, and the compliance layer /v1/models , /status . python -m pytest -q 43 passed