{"slug": "unlimitednim-proxy-that-makes-nvidia-nim-s-40-rpm-feel-unlimited", "title": "UnlimitedNIM – Proxy that makes Nvidia Nim's 40 RPM feel unlimited", "summary": "UnlimitedNIM, a streaming reverse proxy for NVIDIA's NIM free tier, eliminates 429 rate-limit errors for coding agents by enforcing a 40-request-per-minute sliding window and priority queue, with tests showing 260+ requests completed with zero leaked 429s. The proxy, compatible with Cline, Roo Code, and OpenCode, queues requests by priority (user messages, tool calls, debug dumps) and streams responses full-duplex, promoting stuck low-priority requests after a TTL to prevent starvation.", "body_md": "Make NVIDIA's 40 RPM free tier feel unlimited. Smart priority queue proxy for coding agents.\n\n**Works with Cline · Roo Code · OpenCode · Any OpenAI-compatible agent**\n\nNVIDIA's free NIM tier limits you to **40 requests per minute**. Coding agents like\n**Cline, Roo Code, and OpenCode** generate bursts of traffic — an agent loop firing\ntool calls, context reads, and model invocations in parallel will slam straight into\nthat wall. The result: a flood of `429 Rate Limit Exceeded`\n\nerrors, retried work,\nfrustrated agents, and stalls in the middle of a task.\n\nYou don't get more throughput from NVIDIA — but you *can* stop wasting the window\nyou have.\n\nUnlimitedNIM is a thin, streaming reverse proxy that sits between your coding agent\nand NVIDIA's API. It enforces NVIDIA's 40 RPM limit *for you* — in a way that's\ninvisible to your agent:\n\n**Sliding-window rate limiter**— never fires more than 40 requests per minute, so NVIDIA never sees a burst and never returns 429.** Priority queue**— when the window is full, requests wait in line instead of failing:** priority 1**— user messages (you typed it, you're waiting)** priority 3**— agent loop tool calls (the bulk of traffic)** priority 5**— debug dumps and huge stack traces (fire when there's room)\n\n**Full-duplex streaming**— SSE responses stream straight through, chunk by chunk.** Starvation protection**— a low-priority request stuck longer than the TTL is promoted so it eventually fires.** Backpressure**— a bounded queue rejects with`503 + Retry-After`\n\ninstead of leaking memory or flooding NVIDIA.\n\n260+ real NVIDIA requests driven through the proxy — zero leaked 429s, zero rejections.\n\nWave 1 (40 concurrent) fired instantly. Wave 2 (60 more, fired while wave 1 was still in flight) queued and streamed in across the next window — every single request completed, and not one rate-limit error reached a client.\n\n```\nCoding agent ──POST /v1/chat/completions──▶ UnlimitedNIM ──▶ NVIDIA API (40 RPM cap)\n                      ▲                          │\n                      └──── 200 + SSE stream ◄────┘\n                        (or waits in priority queue)\n```\n\n- A request arrives with an\n`X-Priority`\n\nheader (1/3/5). - If the current 60s window has room, it fires immediately.\n- Otherwise it enters the min-heap priority queue — highest priority (lowest number) leaves first; same priority is FIFO.\n- As the window rolls over (or requests finish), the queue drains up to 40/min, never exceeding the limit.\n- Streaming responses relay byte-for-byte; client disconnects abort the upstream call and free the slot (the window slot is never refunded — NVIDIA already counted it, so we never re-fire).\n\n```\n# 1. Set your NVIDIA key\n#    (either export it or copy .env.example → .env and fill it in)\nexport NVIDIA_API_KEY=your-key-here\n\n# 2. Install\npip install -r requirements.txt\n\n# 3. Run\npython main.py\n# INFO: Uvicorn running on http://0.0.0.0:8000\n\n# 4. Point your agent at http://localhost:8000/v1\n```\n\nVerify it's alive:\n\n```\ncurl http://localhost:8000/status\n```\n\nAll options come from environment variables or a `.env`\n\nfile (auto-loaded).\n\n| Variable | Default | Description |\n|---|---|---|\n`NVIDIA_API_KEY` |\n(required) |\nYour NVIDIA API key (used upstream) |\n`NVIDIA_BASE_URL` |\n`https://integrate.api.nvidia.com/v1` |\nUpstream NVIDIA base URL |\n`NVIDIA_DEFAULT_MODEL` |\n`meta/llama-3.3-70b-instruct` |\nModel used when a client sends an unknown model |\n`NVIDIA_MODELS` |\n(built-in list) |\nComma-separated valid models passed through untouched |\n`REWRITE_UNKNOWN_MODELS` |\n`true` |\n`true` = rewrite unknown client models to default; `false` = pass through |\n`MAX_RPM` |\n`40` |\nMax requests per minute (matches NVIDIA free tier) |\n`WINDOW_SIZE` |\n`60` |\nRate-limit window in seconds |\n`MAX_QUEUE` |\n`500` |\nMax queue depth before rejecting with `503 + Retry-After` |\n`RETRY_TTL` |\n`300` |\nSeconds before a queued low-priority request is promoted to priority 1 |\n`HTTP_RETRIES` |\n`3` |\nUpstream `429` retries with exponential backoff |\n`BACKOFF_BASE` |\n`1.0` |\nBackoff base seconds (sleeps 1s, 2s, 4s) |\n`CONNECT_TIMEOUT` |\n`30` |\nhttpx connect timeout (s) |\n`READ_TIMEOUT` |\n`60` |\nhttpx read timeout (s) |\n`WRITE_TIMEOUT` |\n`30` |\nhttpx write timeout (s) |\n`POOL_TIMEOUT` |\n`30` |\nhttpx pool timeout (s) |\n`MAX_STREAM_AGE` |\n`300` |\nMax age of an in-flight stream before the sweeper force-releases it |\n`SWEEP_INTERVAL` |\n`5` |\nSweeper interval (s) |\n`LARGE_BODY_BYTES` |\n`262144` |\nPayload bytes above which a request defaults to priority 5 |\n`HOST` |\n`0.0.0.0` |\nBind address |\n`PORT` |\n`8000` |\nBind port |\n\nAdd to `opencode.json`\n\n(or `~/.config/opencode/opencode.json`\n\n):\n\n```\n{\n  \"$schema\": \"https://opencode.ai/config.json\",\n  \"provider\": {\n    \"nvidia\": {\n      \"npm\": \"@ai-sdk/openai-compatible\",\n      \"name\": \"UnlimitedNIM\",\n      \"options\": {\n        \"baseURL\": \"http://localhost:8000/v1\",\n        \"apiKey\": \"anything-works\"\n      },\n      \"models\": {\n        \"nvidia/nemotron-3.5-lightning-30b-a3b\": {\n          \"name\": \"Nemotron 3.5 Lightning 30B\"\n        }\n      }\n    }\n  }\n}\n```\n\n- Settings → API Provider →\n**OpenAI Compatible** - Base URL:\n`http://localhost:8000/v1`\n\n- API Key: anything (the proxy uses its own NVIDIA key)\n- Model:\n`nvidia/nemotron-3.5-lightning-30b-a3b`\n\n- Settings → API Configuration → Provider →\n**OpenAI Compatible** - Base URL:\n`http://localhost:8000/v1`\n\n- API Key: anything\n- Model:\n`nvidia/nemotron-3.5-lightning-30b-a3b`\n\n`load_test.py`\n\nsimulates heavy agentic traffic (100 concurrent requests, mixed\npriorities, two waves) and reports status counts, leaked 429s, and queue wait times:\n\n```\npython load_test.py                       # full run (40 + 60 waves)\npython load_test.py --total 12 --wave1 5 --wave2 7   # quick smoke\n```\n\nFor safe local runs without touching NVIDIA, start the included mock upstream and point the proxy at it:\n\n```\npython mock_upstream.py                    # terminal 1\nNVIDIA_BASE_URL=http://localhost:9001/v1 python main.py   # terminal 2\napp/\n  config.py        # env-driven settings\n  rate_limiter.py  # sliding-window limiter + priority queue + in-flight tracking\n  proxy.py         # FastAPI app, streaming upstream, 429 retry, validation\nmain.py            # entry point\nload_test.py       # agentic traffic simulator\nmock_upstream.py   # fake NVIDIA upstream for local testing\ntests/             # 43 tests covering all 34 test-case specs\n```\n\nThe suite implements the full 34-case spec — window resets, previous-minute\nin-flight tracking, priority ordering, starvation promotion, clock skew, upstream\n429/500 handling, client disconnects, and the compliance layer (`/v1/models`\n\n,\n`/status`\n\n).\n\n```\npython -m pytest -q   # 43 passed\n```\n\n", "url": "https://wpnews.pro/news/unlimitednim-proxy-that-makes-nvidia-nim-s-40-rpm-feel-unlimited", "canonical_source": "https://github.com/shivnathtathe/UnlimitedNIM", "published_at": "2026-08-20 09:03:31+00:00", "updated_at": "2026-08-20 09:44:35.163107+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "ai-tools"], "entities": ["UnlimitedNIM", "NVIDIA", "Cline", "Roo Code", "OpenCode"], "alternates": {"html": "https://wpnews.pro/news/unlimitednim-proxy-that-makes-nvidia-nim-s-40-rpm-feel-unlimited", "markdown": "https://wpnews.pro/news/unlimitednim-proxy-that-makes-nvidia-nim-s-40-rpm-feel-unlimited.md", "text": "https://wpnews.pro/news/unlimitednim-proxy-that-makes-nvidia-nim-s-40-rpm-feel-unlimited.txt", "jsonld": "https://wpnews.pro/news/unlimitednim-proxy-that-makes-nvidia-nim-s-40-rpm-feel-unlimited.jsonld"}}