# UnlimitedNIM – Proxy that makes Nvidia Nim's 40 RPM feel unlimited

> Source: <https://github.com/shivnathtathe/UnlimitedNIM>
> Published: 2026-08-20 09:03:31+00:00

Make NVIDIA's 40 RPM free tier feel unlimited. Smart priority queue proxy for coding agents.

**Works with Cline · Roo Code · OpenCode · Any OpenAI-compatible agent**

NVIDIA's free NIM tier limits you to **40 requests per minute**. Coding agents like
**Cline, Roo Code, and OpenCode** generate bursts of traffic — an agent loop firing
tool calls, context reads, and model invocations in parallel will slam straight into
that wall. The result: a flood of `429 Rate Limit Exceeded`

errors, retried work,
frustrated agents, and stalls in the middle of a task.

You don't get more throughput from NVIDIA — but you *can* stop wasting the window
you have.

UnlimitedNIM is a thin, streaming reverse proxy that sits between your coding agent
and NVIDIA's API. It enforces NVIDIA's 40 RPM limit *for you* — in a way that's
invisible to your agent:

**Sliding-window rate limiter**— never fires more than 40 requests per minute, so NVIDIA never sees a burst and never returns 429.** Priority queue**— when the window is full, requests wait in line instead of failing:** priority 1**— user messages (you typed it, you're waiting)** priority 3**— agent loop tool calls (the bulk of traffic)** priority 5**— debug dumps and huge stack traces (fire when there's room)

**Full-duplex streaming**— SSE responses stream straight through, chunk by chunk.** Starvation protection**— a low-priority request stuck longer than the TTL is promoted so it eventually fires.** Backpressure**— a bounded queue rejects with`503 + Retry-After`

instead of leaking memory or flooding NVIDIA.

260+ real NVIDIA requests driven through the proxy — zero leaked 429s, zero rejections.

Wave 1 (40 concurrent) fired instantly. Wave 2 (60 more, fired while wave 1 was still in flight) queued and streamed in across the next window — every single request completed, and not one rate-limit error reached a client.

```
Coding agent ──POST /v1/chat/completions──▶ UnlimitedNIM ──▶ NVIDIA API (40 RPM cap)
                      ▲                          │
                      └──── 200 + SSE stream ◄────┘
                        (or waits in priority queue)
```

- A request arrives with an
`X-Priority`

header (1/3/5). - If the current 60s window has room, it fires immediately.
- Otherwise it enters the min-heap priority queue — highest priority (lowest number) leaves first; same priority is FIFO.
- As the window rolls over (or requests finish), the queue drains up to 40/min, never exceeding the limit.
- Streaming responses relay byte-for-byte; client disconnects abort the upstream call and free the slot (the window slot is never refunded — NVIDIA already counted it, so we never re-fire).

```
# 1. Set your NVIDIA key
#    (either export it or copy .env.example → .env and fill it in)
export NVIDIA_API_KEY=your-key-here

# 2. Install
pip install -r requirements.txt

# 3. Run
python main.py
# INFO: Uvicorn running on http://0.0.0.0:8000

# 4. Point your agent at http://localhost:8000/v1
```

Verify it's alive:

```
curl http://localhost:8000/status
```

All options come from environment variables or a `.env`

file (auto-loaded).

| Variable | Default | Description |
|---|---|---|
`NVIDIA_API_KEY` |
(required) |
Your NVIDIA API key (used upstream) |
`NVIDIA_BASE_URL` |
`https://integrate.api.nvidia.com/v1` |
Upstream NVIDIA base URL |
`NVIDIA_DEFAULT_MODEL` |
`meta/llama-3.3-70b-instruct` |
Model used when a client sends an unknown model |
`NVIDIA_MODELS` |
(built-in list) |
Comma-separated valid models passed through untouched |
`REWRITE_UNKNOWN_MODELS` |
`true` |
`true` = rewrite unknown client models to default; `false` = pass through |
`MAX_RPM` |
`40` |
Max requests per minute (matches NVIDIA free tier) |
`WINDOW_SIZE` |
`60` |
Rate-limit window in seconds |
`MAX_QUEUE` |
`500` |
Max queue depth before rejecting with `503 + Retry-After` |
`RETRY_TTL` |
`300` |
Seconds before a queued low-priority request is promoted to priority 1 |
`HTTP_RETRIES` |
`3` |
Upstream `429` retries with exponential backoff |
`BACKOFF_BASE` |
`1.0` |
Backoff base seconds (sleeps 1s, 2s, 4s) |
`CONNECT_TIMEOUT` |
`30` |
httpx connect timeout (s) |
`READ_TIMEOUT` |
`60` |
httpx read timeout (s) |
`WRITE_TIMEOUT` |
`30` |
httpx write timeout (s) |
`POOL_TIMEOUT` |
`30` |
httpx pool timeout (s) |
`MAX_STREAM_AGE` |
`300` |
Max age of an in-flight stream before the sweeper force-releases it |
`SWEEP_INTERVAL` |
`5` |
Sweeper interval (s) |
`LARGE_BODY_BYTES` |
`262144` |
Payload bytes above which a request defaults to priority 5 |
`HOST` |
`0.0.0.0` |
Bind address |
`PORT` |
`8000` |
Bind port |

Add to `opencode.json`

(or `~/.config/opencode/opencode.json`

):

```
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "nvidia": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "UnlimitedNIM",
      "options": {
        "baseURL": "http://localhost:8000/v1",
        "apiKey": "anything-works"
      },
      "models": {
        "nvidia/nemotron-3.5-lightning-30b-a3b": {
          "name": "Nemotron 3.5 Lightning 30B"
        }
      }
    }
  }
}
```

- Settings → API Provider →
**OpenAI Compatible** - Base URL:
`http://localhost:8000/v1`

- API Key: anything (the proxy uses its own NVIDIA key)
- Model:
`nvidia/nemotron-3.5-lightning-30b-a3b`

- Settings → API Configuration → Provider →
**OpenAI Compatible** - Base URL:
`http://localhost:8000/v1`

- API Key: anything
- Model:
`nvidia/nemotron-3.5-lightning-30b-a3b`

`load_test.py`

simulates heavy agentic traffic (100 concurrent requests, mixed
priorities, two waves) and reports status counts, leaked 429s, and queue wait times:

```
python load_test.py                       # full run (40 + 60 waves)
python load_test.py --total 12 --wave1 5 --wave2 7   # quick smoke
```

For safe local runs without touching NVIDIA, start the included mock upstream and point the proxy at it:

```
python mock_upstream.py                    # terminal 1
NVIDIA_BASE_URL=http://localhost:9001/v1 python main.py   # terminal 2
app/
  config.py        # env-driven settings
  rate_limiter.py  # sliding-window limiter + priority queue + in-flight tracking
  proxy.py         # FastAPI app, streaming upstream, 429 retry, validation
main.py            # entry point
load_test.py       # agentic traffic simulator
mock_upstream.py   # fake NVIDIA upstream for local testing
tests/             # 43 tests covering all 34 test-case specs
```

The suite implements the full 34-case spec — window resets, previous-minute
in-flight tracking, priority ordering, starvation promotion, clock skew, upstream
429/500 handling, client disconnects, and the compliance layer (`/v1/models`

,
`/status`

).

```
python -m pytest -q   # 43 passed
```