Published: 2026-09-25 | ~2,200 words | 11 min read
I needed AI agents that could actually do things — run code, call APIs, browse the web, write files — not just chat. Cloud APIs got expensive fast, rate-limited unpredictably, and locked me into someone else's infrastructure.
So I built a local stack on my Windows machine (i7-8700K, 32GB RAM, RTX 3080). Total monthly cost: $0. Everything runs locally or on free tiers.
This article documents what actually works, what broke repeatedly, and the solutions that stuck.
┌─────────────────────────────────────────────────────────────────┐
│ ORCHESTRATION │
│ Windmill (localhost:8000) — 3 containers │
│ - windmill-postgres, windmill-server, windmill-worker │
│ - Workspace: admins, NO_AUTH=true (dev only) │
│ - 13 scripts deployed: 7 pillars + 4 automations + 1 reviewer │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ MODEL LAYER │
│ Primary: nvidia/nemotron-3-super-120b-a12b │
│ Fallback 1: openai/gpt-oss-120b (Groq) │
│ Fallback 2: openai/gpt-oss-20b (Groq) │
│ Reviewer: nvidia/nemotron-3-ultra-550b-a55b (adversarial) │
│ │
│ Local Proxy: 127.0.0.1:8001 (OpenAI-compatible) │
│ - 10 NVIDIA API keys in rotation │
│ - 6-model cascade on failure │
│ - Tool calling + SSE streaming support │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ AGENT INTERFACE │
│ Hermes Agent (Nous Research) — Telegram + local CLI │
│ - Configured with custom provider → local proxy │
│ - Fallback providers: Groq (gpt-oss-120b → gpt-oss-20b) │
│ - Auxiliary model DISABLED (caused rate loops) │
│ - Bot Mode: 3 bots configured on Ultra 550B (see below) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ TOOLS & AUTOMATION │
│ Webwright (Playwright) — browser automation │
│ NVIDIA Reviewer — adversarial code review via Windmill │
│ Edge-TTS — local voice synthesis (Microsoft, free) │
│ FFmpeg — video assembly │
│ Retry watcher — cron every 2 min, Telegram alerts │
└─────────────────────────────────────────────────────────────────┘
NVIDIA NIM free tier: 40 RPM. That's it. One agent loop can burn 10 requests in seconds. Groq free tier is generous but has different models. OpenRouter credits exhausted.
nvidia_key_rotation.py)
class KeyRotator:
def __init__(self, keys: list[str]):
self.keys = keys
self.state = {k: {"fails": 0, "cooldown_until": 0} for k in keys}
self.current_idx = 0
def get_key(self) -> str:
now = time.time()
for _ in range(len(self.keys)):
key = self.keys[self.current_idx]
if self.state[key]["cooldown_until"] <= now:
return key
self.current_idx = (self.current_idx + 1) % len(self.keys)
raise Exception("All keys in cooldown")
def report_result(self, key: str, status: int):
if status == 429:
self.state[key]["cooldown_until"] = time.time() + 30
self.state[key]["fails"] += 1
elif status in (401, 403, 410):
self.state[key]["fails"] = 999 # permanent skip
else:
self.state[key]["fails"] = max(0, self.state[key]["fails"] - 1)
Result: 429 errors become 30-second cooldowns instead of hard stops. Keys that return 401/403/410 (EOL models) are permanently skipped.
nvidia_cascade.py)
When a key works but the model fails (timeout, 500, bad output), cascade to the next model:
ACTIVE_MODELS = [
"nvidia/nemotron-3.5-lightning-30b-a3b", # fastest, good for simple
"openai/gpt-oss-20b", # Groq fallback via proxy
"nvidia/nemotron-3-super-120b-a12b", # primary workhorse
"openai/gpt-oss-120b", # Groq fallback via proxy
"glm-5.2", # backup
"nvidia/nemotron-3-ultra-550b-a55b", # heavy reasoning, reviewer
]
Each model tried in order. First success wins. Logs show which model actually responded.
nvidia_proxy.py)
This was the hardest part. Hermes sends tool calls; the proxy must forward them to NVIDIA and return tool_calls in the response. Also must stream SSE chunks correctly.
Key fixes that took days:
tools parameter in request body to NVIDIAtool_calls present, return as-is (don't wrap in text)data: {json}\n\n, end with data: [DONE]\n\n
finish_reason: tool_calls correctly
Without these, Hermes either got empty streams or couldn't execute tools.
model: "nvidia/nemotron-3-super-120b-a12b"
provider: "custom"
custom:
base_url: "http://127.0.0.1:8001/v1"
fallback_providers:
- name: "groq"
models:
- "openai/gpt-oss-120b"
- "openai/gpt-oss-20b"
Behavior: if primary (proxy) fails, Hermes automatically retries on Groq. Next turn, it tries primary again.
| Pillar | Purpose | Status |
|---|---|---|
| 1 | Approval Gate (human-in-the-loop) | ✅ Working, tested end-to-end |
| 2 | Alternative Supervision | ❌ Blocked — needs API keys |
| 3 | Self-Healing | 📦 Archived — no Docker socket in Windmill CE |
| 4 | Resilience/Queue | ✅ Restored via API (job 01a0d246) |
| 5 | Watchdog/Resume | 📦 Archived — same as Pillar 3 |
| 6 | Cleanup/Mutex | ⚠️ Partial — cleanup paths wrong, mutex OK |
| 7 | Sandbox | ⚠️ Partial — shell escape possible |
| 8, 9, 10 | — | ❌ Never deployed |
Reality: Only Pillars 1 and 4 are production-ready. The rest need Docker socket access (Windmill EE feature) or significant rework.
retry_watcher.py — cron every 2 min, checks failed jobs, retries, notifies Telegramnvidia_reviewer.py — adversarial code review (called via webhook or manually)comando-gabriele — receives signed Telegram commands, executes terminal/file ops
model: "nvidia/nemotron-3-super-120b-a12b"
provider: "custom"
custom:
base_url: "http://127.0.0.1:8001/v1"
api_key: "not-needed" # proxy handles auth
fallback_providers:
- name: "groq"
models:
- "openai/gpt-oss-120b"
- "openai/gpt-oss-20b"
auxiliary:
title_generation:
enabled: false # critical: was causing rate loops
3 bots, all on nvidia/nemotron-3-ultra-550b-a55b (strongest model via proxy):
@direttore — strategic coordinator, custom SOUL.md@analista — market/technical analysis, custom SOUL.md
@critico — adversarial review, custom SOUL.md
Group chat "Progetto ReportForge" completed 3 rounds, converged on positioning. Cron on @direttore runs every 24h.
Limitation: Bots share the same proxy/key pool. Heavy concurrent use hits rate limits. Serial execution (max 3 rounds) mitigates this.
Before: 429 errors killed agent loops. Manual key switching.
After: Automatic. 10 keys, 30s cooldown on 429, cascade across 6 models, Groq fallback. Uptime >99% on free tiers.
Problem: Nemotron-Super-120b hallucinates function signatures, file paths, config options when context grows.
Fix:
/new frequentlynvidia_reviewer (Ultra 550B, temp 0.1)
Problem: Proxy returned text instead of tool_calls. Hermes waited forever.
Fix: Proxy now passes tools param to NVIDIA, returns raw tool_calls array. Verified with Hermes tool execution.
Problem: "Empty stream" errors. NVIDIA sends SSE; proxy wasn't forwarding chunks correctly.
Fix: Proper chunk parsing, data: prefix, [DONE] terminator. Hermes now receives stream correctly.
Burned: meta/llama-3.1-*, meta/llama-3.3-* (Aug 26, 2026), openai/gpt-oss-120b on NVIDIA (Sep 3, 2026).
Process: Before using any model, curl test. 410 = remove from cascade. Current active list maintained in nvidia_cascade.py.
Problem: PowerShell Out-File -Encoding UTF8 adds BOM (EF BB BF). Python json.loads fails silently.
Fix: Always write with [System.IO.File]::WriteAllText($path, $content, $utf8NoBom) where $utf8NoBom = New-Object System.Text.UTF8Encoding $false.
Mistake: Direct psql inserts. Windmill cache didn't refresh. Scripts invisible in UI.
Rule: Only official Windmill API (CLI or HTTP). If API fails, stop and ask.
| Metric | Value | Verified |
|---|---|---|
| NVIDIA API keys | 10 | API_key.txt |
| Active models in cascade | 6 | nvidia_cascade.py |
| Fallback models (Groq) | 2 | Hermes config |
| Windmill containers | 3 healthy | docker ps |
| Deployed scripts | 13 | Windmill UI |
| Pillar 1 test | PASS | Job 01a0d0cf |
| Pillar 4 restored | PASS | Job 01a0d246 |
| NVIDIA Reviewer test | PASS | Found bug in sum(a,b): return a-b |
| Webwright + Reviewer integration | PASS | 10 issues found in git-filter-branch script |
| Monthly cost | $0 | No paid services used |
| Disk C: free | 36% | df -h |
| Disk D: free | 35% | df -h |
| Disk E: free | 66% | df -h |
| Decision | Would Change To |
|---|---|
| Started with 10-pillar design | Start with 2-3 pillars that solve immediate pain |
| Built custom proxy from scratch | Use existing OpenAI-compatible proxy if one supported tool calling + SSE |
| Put all models on same key pool | Separate key pools per model tier |
| Used Windmill CE for self-healing | Accept CE limits; build external watchers instead |
| Tried to automate everything | Keep human-in-the-loop (Pillar 1) for critical gates |
docker-compose up for Windmillbuild.nvidia.com → API_key.txt
pip install webwright → skill for Hermespip install edge-tts (Microsoft, free)
This stack exists because I refused to pay $500+/month for agent infrastructure that randomly rate-limits or changes APIs. It's not magic — it's plumbing. Lots of plumbing. But it works, it's mine, and it costs nothing.
If you build something similar: test every model before trusting it, log every fallback, and keep sessions short. The free tiers are generous but unforgiving.
Built on Windows 11, Docker Desktop, WSL2. Zero cloud dependencies. Zero subscriptions.