How I Built a Local AI Agent Stack for $0/month (Hermes + Windmill + NVIDIA + Groq)

A developer built a $0/month local AI agent stack on a Windows machine (i7-8700K, 32GB RAM, RTX 3080) that runs code, calls APIs, browses the web and writes files, combining Windmill for orchestration, Hermes Agent from Nous Research as the interface, and a local OpenAI-compatible proxy that rotates 10 NVIDIA NIM API keys across a six-model cascade with Groq gpt-oss fallbacks. The writeup documents the failure modes encountered, including NVIDIA's 40 RPM free-tier limit and rate loops caused by Hermes' auxiliary model, which the developer disabled.

Published: 2026-09-25 | ~2,200 words | 11 min read I needed AI agents that could actually do things — run code, call APIs, browse the web, write files — not just chat. Cloud APIs got expensive fast, rate-limited unpredictably, and locked me into someone else's infrastructure. So I built a local stack on my Windows machine i7-8700K, 32GB RAM, RTX 3080 . Total monthly cost: $0 . Everything runs locally or on free tiers. This article documents what actually works, what broke repeatedly, and the solutions that stuck. ┌─────────────────────────────────────────────────────────────────┐ │ ORCHESTRATION │ │ Windmill localhost:8000 — 3 containers │ │ - windmill-postgres, windmill-server, windmill-worker │ │ - Workspace: admins, NO AUTH=true dev only │ │ - 13 scripts deployed: 7 pillars + 4 automations + 1 reviewer │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ MODEL LAYER │ │ Primary: nvidia/nemotron-3-super-120b-a12b │ │ Fallback 1: openai/gpt-oss-120b Groq │ │ Fallback 2: openai/gpt-oss-20b Groq │ │ Reviewer: nvidia/nemotron-3-ultra-550b-a55b adversarial │ │ │ │ Local Proxy: 127.0.0.1:8001 OpenAI-compatible │ │ - 10 NVIDIA API keys in rotation │ │ - 6-model cascade on failure │ │ - Tool calling + SSE streaming support │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ AGENT INTERFACE │ │ Hermes Agent Nous Research — Telegram + local CLI │ │ - Configured with custom provider → local proxy │ │ - Fallback providers: Groq gpt-oss-120b → gpt-oss-20b │ │ - Auxiliary model DISABLED caused rate loops │ │ - Bot Mode: 3 bots configured on Ultra 550B see below │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ TOOLS & AUTOMATION │ │ Webwright Playwright — browser automation │ │ NVIDIA Reviewer — adversarial code review via Windmill │ │ Edge-TTS — local voice synthesis Microsoft, free │ │ FFmpeg — video assembly │ │ Retry watcher — cron every 2 min, Telegram alerts │ └─────────────────────────────────────────────────────────────────┘ NVIDIA NIM free tier: 40 RPM. That's it. One agent loop can burn 10 requests in seconds. Groq free tier is generous but has different models. OpenRouter credits exhausted. nvidia key rotation.py python Simplified logic class KeyRotator: def init self, keys: list str : self.keys = keys self.state = {k: {"fails": 0, "cooldown until": 0} for k in keys} self.current idx = 0 def get key self - str: now = time.time for in range len self.keys : key = self.keys self.current idx if self.state key "cooldown until" <= now: return key self.current idx = self.current idx + 1 % len self.keys raise Exception "All keys in cooldown" def report result self, key: str, status: int : if status == 429: self.state key "cooldown until" = time.time + 30 self.state key "fails" += 1 elif status in 401, 403, 410 : self.state key "fails" = 999 permanent skip else: self.state key "fails" = max 0, self.state key "fails" - 1 Result : 429 errors become 30-second cooldowns instead of hard stops. Keys that return 401/403/410 EOL models are permanently skipped. nvidia cascade.py When a key works but the model fails timeout, 500, bad output , cascade to the next model: ACTIVE MODELS = "nvidia/nemotron-3.5-lightning-30b-a3b", fastest, good for simple "openai/gpt-oss-20b", Groq fallback via proxy "nvidia/nemotron-3-super-120b-a12b", primary workhorse "openai/gpt-oss-120b", Groq fallback via proxy "glm-5.2", backup "nvidia/nemotron-3-ultra-550b-a55b", heavy reasoning, reviewer Each model tried in order. First success wins. Logs show which model actually responded. nvidia proxy.py This was the hardest part. Hermes sends tool calls; the proxy must forward them to NVIDIA and return tool calls in the response. Also must stream SSE chunks correctly. Key fixes that took days: tools parameter in request body to NVIDIA tool calls present, return as-is don't wrap in text data: {json}\n\n , end with data: DONE \n\n finish reason: tool calls correctly Without these, Hermes either got empty streams or couldn't execute tools. Hermes config.yaml relevant section model: "nvidia/nemotron-3-super-120b-a12b" provider: "custom" custom: base url: "http://127.0.0.1:8001/v1" fallback providers: - name: "groq" models: - "openai/gpt-oss-120b" - "openai/gpt-oss-20b" Behavior: if primary proxy fails, Hermes automatically retries on Groq. Next turn, it tries primary again. | Pillar | Purpose | Status | |---|---|---| | 1 | Approval Gate human-in-the-loop | ✅ Working, tested end-to-end | | 2 | Alternative Supervision | ❌ Blocked — needs API keys | | 3 | Self-Healing | 📦 Archived — no Docker socket in Windmill CE | | 4 | Resilience/Queue | ✅ Restored via API job 01a0d246 | | 5 | Watchdog/Resume | 📦 Archived — same as Pillar 3 | | 6 | Cleanup/Mutex | ⚠️ Partial — cleanup paths wrong, mutex OK | | 7 | Sandbox | ⚠️ Partial — shell escape possible | | 8, 9, 10 | — | ❌ Never deployed | Reality : Only Pillars 1 and 4 are production-ready. The rest need Docker socket access Windmill EE feature or significant rework. retry watcher.py — cron every 2 min, checks failed jobs, retries, notifies Telegram nvidia reviewer.py — adversarial code review called via webhook or manually comando-gabriele — receives signed Telegram commands, executes terminal/file ops ~/.config/hermes/config.yaml key sections model: "nvidia/nemotron-3-super-120b-a12b" provider: "custom" custom: base url: "http://127.0.0.1:8001/v1" api key: "not-needed" proxy handles auth fallback providers: - name: "groq" models: - "openai/gpt-oss-120b" - "openai/gpt-oss-20b" auxiliary: title generation: enabled: false critical: was causing rate loops 3 bots, all on nvidia/nemotron-3-ultra-550b-a55b strongest model via proxy : @direttore — strategic coordinator, custom SOUL.md @analista — market/technical analysis, custom SOUL.md @critico — adversarial review, custom SOUL.md Group chat "Progetto ReportForge" completed 3 rounds, converged on positioning. Cron on @direttore runs every 24h. Limitation : Bots share the same proxy/key pool. Heavy concurrent use hits rate limits. Serial execution max 3 rounds mitigates this. Before : 429 errors killed agent loops. Manual key switching. After : Automatic. 10 keys, 30s cooldown on 429, cascade across 6 models, Groq fallback. Uptime 99% on free tiers. Problem : Nemotron-Super-120b hallucinates function signatures, file paths, config options when context grows. Fix : /new frequently nvidia reviewer Ultra 550B, temp 0.1 Problem : Proxy returned text instead of tool calls . Hermes waited forever. Fix : Proxy now passes tools param to NVIDIA, returns raw tool calls array. Verified with Hermes tool execution. Problem : "Empty stream" errors. NVIDIA sends SSE; proxy wasn't forwarding chunks correctly. Fix : Proper chunk parsing, data: prefix, DONE terminator. Hermes now receives stream correctly. Burned : meta/llama-3.1- , meta/llama-3.3- Aug 26, 2026 , openai/gpt-oss-120b on NVIDIA Sep 3, 2026 . Process : Before using any model, curl test. 410 = remove from cascade. Current active list maintained in nvidia cascade.py . Problem : PowerShell Out-File -Encoding UTF8 adds BOM EF BB BF . Python json.loads fails silently. Fix : Always write with System.IO.File ::WriteAllText $path, $content, $utf8NoBom where $utf8NoBom = New-Object System.Text.UTF8Encoding $false . Mistake : Direct psql inserts. Windmill cache didn't refresh. Scripts invisible in UI. Rule : Only official Windmill API CLI or HTTP . If API fails, stop and ask. | Metric | Value | Verified | |---|---|---| | NVIDIA API keys | 10 | API key.txt | | Active models in cascade | 6 | nvidia cascade.py | | Fallback models Groq | 2 | Hermes config | | Windmill containers | 3 healthy | docker ps | | Deployed scripts | 13 | Windmill UI | | Pillar 1 test | PASS | Job 01a0d0cf | | Pillar 4 restored | PASS | Job 01a0d246 | | NVIDIA Reviewer test | PASS | Found bug in sum a,b : return a-b | | Webwright + Reviewer integration | PASS | 10 issues found in git-filter-branch script | | Monthly cost | $0 | No paid services used | | Disk C: free | 36% | df -h | | Disk D: free | 35% | df -h | | Disk E: free | 66% | df -h | | Decision | Would Change To | |---|---| | Started with 10-pillar design | Start with 2-3 pillars that solve immediate pain | | Built custom proxy from scratch | Use existing OpenAI-compatible proxy if one supported tool calling + SSE | | Put all models on same key pool | Separate key pools per model tier | | Used Windmill CE for self-healing | Accept CE limits; build external watchers instead | | Tried to automate everything | Keep human-in-the-loop Pillar 1 for critical gates | docker-compose up for Windmill build.nvidia.com → API key.txt pip install webwright → skill for Hermes pip install edge-tts Microsoft, free This stack exists because I refused to pay $500+/month for agent infrastructure that randomly rate-limits or changes APIs. It's not magic — it's plumbing. Lots of plumbing. But it works, it's mine, and it costs nothing. If you build something similar: test every model before trusting it , log every fallback , and keep sessions short . The free tiers are generous but unforgiving. Built on Windows 11, Docker Desktop, WSL2. Zero cloud dependencies. Zero subscriptions.