{"slug": "how-i-built-a-local-ai-agent-stack-for-0-month-hermes-windmill-nvidia-groq", "title": "How I Built a Local AI Agent Stack for $0/month (Hermes + Windmill + NVIDIA + Groq)", "summary": "A developer built a $0/month local AI agent stack on a Windows machine (i7-8700K, 32GB RAM, RTX 3080) that runs code, calls APIs, browses the web and writes files, combining Windmill for orchestration, Hermes Agent from Nous Research as the interface, and a local OpenAI-compatible proxy that rotates 10 NVIDIA NIM API keys across a six-model cascade with Groq gpt-oss fallbacks. The writeup documents the failure modes encountered, including NVIDIA's 40 RPM free-tier limit and rate loops caused by Hermes' auxiliary model, which the developer disabled.", "body_md": "*Published: 2026-09-25 | ~2,200 words | 11 min read*\n\nI needed AI agents that could actually *do* things — run code, call APIs, browse the web, write files — not just chat. Cloud APIs got expensive fast, rate-limited unpredictably, and locked me into someone else's infrastructure.\n\nSo I built a local stack on my Windows machine (i7-8700K, 32GB RAM, RTX 3080). Total monthly cost: **$0**. Everything runs locally or on free tiers.\n\nThis article documents what actually works, what broke repeatedly, and the solutions that stuck.\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                      ORCHESTRATION                               │\n│  Windmill (localhost:8000) — 3 containers                       │\n│  - windmill-postgres, windmill-server, windmill-worker          │\n│  - Workspace: admins, NO_AUTH=true (dev only)                   │\n│  - 13 scripts deployed: 7 pillars + 4 automations + 1 reviewer  │\n└─────────────────────────────────────────────────────────────────┘\n                                │\n                                ▼\n┌─────────────────────────────────────────────────────────────────┐\n│                      MODEL LAYER                                 │\n│  Primary: nvidia/nemotron-3-super-120b-a12b                     │\n│  Fallback 1: openai/gpt-oss-120b (Groq)                         │\n│  Fallback 2: openai/gpt-oss-20b (Groq)                          │\n│  Reviewer: nvidia/nemotron-3-ultra-550b-a55b (adversarial)      │\n│                                                                  │\n│  Local Proxy: 127.0.0.1:8001 (OpenAI-compatible)                │\n│  - 10 NVIDIA API keys in rotation                               │\n│  - 6-model cascade on failure                                   │\n│  - Tool calling + SSE streaming support                         │\n└─────────────────────────────────────────────────────────────────┘\n                                │\n                                ▼\n┌─────────────────────────────────────────────────────────────────┐\n│                      AGENT INTERFACE                             │\n│  Hermes Agent (Nous Research) — Telegram + local CLI            │\n│  - Configured with custom provider → local proxy                │\n│  - Fallback providers: Groq (gpt-oss-120b → gpt-oss-20b)       │\n│  - Auxiliary model DISABLED (caused rate loops)                 │\n│  - Bot Mode: 3 bots configured on Ultra 550B (see below)        │\n└─────────────────────────────────────────────────────────────────┘\n                                │\n                                ▼\n┌─────────────────────────────────────────────────────────────────┐\n│                      TOOLS & AUTOMATION                          │\n│  Webwright (Playwright) — browser automation                    │\n│  NVIDIA Reviewer — adversarial code review via Windmill         │\n│  Edge-TTS — local voice synthesis (Microsoft, free)             │\n│  FFmpeg — video assembly                                        │\n│  Retry watcher — cron every 2 min, Telegram alerts              │\n└─────────────────────────────────────────────────────────────────┘\n```\n\nNVIDIA NIM free tier: 40 RPM. That's it. One agent loop can burn 10 requests in seconds. Groq free tier is generous but has different models. OpenRouter credits exhausted.\n\n`nvidia_key_rotation.py`)\n\n``` python\n# Simplified logic\nclass KeyRotator:\n    def __init__(self, keys: list[str]):\n        self.keys = keys\n        self.state = {k: {\"fails\": 0, \"cooldown_until\": 0} for k in keys}\n        self.current_idx = 0\n\n    def get_key(self) -> str:\n        now = time.time()\n        for _ in range(len(self.keys)):\n            key = self.keys[self.current_idx]\n            if self.state[key][\"cooldown_until\"] <= now:\n                return key\n            self.current_idx = (self.current_idx + 1) % len(self.keys)\n        raise Exception(\"All keys in cooldown\")\n\n    def report_result(self, key: str, status: int):\n        if status == 429:\n            self.state[key][\"cooldown_until\"] = time.time() + 30\n            self.state[key][\"fails\"] += 1\n        elif status in (401, 403, 410):\n            self.state[key][\"fails\"] = 999  # permanent skip\n        else:\n            self.state[key][\"fails\"] = max(0, self.state[key][\"fails\"] - 1)\n```\n\n**Result**: 429 errors become 30-second cooldowns instead of hard stops. Keys that return 401/403/410 (EOL models) are permanently skipped.\n\n`nvidia_cascade.py`)\nWhen a key works but the model fails (timeout, 500, bad output), cascade to the next model:\n\n```\nACTIVE_MODELS = [\n    \"nvidia/nemotron-3.5-lightning-30b-a3b\",   # fastest, good for simple\n    \"openai/gpt-oss-20b\",                       # Groq fallback via proxy\n    \"nvidia/nemotron-3-super-120b-a12b\",       # primary workhorse\n    \"openai/gpt-oss-120b\",                      # Groq fallback via proxy\n    \"glm-5.2\",                                  # backup\n    \"nvidia/nemotron-3-ultra-550b-a55b\",       # heavy reasoning, reviewer\n]\n```\n\nEach model tried in order. First success wins. Logs show which model actually responded.\n\n`nvidia_proxy.py`)\nThis was the hardest part. Hermes sends tool calls; the proxy must forward them to NVIDIA and return `tool_calls` in the response. Also must stream SSE chunks correctly.\n\n**Key fixes that took days:**\n\n`tools` parameter in request body to NVIDIA`tool_calls` present, return as-is (don't wrap in text)`data: {json}\\n\\n`, end with `data: [DONE]\\n\\n`\n`finish_reason: tool_calls` correctly\nWithout these, Hermes either got empty streams or couldn't execute tools.\n\n```\n# Hermes config.yaml (relevant section)\nmodel: \"nvidia/nemotron-3-super-120b-a12b\"\nprovider: \"custom\"\ncustom:\n  base_url: \"http://127.0.0.1:8001/v1\"\nfallback_providers:\n  - name: \"groq\"\n    models:\n      - \"openai/gpt-oss-120b\"\n      - \"openai/gpt-oss-20b\"\n```\n\nBehavior: if primary (proxy) fails, Hermes automatically retries on Groq. Next turn, it tries primary again.\n\n| Pillar | Purpose | Status | \n|---|---|---|\n| 1 | Approval Gate (human-in-the-loop) | ✅ Working, tested end-to-end | \n| 2 | Alternative Supervision | ❌ Blocked — needs API keys | \n| 3 | Self-Healing | 📦 Archived — no Docker socket in Windmill CE | \n| 4 | Resilience/Queue | ✅ Restored via API (job 01a0d246) | \n| 5 | Watchdog/Resume | 📦 Archived — same as Pillar 3 | \n| 6 | Cleanup/Mutex | ⚠️ Partial — cleanup paths wrong, mutex OK | \n| 7 | Sandbox | ⚠️ Partial — shell escape possible | \n| 8, 9, 10 | — | ❌ Never deployed | \n\n**Reality**: Only Pillars 1 and 4 are production-ready. The rest need Docker socket access (Windmill EE feature) or significant rework.\n\n`retry_watcher.py` — cron every 2 min, checks failed jobs, retries, notifies Telegram`nvidia_reviewer.py` — adversarial code review (called via webhook or manually)`comando-gabriele` — receives signed Telegram commands, executes terminal/file ops\n\n```\n# ~/.config/hermes/config.yaml (key sections)\nmodel: \"nvidia/nemotron-3-super-120b-a12b\"\nprovider: \"custom\"\ncustom:\n  base_url: \"http://127.0.0.1:8001/v1\"\n  api_key: \"not-needed\"  # proxy handles auth\n\nfallback_providers:\n  - name: \"groq\"\n    models:\n      - \"openai/gpt-oss-120b\"\n      - \"openai/gpt-oss-20b\"\n\nauxiliary:\n  title_generation:\n    enabled: false  # critical: was causing rate loops\n```\n\n3 bots, all on `nvidia/nemotron-3-ultra-550b-a55b` (strongest model via proxy):\n\n`@direttore` — strategic coordinator, custom SOUL.md`@analista` — market/technical analysis, custom SOUL.md\n`@critico` — adversarial review, custom SOUL.md\nGroup chat \"Progetto ReportForge\" completed 3 rounds, converged on positioning. Cron on @direttore runs every 24h.\n\n**Limitation**: Bots share the same proxy/key pool. Heavy concurrent use hits rate limits. Serial execution (max 3 rounds) mitigates this.\n\n**Before**: 429 errors killed agent loops. Manual key switching.\n\n**After**: Automatic. 10 keys, 30s cooldown on 429, cascade across 6 models, Groq fallback. Uptime >99% on free tiers.\n\n**Problem**: Nemotron-Super-120b hallucinates function signatures, file paths, config options when context grows.\n\n**Fix**: \n\n`/new` frequently`nvidia_reviewer` (Ultra 550B, temp 0.1)\n**Problem**: Proxy returned text instead of `tool_calls`. Hermes waited forever.\n\n**Fix**: Proxy now passes `tools` param to NVIDIA, returns raw `tool_calls` array. Verified with Hermes tool execution.\n\n**Problem**: \"Empty stream\" errors. NVIDIA sends SSE; proxy wasn't forwarding chunks correctly.\n\n**Fix**: Proper chunk parsing, `data:` prefix, `[DONE]` terminator. Hermes now receives stream correctly.\n\n**Burned**: `meta/llama-3.1-*`, `meta/llama-3.3-*` (Aug 26, 2026), `openai/gpt-oss-120b` on NVIDIA (Sep 3, 2026).\n\n**Process**: Before using any model, `curl` test. 410 = remove from cascade. Current active list maintained in `nvidia_cascade.py`.\n\n**Problem**: PowerShell `Out-File -Encoding UTF8` adds BOM (EF BB BF). Python `json.loads` fails silently.\n\n**Fix**: Always write with `[System.IO.File]::WriteAllText($path, $content, $utf8NoBom)` where `$utf8NoBom = New-Object System.Text.UTF8Encoding $false`.\n\n**Mistake**: Direct psql inserts. Windmill cache didn't refresh. Scripts invisible in UI.\n\n**Rule**: Only official Windmill API (CLI or HTTP). If API fails, stop and ask.\n\n| Metric | Value | Verified | \n|---|---|---|\n| NVIDIA API keys | 10 | `API_key.txt` | \n| Active models in cascade | 6 | `nvidia_cascade.py` | \n| Fallback models (Groq) | 2 | Hermes config | \n| Windmill containers | 3 healthy | `docker ps` | \n| Deployed scripts | 13 | Windmill UI | \n| Pillar 1 test | PASS | Job 01a0d0cf | \n| Pillar 4 restored | PASS | Job 01a0d246 | \n| NVIDIA Reviewer test | PASS | Found bug in `sum(a,b): return a-b` | \n| Webwright + Reviewer integration | PASS | 10 issues found in git-filter-branch script | \n| Monthly cost | $0 | No paid services used | \n| Disk C: free | 36% | `df -h` | \n| Disk D: free | 35% | `df -h` | \n| Disk E: free | 66% | `df -h` | \n\n| Decision | Would Change To | \n|---|---|\n| Started with 10-pillar design | Start with 2-3 pillars that solve immediate pain | \n| Built custom proxy from scratch | Use existing OpenAI-compatible proxy if one supported tool calling + SSE | \n| Put all models on same key pool | Separate key pools per model tier | \n| Used Windmill CE for self-healing | Accept CE limits; build external watchers instead | \n| Tried to automate everything | Keep human-in-the-loop (Pillar 1) for critical gates | \n\n`docker-compose up` for Windmill`build.nvidia.com` → `API_key.txt`\n`pip install webwright` → skill for Hermes`pip install edge-tts` (Microsoft, free)\nThis stack exists because I refused to pay $500+/month for agent infrastructure that randomly rate-limits or changes APIs. It's not magic — it's plumbing. Lots of plumbing. But it works, it's mine, and it costs nothing.\n\nIf you build something similar: **test every model before trusting it**, **log every fallback**, and **keep sessions short**. The free tiers are generous but unforgiving.\n\n*Built on Windows 11, Docker Desktop, WSL2. Zero cloud dependencies. Zero subscriptions.*", "url": "https://wpnews.pro/news/how-i-built-a-local-ai-agent-stack-for-0-month-hermes-windmill-nvidia-groq", "canonical_source": "https://dev.to/gabriele_dimaria_f47e042/how-i-built-a-local-ai-agent-stack-for-0month-hermes-windmill-nvidia-groq-1g3o", "published_at": "2026-09-25 16:00:28+00:00", "updated_at": "2026-09-25 16:31:13.518382+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "large-language-models", "developer-tools"], "entities": ["Windmill", "Hermes Agent", "Nous Research", "NVIDIA", "Groq", "NVIDIA NIM", "OpenRouter", "Playwright"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-local-ai-agent-stack-for-0-month-hermes-windmill-nvidia-groq", "markdown": "https://wpnews.pro/news/how-i-built-a-local-ai-agent-stack-for-0-month-hermes-windmill-nvidia-groq.md", "text": "https://wpnews.pro/news/how-i-built-a-local-ai-agent-stack-for-0-month-hermes-windmill-nvidia-groq.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-local-ai-agent-stack-for-0-month-hermes-windmill-nvidia-groq.jsonld"}}