Hey everyone,
I’ve been developing custom simulation software for my business. Up until recently, my workflow consisted of standard text-based Google Gemini alongside VS Code—with me acting as the human copy-paste middleman for hours on end, using Gemini’s chat interface to help write, iterate, and debug the codebase.
Since I have a substantial amount of local compute sitting idle between simulation runs, I decided to stop relying exclusively on web chat interfaces and start experimenting with local inference using Hermes, Ollama, vLLM, and SGLang.
My current infrastructure consists of two dedicated simulation servers and a collection of high-end GPUs:
Host Setup: Server A (512GB DDR5 RAM) and Server B (1TB DDR4 RAM)
GPUs: 1x NVIDIA RTX Pro 6000 Blackwell (96GB GDDR7) on Server A, 2x NVIDIA L40S, and 4x AMD Instinct MI210s on server B.
Obviously, running open-weights local models doesn’t quite match top-tier frontier APIs right out of the box. However, because I spent hundreds of hours guiding Gemini through the logic and debugging every routine step-by-step, I deeply understand the software architecture and how all the components fit together.
Observations & Issues So Far:
Inference & Web UI: Hermes with an agent dashboard web frontend has been the easiest to spin up and access across my local network. (Side rant: Chrome on Android refusing to auto-scroll chat windows is driving me nuts).
Context & Truncation: I immediately hit issues with output truncation on code blocks and commands. Bumping up the context window helped stabilize this somewhat, but it’s still a work in progress.
Model Selection: Qwen3.8-27B has been my primary model for Hermes and works the best of all the options I’ve tested so far for coding (including DeepSeek V4, Gemma4, and Qwen 3.7 Coder variants).
The Dilemma:
I keep contemplating throwing in the towel and just paying for cloud API tokens. But with an RTX Pro 6000 Blackwell, two L40Ss, four MI210s, and 1.5TB of RAM sitting across my rack, it physically hurts to pay per token. Furthermore, forcing myself to make local models work is helping me write more modular code, structure prompt contexts better, and maintain complete ownership over my codebase.
A few questions for the community:
Is anyone else using Hermes or vLLM/SGLang stacks with Qwen3.8-27B for serious daily coding projects on local hardware?
What IDE integrations or agent frameworks are you using locally to replace the manual “copy-paste middleman” loop (e.g., Continue.dev, Cursor with local OpenAI-compatible endpoints, Aider)?
How are you tuning context limits or stopping sequences in vLLM to prevent mid-code block truncation on large generations?
Would love to hear how others with local hardware are structuring their development workflows.