Over the past year, coding agents have rapidly become the new standard for software development. But while they’ve drastically simplified how we write code, they’ve also introduced hidden costs. As AI vendors shift to usage-based pricing, developers are getting hit with skyrocketing bills.
Fortunately, open-source is striking back. New models have closed the gap with state-of-the-art (SOTA) giants, and they’re optimized enough to run entirely on local consumer hardware — even a 2020-era NVIDIA RTX 3090. In this post, I’ll guide you through setting up a powerful local coding agent where your only operating cost is your electricity.
A few years ago, I picked up an NVIDIA RTX 3090. Thanks to its massive 24GB of VRAM, it’s still an absolute powerhouse today, fully capable of running quantized ~30B parameter LLMs. Inspired by this great post by Moncef Abboud, I decided to finally put it to work and become independent of OpenAI, Google or Anthropic.
There are quite a few incredible open-weight coding models available right now that fit perfectly on this GPU. For this specific setup, we’ll be using Gemma4, though alternatives like Qwen work beautifully too. If you want to explore other options, Hugging Face offers a comprehensive leaderboard of the current top performers:
In this setup, we will use (𝚌𝚢𝚊𝚗𝚔𝚒𝚠𝚒/𝚐𝚎𝚖𝚖𝚊-𝟺–𝟸𝟼𝙱-𝙰𝟺𝙱-𝚒𝚝-𝙰𝚆𝚀-𝟺𝚋𝚒𝚝), which is already quantized and can be used immediately in vLLM out-of-the-box.
The entire setup consists of a vLLM docker container running the LLM and an installation of OpenCode, which is an open source alternative to Claude Code. Hence it is necessary to install the dependencies first.
Hardware is only half the battle. To actually run this local coding agent, we need a way to serve the model and an interface to interact with it. Here is the software stack you’ll need:
Docker
We will be using Docker to host our vLLM image. Running the model in a container keeps our system clean and makes the setup process much smoother. If you don’t have it installed yet, grab it from the official source here.
OpenCode
To replace the functionality of Claude Code, we need an agent framework that can read files, write code, and execute terminal commands. OpenCode is a fantastic open-source harness built exactly for this. You can check out their documentation and installation instructions on the official OpenCode site.
First, we need to configure our vLLM container. Here is the docker-compose.yml:
services: vllm: image: vllm/vllm-openai:latest container_name: vllm deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] volumes: - ~/.cache/huggingface:/root/.cache/huggingface environment: - HF_TOKEN="YOUR HF TOKEN" ports: - "8000:8000" ipc: host command: > cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit --max-model-len 65536 --max-num-batched-tokens 4096 --enable-auto-tool-choice --tool-call-parser gemma4
It is crucial to set the argument —** tool-call-parser gemma4** and enable —
docker compose up -d
a 26B model into VRAM takes a moment, so grab a sip of coffee. Once it’s ready, verify the model is online with a quick curl or wget request:
curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit", "prompt": "Hello World" }'
or
wget -qO- http://localhost:8000/v1/completions \ --header="Content-Type: application/json" \ --post-data='{"model": "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit", "prompt": "Hello World"}'
If the model replies, we are ready to connect it to our coding agent. Ensure OpenCode is installed and accessible by typing opencode in your command line. Next, we just need to point OpenCode to our local vLLM server. Open your config file at ~/.config/opencode/opencode.json (or .jsonc) and define the service like this:
{ "$schema": "https://opencode.ai/config.json", "provider": { "vllm": { "npm": "@ai-sdk/openai-compatible", "name": "Gemma4 vLLM", "options": { "baseURL": "http://localhost:8000/v1" }, "models": { "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit": { "name": "Gemma4-26B-A4B-it-AWQ-4bit", "options": { "max_tokens": 20000 } } } } }}
Launch OpenCode, type ** /models** in the chat, and select
Let’s be realistic: a quantized 26B model isn’t going to have the exact zero-shot magic of a massive SOTA model. To get the most out of it, you need to break your tasks down into smaller steps and provide a bit more supervision. But once you adapt to that workflow, the results are incredible. This setup is perfectly capable of implementing features, summarizing codebases, and finding bugs flawlessly. You get all the productivity benefits of an AI agent, without the anxiety of exploding token costs.
A huge shoutout to this blog post by Moncef Abboud, which was instrumental in putting this setup together. If you don’t have a 24GB local GPU and want to run this coding agent on cloud hardware with ssh instead, I highly recommend checking out his guide!
The Open Source Counter-Strike: Running Local Coding Agents was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.