cd /news/ai-tools/how-to-run-an-open-source-coding-mod… · home › topics › ai-tools › article
[ARTICLE · art-170687] src=coderbaba.si ↗ pub= topic=ai-tools verified=true sentiment=· neutral

How to Run an Open-Source Coding Model Locally with Ollama

A CoderBaba Team tutorial details how to run open-source coding models — Qwen2.5-Coder, Devstral and DeepSeek-Coder-V2 — locally through Ollama, which serves an OpenAI-compatible API at http://localhost:11434/v1 so editors such as the Continue extension in VS Code or JetBrains can use local models for autocomplete and chat. The guide recommends 8 GB of RAM minimum and 16 GB or more for practical use, noting Ollama's default 4-bit quantization shrinks models to roughly a quarter of full-precision size, so qwen2.5-coder:7b downloads at about 4.7 GB while qwen2.5-coder:32b needs about 20 GB. It advises pairing a small 1.5B-to-3B model for tab autocomplete with a larger 7B, 14B or 32B model for chat and refactoring.

read5 min views6 publishedNov 4, 2026
How to Run an Open-Source Coding Model Locally with Ollama
Image: Coderbaba (auto-discovered)

A step-by-step guide to running Qwen2.5-Coder, Devstral or DeepSeek-Coder-V2 on your own machine with Ollama and connecting it to your editor.

CoderBaba Team5 min read

Running a coding model locally gives you privacy (your code never leaves your machine), zero per-token cost and offline access. With Ollama it takes about five minutes to get started, and an afternoon to build a setup that rivals paid assistants for everyday work.

This tutorial takes you from zero to a working local setup: install Ollama, pick a model that fits your hardware, chat with it, connect it to your editor for autocomplete and chat, and finally run an agent.

What you need: macOS, Windows or Linux. 8 GB of RAM at minimum, 16 GB or more recommended. 10 to 20 GB of free disk space.

1. Install Ollama #

On macOS or Windows, download the installer from ollama.com and run it. Ollama starts a background service and adds the ollama command to your terminal.

On Linux:

Checkpoint: run ollama --version. You should see a version number. If the command isn't found, open a new terminal window so your shell picks up the updated PATH.

2. Pick a model that fits your hardware #

The most important rule of local AI: the model must fit in memory, ideally entirely in GPU memory or Apple Silicon unified memory. If it doesn't fit, it either fails to load or runs painfully slowly.

Your hardware Good starting model Approx. download
8 GB RAM laptop qwen2.5-coder:1.5b orcodegemma:2b 1 to 2 GB
16 GB RAM / 8 GB GPU qwen2.5-coder:7b about 4.7 GB
24 GB GPU or 32 GB Mac qwen2.5-coder:14b ordevstral 9 to 14 GB
48 GB+ qwen2.5-coder:32b about 20 GB

Ollama's default tags use 4-bit quantization, which shrinks models to roughly a quarter of their full-precision size with only a small quality loss. That's why a 7B model fits in under 5 GB.

Not sure what you have? On a Mac, check About This Mac for memory. On Windows or Linux with an NVIDIA GPU, run nvidia-smi to see VRAM. Browse the full model directory for licenses and specs.

3. Pull and run it #

The first run downloads the weights. After that you can chat with the model in your terminal. Try a few prompts:

Type /bye to exit. Some useful commands: Checkpoint: while the model is generating, run ollama ps in another terminal. On a machine with a supported GPU, it should show the model running on the GPU. If it says CPU, generation will be slower. See troubleshooting below.

4. Connect it to your editor #

Ollama serves a local API at http://localhost:11434, including an OpenAI-compatible endpoint at http://localhost:11434/v1. Most AI coding extensions accept a custom endpoint, so you can point them at your local model instead of a cloud service.

The two-model setup

The best local editor setup uses two models:

  1. A small, fast model for autocomplete, where suggestions must appear in a fraction of a second.qwen2.5-coder:1.5b is a great choice.
  2. A larger model for chat and edits, where quality matters more than speed:qwen2.5-coder:7b ,14b or32b , depending on your hardware.

Example: Continue in VS Code or JetBrains

Install the Continue extension, open its configuration and add both models with Ollama as the provider: the 14B model as your chat model and the 1.5B model as your tab autocomplete model. Continue talks to Ollama on localhost:11434 by default.

Tip

Use a small model (1.5B to 3B) for autocomplete, where latency matters, and a larger one for chat and refactoring. Checkpoint: open a code file, start typing a function and . You should see a grey inline suggestion within a second or so. Then open the chat panel, select some code and ask for an explanation.

5. Try an agent #

Chat and autocomplete help you write code. Agents take on whole tasks: they read files, make multi-file changes and run your tests. For this, use a model trained for agentic work, such as Devstral:

Then connect an open-source agent:

  • Aider in the terminal, for exampleaider --model ollama_chat/devstral . Every change becomes a git commit you can review or undo.
  • Cline in VS Code: add an OpenAI-compatible provider with base URLhttp://localhost:11434/v1 and approve each step.
  • OpenHands for autonomous runs in a sandbox.

Always commit your work before starting an agent, and work on a branch. To run several agents at once, give each one its own git worktree, as shown in our Conductor alternative guide.

Troubleshooting #

The model runs on the CPU and is slow. Make sure your GPU drivers are installed (NVIDIA drivers with CUDA support on Windows and Linux). If the model is too big for your VRAM, Ollama splits it between GPU and CPU, which is much slower, so try a smaller model.

"Out of memory" or the system freezes. Choose a smaller model or close memory-hungry apps. On Macs, unified memory is shared with the system, so leave a few gigabytes free.

The editor can't connect. Check that Ollama is running (ollama ps) and that the extension uses http://localhost:11434, or /v1 for OpenAI-compatible settings. VPNs and firewalls sometimes block local ports.

Answers ignore earlier context. You may be hitting the context limit. Start a new chat for each task, or include fewer files.

Autocomplete is laggy. Use a smaller autocomplete model and make sure it stays loaded. a model on every request adds noticeable delay.

## [Where to go next](#where-to-go-next)

- Compare models for your hardware in the [model directory](https://coderbaba.si/models)
- Learn the trade-offs in [How to Choose an Open-Source Coding Model](https://coderbaba.si/blog/how-to-choose-open-source-coding-model)
- Read more about running models on your own hardware under [#local-ai](https://coderbaba.si/blog/tag/local-ai)

You now have a private, free coding assistant on your own machine, and you control exactly which model it runs.

── more in #ai-tools 4 stories · sorted by recency
── more on @ollama 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-run-an-open-s…] indexed:0 read:5min 2026-11-04 · —