A step-by-step guide to running Qwen2.5-Coder, Devstral or DeepSeek-Coder-V2 on your own machine with Ollama and connecting it to your editor.
CoderBaba Team5 min read
Running a coding model locally gives you privacy (your code never leaves your machine), zero per-token cost and offline access. With Ollama it takes about five minutes to get started, and an afternoon to build a setup that rivals paid assistants for everyday work.
This tutorial takes you from zero to a working local setup: install Ollama, pick a model that fits your hardware, chat with it, connect it to your editor for autocomplete and chat, and finally run an agent.
What you need: macOS, Windows or Linux. 8 GB of RAM at minimum, 16 GB or more recommended. 10 to 20 GB of free disk space.
1. Install Ollama #
On macOS or Windows, download the installer from ollama.com and run it. Ollama starts a background service and adds the ollama command to your terminal.
On Linux:
Checkpoint: run ollama --version. You should see a version number. If the command isn't found, open a new terminal window so your shell picks up the updated PATH.
2. Pick a model that fits your hardware #
The most important rule of local AI: the model must fit in memory, ideally entirely in GPU memory or Apple Silicon unified memory. If it doesn't fit, it either fails to load or runs painfully slowly.
| Your hardware | Good starting model | Approx. download |
|---|---|---|
| 8 GB RAM laptop | qwen2.5-coder:1.5b orcodegemma:2b |
1 to 2 GB |
| 16 GB RAM / 8 GB GPU | qwen2.5-coder:7b |
about 4.7 GB |
| 24 GB GPU or 32 GB Mac | qwen2.5-coder:14b ordevstral |
9 to 14 GB |
| 48 GB+ | qwen2.5-coder:32b |
about 20 GB |
Ollama's default tags use 4-bit quantization, which shrinks models to roughly a quarter of their full-precision size with only a small quality loss. That's why a 7B model fits in under 5 GB.
Not sure what you have? On a Mac, check About This Mac for memory. On Windows or Linux with an NVIDIA GPU, run nvidia-smi to see VRAM. Browse the full model directory for licenses and specs.
3. Pull and run it #
The first run downloads the weights. After that you can chat with the model in your terminal. Try a few prompts:
Type /bye to exit. Some useful commands:
Checkpoint: while the model is generating, run ollama ps in another terminal. On a machine with a supported GPU, it should show the model running on the GPU. If it says CPU, generation will be slower. See troubleshooting below.
4. Connect it to your editor #
Ollama serves a local API at http://localhost:11434, including an OpenAI-compatible endpoint at http://localhost:11434/v1. Most AI coding extensions accept a custom endpoint, so you can point them at your local model instead of a cloud service.
The two-model setup
The best local editor setup uses two models:
- A small, fast model for autocomplete, where suggestions must appear in a fraction of a second.
qwen2.5-coder:1.5bis a great choice. - A larger model for chat and edits, where quality matters more than speed:
qwen2.5-coder:7b,14bor32b, depending on your hardware.
Example: Continue in VS Code or JetBrains
Install the Continue extension, open its configuration and add both models with Ollama as the provider: the 14B model as your chat model and the 1.5B model as your tab autocomplete model. Continue talks to Ollama on localhost:11434 by default.
Tip
Use a small model (1.5B to 3B) for autocomplete, where latency matters, and a larger one for chat and refactoring. Checkpoint: open a code file, start typing a function and . You should see a grey inline suggestion within a second or so. Then open the chat panel, select some code and ask for an explanation.
5. Try an agent #
Chat and autocomplete help you write code. Agents take on whole tasks: they read files, make multi-file changes and run your tests. For this, use a model trained for agentic work, such as Devstral:
Then connect an open-source agent:
- Aider in the terminal, for example
aider --model ollama_chat/devstral. Every change becomes a git commit you can review or undo. - Cline in VS Code: add an OpenAI-compatible provider with base URL
http://localhost:11434/v1and approve each step. - OpenHands for autonomous runs in a sandbox.
Always commit your work before starting an agent, and work on a branch. To run several agents at once, give each one its own git worktree, as shown in our Conductor alternative guide.
Troubleshooting #
The model runs on the CPU and is slow. Make sure your GPU drivers are installed (NVIDIA drivers with CUDA support on Windows and Linux). If the model is too big for your VRAM, Ollama splits it between GPU and CPU, which is much slower, so try a smaller model.
"Out of memory" or the system freezes. Choose a smaller model or close memory-hungry apps. On Macs, unified memory is shared with the system, so leave a few gigabytes free.
The editor can't connect. Check that Ollama is running (ollama ps) and that the extension uses http://localhost:11434, or /v1 for OpenAI-compatible settings. VPNs and firewalls sometimes block local ports.
Answers ignore earlier context. You may be hitting the context limit. Start a new chat for each task, or include fewer files.
Autocomplete is laggy. Use a smaller autocomplete model and make sure it stays loaded. a model on every request adds noticeable delay.
## [Where to go next](#where-to-go-next)
- Compare models for your hardware in the [model directory](https://coderbaba.si/models)
- Learn the trade-offs in [How to Choose an Open-Source Coding Model](https://coderbaba.si/blog/how-to-choose-open-source-coding-model)
- Read more about running models on your own hardware under [#local-ai](https://coderbaba.si/blog/tag/local-ai)
You now have a private, free coding assistant on your own machine, and you control exactly which model it runs.