How to Run an Open-Source Coding Model Locally with Ollama A CoderBaba Team tutorial details how to run open-source coding models — Qwen2.5-Coder, Devstral and DeepSeek-Coder-V2 — locally through Ollama, which serves an OpenAI-compatible API at http://localhost:11434/v1 so editors such as the Continue extension in VS Code or JetBrains can use local models for autocomplete and chat. The guide recommends 8 GB of RAM minimum and 16 GB or more for practical use, noting Ollama's default 4-bit quantization shrinks models to roughly a quarter of full-precision size, so qwen2.5-coder:7b downloads at about 4.7 GB while qwen2.5-coder:32b needs about 20 GB. It advises pairing a small 1.5B-to-3B model for tab autocomplete with a larger 7B, 14B or 32B model for chat and refactoring. How to Run an Open-Source Coding Model Locally with Ollama A step-by-step guide to running Qwen2.5-Coder, Devstral or DeepSeek-Coder-V2 on your own machine with Ollama and connecting it to your editor. CoderBaba Team5 min read Running a coding model locally gives you privacy your code never leaves your machine , zero per-token cost and offline access . With Ollama https://ollama.com it takes about five minutes to get started, and an afternoon to build a setup that rivals paid assistants for everyday work. This tutorial takes you from zero to a working local setup: install Ollama, pick a model that fits your hardware, chat with it, connect it to your editor for autocomplete and chat, and finally run an agent. What you need: macOS, Windows or Linux. 8 GB of RAM at minimum, 16 GB or more recommended. 10 to 20 GB of free disk space. 1. Install Ollama 1-install-ollama On macOS or Windows , download the installer from ollama.com https://ollama.com and run it. Ollama starts a background service and adds the ollama command to your terminal. On Linux : Checkpoint: run ollama --version . You should see a version number. If the command isn't found, open a new terminal window so your shell picks up the updated PATH. 2. Pick a model that fits your hardware 2-pick-a-model-that-fits-your-hardware The most important rule of local AI: the model must fit in memory , ideally entirely in GPU memory or Apple Silicon unified memory. If it doesn't fit, it either fails to load or runs painfully slowly. | Your hardware | Good starting model | Approx. download | |---|---|---| | 8 GB RAM laptop | qwen2.5-coder:1.5b or codegemma:2b | 1 to 2 GB | | 16 GB RAM / 8 GB GPU | qwen2.5-coder:7b | about 4.7 GB | | 24 GB GPU or 32 GB Mac | qwen2.5-coder:14b or devstral | 9 to 14 GB | | 48 GB+ | qwen2.5-coder:32b | about 20 GB | Ollama's default tags use 4-bit quantization , which shrinks models to roughly a quarter of their full-precision size with only a small quality loss. That's why a 7B model fits in under 5 GB. Not sure what you have? On a Mac, check About This Mac for memory. On Windows or Linux with an NVIDIA GPU, run nvidia-smi to see VRAM. Browse the full model directory https://coderbaba.si/models for licenses and specs. 3. Pull and run it 3-pull-and-run-it The first run downloads the weights. After that you can chat with the model in your terminal. Try a few prompts: Type /bye to exit. Some useful commands: Checkpoint: while the model is generating, run ollama ps in another terminal. On a machine with a supported GPU, it should show the model running on the GPU. If it says CPU, generation will be slower. See troubleshooting below. 4. Connect it to your editor 4-connect-it-to-your-editor Ollama serves a local API at http://localhost:11434 , including an OpenAI-compatible endpoint at http://localhost:11434/v1 . Most AI coding extensions accept a custom endpoint, so you can point them at your local model instead of a cloud service. The two-model setup the-two-model-setup The best local editor setup uses two models : 1. A small, fast model for autocomplete, where suggestions must appear in a fraction of a second. qwen2.5-coder:1.5b is a great choice. 2. A larger model for chat and edits, where quality matters more than speed: qwen2.5-coder:7b , 14b or 32b , depending on your hardware. Example: Continue in VS Code or JetBrains example-continue-in-vs-code-or-jetbrains Install the Continue extension, open its configuration and add both models with Ollama as the provider: the 14B model as your chat model and the 1.5B model as your tab autocomplete model. Continue talks to Ollama on localhost:11434 by default. Tip Use a small model 1.5B to 3B for autocomplete, where latency matters, and a larger one for chat and refactoring. Checkpoint: open a code file, start typing a function and pause. You should see a grey inline suggestion within a second or so. Then open the chat panel, select some code and ask for an explanation. 5. Try an agent 5-try-an-agent Chat and autocomplete help you write code. Agents take on whole tasks: they read files, make multi-file changes and run your tests. For this, use a model trained for agentic work, such as Devstral https://coderbaba.si/models/devstral : Then connect an open-source agent: - Aider in the terminal, for example aider --model ollama chat/devstral . Every change becomes a git commit you can review or undo. - Cline in VS Code: add an OpenAI-compatible provider with base URL http://localhost:11434/v1 and approve each step. - OpenHands for autonomous runs in a sandbox. Always commit your work before starting an agent, and work on a branch. To run several agents at once, give each one its own git worktree, as shown in our Conductor alternative guide https://coderbaba.si/conductor-alternative . Troubleshooting troubleshooting The model runs on the CPU and is slow. Make sure your GPU drivers are installed NVIDIA drivers with CUDA support on Windows and Linux . If the model is too big for your VRAM, Ollama splits it between GPU and CPU, which is much slower, so try a smaller model. "Out of memory" or the system freezes. Choose a smaller model or close memory-hungry apps. On Macs, unified memory is shared with the system, so leave a few gigabytes free. The editor can't connect. Check that Ollama is running ollama ps and that the extension uses http://localhost:11434 , or /v1 for OpenAI-compatible settings. VPNs and firewalls sometimes block local ports. Answers ignore earlier context. You may be hitting the context limit. Start a new chat for each task, or include fewer files. Autocomplete is laggy. Use a smaller autocomplete model and make sure it stays loaded. Loading a model on every request adds noticeable delay. Where to go next where-to-go-next - Compare models for your hardware in the model directory https://coderbaba.si/models - Learn the trade-offs in How to Choose an Open-Source Coding Model https://coderbaba.si/blog/how-to-choose-open-source-coding-model - Read more about running models on your own hardware under local-ai https://coderbaba.si/blog/tag/local-ai You now have a private, free coding assistant on your own machine, and you control exactly which model it runs.