{"slug": "how-to-run-an-open-source-coding-model-locally-with-ollama", "title": "How to Run an Open-Source Coding Model Locally with Ollama", "summary": "A CoderBaba Team tutorial details how to run open-source coding models — Qwen2.5-Coder, Devstral and DeepSeek-Coder-V2 — locally through Ollama, which serves an OpenAI-compatible API at http://localhost:11434/v1 so editors such as the Continue extension in VS Code or JetBrains can use local models for autocomplete and chat. The guide recommends 8 GB of RAM minimum and 16 GB or more for practical use, noting Ollama's default 4-bit quantization shrinks models to roughly a quarter of full-precision size, so qwen2.5-coder:7b downloads at about 4.7 GB while qwen2.5-coder:32b needs about 20 GB. It advises pairing a small 1.5B-to-3B model for tab autocomplete with a larger 7B, 14B or 32B model for chat and refactoring.", "body_md": "# How to Run an Open-Source Coding Model Locally with Ollama\n\nA step-by-step guide to running Qwen2.5-Coder, Devstral or DeepSeek-Coder-V2 on your own machine with Ollama and connecting it to your editor.\n\nCoderBaba Team5 min read\n\nRunning a coding model locally gives you **privacy** (your code never leaves your machine), **zero per-token cost** and **offline access**. With [Ollama](https://ollama.com) it takes about five minutes to get started, and an afternoon to build a setup that rivals paid assistants for everyday work.\n\nThis tutorial takes you from zero to a working local setup: install Ollama, pick a model that fits your hardware, chat with it, connect it to your editor for autocomplete and chat, and finally run an agent.\n\n**What you need:** macOS, Windows or Linux. 8 GB of RAM at minimum, 16 GB or more recommended. 10 to 20 GB of free disk space.\n\n## [1. Install Ollama](#1-install-ollama)\n\nOn **macOS** or **Windows**, download the installer from [ollama.com](https://ollama.com) and run it. Ollama starts a background service and adds the `ollama` command to your terminal.\n\nOn **Linux**:\n\n**Checkpoint:** run `ollama --version`. You should see a version number. If the command isn't found, open a new terminal window so your shell picks up the updated PATH.\n\n## [2. Pick a model that fits your hardware](#2-pick-a-model-that-fits-your-hardware)\n\nThe most important rule of local AI: **the model must fit in memory**, ideally entirely in GPU memory or Apple Silicon unified memory. If it doesn't fit, it either fails to load or runs painfully slowly.\n\n| Your hardware | Good starting model | Approx. download | \n|---|---|---|\n| 8 GB RAM laptop | `qwen2.5-coder:1.5b` or`codegemma:2b` | 1 to 2 GB | \n| 16 GB RAM / 8 GB GPU | `qwen2.5-coder:7b` | about 4.7 GB | \n| 24 GB GPU or 32 GB Mac | `qwen2.5-coder:14b` or`devstral` | 9 to 14 GB | \n| 48 GB+ | `qwen2.5-coder:32b` | about 20 GB | \n\nOllama's default tags use **4-bit quantization**, which shrinks models to roughly a quarter of their full-precision size with only a small quality loss. That's why a 7B model fits in under 5 GB.\n\nNot sure what you have? On a Mac, check **About This Mac** for memory. On Windows or Linux with an NVIDIA GPU, run `nvidia-smi` to see VRAM. Browse the full [model directory](https://coderbaba.si/models) for licenses and specs.\n\n## [3. Pull and run it](#3-pull-and-run-it)\n\nThe first run downloads the weights. After that you can chat with the model in your terminal. Try a few prompts:\n\nType `/bye` to exit. Some useful commands:\n\n**Checkpoint:** while the model is generating, run `ollama ps` in another terminal. On a machine with a supported GPU, it should show the model running on the GPU. If it says CPU, generation will be slower. See troubleshooting below.\n\n## [4. Connect it to your editor](#4-connect-it-to-your-editor)\n\nOllama serves a local API at `http://localhost:11434`, including an **OpenAI-compatible endpoint** at `http://localhost:11434/v1`. Most AI coding extensions accept a custom endpoint, so you can point them at your local model instead of a cloud service.\n\n### [The two-model setup](#the-two-model-setup)\n\nThe best local editor setup uses **two models**:\n\n1. A **small, fast model for autocomplete,** where suggestions must appear in a fraction of a second.`qwen2.5-coder:1.5b` is a great choice.\n2. A **larger model for chat and edits,** where quality matters more than speed:`qwen2.5-coder:7b` ,`14b` or`32b` , depending on your hardware.\n\n### [Example: Continue in VS Code or JetBrains](#example-continue-in-vs-code-or-jetbrains)\n\nInstall the **Continue** extension, open its configuration and add both models with Ollama as the provider: the 14B model as your chat model and the 1.5B model as your **tab autocomplete** model. Continue talks to Ollama on `localhost:11434` by default.\n\nTip\n\nUse a small model (1.5B to 3B) for autocomplete, where latency matters, and a larger one for chat and refactoring.\n\n**Checkpoint:** open a code file, start typing a function and pause. You should see a grey inline suggestion within a second or so. Then open the chat panel, select some code and ask for an explanation.\n\n## [5. Try an agent](#5-try-an-agent)\n\nChat and autocomplete help you write code. **Agents** take on whole tasks: they read files, make multi-file changes and run your tests. For this, use a model trained for agentic work, such as [Devstral](https://coderbaba.si/models/devstral):\n\nThen connect an open-source agent:\n\n- **Aider** in the terminal, for example`aider --model ollama_chat/devstral` . Every change becomes a git commit you can review or undo.\n- **Cline** in VS Code: add an OpenAI-compatible provider with base URL`http://localhost:11434/v1` and approve each step.\n- **OpenHands** for autonomous runs in a sandbox.\n\nAlways commit your work before starting an agent, and work on a branch. To run several agents at once, give each one its own git worktree, as shown in our [Conductor alternative guide](https://coderbaba.si/conductor-alternative).\n\n## [Troubleshooting](#troubleshooting)\n\n**The model runs on the CPU and is slow.** Make sure your GPU drivers are installed (NVIDIA drivers with CUDA support on Windows and Linux). If the model is too big for your VRAM, Ollama splits it between GPU and CPU, which is much slower, so try a smaller model.\n\n**\"Out of memory\" or the system freezes.** Choose a smaller model or close memory-hungry apps. On Macs, unified memory is shared with the system, so leave a few gigabytes free.\n\n**The editor can't connect.** Check that Ollama is running (`ollama ps`) and that the extension uses `http://localhost:11434`, or `/v1` for OpenAI-compatible settings. VPNs and firewalls sometimes block local ports.\n\n**Answers ignore earlier context.** You may be hitting the context limit. Start a new chat for each task, or include fewer files.\n\n**Autocomplete is laggy.** Use a smaller autocomplete model and make sure it stays loaded. Loading a model on every request adds noticeable delay.\n\n## [Where to go next](#where-to-go-next)\n\n- Compare models for your hardware in the [model directory](https://coderbaba.si/models)\n- Learn the trade-offs in [How to Choose an Open-Source Coding Model](https://coderbaba.si/blog/how-to-choose-open-source-coding-model)\n- Read more about running models on your own hardware under [#local-ai](https://coderbaba.si/blog/tag/local-ai)\n\nYou now have a private, free coding assistant on your own machine, and you control exactly which model it runs.", "url": "https://wpnews.pro/news/how-to-run-an-open-source-coding-model-locally-with-ollama", "canonical_source": "https://coderbaba.si/blog/run-open-source-coding-model-locally-ollama", "published_at": "2026-11-04 00:00:00+00:00", "updated_at": "2026-11-16 16:20:25.427901+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools", "ai-products"], "entities": ["Ollama", "Qwen2.5-Coder", "Devstral", "DeepSeek-Coder-V2", "Continue", "VS Code", "JetBrains", "CoderBaba"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-run-an-open-source-coding-model-locally-with-ollama", "markdown": "https://wpnews.pro/news/how-to-run-an-open-source-coding-model-locally-with-ollama.md", "text": "https://wpnews.pro/news/how-to-run-an-open-source-coding-model-locally-with-ollama.txt", "jsonld": "https://wpnews.pro/news/how-to-run-an-open-source-coding-model-locally-with-ollama.jsonld"}}