{"slug": "viability-of-local-models-for-coding", "title": "Viability of Local Models for Coding", "summary": "Thoughtworks technologist Birgitta Böckeler tested running local large language models for agentic coding on Apple M3 Max and M5 Pro machines, finding that while RAM constraints limit model size to 15-25GB for comfortable use, response speeds have improved significantly over the past year. Tool calling remains unreliable but models can self-recover, and overall code quality is hit-or-miss compared to cloud-based models.", "body_md": "# Viability of local models for coding\n\nThis article is part of [“Exploring Gen\nAI”](../exploring-gen-ai.html). A series capturing Thoughtworks technologists' explorations of using gen ai technology for\nsoftware development.\n\n07 July 2026\n\nUntil recently I hadn't tried running models locally for quite a while, the disappointment had just always been too high when I did it. About a month ago though, I dove back in - there were just too many claims out there to ignore, about how far they have come, how it's now much more feasible to run them, and how some of them have become really good at coding. So this is my personal experience with using them, on and off, over the past 4 weeks or so.\n\nIn this memo, I will start with a more general introduction and go through the factors that influence the viability of these models for coding. In a follow-up memo, I will describe my actual experiences in more detail.\n\n## Scope\n\nMy main interest is how useful they are for coding, and not just auto complete, but *\nagentic* coding. Secondly, I'm interested in how *ready and usable* they are more\nbroadly, for developers who don't want to dive into a bunch of specs and extra tooling and\ntweaks to make it work.\n\nIn terms of hardware, I have been running models on these two machines:\n\n- Apple M3 Max, 48GB RAM\n- Apple M5 Pro, 64GB RAM\n\n## Factors that influence viability\n\nThere are a myriad of factors at play that can influence the results, which makes it quite a tedious undertaking to evaluate which setup works best under the resource constraints we have. It also makes it very hard to filter the signals from the noise when people share their successes with these models online.\n\nI found it particularly baffling that in the automated eval setup, one model clearly delivered better outcomes on the stronger machine (not just speed, but better code!), in spite of all other settings being the same.\n\nI'll start with a summary, before going into details of each of these factors.\n\nFigure 1: The many factors that can impact the results - tap or hover over boxes to highlight arrows.\n\n*Runnability:*RAM is the core constraint. I used models between 15-25GB, context windows of maximum 64K, harnesses OpenCode and Pi with zero Skills and MCP servers active)*Response speed:*Is impacted by many factors, but was quite good for some models, and leaps and bounds from where it was a year ago. Setups: LM Studio + 4BIT quantization + M3 Max/M5 Pro + GGUF/MLX both*Viability for agentic coding:*Tool calling was tricky still, the models often failed, but can usually self-recover from their failures. This is a key component of*agentic*coding specifically. Without it, you can still go the old-fashioned way of copying and pasting from a chat window of course. And small models are definitely a lot more viable for auto complete than agentic use.*Quality of outcomes:*Depends on the task (more on that in the next memo), but obviously nowhere near the type of capability we can get from the big models. Overall it was very hit and miss. I only looked at correctness of functionality, didn't go into code quality.\n\n## RAM\n\nThe model weights must fit into the available RAM, or more specifically, VRAM. If they don't, the runtime either crashes (happened to me once!) or drops to unusably slow speeds. On Apple Silicon, virtually all RAM is accessible to the GPU with no separate VRAM limit, this can be different in other machine configurations.\n\n*Impacts:* Model runnability; Speed of responses\n\n*My experience:* On the machine with 48GB, I ran models between 8 and almost 30GB.\nThe 30GB stretches it a lot of course, especially when the context window gets added,\nthe 15-25GB size is more comfortable and I don't have to close quite as many other\napplications. On the machine with 64GB, I once ran a model that was 48GB big - which\nwent great at first, but then it quickly crashed...\n\n## Processing power\n\nMore cores generally means faster token generation, but architecture matters too, and newer chip generations can close the gap even with fewer cores. This is a tough one to compare between machines without diving much deeper into the details of each configuration and architecture.\n\n*Impacts:* Speed of responses\n\n*My experience*: On both the M3 Max and the M5 Pro, all models I ran had quite an\nimpressive speed, compared to where they were a year ago. Speed degrades though the longer a conversation gets. I am okay using them at the current speed for some tasks - if the quality of the output were acceptable.\n\n## Memory bandwidth\n\nMemory bandwidth is a bottleneck for token generation, determining how fast data moves between RAM and the compute units.\n\n*Impacts:* Speed of responses\n\n*My experience:* Both the M3 Max and the M5 Pro I used have a nearly identical\nbandwidth of ~300 GB/s, so I don't really have a comparison to something else. But as\nmentioned before, the speed of all models I tried was quite acceptable.\n\n## Number of parameters\n\nThe parameter count basically represents the size of a model's learned knowledge and capabilities. More parameters generally mean better output quality, but also a larger file size.\n\n*Impacts:* Amount of RAM needed; Quality of results\n\n*My experience:* With 48GB, I used models around 30B parameters, +/- 5B. The\nbiggest model I loaded on the 64GB machine was Qwen3 Coder Next 80B (MoE), which solved\nthe task I gave it a lot better than the smaller models - but then crashed after I\ncontinued the conversation.\n\n## Reasoning capabilities\n\nReasoning models go through a “chain of thought” process before responding, which helps with complex multi-step tasks, but can also generate significantly more tokens and slow responses down.\n\n*Impacts:* Complexity of the tasks; Speed of response; Context window size (and\ntherefore need for RAM)\n\n*My experience:* All of the models I tried had reasoning capabilities, and they were switched on by default throughout most of my experiments. However, I\noften noticed them going in endless circles inside of the reasoning chain, especially\nthe smaller ones. (“Wait, ...”, “Actually, ...”, “But wait, ...”) So I also did a few runs of my automated setup with reasoning off - and lo and behold, it's not only faster (which was to be expected), but also performed the same to slightly better! A good reminder that reasoning is not always necessary, and can sometimes even be counterproductive.\n\n## Tool calling capabilities\n\nFor agentic use, a model must be able to reliably emit structured tool call syntax that matches the schema the harness expects. Models that weren't specifically trained or fine-tuned for tool calling often produce malformed calls.\n\n*Impacts:* Ability to use agentic harnesses\n\n*My experience:* This was a common issue with the models I tried, though they\ncould often self-correct and recover from a failed tool call (e.g. using wrong parameter\nnames like `file.path`\n\ninstead of `filePath`\n\n).\n\n## Format\n\nGGUF is the standard format for llama.cpp-based runtimes like LM Studio and Ollama, and has by far the largest model library. MLX is Apple's own framework built specifically for Apple Silicon and can be faster, but fewer MLX-formatted models are available at the moment.\n\n*Impacts:* Speed of responses\n\n*My experience:* I tried both formats for one or two of the models, but I\npersonally didn't *feel* much of a difference. That could be due to the\nunstructured type of evaluation I did - on the other hand, as any user experience\nresearcher would tell us, the *\nperceived* speed is ultimately what matters, not what the clock says...\n\n## Quantization\n\nQuantization compresses model weights to reduce the file size, trading some quality for a much smaller RAM footprint. The level of quantization is usually marked in model names and descriptions as Q4 / Q6 / Q8, or 4BIT / 6BIT / 8BIT, with lower numbers meaning higher compression. The newest buzz recently have been QAT (Quantization-Aware Training) variants of models, which are trained with quantization simulated during the training. They are supposed to preserve quality better than standard quantization.\n\n*Impacts:* Amount of RAM needed; Speed of responses; Quality of responses\n\n*My experience:* All of the models I downloaded were at Q4 / 4BIT, I didn't get\naround to trying different variations yet. I also haven't gotten around to trying a QAT\none.\n\n## Architecture\n\n*MoE (Mixture of Experts)* models have a large total parameter count but only\nactivate a subsection of their weights at inference time, so a 35B MoE model needs\nsignificantly less RAM and can run faster than a 35B *dense* model.\n\n*Impacts:* Amount of RAM needed; Speed of responses\n\n*My experience:* The Qwen3.6 35B MoE model was by far giving me the best balance between number of parameters and RAM usage, and therefore runnability and quality of outcomes. This could be due to the MoE architecture, I'm not sure. The architecture might also explain my experience of getting better coding abilities out of the model on the 64GB machine than the 48GB - it might be loading more experts there? I'm not sure if that's true, but it's the only sensible hypothesis I have so far.\n\n## Context window size setting\n\nContext window size consumes RAM on top of model weights through the KV cache, which grows with context length. The default size configured in the runtimes is far too small for agentic coding, it has to be set to at least 32K, if not 64K.\n\n*Impacts:* Size and complexity of tasks; Amount of RAM needed; Speed of responses;\nAbility to use reasoning\n\n*My experience:* I tried to see how little I could get away with. For small tasks\nI could sometimes work with 32K, but often I had to increase to 64K, so that seems to be\na good default minimum. As the models themselves were already pushing the boundaries of\nmy available RAM, I'm not sure how how much more I could still increase it, even on the\n64GB machine... So while many of these models in theory support a larger window,\nactually using it is limited by the memory constraints.\n\n## List of models used\n\nThe buzz for coding has been all around Qwen3 and Gemma 4, so those were the ones I went for.\n\n**Qwen3**\n\n- Qwen3.6 35B-A3B MoE Q4 GGUF (22 GB)\n- Qwen3.6 Coder Next 80B MoE GGUF (45 GB)\n\n**Gemma 4**\n\n- Gemma 4 12B Q4 GGUF (7.5 GB)\n- Gemma 4 26B 4BIT MLX (15.6 GB)\n- Gemma 4 31B 4BIT MLX (29 GB)\n\n## Runtime\n\nThe runtime handles model discovery, configuration, and loading. It also determines the\npractical question of how we integrate our harness with the model. Usually this happens by\nstarting a web server that provides a range of typical APIs that the harnesses support,\nand the `localhost`\n\nURL of that web server is then configured as a model\nprovider in the harness. Most widely supported by harnesses is the OpenAI API, but Claude\nCode e.g. expects [Anthropic's\nClaude API](https://platform.claude.com/docs/en/api/messages) .\n\n*Impacts:* Ease of configuration and discoverability; ease of integration with\nharnesses; speed of responses\n\nFigure 2: The “Developer” view in LM Studio, showing a running server and many of the factors described so far (provider url, model size, APIs, context window size configuration)\n\n*My experience:* While I have used other runtimes in the past, I am currently back\nusing [LM Studio](https://lmstudio.ai/), mainly for its user experience. There are lots of ins and outs of which runtime is the most optimised for which hardware and what type of models, to get even more speed out of it. But thinking back to the broader viability of running local models, user experience plays a huge role\nfor that. For what it's worth, the most frequently mentioned alternative from my colleagues was [oMLX](https://omlx.ai/).\n\n## Harness (Claude Code, OpenCode, Pi, ...)\n\nCoding harnesses can vary significantly in how much overhead they inject into the context window (system prompt, number of tools), which becomes more of a problem locally where we're so resource constrained. Our own expanded harness around that also makes a difference, e.g. how many skills or MCP servers are active. A description of each of them will be sent to the model, and again, take up space in the context window.\n\nI mentioned above that small models still struggle with tool calling - and it probably doesn't help that each harness has slightly different schemas for the basic tools. Let's take editing a file as an example:\n\n- Pi:\n`old_text`\n\nand`new_text`\n\n([see here](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/tools/edit.ts)) - OpenCode:\n`oldString`\n\nand`newString`\n\n([see here](https://github.com/anomalyco/opencode/blob/dev/packages/opencode/src/tool/edit.ts)) - Claude Code:\n`old_string`\n\nand`new_string`\n\n(at least that's what it says when I asked it)\n\nFinally, not all harnesses easily support the integration of local models. Open source\ntools are usually the go-to, but Claude Code can also be pointed at local providers. [GitHub\nCopilot seems to support it for their CLI](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/use-byok-models), and I think it's possible in Cursor as well\nto override the OpenAI base URL and point it to localhost.\n\n*Impacts:* Size of context window needed; Tool calling success; Integratability\n\n*My experience:* In my attempts, I used OpenCode and Pi. I avoided Claude Code as it\napparently [would burden the context\nwindow](https://portkey.ai/blog/the-harness-tax/) quite a bit.\n\n## Coming up\n\nIn my next memo, I will dive more deeply into the types of tasks I gave the models, and what I experienced.\n\nA preview of my overall conclusions: Using small models like this is still quite messy and hard to evaluate. It was a frustrating experience to come to conclusions, as the results depend on so many things. I would therefore say that it's still not ready for a simple “plug and play” experience for developers who don't want to spend too much time on it.\n\nHowever, based on this experience I do have a go-to model that I'm using locally now, which is Qwen3.6 35B MoE. It offered the best balance of capability, speed and RAM footprint among what I tried.\n\nMore to come!\n\n## Acknowledgments\n\nThanks in particular to Jigar Jani and Syed Atif Akhtar for their review and valuable inputs.\n\nGenAI (Claude and Claude Code) was used for research and polishing the language.\n\nlatest article (Jul 07):\n\nprevious article:", "url": "https://wpnews.pro/news/viability-of-local-models-for-coding", "canonical_source": "https://martinfowler.com/articles/exploring-gen-ai/local-models-for-coding-factors.html", "published_at": "2026-07-07 18:53:41+00:00", "updated_at": "2026-07-07 18:59:34.274126+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-agents", "developer-tools"], "entities": ["Thoughtworks", "Birgitta Böckeler", "Apple M3 Max", "Apple M5 Pro", "LM Studio", "OpenCode", "Pi"], "alternates": {"html": "https://wpnews.pro/news/viability-of-local-models-for-coding", "markdown": "https://wpnews.pro/news/viability-of-local-models-for-coding.md", "text": "https://wpnews.pro/news/viability-of-local-models-for-coding.txt", "jsonld": "https://wpnews.pro/news/viability-of-local-models-for-coding.jsonld"}}