{"slug": "how-to-get-better-results-from-local-llms-with-ollama", "title": "How to get better results from local LLMs with Ollama", "summary": "Local LLMs have improved sharply, with Simon P. Couch, senior software engineer at Posit, reporting that the April Qwen 3.5 and Gemma 4 releases each scored 90% on his agentic coding eval, up from 0% for any MacBook-runnable model a few months earlier. Open-source developer Simon Willison said in his PyCon US 2026 lightning talk in May that \"laptop-available models, while a lot weaker than the frontier, have started wildly outperforming expectations.\" The guide recommends Ollama for running models locally, noting Google's 4-bit Gemma 4 12B needs 6.7 GB of VRAM or Mac unified memory and Meta's 30B Muse Glimmer, unveiled in August, needs around 24 GB of VRAM.", "body_md": "If you like the idea of running an LLM on your own computer but tried awhile ago and were disappointed, it may be time to give it another chance.\n\n“A few months ago, any LLM that I could run on my Macbook scored 0% on an agentic coding eval I put together,” Simon P. Couch, senior software engineer at Posit, [posted on Bluesky](https://bsky.app/profile/simonpcouch.com/post/3mjmkzey5pk2z) this spring. “[The April] Qwen 3.5 and Gemma 4 releases both scored 90%.”\n\nA model running on your laptop still won’t come close to what a state-of-the-art LLM from Anthropic or OpenAI can do in the cloud. But for defined tasks like answering coding questions, writing functions, or summarizing documents, they can be surprisingly capable.\n\n“Laptop-available models, while a lot weaker than the frontier, have started wildly outperforming expectations,” open-source developer Simon Willison, who follows the AI industry closely, said in his [PyCon US 2026 lightning talk](https://simonwillison.net/2026/may/19/5-minute-llms/) in May. \n\nThere are many ways to run local models on a PC or Mac. [Ollama](https://ollama.com/), while perhaps not the fastest, is among the most popular and easy to set up. It’s also supported out of the box by many mainstream programming tools such as [Visual Studio Code](https://docs.ollama.com/integrations/vscode), [JetBrains AI Assistant](https://www.jetbrains.com/help/ai-assistant/use-custom-models.html), [Zed](https://zed.dev/docs/ai/use-a-local-model), and [Posit Assistant](https://assistant.posit.co/docs/getting-started/providers/). Ollama also can launch Claude Code or Codex with the option to use a local LLM.\n\nI’ll be focusing on Ollama here, but many other tools are available for running LLMs locally, such as [LM Studio](https://lmstudio.ai/), [Jan](https://www.jan.ai/), [Unsloth](https://unsloth.ai/docs), Simon Willison’s [LLM](https://llm.datasette.io/en/stable/), and [llama.cpp](https://llama.app/). \n\nYou can [download Ollama](https://ollama.com/download) and install it as a conventional software program for Windows, Mac, and Linux. There’s both a chat application with a GUI and a terminal CLI. \n\nFrom choosing a model and setting its parameter options to tweaking its runtime configuration, here are some tips for getting the most from your Ollama setup.\n\nChoosing a model is obvious but not always easy, given that the [Ollama model catalog](https://ollama.com/search) has hundreds of listings. What’s best for you depends on both your task and available hardware. For coding, the Gemma 4 family from Google and the latest Qwen models from Alibaba are both worth considering. I’m partial to Gemma, because it writes R code well. In fact, Gemma 4 26B in the cloud is the only open-weight LLM that Posit offers in its Posit AI subscription service that might fit on a laptop. I also lean toward US models for summarizing text, because they’re more likely to be trained in nuances of American English.\n\nNext is selecting a size. The Gemma 4 12B version introduced in June fits comfortably in a lot of systems. Google says a 4-bit version (the Ollama catalog default) requires 6.7 GB of VRAM or Mac unified memory. And, Google’s internal coding benchmarks show it’s close to the larger 26B MoE (mixture of experts) Gemma 4 version from April in some tasks. It’s a decent place to start if you don’t have a ton of VRAM.\n\nAnother possible option: Meta recently announced Muse Glimmer, a 30B model unveiled in August, which is designed for end-to-end agentic work, reliable tool use, and multi-step reasoning. It needs around 24 GB of VRAM to run at a usable clip, though.\n\nIf none of these suggested models work for you, one of the easiest things to do these days is ask a frontier model such as ChatGPT or Claude. Tell it how much GPU VRAM and regular RAM your Windows PC has, or how much unified RAM your Mac has, what LLM family you’re interested in running if you’ve got a favorite, and what you want to do. Summarize short documents? Long documents? Answer coding questions in a terminal? Run a coding agent? Ask for its three top suggestions and start with those. Then try larger or smaller options as appropriate.\n\nFor Windows PCs, if an LLM can’t fit entirely into GPU VRAM, it may run so slowly as to be effectively unusable.\n\nNote, though, that an LLM’s architecture matters along with its size. A so-called mixture-of-experts model like Gemma 26B only uses *some* of its parameters for a task. While the whole LLM needs to be loaded, it may run fine on a PC with less GPU memory than you’d expect—depending on what you’re doing. Gemma 4 26B is pretty snappy for some tasks in a terminal window on my PC with 12 GB of VRAM, even though more than half of the model spills over into RAM.\n\nAnd you don’t always need the largest model that fits in your hardware; smaller ones may be fine, especially for simple, non-mission-critical tasks.\n\nThe takeaway? Selecting a good model is a bit more complicated than “this hardware can run LLMs with that number of parameters, and bigger is always what I want.”\n\nOne more note: If you have a Mac with Apple silicon, choosing an MLX version of an LLM if available will likely give you better performance. [MLX](https://ml-explore.github.io/mlx/build/html/index.html) is a framework that’s optimized for Apple hardware.\n\nOnce you decide on an LLM, the Ollama model listing should show you how to download and run it. For example, the terminal command for Gemma 4 12B is `ollama run gemma4:12b` or `ollama run gemma4:12b-mlx`. If the model is already on your system, the `ollama run` command just loads and launches it; if the LLM isn’t on your system yet, Ollama downloads it first. The `ollama pull` command will download a model without running it. Use `ollama list` to list all the LLMs Ollama has already downloaded on your system. And use `ollama rm [model-name]` to delete a model.\n\nThere are ways to tweak an LLM’s performance with parameter options such as temperature (how “random” its responses are), top_k (if its choices should be restricted to a certain number of best-matching tokens), top_p (how many next tokens the model considers when generating its response), and maximum context window size.\n\nOllama suggests 64000 tokens as a minimum context window size for web search, agents, and coding tools. However, the Ollama default is just 4000 for systems with less than 24 GB of VRAM, even if an LLM has a much larger possible context window.\n\nInstead of specifying parameters on the command line each time you run a specific LLM, you can create a “copy” of the model with your desired parameters using a [Modelfile](https://docs.ollama.com/modelfile). Below is an example Modelfile that creates a version of the Gemma 4 12B LLM with some suggested parameters and a context size of 64000 tokens:\n\n```\nFROM gemma4:12b\n\nPARAMETER temperature 1\nPARAMETER top_p 0.95\nPARAMETER top_k 64\n# For the context window\nPARAMETER num_ctx 64000\n```\n\nGoogle advises setting temperature to 1.0, top_p to 0.95, and top_k to 64 for all the Gemma 4 LLMs. (That’s already what the default Ollama versions are, but it doesn’t hurt to specify again.)\n\nYou can also set a default [system prompt](https://docs.ollama.com/modelfile#system) in a Modelfile with `SYSTEM` instruction. \n\nSave your Modelfile as a plain text file, such as Modelfile-gemma4-12b-64k-optimized.txt, and then create the LLM copy with a terminal command such as\n\n```\nollama create gemma4-12b-64k-optimized -f ./ModelFile-gemma4-12b-64k-optimized.txt\n```\n\nNow you can run your new LLM with\n\n```\nollama run gemma4-12b-64k-optimized\n```\n\nThe `ollama list` command will show your newly created copy as having the same size as the original, but it’s not actually taking up all that additional space on your hard drive. `ollama show [model-name]` will display a number of settings that have been baked in.\n\nOllama defaults to unloading a model out of memory after just five minutes of session inactivity. That may be helpful if you’ve ended your session, but not so great if you want to resume a chat that’s been idle for seven minutes. Having to re-load an LLM into an existing session can make a model feel a *lot* more sluggish than it should.\n\nTo change how long an LLM stays in memory, update your system’s `OLLAMA_KEEP_ALIVE` environment variable. On Windows, you can change the default in a terminal window with `setx OLLAMA_KEEP_ALIVE \"15m\"` or `setx OLLAMA_KEEP_ALIVE 900` (a number without a qualifier such as m for minutes is treated as seconds). \n\nOn a Mac, you can change the default with `launchctl OLLAMA_KEEP_ALIVE 15m`. \n\nRestart Ollama for new settings to take effect.\n\nYou can unload a model manually with the terminal command `ollama stop [model-name]`. To see how much time is left on a running LLM if there’s no additional session activity, use the `ollama ps` command. Note that the results of this command include an “UNTIL” column.\n\nFoundry\n\nAnother important Ollama option is the default context window. If you’re not using a Modelfile-customized LLM, you can set Ollama’s default context window for any LLM you run with `setx OLLAMA_CONTEXT_LENGTH 64000` on Windows or `launchctl OLLAMA_CONTEXT_LENGTH 64000` on a Mac. (If the LLM has a smaller maximum than this setting, Ollama should ignore it.)\n\nIf you’re on a PC, you may also want to change Ollama’s default for KV cache, which controls how a model stores chat history in your current session. The default is `f16` quantization, which gives high precision—but with high memory use. Changing to `q8_0` (8-bit quantization) “uses approximately 1/2 the memory of `f16` with a very small loss in precision, this usually has no noticeable impact on the model’s quality,” according to [Ollama’s documentation](https://docs.ollama.com/faq#how-can-i-set-the-quantization-type-for-the-k/v-cache). Yes please!\n\n```\nsetx OLLAMA_KV_CACHE_TYPE \"q8_0\"\n```\n\nThis usually works with NVIDIA GPUs that should auto-enable a memory optimization called Flash Attention. For other setups, Ollama docs suggest force-enabling Flash Attention by setting `OLLAMA_FLASH_ATTENTION` to 1 in an environment variable. If you’re using an MLX model on a Mac, you probably don’t need to worry about this.\n\nTo check which Ollama variables you’ve set on your PC, you can run these terminal commands:\n\n```\nreg query HKCU\\Environment | findstr /i OLLAMA\nreg query \"HKLM\\SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Environment\" | findstr /i OLLAMA\n```\n\nRunning `ollama ps` in a new terminal window while Ollama is running an LLM elsewhere also shows you how much of a currently running model is in your GPU. If your LLM is slow but it all fits in your VRAM, the problem may be elsewhere.\n\nGemma 4 is a “thinking” model family, which means its models can go through what look like reasoning steps before generating their final response. This “thought process” can improve results but also slow things down—*a lot.*\n\nThinking is on by default in Ollama for all LLMs that have the option. You can turn it off from the Ollama command line with `--think=false`. For example: \n\n```\nollama run gemma4:12b --think=false\n```\n\nOther applications like IDEs may have their own ways of switching thinking on and off, such as a lightbulb icon in the chat window. Unfortunately, you can’t turn thinking off from a Modelfile.\n\nIn a test summarizing documents with Gemma 4 E4B, turning thinking off boosted speed five-fold and result quality was about the same. For Gemma 4 12B, the speed improvement was eight-fold.\n\nHow thinking affects coding will depend on the complexity of the task. You may want to test some real-world work to see whether time spent “thinking” pays off with better results.\n\nOne of the largest sources of open-source LLMs is Hugging Face, and many models there have a “use this model” button that gives the command you need to download and run it on Ollama. Just be aware that not all of those models are as plug-and-play as the Ollama-optimized versions on Ollama.com, especially new ones.\n\nFor example, I tried running the Unsloth project’s 2-bit quantized version of Meta’s new Muse Glimmer. The command was\n\n```\nollama run hf.co/unsloth/Muse-Glimmer-30B-GGUF:UD-Q2_K_XL\n```\n\nHowever, this errored out of the box. ChatGPT helped me fix the issue (with a Modelfile to create a modified version of the LLM) after I told it the error messages.\n\nLocal LLMs have come a long way, but a model that can fit inside a consumer desktop or laptop won’t be able to do the kind of work that top cloud-based models can. Gemma 4 12B may write good basic functions and document summaries, but don’t expect it to do complicated jobs from a single prompt like “create an R package to use the National Weather Service API.” As with frontier LLMs of old, you’ll need to go step by step, be specific, and have some patience. If you never want to go back to those days, stay with the state-of-the-art models in the cloud.\n\nGemma works well for basic data analysis, “importing, tidying, and visualizing data,” and “information retrieval and short factual questions,” according to [Posit AI’s documentation](https://docs.posit.co/posit-ai/user/models/). “It is less reliable than the Claude models on longer, multi-step tasks. You might see a strong response on one turn and a misread on the next. We do not recommend Gemma 4 for long-running agentic coding or complex package development.”\n\nThat was pretty much the status of paid cloud-based LLMs a couple of years ago.\n\nEven if limited compared to current state-of-the-art models, though, free coding help that’s 100% local and private can still be compelling for use cases like exploring sensitive data, summarizing personal documents, doing simple tasks, working on a plane with bad Internet, and saving paid tokens for when they matter. Plus it can feel empowering to have everything on your own system.\n\nAnd, local model capabilities will likely keep getting better.", "url": "https://wpnews.pro/news/how-to-get-better-results-from-local-llms-with-ollama", "canonical_source": "https://www.infoworld.com/article/4218328/how-to-get-better-results-from-local-llms-with-ollama.html", "published_at": "2026-09-15 09:00:00+00:00", "updated_at": "2026-09-15 09:10:21.816926+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "ai-products"], "entities": ["Ollama", "Simon P. Couch", "Posit", "Simon Willison", "Qwen 3.5", "Gemma 4", "Google", "Meta"], "alternates": {"html": "https://wpnews.pro/news/how-to-get-better-results-from-local-llms-with-ollama", "markdown": "https://wpnews.pro/news/how-to-get-better-results-from-local-llms-with-ollama.md", "text": "https://wpnews.pro/news/how-to-get-better-results-from-local-llms-with-ollama.txt", "jsonld": "https://wpnews.pro/news/how-to-get-better-results-from-local-llms-with-ollama.jsonld"}}