This article was originally published on[BuildZn].
Everyone's running local LLMs now, which is great. But then they hit the wall: "Why does my 7B model on Ollama feel dumber than a cloud API?" You've got the tokens/second, but the quality sucks. Figured it out the hard way after pulling my hair out trying to get better local LLM quality improvement for agent tasks.
I’ve shipped FarahGPT to 5,100+ users and built multi-agent systems like NexusOS. I know what it takes to get an LLM to think, not just parrot. When I started building out a 9-agent YouTube automation pipeline locally, the raw output from quantized models was... dismal. Lots of factual errors, incoherent steps, total garbage. You'd think a Qwen 7B Q4_K_M model would at least manage basic reasoning, but default settings often choke it.
The problem isn't always the model itself or your hardware. It's how you talk to it and how you let it configure its own internal world. We’re pushing these models to run on consumer hardware, often with heavy quantization. Expecting them to perform like a 70B cloud model out of the box is naive. They need help to maximize their limited capacity. This is where specific modelfile
tweaks and prompt engineering for better local LLM reasoning come in.
Here’s the thing — most guides tell you to increase num_ctx
. Yeah, sure, more context is usually better. But it’s a blunt instrument. You’re missing the finer controls that genuinely improve how the model processes that context, especially for complex, multi-step agent operations. I don't get why most people stop at num_ctx
.
To actually fix this, you need to hit it from two angles: how you structure your prompts (the input) and how the model internally handles positional embeddings (its understanding of that input).
This isn't just "system, user" stuff. This is about giving the model a structured thinking process within the prompt, making it explicitly think step-by-step. I call it "context-stacking" because you're layering context and instructions in a way that guides its internal monologue. This significantly helps with quantized model prompt engineering.
Here's the pattern that works for me:
Example Prompt Structure (for a coding agent task):
SYSTEM: You are an expert Python developer assistant for building Flask APIs. Your goal is to generate a simple Flask REST API endpoint that returns JSON data.
Here are the strict rules you MUST follow:
- Only use standard Flask libraries. No third-party API frameworks unless explicitly requested.
- Ensure the output is a complete, runnable Flask application.
- Prioritize clear, concise code.
- Do NOT generate any setup or installation instructions.
- Do NOT include example `curl` commands.
- The endpoint must handle GET requests.
The desired endpoint should be `/api/users` and return a list of user dictionaries, each with `id` and `name`.
USER:
Generate the Flask API for `/api/users`.
AGENT THOUGHT PROCESS:
1. **Analyze Request:** The user wants a Flask GET endpoint at `/api/users` returning a list of user dictionaries.
2. **Required Imports:** Need `Flask` and `jsonify`.
3. **App Setup:** Instantiate Flask app.
4. **Route Definition:** Define `@app.route('/api/users', methods=['GET'])`.
5. **Data Structure:** Create a sample list of user dictionaries.
6. **Response:** Use `jsonify` to return the list.
7. **Main Block:** Add `if __name__ == '__main__': app.run(debug=True)`.
8. **Review Constraints:** Check all rules: standard Flask, complete app, concise, no setup/curl. Looks good.
FINAL RESPONSE:
python
from flask import Flask, jsonify
app = Flask(name)
users = [
{"id": 1, "name": "Alice"},
{"id": 2, "name": "Bob"},
{"id": 3, "name": "Charlie"}
]
@app.route('/api/users', methods=['GET'])
def get_users():
"""
Returns a list of all users.
responses:
200:
description: A list of users.
"""
return jsonify(users)
if name == 'main':
app.run(debug=True)
plaintext
See that AGENT THOUGHT PROCESS:
? That's not just for show. You tell the model to output that. It forces it to allocate tokens to internal reasoning before spitting out a FINAL RESPONSE:
. This dramatically improves task completion coherence.
Modelfile
Parameter Tweaks: RoPE Frequencies
This is where things get interesting and where most developers miss the mark. Forget just num_ctx
for a minute. The rope_freq_base
and rope_freq_scale
parameters in your Ollama Modelfile
are critical for how the model understands the position of tokens within its context window. Changing these can impact how well it discerns relationships between widely separated tokens. It's a key part of Ollama quality configuration.
Here’s the deal: many quantized models, especially smaller ones, struggle with long-range dependencies and complex reasoning because their default RoPE (Rotary Positional Embedding) settings might not be optimal for the reduced precision.
My Fix for Qwen 7B Q4_K_M:
I built a custom Modelfile
for qwen:7b-chat-q4_K_M
(downloaded from Ollama) and explicitly set these.
Here’s the Modelfile snippet:
FROM qwen:7b-chat-q4_K_M
PARAMETER num_ctx 4096
PARAMETER rope_freq_base 50000
PARAMETER rope_freq_scale 0.8
To use this, save it as Modelfile
in a directory, then run:
ollama create my-qwen-smart -f ./Modelfile
Then you can use ollama run my-qwen-smart
.
Why these values? Default RoPE settings are often optimized for the full-precision, non-quantized model. When you quantize, you introduce noise and lose precision. Tweaking rope_freq_base
and rope_freq_scale
can essentially "re-tune" the positional encoding to be more robust to this noise, helping the model better understand token relationships across the context. It's like re-calibrating its internal compass. This is a subtle but powerful lever for better local LLM reasoning.
The Numbers (Real Talk):
After combining the context-stacking prompt technique with these rope_freq_base
(set to 50000
from default 10000
) and rope_freq_scale
(set to 0.8
from default 1.0
) modelfile
parameters on my Qwen 7B Q4_K_M
model running via Ollama 0.1.29 on an RTX 4090 (with 16 layers loaded onto VRAM, hitting about 12.4 tok/s for generation after a full context prompt), I observed:
modelfile
with only num_ctx
increased.This isn't about raw speed (which remained consistent at ~12.4 tok/s when measuring over 100 runs for generating ~200 tokens). It's purely about output quality. The llama.cpp smart tips
aren't always about speed.
Initially, I just threw more num_ctx
at the problem and tried longer, more verbose prompts. That helped a bit, but often made the output more convoluted. The model would just fill up the extra context with verbose, but often irrelevant, fluff. It was like giving a confused person more books; they just get more overwhelmed.
Another mistake was blindly copying Modelfile
settings for different models. A rope_freq_base
that works for Llama 2 might completely screw up Mistral or Qwen. The rope_freq_base and rope_freq_scale values are highly model-architecture dependent. You
I also hit a weird behavior with Ollama 0.1.28 where repeated multi-turn conversations would sometimes drop the entire system
prompt context after 3-4 turns, leading to completely nonsensical replies, almost like it had amnesia. Upgrading to 0.1.29 resolved this, so keep your Ollama version updated, folks.
Even with these tweaks, local LLMs still aren't god-tier. The real secret is iterative refinement. After the initial output using the context-stacking and modelfile
tweaks, I often pipe that output back into the model with a "Critique and Refine" prompt.
SYSTEM: You are a meticulous code reviewer. Your task is to identify errors, suggest improvements for clarity, security, and efficiency, and then rewrite the provided code.
USER:
Critique the following Python Flask code. Focus on:
- Adherence to best practices.
- Potential security vulnerabilities.
- Readability and maintainability.
- Correctness of implementation.
CODE:
[Initial code generated by the agent]
AGENT THOUGHT PROCESS:
1. **Review Code:** Read through the Flask code provided.
2. **Check Best Practices:** Is it idiomatic Flask?
3. **Security Scan:** Look for common Flask vulnerabilities (e.g., debug mode in production, unsanitized input, no CSRF protection, if applicable).
4. **Clarity/Maintainability:** Are variable names clear? Is the structure logical? Add docstrings where missing.
5. **Correctness:** Does it actually solve the problem?
6. **Formulate Feedback:** Write a concise critique.
7. **Generate Refined Code:** Provide the improved version.
FINAL CRITIQUE:
...
REFINED CODE:
...
This multi-step approach, where one agent generates and another critiques, is a game-changer for getting genuinely useful output from local models. It mimics how humans collaborate and self-correct.
rope_freq_base
impact LLM quality?
rope_freq_base
directly influences how the model's positional embeddings are calculated. By adjusting it, you can change how quickly positional information "decays" across the sequence, potentially improving the model's ability to track long-range dependencies and token relationships within a large context, especially for quantized models where precision is reduced.
Modelfile
tweaks for any local LLM?
While the concept applies, the specific rope_freq_base
and rope_freq_scale
values are highly dependent on the model's architecture (e.g., Llama, Mistral, Qwen) and its quantization level. You'll need to experiment with different values for your specific model to find the optimal settings. Start with the defaults and make small, incremental changes.
num_ctx
and rope_freq_base
for local LLM performance tips?
num_ctx
simply expands the maximum length of the context window the model can process, allowing more tokens in. rope_freq_base
, on the other hand, tweaks how the model understands the position of those tokens within that context. While num_ctx
provides the capacity, rope_freq_base
refines the model's ability to interpret positional information, leading to better contextual understanding and reasoning quality, not just more tokens.
The default settings on Ollama are good starting points, but they're not optimized for every model or every use case, especially when you're pushing quantized models for complex reasoning. If your local LLM feels dumb, it's probably because you haven't given it the right tools to think. Combine intelligent prompt engineering with targeted modelfile
tweaks like rope_freq_base
and rope_freq_scale
. It's not a silver bullet, but it's the closest thing to a quality upgrade for your local setup that doesn't involve buying a new GPU. Get those models working smarter, not just faster.