If you've tried running local LLMs and felt like they were dumber than they should be... welcome.
This blog is about what breaks when you build with local AI, Claude Code, and Ollama — and what I do to try and fix it.
Any advice is more than appreciated. Please feel free to leave any questions, comments, or any issues that you may be currently facing.
I come from finance, not CS. I've been shipping software with Claude Code and Ollama for the past 9 months, mostly by learning the hard way, noting what breaks, and working with Claude Code to fix it. This is my first blog post, so please bear with me. The point of my blog is to turn those lessons into something useful for other people in the field — and maybe get some help along the way.
It's called Qwellar.
It's a desktop app I've been building that treats a local AI as a long-term presence — persistent memory, chat with local models via Ollama, a workspace designed for working with AI over time. It's genuinely a learning exercise and a side project. It shows up in the debug stories because I keep running into these problems while building it, and I figure other people doing similar things will hit the same ones.
No hot takes. No LinkedIn-style motivation posts. No AI-generated listicles about "The 10 Best Prompts for ChatGPT." No clickbait. Just information, observations, and hypotheses I wish someone had written when I started.
If you catch me wrong — tell me. If you've figured out something I haven't — please share. I'm building this as a learning space, not a broadcast channel. The comments are where the real value happens half the time.
Ollama is a program that runs AI models on your own computer, on your own hardware. Not in the cloud. If you were like me and downloaded a model like llama3.2:3b (that's Meta's Llama 3.2, 3 billion parameters), Ollama gives it an API on localhost:11434. The number is just the port Ollama listens on by default. My project sends prompts to that API and gets responses back. No OpenAI. No Anthropic. No internet needed.
That's the whole promise of "local AI" — the model lives on your machine, so your chats never leave.
What I refer to as a "small model" is really just a model small enough to run on a normal laptop. Llama 3.2 3B fits in ~2GB of RAM. Frontier cloud models like GPT-4 or the Opus 4 series (Anthropic) are reportedly in the hundreds of billions to over a trillion parameters. Llama 3.2 3B is tiny compared to them.
Tiny brains are more sensitive to how you talk to them.
The Ollama API is just HTTP. When my project talks to Ollama, it's just sending a POST request to http://localhost:11434/api/chat with a JSON body. Same as calling any web API.
Here's the minimal version — just a prompt, no options:
POST http://localhost:11434/api/chat
{
"model": "llama3.2:3b",
"messages": [
{ "role": "user", "content": "hey what's up" }
]
}
Ollama receives this, runs the model, streams back the response. Done.
But Ollama lets you add an options object to that same JSON body. This is called "passing options." Same request, but now you're telling Ollama HOW to run the model:
POST http://localhost:11434/api/chat
{
"model": "llama3.2:3b",
"messages": [
{ "role": "user", "content": "hey what's up" }
],
"options": {
"temperature": 0.15,
"num_ctx": 16384
}
}
Three fields total: model, messages, and options. Each entry inside options is one dial — temperature, num_ctx, top_p, top_k, repeat_penalty, seed, num_predict, stop. I'll focus on the two that mattered most.
Now, Ollama runs the same model, on the same prompt, but with my dials instead of its defaults. If you leave options out entirely, Ollama falls back to its defaults (temperature 0.8, num_ctx 2048). That's the trap. Silence equals defaults. Most people never even know the field exists.
Temperature controls how "creative" the model is when picking each next word.
Ollama ships with temperature = 0.8 by default.
Why does that matter? Because users want their AI to do things, not just reply. For my project, I want it to pop up a confirmation card, trigger a search, start a new chat context, and so on. The problem: the model only knows how to output text. It can't press buttons.
So I taught the model code words. In my system prompt, I tell it: "if the user asks to start over or wipe the chat, don't just say 'okay' — output this exact text: <confirm_start_fresh> and then stop." Qwellar's code is watching the output, sees the tag, and shows a confirmation card. I never actually see <confirm_start_fresh> on my screen — my code strips it before rendering. The tag is a signal from the model to my app.
At temperature 0.8, the model would go "eh, close enough" and write <confirm-start-fresh> (dashes instead of underscores) or <confirmStartFresh> (camelCase) or skip the tag entirely and paraphrase. My parser would fail. The feature would appear broken.
At 0.15, the model gets boring and reliable. It emits the tag exactly how I taught it.
This is how many tokens the model can "see" at once. A token is roughly 3/4 of a word.
Ollama ships with num_ctx = 2048 by default. 2048 tokens is ~1500 words, maybe 10 turns of chat.
Here's the killer: Ollama does not tell you when you exceed it. If your conversation is 3000 tokens and num_ctx is 2048, Ollama silently chops off the beginning. The model literally cannot see how the chat started. You ask "what was the first thing I said?" and it makes something up, because the first message is gone.
Users don't see this. There's no error. The AI just gets weirdly amnesic 10 turns in.
Changed the temperature default from 0.8 → 0.15. Instantly, small models stopped screwing up structured output. <confirm_start_fresh> started firing right. JSON handoffs stopped mangling.
Changed num_ctx from 2048 → 16384. 8× more context. The model can see ~80 turns before running out. Full conversations, not amnesic snippets.
Fixed a bug where the options weren't even reaching Ollama. Turns out electron/main.js was dropping the options object on the floor when it forwarded IPC calls to Ollama. So even if I set temperature = 0.15 in the app, Ollama got nothing and used 0.8 anyway. This is the kind of bug you only find when you actually test. There's a reason this bug stayed hidden and I'll explain that in a bit.
Hardened the prompts. Small models take instructions literally and skip nuance. So instead of "please output a JSON handoff," I write "output ONLY this JSON. Do not include any text before or after. Do not include markdown code fences." I add negative examples; I show the model actual bad output that I've seen it produce and label it "do NOT do this." Giving the model something concrete to steer away from works almost as well as showing it what to steer toward. Big models figure it out. Small models need the rails. I'll be honest: I don't love this. Hardening a prompt means I'm paying maintenance cost on English instead of code. Every new feature means more prompt rules. It works, but it's not where I want to end up.
Made the parser more forgiving. Sometimes the model still messes up the output slightly (an extra space, extra text after the tag, or wrapping the whole response in markdown formatting it wasn't supposed to use). My parser now handles those small mistakes instead of failing (tolerates them instead of throwing). This is defense-in-depth — I'm not counting on the model being perfect; I'm counting on my parser being tough enough to catch what the model gets wrong.
Back to the IPC bug for a second. The reason it stayed hidden so long comes down to what "200" actually means.
HTTP responses come with a status code — a number between 100 and 599 that the server sends back with every response. It's a one-number summary of "did this work?"
The common ones:
When a developer tests their code, the lazy version of testing is: "I sent a request to the API. Did it return 200? Yes? Ship it."
That's the trap. The Ollama bug I mentioned returned 200 every single time. The HTTP request succeeded. Ollama got the message. Ollama ran the model. Ollama sent back a response. Code 200. All green. But the response was bad — because the options field never arrived. The model was running at the wrong temperature. The output was mangled. I kept seeing broken behavior. But the API said "200 OK" the whole way through.
So "does the API return 200" is shorthand for the shallow version of testing. The pipes were connected. The water was wrong.
99% of people running Ollama right now are using the stock defaults and don't know it. They try llama3.2:3b, get bad results, and conclude "small models suck." They blame the model. But the model isn't actually as bad as people think. The defaults were sabotaging it.
This is the mechanical part. Four small changes across three files. The code might look dense if you don't write code but it's actually pretty simple once you see it.
Remember from earlier: my project sends a JSON body to Ollama that looks like:
{
"model": "llama3.2:3b",
"messages": [...],
"options": { "temperature": 0.15, "num_ctx": 16384 }
}
The question is: where does that options object come from?
In the file where my project builds the request to Ollama (packages/ai-engine/src/ollama-client.ts), I added:
const BASELINE_MODEL_OPTIONS = {
temperature: 0.15,
num_ctx: 16384
};
That's just a named object. One place in the codebase that says "these are my defaults."
The function that calls Ollama (streamOllamaChat) was changed to merge those baseline options into every request body before sending:
const options = {
...BASELINE_MODEL_OPTIONS, // my defaults first
...callerProvidedOptions // caller can override if needed
};
fetch("http://localhost:11434/api/chat", {
method: "POST",
body: JSON.stringify({ model, messages, options })
});
The spread operator (...) means "copy all these fields in." So options always has temperature 0.15 and num_ctx 16384 unless the caller specifically overrides them.
In the file that forwards UI requests from Qwellar over to Ollama — the Electron main process (apps/electron/electron/main.js) — the handler was missing the options field. One line fix.
Before:
callOllama({ model, messages }) // options silently dropped
After:
callOllama({ model, messages, options }) // options actually forwarded
That's it. Three places touched. The whole thing is maybe 20 lines of code across the codebase.
After the baseline shipped, I added a Settings UI with dropdowns:
Those UI selections save to localStorage, flow through a React hook, and get merged into the options object for every chat message. Users who want creative writing can bump temperature. Users on tiny laptops can lower num_ctx to save RAM. The baseline is the default; the Settings are the override.
I didn't need a CS degree to find this. I needed to:
That's the AI-augmented builder loop: observe → dig → find the real root cause → fix → verify. Claude Code is the pair programmer. I'm the one asking "wait, why is this actually broken?"