Six months ago I wrote up my local AI setup and a reader bookmarked it, tried to reproduce it last week, and emailed me confused because half of it no longer matched what I actually run. Fair. Stacks rot quietly. So here's the mid-2026 state of mine: what's still earning its place on disk, what I deleted, and where I quietly went back to the cloud.
Context for the numbers and opinions below: I do smart contract security work, I run everything on WSL2 on a machine with a modest GPU, and I've been doing the local-model thing daily for over a year, not as a hobby but as part of shipping.
Ollama remains the center of the local stack and honestly it's not close. I've tried the alternatives, llama.cpp directly for control, a couple of the newer serving layers for speed, and I keep coming back for one boring reason: the API is stable and everything I've built talks to it. My audit tooling, my shell scripts, my editor config, they all point at localhost:11434
and they've pointed there for a year without breaking.
That stability matters more than a marginal tokens-per-second win. When a model update lands, ollama pull
and I'm done. The day something meaningfully better appears with the same API shape, I'll switch in an afternoon, which is exactly the position you want to be in.
I run two models and the split has stayed remarkably stable:
qwen2.5-coder:1.5b is the reflex model. It handles anything where speed matters more than depth: quick "what does this diff do" summaries, commit message drafts, pre-filtering files before a heavier pass, and the small classification jobs inside my pipelines ("does this file handle user input, yes or no"). It's fast enough on my machine that I never think about invoking it, and that's the whole point. A model you hesitate to call is a model you stop calling.
qwen2.5-coder:7b is the thinking model. Code review, security triage, structured findings extraction, explaining unfamiliar code. It's the smallest model I trust to follow a JSON schema and to hold an argument across a whole function. The first-pass local analysis in spectr-ai runs on it.
Have newer models tempted me? Constantly. I try most of the coder-tuned releases that fit my VRAM. A few scored better on paper. None have been enough better at my actual tasks to justify re-tuning the prompts and the output parsing that took months to get reliable. Switching costs are real, and "5% better in a benchmark" doesn't pay them.
This one hurt to admit. I spent a while running a local model as inline editor autocomplete, the ghost-text kind, and I turned it off around February and never missed it.
The problem wasn't quality, it was latency psychology. Autocomplete has to be near-instant or it interrupts more than it helps. On my hardware, with the models good enough to suggest something useful, completions arrived just late enough that I'd already started typing my own version. So I was paying GPU memory around the clock for suggestions I raced past. For autocomplete specifically, either you have hardware fast enough for real-time inference or you're better off without it. I write the code myself and save the models for reading it.
At one point I had a lot of models on disk. General chat models, an embedding model I benchmarked once, three abandoned fine-tune experiments, a vision model I used twice. Combined, a triple-digit number of gigabytes doing nothing.
The realization: every model I actually used was one of two. Everything else was collected, not used, the AI equivalent of unread books. I did a purge, kept my two, and made a rule that a new model gets thirty days to earn permanence or it gets ollama rm
. Disk space is cheap but attention isn't, and a zoo invites you to fiddle instead of work.
I built an embedding-based "chat with my repos" setup, and it was fun to build and useless to use. For code, grep and ripgrep with a bit of thought beat semantic search almost every time, because in code I usually know the literal string I'm hunting for. The RAG stack added an index to maintain and returned nearby-but-wrong chunks just often enough to erode trust. Deleted without regret.
Here's the honest part. I'm not local-only and I don't pretend to be.
Long context. When I need a model to hold an entire protocol in its head, multiple contracts, the interactions between them, that's beyond what my hardware runs well. Claude gets those jobs.
Agentic work. Multi-step tasks where the model plans, uses tools, reads results, and adjusts course. Small local models fall apart at step three, they lose the plot or loop. Anything agent-shaped goes to the cloud without a second thought.
Final-pass judgment. My security pipeline is a funnel: local models do the wide, cheap, private first pass over everything, and the highest-signal candidates go to a frontier model for verdicts. The local layer's job is to make the expensive layer's input small.
The pattern that emerged wasn't planned, but it's clean: local models read and filter, cloud models reason and decide.
Before I had local models in the loop, sending everything to cloud APIs during heavy audit work produced monthly bills that made me wince, the kind where you check the dashboard mid-month.
Now the local layer absorbs the bulk of raw token volume: every summarize, every classify, every first-pass review costs me electricity and nothing else. What's left for the cloud is a much smaller number of much harder calls, plus a subscription for interactive work. My total AI spend is a fraction of what the all-cloud version of my workflow would cost, and just as important, the cost is flat and predictable. I don't ration my own tooling anymore, and unrationed tooling gets used more, which compounds.
There's also a non-money line item: contract code under NDA never leaves my machine during the first pass. For security work, some clients care about that more than I do.
Runtime: Ollama on WSL2
Fast model: qwen2.5-coder:1.5b (summaries, classification, pre-filtering)
Deep model: qwen2.5-coder:7b (review, structured findings, triage)
Cloud: Claude (long context, agents, final judgment)
Dropped: local autocomplete, model zoo, local RAG
Rule: new models get 30 days to earn their disk space
If there's a theme to the last six months, it's subtraction. The stack got better by getting smaller, and every deletion made the remaining pieces easier to trust.
What have you dropped from your setup this year, and do you miss it?