Welcome back to Local AI Weekly. I promised a weekly, and then took a month to send the second one. So let's call the first one the pilot episode. From here, the show will run weekly 🤘
This issue leans heavily toward agents, because that's where most of the interesting local AI tooling is landing right now.
Before we see all that, let me thank Atomic Chat, who is supporting this edition of Local AI Weekly.
Atomic Chat lets you run open-source LLMs privately on your own device, fully offline. It has Turboquant built-in providing faster local inference with longer context windows. Which means you can run bigger models smoothly, right on your device.
🧪 On my bench: Unsloth #
I downloaded Unsloth this week. It's an open source framework for fine-tuning and running open models faster and on far less VRAM than the usual setup. It's probably best known for its Dynamic GGUF quants, the ones that let people run 27B-class models in around 17GB of RAM, with 1-bit builds that squeeze onto 8GB.
Yeah...that's the kind of claim I don't want to repeat until I've tested it myself. So consider this a heads-up, not a review or recommendation. I'll try it on my ZimaCube (with Nvidia RTX Ada 2000) and give you the full verdict in a coming issue.
🔍 Discover AI tools #
Three of this week's finds are about making your local agents "behave". tehy are smart, but they have to be accountable, too. All of these tools are open source.
The first is agent-inspect, a local-first debugger for TypeScript AI agents. It turns an agent run into a readable execution tree, so you can see every tool call, model call, and the exact point where things went wrong. It can also fail a CI check when the agent takes the wrong path, and bundle up a redacted trace to share with someone else.
Next is AutoMem, a persistent memory layer that plugs into agents over MCP. Most agents start every session from zero. AutoMem stores what matters in a graph for relationships and a vector index for meaning, so recall gives you the answer and the thread it belongs to. You can run the whole thing locally in Docker, offline, with your data staying on your machine.
The third is BrowserSkill from Tencent. It lets an AI agent use your real, already logged-in broswer instead of a fresh sandbox with throwaway accounts. The interesting part is that it runs tasks in a separate, visible Agent Window and only borrows a tab you have open when it explicitly asks, so you can keep working while it does. When it hits a captcha or a login, it hands control back to you and picks up after. Agents can still not tick all the traffic lights, it seems 😉
🖥 Which model can your hardware actually run? #
This is the question I get asked the most. You see a model you want to try, but you have no idea if it can run on your system. Down and trying all of them is a waste of time and bandwidth.
llmfit helps you there. It's a terminal tool that detects your CPU, RAM, and GPU, then ranks open models by how well they'll actually run on your box, scoring them on size, speed, quality, and context. If you have Ollama running, you can even pull a model straight from its interface. It is open source and written in Rust (if that matters to you).
🧩 Open WebUI goes mainstream #
A quick sign of the times. Open WebUI has become so popular that managed hosting providers are now offering it as a one-click service. PikaPods, for instance, now runs it for you.
Just so you know, I work with PikaPods as its growth consultant, so take the mention for what it is. But the point is that self-hosted AI front-ends have moved from hobbyists' Raspberry Pi to something people want running full time without managing a server.
🎙 Voice transcription lands on Linux #
Back in issue #1 I said voice AI is shaping up to be the next big thing. LM Studio's Bionic release, v1.1.3 proves my point, as it now brings voice transcription to Linux, including on machines with NVIDIA GPUs.
LM Studio is free but closed source, so keep that in mind. If you are on Linux, you will find that the Bionic AppImage is about 1 GB. That's the biggest AppImage I ever downloaded.
📡 Open Model News #
The open model space has kept up the pace since last time. Three releases stand out, and all three are MoE models, which is now clearly the direction the open world is heading.
Qwen3.8-Flash-Next from Alibaba's Qwen team. It's a 125B mixture-of-experts model that only activates 6B parameters per token, and it's an early preview of the architecture behind the upcoming Qwen4 family. Needs serious hardware power, though.
DeepSeek V4.1 Flash is here, too. On paper it's a huge 552B MoE, but it activates only 8B parameters on input and 16B on output, with a new architecture built around an aggressively compressed KV cache, roughly 890 bytes per token. Waiting for community quantization on this one.
Those two are heavyweight so the next one is "relatively affordable" by homelabbers. Ornith-1.5 from DeepReinforce comes in three sizes: a 397B MoE flagship, a 35B MoE that activates just 3B per token and runs on a single 24GB card, and a 9B dense model with a quantized build small enough for a phone. It's trained with a self-improvement loop, where the model writes its own tasks and grades its own attempts.
👀 Big Tech Watch #
Two things worth your attention.
First, OpenRouter is being acquired by Stripe. OpenRouter is the big model-routing gateway, moving over 10 trillion tokens a day across 400 or so models. It says the product, name, and neutral routing won't change. Worth watching, because a lot of the local and open tooling world leans on that gateway staying neutral.
Second, the industry had a rare moment of agreement. Anthropic's Dario Amodei published an essay called "We Must Pace the Frontier," arguing that AI labs should deliberately slow down how fast they push model capabilities.
Within a couple of days, Sam Altman, Elon Musk, and Google DeepMind's Demis Hassabis all said they agreed with the core idea. The notable holdouts weren't the labs at all. The US President Trump rejected the idea flatly, saying "whoever wins AI wins."
If the frontier slows and open weights keep improving, that's a good thing for us Local AI enthusiasts.
🗂 AI Jargon: Harness #
You'll have noticed the word "harness" turning up everywhere lately. Here's what it means.
Think of the LLM as the brain and the harness as the actual body. A model on its own is just a brain sitting in a jar. It reads text and predicts text. A harness is what turns it into an agent that can do things. The harness runs the loop i.e. it sends your request to the model, reads back the tool calls the model wants to make, actually runs those tools, feeds the results back in, and repeats until the job is done. It also manages memory, context, and what the model is and isn't allowed to touch.
The reason everyone is building their own harness is that the model has become the easy part. The value now sits in the harness you wrap around it: your data, your tools, your workflows, your guardrails.
Using an LLM as a glorified chat tool won't help much. Plugging the LLM into a harness tuned to your work or your company's work will let you achieve the actual AI automation.
⚡ Quick Tip: Keep your Ollama models warm #
If you run Ollama and notice the first answer after a break is slow, here's why. By default, Ollama unloads a model from GPU memory after 5 minutes of no requests. I noticed this on my Zima setup. The next request then has to reload the whole model from disk into VRAM, which will delay the next response.
You can change that. Set the OLLAMA_KEEP_ALIVE environment variable on the Ollama service to a longer duration like 30m, or to -1 to keep the model pinned in memory until you restart.
My point is that free GPU memory is only useful if you're going to use it. On a single-user machine where you're the only one using the model, there is little point in freeing up the GPU every five minutes and getting slow responses as a result. However, if you use several large models on one card, the default timeout actually helps.
🎫 Events worth attending #
If agents are your thing, there are two conferences coming up, and It's FOSS readers get a discount on both.
AGNTCon + MCPCon Europe is happening on September 17 and 18 in Amsterdam. Use the code ITSFOSS_50 at checkout, which brings registration to $375. It's valid right up to the event date, which is tomorrow.
AGNTCon + MCPCon North America on October 22 to 23 in San Jose, California. Use the code ITSFOSS for 20% or more off current pricing; it should be around $375 again. This offer expires on October 7.
In the end... #
I hope it was not too long. Not that I was trying to compensate for the long gap between the two editions...or perhaps I was ;)
As always, some of this will be useful to you and some won't, depending on what you're building or what your interests are. Reply and tell me what you're running locally these days. I read every response.
See you next week (for real this time).
If this helped you, consider supporting It's FOSS
It's FOSS has been helping people use Linux for the past 14 years. Help us stay independent from big tech. Become a Plus member, enjoy ad-free reading and get 5 eBooks.