Running the WUIC assistant on a local LLM: Ollama, an MCP server, and a free agentic VS Code

The WUIC framework team replaced the cloud-based generation half of its RAG chatbot with a local LLM running via Ollama, cutting per-token costs and privacy exposure. A side effect emerged: exposing the same WUIC knowledge to VS Code through an MCP server yields a free agentic coding assistant that knows the framework without an API key. The setup uses qwen2.5-coder:32b on a GPU box in the LAN, with retrieval remaining local via ONNX.

Our in-product RAG chatbot https://wuic-framework.com/blog/rag-chatbot-with-claude-and-bge-m3 has two halves. The first is retrieval — hybrid BM25 + bge-m3 + a fine-tuned reranker, running locally as native .NET/ONNX, with zero API surface to anyone. The second is generation — turning the retrieved chunks into an answer, or into a proposed metadata change https://wuic-framework.com/blog/rag-chatbot-tool-use-framework-integration . That second half talked to a cloud API with a per-token bill. This post is about replacing that second half with a local model , and about a side effect that turned out to be more useful than the original goal: the same WUIC knowledge, exposed to VS Code, gives you a free agentic coding assistant that actually knows the framework — no API key, no per-token cost. That side effect later became a product of its own — see the update at the end. It is also an honest post. A local LLM is not "free" in the sense people usually mean. You trade money and privacy gains for quality and latency losses. The last section is the bill. The retrieval half did not move — it was already local. Only the brain changed. bash BEFORE retrieval local ONNX → Claude API $ per token, cloud round-trip AFTER retrieval local ONNX → Ollama / qwen2.5 $0, on a GPU box in the LAN The two are not the same model class, and pretending otherwise would be dishonest — more on that below. But the wiring is clean, because the engine already spoke an OpenAI-compatible dialect. Pointing it at a local server is configuration, not surgery: rag-llm-provider = ollama OpenAI-compatible wire format rag-llm-base-url = http://<gpu-host :11434/v1 Ollama's OpenAI endpoint rag-llm-api-key = ollama dummy, non-empty rag-llm-default-chat-model = qwen2.5-coder:32b Three boxes. Your machine runs the WUIC backend and the local retrieval. A second machine with a GPU runs Ollama it can be the same machine if it has the VRAM . VS Code talks to both through a small bridge. Your machine GPU box same LAN ┌──────────────────────────────┐ ┌──────────────────────────────┐ │ WUIC backend :5000 │ │ Ollama :11434 │ │ ├─ RAG retrieval ONNX │ │ qwen2.5-coder:32b │ │ │ BM25 + bge-m3 + rerank │ ───► │ generation + tool-calling │ │ └─ /api/Rag/Query /Rag/Chat│ └──────────────────────────────┘ └──────────────────────────────┘ ▲ │ stdio MCP server "wuic-rag" │ ┌──────────────────────────────┐ │ VS Code │ │ WUIC Assistant ───────► │ agentic chat: edit files, run commands │ brain = same Ollama │ knowledge = WUIC RAG via the MCP server └──────────────────────────────┘ The arrow from the backend to Ollama is the RAG chatbot's new generation step. The arrow from VS Code back to the backend is the new part: an MCP server that lets any agentic editor query the same WUIC retrieval. MCP https://modelcontextprotocol.io Model Context Protocol is the open standard for giving an LLM client tools . Cline, Continue, Cursor and Claude Code all speak it. So instead of building a bespoke VS Code extension at the time , we shipped a tiny stdio MCP server that wraps the REST API the backend already exposes: You in the editor : "where is the cookie login handled?" │ ▼ The agent local model decides to call a tool │ tools/call wuic codebase search ▼ wuic-rag-mcp.mjs ──POST /api/Rag/Query──► RAG engine ──► top-8 chunks file:line + snippet │ ▼ The agent reads the chunks, answers with real citations — or starts editing It launched with two tools: wuic codebase search retrieval — file paths and snippets and wuic ask a full RAG answer generated by the local model . It is zero-dependency — Node's built-in http , no npm install , runs on any Node ≥ 14.18 — and stateless: every call just proxies to the backend. That matters for distribution: it ships inside every WUIC app and is wired up automatically at first run, so an end developer gets a .mcp.json already pointing at it. The crucial design point of that first iteration: the MCP server is not the agent. It adds knowledge , not autonomy. The editing, the command execution, the plan/act loop belonged to the host editor; reusing a mature open-source harness was the entire point. That is the part that changed — see the update. Retrieval-only worked on day one. The interesting failure was the action path — the chatbot's ability to emit a structured tool call e.g. "add a row-style rule with this JS condition" rather than prose. Cloud models return tool calls in a structured field. Our local serving stack, under the hood, often did not : it put the tool call into the message content as raw text, in inconsistent shapes — sometimes valid JSON without the envelope, sometimes a JavaScript object literal with single quotes and key=value instead of key: value . The model knew what to do; the plumbing dropped it on the floor. The fix was a tolerant parser on our side: detect a known tool name in the content, capture the following {...} block, and normalize that JS-ish literal into JSON — quote-aware, so embedded callback bodies with their own braces and quotes survive. One contained change took routing from unusable to good. The lesson is general: with local models, budget for the serving quirks, not just the model. The weights are capable; the adapter around them is where the work is. We benchmarked against our real tool-routing suite 89 prompt variants, 3 repetitions each, on a 24 GB GPU : | Model | Routing pass-rate | Latency / call | Notes | |---|---|---|---| | qwen2.5-coder:32b | ~91% | ~8 s | best schema fidelity + code/JS generation | | gpt-oss:20b | lower | ~2.6 s | much faster, weaker on complex tool schemas | The residual failures, when we dug in, were mostly not the model: a couple were genuinely correct behavior the single-turn test could not credit the model asked a clarifying question instead of guessing — which the real multi-turn product handles fine , and one was a genuinely hard nested-JSON case. The honest count of real model-quality misses was ~2 of 89. Context window matters too. A cloud model gives you 200k tokens. A 32B model on 24 GB of VRAM, after the weights, leaves room for ~24k tokens of KV cache with q8 cache + flash attention . Plenty for a chat turn — but it changes how aggressively you summarize a long conversation. This is the part most "run an LLM locally " posts skip. What you gain What you pay The honest recommendation is hybrid. Use the local model for the bulk — Q&A over the codebase, scoped edits, the always-on assistant that costs nothing. Keep a frontier model in your pocket for the genuinely hard multi-file work where its reasoning earns the per-token price. Replacing the API key everywhere is a slogan; replacing it where the quality gap does not bite is an architecture. Because it ships in the workspace template, a WUIC app is agentic-ready out of the box: .vsix from .mcp.json is generated at first run, already pointing at the wuic-rag server — any MCP-speaking client picks it up automatically. .clinerules and .cursorrules are generated from the same single source as the agent rules AGENTS.md . WUIC Assistant reads .clinerules itself — that is how it inherits the framework conventions and the nudge to search the codebase before inventing. The same files are also why you are not locked into our extension: point Cline or Cursor at the project and they get identical grounding, with the MCP server covering retrieval. scripts/mcp/ carries the server, a runbook, and an OLLAMA-SETUP.md that walks you through standing up your own Ollama box LAN exposure, num ctx tuning, model pull, persistent startup on Windows or Linux.You install WUIC Assistant, point it at your Ollama host, and you have a WUIC-aware agentic assistant in your editor for the price of the electricity. Local LLMs are not magic and they are not free. But for "an assistant that knows our framework, runs all day, and never sends our code anywhere" — they are exactly the right tool, and the trade-offs are ones you can plan around once you see them written down. A month of running this in anger changed most of the numbers above — for the better. Same architecture, same honest accounting. The model is now a Mixture-of-Experts, fine-tuned on our own mistakes. The dense qwen2.5-coder:32b gave way to qwen3-coder:30b — a MoE with ~3B active parameters that generates at ~197 tokens/s on the same 24 GB GPU the dense 32B did ~30 . Then we DPO-trained it on the assistant's own failure traces https://wuic-framework.com/blog/teaching-a-coding-agent-with-dpo , and that fine-tune is now the default brain for the in-product chatbot too. On the routing suite — grown from 89 to 102 variants — the base MoE scores 88/102; the DPO fine-tune scores 96/102 at identical speed . The ~8 s/call in the table above is now ~2–7 s. The context window doubled, and here is the honest math. We wrote "~24k tokens of KV cache" above. With grouped-query attention 4 KV heads and a q8 cache, this model's KV costs ~52 KB/token — so 49k of context adds only ~2.6 GB on top of ~19 GB of weights. 49152 tokens, still 100% on-GPU , generation speed unchanged. 64k also fits, but leaves ~1.7 GB of headroom on a box that also drives a desktop — we passed. Context is never "free at the same VRAM"; it was simply cheaper than we assumed. A second serving quirk, same lesson as the first. The MoE sometimes serializes array-typed tool arguments as a stringified JS literal — "layout": " {'tool name': ...} " instead of an actual array. The schema validator rightly drops it, and our designer-tool routing quietly fell to 9/21. One more tolerant-parser pass coerce string arguments that parse as arrays/objects back into structure, code fields excluded took it back to 20/21. If you keep one sentence from this post, keep this one: the weights are capable; the adapter around them is where the work is. The MCP server grew from two tools to nine. Alongside retrieval and Q&A it now exposes metadata-aware tools — the real column names of a route, lookup resolution displayed value → ID , sample records, the exact schema of each metadata action, and the framework's skill playbooks — because the single biggest failure mode of a local agent is inventing field names. The chatbot's own tool catalog, meanwhile, is up to 32 tools . The side effect became a product: WUIC Assistant. Borrowing someone else's agentic harness was the right way to start and the wrong way to stay: the loop we needed is metadata-aware, and grafting that onto a general-purpose agent kept leaking. So the assistant is now our own VS Code extension , distributed as a downloadable .vsix — chat sidebar, agent loop, file edits with diff approval, execute command in the integrated terminal, and the same wuic-rag MCP tools wired in. Three interchangeable brains behind it: local Ollama by default no key, no bill , the Anthropic API, or an Anthropic subscription through the claude CLI. It is on the model page https://wuic-framework.com/model . The toolbox grew a dimension. The chatbot now drives the 3D scene designer the same way it drives the dashboard designer: "add a red cube", "kanban wall of the orders by status", "build a warehouse scene bound to the stock table" — one tool call, applied client-side to the live three.js scene, data-bound through the 3D mesh repeater. Same host-registry pattern, new canvas. And the frontier-vs-local check, re-run. We benchmarked a newer dense 27B against the MoE: it is genuinely better at routing perfect on the kinds the MoE fumbles — and 4.5× slower 44 vs 197 tok/s , with no MTP head in the quantized build to claw the speed back. For an interactive assistant we kept the MoE + DPO: 96/102 at full speed beats ~99/102 at thirty seconds a reply. The hybrid recommendation above stands — now the local half is just much closer to the frontier than it was in June. Discussion welcome — if you've run a local model behind an agentic editor and hit different serving quirks than these two, I'd genuinely like to hear about them.