{"slug": "teaching-a-local-coding-agent-from-its-own-mistakes-dpo-on-a-30b-model", "title": "Teaching a local coding agent from its own mistakes: DPO on a 30B model", "summary": "The WUIC Assistant, an agentic VS Code plugin built on the WUIC framework, improved its local coding agent by applying Direct Preference Optimization (DPO) to a 30B-parameter model served via Ollama. The team fixed two environment contaminations that skewed benchmark results, then used DPO to reduce the model's churn (unnecessary redirects) while maintaining high task completion. The model learned from its own mistakes, though not exactly as intended, highlighting the importance of clean evaluation environments and preference-based training over supervised fine-tuning.", "body_md": "The **WUIC Assistant** is our agentic VS Code plugin: it scaffolds Angular components, dashboards, reports, workflows, and metadata patches for apps built on the WUIC framework. Under the hood it runs **qwen3-coder:30b** — a Mixture-of-Experts model with ~3B active parameters — [served locally through Ollama](https://wuic-framework.com/blog/local-llm-ollama-mcp-agentic-vscode) on a single RTX 4090. No API calls, no data leaving the machine.\n\nFor weeks the plugin had been sitting at **100% on its curriculum**: 32 tasks, five green runs each, a 96/96 regression suite. Beautiful numbers, and a problem. A curriculum that always passes has stopped measuring anything. It could no longer tell us whether the model *generalized* to the prompts a real developer types, nor how often it took a wrong turn before correcting itself.\n\nWe wanted two things:\n\n`STOP: use scaffold, not ng generate`\n\n), and only This post is the story of how we got there. It involves five OutOfMemory crashes, a saturated benchmark, two hidden contaminations, and a 30-billion-parameter model that did learn — though not exactly what we asked it to.\n\nA model that evaluates itself on the data it was aligned to is lying to itself. So we built **eval-only** sets, with one iron rule: *never in training, never curricularized*. If you stabilize a task by adding a dedicated mechanism, that task loses its value as a measurement — and it gets replaced.\n\nThe latest of these, the fourth **\"HARD\"** set (Q01–Q12), deliberately raises the bar on the levers where the base model actually struggles:\n\nMeasured on a clean bench, the base model lands at **25/36 (69%)** — failing Q02, Q04 and Q07 on every attempt, and Q05 on two of three. But the most interesting data point was elsewhere: **Q01 passed 3 times out of 3 — with 9 redirects**. It got there, but by bouncing off the guards the whole way. That's the churn we wanted gone.\n\nBefore touching the model, a deceptively innocent question — *\"is the test environment actually clean and in sync?\"* — opened a can of worms.\n\n**Contamination #1: the metadata reset.** Our between-runs reset ran `DELETE WHERE id > baseline`\n\n— it deleted the rows that had been *added*, but never reverted the **UPDATEs** (like `hide_in_list`\n\n) applied to pre-existing rows. Run after run, columns on the `customers`\n\ntable were progressively hidden, until `route_columns`\n\nstarted answering \"no columns found.\" Worse: some of our test prompts had been reworded over time to *compensate* for this degradation — meaning we'd been fixing the symptoms of an environment bug and mistaking it for backend behavior. The fix: restore the full column set from the clean baseline (`metadataTutorial`\n\n) on every reset. The general lesson we kept: **a reset has to cover what you mutate, not just what you add.**\n\n**Contamination #2: the report generator.** A shared file (`gen-report-from-template.mjs`\n\n) had been overwritten by a model run, shrinking from 246 lines to 60 — and our reset didn't cover that directory. Restored, added to the reset scope, and write-guarded.\n\nThe moral: **before you trust the numbers, trust the environment.** A test bench that drifts slowly produces phantom regressions that look exactly like model bugs.\n\nWith a clean bench, the third holdout set went from 22/36 to **36/36**. The fixes were **generic** — synonyms in the detectors, correct precedence between task \"kinds\", scaffold guards for reports — never a patch for an individual prompt. That distinction is the whole game: you generalize the *root condition*, you don't enumerate the cases.\n\nBut the churn remained. And churn isn't solved by adding one more guard — it's solved by teaching the model not to fumble the first move. That meant touching the model itself.\n\nThe right technique here isn't more supervised fine-tuning. The SFT we'd already tried **masked** the rejected turns — the model never got the signal \"*this* move is wrong.\" Redirects dropped only from 43 to 39.\n\n**DPO** (Direct Preference Optimization) learns from the **pair** instead: for the exact same context, *prefer* the accepted action over the rejected one. It's a perfect fit for our problem, because every redirect in the historical transcripts is already a ready-made pair:\n\n`ng generate component…`\n\n→ STOP) = `scaffold widget=map`\n\n→ Saved) = An extractor pulled **559 pairs** from more than a thousand archived runs (including the red ones — a pair is valuable if it contains a valid rejected→corrected transition), deduplicated and balanced so no single task dominates.\n\nHere's where theory meets 80 GB of VRAM. DPO does **four forward passes** per step (policy and reference, over chosen and rejected). And our system prompt — the `.clinerules`\n\nthat give the model all the WUIC context — is **enormous**: ~20,000 tokens. Multiplied by four passes, the 30B model went OOM *before step 0*, five times in a row, first on a 4090 under WSL2 and then even on an **80 GB A100**.\n\nThe debug path, condensed:\n\n`peft`\n\n.`[seq × vocab≈150k]`\n\n→ OOM.`lm_head`\n\n+ loss without ever materializing the full logits) promised to solve it… but on a 4-bit quantized `PeftModel`\n\nit was `SMOKE_FAIL`\n\nwith no clear traceback.The breakthrough was a counterintuitive one: **the real problem wasn't DPO, it was the system prompt.** We **trimmed the system block in the dataset** (head + tail, the parts that matter), bringing sequences from ~24k down to **3–5k tokens**. At that point the fp32 logits take ~6 GB — they fit comfortably in 80 GB — and the liger fused loss was no longer needed. In fact it was the very cause of the last failure: **disabled**, standard DPO ran clean.\n\nThe final smoke test: 8 steps, loss trending down, positive `rewards/margins`\n\n, `accuracies`\n\n0.75–0.88. The model preferred the *chosen*. Green light.\n\nThe full run trained in the background on the A100 (detached, to survive SSH drops), monitored step by step:\n\n| Step | Loss | rewards/accuracies | rewards/margins |\n|---|---|---|---|\n| 5 | 0.62 | 0.63 | +0.30 |\n| 20 | — | 0.73 | +0.59 |\n| 46 | 0.47 | 0.78 | +0.74 |\n| 56 | 0.41 |\n0.88 |\n+1.02 |\n\n**70 steps, 1 epoch over 559 pairs, 4h11m, zero OutOfMemory** from start to finish — even when VRAM read 96% (a false alarm: that was PyTorch's stable *reserved pool*, not a live allocation at the edge; the paged 8-bit optimizer spills to system RAM as a relief valve). Margins above 1.0 and accuracies at 88% say the preference signal was learned cleanly.\n\nThe adapter was merged, converted to GGUF q4_K_M, and served by Ollama as `qwen3-coder-wuic:30b-dpo`\n\n. Then came the **eval gate**, with criteria fixed *before* looking at the results:\n\nA/B on the same clean bench, three rounds per task:\n\n| Criterion | Base | DPO | Gate | Verdict |\n|---|---|---|---|---|\n| PASS on HARD | 25/36 (69%) | 28/36 (78%) |\n≥ 25/36 | pass |\n| Curriculum | 96/96 | 96/96 | 96/96 | pass |\n| Report guard | intact | intact | intact | pass |\n| Redirects/run | 0.67 | 0.83 | < 0.47 | fail |\n| Zero-redirect runs | 53% | 39% | > 53% | fail |\n\nThe capability gains are real: Q02 went from 0/3 to **3/3** — a task the base never solved — and Q05 from 1/3 to 2/3, with zero forgetting. And on Q01, the flagship churn case the pairs were built around, redirects fell from 9 to 6.\n\nBut the primary metric — the whole reason we ran DPO — moved the *wrong* way: +25% redirects overall. Digging in, the number is less damning than it looks. The base scores zero redirects on Q02 because it *fails cleanly*; the DPO model racks up four because it *fights and wins* (0/3 → 3/3). A redirect count that punishes engaging with hard tasks is a metric with a blind spot — a flaw we only saw because the gate forced us to stare at it.\n\nThe pre-registered rule, though, was: no automatic adoption unless the gate clears. It didn't clear. The base stays the plugin's default; the DPO model sits one setting away, and switching in either direction is trivial — exactly the cheap, reversible outcome the gate was designed to buy.\n\nOne epilogue. On a different job — single-shot tool routing for the [in-product RAG chatbot](https://wuic-framework.com/blog/local-llm-ollama-mcp-agentic-vscode), where there are no guards and no multi-turn churn to fall back on — the same DPO model scored **96/102 against the base's 88/102** at identical speed, and *that* gate it cleared. It now ships as the chatbot's default brain. The model didn't fail; it turned out to be better at a different job than the one we trained it for.\n\nBecause that's the real point of this whole story. Not \"we fine-tuned a model,\" but: **we built a ruler that doesn't lie, we cleaned the test bench, we fixed the criteria before looking — and we let the numbers decide.** Twice.\n\nThe model is public. `qwen3-coder-wuic:30b-dpo`\n\nis on [Hugging Face](https://huggingface.co/castricolorenzo/qwen3-coder-wuic-30b-dpo) as a GGUF (q4_K_M, ~19 GB) — pull it straight into Ollama and run it on your own 24 GB GPU. The full setup, end to end — the Ollama tuning, the WUIC backend config, and the WUIC Assistant VS Code extension — is on the [Run the WUIC coding model locally](https://wuic-framework.com/model) page. No API calls, no data leaving your machine.", "url": "https://wpnews.pro/news/teaching-a-local-coding-agent-from-its-own-mistakes-dpo-on-a-30b-model", "canonical_source": "https://dev.to/wuicframework/teaching-a-local-coding-agent-from-its-own-mistakes-dpo-on-a-30b-model-50ac", "published_at": "2026-08-31 11:38:04+00:00", "updated_at": "2026-08-31 11:52:25.646691+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-agents", "developer-tools"], "entities": ["WUIC Assistant", "qwen3-coder:30b", "Ollama", "RTX 4090", "DPO"], "alternates": {"html": "https://wpnews.pro/news/teaching-a-local-coding-agent-from-its-own-mistakes-dpo-on-a-30b-model", "markdown": "https://wpnews.pro/news/teaching-a-local-coding-agent-from-its-own-mistakes-dpo-on-a-30b-model.md", "text": "https://wpnews.pro/news/teaching-a-local-coding-agent-from-its-own-mistakes-dpo-on-a-30b-model.txt", "jsonld": "https://wpnews.pro/news/teaching-a-local-coding-agent-from-its-own-mistakes-dpo-on-a-30b-model.jsonld"}}