Building an AI agent that doesn't just confidently hallucinate but actually writes safe, reliable code is a massive challenge. It is an even bigger challenge when you want to run that agent entirely locally without sending your proprietary code to a closed-source API.
That is exactly why I built Mitii, an AI coding assistant with a unique, multi-mode architecture designed to give developers absolute control.
To see how well this architecture actually works, I recently put Mitii through a brutal manual benchmark featuring 515 distinct tasks ranging from simple bug fixes to adversarial security injections. For this gauntlet, I powered Mitii using qwen3-coder:30b running locally via Ollama.
Here is a deep dive into the architecture and the results.
qwen3-coder:30b
Mitii doesn't force you into a single way of interacting. The SDK features distinct modes to handle different levels of complexity:
We ran Mitii through a comprehensive suite of real-world scenarios across all three modes. The tasks ranged from fixing broken Next.js routes to handling prompt-injection security attacks.
The Final Score: 400 out of 515 tasks passed (78%). Here is the breakdown of how the different modes performed across varying difficulty levels using the Qwen3-Coder model:
| Mode | Severity | Passed | Total | Win Rate |
|---|---|---|---|---|
| Agent | ||||
| Easy | 72 | 82 | 88% | |
| Agent | ||||
| Hard | 66 | 81 | 81% | |
| Agent | ||||
| Medium | 65 | 92 | 71% | |
| Ask | ||||
| Easy | 26 | 37 | 70% | |
| Ask | ||||
| Hard | 27 | 31 | 87% | |
| Ask | ||||
| Medium | 21 | 37 | 57% | |
| Plan | ||||
| Easy | 45 | 52 | 87% | |
| Plan | ||||
| Hard | 35 | 46 | 76% | |
| Plan | ||||
| Medium | 43 | 57 | 75% |
1. Local LLMs are Production-Ready for Agents
Achieving a nearly 80% overall pass rate using a 30-billion parameter local model proves that you don't need to rely on massive cloud providers to get top-tier AI assistance. Mitii's architecture combined with Qwen3-Coder makes local, private AI coding highly capable.
2. "Ask Mode" Shines Under Pressure
Interestingly, Ask Mode scored a massive 87% win rate on Hard tasks. Because Ask Mode explicitly presents an impact analysis detailing the web searches it needs to run and the files it expects to modify, it proves incredibly resilient when dealing with complex, adversarial, or destructive prompts (like the nuke-delete-test-files
or prompt-injection
tasks).
3. Security and Corner Cases are a Strong Suit
In the category breakdown, Mitii and Qwen3 excelled where it matters most:
Achieving a 78% win rate on a highly adversarial benchmark with a local model is a huge milestone, but there is still plenty of room to grow—particularly in improving our semantic retrieval (currently at 63%) and refining our medium-difficulty routing.
If you want to integrate intelligent, context-aware AI agents into your development workflow without sacrificing privacy, check out the Mitii SDK.