Calling frontier models (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro) for narrow, repetitive agent tasks is one of the most expensive architectural patterns in software engineering today.
Most production workflows—triaging alert tickets, converting logs to structured JSON, routing database queries, or interacting with a couple of CLI tools—don't require world-historical reasoning or creative flair. They require:
Yet, teams routinely rack up $500 to $2,000+ monthly API bills because smaller local models (0.5B–1.5B) traditionally collapse into hallucinated chaos when asked to execute multi-step tool calls.
Over the past few weeks, I built and open-sourced Task Model Distiller, a 3-pillar pipeline designed to distill frontier agent capabilities into sub-2B models (Qwen2.5-0.5B, Qwen2.5-1.5B, SmolLM2-1.7B) that run locally on ordinary CPUs at <1.5 GB RAM with zero external API fees.
Here is how the architecture works, why generic fine-tuning fails on small models, and how you can run this flywheel inside your own IDE or terminal.
If you take a base Qwen2.5-0.5B or Llama-3.2-1B model and fine-tune it on standard single-turn prompt-response datasets, you run into three critical failure modes:
system → user → tool_call → tool_output → response). Single-turn pairs fail to teach the model how to inspect prior execution state.{"status": "error", "code": 500}, naive models often ignore the error payload and proceed as if the action succeeded.
To fix this, we need more than just training data—we need adversarial calibration and deterministic evaluators.
┌────────────────────────────────────────┐
│ Frontier Teacher (In-Workspace) │
└───────────────────┬────────────────────┘
│
┌──────────────────────┴──────────────────────┐
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ PILLAR 1 │ │ PILLAR 2 │
│ Full-Session Traces │ │ Adversarial Referee │
│ (Happy + Recovery) │ │ (Stage 1 + Stage 2) │
└────────────┬────────────┘ └────────────┬────────────┘
│ │
│ ┌─────────────────────────┐ │
└────────►│ CPU LoRA Fine-Tuning │◄────────┘
│ (train_cpu_lora.py) │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ PILLAR 3 │
│ The Scoreboard │
│ (Private Locked) │
└────────────┬────────────┘
│ Failures
▼
┌─────────────────────────┐
│ Targeted Flywheel │
│ (Synthesize Fix Traces)│
└─────────────────────────┘
Instead of calling expensive third-party data generation APIs, your frontier AI coding assistant (inside Google Antigravity IDE, Claude Code, or Cursor) acts natively as the Teacher.
We generate complete multi-turn trajectories:
Every trace is verified before training using validate_traces.py to ensure schema integrity and strict ChatML formatting.
Before training or benchmarking, we build and calibrate The Referee:
page_engineer was physically executed and that its return code was not an error.APPROVED vs REJECTED + offending sentence).mutate_traces.py):
We create a private, locked regression benchmark (benchmark_locked.jsonl) containing 30–50 tricky scenarios.
scoreboard.py runs and outputs a categorized failure report (e.g. {"tool_timeout": 3, "unauthorized_flag": 1}). incident-triager
Let's look at an imaginary production scenario:
Qwen/Qwen2.5-0.5B-Instruct (0.5 Billion parameters).
Running the untouched base qwen2.5:0.5b model through scoreboard.py:
6/25 (24%)
We fine-tuned the model using train_cpu_lora.py with 4-bit LoRA target modules:
python scripts/train_cpu_lora.py --base-model Qwen/Qwen2.5-0.5B-Instruct --dataset .distill/incident-triager/dataset/train_traces.jsonl --output-dir .distill/incident-triager/training --epochs 3 --merge
safetensors). 24/25 (96%)
Because train_cpu_lora.py --merge exports a standard Hugging Face checkpoint, you are never locked into a single ecosystem:
| Target Runtime | Command / Export Tool | Best For |
|---|---|---|
| Ollama (4-bit GGUF) | ollama create my-bot -f Modelfile |
Instant local CLI & API (<1 GB RAM) |
| llama.cpp | python llama.cpp/convert_hf_to_gguf.py merged_hf/ |
Native C++ embedded & edge devices |
| ONNX Runtime | optimum-cli export onnx --model merged_hf/ onnx/ |
C#, C++, Rust, Windows DirectML |
| Apple Silicon (MLX) | python -m mlx_lm.convert --hf-path merged_hf/ |
Unified memory Mac inference |
| vLLM / SGLang | Load directly from merged_hf/ |
High-throughput server hosting |
The entire pipeline is packaged as a skill compatible with Google Antigravity IDE, Claude Code, Cursor, and Windsurf.
Install globally:
mkdir -p ~/.gemini/config/skills/task-model-distiller
cp -r * ~/.gemini/config/skills/task-model-distiller/
Then prompt your agent:
"Use the task-model-distiller skill to distill a local 0.5B model for my ticket-triage script."
Add to your project's CLAUDE.md:
When asked to build a local tool model, adhere to `SKILL.md` in `task-model-distiller`.
Then run:
claude "Initialize a model distillation pipeline for my GitHub labeler agent using task-model-distiller."
The entire codebase—including the synthetic trace validator, adversarial referee, scoreboard evaluator, and CPU LoRA training scripts—is open-source under the MIT license:
👉 GitHub Repository: https://github.com/Virat-Ghatge/task-model-distiller
Give it a star ⭐ if you find it helpful, and start replacing expensive frontier agent calls with fast, ultra-reliable 0.5B local models!