Compiling an SOP into a 3B Model

A QLoRA fine-tune of Qwen 2.5 3B that compiles a 3,500-token incident-triage SOP directly into model weights reached 93.0% end-to-end accuracy on 500 test scenarios, edging out Gemini 2.5 Pro's 92.0% with caching and 90.0% raw, according to an experiment published in the arpitbbhayani/workflow-slm repository. The fine-tuned 3B model cut average input tokens per incident by 91.3% (5,200 to 450) by replacing the procedural system prompt with a single trigger string, incident_triage_v1, and reduced cost to $0.22 per 1,000 incidents versus $1.85 for cached Gemini 2.5 Pro and $3.50+ raw, an 8.4x saving. The untuned Qwen 2.5 3B base scored 20.0% (100/500) on the same scenarios.

TL;DR: Compiling a 3,500-token Standard Operating Procedure https://en.wikipedia.org/wiki/Standard operating procedure into a QLoRA https://arxiv.org/abs/2305.14314 -tuned 3B model hit 93.0% end-to-end accuracy vs. 92.0% on Gemini 2.5 Pro https://deepmind.google/technologies/gemini/ while cutting input tokens by 91.3% 5,200 → 450 and cost by 8.4x $1.85 → $0.22 per 1k incidents . End-to-End Success Rate 500 Test Scenarios — Higher is Better ───────────────────────────────────────────────────────────────────────────── Fine-Tuned 3B SLM This Run ████████████████████████████░░ 93.0% 465/500 Gemini 2.5 Pro With Cache ███████████████████████████░░░ 92.0% 460/500 Gemini 2.5 Pro Raw API ███████████████████████████░░░ 90.0% 450/500 Untuned Qwen 2.5 3B Base ██████░░░░░░░░░░░░░░░░░░░░░░░░ 20.0% 100/500 Average Input Tokens per Incident Lower is Better ───────────────────────────────────────────────────────────────────────────── Prompt-Driven Setups SOP ██████████████████████████████ 5,200 tokens Fine-Tuned 3B SLM Compiled ███░░░░░░░░░░░░░░░░░░░░░░░░░░░ 450 tokens -91.3% Cost per 1,000 Incidents Lower is Better ───────────────────────────────────────────────────────────────────────────── Gemini 2.5 Pro Raw API ██████████████████████████████ $3.50+ Gemini 2.5 Pro With Cache ████████████████░░░░░░░░░░░░░░ $1.85 Fine-Tuned 3B SLM This Run ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░ $0.22 8.4x cheaper I got an idea to run that takes things in the opposite direction from what most people are doing right now with language models. Here’s repository for reference arpitbbhayani/workflow-slm https://github.com/arpitbbhayani/workflow-slm . Most AI agent setups today rely on prompt engineering. You take a SOP workflow , paste thousands of tokens of instructions into the system prompt, and send that entire document back and forth on every turn of a conversation. It works, but it got me wondering: what happens if we do the opposite? Instead of giving the model any procedural instructions in the prompt at all, could we compile the entire multi-step decision tree directly into the weights of a small 3-billion-parameter model? Incident triage was a simple workflow I tried to convert into an SLM. This was just a fun experiment I decided to run to see if it would work, how hard it would be to train, and what the resulting performance and token numbers would look like. Workflow Incident triage <https://en.wikipedia.org/wiki/Incident management ITSM was a simple workflow I chose because it operates cleanly as a finite state machine where each step depends on the output of the previous step: 1. An alert arrives as noisy text: FIRING checkout api 5xx rate way above normal in us-east, started about 12 min ago . 2. Extract the canonical service name, symptom, region, and start time. 3. Query metric telemetry to check whether the anomaly is real or transient noise. 4. If metrics are critical, query recent releases across a forty-five-minute correlation window to locate suspect deploys. 5. Query service logs to match error signatures and stack traces. 6. Conclude the root cause, culprit deployment identifier, and remediation runbook. 7. Compute incident severity and on-call routing using deterministic code. 8. Draft a concise status update citing only verified evidence. In a traditional prompt-driven setup, the operational specification for this procedure takes about 3,500 tokens of Markdown. Across five or six conversational turns, that means sending roughly 5,200 input tokens per incident to an API. Zero Procedural Prompt The idea was to strip the system prompt down to almost nothing. In my experiment, the system prompt contains zero instructions, zero schemas, and zero runbooks. It is just a single trigger string: incident triage v1 . Prompt Setup Baseline — ~5,200 input tokens / incident 3,500-Token SOP Manual + Alert + Tools -- Action JSON Compiled Setup This Experiment — ~450 input tokens / incident "incident triage v1" + Alert + Tools -- Action JSON The goal was to see if the neural network could learn the state machine so thoroughly that it would execute the diagnostic turns in the correct sequence purely from its weights. Building A Simulator And Generating Training Data To train and test this cleanly, I built a deterministic simulator with twelve microservices and nine root causes, such as bad code deploys, configuration errors, database connection pool exhaustion, and transient metric spikes. I also wrote an oracle script that knows the mathematically optimal sequence through the state machine. The oracle generated 5,116 multi-turn training conversations across five different alert styles PagerDuty alerts, Slack messages, email alerts, SMS messages, and messy freeform notes . Every training trace had to pass eight programmatic checks: valid step ordering, strict JSON schemas, grounded entity extraction, true root cause identification, valid runbook selection, verified culprit commit hashes, log evidence confirmation, and hallucination-free channel summaries. The Training Setup: Loss Masking On Assistant Turns I used Qwen/Qwen2.5-3B-Instruct https://huggingface.co/Qwen/Qwen2.5-3B-Instruct as the base model and trained it locally on my desktop with an NVIDIA RTX 3060 graphics card 12 GB of VRAM using QLoRA https://arxiv.org/abs/2305.14314 . The one crucial technical detail during fine-tuning <https://en.wikipedia.org/wiki/Fine-tuning machine learning was token loss masking. In standard causal language modeling https://en.wikipedia.org/wiki/Autoregressive language model , loss is calculated across all tokens. In a multi-turn tool-calling dataset, you do not want the model to learn to predict what the diagnostic tools will return. If you do not mask tool outputs, the model wastes its capacity trying to memorize random telemetry numbers and arbitrary commit hashes. To prevent this, the dataset collator assigned label to all tokens representing the user alert, system triggers, and tool responses, leaving only the assistant actions active during loss calculation. Gradients only flow through the decisions of the assistant: The base model was quantized in 4-bit NormalFloat NF4 https://arxiv.org/abs/2305.14314 format. I attached low-rank adapters LoRA https://arxiv.org/abs/2106.09685 , across all linear projection layers, resulting in an adapter file of just 115 MB. The entire training run took three epochs and finished in under four hours. What An Interactive Run Looks Like Once trained, I wired the adapter into a simple command-line runtime loop. When you pass it a raw alert string, here is what happens turn by turn: Turn 1: The model parses the messy alert and extracts canonical keys: { "step": "extract alert", "service": "checkout-api", "symptom": "high error rate", "region": "us-east", "started at": null, "started minutes ago": 12 } Turn 2: Without any prompt telling it what to do next, the model calls the metric tool: { "step": "get metrics", "service": "checkout-api", "metric": "error rate", "window minutes": 30 } The simulator returns an error rate of 0.18 against a baseline of 0.01, marking the status as critical. Turn 3: Recognizing the metric anomaly, the model inspects recent releases: { "step": "get recent deploys", "service": "checkout-api", "window minutes": 120 } The simulator finds deployment d-7731 rolled out fifteen minutes prior. Turn 4: The model inspects the error logs: { "step": "query logs", "service": "checkout-api", "window minutes": 30 } The simulator returns NullPointerException stack traces. Turn 5: The model concludes the root cause: { "step": "conclude", "root cause": "bad deploy", "runbook id": "RB-101", "culprit": "d-7731", "log signature": "unhandled exception" } A deterministic Python function runs to assign incident severity and paging actions: { "severity": "sev1", "action": "page oncall immediately", "team": "team-checkout", "runbook title": "Roll back the deploy" } Turn 6: The model drafts the final channel broadcast: { "step": "draft summary", "text": "SEV1: checkout-api root cause bad deploy. Follow RB-101. Team team-checkout. Culprit: d-7731." } The Experimental Numbers I evaluated the trained model on 500 held-out test scenarios and compared the results alongside Gemini 2.5 Pro and an untuned Qwen 2.5 3B model. The numbers surprised me: - The fine-tuned 3B model achieved a 93% end-to-end success rate, right on par with Gemini 2.5 Pro 92% . - The untuned 3B model with the full prompt failed on most traces 20% success , getting overwhelmed by the long procedural instructions. - Because the 3,500-token manual was absent from the prompt, input token volume dropped from 5,200 down to 450 tokens per incident a 91.3% reduction . - At standard local GPU or small-model inference rates, triaging 1,000 incidents cost $0.22 compared to $1.85 on frontier prompt caching https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview , an 8.4x difference. What I Learned From The Experiment This was a fun technical exploration that showed compiling a multi-turn state machine into model weights is entirely feasible on everyday hardware: - A 3B model can internalize a complex, branching procedural sequence without needing continuous prompt reminders. - Proper loss masking during training is essential to stop the model from trying to predict simulated environment outputs. - The tradeoff is simple: prompts are easy to edit when rules change, while compiled weights give you high speed, zero prompt overhead, and predictable local execution. Footnote It was an entertaining holiday build that proved you do not always need a massive prompt to get disciplined, multi-turn reasoning out of a language model.