The LLM Didn't Win Everywhere — and That's What Made the Project Interesting

A developer closed a key stage of ops-triage-ai, an operational triage system that combines a deterministic baseline, a local LLM, and a hybrid policy with human review. Evaluated on a frozen held-out set of 70 synthetic tickets in a single run, the deterministic baseline still beat the local LLM on overall risk accuracy, 95.7% versus 91.4%, and the regression was published alongside the gains. The project reports a hybrid latency of roughly 6 to 7 seconds and 13 published limitations, and has not been validated under real production traffic.

I closed an important stage of ops-triage-ai , an operational triage system that combines a deterministic baseline, a local LLM, and a hybrid policy with human review. The most interesting result was not simply "the LLM was better." Evaluation on a frozen held-out set of 70 synthetic tickets, in a single run. Deterministic baseline versus the local LLM: And the point that matters: the deterministic baseline still won on overall risk accuracy — 95.7% against the LLM's 91.4%. The LLM did not win everywhere. The regression is published alongside the gains, with nothing smoothed over. Instead of: "How do we replace rules with AI?" the architecture answers: "How do we combine different behaviors in a way that is safe, observable, and auditable?" In the final benchmark: Review rate describes operations here — how many cases asked for review — not review precision/recall. The held-out set has no independent ground truth for that. Synthetic dataset, in English, with 70 examples. A single official run, with no measurement of LLM variance. The system has not been validated under real production traffic. Hybrid latency sits around 6 to 7 seconds: fine for asynchronous triage, not for a critical synchronous path. The full details — official metrics, trade-offs, and the 13 published limitations — are in the project case https://marcelotaparelli.com.br/en/projects/ops-triage-ai/ . A useful AI system does not need to trust the model blindly. Sometimes the best architecture comes precisely from understanding where the model is better — and where it is not.