I closed an important stage of ops-triage-ai, an operational triage
system that combines a deterministic baseline, a local LLM, and a hybrid
policy with human review.
The most interesting result was not simply "the LLM was better."
Evaluation on a frozen held-out set of 70 synthetic tickets, in a single
run. Deterministic baseline versus the local LLM:
And the point that matters: the deterministic baseline still won on
overall risk accuracy — 95.7% against the LLM's 91.4%.
The LLM did not win everywhere. The regression is published alongside the
gains, with nothing smoothed over.
Instead of:
"How do we replace rules with AI?"
the architecture answers:
"How do we combine different behaviors in a way that is safe, observable,
and auditable?"
In the final benchmark:
Review rate describes operations here — how many cases asked for review —
not review precision/recall. The held-out set has no independent ground
truth for that.
Synthetic dataset, in English, with 70 examples. A single official run,
with no measurement of LLM variance. The system has not been validated
under real production traffic. Hybrid latency sits around 6 to 7 seconds:
fine for asynchronous triage, not for a critical synchronous path. The
full details — official metrics, trade-offs, and the 13 published
limitations — are in the project case. A useful AI system does not need to trust the model blindly.
Sometimes the best architecture comes precisely from understanding where
the model is better — and where it is not.