# The LLM Didn't Win Everywhere — and That's What Made the Project Interesting

> Source: <https://dev.to/marcelotaparelli/the-llm-didnt-win-everywhere-and-thats-what-made-the-project-interesting-a1a>
> Published: 2026-09-15 09:33:21+00:00

I closed an important stage of `ops-triage-ai`, an operational triage

system that combines a deterministic baseline, a local LLM, and a hybrid

policy with human review.

The most interesting result was not simply "the LLM was better."

Evaluation on a frozen held-out set of 70 synthetic tickets, in a single

run. Deterministic baseline versus the local LLM:

And the point that matters: the deterministic baseline still won on

overall risk accuracy — 95.7% against the LLM's 91.4%.

The LLM did not win everywhere. The regression is published alongside the

gains, with nothing smoothed over.

Instead of:

"How do we replace rules with AI?"

the architecture answers:

"How do we combine different behaviors in a way that is safe, observable,

and auditable?"

In the final benchmark:

Review rate describes operations here — how many cases asked for review —

not review precision/recall. The held-out set has no independent ground

truth for that.

Synthetic dataset, in English, with 70 examples. A single official run,

with no measurement of LLM variance. The system has not been validated

under real production traffic. Hybrid latency sits around 6 to 7 seconds:

fine for asynchronous triage, not for a critical synchronous path. The

full details — official metrics, trade-offs, and the 13 published

limitations — are in the [project case](https://marcelotaparelli.com.br/en/projects/ops-triage-ai/).

A useful AI system does not need to trust the model blindly.

Sometimes the best architecture comes precisely from understanding where

the model is better — and where it is not.