I set out to have a language model classify integration failures. I built an evaluation harness to prove it worked. The harness proved it wasn't worth using.
Final architecture: deterministic rules do the classification and structured extraction. The LLM does exactly one thing, writing plain-English incident summaries. Cheap rules to sort, AI to explain.
Harness is here: https://github.com/fissible/llm-triage-eval
Here are the numbers, then how I got them.
A legacy integration layer is being retired and rebuilt. Its failures are voluminous, repetitive, and hard to reason about: a wall of stack traces where the same handful of root causes repeat thousands of times.
I wanted a failure taxonomy and triage tool. Feed it exported logs, get back a categorized set of distinct problems rather than an undifferentiated pile.
Secondary benefit that turned out to matter: the labeled failure set doubles as a migration-acceptance suite. If the replacement handles all 58 distinct failures correctly, it covers what the old system was actually failing on.
Roughly 4,000 parsed failures collapse to 58 distinct signatures, deduplicated by error type plus normalized message plus originating element. In one case, 675 expression errors were approximately one bug.
Two things mattered more than anything model-related.
Streaming parse. Line-by-line via a generator, constant memory on 100MB+ files. Header regex plus block-field regex.
Correlated-detail enrichment. This is the key move and it's entirely deterministic. A failing transaction logs a generic error at ERROR level, and separately, at WARN, a JSON blob keyed by correlation ID carrying the actual root cause. A two-pass scan collects the detail blocks by correlation ID and attaches them to the error case.
The impact was large. One generic routing category de-aliased from 287 cases to 85. A database timeout category went from 0 to 60, because the real cause had been invisible behind a wrapper.
That is worth sitting with. The single biggest improvement in classification quality came from stitching correlated log lines together with regex, before any model saw anything.
Thirteen categories, derived from frequency counts on real logs, defined by root cause rather than error code.
The labeling standard, which I'll call the doctrine: classify by the deepest root cause visible in the correlated details, not the generic wrapper. A routing error whose underlying detail is a duplicate key violation is a constraint violation, not a routing error. With one nuance: a timeout calling a downstream service over HTTP is a downstream timeout; a true database timeout requires the JDBC call to be inside the failing component.
A rule-based classifier pre-tagged every case, which made building the golden set a review task rather than blind labeling. That weak labeler also became the deterministic baseline the LLM had to beat, which turned out to be the most consequential design decision in the project.
The 58 cases are sanitized, environment-tagged, and human-reviewed. The original error type is withheld from the model's input, since including it would trivialize the task.
Three eval types:
Discipline throughout: temperature 0, structured output via schemas, every run writes a timestamped JSON report, prompts are versioned so regressions are diffable, and the golden set is the regression target.
php artisan triage:eval --source=db --only-reviewed --prompt=v3
php artisan triage:eval-extract
php artisan triage:eval-summary
Rules beat the model at classification. 89.7% to 87.9%. Not a rout, but the model has to be better to justify latency, cost, and nondeterminism, and it wasn't better at all.
Prompt engineering did nothing. Three prompt versions, identical accuracy. The entire improvement in this project, an 8.7 point jump from 81% to 89.7%, came from making the labeling doctrine coherent. Not from prompts. Once I could state clearly what the correct label was, both the rules and the model improved, and the rules improved more.
The harness caught a bug in itself. Running with the v2 prompt flag produced a report labeled v2, but the runner never threaded the version through to the classifier, so it silently re-ran v1. The tell was identical token counts across supposedly different runs. Without the harness recording token counts per run, three prompt versions would have looked like a real negative result rather than a wiring bug producing a fake one.
A fair baseline erased the model's apparent edge on extraction. With a shallow regex, the model looked dramatically better, because it dug into correlated details the regex ignored. Making the regex details-aware closed the gap: agreement of 96.5% on HTTP method, 94.8% on status, 93.1% on constraint, 89.7% on operation. The two fields that still diverge, error type at 62% and target entity at 83%, differ for definitional reasons rather than accuracy: when a failure chain carries several error codes, which one is "the" type is a question about the taxonomy, not the extractor.
I had previously claimed extraction was where the LLM earned its place. That claim was an artifact of a weak baseline.
Summaries are where the model wins. Given a failure with correlated details, it produces a readable root-cause-and-cascade narrative that no regex is going to write.
Scoring summaries with an LLM judge that shares a model with the summarizer is self-evaluation, and I didn't want to report those numbers without checking them. So I scored 15 summaries by hand and compared.
| Axis | Exact | Within-1 | Mean bias | Quadratic-weighted κ | Verdict |
|---|---|---|---|---|---|
| Faithfulness | 80% | 100% | -0.07 | 0.57 | Moderate |
| Completeness | 40% | 80% | +0.13 | 0.10 | Do not trust |
Faithfulness holds. Never off by more than one point, near-zero bias, κ of 0.57. The 4.62 stands with moderate confidence. This is the axis that matters most, since it's asking whether the model invented facts.
Completeness does not. κ of 0.10, exact agreement 40%. Don't cite the 4.26. Two things are likely tangled: completeness is genuinely more subjective, and with only 15 items scored mostly 4s and 5s, κ becomes unstable and reads low even where within-1 agreement is 80%. Either way the honest call is inconclusive.
The most useful thing I learned here wasn't statistical. Scoring completeness by hand was hard, and I disagreed with my own earlier judgments. That's not a judge problem, it's a rubric problem. I had written a coherent doctrine for classification labels and never wrote one for what "complete" means in a summary. The judge disagreed with me because I disagreed with myself.
LLM-as-judge worked on the objective axis and failed on the subjective one, which is roughly what you'd hope, and worth knowing rather than assuming.
Deterministic rules and parsers do classification and structured extraction. The LLM writes the plain-English summary on demand.
The LLM classifier and extractor still exist, but only inside the eval harness, as the measured contender. Swapping to a stronger model is a two-value config change, so when I want to know whether a frontier model changes the answer, the benchmark is already there. It just isn't in the live path.
The uncomfortable version of all this: I spent most of the project building the thing that told me not to build the thing. That was the project working correctly.
The harness is on GitHub: https://github.com/fissible/llm-triage-eval