A Better Model Improved the Numbers. It Didn't Fix the Product.

Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you 25+ CLI commands, an MCP server, 7 export formats, and a bundled git rule pack. The field test report is the source for every number in this article: 4 models, 13 corpus types, 394 trajectories, 844+ deterministic tests passing. The strongest cloud model tested was meta-llama/llama-3.1-8b-instruct with 123 pass / 168 inconclusive / 101 fail — better than every other model, but still not enough to solve the safety problem on its own. Release notes · Changelog

There is a comforting fantasy in AI engineering. When a system struggles, you tell yourself the problem is the model. Spend a little more, use a stronger one, and the product gets better.

Sometimes that is true. One of the most useful outcomes from the CauterRule field test was learning exactly where that story stops being true.

After fixing the parser, prompt, result-reset, and corpus timestamp issues, we finally had a benchmark we could trust. That was the moment cloud comparison became worth doing. We ran two working cloud models through the full corpus: openai/gpt-4o-mini and meta-llama/llama-3.1-8b-instruct.

The point was not just to get better numbers. It was to answer a sharper question:

Are the remaining problems mostly local-model weakness, or are they product problems?

The answer turned out to be both, but mostly the second.

The broadest comparison is the full-corpus view.

| Model | Type | Total rows | Total candidates | Pass | Pass % | Inconclusive | Fail |

|---|---|---|---|---|---|---|---|
| `omlx-openai-Llama-3.2-3B-Instruct-4bit` | local | 393 | 379 | 72 | 19.0% | 189 | 118 | 
| `omlx-openai-Qwen3-4B-Instruct-2507-4bit` | local | 373 | 373 | 93 | 24.9% | 209 | 71 | 

| openai/gpt-4o-mini | cloud | 394 | 394 | 77 | 19.5% | 248 | 69 | | meta-llama/llama-3.1-8b-instruct | cloud | 394 | 392 | 123 | 31.4% | 168 | 101 |

Three things jump out. First, the cloud models removed most of the remaining availability and formatting ambiguity — both produced near-perfect parse reliability. Second, meta-llama/llama-3.1-8b-instruct produced the strongest overall pass count of any model we tested: 123 passes from 392 candidates. Third, gpt-4o-mini was still operationally valuable because it was stable, cheap, and fully runnable across all 394 trajectories.

But there is something more subtle in this table that matters more than the ranking. Even the strongest tested model still produced 168 inconclusives and 101 hard fails. That is 269 out of 392 candidates — 69% of the output was not clearly usable. Better models improved the system. They did not settle the product.

The curated corpora are the cleanest lens for this comparison.

If you want the shortest honest reading of that table: cloud models clearly help on quality. meta-llama/llama-3.1-8b-instruct was the strongest model tested. And the positive-case corpora improved much faster than the safety corpora. That last point matters more than the ranking itself.

This view is a better way to summarize what actually improved.

The cloud models won where learning-from-failure was easiest. They did not decisively win where restraint mattered most.

The raw corpora add volume and show the same pattern at scale.

Cloud Llama 3.1 8B led on raw/opencode (18P), raw/synthetic (38P), raw/corrections (3P), and raw/cross-session (4P). But on raw/ci, it still produced 60 inconclusives and 39 hard fails from 110 trajectories. That is a lot of noise from the strongest model in the batch.

The report makes an observation about this that is worth pulling out: raw breadth without replay clarity is not yet evidence. The product can process broad raw corpora, but if it mostly yields inconclusive judgments, that is not the same as producing trustworthy results. Broad processing coverage is good. But broad coverage that produces mostly inconclusives is not yet the strongest evidence of product trustworthiness.

The cloud runs gave us much cleaner evidence than the early local-only picture:

golden and failures/positive corrections and broad raw corpora One model stood out: meta-llama/llama-3.1-8b-instruct was the strongest cost-effective model in the entire batch. On failures/positive, it produced 22 pass / 5 inconclusive / 3 fail — the best single result on the most practical extraction benchmark. On noisy, it produced 5 pass / 0 inconclusive / 0 fail — a clean sweep. On raw/opencode, it produced 18 pass — more than any other model.

The report's verdict: best current cost/performance benchmark model. Not because it was the most expensive or the largest, but because it consistently outperformed gpt-4o-mini on the corpora that matter most for practical rule extraction.

This is the more important half of the story.

Even with stronger cloud models, the system still struggled on the corpora that matter most for trust: successes, failures/negative, and nearmiss.

The simplest local-vs-cloud safety view:

On successes, the strongest cloud model produced 3 pass / 6 inconclusive / 11 fail. Those 3 "passes" are not a win — they are false positives. A "pass" on the successes corpus means the system extracted a rule from a trajectory that should not have produced one. Every pass on successes is a safety violation.

On failures/negative, the strongest cloud model produced 3 pass / 1 inconclusive / 6 fail. Every model in the field test produced 5-6 hard fails on this corpus, and no model produced more than 3 passes. The rejection gate is weak regardless of which model drives extraction.

On nearmiss, Cloud Llama 3.1 8B produced 6 pass / 1 inconclusive / 7 fail. Better than the locals, but still more failures than passes. The trigger is firing on cases that look similar but should not trigger.

If cloud models had cleaned those up, I would be writing a very different article. They did not. This is the insight that changed how I think about the roadmap.

Before the cloud runs, it was possible to believe the remaining weakness was mostly a model problem. Small local models, limited instruction-following, noisy output — maybe a bigger model would fix it.

After the cloud runs, that explanation is dead. A stronger model with near-perfect parse reliability, running on a now-trustworthy benchmark, still produced 11 fails on successes and could not reliably reject bad candidates on failures/negative. The problem is not that the model cannot generate rules. The problem is that the product's judgment — when to extract, when to reject, when to stay silent — is not strong enough.

The report identifies six specific gaps between the current state and a stronger release claim:

successes, failures/negative, and nearmiss. The first three are the real release blockers. The others matter, but safety, replay trust, and promotion confidence are what define whether the product can honestly claim v0.1.0 field-test success in the strongest sense.

Better models improve the ceiling, not the floor. The cloud models lifted the best results — more passes on failures/positive, cleaner noisy handling, stronger raw corpus extraction. But they did not lift the worst results. successes and failures/negative stayed weak. That tells you the floor is a product problem, not a model problem.

A "pass" on a safety corpus is not always a win. On successes, a pass means the system extracted a rule from a trajectory that should not have produced one. Counting passes without reading the corpus label gives you a false sense of progress. You have to know what each corpus is testing before you interpret the number.

Inconclusives are the silent killer. gpt-4o-mini produced 248 inconclusives out of 394 candidates — 63% of its output was uncommitted. Those inconclusives did not show up as failures, so they did not trigger alarm. But they also did not produce usable rules. A model that is 63% inconclusive is not producing 77 passes. It is producing 77 passes and 248 maybes. Maybes do not build trust. And the report is clear that some of those maybes are a replay-engine limitation, not a model limitation — the matcher is the named #1 bottleneck. But on the safety corpora, inconclusives are a different signal: they mean the model extracted something from a trajectory where it should have stayed silent, and the matcher could not cleanly reject it. Those are model problems hiding behind engine weakness. You have to split the inconclusive bucket by corpus before you know which ones are the matcher's fault and which ones are the model's.

The field test succeeded even though the product did not fully pass. This sounds contradictory but it is not. The field test succeeded because it forced the system through realistic comparisons, exposed tooling defects, validated that fixes improved signal, and produced a coherent map of what remains. The product improved materially. The product is not yet fully field-test complete by a strict release gate. Both of those things are true. That is a good outcome for an honest engineering report.

gpt-4o-mini produced zero fails on If the cloud models had solved everything, the conclusion would have been easy: just use a better model. That would be convenient, but it would teach us less.

What happened instead was better. The cloud runs improved the benchmark enough to expose the real work that remains. They showed that stronger models help a lot, but they also showed that the core challenge now lives in replay trust, safety, and promotion confidence.

That is exactly what a serious field test should surface.

The cloud models made the product look better. They also made its remaining weaknesses harder to deny. That is why we ran them.

CauterRule v0.1.0 is released. The full all-model comparison — local vs cloud, curated and raw corpora, winner-by-corpus view, and model recommendations — is in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes