{"slug": "a-better-model-improved-the-numbers-it-didn-t-fix-the-product", "title": "A Better Model Improved the Numbers. It Didn't Fix the Product.", "summary": "CauterRule, an open-source tool that converts repeated agent failures into standing rules, released v0.1.0 after a field test comparing four models across 394 trajectories. The test found that even the strongest model, meta-llama/llama-3.1-8b-instruct, produced only 31.4% clearly usable output, indicating that better models improve but do not fix underlying product issues.", "body_md": "**Update — v0.1.0 released.** CauterRule is now live on [GitHub](https://github.com/deghosal-2026/CauterRule) and [PyPI](https://pypi.org/project/cauterule/). It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. `pip install cauterule` gives you 25+ CLI commands, an MCP server, 7 export formats, and a bundled git rule pack. The [field test report](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT.md) is the source for every number in this article: 4 models, 13 corpus types, 394 trajectories, 844+ deterministic tests passing. The strongest cloud model tested was `meta-llama/llama-3.1-8b-instruct` with 123 pass / 168 inconclusive / 101 fail — better than every other model, but still not enough to solve the safety problem on its own. [Release notes](https://github.com/deghosal-2026/CauterRule/releases/tag/v0.1.0) · [Changelog](https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md)\n\nThere is a comforting fantasy in AI engineering. When a system struggles, you tell yourself the problem is the model. Spend a little more, use a stronger one, and the product gets better.\n\nSometimes that is true. One of the most useful outcomes from the CauterRule field test was learning exactly where that story stops being true.\n\nAfter fixing the parser, prompt, result-reset, and corpus timestamp issues, we finally had a benchmark we could trust. That was the moment cloud comparison became worth doing. We ran two working cloud models through the full corpus: `openai/gpt-4o-mini` and `meta-llama/llama-3.1-8b-instruct`.\n\nThe point was not just to get better numbers. It was to answer a sharper question:\n\nAre the remaining problems mostly local-model weakness, or are they product problems?\n\nThe answer turned out to be both, but mostly the second.\n\nThe broadest comparison is the full-corpus view.\n\n| Model | Type | Total rows | Total candidates | Pass | Pass % | Inconclusive | Fail | \n|---|---|---|---|---|---|---|---|\n| `omlx-openai-Llama-3.2-3B-Instruct-4bit` | local | 393 | 379 | 72 | 19.0% | 189 | 118 | \n| `omlx-openai-Qwen3-4B-Instruct-2507-4bit` | local | 373 | 373 | 93 | 24.9% | 209 | 71 | \n| `openai/gpt-4o-mini` | cloud | 394 | 394 | 77 | 19.5% | 248 | 69 | \n| `meta-llama/llama-3.1-8b-instruct` | cloud | 394 | 392 | 123 | 31.4% | 168 | 101 | \n\nThree things jump out. First, the cloud models removed most of the remaining availability and formatting ambiguity — both produced near-perfect parse reliability. Second, `meta-llama/llama-3.1-8b-instruct` produced the strongest overall pass count of any model we tested: 123 passes from 392 candidates. Third, `gpt-4o-mini` was still operationally valuable because it was stable, cheap, and fully runnable across all 394 trajectories.\n\nBut there is something more subtle in this table that matters more than the ranking. Even the strongest tested model still produced 168 inconclusives and 101 hard fails. That is 269 out of 392 candidates — **69% of the output was not clearly usable**. Better models improved the system. They did not settle the product.\n\nThe curated corpora are the cleanest lens for this comparison.\n\n| Curated corpus | Local Llama 3.2B | Local Qwen 4B | Cloud GPT-4o-mini | Cloud Llama 3.1 8B | \n|---|---|---|---|---|\n| `golden` | 8P / 2F | 3P / 4I / 3F | 6P / 4I / 0F | 7P / 0I / 3F | \n| `failures/positive` | 15P / 3I / 12F | 15P / 9I / 6F | 14P / 13I / 3F | 22P / 5I / 3F | \n| `failures/negative` | 1P / 2I / 5F | 1P / 4I / 5F | 0P / 4I / 6F | 3P / 1I / 6F | \n| `successes` | 1P / 16I / 2F | 2P / 8I / 10F | 0P / 6I / 14F | 3P / 6I / 11F | \n| `nearmiss` | 4P / 4I / 6F | 5P / 5I / 4F | 2P / 4I / 8F | 6P / 1I / 7F | \n| `noisy` | 1P / 3I / 1F | 4P / 1I / 0F | 1P / 3I / 1F | 5P / 0I / 0F | \n| `corrections` | 2P / 2I / 1F | 2P / 1I / 2F | 3P / 2I / 0F | 3P / 1I / 1F | \n\nIf you want the shortest honest reading of that table: cloud models clearly help on quality. `meta-llama/llama-3.1-8b-instruct` was the strongest model tested. And the positive-case corpora improved much faster than the safety corpora.\n\nThat last point matters more than the ranking itself.\n\nThis view is a better way to summarize what actually improved.\n\n| Corpus | Best model | Why it matters | \n|---|---|---|\n| `golden` | Local Llama 3.2B by raw pass count, Cloud Llama 3.1 8B by strongest cloud showing | regression anchor / specificity | \n| `failures/positive` | Cloud Llama 3.1 8B | best practical extraction benchmark | \n| `successes` | none were convincingly strong | safety / over-triggering check | \n| `failures/negative` | none were convincingly strong (all 5-6 fails) | rejection quality | \n| `nearmiss` | Cloud Llama 3.1 8B, but still mixed | trigger precision | \n| `noisy` | Cloud Llama 3.1 8B | strongest clutter-handling | \n| `corrections` | Cloud GPT-4o-mini and Cloud Llama 3.1 8B both strong | correction-to-rule synthesis | \n\nThe cloud models won where learning-from-failure was easiest. They did not decisively win where restraint mattered most.\n\nThe raw corpora add volume and show the same pattern at scale.\n\n| Raw corpus | Local Llama 3.2B | Local Qwen 4B | Cloud GPT-4o-mini | Cloud Llama 3.1 8B | \n|---|---|---|---|---|\n| `raw/opencode` | 13P / 2I / 9F | 13P / 7I / 5F | 12P / 10I / 3F | 18P / 3I / 4F | \n| `raw/synthetic` | 18P / 101I / 26F | 36P / 87I / 22F | 28P / 91I / 26F | 38P / 81I / 25F | \n| `raw/ci` | 8P / 49I / 52F | 9P / 90I / 11F | 7P / 99I / 4F | 11P / 60I / 39F | \n| `raw/sibling-repos` | 0P / 9I / 1F | 0P / 9I / 1F | 0P / 7I / 3F | 0P / 10I / 0F | \n| `raw/corrections` | 2P / 2I / 1F | 3P / 1I / 1F | 2P / 3I / 0F | 3P / 0I / 1F | \n| `raw/cross-session` | 2P / 1I / 2F | 1P / 2I / 2F | 2P / 2I / 1F | 4P / 0I / 1F | \n\nCloud Llama 3.1 8B led on `raw/opencode` (18P), `raw/synthetic` (38P), `raw/corrections` (3P), and `raw/cross-session` (4P). But on `raw/ci`, it still produced 60 inconclusives and 39 hard fails from 110 trajectories. That is a lot of noise from the strongest model in the batch.\n\nThe report makes an observation about this that is worth pulling out: raw breadth without replay clarity is not yet evidence. The product can *process* broad raw corpora, but if it mostly yields inconclusive judgments, that is not the same as producing trustworthy results. Broad processing coverage is good. But broad coverage that produces mostly inconclusives is not yet the strongest evidence of product trustworthiness.\n\nThe cloud runs gave us much cleaner evidence than the early local-only picture:\n\n`golden` and `failures/positive`\n`corrections` and broad raw corpora\nOne model stood out: `meta-llama/llama-3.1-8b-instruct` was the strongest cost-effective model in the entire batch. On `failures/positive`, it produced 22 pass / 5 inconclusive / 3 fail — the best single result on the most practical extraction benchmark. On `noisy`, it produced 5 pass / 0 inconclusive / 0 fail — a clean sweep. On `raw/opencode`, it produced 18 pass — more than any other model.\n\nThe report's verdict: best current cost/performance benchmark model. Not because it was the most expensive or the largest, but because it consistently outperformed `gpt-4o-mini` on the corpora that matter most for practical rule extraction.\n\nThis is the more important half of the story.\n\nEven with stronger cloud models, the system still struggled on the corpora that matter most for trust: `successes`, `failures/negative`, and `nearmiss`.\n\nThe simplest local-vs-cloud safety view:\n\n| Safety-sensitive corpus | Best local outcome | Best cloud outcome | What it suggests | \n|---|---|---|---|\n| `successes` | still weak | still weak | replay and restraint remain product issues | \n| `failures/negative` | none were convincingly strong | none were convincingly strong | stronger generation does not solve rejection quality | \n| `nearmiss` | mixed | mixed | trigger specificity remains a hard problem | \n\nOn `successes`, the strongest cloud model produced 3 pass / 6 inconclusive / **11 fail**. Those 3 \"passes\" are not a win — they are false positives. A \"pass\" on the `successes` corpus means the system extracted a rule from a trajectory that should not have produced one. Every pass on `successes` is a safety violation.\n\nOn `failures/negative`, the strongest cloud model produced 3 pass / 1 inconclusive / 6 fail. Every model in the field test produced 5-6 hard fails on this corpus, and no model produced more than 3 passes. The rejection gate is weak regardless of which model drives extraction.\n\nOn `nearmiss`, Cloud Llama 3.1 8B produced 6 pass / 1 inconclusive / 7 fail. Better than the locals, but still more failures than passes. The trigger is firing on cases that look similar but should not trigger.\n\nIf cloud models had cleaned those up, I would be writing a very different article. They did not.\n\nThis is the insight that changed how I think about the roadmap.\n\nBefore the cloud runs, it was possible to believe the remaining weakness was mostly a model problem. Small local models, limited instruction-following, noisy output — maybe a bigger model would fix it.\n\nAfter the cloud runs, that explanation is dead. A stronger model with near-perfect parse reliability, running on a now-trustworthy benchmark, still produced 11 fails on `successes` and could not reliably reject bad candidates on `failures/negative`. The problem is not that the model cannot generate rules. The problem is that the product's judgment — when to extract, when to reject, when to stay silent — is not strong enough.\n\nThe report identifies six specific gaps between the current state and a stronger release claim:\n\n`successes`, `failures/negative`, and `nearmiss`.\nThe first three are the real release blockers. The others matter, but safety, replay trust, and promotion confidence are what define whether the product can honestly claim v0.1.0 field-test success in the strongest sense.\n\n**Better models improve the ceiling, not the floor.** The cloud models lifted the best results — more passes on `failures/positive`, cleaner `noisy` handling, stronger raw corpus extraction. But they did not lift the worst results. `successes` and `failures/negative` stayed weak. That tells you the floor is a product problem, not a model problem.\n\n**A \"pass\" on a safety corpus is not always a win.** On `successes`, a pass means the system extracted a rule from a trajectory that should not have produced one. Counting passes without reading the corpus label gives you a false sense of progress. You have to know what each corpus is testing before you interpret the number.\n\n**Inconclusives are the silent killer.** `gpt-4o-mini` produced 248 inconclusives out of 394 candidates — 63% of its output was uncommitted. Those inconclusives did not show up as failures, so they did not trigger alarm. But they also did not produce usable rules. A model that is 63% inconclusive is not producing 77 passes. It is producing 77 passes and 248 maybes. Maybes do not build trust. And the report is clear that some of those maybes are a replay-engine limitation, not a model limitation — the matcher is the named #1 bottleneck. But on the safety corpora, inconclusives are a different signal: they mean the model extracted something from a trajectory where it should have stayed silent, and the matcher could not cleanly reject it. Those are model problems hiding behind engine weakness. You have to split the inconclusive bucket by corpus before you know which ones are the matcher's fault and which ones are the model's.\n\n**The field test succeeded even though the product did not fully pass.** This sounds contradictory but it is not. The field test succeeded because it forced the system through realistic comparisons, exposed tooling defects, validated that fixes improved signal, and produced a coherent map of what remains. The product improved materially. The product is not yet fully field-test complete by a strict release gate. Both of those things are true. That is a good outcome for an honest engineering report.\n\n`gpt-4o-mini` produced zero fails on If the cloud models had solved everything, the conclusion would have been easy: just use a better model. That would be convenient, but it would teach us less.\n\nWhat happened instead was better. The cloud runs improved the benchmark enough to expose the real work that remains. They showed that stronger models help a lot, but they also showed that the core challenge now lives in replay trust, safety, and promotion confidence.\n\nThat is exactly what a serious field test should surface.\n\nThe cloud models made the product look better. They also made its remaining weaknesses harder to deny. That is why we ran them.\n\n**CauterRule v0.1.0 is released.** The full all-model comparison — local vs cloud, curated and raw corpora, winner-by-corpus view, and model recommendations — is in the [field test report](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT.md). The [repo](https://github.com/deghosal-2026/CauterRule) is public. Install with `pip install cauterule`. [Changelog](https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md) · [Release notes](https://github.com/deghosal-2026/CauterRule/releases/tag/v0.1.0)", "url": "https://wpnews.pro/news/a-better-model-improved-the-numbers-it-didn-t-fix-the-product", "canonical_source": "https://dev.to/debashish_ghosal/better-models-showed-us-what-to-build-next-1oj6", "published_at": "2026-09-07 04:50:00+00:00", "updated_at": "2026-09-07 04:57:28.247889+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "developer-tools", "machine-learning"], "entities": ["CauterRule", "meta-llama/llama-3.1-8b-instruct", "openai/gpt-4o-mini", "GitHub", "PyPI"], "alternates": {"html": "https://wpnews.pro/news/a-better-model-improved-the-numbers-it-didn-t-fix-the-product", "markdown": "https://wpnews.pro/news/a-better-model-improved-the-numbers-it-didn-t-fix-the-product.md", "text": "https://wpnews.pro/news/a-better-model-improved-the-numbers-it-didn-t-fix-the-product.txt", "jsonld": "https://wpnews.pro/news/a-better-model-improved-the-numbers-it-didn-t-fix-the-product.jsonld"}}