{"slug": "4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs", "title": "4,768 LLM Runs, Zero Lost Sweeps: Hardening a Field-Test Runner for Timeouts, Hangs, and Cost", "summary": "A developer released CauterRule v0.3.0, an open-source sidecar that learns standing rules from repeated AI agent failures by extracting lessons from trajectories, replay-testing them, and promoting reusable guidance. The project's field test completed 4,768 trajectory-runs across 40 corpora and two models (gpt-4o-mini and llama-3.1-8b) after adding five runner guards — per-trajectory timeouts, non-retryable timeout classification, token caps, quarantine lists, and cancel-on-shutdown — to prevent hung requests from killing entire sweeps.", "body_md": "**Update — v0.3.0 released.** CauterRule is now live on [GitHub](https://github.com/deghosal-2026/CauterRule) and [PyPI](https://pypi.org/project/cauterule/). It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. `pip install cauterule` gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The [v0.3.0 field test report](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md) evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. [Release notes](https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.0/release-notes.md) · [Changelog](https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md)\n\n**CauterRule** is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\n\nHere is the failure mode that never makes it into a paper. You launch a 4,768-run benchmark. It churns for an hour. Then one request to the model provider simply never returns — no exception, no error, no timeout. The worker is blocked. The queue drains to zero throughput. You have not lost one trajectory. You have lost the sweep, and the API spend it already consumed.\n\nv0.3.0 is the first CauterRule field test that finished. The reason is not a better prompt. It is five small guards in the runner.\n\nOur original runner encoded three assumptions that hold at 50 runs and fail at 4,768:\n\n**Assumption 1:** the provider will always respond.\n\n**Assumption 2:** a timeout is a transient error, so retry it.\n\n**Assumption 3:** a failed trajectory should be retried until it succeeds.\n\nAt scale, assumption 1 produces infinite hangs, assumption 2 doubles the cost of a request that was already dead, and assumption 3 turns one bad input into an unbounded retry loop. None of these are model problems. They are harness problems — and they compound.\n\nThe scale that breaks them: **40 corpora × 2 models = 4,768 runs**, roughly **2,384 trajectories per model**, several LLM calls per trajectory.\n\nNone of this is clever. It is the set of things you must do before you can trust a large sweep:\n\n| Guard | Implementation | Failure it removes | \n|---|---|---|\n| Per-trajectory timeout | `create(..., timeout=120)` | A hung call blocks a worker forever | \n| Non-retryable timeouts | classify timeout as terminal | Retrying a dead request multiplies cost | \n| Token cap | `max_tokens=4096` | Runaway generations blow latency and spend | \n| Quarantine | `CAUTERULE_QUARANTINE_IDS` | Known-bad IDs are skipped, not retried | \n| Cancel on shutdown | `executor.shutdown(wait=False, cancel_futures=True)` | Interrupt hangs on queued work | \n\nThe shape of the concurrency fix:\n\n```\nwith ThreadPoolExecutor(max_workers=6) as executor:\n    futures = {executor.submit(run_trajectory, t): t for t in trajectories}\n    ...\n# on KeyboardInterrupt / SIGTERM:\nexecutor.shutdown(wait=False, cancel_futures=True)\n```\n\nThe principle: **one bad request costs one trajectory, not the sweep.**\n\nWhat the hardening bought, concretely:\n\n| Metric | Value | \n|---|---|\n| Trajectory-runs completed | 4,768 | \n| Models | 2 (gpt-4o-mini, llama-3.1-8b) | \n| Corpora | 40 | \n| Safety gate dropped (per model) | 580 trajectories | \n| LLM calls avoided by the gate (per model) | 1,160 | \n| Safety silence (successes / negatives) | 60/60 · 60/60 | \n| Adversarial promotions | 0 | \n\nThe safety number is only credible because every corpus finished. A benchmark that completes its easy corpora and dies on the hard ones is measuring the corpora it could finish.\n\nOnce runs stop dying, you can afford to measure spend. v0.3.0 captures token usage on every call and prices it with real provider rates:\n\n```\n@dataclass\nclass LLMResponse:\n    text: str\n    prompt_tokens: int = 0\n    completion_tokens: int = 0\n```\n\n| Model | Input ($/1M) | Output ($/1M) | \n|---|---|---|\n| gpt-4o-mini | 0.15 | 0.60 | \n| llama-3.1-8b | 0.06 | 0.06 | \n\nThe SDK was already returning `usage` on every response. We were discarding it. Adding two fields turns \"what does 1,000 trajectories cost?\" from a shrug into a measured answer — the question that actually gates adoption.\n\n`raw/ci`. The field test went cloud-only. The \"ship on local models\" path from the v0.1.0 report did not hold at 40 corpora.`cross_session.py`, `human_agreement.py`) and runner flags. Neither protocol was executed. A flag that was never pulled is not evidence.\n**Throughput is a product feature for evaluation.** Your benchmark's quality ceiling is whether it finishes. A harness that can't survive a hung call can't run the corpora where the interesting failures live.\n\n**Fail-soft beats fail-hard.** Retries that assume transience are a cost bug waiting to happen. Classify failures as terminal vs. transient explicitly.\n\n**Capture usage while you're already holding the client.** Token accounting is trivial when you own the call site and impossible when you don't. Do it from day one.\n\n**Building the instrument is not taking the reading.** We shipped cross-session and agreement tooling and left both protocols unrun, so those questions are still open. Tooling is not measurement.\n\nIf you run large evaluations against a flaky external API, the infrastructure is not a distraction from the science — it *is* part of the science.\n\nThe most valuable code in CauterRule's v0.3.0 field test was not a matcher or a prompt. It was five guards that let 4,768 requests finish, plus two dataclass fields that let us price them. That is what turned a benchmark that kept dying into one you can trust — and, conveniently, one that can tell you what it costs.\n\n**CauterRule v0.3.0 is released.** The methodology, hardening notes, and cost report are in the [field test report](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md) and the [cost measurement doc](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/cost-measurement.md). The [repo](https://github.com/deghosal-2026/CauterRule) is public. Install with `pip install cauterule`. [Changelog](https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md) · [Release notes](https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.0/release-notes.md)", "url": "https://wpnews.pro/news/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs", "canonical_source": "https://dev.to/debashish_ghosal/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs-and-cost-1k24", "published_at": "2026-09-12 18:07:01+00:00", "updated_at": "2026-09-12 18:19:49.991827+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "mlops"], "entities": ["CauterRule", "GitHub", "PyPI", "gpt-4o-mini", "llama-3.1-8b"], "alternates": {"html": "https://wpnews.pro/news/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs", "markdown": "https://wpnews.pro/news/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs.md", "text": "https://wpnews.pro/news/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs.txt", "jsonld": "https://wpnews.pro/news/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs.jsonld"}}