# 4,768 LLM Runs, Zero Lost Sweeps: Hardening a Field-Test Runner for Timeouts, Hangs, and Cost

> Source: <https://dev.to/debashish_ghosal/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs-and-cost-1k24>
> Published: 2026-09-12 18:07:01+00:00

**Update — v0.3.0 released.** CauterRule is now live on [GitHub](https://github.com/deghosal-2026/CauterRule) and [PyPI](https://pypi.org/project/cauterule/). It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. `pip install cauterule` gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The [v0.3.0 field test report](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md) evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. [Release notes](https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.0/release-notes.md) · [Changelog](https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md)

**CauterRule** is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

Here is the failure mode that never makes it into a paper. You launch a 4,768-run benchmark. It churns for an hour. Then one request to the model provider simply never returns — no exception, no error, no timeout. The worker is blocked. The queue drains to zero throughput. You have not lost one trajectory. You have lost the sweep, and the API spend it already consumed.

v0.3.0 is the first CauterRule field test that finished. The reason is not a better prompt. It is five small guards in the runner.

Our original runner encoded three assumptions that hold at 50 runs and fail at 4,768:

**Assumption 1:** the provider will always respond.

**Assumption 2:** a timeout is a transient error, so retry it.

**Assumption 3:** a failed trajectory should be retried until it succeeds.

At scale, assumption 1 produces infinite hangs, assumption 2 doubles the cost of a request that was already dead, and assumption 3 turns one bad input into an unbounded retry loop. None of these are model problems. They are harness problems — and they compound.

The scale that breaks them: **40 corpora × 2 models = 4,768 runs**, roughly **2,384 trajectories per model**, several LLM calls per trajectory.

None of this is clever. It is the set of things you must do before you can trust a large sweep:

| Guard | Implementation | Failure it removes | 
|---|---|---|
| Per-trajectory timeout | `create(..., timeout=120)` | A hung call blocks a worker forever | 
| Non-retryable timeouts | classify timeout as terminal | Retrying a dead request multiplies cost | 
| Token cap | `max_tokens=4096` | Runaway generations blow latency and spend | 
| Quarantine | `CAUTERULE_QUARANTINE_IDS` | Known-bad IDs are skipped, not retried | 
| Cancel on shutdown | `executor.shutdown(wait=False, cancel_futures=True)` | Interrupt hangs on queued work | 

The shape of the concurrency fix:

```
with ThreadPoolExecutor(max_workers=6) as executor:
    futures = {executor.submit(run_trajectory, t): t for t in trajectories}
    ...
# on KeyboardInterrupt / SIGTERM:
executor.shutdown(wait=False, cancel_futures=True)
```

The principle: **one bad request costs one trajectory, not the sweep.**

What the hardening bought, concretely:

| Metric | Value | 
|---|---|
| Trajectory-runs completed | 4,768 | 
| Models | 2 (gpt-4o-mini, llama-3.1-8b) | 
| Corpora | 40 | 
| Safety gate dropped (per model) | 580 trajectories | 
| LLM calls avoided by the gate (per model) | 1,160 | 
| Safety silence (successes / negatives) | 60/60 · 60/60 | 
| Adversarial promotions | 0 | 

The safety number is only credible because every corpus finished. A benchmark that completes its easy corpora and dies on the hard ones is measuring the corpora it could finish.

Once runs stop dying, you can afford to measure spend. v0.3.0 captures token usage on every call and prices it with real provider rates:

```
@dataclass
class LLMResponse:
    text: str
    prompt_tokens: int = 0
    completion_tokens: int = 0
```

| Model | Input ($/1M) | Output ($/1M) | 
|---|---|---|
| gpt-4o-mini | 0.15 | 0.60 | 
| llama-3.1-8b | 0.06 | 0.06 | 

The SDK was already returning `usage` on every response. We were discarding it. Adding two fields turns "what does 1,000 trajectories cost?" from a shrug into a measured answer — the question that actually gates adoption.

`raw/ci`. The field test went cloud-only. The "ship on local models" path from the v0.1.0 report did not hold at 40 corpora.`cross_session.py`, `human_agreement.py`) and runner flags. Neither protocol was executed. A flag that was never pulled is not evidence.
**Throughput is a product feature for evaluation.** Your benchmark's quality ceiling is whether it finishes. A harness that can't survive a hung call can't run the corpora where the interesting failures live.

**Fail-soft beats fail-hard.** Retries that assume transience are a cost bug waiting to happen. Classify failures as terminal vs. transient explicitly.

**Capture usage while you're already holding the client.** Token accounting is trivial when you own the call site and impossible when you don't. Do it from day one.

**Building the instrument is not taking the reading.** We shipped cross-session and agreement tooling and left both protocols unrun, so those questions are still open. Tooling is not measurement.

If you run large evaluations against a flaky external API, the infrastructure is not a distraction from the science — it *is* part of the science.

The most valuable code in CauterRule's v0.3.0 field test was not a matcher or a prompt. It was five guards that let 4,768 requests finish, plus two dataclass fields that let us price them. That is what turned a benchmark that kept dying into one you can trust — and, conveniently, one that can tell you what it costs.

**CauterRule v0.3.0 is released.** The methodology, hardening notes, and cost report are in the [field test report](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md) and the [cost measurement doc](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/cost-measurement.md). The [repo](https://github.com/deghosal-2026/CauterRule) is public. Install with `pip install cauterule`. [Changelog](https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md) · [Release notes](https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.0/release-notes.md)