{"slug": "best-of-n-is-prepaid-retries-the-cost-math-of-racing-parallel-attempts", "title": "Best-of-N is prepaid retries: the cost math of racing parallel attempts", "summary": "A developer's analysis of best-of-N sampling for LLM agents shows that firing N parallel attempts costs 2.9x the tokens of sequential retry for the same success rate, because best-of-N pays for all attempts unconditionally. The tradeoff is only worthwhile when latency has a price and attempts fail independently, a variable most teams never measure.", "body_md": "*Originally published on Loop & Retry — field notes on building LLM agents that survive production.*\n\nHere's the pitch for best-of-N: instead of trying once and retrying on failure, fire off N attempts at the same task simultaneously and keep whichever one finishes first (or scores highest). You've turned a serial wait into a parallel one, so your tail latency drops — no more [waiting through however many retries it takes](https://loopandretry.github.io/posts/your-agents-p99-is-a-different-animal/?ref=devto) before one succeeds. The catch that doesn't show up in the pitch: you pay for all N attempts every single time, whether you needed them or not, and *when* that payment stops being worth it depends entirely on a variable most teams never measure — whether your attempts actually fail independently of each other.\n\n[The retry-budgets post](https://loopandretry.github.io/posts/retry-budgets/?ref=devto) modeled sequential retry cost: try, and on failure, try again, accumulating the failed attempt's tokens into the transcript each time. Best-of-N is structurally different — there's no accumulation, because the N attempts don't see each other. Each one starts fresh from the same prompt and runs to completion independently. That makes the accounting simpler, which is exactly what makes the tradeoff easy to misjudge.\n\n``` python\nimport random\n\ndef cost_sequential(p_fail, tokens_per_attempt, max_retries=4):\n    \"\"\"Expected token cost of retrying serially until success or cap.\"\"\"\n    cost = 0\n    for attempt in range(max_retries):\n        cost += tokens_per_attempt\n        if random.random() >= p_fail:\n            return cost, attempt + 1   # succeeded on this attempt\n    return cost, max_retries           # exhausted retries, still failed\n\ndef cost_best_of_n(p_fail, tokens_per_attempt, n):\n    \"\"\"Best-of-N always pays for all N attempts, launched in parallel.\"\"\"\n    cost = tokens_per_attempt * n\n    succeeded = any(random.random() >= p_fail for _ in range(n))\n    return cost, succeeded\n```\n\nRun both at `p_fail = 0.3`\n\n, `tokens_per_attempt = 2000`\n\n, over 20,000 trials, and the naive comparison looks like this:\n\n| Strategy | Mean tokens spent | P(eventual success) |\n|---|---|---|\n| Sequential retry, cap 4 | ~2,800 | 99.2% |\n| Best-of-N, N=4 | 8,000 | 99.2% |\n\nSame success rate, **2.9x the tokens**, every time — not just on the runs that needed all four attempts. Sequential retry only pays for extra attempts when the first one actually fails; best-of-N pays for N attempts unconditionally, because it doesn't know in advance which one will win. That's the fee for parallelism: sequential is priced by *expected* attempts (close to 1 when `p_fail`\n\nis small), best-of-N is priced by *worst-case* attempts, always.\n\nThe fee buys something sequential retry can't: bounded latency. A sequential retry loop's wall-clock time is a sum — it's a random variable whose tail gets long exactly [the way p99 posts warn about](https://loopandretry.github.io/posts/your-agents-p99-is-a-different-animal/?ref=devto), because a bad run means the sum of every failed attempt's latency plus the final success. Best-of-N's wall-clock time is a max across N attempts running concurrently, which is a much better-behaved random variable — its tail barely grows as N increases, because you only need the *fastest* of N to land, not the *last* of a serial chain to succeed. If a human is staring at a loading spinner, that's the number that matters, and it's the reason best-of-N sampling and self-consistency voting are real, published techniques and not just an expensive mistake.\n\nSo the fee is legitimate when latency has a price and independent attempts genuinely raise your odds. The question that decides whether it's a good trade is the one the pitch skips: *how independent are your attempts, really?*\n\n[The $200 postmortem](https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto) turned on one fact: retryability is a property of the specific error, not a default you apply to every error. An HTTP 400 from a malformed request will fail identically no matter how many times or how quickly you retry it, because nothing about the retry changes the thing that's wrong. That fact doesn't go away when you switch from sequential to parallel — it gets *worse*, because parallel execution removes the one thing that occasionally saves you in a sequential loop: a later attempt happening after some upstream state has changed.\n\nExtend the model to make failures correlated instead of independent — a shared cause (a bad system prompt, a broken tool schema, a poisoned upstream fact) that fails every attempt the same way with probability `p_shared`\n\n, on top of ordinary independent noise `p_indep`\n\n:\n\n``` python\ndef cost_best_of_n_correlated(p_shared, p_indep, tokens_per_attempt, n):\n    \"\"\"A shared-cause failure dooms every attempt identically; only the\n    remaining slice of runs benefits from N independent rolls.\"\"\"\n    cost = tokens_per_attempt * n\n    if random.random() < p_shared:\n        return cost, False   # every one of the N attempts fails the same way\n    succeeded = any(random.random() >= p_indep for _ in range(n))\n    return cost, succeeded\n```\n\nAt `p_shared = 0.15`\n\n— a modest 15% chance the failure is systemic rather than transient — best-of-N's success rate caps at 85% no matter how large you make N, because the correlated slice of runs fails identically on every single attempt. You've spent `tokens_per_attempt * n`\n\nrestating the same doomed request N times, in parallel, instead of once. Sequential retry wastes tokens on the same correlated failures too, but it wastes `tokens_per_attempt * max_retries`\n\nat most — best-of-N wastes exactly that much on *every* correlated failure, with no cap-based early exit, because all N attempts fire before any of them can report back.\n\nThis is the same lesson the postmortem already taught, arriving through a different door: **N parallel attempts amortize independent noise and do nothing for a shared cause.** If you don't know your `p_shared`\n\n, adding N doesn't just fail to help — it multiplies the cost of every failure that N can't fix by exactly N.\n\nBest-of-N is a legitimate lever, not a trap to avoid — but only once you've priced it against sequential retry using your own numbers, not the pitch's:\n\n`p_shared`\n\nbefore picking N.The honest framing: best-of-N doesn't dodge the retry-budget math from [the earlier post](https://loopandretry.github.io/posts/retry-budgets/?ref=devto) — it prepays it, in full, on every request, in exchange for a flatter latency tail. That's sometimes exactly what you want. It's never free, and for a shared-cause failure it isn't even a discount.", "url": "https://wpnews.pro/news/best-of-n-is-prepaid-retries-the-cost-math-of-racing-parallel-attempts", "canonical_source": "https://dev.to/loopandretry/best-of-n-is-prepaid-retries-the-cost-math-of-racing-parallel-attempts-2h9i", "published_at": "2026-08-09 15:50:44+00:00", "updated_at": "2026-08-09 16:18:49.997362+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-infrastructure"], "entities": ["Loop & Retry"], "alternates": {"html": "https://wpnews.pro/news/best-of-n-is-prepaid-retries-the-cost-math-of-racing-parallel-attempts", "markdown": "https://wpnews.pro/news/best-of-n-is-prepaid-retries-the-cost-math-of-racing-parallel-attempts.md", "text": "https://wpnews.pro/news/best-of-n-is-prepaid-retries-the-cost-math-of-racing-parallel-attempts.txt", "jsonld": "https://wpnews.pro/news/best-of-n-is-prepaid-retries-the-cost-math-of-racing-parallel-attempts.jsonld"}}