{"slug": "a-lesson-about-retries-hidden-in-the-deepseek-v4-paper", "title": "A lesson about retries, hidden in the DeepSeek-V4 paper", "summary": "An experiment by an unnamed researcher found that retrying failed requests in LLM benchmarks introduces selection bias, shortening average response lengths by 19.2% and reducing long-form outputs by up to 32.5%, as warned in the DeepSeek-V4 paper. The researcher generated 100,000 poems with DeepSeek-V4-Flash for $6.06, simulated 10% random failures using a Poisson process, and observed that retries disproportionately replaced long responses with shorter ones. The findings suggest that retries can skew benchmark results, though the impact is negligible for most consumer applications.", "body_md": "The DeepSeek-V4 paper contains an unexpected lesson for anyone running LLM benchmarks: failed requests are not always safe to retry.\n\nSo I decided to check it myself. 100,000 AI poems later, here’s what I found.\n\nDeepSeek’s warning\n\nHere’s the paragraph from the DeepSeek-V4 paper that made me curious:\n\nAdding retries is an obvious way to fix issues with reliability. But as DeepSeek noticed, longer requests have a higher chance of being interrupted. When you retry a failed long request, there’s a chance you’ll get a short response as a replacement.\n\nRetries make AI poems shorter\n\nExperiment\n\nSince statistical biases can often be hard to grasp, let’s see how this affects a real run.\n\nI asked DeepSeek-V4-Flash to generate 100,000 poems, haikus, or other literary works for me:\n\nWrite a complete piece of literature in one randomly chosen form: a one-line poem, a haiku, a novel chapter, or a limerick.\n\nIt cost me $6.06 (V4-Flash is cheap!) and generated a large variety of responses:\n\nOn average it generated a 74-word text, but the results varied widely:\n\n73.2% haikus. Very short: 15 words on average.\n\n11.5% novel chapters. 34 times longer: 503 words on average.\n\nSimulating failures\n\nNow let’s simulate what would happen if the LLM inference were really unreliable and 10% of requests failed by being interrupted randomly during generation.\n\nI used a statistical model called a Poisson process to model this behavior; an average time between interruptions of 31 seconds makes 10% of requests fail in my experiment.\n\nAs the DeepSeek paper noted, longer requests are affected more often. You can use this formula for a Poisson process to calculate it:\n\nP(failure during request)=1−e−request time/mean time between interruptions\n\nFor example, haikus take 2.38 seconds on average to generate, so their failure rate is 1−e−2.38/31≈7.4%. But novel chapters are longer (11.52 seconds), so their failure rate is higher: 1−e−11.52/31≈31.0%.\n\nYou can see how the failure rate rises for longer requests:\n\nAdding retries\n\nSo let’s see what would happen if I added retries. I artificially simulate failures and perform retries on the failed requests until they succeed.\n\nAfter running the simulation, the resulting dataset looks noticeably different! Just as the DeepSeek authors warned us:\n\nNo failures\n\nFailures + retries\n\nChange\n\nAverage word count\n\n73.51\n\n59.40\n\n−19.2%\n\nResponses ≥ 600 words\n\n2,904\n\n1,961\n\n−32.5%\n\nNovel chapters\n\n11,499\n\n8,919\n\n−22.4%\n\nThe average response is now noticeably shorter and there are fewer longer-form responses. To better understand why it changed so much, let’s take a look at the requests that failed and what happened when they were retried:\n\nAfter retries, (would-be) long responses often become shorter responses. The right-hand side of the chart gets affected the most.\n\nLooking at it with a statistics toolset, adding retries changed the final distribution of the data. What we’re seeing here is a form of selection bias: the retried sample was not representative - long responses were overrepresented in it. Survivorship bias is also a good perspective: the final dataset includes only responses that survived some filtering process - in this case, interruptions that penalized long requests.\n\nConclusion\n\nShorter poems might sound innocent, but the same problem could be dangerous in a real benchmark. A longer request might be stuck in an endless reasoning loop or headed down the wrong path, and retrying it might give the model a second chance - raising the score.\n\nArmed with that knowledge, we have 3 ways to deal with the issue:\n\nThe DeepSeek authors architected their system to resume interrupted requests rather than regenerate them from scratch.\n\nFailures that happen truly randomly (independently of request length) can be safely retried.\n\nThis problem can be important for benchmarks or research, but for most consumer-facing applications the difference doesn’t really matter.\n\nWhat started as a curious warning in the DeepSeek-V4 paper became a surprisingly intuitive lesson in statistics for me.", "url": "https://wpnews.pro/news/a-lesson-about-retries-hidden-in-the-deepseek-v4-paper", "canonical_source": "https://quesma.com/blog/hidden-lesson-deepseek-paper/", "published_at": "2026-07-31 00:00:00+00:00", "updated_at": "2026-07-31 15:43:36.334442+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-safety"], "entities": ["DeepSeek-V4", "DeepSeek-V4-Flash"], "alternates": {"html": "https://wpnews.pro/news/a-lesson-about-retries-hidden-in-the-deepseek-v4-paper", "markdown": "https://wpnews.pro/news/a-lesson-about-retries-hidden-in-the-deepseek-v4-paper.md", "text": "https://wpnews.pro/news/a-lesson-about-retries-hidden-in-the-deepseek-v4-paper.txt", "jsonld": "https://wpnews.pro/news/a-lesson-about-retries-hidden-in-the-deepseek-v4-paper.jsonld"}}