The DeepSeek-V4 paper contains an unexpected lesson for anyone running LLM benchmarks: failed requests are not always safe to retry.
So I decided to check it myself. 100,000 AI poems later, here’s what I found.
DeepSeek’s warning
Here’s the paragraph from the DeepSeek-V4 paper that made me curious:
Adding retries is an obvious way to fix issues with reliability. But as DeepSeek noticed, longer requests have a higher chance of being interrupted. When you retry a failed long request, there’s a chance you’ll get a short response as a replacement.
Retries make AI poems shorter
Experiment
Since statistical biases can often be hard to grasp, let’s see how this affects a real run.
I asked DeepSeek-V4-Flash to generate 100,000 poems, haikus, or other literary works for me:
Write a complete piece of literature in one randomly chosen form: a one-line poem, a haiku, a novel chapter, or a limerick.
It cost me $6.06 (V4-Flash is cheap!) and generated a large variety of responses:
On average it generated a 74-word text, but the results varied widely:
73.2% haikus. Very short: 15 words on average.
11.5% novel chapters. 34 times longer: 503 words on average.
Simulating failures
Now let’s simulate what would happen if the LLM inference were really unreliable and 10% of requests failed by being interrupted randomly during generation.
I used a statistical model called a Poisson process to model this behavior; an average time between interruptions of 31 seconds makes 10% of requests fail in my experiment.
As the DeepSeek paper noted, longer requests are affected more often. You can use this formula for a Poisson process to calculate it:
P(failure during request)=1−e−request time/mean time between interruptions
For example, haikus take 2.38 seconds on average to generate, so their failure rate is 1−e−2.38/31≈7.4%. But novel chapters are longer (11.52 seconds), so their failure rate is higher: 1−e−11.52/31≈31.0%. You can see how the failure rate rises for longer requests:
Adding retries
So let’s see what would happen if I added retries. I artificially simulate failures and perform retries on the failed requests until they succeed.
After running the simulation, the resulting dataset looks noticeably different! Just as the DeepSeek authors warned us:
No failures
Failures + retries
Change
Average word count
73.51
59.40
−19.2%
Responses ≥ 600 words
2,904
1,961
−32.5%
Novel chapters
11,499
8,919
−22.4%
The average response is now noticeably shorter and there are fewer longer-form responses. To better understand why it changed so much, let’s take a look at the requests that failed and what happened when they were retried:
After retries, (would-be) long responses often become shorter responses. The right-hand side of the chart gets affected the most.
Looking at it with a statistics toolset, adding retries changed the final distribution of the data. What we’re seeing here is a form of selection bias: the retried sample was not representative - long responses were overrepresented in it. Survivorship bias is also a good perspective: the final dataset includes only responses that survived some filtering process - in this case, interruptions that penalized long requests.
Conclusion
Shorter poems might sound innocent, but the same problem could be dangerous in a real benchmark. A longer request might be stuck in an endless reasoning loop or headed down the wrong path, and retrying it might give the model a second chance - raising the score.
Armed with that knowledge, we have 3 ways to deal with the issue:
The DeepSeek authors architected their system to resume interrupted requests rather than regenerate them from scratch.
Failures that happen truly randomly (independently of request length) can be safely retried.
This problem can be important for benchmarks or research, but for most consumer-facing applications the difference doesn’t really matter.
What started as a curious warning in the DeepSeek-V4 paper became a surprisingly intuitive lesson in statistics for me.