Python requests timeout=60 Let My LLM Stream Run for 38 Minutes

My nightly summarization job had timeout=60 on every request. One call ran for 38 minutes and 12 seconds anyway. Nothing crashed and nothing logged an error. The 6:00 a.m. report just didn't show up.

The Python requests timeout is not a deadline for the whole call. It caps how long you wait for the next bytes. Point it at a streaming LLM endpoint that emits a token every 70 milliseconds and that timeout never fires.

This post covers what I found over 14 nights and 33,600 calls, how I tracked it down, and the 20 lines that fixed it.

requests.post(..., timeout=60) sets a httpx works the same way: Timeout(60) covers connect, read, write and pool separately. There is no total-time cap by default.max_tokens, keep a short read timeout, asyncio.timeout() around the whole call). Nothing exotic here:

The call looked like this, and I'd guess half the LLM scripts on GitHub look the same:

r = requests.post(URL, json=payload, stream=True, timeout=60)
for line in r.iter_lines():
    if line.startswith(b"data: ") and line != b"data: [DONE]":
        chunk = json.loads(line[6:])
        parts.append(chunk["choices"][0]["delta"].get("content") or "")

I read timeout=60 as "this call dies after a minute." That was wrong.

The Python requests timeout sets two limits: how long to wait for the TCP connection, and how long to wait between bytes on the socket. Passing one number sets both to that value. Passing a tuple like (5, 30) sets them separately. Neither one limits total request time.

The requests docs say this outright. The read timeout is the number of seconds the client will wait for the server to send a byte. Every byte that arrives resets that clock.

For a normal JSON API this almost never matters. The server is silent until it has an answer, so "time between bytes" and "total time" are basically the same number.

Streaming changes that. A server streaming tokens is never silent for long. My model produced a chunk roughly every 70 ms under load, so the read timeout restarted about 14 times a second. A 60-second read timeout couldn't fire unless the model stopped completely.

It ran that long because the model got stuck repeating itself, and nothing told it to stop. I hadn't set max_tokens. vLLM's OpenAI-compatible server then allows generation up to whatever context window is left after the prompt. My context was 32k tokens, the prompts were about 1.5k, and that left roughly 30,000 tokens of room.

At around 15 tokens per second per request under batch load, 30,000 tokens takes about 33 minutes. My worst call ran 38. The numbers line up.

The outputs from those calls looked like what you'd expect:

"summary": "Customer reports login failure. Customer reports login failure.
Customer reports login failure. Customer reports login failure. ...

That repeated a few thousand times. Some tickets (very long ones, pasted stack traces, logs full of near-identical lines) push small models into these loops. It's rare, but over 2,400 calls a night it happens.

Here's what 14 nights of logs showed:

0.18% sounds harmless until you remember there were only 6 workers. When 2 or 3 runaway calls landed on the same night, a third to half of my capacity spent half an hour generating "Customer reports login failure." The other workers also slowed down, because every runaway sequence took a slot in vLLM's batch and a chunk of KV cache.

On the 3 late nights, 4 or more workers were stuck at once.

The job wasn't failing, so I had no errors to read. What I had:

py-spy dump on the stuck process.iter_lines() → urllib3 → socket.recv. So they weren't deadlocked. They were busy receiving data./metrics endpoint.vllm:num_requests_running sat at 6 with nothing waiting. The server thought it was doing useful work. Then I wrote a 15-line test server that sent one SSE line every 30 seconds and pointed requests.post(..., timeout=60) at it. It sat there for an hour until I killed it. That settled it.

Set a total timeout by enforcing a wall-clock deadline in your own code. Neither requests nor httpx gives you one. Check elapsed time on every chunk, cap generation with max_tokens, and close the connection when you give up so the server stops generating.

Here's what I run now:

import json
import time
import requests

def stream_completion(payload, deadline_s=90, read_timeout=20):
    payload = {**payload, "max_tokens": 1024, "stream": True}
    start = time.monotonic()
    parts, finish = [], None

    with requests.post(URL, json=payload, stream=True,
                       timeout=(5, read_timeout)) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if time.monotonic() - start > deadline_s:
                raise TimeoutError(f"exceeded {deadline_s}s wall clock")
            if not line.startswith(b"data: ") or line == b"data: [DONE]":
                continue
            choice = json.loads(line[6:])["choices"][0]
            parts.append(choice["delta"].get("content") or "")
            finish = choice.get("finish_reason") or finish

    return "".join(parts), finish

Four things changed, and each one matters:

max_tokens=1024. timeout=(5, 20). deadline_s + read_timeout, here 110 seconds. Know that number before you promise anyone an SLA.with block. Yes. With async code, wrap the entire call in asyncio.timeout() (Python 3.11+). It cancels the task at the deadline whether or not bytes are flowing:

import asyncio
import httpx

async def stream_completion(client: httpx.AsyncClient, payload):
    async with asyncio.timeout(90):
        async with client.stream("POST", URL, json=payload) as r:
            async for line in r.aiter_lines():
                ...

Cancellation unwinds the async with and closes the stream. This is my preferred version. I kept the sync one above because the batch job is threaded and I didn't want to rewrite it at 2 a.m.

If you use an SDK built on httpx, like the OpenAI Python client, read what its timeout argument means for streams in the version you have installed. Don't assume it's a total deadline. Wrap the call in your own deadline either way. It costs one line.

Capping generation doesn't fix the loop. It just makes the loop cheap. In the 14 nights after the fix, 58 calls ended with finish_reason == "length", which in this job always meant a repetition loop.

I handle those explicitly now:

That's 9 tickets a fortnight for a person to look at, instead of 9 GPU-hours of "Customer reports login failure."

Same job, same model, next 14 nights:

finish_reason == "length" as a failure until proven otherwise. Because the Python requests timeout only limits connecting and the gap between received bytes. It never limits total request duration. A streaming LLM response sends a byte every few milliseconds, so the read timeout keeps resetting and a call can run as long as the model keeps generating: 38 minutes in my case, with no max_tokens set. To actually bound a streaming call, set max_tokens, keep a short read timeout, enforce a wall-clock deadline in your own loop or with asyncio.timeout(), and close the connection when the deadline hits so the server stops generating.

Written by the developer behind Preterview, an interview prep platform.