Next.js API Proxy Times Out After Long ML Inference (502) — Navigating Undici's Timeout Quagmire

A developer at Workstyle Tech documented how Next.js API routes proxying requests to a FastAPI inference service returned 502 errors during long-running audio generation because the global fetch (undici) has a default timeout of about 300 seconds. The team rewrote the proxy using Node's http/https modules to disable response timeouts while enforcing only connection timeouts, and added configuration to suppress Next.js's response limit warnings. The fix ensures that inference jobs lasting minutes or hours are no longer cut off by the proxy layer.

📝 Originally published in Japanese at forge.workstyle.tech . When using Next.js to relay requests from the frontend to an inference service FastAPI running in a separate process via API routes, it's a common BFF Backend for Frontend architecture. However, when running audio generation with a 44.1kHz model on the CPU, processing can take several minutes — or even hours. And then, at some point, this happens: The frontend receives a 502error even though the generation hasn't finished. Checking the inference service logs shows that processing is still running smoothly. The issue isn't with the inference itself — the bottleneck is the proxy in between . This article documents how we identified that the culprit was Next.js's global fetch powered by undici and its default timeout, and how we rewrote the proxy using Node's standard http / https modules to bypass it. The key observations during debugging were: The inference service continues running, but the proxy layer gives up first. The fact that it fails "after a fixed time" is a strong hint: there’s a hardcoded timeout somewhere. fetch undici When you use fetch directly in a Next.js API route, under the hood it uses undici , Node.js’s built-in HTTP client. Undici has a default timeout for receiving headers around 300 seconds , and if no response comes back within that window, it forcibly closes the connection. When inference takes more than 5 minutes, it gets cut off right there — resulting in a 502. "Just disable undici’s timeout then " We tried adjusting headersTimeout / bodyTimeout via an Agent , but ran into another wall: it’s hard to directly import and inject undici in this setup . Modifying the internal implementation of global fetch isn’t clean, and the override may not work across environments. Temporarily increasing the timeout just kicks the can down the road — eventually, another long-running generation will hit the same limit. The real fix wasn’t to tweak the timeout — it was to rewrite the proxy layer using a different timeout model entirely . http / https Instead of fighting with undici, we rewrote the proxy using Node’s built-in http / https modules. With standard modules, we gain full control over timeout behavior. The key design principle: distinguish between connection setup and response waiting . This separation is critical. // Disable response timeout entirely allow long-running generation . Only enforce connection timeout. preq.setTimeout 0 ; preq.on 'socket', s = { s.setTimeout 30 000, = { if s.destroyed && s.connecting || s.writable s.destroy new Error 'connect timeout' ; } ; s.once 'connect', = s.setTimeout 0 ; } ; Breaking it down: preq.setTimeout 0 — disables the overall request timeout, allowing the response to be awaited indefinitely 30 000ms timeout that triggers only if the socket is still connecting or not writable — i.e., connect fires, we immediately disable the socket timeout setTimeout 0 , letting the response stream in however long it takesThis way: We also added proper error handling: js preq.on 'error', e = { if res.headersSent res.status 502 .json { error: backend API unreachable: ${String e } } ; resolve ; } ; Even after removing the timeout, Next.js API routes have a built-in mechanism that warns when a handler takes too long to respond. To suppress this warning for long-running proxies, we declare: js export const config = { api: { bodyParser: { sizeLimit: '25mb' }, responseLimit: '25mb', externalResolver: true, // Tell Next.js: "This route resolves externally; don't monitor it" }, }; We also increased the body and response size limits since we’re dealing with audio data — the defaults would reject large recordings. fetch uses undici under the hood fetch in API routes without realizing it has default timeouts, you’ll end up with a hard-to-debug scenario: "The service is running, but the proxy returns 502." http / https and managing the socket directly is more reliable and readable for this kind of requirement. externalResolver: true fetch undici http / https setTimeout 0 on connect to fail fast only when unreachable externalResolver: true