This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
It was 11:14 PM.
My friend DM'd me on Twitter: "Your app just hung for 30 seconds, spun indefinitely, and then completely died."
I opened the browser DevTools console pointed at production and saw it — the screen flooded in red:
GET https://careerpilot-ai.run.app/api/analyze-career
net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)
The Server-Sent Events stream powering CareerPilot AI was systematically collapsing on Google Cloud Run. And I had no idea why.
Locally on localhost:3000
, the agentic pipeline was a masterpiece. The multi-stage reasoning logs streamed gracefully — Step 1 flowed into Step 6, the final structured JSON payload arrived within seconds, the UI lit up with a complete personalized career roadmap. Beautiful.
But once deployed behind Google's Front End (GFE) proxy, the pipeline was a graveyard of broken sockets.
CareerPilot AI runs a six-stage agentic pipeline on every career analysis request. Instead of firing a single long-running prompt to Gemini and making the user stare at a blank screen for 20+ seconds, we designed a Server-Sent Events logging stream to broadcast real-time reasoning steps directly to the browser — giving the interface the feel of a live, active mentor thinking out loud.
Once the final stage (Self-Evaluation & Constraint Validation) completed, the backend constructed a massive, nested 15KB JSON payload containing the personalized roadmap: skill weightings, role benchmarks, resource links, and a 30-day milestone calendar.
Here was the delivery mechanism — and the landmine hiding inside it:
// server.ts — The vulnerable streaming channel
app.get("/api/analyze-career", async (req, res) => {
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
// Stream intermediate reasoning logs per step
for (let step = 1; step <= 6; step++) {
const log = await executeAgentStep(step);
res.write(`data: ${JSON.stringify({ step, text: log })}\n\n`);
}
// The fatal moment: sending a 15KB structured JSON payload
res.write(`data: ${JSON.stringify({ finalResult })}\n\n`);
res.end();
});
Everything looked fine. Until it wasn't.
Our first instinct was reasonable: the Express response buffer or the compression middleware was throttling chunks and causing a timeout. The fix seemed obvious — force-flush every payload as soon as it was written.
We imported the compression middleware and started manually calling res.flush()
after every single log line:
// The "intuitive" (but catastrophic) mistake
res.write(`data: ${JSON.stringify({ step, text: log })}\n\n`);
res.flush(); // Force-flushed chunks into arbitrary TCP packet boundaries!
It made things infinitely worse.
Instead of hanging for 30 seconds and then failing, the client now crashed immediately on almost every single attempt. By aggressively forcing flushes on every minor string, we triggered something far uglier: violent TCP packet fragmentation.
The large final JSON payload was being split mid-byte across arbitrary network packet boundaries. Our client-side reader, which naively assumed that each incoming SSE chunk was a complete, fully-formed JSON object, was receiving this:
// Chunk 1 — Mutilated
{"finalResult":{"skills":[{"name":"Python","category":"foun
// Chunk 2 — The rest, arriving milliseconds later
dational","matched":true}],"weeklyPlan":[]}}
JSON.parse()
threw a SyntaxError: Unexpected end of JSON input
on Chunk 1. The entire React UI state machine shattered. The app froze. The user saw a broken screen.
We had fixed a timeout by turning it into an instant crash. Brilliant.
That's when it clicked — we weren't fighting one bug. We were fighting two completely separate infrastructure forces compounding each other in production:
1. Proxy Buffering
Google Cloud Run's GFE proxy holds small SSE packets in a buffer waiting for more data before flushing downstream. Unless explicitly instructed otherwise, it optimizes throughput by batching chunks — not passing them through immediately. This explained the initial 30-second hang: the proxy was collecting all six of our log lines and sitting on them before sending anything to the client.
2. TCP Stream Fragmentation
An SSE stream is not a guaranteed series of atomic messages — it is a continuous raw byte stream. When sending a 15KB payload, the network transport layer is fully entitled — and will — split that across multiple TCP packets. The frontend must treat incoming data as a byte accumulator, not a ready-to-parse event queue.
We had fixed neither. Our flush attempt had only worsened the second while accidentally bypassing the first.
The solution required changes on both sides of the wire simultaneously.
We added the HTTP header that explicitly instructs modern reverse proxies — including Nginx and GFE — to pass bytes through immediately without buffering:
// server.ts — Patched SSE initialization
res.writeHead(200, {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache, no-transform", // 'no-transform' blocks compression proxies from buffering
"Connection": "keep-alive",
"X-Accel-Buffering": "no" // Explicit bypass for Nginx/GFE
});
We also disabled global Gzip compression specifically for the /api/analyze-career
route, ensuring the compression layer wasn't holding bytes in its own internal buffer before the proxy even saw them.
onmessage
, Build a Byte Accumulator
We abandoned the naive EventSource.onmessage
listener entirely. Instead, we wrote a low-level stream reader that accumulates incoming bytes in a staging buffer and only processes complete SSE frames — identified by double-newline (\n\n
) boundaries — never attempting to parse a fragment:
// src/App.tsx — Boundary-safe chunk reassembly
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split("\n\n");
// Keep the last, potentially incomplete fragment staged in the buffer
buffer = lines.pop() || "";
for (const line of lines) {
if (!line.startsWith("data: ")) continue;
const jsonStr = line.replace("data: ", "").trim();
try {
const parsed = JSON.parse(jsonStr);
// Safe: only complete, boundary-verified JSON ever reaches state
} catch (e) {
// Fragment guard: push back and wait for remaining bytes
console.warn("Partial chunk detected — staging for next read cycle.");
}
}
}
The key insight: lines.pop()
pulls the trailing incomplete fragment out of the processing loop and holds it in buffer
until the next network read brings the rest of the bytes. No more mid-payload parsing. No more crashes.
After pushing both fixes to production, we ran a load test simulating 100 concurrent users hitting the full six-stage agentic pipeline simultaneously.
The numbers were stark:
| Metric | Before | After |
|---|---|---|
| Pipeline Completion Rate | 67% | 99.2% |
| Time-to-First Log (FCP) | 28 seconds | 1.1 seconds |
| JSON Syntax Crashes | Constant | Zero |
I DM'd the user back at 1:30 AM. They tested it again and replied: "okay yeah this is actually really cool now."
The real lesson here wasn't about SSE or TCP fragmentation specifically. It was about the gap between local development and containerized production.
Locally, you are your own proxy. There is no GFE, no Nginx, no buffering layer between your Node process and the browser. Every byte arrives instantly. Every test passes. Everything feels perfect.
The moment you deploy behind a reverse proxy, your assumptions about when data arrives collapse. Streaming architectures that rely on chunk atomicity — SSE, WebSockets, chunked transfer encoding — need to be designed defensively from the first line of code, not patched reactively at 11 PM when a user DM slides in.
Build your readers as byte accumulators. Treat every incoming chunk as potentially incomplete. Use X-Accel-Buffering: no
from day one on any SSE endpoint that goes near a proxy. And test in an environment that actually mirrors production — because localhost will lie to you every time.
CareerPilot AI is an open-source AI-powered career roadmap agent. You can explore the full codebase at github.com/aashitanegii/-CareerPilotAI and try the live app at careerpilot-ai-667889155113.asia-southeast1.run.app.