OpenAI engineers spent weeks trying to explain mysterious crashes in Rockset, the C++ data infrastructure service that powers ChatGPT's search and data plugins. Functions appeared to return to bogus memory addresses. Stack pointers seemed to shift by 8 bytes mid-execution. Every hypothesis the team could construct had strong evidence against it. The bug seemed impossible.
What they assumed was one bug turned out to be two unrelated bugs, coincidentally discovered at the same time. The breakthrough came not from deeper inspection of individual crashes but from switching to what the team calls epidemiological debugging: building a pipeline to automatically analyze every production core dump from the past year, then looking for population-level patterns instead of reasoning about individual cases.
The team had ChatGPT write a script that downloaded a prefix of each core file, extracted registers, filtered known false positives, and labeled each crash as return-to-null, misaligned-stack, or other. They ran it in parallel over every Rockset core dump from the previous year. Correlations appeared immediately. What looked like one syndrome was actually two distinct crash populations with completely different signatures.
The misaligned-stack crashes all came from one Azure region, had a clear start date, and never appeared on long-running nodes. The team traced them to a single physical host where the CPU was silently producing incorrect results. Not overheating, not throwing machine-check exceptions just quietly getting math wrong. Once that host was removed from service, the misaligned-stack crashes disappeared entirely.
With the hardware crashes separated out, the remaining return-to-null crashes became tractable. The team had previously ruled out C++ exception unwinding as a cause because they thought they had counterexamples: crashes in code paths that didn't use exceptions. But those counterexamples all came from the hardware-corruption cluster. Once that contamination was removed, every remaining crash was happening during exception unwinding.
The root cause was a race condition in GNU libunwind's _Ux86_64_setcontext function that had been present for 18 years. During C++ exception unwinding, libunwind synthesizes a ucontext_t struct on the stack, fills in the desired register state, then calls _Ux86_64_setcontext to transfer control to the cleanup handler. The problem: _Ux86_64_setcontext updates the stack pointer (%rsp) to point to the new stack frame before it finishes reading the instruction pointer from the old struct. The moment %rsp changes, the struct is no longer part of the active stack and is no longer protected by the kernel's red zone guarantee. If a signal arrives in that exact window between the %rsp update and the %rip read the kernel builds its signal frame on top of the struct, corrupting the instruction pointer. The function then jumps to NULL or garbage.
The race window is exactly one instruction wide. At modern clock speeds, that's roughly 100 picoseconds. In most programs it would never trigger. OpenAI's Rockset uses timer_create to deliver a SIGUSR2 signal every few milliseconds of CPU time for lightweight per-query accounting, creating far more signal delivery events than typical applications. That frequency turned a theoretically possible race into a production crash.
The team upstreamed a fix and a self-contained reproducer to GNU libunwind, and verified that other unwinders (libgcc) don't have the same issue. The fix reorders the instructions so that %rip is read before %rsp is updated, eliminating the window entirely.
The team's own summary of the lesson is worth repeating in full:
The most important step was not the clever assembly reading or deep knowledge of the details. It was building a high-quality data set. In the absence of this data set, we were mixing two distinct phenomena into one story and trying to reason our way out of the confusion. Once we had accurate and complete population data, the structure of the problem became obvious.
For any team debugging production crashes that resist explanation: check whether you're conflating multiple bugs. Symptoms that seem inconsistent with every hypothesis might not be inconsistent at all ; they might be consistent with two different hypotheses that you're accidentally mixing. The fastest path to seeing the structure is not deeper analysis of individual cases but complete, labeled data across the entire population of failures. The full engineering blog post includes detailed stack diagrams, the vulnerable assembly instructions, and the crash-rate visualizations that revealed the two populations.