Core dump epidemiology: fixing an 18-year-old bug

Rare crashes are the worst kind of bug. They happen once every few thousand runs, leave behind a core dump nobody has time to read, and vanish before anyone can reproduce them. OpenAI's infrastructure team decided the fix wasn't better luck — it was better statistics. Instead of poring over a single crash dump like a detective with one clue, they pulled together large batches of them and looked for patterns across the whole set, treating the problem less like debugging and more like epidemiology.

That shift in approach paid off fast. Aggregating core dumps let engineers spot commonalities that were invisible in isolation: the same memory address showing up, the same stack pattern, the same subtle timing. It pointed them toward a faulty piece of hardware quietly corrupting data in a way that only showed up under specific load conditions, the kind of failure that's nearly impossible to catch with a single dump and a debugger.

But the more striking find was a software bug that had apparently been sitting in the codebase, or in code inherited from it, for 18 years. Nobody had noticed because it triggered so rarely and so unpredictably that it never surfaced in normal testing or monitoring. It took the volume and consistency of analyzing many crashes together to make an 18-year-old fault finally visible.

The episode says something about the scale OpenAI now operates at. Running enough machines, for enough hours, means even one-in-a-million failure modes show up often enough to study statistically. That's a very different debugging regime than most software teams ever have to deal with, and it changes what counts as a solvable problem versus permanent background noise.