How we keep GPUs reliable across Databricks AI

Distributed GPU training used to be the domain of a handful of frontier labs. Now it's just Tuesday for a lot of engineering teams, and that shift has quietly turned hardware reliability into one of the hardest problems in AI infrastructure. Databricks AI just published a detailed look at how it keeps its fleet running, and the numbers alone explain why this matters: assume a conservative 1% annualized failure rate per GPU, and a 256-GPU job running for 30 days has roughly a 19% chance of hitting a failure. Push that to 1,024 GPUs and the odds jump to 57%. At that scale, something breaking isn't an edge case. It's the baseline assumption.

The failures themselves split into three flavors, and only one of them is easy. Crashed jobs announce themselves loudly, usually via the dreaded NCCL watchdog timeout, though the timeout message itself says almost nothing about the actual cause buried somewhere in hardware, fabric, or software. Silent slowdowns are worse: a degraded GPU keeps chugging, loss keeps trending down, and nobody notices that one bad card is quietly bottlenecking an entire cluster's throughput. Numerical corruption is the nastiest of the three — bit flips that ECC can't catch, propagating through training until they surface as NaN losses or, more insidiously, a model that looks fine and just quietly underperforms.

One anecdote in the post captures how brutal the edge cases get. A training run crashed seven hours in with a standard NCCL timeout. The root cause: a single InfiniBand port flapped once, recovered, and never flapped again. That shouldn't have mattered — except NCCL_IB_TIMEOUT, the transport-layer timeout that governs how long a connection waits for a downed port, works out to about seven seconds by default. Far shorter than the ten-minute PyTorch watchdog most teams tune for and assume is the relevant number. Once that seven-second window blew past, the collective was already dead; the watchdog just showed up late to confirm it. Databricks retuned its IB timeout defaults and now treats cumulative port downtime, not flap count, as the real signal worth acting on.

The fix isn't a single clever patch, it's a system: gpu-monitor, a three-layer health check that runs across the entire node lifecycle. Active bootstrap checks run every time a node is provisioned or cleaned, testing everything from NVLink peer connectivity to ECC memory headroom, and any node that fails gets yanked before a workload ever touches it. Passive continuous checks watch running nodes for the slow-burn stuff — thermal throttling, PCIe errors, NVSwitch faults — the failures that only show up under real load. And periodic multi-node checks run NCCL bandwidth probes across idle nodes, sweeping payload sizes from 8 bytes to 2 gigabytes, because a fabric issue that shows up at 1KB messages can be invisible at 128MB and vice versa.

What's notable is how Databricks stress-tests all of this: not with synthetic benchmarks, but by running its own weird, demanding workloads — RL training loops, agentic coding evaluations, document-intelligence pipelines chewing through images — on the same platform customers use. Odd workloads surface odd failure modes faster than clean ones do, and that seems to be the actual point of the exercise.