How we keep GPUs reliable across Databricks AI

Databricks · 3 months ago · 31

Databricks describes its gpu-monitor system for maintaining GPU reliability across thousands of processors running distributed training workloads. The system uses three layers of health checks: active bootstrap tests when nodes start (GPU compute validation, peer connectivity, NCCL bandwidth), passive continuous monitoring during workloads (thermal throttling, fabric port status, XID errors), and periodic multi-node tests on idle nodes with NCCL collectives at payload sizes from 8 bytes to 2 GB. By catching failures before they crash training runs, the health check architecture enables reliable operation despite GPU failure rates approximately 1% per year, meaning a 1,024-GPU job has a 57% probability of encountering at least one failure event over a 30-day run.