How we keep GPUs reliable across Databricks AI
TLDR Dev
Databricks describes its gpu-monitor system for maintaining GPU reliability across thousands of processors running distributed training workloads. The system uses three layers of health checks: active bootstrap tests when nodes start (GPU compute validation, peer connectivity, NCCL bandwidth), passive continuous monitoring during workloads (thermal throttling, fabric port status, XID errors), and periodic multi-node tests on idle nodes with NCCL collectives at payload sizes from 8 bytes to 2 GB. By catching failures before they crash training runs, the health check architecture enables reliable operation despite GPU failure rates approximately 1% per year, meaning a 1,024-GPU job has a 57% probability of encountering at least one failure event over a 30-day run.
Why it matters
GPU failures during distributed training are addressed through a comprehensive health check system at Databricks AI using proactive stress testing and continuous monitoring to detect and mitigate failures.
.png)
