Decoupled DiLoCo: A new frontier for resilient, distributed AI training

Google DeepMind · 2 months ago

Google introduced Decoupled DiLoCo, a distributed training architecture that divides large language model training across separate compute clusters with asynchronous data flow instead of requiring tight synchronization across thousands of chips. The system trained a 12 billion parameter model across four U.S. regions using only 2-5 Gbps of bandwidth while achieving training more than 20 times faster than conventional synchronization methods. The approach enables AI training to tolerate hardware failures in isolated clusters without interrupting overall progress, and allows mixing different hardware generations in a single training run.