Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL

Hugging Face Blog · 1 month ago

Hugging Face researchers developed a method to reduce weight synchronization costs in reinforcement learning by sending only changed parameters instead of full model checkpoints, exploiting the fact that 98-99% of weights remain identical between training steps in bf16 format. The per-step payload for a 0.6B model dropped from 1.2 GB to 20-35 MB by encoding sparse deltas as safetensors files uploaded to a Hugging Face Bucket, with the sparse encoding working because bf16's limited precision absorbs most optimizer updates at typical RL learning rates. This enables distributed training where the trainer, inference server, and environment can operate independently in different regions by routing weight updates through a shared object store rather than requiring direct connectivity.