From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub

Hugging Face Blog · 1 year ago

Hugging Face's Xet team implemented a block-based deduplication system using content-defined chunking to accelerate file uploads and downloads on its model hub. For a 191GB repository of quantized model variants, the system reduced stored size to 97GB and cut upload time from 509 minutes to 258 minutes at 50MB/s, roughly a 2x speedup. The approach bundles chunks into 64MB blocks and uses key chunks as an index to avoid making millions of individual network requests, enabling faster iteration for AI builders uploading and downloading large files.