From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub

Hugging Face has a storage problem that most companies would love to have: nearly 45 petabytes spread across 2 million model, dataset, and space repos. The obvious fix for all that redundant data is deduplication, chopping files into small chunks and only storing the unique ones. But the Xet team learned the hard way that going too granular breaks everything else. Chunk everything at roughly 64KB, and a single 200GB repo alone generates 3 million entries. Scale that across the whole Hub and you're staring at 690 billion chunks, a number that would bankrupt any database and choke any network trying to request them one at a time.

So the team's actual insight wasn't about deduplicating harder. It was about admitting deduplication is a means, not the goal. What builders actually want is to upload and download faster and stop babysitting massive files while iterating on models. Once you accept that framing, the fix becomes obvious: stop scaling anything 1:1 with chunk count. Instead of tracking millions of tiny chunks individually, Hugging Face now bundles them into blocks up to 64MB, cutting content-addressed-store entries by roughly a thousandfold. Shards then map files to the chunks inside those blocks, so the system can tell exactly what changed between versions without re-touching everything else.

The cleverest bit is how they avoid querying every single chunk to check if it already exists. They only index a 0.1% sample, so-called key chunks, picked via a simple hash-based rule. Because similar data tends to cluster together, finding one key chunk in a shard usually means its neighbors are sitting right there too, unlocking deduplication without a firehose of network calls. It's a bet on locality over brute force, and it apparently pays off.

The numbers get concrete with GGUF quantized models, a format where Hugging Face already stores 3.5 petabytes. Take bartowski/gemma-2-9b-it-GGUF, 29 quantized variants of Gemma 2 9B totaling 191GB. Because quantization compresses values into narrow integer ranges, and because different quant variants overlap heavily, there's a ton of redundant weight data across versions. Running this through the new Xet pipeline dropped storage to about 97GB, a savings of roughly 94GB from one repo alone. At 50MB/s that's the difference between an 8.5-hour upload and a 4.3-hour one, cutting the wait nearly in half. Downloads benefit too, since local chunk caching means only genuinely new data needs to move.

Hugging Face is rolling this out gradually rather than flipping a switch across the whole Hub, which tracks with how quietly infrastructure work like this tends to ship. But if it holds up at scale, this is the kind of unglamorous engineering that quietly saves thousands of researchers real hours every week.