Xet is on the Hub

Hugging Face just proved out something it's been building for months: a replacement for Git LFS called Xet, designed specifically for the multi-gigabyte models and datasets that clog up the Hub. Rather than treating a file as one giant blob that must be re-uploaded whenever anything changes, Xet chops files into roughly 64KB chunks and only pushes the bytes that actually changed. The team's own example is a good gut-check: a 5GB internal SQLite database that gets a 1MB update. Under LFS, that meant re-uploading all 5GB. Under Xet, it takes a tenth of a second instead of 13 minutes.

On February 20th, the team migrated 4.5TB of repositories from LFS to Xet in a single day, moving roughly 6% of the Hub's total download traffic onto the new infrastructure. That's not a toy test — with nearly two million developers touching over two million public repos, the Hub is a genuinely brutal proving ground, and no staging environment could have surfaced what production traffic did within hours.

And surface it did. CAS, the content-addressed store sitting between clients and S3, was pulling down four times more data than it actually delivered to users. The culprit was hf_transfer users requesting 10MB ranges that didn't line up with block boundaries, forcing CAS to stream from the start of a ~60MB block just to find the requested chunk. Fixing it meant rewriting the block format to store chunk-length metadata across CAS's APIs, its metadata format, and over 65,000 existing blocks on S3 — all without downtime. That single fix cut GET latency by about 35% across the board.

A second, gnarlier problem hit during uploads: one CAS pod would suddenly handle hundreds of concurrent uploads while its neighbors sat nearly idle. Swapping load-balancing algorithms didn't help. The real cause was subtler — temp files being written without fsync, so unflushed data piled up in the OS page cache until disk throttling kicked in and created a feedback loop of backlogged uploads. The fix was blunt but effective: cap concurrent uploads per pod and spill overflow to the rest of the cluster, with autoscaling as a backstop.

What's notable here isn't that Hugging Face hit bugs — everyone does — it's how deliberately they staged the rollout to hit them early, on a fraction of traffic, before betting the whole Hub on it. Xet is now open via waitlist, with a coming hf_xet package meant to slot into existing huggingface_hub workflows without forcing anyone to change how they work. Legacy LFS clients keep working too, through a bridge, so nobody's forced to jump before they're ready.