Xet is on the Hub

Hugging Face Blog · 1 year ago

Hugging Face migrated the first model and dataset repositories from LFS storage to its new Xet system, which deduplicates data at the byte-chunk level rather than the file level. The initial migration on February 20th shifted 4.5 terabytes across approximately 6% of the Hub's download traffic, revealing that uploading a 1MB change to a 5GB database takes a tenth of a second instead of 13 minutes. The system identified and resolved performance issues including unnecessary block downloading and upload pod imbalances, enabling faster iterations on large files for developers across Hugging Face's two million repositories.