Introducing Storage Buckets on the Hugging Face Hub

Hugging Face Blog · 4 months ago

Hugging Face introduced Storage Buckets, mutable object storage on its Hub designed for intermediate ML artifacts like checkpoints, optimizer states, and processed data that don't require version control. Buckets are built on Xet, a chunk-based backend that deduplicates content across related files, reducing bandwidth and storage costs, with pre-warming available to bring frequently-accessed data closer to compute infrastructure in AWS and GCP regions. Users can now manage training pipelines, data processing, and agent traces in a single Hub-native workflow, with a clear separation between mutable working storage and versioned repos for final deliverables.