Introducing Storage Buckets on the Hugging Face Hub

Hugging Face just admitted something a lot of ML teams already knew: Git repos are a lousy home for the messy stuff that training jobs actually produce. Checkpoints, optimizer states, processed shards, agent traces, logs — none of it needs a commit history, and forcing it into one just slows everyone down. So the company built Storage Buckets, a new non-versioned storage layer that sits on the Hub alongside Models and Datasets repos, but behaves more like an S3 bucket than a Git remote.

Under the hood, Buckets run on Xet, Hugging Face's chunk-based storage backend that's been quietly powering repo uploads for a while now. Instead of storing files as whole blobs, Xet slices content into chunks and deduplicates across them. That's a genuinely good fit for training workflows, where a processed dataset overlaps heavily with its raw source, or where checkpoint 47 shares most of its weights with checkpoint 46. Buckets just skip re-uploading the parts that already exist, which cuts bandwidth and, for Enterprise customers billed on deduplicated storage, actually lowers the invoice.

There's also a pre-warming feature aimed at the throughput-sensitive crowd — distributed training clusters and multi-region pipelines where pulling data across clouds on every read becomes the bottleneck. You declare where your compute lives, and Hugging Face makes sure the hot data is already sitting in that region, in partnership with AWS and GCP for now, with more providers promised later.

Getting one running is deliberately unglamorous. Install the hf CLI, log in, run hf buckets create, then sync a local checkpoints folder straight into hf://buckets/username/my-training-bucket. There's a dry-run flag for previewing large syncs before committing to them, and the whole thing is mirrored in Python via huggingface_hub 1.5.0 and in JavaScript through @huggingface/hub. Because Buckets also plug into fsspec's HfFileSystem, tools like pandas, Polars and Dask can read and write hf:// paths without any special handling — which quietly makes Buckets usable in code that was never written with Hugging Face in mind.

The bigger idea here is a two-layer workflow: Buckets for anything still in motion, versioned repos for anything final. Hugging Face says direct transfers between the two are coming, so a finished checkpoint could get promoted into a model repo without manual re-uploading. Jasper, Arcee, IBM and PixAI tested it during a private beta before this general release, and Buckets now ship as part of existing Hub storage plans, free tier included.