Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

The split between where your data lives and where your GPUs live has quietly been costing AI teams real money. You reserve H100s on one cloud, your dataset sits in a bucket on another, and every epoch of training racks up a transfer bill just to get bytes onto the chips you already paid for. Hugging Face and SkyPilot just closed that gap with a new integration that treats the Hub as a native storage backend for SkyPilot jobs, addressable with a simple hf:// URL.

The mechanics are refreshingly boring, which is the point. You add store: hf to a SkyPilot file_mounts block, point it at a Hugging Face Bucket or any model, dataset, or Space repo, and pick MOUNT or COPY. Authentication reuses the HF_TOKEN you probably already have sitting in your environment. SkyPilot then schedules the job across whichever of its 20-plus supported clouds, Kubernetes clusters, or on-prem racks has free capacity, and the job reads that same Hugging Face storage no matter where it lands. Under the hood, MOUNT relies on an hf-mount FUSE driver that Hugging Face engineers fixed up to run in unprivileged containers, the default setup on most Kubernetes clusters, so reads happen lazily at the byte level instead of forcing a full download before training starts.

The economics are the real story. AWS charges roughly nine cents per gigabyte to move data out, and that adds up fast when you're streaming a dataset over many epochs or pushing a base model onto every inference node in a fleet. Hugging Face Storage charges no egress or CDN fees at all, and prices storage itself at $12 to $18 per terabyte per month against S3's roughly $23. In a benchmark fine-tuning Qwen3.5-4B on the Multilingual-Thinking dataset across AWS, GCP, and Lambda, the model loaded in about 30 seconds at up to 500 MB/s on every cloud, and checkpoints wrote back to the bucket at speeds ranging from 112 MB/s on Lambda to 168 MB/s on AWS. Had that model lived in S3 instead, every one of those cross-cloud reads would have carried an egress charge.

There's a second layer of savings baked into how the storage works. Hugging Face Buckets run on Xet, which chunks files at roughly 64 KB boundaries based on content rather than position, so only the parts of a file that actually changed get re-uploaded. Hugging Face tested this by appending 10,000 rows to a 100,000-row Parquet table and found it moved about 10 MB instead of re-sending the full 106 MB file. Re-uploading an already-stored 8.43 GB checkpoint took 8 seconds versus 24 for the first upload, since only chunk hashes needed to move. For teams running adapter fine-tunes or shipping frequent checkpoints, that dedup adds up on top of the egress savings.

The collaboration itself has a nice open-source backstory: the initial store: hf support came from a contributor named Nikhil Jha, and Hugging Face's team carried it forward, upstreaming the FUSE fixes needed to make it work broadly. Every piece involved, SkyPilot, hf-mount, and the huggingface_hub client, is open source, which fits the general shape of infrastructure tooling in this space lately: less proprietary lock-in, more plumbing that just quietly works across vendors.