How 🤗 Accelerate runs very large models thanks to PyTorch
Hugging Face Blog
Hugging Face developed Accelerate, a tool that lets users run very large language models like OPT-6.7B on consumer hardware without enough RAM by loading model weights progressively onto different devices instead of loading the entire model at once. OPT-6.7B has 6.7 billion parameters, which would require 26.8GB of RAM just to create the empty model in default precision, but the new approach avoids this bottleneck. The system automatically decides which model layers go on GPU, CPU RAM, or disk storage, then loads and places weights incrementally, freeing memory after each step.