Hugging Face Blog
·
2 years ago
GaLore reduces memory requirements for training large language models by projecting gradients into lower-dimensional subspaces before optimizer processing. The technique achieves an 82.5% reduction in memory for optimizer states and enables training of 7-billion-parameter models on consumer GPUs like the NVIDIA RTX 4090. When combined with 8-bit quantization, GaLore allows researchers with limited computational resources to train larger models or use larger batch sizes on standard hardware.
Hugging Face Blog
·
2 years ago
Researchers at Hugging Face created Cosmopedia, an open synthetic dataset containing 25 billion tokens generated using Mixtral-8x7B to replicate the training data behind Microsoft's Phi-1.5 language model. The dataset comprises over 30 million files across textbooks, blog posts, stories, and WikiHow articles, with less than 1% duplicate content achieved through extensive prompt engineering across 145 web-clustered topics and curated educational sources. The release includes the generation code, the full dataset, and a 1-billion-parameter model trained on it, enabling the community to reproduce high-performance language model training without proprietary data or models.
