Large Language Models: A New Moore's Law?

Hugging Face Blog · 4 years ago

Microsoft and NVIDIA unveiled Megatron-Turing NLG 530B, a 530-billion parameter language model that the author argues represents an impractical approach to AI development despite its engineering achievement. Training such a model requires hundreds of DGX A100 servers costing approximately $100 million total, with each server consuming up to 6.5 kilowatts of power and producing a carbon footprint comparable to multiple trans-American flights. The author contends that resources would be better spent on practical techniques like fine-tuning smaller pretrained models, which DistilBERT demonstrates by retaining 97% of BERT's language understanding while being 40% smaller and 60% faster.