Pre-Train BERT with Hugging Face Transformers and Habana Gaudi

Hugging Face Blog · 3 years ago

Hugging Face released a tutorial for pre-training BERT-base from scratch using Habana Gaudi accelerators on AWS, leveraging the Transformers and Optimum Habana libraries with masked-language modeling. The training ran for 100,000 steps with a global batch size of 256 over approximately 12.5 hours on a DL1 instance with 8 HPU-cores. Users can now train custom BERT models on Gaudi hardware by following the guide's four steps: dataset preparation, tokenizer training, dataset preprocessing, and distributed model pre-training.