Training CodeParrot 🦜 from Scratch

Hugging Face Blog · 4 years ago

Researchers trained CodeParrot, a GPT-2 model for Python code generation, from scratch using a cleaned dataset of GitHub code. The training dataset comprised 50 GB of Python code after removing duplicates that initially made up 66% of files in the original 180 GB collection. The model uses streaming data loading and distributed training across multiple GPUs to enable practical training without downloading the full dataset to disk.