No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL

Hugging Face Blog · 1 year ago

TRL integrated vLLM into its GRPO training algorithm to allow training and inference to run on the same GPUs instead of separate ones, eliminating idle GPU time caused by previous server-mode setups. The co-located approach achieved up to 1.73× speedup on a 7B model and enabled training of a 72B model by combining vLLM's sleep mode with DeepSpeed ZeRO Stage 3 optimizations. This reduces hardware requirements and cost while improving overall training throughput by allowing GPUs to switch between training and generation tasks without waiting periods.