Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries
Hugging Face
Researchers surveyed 16 open-source reinforcement learning libraries to understand how they handle asynchronous training, where inference and gradient computation run in parallel on separate GPU pools. The generation phase for reasoning models can consume 28 minutes to 3.7 hours per training step on a single GPU, leaving training GPUs idle while waiting for data. The standard solution separates inference and training onto different hardware, connects them with a rollout buffer, and synchronizes model weights asynchronously so neither process blocks the other.
