vLLM V0 to V1: Correctness Before Corrections in RL

Hugging Face Blog · 2 months ago

PipelineRL's RL training system experienced discrepancies between vLLM V0 and V1 inference engines that affected rollout logprobs used in policy gradient computations, requiring systematic debugging before objective changes. The team fixed four specific issues: logprobs semantics (enabling processed_logprobs mode), runtime defaults (disabling prefix caching and async scheduling), inflight weight-update synchronization (using pause/resume with cache preservation), and fp32 precision for the language-model head projection. The final vLLM V1 configuration now produces training metrics matching the V0 reference, enabling future objective-level improvements on a correct backend foundation.