vLLM V0 to V1: Correctness Before Corrections in RL

When the PipelineRL team moved their inference engine from vLLM V0 to V1, training metrics like KL divergence, clip rate, and reward started drifting away from the known-good V0 baseline almost immediately. The instinct in most RL shops would be to start tweaking the objective — maybe add some importance-ratio correction, maybe assume the rollouts had gone stale. This team resisted that instinct, and the writeup they published is basically a case study in why that restraint paid off.

The first bug was semantic: V1 returns logprobs from raw model outputs by default, before temperature scaling or top-k/top-p filtering gets applied. PipelineRL's trainer expected logprobs from the post-processed distribution actually used for sampling. Flipping a single flag, logprobs-mode=processed_logprobs, killed the obvious mean bias in the policy ratio. But the training curves still didn't match V0, which meant something else was hiding in the inference path itself.

That something turned out to be a pile of V1 runtime defaults nobody had pinned down explicitly — prefix caching and async scheduling chief among them. Prefix caching is usually harmless, a free optimization for a model whose weights aren't changing. But in an online RL loop where weights update mid-flight, a cached prefix computed before an update can get reused after it, silently corrupting the signal. Turning prefix caching off, along with pinning async scheduling, removed that variable. The team also had to rewrite how weight updates propagate through the engine, using vLLM's pause_generation with mode=