Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL
Hugging Face
AsyncGRPOTrainer in TRL v1.14 was updated to train a LoRA adapter and sync only the adapter to vLLM replicas running in separate Hugging Face Jobs, using a Storage Bucket mount plus a proxy rather than NCCL. Five runs using the same recipe for 500 steps fell from 3 h 27 min to 53 min. The setup changes training-inference coordination by persisting versioned adapters to shared storage, having the proxy route rollouts to replicas with matching KV prefixes, and requiring vLLM to load/unload adapters at runtime while checkpoints can resume after preemption.