Ulysses Sequence Parallelism: Training with Million-Token Contexts

Hugging Face · 7 months ago · 50

Researchers developed Ulysses Sequence Parallelism, a technique that distributes attention computation across multiple GPUs using attention head partitioning to enable training on million-token sequences. The method requires two all-to-all communication operations per attention layer with communication volume of O(n·d/P) per GPU, compared to Ring Attention's O(n·d)—a factor of P times more data. The technique has been integrated into Hugging Face's Accelerate, Transformers Trainer, and TRL's SFTTrainer, allowing standard training workflows to handle sequences of 32,768 tokens or longer without exceeding single-GPU memory limits.