Taming Outlier Tokens in Diffusion Transformers

Apple Machine Learning Research · 3 weeks ago · 50

Researchers found that Diffusion Transformers for image generation develop outlier tokens—high-norm tokens that attract attention despite carrying little useful information—in both their encoder and denoising components. The outlier phenomenon appears particularly in intermediate layers of DiTs and in pretrained ViT encoders within RAE-DiT pipelines. Understanding and controlling these outliers could improve the efficiency and quality of transformer-based image generation models.