The Design Space of Tri-Modal Masked Diffusion Models
Apple Machine Learning Research
Tri-modal Masked Diffusion Models were introduced, pretrained from scratch on text, image-text, and audio-text data to enable generation across three modalities. The work presents a batch-size analysis linked to a novel stochastic differential equation (SDE) based method. It changes how these diffusion models are scaled and sampled, by adding guidance on mixing ratios, noise schedules, batch effects, and optimized inference defaults.
Why it matters
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and finetuning a base unimodal model for bi-modal generation. Diverging from previous approaches, we introduce the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text, and audio-text data. We systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects and provide optimized inference sampling defaults. Our batch-size analysis yields a novel stochastic differential equation (SDE) based…