One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation
Apple ML Research
Researchers propose FAE, a framework that adapts pre-trained visual encoders for image generation by using a single attention layer to convert high-dimensional features into low-dimensional latents suitable for generative models. On ImageNet 256×256, FAE achieves an FID score of 1.48 without classifier-free guidance after 800 epochs and 2.08 after 80 epochs. The approach enables simpler adaptation of pre-trained representations across different generative model families including diffusion models and normalizing flows.
Why it matters
Visual generative models (e.g., diffusion models) typically operate in compressed latent spaces to balance training efficiency and sample quality. In parallel, there has been growing interest in leveraging high-quality pre-trained visual representations—either by aligning them inside VAEs or directly within the generative model. However, adapting such representations remains challenging due to fundamental mismatches between understanding-oriented features and generation-friendly latent spaces. Representation encoders benefit from high-dimensional latents that capture diverse hypotheses for…