One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation

Apple · 3 months ago · 16

Researchers propose FAE, a framework that adapts pre-trained visual encoders for image generation by using a single attention layer to convert high-dimensional features into low-dimensional latents suitable for generative models. On ImageNet 256×256, FAE achieves an FID score of 1.48 without classifier-free guidance after 800 epochs and 2.08 after 80 epochs. The approach enables simpler adaptation of pre-trained representations across different generative model families including diffusion models and normalizing flows.