Perceiver IO: a scalable, fully-attentional model that works on any modality

Hugging Face · 4 years ago · 34

DeepMind's Perceiver IO model has been integrated into HuggingFace Transformers as a fully-attentional architecture that handles any modality—text, images, audio, video, and point clouds—without requiring modality-specific preprocessing. The model uses 256 latent variables with a dimensionality of 1280, performing cross-attention between inputs and latents followed by 26 self-attention layers to process information independently of input size. This design eliminates the quadratic scaling problem of standard Transformers by conducting bulk computation in a compact latent space, enabling the same architecture to work across different data types with only different preprocessors and decoders.