From Autoencoder to Beta-VAE
Lilian Weng
Autoencoders use neural networks with bottleneck layers to reconstruct high-dimensional data while compressing it into lower-dimensional latent representations. The bottleneck layer typically reduces data dimensionality significantly, enabling applications in search, compression, and factor discovery. This compressed representation allows systems to learn underlying patterns in data that can be applied across multiple downstream tasks.
Why it matters
[Updated on 2019-07-18: add a section on VQ-VAE & VQ-VAE-2.] [Updated on 2019-07-26: add a section on TD-VAE.] Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer in the middle (oops, this is probably not true for Variational Autoencoder, and we will investigate it in details in later sections). A nice byproduct is dimension reduction: the bottleneck layer captures a compressed latent encoding. Such a low-dimensional representation can be used as en embedding vector in various applications (i.e. search), help data compression, or reveal the underlying data generative factors.
