From GAN to WGAN
Lilian Weng
The article explains the mathematical foundations of Generative Adversarial Networks (GANs), which use a generator and discriminator in competition to create synthetic data, and discusses why GAN training is unstable including problems like difficulty achieving Nash equilibrium, vanishing gradients, and low-dimensional manifold supports. The optimal discriminator outputs D*(x) = 0.5 when the generator perfectly matches real data distribution, and the loss function equals -2log2 at global optimality. Training instability occurs because both the real data distribution and generated distribution occupy low-dimensional manifolds that rarely overlap, causing the discriminator to become too perfect and gradients to vanish during backpropagation.
Why it matters
[Updated on 2018-09-30: thanks to Yoonju, we have this post translated in Korean!] [Updated on 2019-04-18: this post is also available on arXiv.] Generative adversarial network (GAN) has shown great results in many generative tasks to replicate the real-world rich content such as images, human language, and music. It is inspired by game theory: two models, a generator and a critic, are competing with each other while making each other stronger at the same time. However, it is rather challenging to train a GAN model, as people are facing issues like training instability or failure to converge.
