Are Deep Neural Networks Dramatically Overfitted?
Lilian Weng
The article examines theoretical frameworks for understanding why deep neural networks generalize despite having many parameters and zero training error, discussing classical principles like Occam's Razor, Minimum Description Length, and Kolmogorov Complexity. A key concrete finding is that a two-layer neural network with ReLU activations requires only 2n + d weights to represent any function on a sample of size n in d dimensions. The analysis suggests that model simplicity and compression principles, rather than parameter count alone, determine generalization capability.
Why it matters
[Updated on 2019-05-27: add the section on Lottery Ticket Hypothesis.] If you are like me, entering into the field of deep learning with experience in traditional machine learning, you may often ponder over this question: Since a typical deep neural network has so many parameters and training error can easily be perfect, it should surely suffer from substantial overfitting. How could it be ever generalized to out-of-sample data points?
