Are Deep Neural Networks Dramatically Overfitted?

Lilian Weng · 7 years ago

The article examines theoretical frameworks for understanding why deep neural networks generalize despite having many parameters and zero training error, discussing classical principles like Occam's Razor, Minimum Description Length, and Kolmogorov Complexity. A key concrete finding is that a two-layer neural network with ReLU activations requires only 2n + d weights to represent any function on a sample of size n in d dimensions. The analysis suggests that model simplicity and compression principles, rather than parameter count alone, determine generalization capability.