Some Math behind Neural Tangent Kernel
Lilian Weng
Neural Tangent Kernel (NTK) is a mathematical framework that explains how neural networks evolve during gradient descent training and why wide networks consistently converge to global minima. The theory demonstrates that networks with infinite width exhibit deterministic convergence regardless of random initialization. This provides theoretical understanding of why overparameterized neural networks can achieve good generalization despite having more parameters than training data points.
Why it matters
Neural networks are well known to be over-parameterized and can often easily fit data with near-zero training loss with decent generalization performance on test dataset. Although all these parameters are initialized at random, the optimization process can consistently lead to similarly good outcomes. And this is true even when the number of model parameters exceeds the number of training data points. Neural tangent kernel (NTK) (Jacot et al. 2018) is a kernel to explain the evolution of neural networks during training via gradient descent. It leads to great insights into why neural networks with enough width can consistently converge to a global minimum when trained to minimize an empirical loss. In the post, we will do a deep dive into the motivation and definition of NTK, as well as the proof of a deterministic convergence at different initializations of neural networks with infinite width by characterizing NTK in such a setting.
