Some Math behind Neural Tangent Kernel

Lilian Weng · 3 years ago

Neural Tangent Kernel (NTK) is a mathematical framework that explains how neural networks evolve during gradient descent training and why wide networks consistently converge to global minima. The theory demonstrates that networks with infinite width exhibit deterministic convergence regardless of random initialization. This provides theoretical understanding of why overparameterized neural networks can achieve good generalization despite having more parameters than training data points.