Deep double descent
OpenAI Blog
Researchers observed that neural networks including CNNs, ResNets, and transformers exhibit a double descent pattern where performance improves, then deteriorates, then improves again as model size, data size, or training time increases. The effect occurs across multiple architectures and can typically be mitigated through regularization techniques. Understanding the underlying causes of this pattern remains an open research question with implications for model design and training strategies.
Why it matters
We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction.
