Anatomize Deep Learning with Information Theory

Lilian Weng · 8 years ago

Naftali Tishby's information bottleneck method applies information theory to analyze how deep neural networks learn during training. The approach proposes new learning bounds for networks with exponentially large parameters where traditional learning theory breaks down. Training occurs in two phases: first representing input data to minimize error, then compressing representations by forgetting irrelevant details.