Understanding Internal Covariate Shift and Residual Connections

This article explores two key challenges in training deep neural networks - internal covariate shift and vanishing gradients - and the solutions of batch normalization and residual connections.

Details

The article discusses how going deeper with neural networks can sometimes lead to worse performance, due to two key issues: internal covariate shift and vanishing gradients. Internal covariate shift refers to the problem where the distribution of each layer's input keeps changing as the weights are updated, making it hard for the network to learn. Batch normalization is presented as a solution, normalizing the inputs to each layer to have zero mean and unit variance. This stabilizes the activations and allows for faster training and more robust models. The second issue is vanishing gradients, where backpropagation multiplies small derivatives, causing the gradient to shrink with each layer. This can lead to the early layers in a deep network effectively stopping learning. Residual connections are introduced as a way to bypass layers, allowing the gradient to flow more effectively through the network.