Explaining AlexNet Convolutional Neural Network
Overfitting Prevention

Having tackled normalization and pooling AlexNet was faced with a huge overfitting challenge. Their 60-million parameter model was bound to overfit. They needed to come up with an overfitting prevention strategy that could work at this scale.

Whenever a system has huge number of parameters, it becomes prone to overfitting.

Overfitting — Given a question that you’ve already seen you can answer perfectly but you’ll perform poorly on unseen questions.

They employed two methods to battle overfitting

Data Augmentation

Dropout

Data Augmentation

Data augmentation is increasing the size of your dataset by creating transforms of each image in your dataset. These transforms can be simple scaling of size or reflection or rotation.

See how no. 6 is rotated in various directions Source

These schemes led to an error reduction of 1% in their top-1 error metric. By augmenting the data you not only increase the dataset but the model tries to become rotation invariant, color invariant etc. and prevents overfitting

Dropout

The second technique that AlexNet used to avoid overfitting was dropout. It consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in back- propagation. So every time an input is presented, the neural network samples a different architecture.

This new-architecture-everytime is akin to using multiple architectures without expending additional resources. The model, therefore, forced to learn more robust features.
['Rishi Sidhu']
['Machine Learning', 'Data Science', 'Technology', 'Neural Networks', 'Artificial Intelligence']