[Classic] ImageNet Classification with Deep Convolutional Neural Networks (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper classic alexnet hinton geoff hinton imagenet convolution convolutional neural network architecture dropout data augmentation cnns computer vision image classification object recognition classifier max pool pretraining deep neural networks
#ai #research #alexnet

AlexNet was the start of the deep learning revolution. Up until 2012, the best computer vision systems relied on hand-crafted features and highly specialized algorithms to perform object classification. This paper was the first to successfully train a deep convolutional neural network on not one, but two GPUs and managed to outperform the competition on ImageNet by an order of magnitude.

OUTLINE:
0:00 - Intro & Overview
2:00 - The necessity of larger models
6:20 - Why CNNs?
11:05 - ImageNet
12:05 - Model Architecture Overview
14:35 - ReLU Nonlinearities
18:45 - Multi-GPU training
21:30 - Classification Results
24:30 - Local Response Normalization
28:05 - Overlapping Pooling
32:25 - Data Augmentation
38:30 - Dropout
40:30 - More Results
43:50 - Conclusion

Paper: http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf

Abstract:
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

Authors: Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar (preferred to Patreon): https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there! Today we'll look at ImageNet classification with deep convolutional neural networks by Alex Kruschevsky, Ilya Sutskever and Jeffrey E. Hinton. This paper is another one in the installment of our historical paper overview, where we go through kind of old papers that were or weren't very impactful and see what people knew at the time already, how this developed and so on. Of course this paper here, also known as AlexNet, was the one that started the deep learning revolution, so to say, or at least contributed in large part to it. It was the first paper that showed that you could train these very deep neural networks, and very deep in here is a relative term, but the first one that showed that you could actually use CUDA, GPUs, to train those large networks efficiently, and it won the ImageNet competition that year, and it did so by a very very large margin. So it kind of shook the world, because previously computer vision was still doing like hand engineered features and then using some kind of classifiers on top of those. This paper basically changed everything. So we'll go through the paper and we'll see what was already known, and especially I always enjoy with these papers how did the choices that people make back then, how did they pull through to today, sort of what arbitrary choices that Alex Kruschevsky made right here are we still doing today and what have we learned since then. So the paper is written relatively straightforward, I have to say. It's a good read if you want to read it, and you know straightforward, and sort of gives you a little bit of an intuition of how much work must have gone into this, which is I guess a lot. So they start off by saying that that current approaches to object recognition make essential use of machine learning methods. This was also new, right? Object recognition wasn't always learned. The object recognizers, you could even do it in the indifferent way, like matching templates and so on. Machine learning was still one of the methods used, and of course today it's the method used. To improve their performance we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Until recently datasets of labeled images were relatively small, on the orders of tens of thousands of images. So this especially at NORP, or here the C410, or C4100, these are relatively small datasets with relatively small images as well, like C410 is 32 by 32 pixels. So they're saying that in these small datasets you can solve it with classical computer vision models, but if you have larger datasets, and especially more realistic datasets like bigger resolution and so on, you need bigger models. So they say but objects in realistic settings exhibit considerable variability to learn to recognize them, it is necessary to use much larger training sets. So they say that this ImageNet dataset is one of those larger datasets, consists of 15 million labeled high resolution images in over 22,000 categories. People keep forgetting this, and I am included in that group of people, that the ImageNet dataset is actually much larger than we know, than we when we talk of ImageNet. When we speak of ImageNet we think of the ImageNet that has a thousand classes and about one or one and a half million images. However that's only a subset of the much much larger ImageNet dataset within many many more categories. It's just that the ImageNet competitions were performed on this subset, because I guess people thought well a thousand classes and a million images is already plenty, so we'll do that. So that's I guess how that came to be. So their argument is right here, to learn about thousands of objects from millions of images we need a model with a large learning capacity. However the immense complexity of object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet. So our model should also have lots of prior knowledge to compensate for all the data we don't have. So their main argument for using neural networks is that the size of the dataset is so large, therefore we need a large model. Granted they already recognize the inherent connection between large models and a lot of complex data, but in the opposite they say well even if we have that much data the task we are trying to solve, object recognition, is way more complicated than the amount of data we have. So our model should also have lots of prior knowledge to compensate for all the data we don't have. Remember at this time convolutional neural networks weren't really known to do anything. I guess they were used for handwritten digit recognition and so on and were kind of on par with other methods. However it wasn't like obviously clear that you would use them for image recognition. So here they have to make like a argument to convince people that okay we can use neural networks for this task because they have such a high capacity. However neural networks, feed-forward neural networks, are already too powerful. They don't know anything about the data. Everything's connected to everything and they argue right here our model should have lots of prior knowledge to compensate for all the data we don't have. So they allude to the convolutional neural networks constitute one such class of models. Their capacity can be controlled by varying the depth and breadth and they also make strong and mostly correct assumptions about the nature of images, namely stationarity of statistics and locality of pixel dependencies. So their argument here is that the convolutional operation is such a strong prior that is mostly consistent with what we know about images that they are very well suited to computer vision. Again something that was not abundantly clear at the time as it is right now. It's interesting to see how they get to this point where they say we need lots of capacity but we also need a model with lots of prior knowledge and of course CNNs fit that very well. So they go into the problems of CNN despite the attractive qualities and despite the relative efficiency of their local architecture they are prohibitively expensive to apply in large-scale high-resolution images. Luckily current GPUs paired with a highly optimized implementation of 2D convolution are powerful enough to facilitate the training of interestingly large CNNs and recent data sets such as ImageNet contain enough labeled example to train such model without severe overfitting. So overfitting was also still like very much at the forefront of people's minds back then. Right now we don't really care about overfitting that much anymore. Basically we figured out that if we just build large enough models we don't overfit which is strange in itself like this double descent phenomenon and so on but overfitting was still very much at the forefront of people's minds and they do a lot of things here to prevent overfitting which gives them kind of a boost in the test accuracy which might actually not have been the overfitting that they're combating. So they do for example in data augmentation already in this paper and they always allude to how this is to prevent overfitting. However we know nowadays that it might not be the overfitting that's combated by data augmentation. It might actually have something to do with regularizing your function making it more smooth and so on. So you just see how coming from a classical machine learning perspective overfitting was like the number one or one of the number one problems in classical machine learning in SVMs and things like this. So it's safe to say that they thought if we built these large models we're gonna have a huge overfitting problem and yeah so that's why this pulls through right here. Also I guess one of the main contributions of this paper is to show to combine this CNN training with GPUs. Also not very non-clear at the time like it was known that you could do computation on GPUs but the fact that these are you know very capable for training these CNNs or generally neural networks wasn't something that was you know known at the time. So this paper basically showed that if you use a GPU you can get that much faster and that makes it possible to train these big neural networks. Again right here the size of our network made overfitting a significant problem even with 1.2 million labeled training examples so we use several effective techniques for preventing overfitting and we'll look at those. And the end they say the network's size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 GPUs. All of our experiments suggest that our results can be improved by simply waiting for faster GPUs and bigger data sets to become available. And I mean that proved to be absolutely true. We don't necessarily have bigger data sets right now though we do but certainly with faster GPUs and bigger GPUs this became a this became these networks became better simply by increasing their depth and as you know then ResNets came along increasing the depth by an order of magnitude and that gave another boost to computer vision. Alright so they talk about the ImageNet data set here and the main point in the ImageNet data set right here is the fact that the images are plenty so there are over a million training images in this subset with a thousand classes which was you know a very big that was that was on like CIFAR 10 had 10 classes, CIFAR 100 had a hundred classes that was already a lot. A thousand classes that is like unheard of before this data set. I guess not unheard of but yeah and a million training images. Completely crazy and also not only was it a lot of images they were resolution was really big so in the order of 256 by 256 whereas previous methods all were like 32 by 32 so definitely challenging data set even today it's a challenging data set. Alright so the architecture. The architecture and there's this famous graphic right here of the AlexNet architecture so briefly they described these convolutional layers right here as you can see there's max pooling already here they have dense layers at the end they do generally increase the number of feature maps right here while decreasing the resolution with max pooling so all of this has sort of you know kept until today I guess they also took it from earlier work on convolutional neural networks that generally found this to be a good idea and the important part here that is kind of special to AlexNet is you can see there are these two different pipelines and Alex for cutting off this part right here I mean you just know like this has the eight pages we need to like we have like three lines too much how can we fit the three lines we've already cropped everything let's just cut off the top half here it's essentially the same as the bottom yeah so space constraints and PDFs for conference submissions ruining yet another paper alright but you can see there is this two this this two column architecture right here so this network was so large that it didn't fit on one GPU so they had to split it onto two GPUs with the occasional intercommunication right you can see here there is intercommunication between the two GPUs and there is also no intercommunication right here on this layer this was very intricate that was one thing that really didn't hold until today I guess until now with things like I don't know G shard or so where you have different weights on different GPUs again I guess the the invention of bigger GPUs made that sort of super fluid but just imagine the amount of code they had to write there was no tensor flow at this point there I don't think there was even cafe around there was just CUDA and yeah just this cross GPU memory writing I just imagined this to be so so ugly and big respect for writing all of this all of this code alright so they they go through a number of important things and most of the things here aren't their invention let's say but they cleverly combine things that were already known about neural networks and things that were maybe developed somewhere that they have found to work really well so the first one is the relu non-linearity now of course relu is nowadays all like abundant everyone uses relu's non-linearities but at that time it was still very much in fashion to use something like the sigmoid right here or the hyperbolic tangent and why is that because the neural networks were still kind of inspired by the neurons where you had the soma of the neuron and then the input dendrites sorry the dendrites with the input axons and then you would sum up all the incoming signals and then that would go over so in the true neuron you have this this this kind of curve where if the input rises above this border right here the action potential maybe I don't know what the the English term is then if it rise above that then the neuron would start to spike right and if it's below that it wouldn't so people wanted to approximate this using some sort of a a kind of differentiable but something that's very similar to this step function and that ultimately led to something like a sigmoid or an hyperbolic tangent so people trying to stay close to biological neurons did this but that gives you the problem that in this region and in this region right here you have almost no gradient to learn from so you can see that they argue that in terms of training time with gradient descent the saturating non-linearity so the hyperbolic tangent and the sigmoid are much slower than the non saturating lean non-linearity this one following Narendt Hinton we refer to neurons with this non-linearity as rectified linear units so taken from this this other paper they say okay we use these relu's these rectified linear units which are not exactly like real biological neurons but they train much faster right and of course relu's are used until this day so you can see right here that a this is on a C for 10 and they measure the time to reach 25% of the training error and this here is with the relu's and this here is with the hyperbolic tangent and it takes much longer to reach the hyperbolic tangent especially it takes six times faster to with the relu's and they say that's one of the main components that allows them to learn this fast to even experiment with these big networks because their entire training time is six days right but they probably didn't train it only once they experimented with it and saw what works so if you have a couple of months of time and he takes you a week to train one of these things you know you don't you can't afford a six times slowdown because that would mean you can only train like two models in the entire course of research and that would severely hinder your progress now we are at the point where that becomes true again with these giant giant transformer language models where people can train it once and then you know like GPT-3 they say oh we made we discovered a bug halfway through and we've kind of fixed it but we're not sure we couldn't restart because it was too expensive yeah maybe we're waiting for a moment I'm still saying we're waiting for the resonant moment in the transformers but yeah relu's in you know here in not introduced here but used here and have been prevailing until today training on multiple GPUs something as I said that didn't didn't really get forward from here especially the kind of GPU training so if we train on multiple GPUs today what we mean is that we have our model right and then we distribute that to multiple GPUs like this and then we take a mini batch from the training data and we simply split it up let each GPU do its thing on its subset of the mini batch and then at the end kind of calculate the loss and then back propagate the gradients and synchronize the gradients between that so we have one model that is on both GPUs here they distribute a model to two GPUs and I'm also thinking that with frameworks like G shard this could potentially have a revival right here this kind of distributing your models especially within the same layer across many GPUs and then having cross communication only at some points so their argument is this only has three gigabytes of memory which limits the maximum size of networks can be trained on it turns out that 1.2 train million training samples are enough to train networks which are too big to fit on one GPU therefore we spread the net across two GPUs current GPUs are particularly well suited to cross GPU parallelization as they're able to read from and write to one another's memory directly without going through the host machine okay so this means that for so sorry here they say the parallelization scheme that we employ essentially puts half the kernels or neurons on each GPU with one additional trick the GPUs communicate only in certain layers that means that for example the kernels of layer 3 take input from all kernel maps in layer 2 however the kernels in layer 4 take input only from the kernel maps in layer 3 which reside on the same GPU so very very interesting choice right here and they they justify this here or they say the results this scheme reduces our top one top five error rates by 1.7 and 1.2 percent respectively as compared with a net with half as many kernels in each computational layer in each convolutional layer on one GPU the two GPU net takes slightly less time to train than the one a GPU net so first of all I have to say big respect right here like like I can imagine they did this you know with the relu's and stuff and they were already better than previous because they're so just to go to the results the pre they beat the error rates of previous models by ginormous amount so this is what they knew right here this is on the 2010 image net split so the previous best ones were like at around 28 25 percent and here their best one is at 17 percent top five error rate I'm gonna imagine that they trained it first and we're already better than the 25 percent and I guess lots of people would just call it a day would be like oh cool we have this entirely new method not only did we show that we can train it we actually showed that it's better and bad a boom I have point one percent better error rate and everything else can be a separate paper no they stuck with it and they pushed it each so each of these things right here they say oh this reduces the error rate by 1% this reduces the error rate by 2% and you know really they they went about it how far can we push this with everything I mean just imagine you come and you train a network I'm pretty sure they first trained on one GPU right and and then they thought oh you know maybe we can train an even bigger network by using two GPUs and then they realized what it's gonna take like a crap ton amount of dumb code to cross synchronize and keep them in lockstep and blah blah blah like it's not even easy to write multi GPU code today with all the frameworks just imagine that and for them to having already observed that their network does better than everything that was previously to sit down and do the cross GPU thing experiment with okay when do we cross communicate and whatnot that is very very respectable right here so maybe a lesson to be learned or or just the mentality of the people maybe they just had more time they were like okay it's still like two months out this competition deadline I don't know but you know I'm this this is not something that I see today very often this this kind of persistence and additional pushing and reporting of what works in these kinds of things I mean some some papers do it but most papers do it because only with all the tricks they can get that point one percent improvement and this one already had the improvement and did it anyway okay but multi GPU training didn't really it's like splitting the models across GPUs didn't really didn't really stick around mainly because I guess the GPUs got larger in memory pretty quickly so it wasn't that necessary but also I guess because the frameworks were just too clunky and now maybe with G-shard this is coming back so worth another shot I guess next one local response normalization this also didn't really stick around I cut kind of dumped in favor of things like batch normalization but with the resurfacing of things like layer normalization this it comes back to this thing here again a little bit so what they say is that what they want to do is they want to kind of normalize the response of these of these relu's so what they do is each response which is this alpha they are these a here is normalized by the following quantity and it's the all the responses of the other neurons around them or of the other kernels around them and you can see the sum is over this weird quantity right here so what does it mean if they have a bunch of convolutional filters and these are the activation so these are the feature maps after the convolution and yeah so if I have like 10 convolutional filters in my layer this is going to be the output the way they normalizes they normalize each filter sorry each output channel by averaging by you see here dividing by the average response of the channels around them right so let's maybe say the five channels though two channels in front of them and two channels behind them this is going to be they take the average across this one and then for another channel right here for this one you would take the average of the five around that this isn't really something that stuck around I guess mainly because of the really dynamic situation right here what people do today is they have things like layer normalization that simply averages across all of the channels or they have group normalization that pre defines these groups like here is there's two groups and we only normalize within this group and within this group also always the same this kind of dynamic normalization on across neighboring filters as I said didn't really stick around not really sure why but I guess it was just easier to implement it otherwise or it just worked better again here they say this this it was motivated well right this scheme bears some resemblance to the local contrast normalization scheme of that but ours would be more correctly termed brightness normalization since we do not subtract the mean activity and oh they make it connection to biological neurons where is it this sort of response normalization implements a form of lateral inhibition inspired by type found in real neurons creating competition for big activities amongst neuron outputs computed using different kernels okay so kind of inspired by real neurons but also kind of inspired by other people doing also some kind of normalization so people already knew that normalization was helpful at some times and this is what they employed right here again reducing the top error rates by 1.4 and 1.2 percent respectively so not a big improvement but still an improvement the last thing overlapping pooling again a thing that didn't really stick around that much where they say okay instead of having a pooling layer so if this is your image and instead of pooling 2x2 in the stride of 2 like we do today and you know pull it down to a smaller image what we can do instead is we can pool with overlapping windows so in that case they pool with a 3x3 window but they do always do stride of 2 so they have like these overlaps right here resulting in the same size but then each pixel right here has some sort of overlapping information from the pixels around it again they say it reduces the top one and top five error rates by 0.4 percent and 0.3 percent maybe this this didn't stick around because I'm not sure maybe because people found it doesn't work in other problems who knows so the overall architecture as we said is described in this picture right here so you have the input image which you can see has three channels and they use convolutional filters with a here with a stride of four at the beginning to reduce the size so at the beginning it's 224 by 224 and then it's 48 by sorry it's 55 by 55 that thing here 55 by 55 48 feature maps you can already see as we said before the feature maps keep increasing while the number of the dimension the resolution of the image keeps decreasing the stride of four convolution here already employed in order to down sample the image at the same time as convolving it nowadays a lot of architectures will simply not do max pooling at all but always use the kind of strided convolution to down sample image while convolving it what you also see here is that they thought that the feature map size should be should also be large at the beginning and then decrease which is a reasonable assumption right because if you have higher resolution images you're probably going to need higher resolution feature maps this didn't really come through until today as you know most architectures today they just go with like three by three kernels from the very start and don't really care about you know also downsizing their their filters I don't really know why whether it's just more convenient or less parameters or whether there's really something to having small filters but I just know you know this is something the large filters at the beginning is something that didn't didn't hold over time also you can see right here they have multiple dense layers at the end I believe most architectures today simply go with two of those instead of three so one like hidden layer and then one classification layer but it's you know it's it's very close to the architectures today right there hasn't changed that much like the difference between this and the VGG 16 VGG 19 network is just depth and then the difference between those and the ResNet is just the whatever these skip connections right here and that's where we are today so so there hasn't hasn't changed that much honestly they also allude to the fact that actually even though it doesn't look like it most parameters are here in these dense layers those are most parameters of the network this right here a convolution layer is like 1% of the parameters even though it takes up a lot of space in the in the drawing so maybe the reduction in the number of classification layers at the end also has something to do with the fact that that's where most parameters are so if you get rid of one of those dense layers you can like get many many more convolutional layers all right so the last part here is on reducing overfitting again they didn't really investigate whether or not really their network was overfitting like really establishing the overfitting it was I think maybe they did and maybe it was actually overfitting but we now we we don't care about overfitting too much anymore maybe because we already use these augmentations naturally but also because we built these deep models so we somehow have an idea that they generalize naturally I'm not sure whether they actually were only worried about it that much because of the history of machine learning or whether they actually did see that everything was overfitting constantly okay they say our neural network architecture has 60 million parameters although the thousand classes make each training example impose 10 bits of constraints on the mapping from image to label this turns out to be insufficient to learn many parameters without considerable overfitting below we describe two primary ways in which we combat overfitting again there's no one today no one today makes this argument anymore this oh we have this many parameters and there are that many images right we have 60 million parameters we have 1.2 million images a thousand classes how you know when how many parameters per sample is that and so on how many bits of constraint we don't care about we're fine with having like a billion times more parameters than training samples we we don't worry about it anymore so the first thing they do is data augmentation already I mean this was already known again like lots of these things here were already known but the combination is just so cool in this paper where so first of all again they say the transformed images are generating Python code on the CPU while the GPU is training on the previous batch of images so these data augmentation schemes are in effect computationally free again this code must have been ugly the first form of data augmentation consists of generating image translations and horizontal reflections we do this by extracting random 224 by 224 patches and their horizontal reflections from the 256 by 256 images okay so random so this was already this these are the most valuable data augmentations that still we have today random horizontal flipping is still used in every pipeline of computer vision except if you want to read text I guess and random cropping is still the most powerful data augmentation technique for images today and the it's crazy that this was already discovered and I I don't know whether they say right here how much this particular thing improves I don't think they have a stat on how much this improves they just say how much this this next thing improves but I'm going to guess this was one of the vital things for pushing the performance because now we know cropping is very important I guess they thought that they they would you know translation was the important part and so they focused on generating image translations and to generate an image translation from a single image naturally you have to crop it however we we we now focus much more on the fact that we crop it and kind of have different sub images of the same image especially in you know self-supervised learning and things like this we know that cropping is what is like the the power horse of these methods so the fact that they extract random patches right here means that their network only operates on these sub patches and then they compensate by a test time the networks makes a prediction by extracting five patches the four corner patches and the center patch as well as their horizontal reflections and averaging the prediction made by the networks softmax layer on the ten patches I also believe that people don't do this too much nowadays they most of the time they simply rescale the test images or something like this or a fine-tune at the end on the kind of scale training images there are various techniques for doing this but random cropping and horizontal flipping already employed right here also color kind of color jittering a form of color jittering a very special form altering the intensities of RGB channels in training images specifically we perform PCA on the set of RGB pixel values throughout the image in a training set to each training image we add multiples of the found principal components with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a gauss with zero mean and standard deviation point one this is I believe this has gone out of fashion so people do color jitter and kind of brightness jitter and so on but I don't think they particularly do this kind of PCA based image image augmentation right here anymore they say this scheme reduces the top one error rate by over 1% I wonder why why this isn't or maybe because you need these stats over the entire data set and the other things may be working equivalently well but you you can simply apply them without knowing kind of your your principal components okay next thing dropout dropout has been you know one of the things that was very important throughout the early stages of deep learning isn't that important anymore now dropout some people still use it but most people I think don't use dropout anymore and it's very interesting to see but it definitely was a technique that was used a lot during like from Alex net to basically like now or like the last very few years so they say combining the predictions of many different models is a very successful way to reduce test errors but it appears to be too expensive for big neural networks that already take several days to train there is however a very efficient version of model combination that only costs about a factor of two during training so there's this take this technique called dropout then they explain it to set to zero the output of each hidden neuron with probability 0.5 again people didn't know about dropout as they do now but they introduced this right here and they say it reduces their not sure that they also don't say how they how much they by how much this reduces the training error but they say we use drop out in the first two fully connected layers without dropout our network exhibits substantial overfitting dropout roughly doubles the number of iterations required to converge so okay so they did actually make sure or they did find the actual evidence of overfitting and saw that dropout reduces that and I wonder why this doesn't happen nowadays maybe because we have the we have less of these fully connected layers but I can't really imagine maybe because we do more augmentation I don't I don't know or maybe dropout is still used and I'm just I just don't know it and don't see it yeah so here they use momentum to train this and they do some qualitative analysis they do some qualitative analysis so first of all they say okay they shatter all of the previous approaches especially also then they build kind of ensemble methods and they pre-trained they already do transfer learning they already pre-trained on image net 2011 and fine-tune then on the image net 2012 right here the image net 2011 and then fine-tuning on the image net 2012 to reduce that error even further like pulling all the tricks all these things are around still very cool and then they look into what their network learned so they find that there are a number of these kind of filters you see these 11 by 11 filters in the first layer where they show okay this really and this was kind of already known that these neural networks extract filters like this like color gradients or edge detectors in various forms and directions and it's cool to see that this one also does so this one here is also a very cool investigation where they look at examples and the red bar the red one is always the correct label and the bars are basically what their model says are the top five things and it's cool to look at so for example here you have might as the top one but then also black widow cockroach tick starfish but the top labels are usually also very very good labels you can see here grill and it assigns convertible which you know by all means is correct it's just not the class that the annotators assigned to this particular image as well as here Dalmatian was the highest prediction of the network where the label was actually cherry and this is this is quite debatable right so you can see that a lot of the mistakes the network does is are are you know forgivable let's say and you can see that for when the network doesn't do mistakes the not only the top label is good but a lot of the top five labels are also very very adequate lastly they look at a given training set image which these are the training set images right here and they look at the last layers feature vector and the five nearest or the nearest neighbors in Euclidean space of the entire training data set and here's what you come up with so you can see for the elephant the nearest neighbors are all other elephants and regard that they are in different poses right they don't always look the same way these elephants also these dogs right here so it's pretty cool to see that the network actually learns some invariances across the class and puts images with the same label into the same area in the embedding space yeah so that's their that's their paper that they they already allude to the fact that depth is very important it is notable that our networks performance degrades if a single convolutional layer is removed for example removing any of the middle layers results in a loss of about 2% for the top one performance of the network so the depth really is important for achieving our results and as you know this spurred an area of this burden area of trying to build deeper and deeper networks until Resnets came along and built ultra deep networks they also say we did not use any unsupervised pre-training even though we expect that it will help especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase of the amount of labeled data thus far our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infrared temporal pathway of the human visual system ultimately with ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing of our far less obvious in static images so already the previewing of future research here with the self supervised with the many more layers and so on astounding that this kind of foresight and of course all of this proved to be you know very very adequate predictions right here and yeah so this was the paper right here the paper that kicked off deep learning I enjoy reading kind of these old papers especially looking back at what was already known what still is around which turns out to be a lot a lot is still around and the choices that people made back then some of them defined our modern field so that was it for Alex net let me know what you think in the comments and I'll see you next time bye
 Hi there! Today we'll look at ImageNet classification with deep convolutional
 neural networks by Alex Kruschevsky, Ilya Sutskever and Jeffrey E. Hinton.
 This paper is another one in the installment of our historical paper
 overview, where we go through kind of old papers that were or weren't very
 impactful and see what people knew at the time already, how this developed and
 so on. Of course this paper here, also known as AlexNet, was the one that
 started the deep learning revolution, so to say, or at least contributed in large
 part to it. It was the first paper that showed that you could train these very
 deep neural networks, and very deep in here is a relative term, but the first
 one that showed that you could actually use CUDA, GPUs, to train those large
 networks efficiently, and it won the ImageNet competition that year, and it
 did so by a very very large margin. So it kind of shook the world, because
 previously computer vision was still doing like hand engineered features and
 then using some kind of classifiers on top of those. This paper basically
 changed everything. So we'll go through the paper and we'll see what was already
 known, and especially I always enjoy with these papers how did the choices that
 people make back then, how did they pull through to today, sort of what arbitrary
 choices that Alex Kruschevsky made right here are we still doing today and what
 have we learned since then. So the paper is written relatively
 straightforward, I have to say. It's a good read if you want to read it, and you
 know straightforward, and sort of gives you a little bit of an intuition of how
 much work must have gone into this, which is I guess a lot. So they start off
 by saying that that current approaches to object recognition make essential use
 of machine learning methods. This was also new, right? Object recognition
 wasn't always learned. The object recognizers, you could even do it in
 the indifferent way, like matching templates and so on. Machine learning
 was still one of the methods used, and of course today it's the method used. To
 improve their performance we can collect larger datasets, learn more powerful
 models, and use better techniques for preventing overfitting. Until recently
 datasets of labeled images were relatively small, on the orders of tens
 of thousands of images. So this especially at NORP, or here the C410,
 or C4100, these are relatively small datasets with relatively small
 images as well, like C410 is 32 by 32 pixels. So they're saying that
 in these small datasets you can solve it with classical computer
 vision models, but if you have larger datasets, and especially more realistic
 datasets like bigger resolution and so on, you need bigger models. So they say
 but objects in realistic settings exhibit considerable variability to
 learn to recognize them, it is necessary to use much larger training sets. So
 they say that this ImageNet dataset is one of those larger datasets, consists of
 15 million labeled high resolution images in over 22,000 categories.
 People keep forgetting this, and I am included in that group of people, that
 the ImageNet dataset is actually much larger than we know, than we when we
 talk of ImageNet. When we speak of ImageNet we think of the ImageNet that
 has a thousand classes and about one or one and a half million images. However
 that's only a subset of the much much larger ImageNet dataset within many
 many more categories. It's just that the ImageNet competitions were performed on
 this subset, because I guess people thought well a thousand classes and
 a million images is already plenty, so we'll do that. So that's I guess how that
 came to be. So their argument is right here, to learn about thousands of objects
 from millions of images we need a model with a large learning capacity. However
 the immense complexity of object recognition task means that this
 problem cannot be specified even by a dataset as large as ImageNet. So
 our model should also have lots of prior knowledge to compensate for all the
 data we don't have. So their main argument for using neural
 networks is that the size of the dataset is so large, therefore we need a large
 model. Granted they already recognize the inherent connection
 between large models and a lot of complex data, but in the opposite they
 say well even if we have that much data the task we are trying to solve, object
 recognition, is way more complicated than the amount of data we have. So our model
 should also have lots of prior knowledge to compensate for all the data we don't
 have. Remember at this time convolutional neural networks weren't really
 known to do anything. I guess they were used for handwritten
 digit recognition and so on and were kind of on par with other methods.
 However it wasn't like obviously clear that you would use them for image
 recognition. So here they have to make like a argument to convince
 people that okay we can use neural networks for this task because they have
 such a high capacity. However neural networks, feed-forward neural
 networks, are already too powerful. They don't know anything about the data.
 Everything's connected to everything and they argue right here our model should
 have lots of prior knowledge to compensate for all the data we don't have.
 So they allude to the convolutional neural networks constitute one such class
 of models. Their capacity can be controlled by varying the depth and
 breadth and they also make strong and mostly correct assumptions about the
 nature of images, namely stationarity of statistics and locality of pixel
 dependencies. So their argument here is that the convolutional operation is such
 a strong prior that is mostly consistent with what we know about images that
 they are very well suited to computer vision. Again something that was not
 abundantly clear at the time as it is right now. It's interesting to see how
 they get to this point where they say we need lots of capacity but we also need
 a model with lots of prior knowledge and of course CNNs fit that very well.
 So they go into the problems of CNN despite the attractive qualities
 and despite the relative efficiency of their local architecture they are
 prohibitively expensive to apply in large-scale high-resolution images.
 Luckily current GPUs paired with a highly optimized implementation of 2D
 convolution are powerful enough to facilitate the training of interestingly
 large CNNs and recent data sets such as ImageNet contain enough labeled example
 to train such model without severe overfitting. So overfitting was also
 still like very much at the forefront of people's minds back then. Right now we
 don't really care about overfitting that much anymore. Basically we figured out
 that if we just build large enough models we don't overfit which is strange
 in itself like this double descent phenomenon and so on but overfitting was
 still very much at the forefront of people's minds and they do a lot of
 things here to prevent overfitting which gives them kind of a boost in the test
 accuracy which might actually not have been the overfitting that they're
 combating. So they do for example in data augmentation already in this paper
 and they always allude to how this is to prevent overfitting. However we know
 nowadays that it might not be the overfitting that's combated by data
 augmentation. It might actually have something to do with regularizing
 your function making it more smooth and so on. So you just see how
 coming from a classical machine learning perspective overfitting was like the
 number one or one of the number one problems in classical machine learning
 in SVMs and things like this. So it's safe to say that they thought
 if we built these large models we're gonna have a huge overfitting problem
 and yeah so that's why this pulls through right here. Also I guess
 one of the main contributions of this paper is to show to combine this CNN
 training with GPUs. Also not very non-clear at the time like it was known
 that you could do computation on GPUs but the fact that these are you know very
 capable for training these CNNs or generally neural networks wasn't
 something that was you know known at the time. So this paper basically showed that
 if you use a GPU you can get that much faster and that makes it
 possible to train these big neural networks. Again right here the size of
 our network made overfitting a significant problem even with 1.2
 million labeled training examples so we use several effective techniques for
 preventing overfitting and we'll look at those. And the end they say the
 network's size is limited mainly by the amount of memory available on current
 GPUs and by the amount of training time that we are willing to tolerate. Our
 network takes between five and six days to train on two GTX 580 GPUs. All of our
 experiments suggest that our results can be improved by simply waiting for faster
 GPUs and bigger data sets to become available. And I mean that proved to be
 absolutely true. We don't necessarily have bigger data sets right now though
 we do but certainly with faster GPUs and bigger GPUs this became a this became
 these networks became better simply by increasing their depth and as you know
 then ResNets came along increasing the depth by an order of magnitude and that
 gave another boost to computer vision. Alright so they talk about the ImageNet
 data set here and the main point in the ImageNet data set right here is the fact
 that the images are plenty so there are over a million training images in this
 subset with a thousand classes which was you know a very big that was that was on
 like CIFAR 10 had 10 classes, CIFAR 100 had a hundred classes that was already a
 lot. A thousand classes that is like unheard of before this data set. I guess
 not unheard of but yeah and a million training images. Completely crazy and
 also not only was it a lot of images they were resolution was really big so
 in the order of 256 by 256 whereas previous methods all were like 32 by 32
 so definitely challenging data set even today it's a challenging data set. Alright
 so the architecture. The architecture and there's this famous graphic right
 here of the AlexNet architecture so briefly they described these
 convolutional layers right here as you can see there's max pooling already
 here they have dense layers at the end they do generally increase the number
 of feature maps right here while decreasing the resolution with max
 pooling so all of this has sort of you know kept until today I guess they also
 took it from earlier work on convolutional neural networks that
 generally found this to be a good idea and the important part here that is kind
 of special to AlexNet is you can see there are these two different pipelines
 and Alex for cutting off this part right here I mean you just know like this has
 the eight pages we need to like we have like three lines too much how can we fit
 the three lines we've already cropped everything let's just cut off the top
 half here it's essentially the same as the bottom yeah so space constraints and
 PDFs for conference submissions ruining yet another paper alright but you can
 see there is this two this this two column architecture right here so this
 network was so large that it didn't fit on one GPU so they had to split it onto
 two GPUs with the occasional intercommunication right you can see
 here there is intercommunication between the two GPUs and there is also no
 intercommunication right here on this layer this was very intricate that was
 one thing that really didn't hold until today I guess until now with things like
 I don't know G shard or so where you have different weights on different GPUs
 again I guess the the invention of bigger GPUs made that sort of super
 fluid but just imagine the amount of code they had to write there was no
 tensor flow at this point there I don't think there was even cafe around there
 was just CUDA and yeah just this cross GPU memory writing I just imagined this
 to be so so ugly and big respect for writing all of this all of this code
 alright so they they go through a number of important things and most of the
 things here aren't their invention let's say but they cleverly combine things
 that were already known about neural networks and things that were maybe
 developed somewhere that they have found to work really well so the first one is
 the relu non-linearity now of course relu is nowadays all like abundant
 everyone uses relu's non-linearities but at that time it was still very much in
 fashion to use something like the sigmoid right here or the hyperbolic
 tangent and why is that because the neural networks were still kind of
 inspired by the neurons where you had the soma of the neuron and then the
 input dendrites sorry the dendrites with the input axons and then you would sum
 up all the incoming signals and then that would go over so in the true neuron
 you have this this this kind of curve where if the input rises above this
 border right here the action potential maybe I don't know what the the English
 term is then if it rise above that then the neuron would start to spike right
 and if it's below that it wouldn't so people wanted to approximate this using
 some sort of a a kind of differentiable but something that's very similar to
 this step function and that ultimately led to something like a sigmoid or an
 hyperbolic tangent so people trying to stay close to biological neurons did
 this but that gives you the problem that in this region and in this region right
 here you have almost no gradient to learn from so you can see that they
 argue that in terms of training time with gradient descent the saturating
 non-linearity so the hyperbolic tangent and the sigmoid are much slower than the
 non saturating lean non-linearity this one following Narendt Hinton we refer to
 neurons with this non-linearity as rectified linear units so taken from
 this this other paper they say okay we use these relu's these rectified linear
 units which are not exactly like real biological neurons but they train much
 faster right and of course relu's are used until this day so you can see right
 here that a this is on a C for 10 and they measure the time to reach 25% of
 the training error and this here is with the relu's and this here is with the
 hyperbolic tangent and it takes much longer to reach the hyperbolic tangent
 especially it takes six times faster to with the relu's and they say that's one
 of the main components that allows them to learn this fast to even experiment
 with these big networks because their entire training time is six days right
 but they probably didn't train it only once they experimented with it and saw
 what works so if you have a couple of months of time and he takes you a week
 to train one of these things you know you don't you can't afford a six times
 slowdown because that would mean you can only train like two models in the entire
 course of research and that would severely hinder your progress now we are
 at the point where that becomes true again with these giant giant transformer
 language models where people can train it once and then you know like GPT-3
 they say oh we made we discovered a bug halfway through and we've kind of fixed
 it but we're not sure we couldn't restart because it was too expensive
 yeah maybe we're waiting for a moment I'm still saying we're waiting for the
 resonant moment in the transformers but yeah relu's in you know here in not
 introduced here but used here and have been prevailing until today training on
 multiple GPUs something as I said that didn't didn't really get forward from
 here especially the kind of GPU training so if we train on multiple GPUs today
 what we mean is that we have our model right and then we distribute that to
 multiple GPUs like this and then we take a mini batch from the training data and
 we simply split it up let each GPU do its thing on its subset of the mini batch
 and then at the end kind of calculate the loss and then back propagate the
 gradients and synchronize the gradients between that so we have one model that
 is on both GPUs here they distribute a model to two GPUs and I'm also thinking
 that with frameworks like G shard this could potentially have a revival right
 here this kind of distributing your models especially within the same layer
 across many GPUs and then having cross communication only at some points so
 their argument is this only has three gigabytes of memory which limits the
 maximum size of networks can be trained on it turns out that 1.2 train million
 training samples are enough to train networks which are too big to fit on one
 GPU therefore we spread the net across two GPUs current GPUs are particularly
 well suited to cross GPU parallelization as they're able to read from and write
 to one another's memory directly without going through the host machine okay so
 this means that for so sorry here they say the parallelization scheme that we
 employ essentially puts half the kernels or neurons on each GPU with one
 additional trick the GPUs communicate only in certain layers that means that
 for example the kernels of layer 3 take input from all kernel maps in layer 2
 however the kernels in layer 4 take input only from the kernel maps in layer
 3 which reside on the same GPU so very very interesting choice right here and
 they they justify this here or they say the results this scheme reduces our top
 one top five error rates by 1.7 and 1.2 percent respectively as compared with a
 net with half as many kernels in each computational layer in each
 convolutional layer on one GPU the two GPU net takes slightly less time to
 train than the one a GPU net so first of all I have to say big respect right here
 like like I can imagine they did this you know with the relu's and stuff and
 they were already better than previous because they're so just to go to the
 results the pre they beat the error rates of previous models by ginormous
 amount so this is what they knew right here this is on the 2010 image net split
 so the previous best ones were like at around 28 25 percent and here their best
 one is at 17 percent top five error rate I'm gonna imagine that they trained it
 first and we're already better than the 25 percent and I guess lots of people
 would just call it a day would be like oh cool we have this entirely new method
 not only did we show that we can train it we actually showed that it's better
 and bad a boom I have point one percent better error rate and everything else
 can be a separate paper no they stuck with it and they pushed it each so each
 of these things right here they say oh this reduces the error rate by 1% this
 reduces the error rate by 2% and you know really they they went about it how
 far can we push this with everything I mean just imagine you come and you train
 a network I'm pretty sure they first trained on one GPU right and and then
 they thought oh you know maybe we can train an even bigger network by using
 two GPUs and then they realized what it's gonna take like a crap ton amount
 of dumb code to cross synchronize and keep them in lockstep and blah blah blah
 like it's not even easy to write multi GPU code today with all the frameworks
 just imagine that and for them to having already observed that their network does
 better than everything that was previously to sit down and do the cross
 GPU thing experiment with okay when do we cross communicate and whatnot that is
 very very respectable right here so maybe a lesson to be learned or or just
 the mentality of the people maybe they just had more time they were like okay
 it's still like two months out this competition deadline I don't know but
 you know I'm this this is not something that I see today very often this this
 kind of persistence and additional pushing and reporting of what works in
 these kinds of things I mean some some papers do it but most papers do it
 because only with all the tricks they can get that point one percent improvement
 and this one already had the improvement and did it anyway okay but multi GPU
 training didn't really it's like splitting the models across GPUs didn't
 really didn't really stick around mainly because I guess the GPUs got larger in
 memory pretty quickly so it wasn't that necessary but also I guess because the
 frameworks were just too clunky and now maybe with G-shard this is coming back
 so worth another shot I guess next one local response normalization this also
 didn't really stick around I cut kind of dumped in favor of things like batch
 normalization but with the resurfacing of things like layer normalization this
 it comes back to this thing here again a little bit so what they say is that what
 they want to do is they want to kind of normalize the response of these of these
 relu's so what they do is each response which is this alpha they are these a
 here is normalized by the following quantity and it's the all the responses
 of the other neurons around them or of the other kernels around them and you can
 see the sum is over this weird quantity right here so what does it mean if they
 have a bunch of convolutional filters and these are the activation so these are
 the feature maps after the convolution and yeah so if I have like 10
 convolutional filters in my layer this is going to be the output the way they
 normalizes they normalize each filter sorry each output channel by averaging
 by you see here dividing by the average response of the channels around them
 right so let's maybe say the five channels though two channels in front of
 them and two channels behind them this is going to be they take the average
 across this one and then for another channel right here for this one you
 would take the average of the five around that this isn't really something
 that stuck around I guess mainly because of the really dynamic situation right
 here what people do today is they have things like layer normalization that
 simply averages across all of the channels or they have group normalization
 that pre defines these groups like here is there's two groups and we only
 normalize within this group and within this group also always the same this
 kind of dynamic normalization on across neighboring filters as I said didn't
 really stick around not really sure why but I guess it was just easier to
 implement it otherwise or it just worked better again here they say this this it
 was motivated well right this scheme bears some resemblance to the local
 contrast normalization scheme of that but ours would be more correctly termed
 brightness normalization since we do not subtract the mean activity and oh they
 make it connection to biological neurons where is it this sort of response
 normalization implements a form of lateral inhibition inspired by type
 found in real neurons creating competition for big activities amongst
 neuron outputs computed using different kernels okay so kind of inspired by real
 neurons but also kind of inspired by other people doing also some kind of
 normalization so people already knew that normalization was helpful at some
 times and this is what they employed right here again reducing the top error
 rates by 1.4 and 1.2 percent respectively so not a big improvement but still an
 improvement the last thing overlapping pooling again a thing that didn't really
 stick around that much where they say okay instead of having a pooling layer
 so if this is your image and instead of pooling 2x2 in the stride of 2 like
 we do today and you know pull it down to a smaller image what we can do instead
 is we can pool with overlapping windows so in that case they pool with a 3x3
 window but they do always do stride of 2 so they have like these overlaps right
 here resulting in the same size but then each pixel right here has some sort of
 overlapping information from the pixels around it again they say it reduces the
 top one and top five error rates by 0.4 percent and 0.3 percent maybe this this
 didn't stick around because I'm not sure maybe because people found it doesn't
 work in other problems who knows so the overall architecture as we said is
 described in this picture right here so you have the input image which you can
 see has three channels and they use convolutional filters with a here with a
 stride of four at the beginning to reduce the size so at the beginning it's
 224 by 224 and then it's 48 by sorry it's 55 by 55 that thing here 55 by 55
 48 feature maps you can already see as we said before the feature maps keep
 increasing while the number of the dimension the resolution of the image
 keeps decreasing the stride of four convolution here already employed in
 order to down sample the image at the same time as convolving it nowadays a
 lot of architectures will simply not do max pooling at all but always use the
 kind of strided convolution to down sample image while convolving it what
 you also see here is that they thought that the feature map size should be
 should also be large at the beginning and then decrease which is a reasonable
 assumption right because if you have higher resolution images you're
 probably going to need higher resolution feature maps this didn't really come
 through until today as you know most architectures today they just go with
 like three by three kernels from the very start and don't really care about
 you know also downsizing their their filters I don't really know why whether
 it's just more convenient or less parameters or whether there's really
 something to having small filters but I just know you know this is something the
 large filters at the beginning is something that didn't didn't hold over
 time also you can see right here they have multiple dense layers at the end I
 believe most architectures today simply go with two of those instead of three
 so one like hidden layer and then one classification layer but it's you know
 it's it's very close to the architectures today right there hasn't
 changed that much like the difference between this and the VGG 16 VGG 19
 network is just depth and then the difference between those and the
 ResNet is just the whatever these skip connections right here and that's where
 we are today so so there hasn't hasn't changed that much honestly they also
 allude to the fact that actually even though it doesn't look like it most
 parameters are here in these dense layers those are most parameters of the
 network this right here a convolution layer is like 1% of the parameters even
 though it takes up a lot of space in the in the drawing so maybe the reduction in
 the number of classification layers at the end also has something to do with
 the fact that that's where most parameters are so if you get rid of one
 of those dense layers you can like get many many more convolutional layers
 all right so the last part here is on reducing overfitting again they didn't
 really investigate whether or not really their network was overfitting like
 really establishing the overfitting it was I think maybe they did and maybe it
 was actually overfitting but we now we we don't care about overfitting too much
 anymore maybe because we already use these augmentations naturally but also
 because we built these deep models so we somehow have an idea that they
 generalize naturally I'm not sure whether they actually were only worried
 about it that much because of the history of machine learning or whether
 they actually did see that everything was overfitting constantly okay they say
 our neural network architecture has 60 million parameters although the thousand
 classes make each training example impose 10 bits of constraints on the
 mapping from image to label this turns out to be insufficient to learn many
 parameters without considerable overfitting below we describe two
 primary ways in which we combat overfitting again there's no one today
 no one today makes this argument anymore this oh we have this many parameters and
 there are that many images right we have 60 million parameters we have 1.2 million
 images a thousand classes how you know when how many parameters per sample is
 that and so on how many bits of constraint we don't care about we're fine
 with having like a billion times more parameters than training samples we we
 don't worry about it anymore so the first thing they do is data
 augmentation already I mean this was already known again like lots of these
 things here were already known but the combination is just so cool in this
 paper where so first of all again they say the transformed images are generating
 Python code on the CPU while the GPU is training on the previous batch of images
 so these data augmentation schemes are in effect computationally free again
 this code must have been ugly the first form of data augmentation consists of
 generating image translations and horizontal reflections we do this by
 extracting random 224 by 224 patches and their horizontal reflections from the
 256 by 256 images okay so random so this was already this these are the most
 valuable data augmentations that still we have today random horizontal flipping
 is still used in every pipeline of computer vision except if you want to
 read text I guess and random cropping is still the most powerful data
 augmentation technique for images today and the it's crazy that this was already
 discovered and I I don't know whether they say right here how much this
 particular thing improves I don't think they have a stat on how much this
 improves they just say how much this this next thing improves but I'm going
 to guess this was one of the vital things for pushing the performance
 because now we know cropping is very important I guess they thought that they
 they would you know translation was the important part and so they focused on
 generating image translations and to generate an image translation from a
 single image naturally you have to crop it however we we we now focus much more
 on the fact that we crop it and kind of have different sub images of the same
 image especially in you know self-supervised learning and things like
 this we know that cropping is what is like the the power horse of these
 methods so the fact that they extract random patches right here means that
 their network only operates on these sub patches and then they compensate by a
 test time the networks makes a prediction by extracting five patches
 the four corner patches and the center patch as well as their horizontal
 reflections and averaging the prediction made by the networks softmax layer on
 the ten patches I also believe that people don't do this too much nowadays
 they most of the time they simply rescale the test images or something
 like this or a fine-tune at the end on the kind of scale training images there
 are various techniques for doing this but random cropping and horizontal
 flipping already employed right here also color kind of color jittering a
 form of color jittering a very special form altering the intensities of RGB
 channels in training images specifically we perform PCA on the set of RGB pixel
 values throughout the image in a training set to each training image we
 add multiples of the found principal components with magnitudes proportional
 to the corresponding eigenvalues times a random variable drawn from a gauss with
 zero mean and standard deviation point one this is I believe this has gone out
 of fashion so people do color jitter and kind of brightness jitter and so on but
 I don't think they particularly do this kind of PCA based image image
 augmentation right here anymore they say this scheme reduces the top one error
 rate by over 1% I wonder why why this isn't or maybe because you need these
 stats over the entire data set and the other things may be working equivalently
 well but you you can simply apply them without knowing kind of your your
 principal components okay next thing dropout dropout has been you know one of
 the things that was very important throughout the early stages of deep
 learning isn't that important anymore now dropout some people still use it but
 most people I think don't use dropout anymore and it's very interesting to see
 but it definitely was a technique that was used a lot during like from Alex net
 to basically like now or like the last very few years so they say combining the
 predictions of many different models is a very successful way to reduce test
 errors but it appears to be too expensive for big neural networks that
 already take several days to train there is however a very efficient version of
 model combination that only costs about a factor of two during training so
 there's this take this technique called dropout then they explain it to set to
 zero the output of each hidden neuron with probability 0.5 again people
 didn't know about dropout as they do now but they introduced this right here and
 they say it reduces their not sure that they also don't say how they how much
 they by how much this reduces the training error but they say we use drop
 out in the first two fully connected layers without dropout our network
 exhibits substantial overfitting dropout roughly doubles the number of iterations
 required to converge so okay so they did actually make sure or they did find the
 actual evidence of overfitting and saw that dropout reduces that and I wonder
 why this doesn't happen nowadays maybe because we have the we have less of
 these fully connected layers but I can't really imagine maybe because we do more
 augmentation I don't I don't know or maybe dropout is still used and I'm just
 I just don't know it and don't see it yeah so here they use momentum to train
 this and they do some qualitative analysis they do some qualitative
 analysis so first of all they say okay they shatter all of the previous
 approaches especially also then they build kind of ensemble methods and they
 pre-trained they already do transfer learning they already pre-trained on
 image net 2011 and fine-tune then on the image net 2012 right here the image net
 2011 and then fine-tuning on the image net 2012 to reduce that error even
 further like pulling all the tricks all these things are around still very cool
 and then they look into what their network learned so they find that there
 are a number of these kind of filters you see these 11 by 11 filters in the
 first layer where they show okay this really and this was kind of already
 known that these neural networks extract filters like this like color gradients
 or edge detectors in various forms and directions and it's cool to see that
 this one also does so this one here is also a very cool investigation where
 they look at examples and the red bar the red one is always the correct label
 and the bars are basically what their model says are the top five things and
 it's cool to look at so for example here you have might as the top one but then
 also black widow cockroach tick starfish but the top labels are usually also very
 very good labels you can see here grill and it assigns convertible which you
 know by all means is correct it's just not the class that the annotators
 assigned to this particular image as well as here Dalmatian was the highest
 prediction of the network where the label was actually cherry and this is
 this is quite debatable right so you can see that a lot of the mistakes the
 network does is are are you know forgivable let's say and you can see
 that for when the network doesn't do mistakes the not only the top label is
 good but a lot of the top five labels are also very very adequate lastly they
 look at a given training set image which these are the training set images right
 here and they look at the last layers feature vector and the five nearest or
 the nearest neighbors in Euclidean space of the entire training data set and
 here's what you come up with so you can see for the elephant the nearest
 neighbors are all other elephants and regard that they are in different poses
 right they don't always look the same way these elephants also these dogs
 right here so it's pretty cool to see that the network actually learns some
 invariances across the class and puts images with the same label into the same
 area in the embedding space yeah so that's their that's their paper that they
 they already allude to the fact that depth is very important it is notable
 that our networks performance degrades if a single convolutional layer is
 removed for example removing any of the middle layers results in a loss of about
 2% for the top one performance of the network so the depth really is important
 for achieving our results and as you know this spurred an area of this burden
 area of trying to build deeper and deeper networks until Resnets came along
 and built ultra deep networks they also say we did not use any unsupervised
 pre-training even though we expect that it will help especially if we obtain
 enough computational power to significantly increase the size of the
 network without obtaining a corresponding increase of the amount of
 labeled data thus far our results have improved as we have made our network
 larger and trained it longer but we still have many orders of magnitude to
 go in order to match the infrared temporal pathway of the human visual
 system ultimately with ultimately we would like to use very large and deep
 convolutional nets on video sequences where the temporal structure provides
 very helpful information that is missing of our far less obvious in static images
 so already the previewing of future research here with the self supervised
 with the many more layers and so on astounding that this kind of foresight
 and of course all of this proved to be you know very very adequate predictions
 right here and yeah so this was the paper right here the paper that kicked
 off deep learning I enjoy reading kind of these old papers especially looking
 back at what was already known what still is around which turns out to be a
 lot a lot is still around and the choices that people made back then some
 of them defined our modern field so that was it for Alex net let me know what you
 think in the comments and I'll see you next time bye