[Classic] Deep Residual Learning for Image Recognition (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper vision computer vision kaiming he google resnet resnet50 resnet151 deep neural network imagenet residual identity function very deep convolutional neural network bottleneck overfitting
#ai #research #resnet

ResNets are one of the cornerstones of modern Computer Vision. Before their invention, people were not able to scale deep neural networks beyond 20 or so layers, but with this paper's invention of residual connections, all of a sudden networks could be arbitrarily deep. This led to a big spike in the performance of convolutional neural networks and rapid adoption in the community. To this day, ResNets are the backbone of most vision models and residual connections appear all throughout deep learning.

OUTLINE:
0:00 - Intro & Overview
1:45 - The Problem with Depth
3:15 - VGG-Style Networks
6:00 - Overfitting is Not the Problem
7:25 - Motivation for Residual Connections
10:25 - Residual Blocks
12:10 - From VGG to ResNet
18:50 - Experimental Results
23:30 - Bottleneck Blocks
24:40 - Deeper ResNets
28:15 - More Results
29:50 - Conclusion & Comments

Paper: https://arxiv.org/abs/1512.03385

Abstract:
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers.
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun


Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there! Today we'll look at deep residual learning for image recognition by Kai Ming He, Xiang Yu Cheng, Shao Qing Ren and Jian Sun. So this, you know it, this is an old paper. It is from 2015 but I thought we'd still look at it because this not only is it one of the most influential papers in modern deep learning, it is also a very well written paper and I remember it like it was yesterday when this came out. This was like a bomb. So around that time this this meme was going around. I was winning ImageNet but then someone made a deeper net. This was a this was a the time when after AlexNet people were trying to build bigger and bigger networks and every time someone managed to build a bigger network the accuracy on ImageNet data set would increase pretty much in lockstep with how much bigger the network was but people got to the limit of building big networks and then this paper drops and changed everything and now residual connections are everywhere not only in image recognition they are in transformers they are in whatever wherever you go you'll probably find some residual connections somewhere in there. So yeah let's let's look at this paper and let's revisit what kind of problems people had then and how they solved it. So here they go directly into into this problem of deep neural networks and the problem that people had was they knew that if you can increase the if you can increase the depth of a neural network you can make it perform better you can make it generalize better you can reach lower training loss but optimizing it was hard. Specifically this was a phenomenon that people observed. So if you have a 20 layer neural network you could train it and you know there is this learning rate drop people have all had already figured out that you need to drop drop the learning rate and it would reach a certain level and here this would be the test error over here. However if after a certain point if they increase the depth even more the training error would actually go up again and so would the test error and this is not a problem of overfitting because overfitting would be when the training error is lower or as low and then the test error went up. So this is the first thing this is not a phenomenon of overfitting of too many parameters so why can't we train bigger layers networks until that time have very much followed kind of the original network design that was envisioned by sort of people like Jan LeCun and also Alex Net and the most popular ones were these VGG nets and they were very much of the philosophy that you'd have like some you have the image here and you input that into convolutional layers which first would kind of keep a big resolution but would increase the channel size by you know some amount and then you would sort of downscale the image as you increase the number of filters so you would stack more and more filters and draw more filters you would stack more and more filters while downscaling the resolution of the image the reasoning was that if you do image classification right then you know where on this where on this maybe you want to classify this into a Lego tower or whatever that is it's not that important where it is so on the lower levels you would want to parse out like very low layer features like edges and so on and these are still important where they are right the fact that here's an edge here's an edge here's an edge but then as you go higher up and go to more and more abstracted features and we already knew that these neural network they tend to learn more and more abstract features as you go up the layers the hypothesis was that the exact localization of these abstract features would be less and less important so if there is if you recognize that there is a rectangle it's not that important where it is just that it's somewhere there and maybe where it is in relation to the other so if you have if you recognize want to recognize a car the lower layers would recognize the fact that there are edges and then the intermediate layers would recognize the geometric shapes of maybe here the wheels and these bodies but it's not that important where exactly they are and then the higher layers would learn to combine the individual parts to each other and again it becomes less and less important where these things are and more and more important that you build more expressive features so people would downscale the resolution upscale the number of filters now that's a good heuristic but this is based this was basically the architecture of these networks and we would question why would if we increase the number of layers so if we instead of one here we have two of these layers right we simply have two of these layers and here we have two of these layers why does it get worse especially this paper here makes an interesting observation so it is not caused caused by overfitting and adding more layers leads to a higher training error the degradation indicates that is not all systems are similarly easy to optimize let us consider a shallower architecture and its deeper counterparts that adds more layers onto it there exists a solution by construction to the deeper model the added layers are identity mapping and the other layers are copied from the learned shallower model so pretty easy if you have a shallow model like five layers that learns a particular function I can pretty easily prove that there is a deep model that learns the same function by simply copying over these five layers and having these here learn the identity function okay so if we are able to learn this we should be able to train this network to at least the same accuracy that's what this paper argues because it can you know these layers can simply learn the identity function so it must have something to do with the easiness of optimizing these deep architectures not with overfitting this is I think if you read the entire text here it's very very clear if you read it they lead you through this reasoning saying that look all these layers have to do is learn the identity function and then we could at least get the same accuracy so so what why don't they learn the identity function well because we initialize most weights you know towards zero we initialize them randomly but mostly we initialize them around zero our initialization procedure usually sample from some Gaussian with some kind of a standard deviation but around the mean of zero and also if we use things like weight decay L2 regularization all of these things they do they bias the weights towards zero so if there is any natural thing that these networks are good at is they learn the zero function really well learning the identity function is as difficult as learning any other function the identity function the convolutional filter is actually pretty difficult to learn because you know if I have a my if I have my three by three filter where is my no no this is my three by three filter the identity function is it like a one here and zeros everywhere else that's the that that would be one of the things it's not that easy you need to learn nine weights in the correct way so this paper says can we do something to make the default function of the network not be the zero function or whatever the randomly initialized function can we make the default function the one function can we make the default function the identity function and that brings you to residual connection so instead of learning to transform X via a neural network into X which is the identity function why don't we have X stay X and then learn whatever we need to change okay so if let's call that tilde if the assumption is that it's a good default to not change much so this is almost the same as this we might make this build this directly into the architecture the fact that these two are equal plus plus some deviation that is learned right here and the hypothesis is that especially the deeper you go if you go very deep each function here will actually learn not that much it will learn to basically change the signal a little bit but mostly it will learn the identity function if it behaves well and therefore it might be you know reasonable to build this into the architecture and of course this has turned out to be very accurate it has actually been reasonable to build this into the architecture so that's what they propose right here so instead of just having weight layers one after another what they propose is to have these skip connections in here so these skip connections they will instead of learning the function they call this entire function H of X which might be very complicated they learn the function whatever F and F is whatever you need to change about X you see at the end you add X to it so these weight layers here they simply learn whatever makes this next this output different from this input and learning differences now you have the desired property because what do we know about weight layers from before well they tend towards the zero function right if we use weight decay or generally how we initialize them they tend towards the zero function well if F tends towards the zero function then H becomes the identity function so the default function of this network is the identity function and whenever we learn something we learn how to deviate from the identity function and that is that is a much better default function now it's not entirely true that the default function is the identity function you see that here for example there's after the skip connection there is actually a relu so there's still it's still a nonlinear function in total the network in total but the default for the individual blocks here is the identity okay now if you chain these blocks you get a residual network and that's what they propose right here so on the left you see this original VGG architecture like we described it so you can see you have an image which has four channels and you first up it to 64 channels you keep the resolution and then you max pool which halves the resolution but you go up with the filters to 128 you max pool again go up with the filters and so on now this has even though it doesn't look like it this has a lot of parameters and it needs a lot of computation so it has 19.6 billion floating point operation for a forward pass in contrast the networks we're going to build here the residual networks have 3.6 billion flops so they are much much less in terms of complexity than the old VGG networks while still being much deeper okay the hypothesis is the deeper the better and as a trade-off per layer you don't actually need to have that many parameters because you don't learn that much per layer but the succession of layers it gains you much more than simply having single massive layers you can see at the same size of resolution here you the the Resnets can get away with much less amounts of filters and that's why they are less they are of less size so this is the comparison the VGG 19 now they do build this 34 layer network which they call plane and you can see it is simply a 34 layer network with no pooling right here and here instead of pooling they do a stride to convolution which has also become this has become kind of more standard than doing max or average pooling to downscale to do simply stride to convolution so this paper has actually set the standards for a lot of things in modern deep learning so our goal is to going to be to compare first of all the VGG 19 to the 34 layer plane to show that you will lose performance when you simply up the number of layers but then when you introduce the residual connections as you can see right here so there is always this jumping connection right here so along these jumping connections the signal can travel as the identity function what we're going to see is that if we go from plane to residual introducing no extra parameters just these skip connections will change everything will make this network all of a sudden trainable and make the deeper networks the better networks okay the only little caveat here is of course in order to build a residual connection the output has to be of the same size as the input because you need to add the input to the output and this here for example is not given so here you can see this signal after this layer is going to be half as big because it's a stride to convolution so the output right here is only half the size but it is it is twice the number of filters you can see right here this has 64 filters and here we go to 128 filters that's why this connection right here has parameters in order to simply expand the number of filters these are these one by one convolutions that simply up that simply project the 64 filters to 128 filters however this doesn't introduce too many parameters because it's only one by one in fact here the 34 parameters residual network no I'm wrong you have different options so the world has ended up at the option of doing one by one convolutions but in this paper they still they still explore three different options and I guess here in this particular experiment the option a is simply to zero pad so to leave the first 64 channels but to simply append 128 zero padded filters there or channels option B is the one by one convolution and option C is actually that all of these connections right here also have the one by one convolutions which introduces extra parameters and they they realize that option C isn't improving over option B substantially and in fact is only improving marginally and they say okay that's probably just because we have more parameters so ultimately they went with option B and I think that's what the world does right now I also I when I read this first I particularly enjoyed this paragraph right here let's read it together our implementation for image net follows the practice in the data the image is resized with shorter randomly sampled in this for scale augmentation a this crop is randomly sampled from the image or its horizontal flip with the per pixel has been subtracted the standard color augmentation is used we adopt the batch normalization right after each convolution before activation this an age-old discussion was born when to use batch normalization before come before the activation or after the activation I still I think people are still fighting over this today we initialize the weights as in 13 and train all plain residual nets from scratch use SGD data data data the learning rate starts from this is divided by then so here in this perhaps in this paragraph they detail basically all the training procedure and all the tricks that they use that I remember specifically that you know I've read all of this which was the idea and I could follow like oh this is super well explained this is so cool and so on and then I expect basically an implementation of that and then there's one single paragraph with like 20 lines saying oh and by the way we use these 50 tricks from these other papers and yeah that's when it I guess it was already happening you needed to do all the modern tricks in order to really reach the top accuracies but you know in hindsight we know it wasn't the tricks that helped them it was actually their idea I just I just thought it was rather funny so you can see right here the results of this if you look at the left these are the plane networks and we've already sort of seen this now this is on image net right here you can see the 18 layer network simply has lower train and validation accuracy so the solid line here is the validation on image net bold curves denote the validation error of the center crops so I guess they do yeah they do center crops so the training error is going to be higher because they do these different augmentations but you can see the training and the validation error are higher in the deeper network if you don't use residual connections again this is not due to overfitting and it this is because we can't train these deep networks because we should be able to the solution space of the 18 layer network is a subspace of the solution space of the 34 layer network everything tells us we should be able to learn the 34 layers to at least the accuracy of the 18 layers but we can't however introduce residual connections bum bum bum bum and you can see that the trend is exactly reversed now the 34 layer with residual connections has a much much lower training and validation error than the 18 layer in fact look at this table right here if you introduce the residual connections to the 18 layers it's marginally better however if you introduce the residual connections to the 34 layers it is a lot better and this is another testament to the fact that these residual connections they really help more and more the deeper you go you can see the effect in so this 18 layers this is sort of a VGG 19 depth network well if and there we already know we can train these without residual connections right because we were able to train VGG 19 however if we go higher to more layers we can these residual connections all of a sudden make it a lot a lot better you can see that it's not it's not that we can't train the 34 layers but the residual connections just help a lot more and it most of a sudden it most of most importantly they don't degrade the performance from the shallower network so they they explore the different options right here and compare it to others different options as I said being a B and C where a is the zero padding for the projection B is having projections simply between where the channels don't fit and C being having projections in every single residual connection and you can see right here that the option B gives you quite a bit of a boost well option C doesn't give you that much of a boost introduces many more parameters and you know overall is I guess they decided against it which since then the world has also decided against it they also do deeper networks so they built deeper networks like 50 layer resnet 101 layer resnet and 152 layer resnet and the 152 layer resnet ended up being the best one as you can see here and you can see a pretty gain like it almost almost lock step gain depth more depth means better network in this at the time this these numbers they were unheard of like even 50 layer deep neural network was bombastic but a hundred and fifty two layers it was it was crazy and the fact that still it has less parameters than the VGG 19 and performs better that was mind mind-blowing absolutely mind-blowing and then at the end they built an ensemble of these models and ended up taking the 2015 ImageNet competition winner that was still like very important back then it was still very important who wins who wins ImageNet that year where I think I haven't even followed up on the last few years it's some kind of wide fixed resnet whatnot with pre-trained and 50 billion extra data yeah so for the deeper networks they decide that they are computationally rather become rather expensive so they introduce these bottleneck blocks here on the right where as you can see so here if you have a 64 dimensional input you do 64 feature channels in your convolution have a 64 dimensional output you can save computation if you first project the higher so here you have a 256 dimensional input and they say we can save computational power by pretty much projecting down to 64 first because then our complexity of this layer which is the expensive layer will be the same as the complexity of one of these layers and then we can project up again the one by one convolution they are significantly lower computational intensive than the three by three convolutions like it's nine times less operations if you think about it so that's what they use to build the deeper residual networks and these residual networks the ResNet 50, 101, 152 they are still staples today you can have these are you can have pre-trained versions of those and people still use it like ResNet 50 is used in every segmentation whatnot application so yeah this has turned out these decisions here have you know made it long way here you can see the number of parameters in these residual networks and this was the absolute craziest thing right here 1202 layers okay so you can see still until here ResNet 110 now this is on CIFAR 10 right here not on image net anymore but you can see that even 110 layers still had less parameters or actually the select the same order of parameters than these previous networks that were only 19 layers deep this was unheard of and much more unheard of 1202 layer network to train on CIFAR 10 it's a bit of an overkill but they say their goal was explicitly to study depth and you can see here that with the deeper and deeper networks they outperformed all of the previous networks so all of the baselines and themselves as they went deeper and deeper and deeper however once you go to 1002 layers you go up again so here's the question was this all just kind of a trick a hack and do we run into the same problem again and that's the question they ask themselves and the answer is no so if you look right here so here you see again the plane networks in the plane networks you can pretty easily see that the more layers you have the higher your error goes whereas in the residual network it's exactly the opposite way the more layers you have the lower your error and if you compare this 110 layer network with the 1200 layer network you see your validation error going up again however your training error and I can't zoom in more but it's the same it's the same and it's at zero so here they conclude and the the here they conclude now we are overfitting they don't use like the biggest data augmentation like we use today so overfitting was still a thing back then so now they conclude okay now we have actually built a large enough network that is overfitting and then and the fact that we go up again in the training error is due to the fact that we are probably overfitting so not only have they enabled us to build deeper networks they have effectively shown that this can get you to the to the point where you don't need deeper networks anymore at least on C410 because you are overfitting and it can effectively get you there this is a lot of evidence for the fact that this biasing the networks towards the identity function is a very valid thing to do and is the solution to the we can't train deep networks problems lastly they investigate the size of the responses so their hypothesis is that if if it is really beneficial to bias the network towards the identity function and if it is really true that each of these layers only learns a little bit right because the identity function is already very good each of these layers only needs to learn kind of a small function they look at the responses of these things so the the response magnitude of these layers right here of the signal through the layers and they compare those with the response magnitude of the other neural networks where you don't have the skip connection the hypothesis is if we look at these then the responses of these layers should be much larger because they have to learn much more and the responses here will be much smaller because the identity function is already doing most of the work and that's exactly what you find so here the layers are ordered by response and you can see the plane networks in the dashed lines are significantly above the residual network even and that's not a function of the depth because if the depth was actually equal here you would expect that the dashed lines would would stretch like this right they would kind of stretch out however exactly the opposite is happening you can see that the residual networks even at the beginning their responses are very much smaller and this is kind of what I like about this paper it's it's one narrative it is a hypothesis and then every single like the the hypothesis is taken and they make predictions from the hypothesis they say okay if we are right with our hypothesis not only should our idea get us better accuracy that's what most people most papers do today but also you know but also it should be that we can for example push our network to the brink of where we actually are overfitting like here and it should also be that the responses of our signal through our layers is smaller and yeah that's research like this is just pretty pretty cool and it's I think a lesson for us that sadly the world has taken the resonance but the world hasn't all taken the research methodology of this paper yeah if you again if you want a good read it's very well written you I'm very sure you can follow it even if you have read very few papers and with that yeah I hope you enjoyed this please tell me what you think of going through kind of old papers looking at whether or not they have stood the test of time and yeah any other comments leave them in the comments I do read them and I'll see you next time bye bye
 Hi there! Today we'll look at deep residual learning for image recognition
 by Kai Ming He, Xiang Yu Cheng, Shao Qing Ren and Jian Sun. So this, you know it, this is an
 old paper. It is from 2015 but I thought we'd still look at it because this not
 only is it one of the most influential papers in modern deep learning, it is
 also a very well written paper and I remember it like it was yesterday when
 this came out. This was like a bomb. So around that time this this meme was
 going around. I was winning ImageNet but then someone made a deeper net. This
 was a this was a the time when after AlexNet people were trying to build
 bigger and bigger networks and every time someone managed to build a bigger
 network the accuracy on ImageNet data set would increase pretty much in lockstep
 with how much bigger the network was but people got to the limit of building big
 networks and then this paper drops and changed everything and now residual
 connections are everywhere not only in image recognition they are in
 transformers they are in whatever wherever you go you'll probably find
 some residual connections somewhere in there. So yeah let's let's look at this
 paper and let's revisit what kind of problems people had then and how they
 solved it. So here they go directly into into this problem of deep neural
 networks and the problem that people had was they knew that if you can increase
 the if you can increase the depth of a neural network you can make it perform
 better you can make it generalize better you can reach lower training loss but
 optimizing it was hard. Specifically this was a phenomenon that people observed. So
 if you have a 20 layer neural network you could train it and you know there is
 this learning rate drop people have all had already figured out that you need to
 drop drop the learning rate and it would reach a certain level and here this would
 be the test error over here. However if after a certain point if they increase
 the depth even more the training error would actually go up again and so would
 the test error and this is not a problem of overfitting because overfitting would
 be when the training error is lower or as low and then the test error went up. So
 this is the first thing this is not a phenomenon of overfitting of too many
 parameters so why can't we train bigger layers networks until that time have
 very much followed kind of the original network design that was envisioned by
 sort of people like Jan LeCun and also Alex Net and the most popular ones were
 these VGG nets and they were very much of the philosophy that you'd have like
 some you have the image here and you input that into convolutional layers
 which first would kind of keep a big resolution but would increase the
 channel size by you know some amount and then you would sort of downscale the
 image as you increase the number of filters so you would stack more and more
 filters and draw more filters you would stack more and more filters while
 downscaling the resolution of the image the reasoning was that if you do image
 classification right then you know where on this where on this maybe you want to
 classify this into a Lego tower or whatever that is it's not that important
 where it is so on the lower levels you would want to parse out like very low
 layer features like edges and so on and these are still important where they are
 right the fact that here's an edge here's an edge here's an edge but then
 as you go higher up and go to more and more abstracted features and we already
 knew that these neural network they tend to learn more and more abstract features
 as you go up the layers the hypothesis was that the exact localization of
 these abstract features would be less and less important so if there is if you
 recognize that there is a rectangle it's not that important where it is just that
 it's somewhere there and maybe where it is in relation to the other so if you
 have if you recognize want to recognize a car the lower layers would recognize
 the fact that there are edges and then the intermediate layers would recognize
 the geometric shapes of maybe here the wheels and these bodies but it's not that
 important where exactly they are and then the higher layers would learn to
 combine the individual parts to each other and again it becomes less and less
 important where these things are and more and more important that you build
 more expressive features so people would downscale the resolution upscale the
 number of filters now that's a good heuristic but this is based this was
 basically the architecture of these networks and we would question why would
 if we increase the number of layers so if we instead of one here we have two of
 these layers right we simply have two of these layers and here we have two of
 these layers why does it get worse especially this paper here makes an
 interesting observation so it is not caused caused by overfitting and adding
 more layers leads to a higher training error the degradation indicates that is
 not all systems are similarly easy to optimize let us consider a shallower
 architecture and its deeper counterparts that adds more layers onto it there
 exists a solution by construction to the deeper model the added layers are
 identity mapping and the other layers are copied from the learned shallower
 model so pretty easy if you have a shallow model like five layers that
 learns a particular function I can pretty easily prove that there is a deep
 model that learns the same function by simply copying over these five layers and
 having these here learn the identity function okay so if we are able to learn
 this we should be able to train this network to at least the same accuracy
 that's what this paper argues because it can you know these layers can simply
 learn the identity function so it must have something to do with the easiness
 of optimizing these deep architectures not with overfitting this is I think if
 you read the entire text here it's very very clear if you read it they lead you
 through this reasoning saying that look all these layers have to do is learn the
 identity function and then we could at least get the same accuracy so so what
 why don't they learn the identity function well because we initialize most
 weights you know towards zero we initialize them randomly but mostly we
 initialize them around zero our initialization procedure usually sample
 from some Gaussian with some kind of a standard deviation but around the mean
 of zero and also if we use things like weight decay L2 regularization all of
 these things they do they bias the weights towards zero so if there is any
 natural thing that these networks are good at is they learn the zero function
 really well learning the identity function is as difficult as learning any
 other function the identity function the convolutional filter is actually pretty
 difficult to learn because you know if I have a my if I have my three by three
 filter where is my no no this is my three by three filter the identity
 function is it like a one here and zeros everywhere else that's the that that
 would be one of the things it's not that easy you need to learn nine weights in
 the correct way so this paper says can we do something to make the default
 function of the network not be the zero function or whatever the randomly
 initialized function can we make the default function the one function can we
 make the default function the identity function and that brings you to
 residual connection so instead of learning to transform X via a neural
 network into X which is the identity function why don't we have X stay X and
 then learn whatever we need to change okay so if let's call that tilde if the
 assumption is that it's a good default to not change much so this is almost the
 same as this we might make this build this directly into the architecture the
 fact that these two are equal plus plus some deviation that is learned right
 here and the hypothesis is that especially the deeper you go if you go
 very deep each function here will actually learn not that much it will
 learn to basically change the signal a little bit but mostly it will learn the
 identity function if it behaves well and therefore it might be you know
 reasonable to build this into the architecture and of course this has
 turned out to be very accurate it has actually been reasonable to build this
 into the architecture so that's what they propose right here so instead of
 just having weight layers one after another what they propose is to have
 these skip connections in here so these skip connections they will instead of
 learning the function they call this entire function H of X which might be
 very complicated they learn the function whatever F and F is whatever you need to
 change about X you see at the end you add X to it so these weight layers here
 they simply learn whatever makes this next this output different from this
 input and learning differences now you have the desired property because what
 do we know about weight layers from before well they tend towards the zero
 function right if we use weight decay or generally how we initialize them they
 tend towards the zero function well if F tends towards the zero function then H
 becomes the identity function so the default function of this network is the
 identity function and whenever we learn something we learn how to deviate from
 the identity function and that is that is a much better default function now
 it's not entirely true that the default function is the identity function you
 see that here for example there's after the skip connection there is actually a
 relu so there's still it's still a nonlinear function in total the network
 in total but the default for the individual blocks here is the identity
 okay now if you chain these blocks you get a residual network and that's what
 they propose right here so on the left you see this original VGG architecture
 like we described it so you can see you have an image which has four channels
 and you first up it to 64 channels you keep the resolution and then you max
 pool which halves the resolution but you go up with the filters to 128 you max
 pool again go up with the filters and so on now this has even though it doesn't
 look like it this has a lot of parameters and it needs a lot of
 computation so it has 19.6 billion floating point operation for a forward
 pass in contrast the networks we're going to build here the residual
 networks have 3.6 billion flops so they are much much less in terms of complexity
 than the old VGG networks while still being much deeper okay the hypothesis is
 the deeper the better and as a trade-off per layer you don't actually need to
 have that many parameters because you don't learn that much per layer but the
 succession of layers it gains you much more than simply having single massive
 layers you can see at the same size of resolution here you the the Resnets can
 get away with much less amounts of filters and that's why they are less they
 are of less size so this is the comparison the VGG 19 now they do build
 this 34 layer network which they call plane and you can see it is simply a 34
 layer network with no pooling right here and here instead of pooling they do a
 stride to convolution which has also become this has become kind of more
 standard than doing max or average pooling to downscale to do simply stride
 to convolution so this paper has actually set the standards for a lot of
 things in modern deep learning so our goal is to going to be to compare first
 of all the VGG 19 to the 34 layer plane to show that you will lose performance
 when you simply up the number of layers but then when you introduce the residual
 connections as you can see right here so there is always this jumping connection
 right here so along these jumping connections the signal can travel as the
 identity function what we're going to see is that if we go from plane to
 residual introducing no extra parameters just these skip connections will change
 everything will make this network all of a sudden trainable and make the deeper
 networks the better networks okay the only little caveat here is of course in
 order to build a residual connection the output has to be of the same size as
 the input because you need to add the input to the output and this here for
 example is not given so here you can see this signal after this layer is going to
 be half as big because it's a stride to convolution so the output right here is
 only half the size but it is it is twice the number of filters you can see right
 here this has 64 filters and here we go to 128 filters that's why this
 connection right here has parameters in order to simply expand the number of
 filters these are these one by one convolutions that simply up that simply
 project the 64 filters to 128 filters however this doesn't introduce too many
 parameters because it's only one by one in fact here the 34 parameters residual
 network no I'm wrong you have different options so the world has ended up at the
 option of doing one by one convolutions but in this paper they still they still
 explore three different options and I guess here in this particular experiment
 the option a is simply to zero pad so to leave the first 64 channels but to
 simply append 128 zero padded filters there or channels option B is the one by
 one convolution and option C is actually that all of these connections right here
 also have the one by one convolutions which introduces extra parameters and
 they they realize that option C isn't improving over option B
 substantially and in fact is only improving marginally and they say okay
 that's probably just because we have more parameters so ultimately they went
 with option B and I think that's what the world does right now I also I when I
 read this first I particularly enjoyed this paragraph right here let's read it
 together our implementation for image net follows the practice in the data
 the image is resized with shorter randomly sampled in this for scale
 augmentation a this crop is randomly sampled from the image or its horizontal
 flip with the per pixel has been subtracted the standard color
 augmentation is used we adopt the batch normalization right after each
 convolution before activation this an age-old discussion was born when to use
 batch normalization before come before the activation or after the activation I
 still I think people are still fighting over this today we initialize the weights
 as in 13 and train all plain residual nets from scratch use SGD data data
 data the learning rate starts from this is divided by then so here in this
 perhaps in this paragraph they detail basically all the training procedure and
 all the tricks that they use that I remember specifically that you know I've
 read all of this which was the idea and I could follow like oh this is super
 well explained this is so cool and so on and then I expect basically an
 implementation of that and then there's one single paragraph with like 20 lines
 saying oh and by the way we use these 50 tricks from these other papers and yeah
 that's when it I guess it was already happening you needed to do all the
 modern tricks in order to really reach the top accuracies but you know in
 hindsight we know it wasn't the tricks that helped them it was actually their
 idea I just I just thought it was rather funny so you can see right here the
 results of this if you look at the left these are the plane networks and we've
 already sort of seen this now this is on image net right here you can see the 18
 layer network simply has lower train and validation accuracy so the solid line
 here is the validation on image net bold curves denote the validation error of
 the center crops so I guess they do yeah they do center crops so the training
 error is going to be higher because they do these different augmentations but you
 can see the training and the validation error are higher in the deeper network
 if you don't use residual connections again this is not due to overfitting
 and it this is because we can't train these deep networks because we should be
 able to the solution space of the 18 layer network is a subspace of the
 solution space of the 34 layer network everything tells us we should be able to
 learn the 34 layers to at least the accuracy of the 18 layers but we can't
 however introduce residual connections bum bum bum bum and you can see that the
 trend is exactly reversed now the 34 layer with residual connections has a
 much much lower training and validation error than the 18 layer in fact look at
 this table right here if you introduce the residual connections to the 18
 layers it's marginally better however if you introduce the residual connections
 to the 34 layers it is a lot better and this is another testament to the fact
 that these residual connections they really help more and more the deeper you
 go you can see the effect in so this 18 layers this is sort of a VGG 19 depth
 network well if and there we already know we can train these without residual
 connections right because we were able to train VGG 19 however if we go higher
 to more layers we can these residual connections all of a sudden make it a
 lot a lot better you can see that it's not it's not that we can't train the 34
 layers but the residual connections just help a lot more and it most of a sudden
 it most of most importantly they don't degrade the performance from the
 shallower network so they they explore the different options right here and
 compare it to others different options as I said being a B and C where a is the
 zero padding for the projection B is having projections simply between where
 the channels don't fit and C being having projections in every single
 residual connection and you can see right here that the option B gives you
 quite a bit of a boost well option C doesn't give you that much of a boost
 introduces many more parameters and you know overall is I guess they decided
 against it which since then the world has also decided against it they also
 do deeper networks so they built deeper networks like 50 layer resnet 101 layer
 resnet and 152 layer resnet and the 152 layer resnet ended up being the best one
 as you can see here and you can see a pretty gain like it almost almost lock
 step gain depth more depth means better network in this at the time this these
 numbers they were unheard of like even 50 layer deep neural network was
 bombastic but a hundred and fifty two layers it was it was crazy and the fact
 that still it has less parameters than the VGG 19 and performs better that was
 mind mind-blowing absolutely mind-blowing and then at the end they
 built an ensemble of these models and ended up taking the 2015 ImageNet
 competition winner that was still like very important back then it was still
 very important who wins who wins ImageNet that year where I think I
 haven't even followed up on the last few years it's some kind of wide fixed resnet
 whatnot with pre-trained and 50 billion extra data yeah so for the deeper
 networks they decide that they are computationally rather become rather
 expensive so they introduce these bottleneck blocks here on the right
 where as you can see so here if you have a 64 dimensional input you do 64 feature
 channels in your convolution have a 64 dimensional output you can save
 computation if you first project the higher so here you have a 256
 dimensional input and they say we can save computational power by pretty much
 projecting down to 64 first because then our complexity of this layer which is
 the expensive layer will be the same as the complexity of one of these layers
 and then we can project up again the one by one convolution they are
 significantly lower computational intensive than the three by three
 convolutions like it's nine times less operations if you think about it so
 that's what they use to build the deeper residual networks and these residual
 networks the ResNet 50, 101, 152 they are still staples today you can have these
 are you can have pre-trained versions of those and people still use it like
 ResNet 50 is used in every segmentation whatnot application so yeah this has
 turned out these decisions here have you know made it long way here you can see
 the number of parameters in these residual networks and this was the
 absolute craziest thing right here 1202 layers okay so you can see still until
 here ResNet 110 now this is on CIFAR 10 right here not on image net anymore but
 you can see that even 110 layers still had less parameters or actually the
 select the same order of parameters than these previous networks that were only
 19 layers deep this was unheard of and much more unheard of 1202 layer network
 to train on CIFAR 10 it's a bit of an overkill but they say their goal was
 explicitly to study depth and you can see here that with the deeper and deeper
 networks they outperformed all of the previous networks so all of the
 baselines and themselves as they went deeper and deeper and deeper however once
 you go to 1002 layers you go up again so here's the question was this all just
 kind of a trick a hack and do we run into the same problem again and that's
 the question they ask themselves and the answer is no so if you look right here
 so here you see again the plane networks in the plane networks you can pretty
 easily see that the more layers you have the higher your error goes whereas in
 the residual network it's exactly the opposite way the more layers you have
 the lower your error and if you compare this 110 layer network with the 1200
 layer network you see your validation error going up again however your
 training error and I can't zoom in more but it's the same it's the same and it's
 at zero so here they conclude and the the here they conclude now we are
 overfitting they don't use like the biggest data augmentation like we use
 today so overfitting was still a thing back then so now they conclude okay now
 we have actually built a large enough network that is overfitting and then and
 the fact that we go up again in the training error is due to the fact that
 we are probably overfitting so not only have they enabled us to build deeper
 networks they have effectively shown that this can get you to the to the point
 where you don't need deeper networks anymore at least on C410 because you are
 overfitting and it can effectively get you there this is a lot of evidence for
 the fact that this biasing the networks towards the identity function is a very
 valid thing to do and is the solution to the we can't train deep networks
 problems lastly they investigate the size of the responses so their hypothesis
 is that if if it is really beneficial to bias the network towards the identity
 function and if it is really true that each of these layers only learns a
 little bit right because the identity function is already very good each of
 these layers only needs to learn kind of a small function they look at the
 responses of these things so the the response magnitude of these layers right
 here of the signal through the layers and they compare those with the response
 magnitude of the other neural networks where you don't have the skip
 connection the hypothesis is if we look at these then the responses of these
 layers should be much larger because they have to learn much more and the
 responses here will be much smaller because the identity function is already
 doing most of the work and that's exactly what you find so here the layers
 are ordered by response and you can see the plane networks in the dashed lines
 are significantly above the residual network even and that's not a function
 of the depth because if the depth was actually equal here you would expect
 that the dashed lines would would stretch like this right they would kind
 of stretch out however exactly the opposite is happening you can see that
 the residual networks even at the beginning their responses are very much
 smaller and this is kind of what I like about this paper it's it's one narrative
 it is a hypothesis and then every single like the the hypothesis is taken and
 they make predictions from the hypothesis they say okay if we are right
 with our hypothesis not only should our idea get us better accuracy that's what
 most people most papers do today but also you know but also it should be that
 we can for example push our network to the brink of where we actually are
 overfitting like here and it should also be that the responses of our signal
 through our layers is smaller and yeah that's research like this is just
 pretty pretty cool and it's I think a lesson for us that sadly the world has
 taken the resonance but the world hasn't all taken the research methodology of
 this paper yeah if you again if you want a good read it's very well written you
 I'm very sure you can follow it even if you have read very few papers and with
 that yeah I hope you enjoyed this please tell me what you think of going through
 kind of old papers looking at whether or not they have stood the test of time and
 yeah any other comments leave them in the comments I do read them and I'll
 see you next time bye bye