SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper vision recognition localization resnet resnet50 fpn backbone permuation upsampling stride convolution convolutional neural network google spine spine net imagenet coco segmentation bounding box skip connections residual bottleneck
#machinelearning #ai #google

The high-level architecture of CNNs has not really changed over the years. We tend to build high-resolution low-dimensional layers first, followed by ever more coarse, but deep layers. This paper challenges this decades-old heuristic and uses neural architecture search to find an alternative, called SpineNet that employs multiple rounds of re-scaling and long-range skip connections.

OUTLINE:
0:00 - Intro & Overview
1:00 - Problem Statement
2:30 - The Problem with Current Architectures
8:20 - Scale-Permuted Networks
11:40 - Neural Architecture Search
14:00 - Up- and Downsampling
19:10 - From ResNet to SpineNet
24:20 - Ablations
27:00 - My Idea: Attention Routing for CNNs
29:55 - More Experiments
34:45 - Conclusion & Comments

Papers: https://arxiv.org/abs/1912.05027
Code: https://github.com/tensorflow/tpu/tree/master/models/official/detection

Abstract:
Convolutional neural networks typically encode an input image into a series of intermediate features with decreasing resolutions. While this structure is suited to classification tasks, it does not perform well for tasks requiring simultaneous recognition and localization (e.g., object detection). The encoder-decoder architectures are proposed to resolve this by applying a decoder network onto a backbone model designed for classification tasks. In this paper, we argue encoder-decoder architecture is ineffective in generating strong multi-scale features because of the scale-decreased backbone. We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search. Using similar building blocks, SpineNet models outperform ResNet-FPN models by ~3% AP at various scales while using 10-20% fewer FLOPs. In particular, SpineNet-190 achieves 52.5% AP with a MaskR-CNN detector and achieves 52.1% AP with a RetinaNet detector on COCO for a single model without test-time augmentation, significantly outperforms prior art of detectors. SpineNet can transfer to classification tasks, achieving 5% top-1 accuracy improvement on a challenging iNaturalist fine-grained dataset. Code is at: this https URL.

Authors: Xianzhi Du, Tsung-Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V. Le, Xiaodan Song

Thumbnail art by Lucas Ferreira

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we'll look at SpineNet Learning Scale-Permuted Backbone for Recognition and Localization by Xianze Du at Aal of Google Research. On a high level this paper proposes to take current recognition and localization networks which have a CNN backbone, usually something like a ResNet, and switch up the order of the blocks in the ResNet and cross-connect them in a different way, such that they reach a higher accuracy with the new network that has the same amount of parameters or almost the same amount of parameters. They then further modify this network such that it reaches that higher accuracy with less compute than the original network. So if you want to know how it's done, you know, stick around. You can help me by sharing out this video if you liked it, if you didn't like it, leave a comment and tell me what you didn't like, otherwise I have no chance of improving. So that's the deal, okay? Cool. So the task here is a recognition and localization as you can see here, which basically means that you have an image and there's stuff on the image. Maybe there's a cat right here and maybe there is some kind of a house right here. And the tasks, these tasks come in various forms, but some of the tasks are to say what's on the image, so in this case cat and house, and also where is it? Now this could be a point, this could be a bounding box, or this could actually be a pixel segmentation. All of this sort of tasks exists in various forms. What usually is done in these tasks is you want to go in some way through a neural network and the neural network will output the same image again or the same shape. So it will output an image that is of the same shape, that if this is your input image, I'm just gonna quickly redraw without the labels, if this is your input image then the output image, let's say we're doing bounding boxes, the output would say something like here are bounding boxes and also the output would be cat and house. So these are the two outputs that the neural network would generate. This is some sort of a convolutional neural network because they deal with images fairly well. Now usually when we do image processing and we know this from for example image classification, so if we just have image classification just to classify here, if we just want the outputs cat or house or even just one single thing like an image net, our convolutional neural networks have a particular architecture. Namely what we do is we have the first convolutional neural net, the first layers will take the image and run these convolutional filters across them which gives you the same shape image back but then with time we scale it down. We have a max pooling or a convolution with stride 2 so that the image is only this big anymore and then we have a bunch of further layers and then we scale it down again and so on. Now as we scale it down the number of channels goes up. Of course at the beginning you have three channels for the three colors but then after the first convolution you might have whatever 32 channels right here. This is no longer the original image, this now is of course for each pixel you have a stack of features right you have a stack of features right here because that's what your convolutional layer does and then when you scale it down you have even more feature maps so we tend to scale down the resolution of our feature maps but we tend to increase the number of feature maps right here. The reasoning behind this is if you look at these bounding boxes they don't really... sorry if you look at the labels right here the fact that there's a cat on the image shouldn't depend on the exact pixel location of the cat right so even if I scale this down a bit I'll still recognize that there's a cat somewhere it can still aggregate that information in fact I could deal with scaling this down successively up to a single pixel and that's ultimately what an image classifier does you simply have a single vector at the end with the features in it and from that you classify. So the reasoning is that as you go through the network you pick up the low level features first like here you pick up the edges and the kind of low level shapes as you go higher through the network your features become more abstract but less localized which means that it's less important where they are and that's why you look at this image through a coarser and coarser segmentation and at the end your segmentation might be something like this. Okay so we have had a lot of success building image classifiers with this reasoning and this is sort of a human heuristic that just has worked well. Now when we do something like this bounding box classification or even per pixel classification all of a sudden it is very important where the things are right it is very important that it's this pixel and this pixel and this pixel and this pixel forming the bounding box because the more accurate you are the better your bounding box classifier you still have this right here this recognition but the localization part we can't just scale down anymore because we need to output something that's of the same size so what people have done is they've gone from this kind of from architecture that scales down because we know that works well we know the downscaling works well so we take that and then we scale up again and there is some reasoning behind this right so that's what we can do because we know this part works very well for extracting high-level features that are not that localized so our reasoning is going to be something like okay we'll force the network through this kind of bottleneck right here we'll force it to learn some high-level features we because otherwise you can just you know kind of remember the individual pixels and that won't work as well we'll force it to remember the high-level we'll force it to remember what a cat is and then it will help in the pixel segmentation to know what a cat is this is very valid assumption but it doesn't need to be the case and so there is one additional thing that these networks usually do is that they have like some skip connections here from the layers that are of the same size to the layers that are of the same size right here to here in order to kind of recover these high-level features because if you only look at an image through the lens of this right here and you're a you have to segment the ear of the cat you know you can only either color an entire pixel or not so you want to gain back some of that some of those high-level features and that's what you do with skip connections and that's why these networks usually look like this now in this work the authors sort of criticize this they say why why are we doing it like this isn't there a better way to do it specifically we want to look at this part right here which is called the backbone so we assume that we have these these output layers that give you at different scales different features and what we have to do is we have to construct a backbone that somehow feeds features either you know through this direct way or through these connections right here and feeds features to this these ultimate classifiers so these classifiers will then be used to classify the bounding boxes and classify the output classes for recognition and localization this is an illustration of this on the left you see a typical backbone and they call this a scale decreased network so an example of scale decreased network on the left versus a scale permuted network on the right the width of the block indicates feature resolution and the height indicates feature dimension dotted arrows represent connections from two blocks not plotted okay so on the left you have the typical architecture you see that the the width so this is the resolution is very high and as you go through the layers that resolution gets smaller and smaller and smaller but the number of features indicated by the height gets higher and higher and higher as you go through this is your typical architecture we are looking into that say that this is not the only one what we can do is we can build any sort of backbone and here they restrict themselves they say okay in order to make it comparable in order to be you know scientifically a bit more rigorous than just building anything what we restrict ourselves are simply permutations of this so we only allow us to permute these things so all the you know this goes here and this goes here as you see and that ensures that you still have roughly the same amount of parameters now there is it sort of a parameter difference because these connections here you need to up and down sample the images and sometimes that introduces parameters but in essence you have the rough same amount of parameters and then you can really research what can we improve a network simply by rearranging its blocks because that would give evidence that this scaling down architecture isn't really the best one okay so here you can see an example of this this is what they call a scale permuted network sorry this scale permuted network right here so in a scale permuted network what you're allowed to do is you're allowed you have these blocks on the left and you're allowed to put them anywhere you want in in any in any sort of I don't want to say order but yes in any sort of order yes it's an order actually so it goes from here down if this is one two three four five any block for any block you first place you first place this block you're allowed to connect it to any other block before it now here we don't see but you can see there's two incoming connections right here so we make use of more than one connection on the left you see there's always one connection between the blocks and on the right you see we allow up to two blocks to connect to a given block okay then you're done with this block you place the next block this one here also you're allowed to have two incoming connections this one here and this one here and you place the next block and so on now how you make this you also see that there doesn't need to be like a straight linear path because there is no connection right here if you can see that so you might be wondering how do I decide which block goes where and how do I decide on which connections connect where and that is going to be the idea here to use neural architecture search so neural architecture search right now is still a fancy way of saying let's try stuff out and so what you'll do is you will initialize a reinforcement learning controller that decides on the ordering and on the connections and it has some action space and you basically let it run so it it you know proposes a couple of architectures and and then you measure all of them you train all of these architectures and you see how well they fare and then you go back to the controller that's the reward signal and so we can draw so you have an agent which builds the building plan so the agent in the agent will emit as an action a building plan like big small big small big with connections like this and like this and like this and then that will go to the environment the environment here simply takes the architecture and trains train the architecture and then the let's say the eval loss or the validation loss the validation accuracy is equal is going to be your reward signal so you simply train a reinforcement learning agent to solve this particular problem which is training this in recognition and localization on the particular data set as well as possible to basically come up with the best architecture which you know it's it's fancy and it's a bit better than trying everything out but it's not much better right now and it takes a lot of compute to run these experiments because it takes a lot of iterations of this and every iteration consists of training one of these networks fully once now you can do something with like early stopping and stuff but so you get the idea this is what they what they propose and this is you know how they get better so there are a number of challenges in this namely we we said okay when you input a signal for example when you input a signal from this layer to this layer you can see that you have to shrink the resolution and you have to up the number of features and this was already sort of solved in the resonant original resonant paper but they reiterate how they do this here basically you have we have this layer and it is connected to these two layers we said every layer can receive inputs from two layers you see at the very end these are just added together okay so we have two things first of all the number of features is different you can see right over here the number of features the number of channels is different than the number of channels in the output image let's say right here so those are different and in fact they're different in both inputs and we have this method of one by one convolutions that was introduced in the original resonant paper if we do one by one convolutions it's basically a a learned transformation from a number of input channels to a number of output channels without change without doing any actual convolution operation this is simply linear operation up scaling or up up upping the number of feature maps you can see these one by one convolutions are employed here in various ways so because this is fairly compute intensive or so they claim what they do first is they always first go to less features so here we have a number of features which is maybe let's say this is f or sorry this is c0 you can see very small here maybe that there is this first we go to alpha times c0 and alpha is I think in the default setting it's one half so first we always go to one half the number of features before we do this switch here and then we have two options either so you go to one half the features and at the end you go to the number of target features so it could be if the target features are more than you currently have it could be that you first go to less features and then you go to even more features right as if you the the current one has more features than the end it's probably not as bad because you first go to less features and to even less features this is probably one of the things they did to save computation but which you can imagine that it hurts because here you simply have to you have to basically throw away half the features or you have to like linearly combine them in every step where you connect two layers to each other you know okay so there's two situations first situation your current resolution here is higher than the target resolution in that case we can simply do a convolution with a bigger stride than one right if you have an image and you do a convolution usually you have this overlapping convolution such that the result is the same size as you started with but you can also do a bigger stride and I'm a bit over drawing this here but you can do a bigger stride such that the final resolution is smaller and you can also do this max pooling right here so the max pooling is also a way to reduce the number of of the resolution of the image so if we're bigger we can do that if we are currently smaller than the the target what we can do is we can up sample and up sample you can do by doing nearest neighbor or things like this you can also do a learned up sample there are various ways I believe here they do a nearest neighbor but I'm not sure anymore actually let's check it out that's here somewhere resampling in cross-scale connections yada yada yada yada yada yada yada it's important to keep the computational resampling low we introduce a scaling factor alpha we had that then we use a nearest neighbor interpolation for up sampling or a stride to three by three convolution okay so it's nearest neighbor by up sampling alright so that's that's how they up and down sample the feature maps to the correct shapes either using nearest neighbor up sampling or using multi stride convolution followed by max pooling so what does that give them now they do several different steps in this so the first architecture this resnet 50 is the original architecture and remember we're only talking about the backbone right here now in the original resnet 50 architecture you have this resnet 50 fpn and this fpn is these are called the output layers this is what then goes and classifies the bounding boxes and the labels and so on now here you can see the resnet 50 is continuously getting smaller and more features they do an intermediate step so this this right here is their final thing where they let this let this algorithm go wild and you can see that it's pretty pretty fuzzy so this RL controller finds this architecture to be the best architecture and you can see it's continuously down and up and down and sorry and up and down and there is considerable cross connections between all of these things and then here you have the you have the different output layers built into the network rather than next to the network right so these are the ones that are now the red border ones are now the features that are used for going in classifying as an intermediate step they also consider this architecture where they basically built a smaller resnet right here and then let the algorithm decide on the rest right here so it still has the same amount of parameters roughly but they can investigate what happens if we go to these to this lower if we have this structure at the beginning but then part of it we can do with our algorithm and lastly they also consider this architecture now this architecture again their algorithm has control over the whole network but there is an additional thing that the algorithm can do the algorithm can also decide to change the number of features and to change the type of block so here you can see these are all residual blocks and these are these called bottleneck blocks they're simply a different way of of doing a residual block it was introduced in the original resnet paper but the the controller can simply switch to that and that can save some computation if you go through these bottleneck blocks so what does that give you you can see below that the resnet 50 is at 37.8 percent average precision if you liberate the top part to leave it to the algorithm it's at 39 if you liberate the entire network it's at 40.7 and remember these are like roughly the same amount of parameters and then if you if you also let the network control a bit of the feature size and the type of block you get a 40.8 which is the same as before but now this one I believe has about oh yeah here we go with 10% fewer flops okay so that's that's pretty cool though remember that the left thing this is this is made by humans this is just our heuristic and the right things they are made by RL and they are you know for these particular data sets though they do find that generally this also transfers to image net classification but still this is sort of a it works well for the type of data we work with and so on so I don't know how much I would trust it how far we should go of building spine net 49 as our new backbone for every image task that we have it remains to be seen I believe before actually we go to the experiments before we go to the experiments I want to state my idea right here so you get the general gist here and so another kind of coral I have with this is that you know in here you always have these single connections and here you always have these these double connections and I've looked through the experiment it seems like nowhere do they ablate or anything what what it means to only have single connections or if they so if they let the resnet run with double connections so if their controller could not switch the order but only introduce the connections they might have done this they have a lot of experiments where they do the different ablations so I would be interested what happens when you let it run on the resnet but let it have two connections per per layer is it then better or not so here the importance I'll get to my idea later the importance of scale permutation that's where they investigate how important is it that you permute the layers and that turns out to be fairly important then the importance of cross scale connections that's how they investigate here so these are these connections they say the cross scale connections play a crucial role in fusing features at different resolutions throughout the scale permuted network so that's the reasoning behind it we we take features from different kind of resolutions and we can also scale up again and then scale down again to gain some additional features from the from the higher resolutions we study it's important by graph damage so either they remove the short-term connections or they remove the long-range connections or they remove both and then connect one block to the previous block via a sequential connection so this is only this is only in the things that they learned right so this model is where they fully give their model control over the ordering and connections you can see that as this forty point seven percent now if they delete the short-range connections they drop to thirty five if they delete the only the long-range they drop to even more so here you can see that these long-range connections which I guess are connections that are going across multiple blocks skipping multiple blocks these tend to be very important so you can make the case that it might be very important to fuse these things from different layers to fuse the features from different resolutions because these long-range connections tend to be important though it's one thing to say that if we just leave them away with our model if we just damage it and then let it run it it it drops in accuracy it's not entirely the same thing as to say that these are important because you don't really know what happens like if you train without them maybe you could if you train without them you could reach as good an accuracy so this graph damage investigation it has something but not I wouldn't trust it too much and yeah I think they haven't investigated what happens if they keep the resinette order but let the connections be twice but you get the general the general idea of the paper right here of of what they do now they do this with architecture search right here but here's an idea okay I propose the following you have an image right here and we are wondering here should we let it go through a layer that's wide and with less features should we let it go through a layer that's you know very many features but not as wide but we have to downscale the image or should we let it go first through something intermediate let's see it like this okay so we're wondering how should we order these blocks why can't we do all of them at the same time why can't we do this this and this okay and then in the next layer again do all of them at the same time and you you you can already see where this is going I hope I hope you can see where this is going so you have a routing right here and how do we do routing in modern times in deep learning with attention so I propose you have layers with different attention hey let's say these are these are now your your sequences or you can also make them as attention heads okay these are you these and the lower level features are routed to the higher level features with an attention mechanism and you do this layer by layer by layer so you let because what's the problem here the problem here is that the same data point has to go you know you find these good connections but the all the data points have to go through the same connections and it might actually be that you need different routing depending on the data point it might be that what this is this is good for the average data point but it would be much better if whenever there's a cat you take one path and whenever there's a dog you take a different path so this will allow for that you basically have the routing parameterized by an attention mechanism this I have no clue how much compute this would take it doesn't seem that outrageous because what's your sequence length here your sequence length is going to be the number of layers maybe and maybe times the number of feature maps maybe have different attention head so you maybe want to replicate some of those here but ultimately I would guess the attention mechanism itself isn't that much of an overhead maybe it's an overhead that you have so many in parallel yeah but you know it remains to be seen that's that's the idea yeah you heard it here first okay so they have more experiments so they also build here is where they say okay we have the spine net 49 now and we found this to work we found this to work really well this is our spine net 49 architecture cool and we want to make it bigger but I guess they didn't have the computational resources to run the neural architecture search for bigger networks this is now as about as big as a resonant 50 right but what if you wanted to go to a resonant 100 or a resonant 150 there you you don't have the computational resources do neural architectures imagine this Google has doesn't have the computational resources to do neural architecture search on this thing so this must be expensive or I'm just I have no idea but what they do is they kind of do a trick so here they take the the spine net 49 and they say we build a spine net 60 96 by simply repeating each block twice so all the incoming connections would go to the first block and all the outgoing connections would come from the second block right here you had two in and maybe there's actually no limit to how many outgoing connections you can have and also you can also do this three times which I think is a bit of a cheap way and it kind of defeats the entire purpose right couldn't you make the exact same argument again here that maybe it's helpful to route from this block right here or maybe it's helpful that these don't have the same scale right after one another it just seems but okay so they say we found this good structure and we simply duplicate each block I'm not that big of a fan in any case so they train this and it of course outperforms everything else if you compare with kind of models of the same size so here you compare this spine net 49 to the resnet 50 and you can see there's about the same number of parameters how about it outperforms the resnet 50 pretty much and as you go up the number of parameters here the performance goes up yet again and I believe these dagger ones here are simply trained with a special schedule with here with applying stochastic depth and swish activation for a longer training schedule so you can see that not only do are these spine nets sorry of the number of parameters is here not only are the spine nets slightly smaller than the resnets they do require less flops and they reach better accuracy so you know every everything is a win here yeah so they apply this to these data sets I don't want to go you know too much into into that but in the last part they also apply this to image net so there's image classification where they basically say okay we can just go to our architecture and we can just add up all the output blocks we scale them appropriately and add up all the output blocks right here because these are good features for localization and so on and we can train it to do image classification so all of these go into a big combination classifier that does the 1000 classes of image net image classification and that also works pretty well with this network so they basically argue what they found is sort of a better image image processing network than the resnet 50 and I guess they would argue that from now on you should take this as your backbone for image classification and recognition and so on which it's entirely possible that this works better there's no particular reason why the resnet 50 should work at all right it's just a heuristic but I guess the I it remains to be to be seen whether that's generally true or just in the things they considered so you can see right here the spine net generally improving over the image net which isn't is not stated right here but it does generally improve and you can see as you go higher and higher spine net the the numbers tend to improve as well and this is already pretty respectable respectable number for image net right all right so this was it for this paper for this particular paper they do have you know two different of these object detection recognition datasets and I invite you to check out the experiments more closely if you're interested in that sort of thing I was mainly interested in the method of doing and arranging these layers and so on it seems like it's a cool engineering project cool investigative project the experiments are done well and in the end they reach a better you know they achieve to get a better model out of that and if it turns out that this model is a good model the entire community will be better off unfortunately there's no broader impact statement to tell us that also the terrorists will be able to use this for purposes but you can imagine that yourself all right that was it for me again leave a comment if you want me to change anything or have suggestions leave a like if you like the video share it out bye bye
 Hi there, today we'll look at SpineNet Learning Scale-Permuted Backbone for
 Recognition and Localization by Xianze Du at Aal of Google Research. On a high
 level this paper proposes to take current recognition and localization
 networks which have a CNN backbone, usually something like a ResNet, and
 switch up the order of the blocks in the ResNet and cross-connect them in a
 different way, such that they reach a higher accuracy with the new network
 that has the same amount of parameters or almost the same amount of parameters.
 They then further modify this network such that it reaches that higher
 accuracy with less compute than the original network. So if you want to know
 how it's done, you know, stick around. You can help me by sharing out this video if
 you liked it, if you didn't like it, leave a comment and tell me what you didn't
 like, otherwise I have no chance of improving. So that's the deal, okay? Cool.
 So the task here is a recognition and localization as you can see here, which
 basically means that you have an image and there's stuff on the image. Maybe
 there's a cat right here and maybe there is some kind of a house right here. And
 the tasks, these tasks come in various forms, but some of the tasks are to say
 what's on the image, so in this case cat and house, and also where is it? Now this
 could be a point, this could be a bounding box, or this could actually be a
 pixel segmentation. All of this sort of tasks exists in various forms. What
 usually is done in these tasks is you want to go in some way through a neural
 network and the neural network will output the same image again or the same
 shape. So it will output an image that is of the same shape, that if this is your
 input image, I'm just gonna quickly redraw without the labels, if this is
 your input image then the output image, let's say we're doing bounding boxes, the
 output would say something like here are bounding boxes and also the output would
 be cat and house. So these are the two outputs that the neural network would
 generate. This is some sort of a convolutional neural network because
 they deal with images fairly well. Now usually when we do image processing and
 we know this from for example image classification, so if we just have image
 classification just to classify here, if we just want the outputs cat or house or
 even just one single thing like an image net, our convolutional neural networks
 have a particular architecture. Namely what we do is we have the first
 convolutional neural net, the first layers will take the image and run these
 convolutional filters across them which gives you the same shape image back but
 then with time we scale it down. We have a max pooling or a convolution with
 stride 2 so that the image is only this big anymore and then we have a bunch of
 further layers and then we scale it down again and so on. Now as we scale it down
 the number of channels goes up. Of course at the beginning you have three channels
 for the three colors but then after the first convolution you might have
 whatever 32 channels right here. This is no longer the original image, this now is
 of course for each pixel you have a stack of features right you have a stack
 of features right here because that's what your convolutional layer does and
 then when you scale it down you have even more feature maps so we tend to
 scale down the resolution of our feature maps but we tend to increase the number
 of feature maps right here. The reasoning behind this is if you look at these
 bounding boxes they don't really... sorry if you look at the labels
 right here the fact that there's a cat on the image shouldn't depend on the exact
 pixel location of the cat right so even if I scale this down a bit I'll still
 recognize that there's a cat somewhere it can still aggregate that information
 in fact I could deal with scaling this down
 successively up to a single pixel and that's ultimately what an image classifier
 does you simply have a single vector at the end with the features in it and from
 that you classify. So the reasoning is that as you go through the network you
 pick up the low level features first like here you pick up the edges and the
 kind of low level shapes as you go higher through the network your features
 become more abstract but less localized which means that it's less important
 where they are and that's why you look at this image through a coarser and
 coarser segmentation and at the end your segmentation might be something like
 this. Okay so we have had a lot of success building image classifiers with
 this reasoning and this is sort of a human heuristic that just has worked
 well. Now when we do something like this bounding box classification or even per
 pixel classification all of a sudden it is very important where the things are
 right it is very important that it's this pixel and this pixel and this pixel
 and this pixel forming the bounding box because the more accurate you are the
 better your bounding box classifier you still have this right here this
 recognition but the localization part we can't just scale down anymore because
 we need to output something that's of the same size so what people have done
 is they've gone from this kind of from architecture that scales down because we
 know that works well we know the downscaling works well so we take that
 and then we scale up again and there is some reasoning behind this right so
 that's what we can do because we know this part works very well for
 extracting high-level features that are not that localized so our reasoning is
 going to be something like okay we'll force the network through this kind of
 bottleneck right here we'll force it to learn some high-level features we because
 otherwise you can just you know kind of remember the individual pixels and that
 won't work as well we'll force it to remember the high-level we'll force it
 to remember what a cat is and then it will help in the pixel segmentation to
 know what a cat is this is very valid assumption but it doesn't need to be the
 case and so there is one additional thing that these networks usually do is
 that they have like some skip connections here from the layers that
 are of the same size to the layers that are of the same size right here to here
 in order to kind of recover these high-level features because if you only
 look at an image through the lens of this right here and you're a you have to
 segment the ear of the cat you know you can only either color an entire pixel or
 not so you want to gain back some of that some of those high-level features
 and that's what you do with skip connections and that's why these
 networks usually look like this now in this work the authors sort of criticize
 this they say why why are we doing it like this isn't there a better way to do
 it specifically we want to look at this part right here which is called the
 backbone so we assume that we have these these output layers that give you at
 different scales different features and what we have to do is we have to
 construct a backbone that somehow feeds features either you know through this
 direct way or through these connections right here and feeds features to this
 these ultimate classifiers so these classifiers will then be used to
 classify the bounding boxes and classify the output classes for
 recognition and localization this is an illustration of this on the left you see
 a typical backbone and they call this a scale decreased network so an example of
 scale decreased network on the left versus a scale permuted network on the
 right the width of the block indicates feature resolution and the height
 indicates feature dimension dotted arrows represent connections from two
 blocks not plotted okay so on the left you have the typical architecture you
 see that the the width so this is the resolution is very high and as you go
 through the layers that resolution gets smaller and smaller and smaller but the
 number of features indicated by the height gets higher and higher and higher
 as you go through this is your typical architecture we are looking into that
 say that this is not the only one what we can do is we can build any sort of
 backbone and here they restrict themselves they say okay in order to
 make it comparable in order to be you know scientifically a bit more rigorous
 than just building anything what we restrict ourselves are simply permutations
 of this so we only allow us to permute these things so all the you know this
 goes here and this goes here as you see and that ensures that you still have
 roughly the same amount of parameters now there is it sort of a parameter
 difference because these connections here you need to up and down sample the
 images and sometimes that introduces parameters but in essence you have the
 rough same amount of parameters and then you can really research what can we
 improve a network simply by rearranging its blocks because that would give
 evidence that this scaling down architecture isn't really the best one
 okay so here you can see an example of this this is what they call a scale
 permuted network sorry this scale permuted network right here so in a
 scale permuted network what you're allowed to do is you're allowed you have
 these blocks on the left and you're allowed to put them anywhere you want in
 in any in any sort of I don't want to say order but yes in any sort of order
 yes it's an order actually so it goes from here down if this is one two three
 four five any block for any block you first place you first place this block
 you're allowed to connect it to any other block before it now here we don't
 see but you can see there's two incoming connections right here so we make use of
 more than one connection on the left you see there's always one connection
 between the blocks and on the right you see we allow up to two blocks to connect
 to a given block okay then you're done with this block you place the next block
 this one here also you're allowed to have two incoming connections this one
 here and this one here and you place the next block and so on now how you make
 this you also see that there doesn't need to be like a straight linear path
 because there is no connection right here if you can see that so you might
 be wondering how do I decide which block goes where and how do I decide on which
 connections connect where and that is going to be the idea here to use neural
 architecture search so neural architecture search right now is still a
 fancy way of saying let's try stuff out and so what you'll do is you will
 initialize a reinforcement learning controller that decides on the ordering
 and on the connections and it has some action space and you basically let it
 run so it it you know proposes a couple of architectures and and then you measure
 all of them you train all of these architectures and you see how well they
 fare and then you go back to the controller that's the reward signal and
 so we can draw so you have an agent which builds the building plan so the
 agent in the agent will emit as an action a building plan like big small
 big small big with connections like this and like this and like this and then
 that will go to the environment the environment here simply takes the
 architecture and trains train the architecture and then the let's say the
 eval loss or the validation loss the validation accuracy is equal is going to
 be your reward signal so you simply train a reinforcement learning agent to
 solve this particular problem which is training this in recognition and
 localization on the particular data set as well as possible to basically come up
 with the best architecture which you know it's it's fancy and it's a bit
 better than trying everything out but it's not much better right now and it
 takes a lot of compute to run these experiments because it takes a lot of
 iterations of this and every iteration consists of training one of these
 networks fully once now you can do something with like early stopping and
 stuff but so you get the idea this is what they what they propose and this is
 you know how they get better so there are a number of challenges in this
 namely we we said okay when you input a signal for example when you input a
 signal from this layer to this layer you can see that you have to shrink the
 resolution and you have to up the number of features and this was already sort of
 solved in the resonant original resonant paper but they reiterate how
 they do this here basically you have we have this layer and it is connected to
 these two layers we said every layer can receive inputs from two layers you see
 at the very end these are just added together okay so we have two things first
 of all the number of features is different you can see right over here
 the number of features the number of channels is different than the number of
 channels in the output image let's say right here so those are different and in
 fact they're different in both inputs and we have this method of one by one
 convolutions that was introduced in the original resonant paper if we do one by
 one convolutions it's basically a a learned transformation from a number of
 input channels to a number of output channels without change without doing
 any actual convolution operation this is simply linear operation up scaling or up
 up upping the number of feature maps you can see these one by one convolutions
 are employed here in various ways so because this is fairly compute intensive
 or so they claim what they do first is they always first go to less features so
 here we have a number of features which is maybe let's say this is f or sorry
 this is c0 you can see very small here maybe that there is this first we go to
 alpha times c0 and alpha is I think in the default setting it's one half so
 first we always go to one half the number of features before we do this
 switch here and then we have two options either so you go to one half the
 features and at the end you go to the number of target features so it could be
 if the target features are more than you currently have it could be that you
 first go to less features and then you go to even more features right as if you
 the the current one has more features than the end it's probably not as bad
 because you first go to less features and to even less features this is
 probably one of the things they did to save computation but which you can
 imagine that it hurts because here you simply have to you have to basically
 throw away half the features or you have to like linearly combine them in every
 step where you connect two layers to each other you know okay so there's two
 situations first situation your current resolution here is higher than the
 target resolution in that case we can simply do a convolution with a bigger
 stride than one right if you have an image and you do a convolution usually
 you have this overlapping convolution such that the result is the same size as
 you started with but you can also do a bigger stride and I'm a bit over drawing
 this here but you can do a bigger stride such that the final resolution is
 smaller and you can also do this max pooling right here so the max pooling is
 also a way to reduce the number of of the resolution of the image so if we're
 bigger we can do that if we are currently smaller than the the target
 what we can do is we can up sample and up sample you can do by doing nearest
 neighbor or things like this you can also do a learned up sample there are
 various ways I believe here they do a nearest neighbor but I'm not sure anymore
 actually let's check it out that's here somewhere resampling in cross-scale
 connections yada yada yada yada yada yada yada it's important to keep the
 computational resampling low we introduce a scaling factor alpha we had
 that then we use a nearest neighbor interpolation for up sampling or a
 stride to three by three convolution okay so it's nearest neighbor by up
 sampling alright so that's that's how they up and down sample the feature maps
 to the correct shapes either using nearest neighbor up sampling or using
 multi stride convolution followed by max pooling so what does that give them now
 they do several different steps in this so the first architecture this resnet
 50 is the original architecture and remember we're only talking about the
 backbone right here now in the original resnet 50 architecture you have this
 resnet 50 fpn and this fpn is these are called the output layers this is what
 then goes and classifies the bounding boxes and the labels and so on now here
 you can see the resnet 50 is continuously getting smaller and more
 features they do an intermediate step so this this right here is their final
 thing where they let this let this algorithm go wild and you can see that
 it's pretty pretty fuzzy so this RL controller finds this architecture to be
 the best architecture and you can see it's continuously down and up and down
 and sorry and up and down and there is considerable cross connections between
 all of these things and then here you have the you have the different output
 layers built into the network rather than next to the network right so these
 are the ones that are now the red border ones are now the features that are used
 for going in classifying as an intermediate step they also consider
 this architecture where they basically built a smaller resnet right here and
 then let the algorithm decide on the rest right here so it still has the same
 amount of parameters roughly but they can investigate what happens if we go to
 these to this lower if we have this structure at the beginning but then part
 of it we can do with our algorithm and lastly they also consider this
 architecture now this architecture again their algorithm has control over the
 whole network but there is an additional thing that the algorithm can do the
 algorithm can also decide to change the number of features and to change the
 type of block so here you can see these are all residual blocks and these are
 these called bottleneck blocks they're simply a different way of of doing a
 residual block it was introduced in the original resnet paper but the the
 controller can simply switch to that and that can save some computation if you go
 through these bottleneck blocks so what does that give you you can see below
 that the resnet 50 is at 37.8 percent average precision if you liberate the
 top part to leave it to the algorithm it's at 39 if you liberate the entire
 network it's at 40.7 and remember these are like roughly the same amount of
 parameters and then if you if you also let the network control a bit of the
 feature size and the type of block you get a 40.8 which is the same as before
 but now this one I believe has about oh yeah here we go with 10% fewer flops
 okay so that's that's pretty cool though remember that the left thing this is
 this is made by humans this is just our heuristic and the right things they are
 made by RL and they are you know for these particular data sets though they
 do find that generally this also transfers to image net classification but
 still this is sort of a it works well for the type of data we work with and so
 on so I don't know how much I would trust it how far we should go of building
 spine net 49 as our new backbone for every image task that we have it
 remains to be seen I believe before actually we go to the experiments before
 we go to the experiments I want to state my idea right here so you get the
 general gist here and so another kind of coral I have with this is that you know
 in here you always have these single connections and here you always have
 these these double connections and I've looked through the experiment it seems
 like nowhere do they ablate or anything what what it means to only have single
 connections or if they so if they let the resnet run with double connections
 so if their controller could not switch the order but only introduce the
 connections they might have done this they have a lot of experiments where
 they do the different ablations so I would be interested what happens when
 you let it run on the resnet but let it have two connections per per layer is it
 then better or not so here the importance I'll get to my idea later
 the importance of scale permutation that's where they investigate how
 important is it that you permute the layers and that turns out to be fairly
 important then the importance of cross scale connections that's how they
 investigate here so these are these connections they say the cross scale
 connections play a crucial role in fusing features at different resolutions
 throughout the scale permuted network so that's the reasoning behind it we we
 take features from different kind of resolutions and we can also scale up
 again and then scale down again to gain some additional features from the from
 the higher resolutions we study it's important by graph damage so either they
 remove the short-term connections or they remove the long-range connections or
 they remove both and then connect one block to the previous block via a
 sequential connection so this is only this is only in the things that they
 learned right so this model is where they fully give their model control over
 the ordering and connections you can see that as this forty point seven percent
 now if they delete the short-range connections they drop to thirty five if
 they delete the only the long-range they drop to even more so here you can see
 that these long-range connections which I guess are connections that are going
 across multiple blocks skipping multiple blocks these tend to be very important
 so you can make the case that it might be very important to fuse these things
 from different layers to fuse the features from different resolutions
 because these long-range connections tend to be important though it's one
 thing to say that if we just leave them away with our model if we just damage it
 and then let it run it it it drops in accuracy it's not entirely the same
 thing as to say that these are important because you don't really know what
 happens like if you train without them maybe you could if you train without
 them you could reach as good an accuracy so this graph damage investigation it
 has something but not I wouldn't trust it too much and yeah I think they haven't
 investigated what happens if they keep the resinette order but let the
 connections be twice but you get the general the general idea of the paper
 right here of of what they do now they do this with architecture search right
 here but here's an idea okay I propose the following you have an image right
 here and we are wondering here should we let it go through a layer that's wide
 and with less features should we let it go through a layer that's you know very
 many features but not as wide but we have to downscale the image or should we
 let it go first through something intermediate let's see it like this okay
 so we're wondering how should we order these blocks why can't we do all of
 them at the same time why can't we do this this and this okay and then in the
 next layer again do all of them at the same time and you you you can already
 see where this is going I hope I hope you can see where this is going so you
 have a routing right here and how do we do routing in modern times in deep
 learning with attention so I propose you have layers with different attention
 hey let's say these are these are now your your sequences or you can also make
 them as attention heads okay these are you these and the lower level features
 are routed to the higher level features with an attention mechanism and you do
 this layer by layer by layer so you let because what's the problem here the
 problem here is that the same data point has to go you know you find these good
 connections but the all the data points have to go through the same connections
 and it might actually be that you need different routing depending on the data
 point it might be that what this is this is good for the average data point but
 it would be much better if whenever there's a cat you take one path and
 whenever there's a dog you take a different path so this will allow for
 that you basically have the routing parameterized by an attention mechanism
 this I have no clue how much compute this would take it doesn't seem that
 outrageous because what's your sequence length here your sequence length is
 going to be the number of layers maybe and maybe times the number of feature
 maps maybe have different attention head so you maybe want to replicate some of
 those here but ultimately I would guess the attention mechanism itself isn't
 that much of an overhead maybe it's an overhead that you have so many in
 parallel yeah but you know it remains to be seen that's that's the idea yeah you
 heard it here first okay so they have more experiments so they also build
 here is where they say okay we have the spine net 49 now and we found this to
 work we found this to work really well this is our spine net 49 architecture
 cool and we want to make it bigger but I guess they didn't have the
 computational resources to run the neural architecture search for bigger
 networks this is now as about as big as a resonant 50 right but what if you
 wanted to go to a resonant 100 or a resonant 150 there you you don't have
 the computational resources do neural architectures imagine this Google has
 doesn't have the computational resources to do neural architecture
 search on this thing so this must be expensive or I'm just I have no idea
 but what they do is they kind of do a trick so here they take the the spine net
 49 and they say we build a spine net 60 96 by simply repeating each block twice
 so all the incoming connections would go to the first block and all the outgoing
 connections would come from the second block right here you had two in and
 maybe there's actually no limit to how many outgoing connections you can have
 and also you can also do this three times which I think is a bit of a cheap
 way and it kind of defeats the entire purpose right couldn't you make the exact
 same argument again here that maybe it's helpful to route from this block right
 here or maybe it's helpful that these don't have the same scale right after
 one another it just seems but okay so they say we found this good structure
 and we simply duplicate each block I'm not that big of a fan in any case so
 they train this and it of course outperforms everything else if you
 compare with kind of models of the same size so here you compare this spine net
 49 to the resnet 50 and you can see there's about the same number of
 parameters how about it outperforms the resnet 50 pretty much and as you go up
 the number of parameters here the performance goes up yet again and I
 believe these dagger ones here are simply trained with a special schedule
 with here with applying stochastic depth and swish activation for a longer
 training schedule so you can see that not only do are these spine nets sorry
 of the number of parameters is here not only are the spine nets slightly smaller
 than the resnets they do require less flops and they reach better accuracy so
 you know every everything is a win here yeah so they apply this to these data
 sets I don't want to go you know too much into into that but in the last
 part they also apply this to image net so there's image classification where
 they basically say okay we can just go to our architecture and we can just add
 up all the output blocks we scale them appropriately and add up all the output
 blocks right here because these are good features for localization and so on and
 we can train it to do image classification so all of these go into a
 big combination classifier that does the 1000 classes of image net image
 classification and that also works pretty well with this network so they
 basically argue what they found is sort of a better image image processing
 network than the resnet 50 and I guess they would argue that from now on you
 should take this as your backbone for image classification and recognition and
 so on which it's entirely possible that this works better there's no particular
 reason why the resnet 50 should work at all right it's just a heuristic but I
 guess the I it remains to be to be seen whether that's generally true or just in
 the things they considered so you can see right here the spine net generally
 improving over the image net which isn't is not stated right here but it does
 generally improve and you can see as you go higher and higher spine net the the
 numbers tend to improve as well and this is already pretty respectable
 respectable number for image net right all right so this was it for this paper
 for this particular paper they do have you know two different of these object
 detection recognition datasets and I invite you to check out the experiments
 more closely if you're interested in that sort of thing I was mainly
 interested in the method of doing and arranging these layers and so on it
 seems like it's a cool engineering project cool investigative project the
 experiments are done well and in the end they reach a better you know they
 achieve to get a better model out of that and if it turns out that this model
 is a good model the entire community will be better off unfortunately there's
 no broader impact statement to tell us that also the terrorists will be able to
 use this for purposes but you can imagine that yourself all right that was
 it for me again leave a comment if you want me to change anything or have
 suggestions leave a like if you like the video share it out bye bye