DETR: End-to-End Object Detection with Transformers (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper facebook fair fb facebook ai object detection coco bounding boxes hungarian matching bipartite cnn transformer attention encoder decoder images vision pixels segmentation classes stuff things attention mechanism squared unrolled overlap threshold rcnn
Object detection in images is a notoriously hard task! Objects can be of a wide variety of classes, can be numerous or absent, they can occlude each other or be out of frame. All of this makes it even more surprising that the architecture in this paper is so simple. Thanks to a clever loss function, a single Transformer stacked on a CNN is enough to handle the entire task!

OUTLINE:
0:00 - Intro & High-Level Overview
0:50 - Problem Formulation
2:30 - Architecture Overview
6:20 - Bipartite Match Loss Function
15:55 - Architecture in Detail
25:00 - Object Queries
31:00 - Transformer Properties
35:40 - Results

ERRATA:
When I introduce bounding boxes, I say they consist of x and y, but you also need the width and height.

My Video on Transformers: https://youtu.be/iDulhoQ2pro

Paper: https://arxiv.org/abs/2005.12872
Blog: https://ai.facebook.com/blog/end-to-end-object-detection-with-transformers/
Code: https://github.com/facebookresearch/detr

Abstract:
We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at this https URL.

Authors: Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're going to look at end-to-end object detection with transformers by Nicolas Carillon, Francisco Massa and others at Facebook AI Research. So on a high level this paper does object detection in images using first a CNN and then a transformer to detect objects and it does so via a bipartite matching training objective. And this leaves you basically with an architecture that is super super simple compared to the previous architectures that had all kinds of engineering hurdles and thresholds and hyper parameters. So really excited for this. As always if you like content like this consider leaving a like comment or subscribing. Let's get into it. So let's say you have a picture like this here and you're supposed to detect all the objects in it and also where they are and what they are. This task is called object detection. So a good classifier here would say there's a bird right here and so this is a bird and then this here is also a bird. Right? They can be overlapping these bounding boxes so this is you see the the first problem that bird why is that green? Nevermind. Okay and those are the only two objects. So there's a number of very difficult things here. First of all you need to sort of detect the objects. You need to know how many there are. It's all it's not always the same in each image. There can be multiple objects of the same class. There can be multiple objects of different classes. They can be anywhere of any size. They can be overlapping in the background small or across the entire image. They can occlude each other partially. So the problem is a very very difficult problem and previous work has done a lot of engineering on this like building detectors and then kind of you want to classify every single pixel here and then you get like two detections right here that are very close for the same class. You say ah that must maybe be the same instance. Right? So there's only one thing here and not two things and so on. So there used to be very complicated architectures that solve these problems and this paper here comes up with a super simple architecture and will kind of go from the high level to the low to the implementation of each of the parts. So what does this paper propose? How do we solve a task like this? First of all we put the image and the image here without the labels of course. We put it through a convolutional neural network encoder. Since this is an image task it's you know kind of understandable that we do this mostly because CNNs just work so well for images. So this gives us this set of image features and I think this vector here is not really representative of what's happening. So let's actually take this picture right here and draw it in kind of an angled way and what we'll do with CNN is we'll simply sort of scale it down but have it multiple. So here it's three channels right? It's red, green and blue like this. Three channels but we'll scale it down but we make it more channels. So yeah so more channels. Okay but it's still sort of an image right here. It still has the image form. So the CNN basically gives us this thing which is sort of a higher level representation of the image with many more feature channels but still kind of information where in the image those features are. This is going to be important in a second because now this thing which is this set of image features goes into a transformer encoder decoder and this is sort of the magic thing here as a component. We'll look into that in a second but what they get out right here is this set of box predictions. So outcomes, each of these boxes here is going to be consisting of a tuple and the tuple is going to be the class and the bounding box. So an example for this could be bird at x equals 2, y equals 5. That's an example. Another example of this could also be there is nothing at x equals 7, y equals 9. So the nothing class is a valid class right here and that's also important. But safe to say there is this set of box predictions and then that is basically your output. These things are your output. If you have those things you can draw these bounding boxes, you can assign the labels. The question is how do you train it? Now what you're given is a database of images and these images as you see here on the right, these images already have by human annotators drawn these bounding boxes in and also labels. So this here would be annotated with bird and this here would be annotated with bird. But it doesn't have any of these like it doesn't annotate the nothing classes and so on. So the question is how do you compare the two? Can you simply say okay if the first one here is the bird and then and the second one is this bird then it's good but then you know that the ordering shouldn't matter. You simply care whether you have the correct bounding boxes, you don't care whether you output them in the correct order. And also what if your classifier does something like this? It outputs those two boxes we see here but it also outputs this here and says bird or like one that is slightly off and says bird and so on. So how do you deal with all of these cases? So the way that this paper deals with all of these cases is with their bipartite matching loss this thing right here. So how does it work? Let's say your... where can we go? Let's say your classifier, so here is an image. I have to wait for this to catch up. Here is an image and we put it through this entire pipeline and we get a set of predictions and they're going to be class bounding box, class bounding box, class bounding box. Now the first thing you need to know is that there are always the same amount of predictions. There are always this size here is fixed, that's large n. That's kind of a maximum of predictions. Since you can always predict either a class or the nothing class, in this case you could predict anywhere from zero to five objects in the scene. And then the second thing is from your database you get out an image with its bounding box annotations that are made by human labellers. Let's say these two. And you also do class bounding box, class bounding box. But now you see we only have two instances, so here we just pad with the nothing class. So I don't know what the bounding box should be for the nothing class. It doesn't really matter. Nothing, no bounding box, nothing, no bounding box, no bounding box. So your ground truth labels, if you will, are also of size n. So you always compare n things here on the left that your classifier output with n things on the right. Now as we already said the question is how do you deal with... you can't simply compare one by one because the ordering should not be important. But also you don't want to encourage your classifier to always kind of... if the one bird is very prominent, you don't want to encourage your classifier to say, here's a bird, here's a bird, there's a bird right here, hey, hey, there's a bird, there's a bird, there's a bird. And basically just because the signal for that bird is stronger and basically ignore the other bird, what you want to do is you want to encourage some sort of your classifier to detect if it has already detected an object, it shouldn't detect it again in a slightly different place. So the way you do this is with this bipartite matching loss. So at the time when you compute a loss, you go here and you compute what's called a maximum matching. Now what you have to provide is a loss function. So we can... there's a loss function L and L will take two of these things. L will take the red, the predicted thing of your model and L will take the true under... one of the true underlying things and L will compute a number and will say how well do these two agree. So you can say for example if either of them is the nothing class then I have no loss, like I don't care about them, that gives you no loss. But if the two classes agree and the two bounding boxes agree then it's very good right? Then we maybe even give like some negative loss or give loss zero. But if the bounding boxes agree but the classes don't agree then you say that's bad. Or the other way around if the classes agree in the bounding... or even if everything disagrees it's the worst. What you're basically saying is if these two would correspond to each other, if the thing on the left were the prediction for the thing on the right, which we don't know, it could be that the thing on the right refers to the bird on the right and the thing on the left refers to the bird on the left. So it would be natural that the bounding boxes are the same. But you say if these were corresponding to each other what would the loss be? How well would they do? And now if you compute this bipartite matching, what you want, I guess it's a it's a minimum matching in this case, what you want is you want to find an assignment of things on the left to things on the right. A one-to-one assignment. This is an example of a one-to-one assignment. Everything on the left is assigned exactly one thing on the right such that the total loss is minimized. So you're going to say I'm going to align the things on the left with the things on the right such that it's maximally favorable. I give you the maximum benefit of the doubt by aligning these things. So in the best possible case what's the loss? This is somehow clear. So this you're trying to find the assignment from the left to the right that makes that basically is the best case for this output right here. Where you really say oh okay here you output a bird very close to the bird here in the in a ground truth label. That's this here. So I'm going to connect these two because that's sort of it's it gives a model the most benefit of the doubt. And the loss that you have at the end of that matching, so this loss here would only then count wherever these connections are, that loss is going to be your training loss. So this solves the problems we had before. It is not dependent on the order because if you reorder the things your minimum matching will simply swap with it. If you output the same bird multiple times only one of these is going to be assigned. So if this here is that bird only one of them, only this one maybe, is going to be assigned to that one. And the other ones can't be assigned to that one, are forced to be assigned to a different one. Let's say this one here and are going to incur a loss. So you encourage your model to output let's say diverse bounding boxes, different bounding boxes for things. So this solves these problems and it's very clever. And there are algorithms to compute these minimum matchings. They use the Hungarian algorithm which will give you exactly such a matching. Again this is possible because you have n things on each side and the n is in effect here is the maximum of objects that you can detect at once. I guess if there is less you can simply pad right here. And then the model of course is encouraged to come up with the equal number of no class predictions. Because if it outputs a prediction when it shouldn't, if it already predicts two things and these are assigned to these two things and then it outputs one more thing it is going to be penalized because it should output three things with no class but it has output one too many with a class, it's going to be penalized. Okay so this is a pretty pretty cool thing. Again it relies on the fact that you have n on both sides but you can make n so large that basically it covers all of the cases. So you can make n like 50. So you can detect up to 50 things in a scene. Alright that's the algorithm in a high level. They do show their loss here. You see the loss ultimately is going to be over this matching right here. That's the minimum bipartite assignment that basically minimizes this total loss over your prediction and label matchings. And the loss they come up with here, I said you have to give the algorithm a loss, is this one. And they kind of go into how they do it. I don't think it's super important so the class algorithm, sorry the loss on the class labels I think is going to be a softmax or a sorry a cross entropy loss like an usual classification. And the loss on the to say whether two bounding boxes agree is a mixture of the L1 loss that compares two bounding boxes and this IOU loss which is not dependent on the scale of the bounding boxes. It kind of computes how much fraction of the two bounding boxes overlap. But in any case the loss basically consists of saying how how much do the labels agree and how much do the bounding boxes agree. Again this is only possible because after that you compute this matching otherwise you would have no clue which predictions to compare to which other predictions. So let's look at this architecture a bit more in detail. As we said you have this what they call the backbone which is a convolutional neural network. And with that you put in some positional encodings. Now I already said the you should look at these features right here as just smaller feature versions of the image but they still have some image nature. Then they are flattened. So once they are put in the transformer encoder because the transformer is naturally a sequence processing unit okay so it takes in just a sequence of vectors right here. And since an image is not a sequence what you'll do is if you have your image features and we said we have a bunch of channels let's say we have four channels and they're height and width and C you're going to unroll and flatten that into one sequence. So this is height times width you basically unroll across these axis right here into this axis and it's channel size. So basically you have a sequence here of C dimensional feature vectors that you then put into your encoder. So your encoder will now transform this sequence into an equally long sequence yet again of features. And the good thing about a transformer because why do you use a transformer? The good thing about the transformer is that in such a sequence and I've done videos on transformers you can basically mainly look at the video attention is all you need if you want to understand this more fully. This thing can basically have attention so it has attention layers it can attend from each position to each position in a one-shot manner. So as it transforms this representation up the transformer layers at each step it can basically aggregate information from everywhere in the sequence to anywhere else and therefore it's very it's very powerful if you have a sequence and you need sort of global connections across the sequence. This is very good for a language processing because in a sentence let's look at this sentence the input images are batched together. Applying blah blah blah blah blah blah blah blah blah and then there's they right the word they and you need you need to know that they refers to the input images okay and but you see this is very very far away in the sentence so you need a model that makes use of long range dependencies and they make the case that in such a task right here you also need the long range dependencies because these bounding boxes as you see right here they can be quite large so if you have an image you need that this part here communicates with these and this and this and this part basically anywhere in the bounding box and these bounding boxes can be quite large so the transformer architecture actually makes sense here. Now I want to go a bit later into why I think it actually makes even more sense for bounding box detection but right now I just want to keep going through this through this architecture right here so if my computer here decides to come back yes we can go on so what we'll get out is yet another so in here we put this thing we put down here we put into the transformer encoder and we get an equally sized equally shaped sequence out of the transformer encoder and you see that this thing here goes as a side input into this transformer decoder so the transformer encoder here is just a bit more of a feature mapping technically just for the architecture you could think of just putting this into here but of course it's going to go better with the transformer encoder the transformer decoder now does something similar but you see it has the encoder as a side input this is very much like this is not like BERT BERT is like a only encoder transformer whereas this is much like the original attention is all you need transformer that has an encoder and then the decoder as a side input basically as conditioning information has the encoder output what does the decoder do again since it's a transformer it's going to take a sequence and output a sequence the sequence it takes is right here is what they call object queries and this also is different from the attention is all you need papers and they don't do it autoregressively they just do it one shot what does it mean it means that you start with a sequence here of four things and these are these are the this is this big N and you out you output the sequence of a sequence of four things and it's important to see what they're going to end up so these things are then directly going through a classifier that now outputs the so these things here are these class label bounding box outputs okay so each of these things is going to after transformation end up being one of these bounding boxes either defining an object or saying that there isn't an object somewhere okay you see here this bounding box refers to this bird this bounding box refers to this bird so each of these things is going to be one bounding box and the what they call object queries is the question of course is what do you input here right I actually I want to transform this image information that comes from the left here I want to transform that into the bounding boxes what do I input here and the answer is you just input at the start you just input n random vectors because what's that gonna give you is basically n outputs you want an outputs because you want n of these bounding box classifications so you need n things and if I input n things into a transformer it's going to give me n things as an output and then in each step I can simply condition on the information that comes in the images and it it'll give me right then I can incorporate that information it's a very deep learning way of thinking about it actually so that you just need the information somewhere in there and I need n things now they go more into detail into this transformer architecture help help in a helpful fashion in the appendix and we'll go there quickly so this I think here makes more sense so the image features come in here right and you see this is just a transformer stack an encoder stack of multi-head self-attention and feed forward in instance wise or like token wise feed forward network and then that information is taken and is given as conditioning information over here now in here as I said you input these object queries which at the beginning are just n random vectors and what you're going to do you are also going to feature encode them and then you combine it with this image information so ultimately if you think of this one of these things one of these things is going to be a vector right and then that vector is going to be transformed and then it will have as it is transformed it will have the opportunity to basically look at features that come from here the arrow is in the wrong direction so you have already taken the image and you've transformed it into a feature representation which is also a vector right you have the features of the image right here now as you transform this vector this object query Q you have the opportunity to look at the image features right and that's how do you get the image information in there so the image features will come in here transform that through attention so this is an attention mechanism on the image and then what you will output is a bounding box and a class label it's really hard to explain I would guess you need to understand really what attention mechanisms are and of course the crucial part of course is what what's this what do you input at the beginning and these object queries aren't actually random as I said they are learned so what you're going to do is you're going to learn in dependent of the input image you're going to learn n different object queries and these object queries now it's very it's very interesting because these object queries are sort of going to be different it's like you have different people that can ask the input image different questions right and this they have so their end is 100 but they show 20 of these object queries that they learn and so they have visualization of all bounding box predictions on all images so it's it's sort of like you have n different people at your disposal and you train these n different people to kind of ask different questions of the input image okay you say this person up here will always ask irrespective of what the input images will always ask sort of hey input image what's what's on your bottom left right that's I'm really interested what's on your bottom left and sometimes I'm a bit interested in what's here but I'm mainly interested what's on the bottom left of the image whereas this person right here sorry this person right here is more interested in what's in the center of the different colors here is referred to different sizes of bounding boxes so this person is also interested so the person on the top left is interested mainly in I think small bounding boxes that are on the bottom left and the person here is mostly interested in what's I'm really interested what's in the center what's large in the center I want give me large things that are in the center right and then this person right here is really interested on stuff that's on the right side of the image so you see in order to get different sort of a difference in bounding box predictions you train n different people to ask different questions of the of the input image and this asking of questions is exactly what an attention mechanism is so this person right here let's let's take this this person and I'm saying person these are vectors these are learned object queries but this person first they will simply ask the question what's on what's on the right side and then the the image features right getting drawing the image features it will have an attention mechanism to this part of the image features and then it will get back some signal right and then it will transform that with its own signal up and then it will ask maybe again okay now that I know more because you see that person is interested in multiple things it's interested in those things and those things so at first it will focus on these things but then it says on now I'm now I know more right there is there I know I see there is actually something on the right side so in the higher layers it can then go back and ask the image more questions by sending these Q vectors of the attention mechanism and it will get back the V vectors from the image features that correspond to these Q things so up and up the layers this person can ask more refined questions about what that particular person is interested in okay and since you have the different people here that ask different questions you basically learn the people in a way such that across the data set they all together they cover every possible image pretty well again these people what they're interested in initially is not dependent on the picture you simply learn this in a global manner all right this is the best way I have of describing it he basically learned and people that are each one is interested in different things different classes and different regions in the image and each one of these people is going to output their best guess of what is where based on what they're interested in so that person might say I'm you know I'm the person that's interested kind of in the left side of things so I I'm going to output that there is a bird right here now these people if this is a transformer right and everything can attend to everything they can actually communicate with each other as they incorporate information from the image so in each layer they can do both they can incorporate information from the image and they can communicate with each other and then in the next layer they can do it again and again and again and thereby they can sort of they can sort of say well you already got the left side I will take the right side you already got the bird class I will take the elephant class and so on so you see here how the the architecture of the transformer actually is also very conducive to doing this bounding box prediction in that these different things can sort of attend to each other and therefore communicate with each other all right I hope that sort of makes sense now before we get into the experiments I want to list a third reason of why the transformer especially the encoders might actually also make a giant amount of sense here since you unroll the image into height and width and you have to imagine what does the transformer do the transformer as we said here has this notion of a tension where from any point in the sequence it can gather information from any other point in the sequence and this that's usually one of the downsides of the transformers is done via a quadratic attention mechanism so if I just list one feature channel go over here if I just list one feature channel right here this is height times width of the image right this is this is the entire image unrolled in one vector height times width and here I unroll it again height times width then this this matrix that I can build right here which is called the attention matrix right here it will tell me which parts of the sequence attend to which other parts okay so if you have an image that has the let's say the number three and you really want to figure out whether or not this is a three then the bow up here must communicate with the bow down here right they need to share information is it ah there's a bow here there's a bow here and there is a spiky thing here that must be a three so you want something this is rather at the beginning of the sequence you want this to attend first of all it will attend itself so you get fairly high values along the diagonal maybe 10 10 10 11 11 12 I saw this olig skit 100 million nine nine but it also like this this part here at the beginning of the sequence let's say it's here because this is unrolled right needs to attend to the end so this needs to attend to the end which we will put an 11 here and the other way around doesn't always need to be symmetrical by the way okay but in any case this is going to be a h times w squared matrix because everything can attend to everything and that's the attention mechanism why do I think this is so good for bounding boxes because let's let's imagine you actually have a matrix that is like this okay height times width times height times width every single point in here actually defines a bounding box because this point this point right here in this dimension corresponds to one location in the image and on this axis it corresponds to another location now in the attention matrix simply means these two points need to communicate but if you have two pixels you actually have defined a bounding box right here right you you are actually you're defining a bounding box and the the fact that this is happening in the exact same matrices could mean that the transformers are uniquely well the transformers across sequences of these height times width unrolled images are uniquely well conducive to these bounding box prediction tasks I'm actually a bit astounded because when I first just read the title this immediately popped to my mind I'm like oh yes of course and they're going to predict the bounding boxes by simply training so what you would do what I thought this was gonna be is out you output an actual matrix like this and then you simply each point you can you can simply classify right so you can classify here whether whether or not like at in this direction there is a bird right and then if you have two points like this for example you and you also classify whether in this direction there is a bird right and this naturally defines a bounding box or you could like take this matrix and actually just classify individual points in this matrix to be the bounding boxes because they already define bounding boxes so I just I think these these quadratic things are are uniquely I mean someone must have thought of this or if not cite the YouTube channel it would be funny first paper ever to actually have to cite the YouTube channel but again yeah so transformers seem to be a good idea for these kinds of things so how do they do of course they do well they are on par with these other much much much more complex architectures these faster are CNN models they are apparently much more complex but they are on par with this they do however train forever I think they train for like six days on eight GPUs is not that much if you compare to like language models on hundreds of TPUs but still okay I don't want to go into the numbers of experiments but what is pretty cool is that they can now visualize this sort of attention and you can see right here that if they look at a particular point in the image and visualize the attention it will actually attend to the instance itself so it will like these are usually the problems for these detection algorithms when things overlap and are partially occluded but you can see right here that the attention is on the part of the image that makes the instance in the back and the attention here is on the part of this and it doesn't sort of overlap into the others so that is one thing that's pretty impressive about these architectures the other thing they show is for example it can generalize to many many instances so here it has never seen 24 giraffes in one image but yet it can absolutely do that and giraffe giraffe giraffe giraffe giraffe and the one of the coolest images I find are these here where you can see right here again attention visualization and you see that even within the bounding box of the front elephant here you see that the attention on this foot of the back elephant is is is assigned to this blue bounding box so this is the blue basically the blue bounding box person that is attending to that back foot that means they they these things really sort of understand or they learn these things like occlusion and just hard I have a hard time describing it but you can see it visually here right like how it clearly learns that these are two instances that are sort of occluding each other but this this this instance can actually appear within the bounding box of the other instance and the same goes for the zebra here that are partially occluding each other and you can see that the attention is correctly like even here that this back foot of this zebra is correctly labeled so all in all that is pretty cool and they take it a step further and they say well with this architecture we can actually pretty easily do pixel wise classification so this is this cocoa stuff and things data set where I don't know which one is the stuff and which one is the things I think things is the objects and stuff is like sky and mountains and so on and so this is a classification task where you actually have to label every single pixel so what they do is they simply input this through their detector and they detect the instances they take the attention maps of the instances and then they scale it up this right here is just a CNN sort of in reverse that scales up the image because they have scaled it down as we said they scale it up again and then they can simply classify each pixel where each of these you remember we had these different people here that it that cared about different things in the image each of these people will classify their respective pixels the pixels they feel responsible for and then you simply merge all of these people's predictions together into this prediction and again this gives pretty pretty impressive results I am I mean this is this is fun this looks like it sort of works haven't they do quantitative analysis of course but I'm just impressed by the examples right here alright that was sort of it I really enjoyed reading this papers the simplicity is pretty cool they do have not only do they have code in the paper to show how ridiculously easy it is to get this to run this is all you need in pytorch but they do actually have code and as I understand they also have pre-trained models so they have this model zoo right here where they give you the pre-trained models so you can play with it and you can even load it from torch hub yourself and you can train it yourself they have a collab all is there all right again if you enjoyed this video consider leaving a like subscribing and I'll see you next time bye bye
 Hi there! Today we're going to look at end-to-end object detection with
 transformers by Nicolas Carillon, Francisco Massa and others at Facebook AI
 Research. So on a high level this paper does object detection in images using
 first a CNN and then a transformer to detect objects and it does so via a
 bipartite matching training objective. And this leaves you basically with an
 architecture that is super super simple compared to the previous architectures
 that had all kinds of engineering hurdles and thresholds and hyper
 parameters. So really excited for this. As always if you like content like this
 consider leaving a like comment or subscribing. Let's get into it. So let's
 say you have a picture like this here and you're supposed to detect all the
 objects in it and also where they are and what they are. This task is called
 object detection. So a good classifier here would say there's a bird right here
 and so this is a bird and then this here is also a bird. Right? They can be
 overlapping these bounding boxes so this is you see the the first problem that
 bird why is that green? Nevermind. Okay and those are the only two objects. So
 there's a number of very difficult things here. First of all you need to
 sort of detect the objects. You need to know how many there are. It's all it's
 not always the same in each image. There can be multiple objects of the same
 class. There can be multiple objects of different classes. They can be anywhere
 of any size. They can be overlapping in the background small or across the
 entire image. They can occlude each other partially. So the problem is a very very
 difficult problem and previous work has done a lot of engineering on this like
 building detectors and then kind of you want to classify every single pixel here
 and then you get like two detections right here that are very close for the
 same class. You say ah that must maybe be the same instance. Right? So there's only
 one thing here and not two things and so on. So there used to be very
 complicated architectures that solve these problems and this paper here comes
 up with a super simple architecture and will kind of go from the high level to
 the low to the implementation of each of the parts. So what does this paper
 propose? How do we solve a task like this? First of all we put the image and the
 image here without the labels of course. We put it through a convolutional neural
 network encoder. Since this is an image task it's you know kind of understandable
 that we do this mostly because CNNs just work so well for images. So this gives us
 this set of image features and I think this vector here is not really
 representative of what's happening. So let's actually take this picture right
 here and draw it in kind of an angled way and what we'll do with CNN is we'll
 simply sort of scale it down but have it multiple. So here it's three channels
 right? It's red, green and blue like this. Three channels but we'll scale it down
 but we make it more channels. So yeah so more channels. Okay but it's still sort
 of an image right here. It still has the image form. So the CNN
 basically gives us this thing which is sort of a higher level representation of
 the image with many more feature channels but still kind of information
 where in the image those features are. This is going to be important in a
 second because now this thing which is this set of image features goes into a
 transformer encoder decoder and this is sort of the magic thing here as a
 component. We'll look into that in a second but what they get out right
 here is this set of box predictions. So outcomes, each of these boxes here is
 going to be consisting of a tuple and the tuple is going to be the class and
 the bounding box. So an example for this could be bird at x equals 2, y
 equals 5. That's an example. Another example of this could also be there is
 nothing at x equals 7, y equals 9. So the nothing class is a valid class
 right here and that's also important. But safe to say there is this set of box
 predictions and then that is basically your output. These things are
 your output. If you have those things you can draw these bounding boxes, you can
 assign the labels. The question is how do you train it? Now what you're given is a
 database of images and these images as you see here on the right, these images
 already have by human annotators drawn these bounding boxes in and also labels.
 So this here would be annotated with bird and this here would be annotated
 with bird. But it doesn't have any of these like it doesn't annotate the
 nothing classes and so on. So the question is how do you compare the two?
 Can you simply say okay if the first one here is the bird and then and the second
 one is this bird then it's good but then you know that the ordering shouldn't
 matter. You simply care whether you have the correct bounding boxes, you
 don't care whether you output them in the correct order. And also what if your
 classifier does something like this? It outputs those two boxes we see here but
 it also outputs this here and says bird or like one that is slightly off and
 says bird and so on. So how do you deal with all of these cases? So the way that
 this paper deals with all of these cases is with their bipartite matching loss
 this thing right here. So how does it work? Let's say your... where can we go?
 Let's say your classifier, so here is an image. I have to wait for this to catch
 up. Here is an image and we put it through this entire pipeline and we
 get a set of predictions and they're going to be class bounding box, class
 bounding box, class bounding box. Now the first thing you need to know is that
 there are always the same amount of predictions. There are always this
 size here is fixed, that's large n. That's kind of a maximum
 of predictions. Since you can always predict either a class or the nothing
 class, in this case you could predict anywhere from zero to five objects in
 the scene. And then the second thing is from your database you
 get out an image with its bounding box annotations that are made by human
 labellers. Let's say these two. And you also do class bounding box, class bounding
 box. But now you see we only have two instances, so here we just pad with
 the nothing class. So I don't know what the bounding box should be for the
 nothing class. It doesn't really matter. Nothing, no bounding box, nothing, no
 bounding box, no bounding box. So your ground truth labels, if you will, are also
 of size n. So you always compare n things here on the left that your classifier
 output with n things on the right. Now as we already said the question is how do
 you deal with... you can't simply compare one by one because the ordering should
 not be important. But also you don't want to encourage your classifier to always
 kind of... if the one bird is very prominent, you don't want to
 encourage your classifier to say, here's a bird, here's a bird, there's a bird
 right here, hey, hey, there's a bird, there's a bird, there's a bird. And
 basically just because the signal for that bird is stronger and basically
 ignore the other bird, what you want to do is you want to encourage some sort of
 your classifier to detect if it has already detected an object, it shouldn't
 detect it again in a slightly different place. So the way you do this is
 with this bipartite matching loss. So at the time when you compute a loss,
 you go here and you compute what's called a maximum matching. Now what you
 have to provide is a loss function. So we can... there's a loss function L and L
 will take two of these things. L will take the red, the predicted thing of your
 model and L will take the true under... one of the true underlying things and L
 will compute a number and will say how well do these two agree. So you can say
 for example if either of them is the nothing class then I have no loss, like I
 don't care about them, that gives you no loss. But if the two classes
 agree and the two bounding boxes agree then it's very good right? Then we maybe
 even give like some negative loss or give loss zero. But if the bounding
 boxes agree but the classes don't agree then you say that's bad. Or the other way
 around if the classes agree in the bounding... or even if everything disagrees it's
 the worst. What you're basically saying is if these two would
 correspond to each other, if the thing on the left were the prediction for
 the thing on the right, which we don't know, it could be that the thing on
 the right refers to the bird on the right and the thing on the left refers to
 the bird on the left. So it would be natural that the bounding boxes are the
 same. But you say if these were corresponding to each other what would
 the loss be? How well would they do? And now if you compute this bipartite
 matching, what you want, I guess it's a it's a minimum matching in this case,
 what you want is you want to find an assignment of things on the left to
 things on the right. A one-to-one assignment. This is an example of a
 one-to-one assignment. Everything on the left is assigned exactly one thing on
 the right such that the total loss is minimized. So you're going to say I'm
 going to align the things on the left with the things on the right such that
 it's maximally favorable. I give you the maximum benefit of the doubt by
 aligning these things. So in the best possible case what's the loss?
 This is somehow clear. So this you're trying to find the assignment
 from the left to the right that makes that basically is the best case for this
 output right here. Where you really say oh okay here you output a bird
 very close to the bird here in the in a ground truth label. That's this here. So
 I'm going to connect these two because that's sort of it's
 it gives a model the most benefit of the doubt. And the loss that you have at the
 end of that matching, so this loss here would only then count wherever these
 connections are, that loss is going to be your training loss. So this solves
 the problems we had before. It is not dependent on the order because if you
 reorder the things your minimum matching will simply swap
 with it. If you output the same bird multiple times only one of
 these is going to be assigned. So if this here is that bird only one of them,
 only this one maybe, is going to be assigned to that one. And the other ones
 can't be assigned to that one, are forced to be assigned to a different one. Let's
 say this one here and are going to incur a loss. So you encourage your model to
 output let's say diverse bounding boxes, different bounding boxes for things.
 So this solves these problems and it's very clever. And there are
 algorithms to compute these minimum matchings. They use the Hungarian
 algorithm which will give you exactly such a matching. Again this is possible
 because you have n things on each side and the n is in effect here is the
 maximum of objects that you can detect at once. I guess if there is less you can
 simply pad right here. And then the model of course is encouraged to come up with
 the equal number of no class predictions. Because if it outputs a prediction when
 it shouldn't, if it already predicts two things and these are assigned to
 these two things and then it outputs one more thing it is going to be penalized
 because it should output three things with no class but it has output one too
 many with a class, it's going to be penalized. Okay so this is a pretty
 pretty cool thing. Again it relies on the fact that you have n on both sides
 but you can make n so large that basically it covers all of the cases. So
 you can make n like 50. So you can detect up to 50 things in a scene. Alright
 that's the algorithm in a high level. They do show their loss here. You see the
 loss ultimately is going to be over this matching right here.
 That's the minimum bipartite assignment that basically minimizes this total loss
 over your prediction and label matchings. And the loss they come up with here, I
 said you have to give the algorithm a loss, is this one. And they kind of go
 into how they do it. I don't think it's super important so the class algorithm,
 sorry the loss on the class labels I think is going to be a softmax or a
 sorry a cross entropy loss like an usual classification. And the loss on the to
 say whether two bounding boxes agree is a mixture of the L1 loss that compares
 two bounding boxes and this IOU loss which is not dependent on the scale of
 the bounding boxes. It kind of computes how much fraction of the two bounding
 boxes overlap. But in any case the loss basically consists of saying how
 how much do the labels agree and how much do the bounding boxes agree.
 Again this is only possible because after that you compute this matching
 otherwise you would have no clue which predictions to
 compare to which other predictions. So let's look at this architecture a bit
 more in detail. As we said you have this what they call the backbone which is a
 convolutional neural network. And with that you put in some positional encodings.
 Now I already said the you should look at these features right here as just
 smaller feature versions of the image but they still have some image nature.
 Then they are flattened. So once they are put in the transformer encoder
 because the transformer is naturally a sequence processing unit okay so it
 takes in just a sequence of vectors right here. And since an image is not a
 sequence what you'll do is if you have your image features and we said we have
 a bunch of channels let's say we have four channels and they're height and
 width and C you're going to unroll and flatten that into one sequence. So this
 is height times width you basically unroll across these axis right here into
 this axis and it's channel size. So basically you have a sequence here of
 C dimensional feature vectors that you then put into your encoder. So
 your encoder will now transform this sequence into an equally long sequence
 yet again of features. And the good thing about a transformer because why do you
 use a transformer? The good thing about the transformer is that in such a
 sequence and I've done videos on transformers you can basically mainly
 look at the video attention is all you need if you want to understand this more
 fully. This thing can basically have attention so it has attention layers it
 can attend from each position to each position in a one-shot manner. So as it
 transforms this representation up the transformer layers at each step it can
 basically aggregate information from everywhere in the sequence to anywhere
 else and therefore it's very it's very powerful if you have a sequence and you
 need sort of global connections across the sequence. This is very good for a
 language processing because in a sentence let's look at this sentence the
 input images are batched together. Applying blah blah blah blah blah blah
 blah blah blah and then there's they right the word they and you need you
 need to know that they refers to the input images okay and but you see this
 is very very far away in the sentence so you need a model that makes use of long
 range dependencies and they make the case that in such a task right here you
 also need the long range dependencies because these bounding boxes as you see
 right here they can be quite large so if you have an image you need that this
 part here communicates with these and this and this and this part basically
 anywhere in the bounding box and these bounding boxes can be quite large so the
 transformer architecture actually makes sense here. Now I want to go a bit later
 into why I think it actually makes even more sense for bounding box detection
 but right now I just want to keep going through this through this architecture
 right here so if my computer here decides to come back yes we can go on so
 what we'll get out is yet another so in here we put this thing we put down here
 we put into the transformer encoder and we get an equally sized equally shaped
 sequence out of the transformer encoder and you see that this thing here goes as
 a side input into this transformer decoder so the transformer encoder here
 is just a bit more of a feature mapping technically just for the architecture
 you could think of just putting this into here but of course it's going to go
 better with the transformer encoder the transformer decoder now does something
 similar but you see it has the encoder as a side input this is very much like
 this is not like BERT BERT is like a only encoder transformer whereas this
 is much like the original attention is all you need transformer that has an
 encoder and then the decoder as a side input basically as conditioning
 information has the encoder output what does the decoder do again since it's a
 transformer it's going to take a sequence and output a sequence the
 sequence it takes is right here is what they call object queries and this also
 is different from the attention is all you need papers and they don't do it
 autoregressively they just do it one shot what does it mean it means that you
 start with a sequence here of four things and these are these are the this
 is this big N and you out you output the sequence of a sequence of four things
 and it's important to see what they're going to end up so these things are then
 directly going through a classifier that now outputs the so these things here are
 these class label bounding box outputs okay so each of these things is going to
 after transformation end up being one of these bounding boxes either defining an
 object or saying that there isn't an object somewhere okay you see here this
 bounding box refers to this bird this bounding box refers to this bird so each
 of these things is going to be one bounding box and the what they call
 object queries is the question of course is what do you input here right I
 actually I want to transform this image information that comes from the left
 here I want to transform that into the bounding boxes what do I input here and
 the answer is you just input at the start you just input n random vectors
 because what's that gonna give you is basically n outputs you want an outputs
 because you want n of these bounding box classifications so you need n things and
 if I input n things into a transformer it's going to give me n things as an
 output and then in each step I can simply condition on the information that
 comes in the images and it it'll give me right then I can incorporate that
 information it's a very deep learning way of thinking about it actually so that
 you just need the information somewhere in there and I need n things now they go
 more into detail into this transformer architecture help help in a helpful
 fashion in the appendix and we'll go there quickly so this I think here makes
 more sense so the image features come in here right and you see this is just a
 transformer stack an encoder stack of multi-head self-attention and feed
 forward in instance wise or like token wise feed forward network and then that
 information is taken and is given as conditioning information over here now
 in here as I said you input these object queries which at the beginning are just
 n random vectors and what you're going to do you are also going to feature
 encode them and then you combine it with this image information so ultimately if
 you think of this one of these things one of these things is going to be a
 vector right and then that vector is going to be transformed and then it
 will have as it is transformed it will have the opportunity to basically look
 at features that come from here the arrow is in the wrong direction so you
 have already taken the image and you've transformed it into a feature
 representation which is also a vector right you have the features of the image
 right here now as you transform this vector this object query Q you have the
 opportunity to look at the image features right and that's how do you get
 the image information in there so the image features will come in here
 transform that through attention so this is an attention mechanism on the image
 and then what you will output is a bounding box and a class label it's
 really hard to explain I would guess you need to understand really what attention
 mechanisms are and of course the crucial part of course is what what's this what
 do you input at the beginning and these object queries aren't actually random as
 I said they are learned so what you're going to do is you're going to learn in
 dependent of the input image you're going to learn n different object
 queries and these object queries now it's very it's very interesting because
 these object queries are sort of going to be different it's like you have
 different people that can ask the input image different questions right and this
 they have so their end is 100 but they show 20 of these object queries that
 they learn and so they have visualization of all bounding box
 predictions on all images so it's it's sort of like you have n different people
 at your disposal and you train these n different people to kind of ask
 different questions of the input image okay you say this person up here will
 always ask irrespective of what the input images will always ask sort of hey
 input image what's what's on your bottom left right that's I'm really interested
 what's on your bottom left and sometimes I'm a bit interested in what's here but
 I'm mainly interested what's on the bottom left of the image whereas this
 person right here sorry this person right here is more interested in what's
 in the center of the different colors here is referred to different sizes of
 bounding boxes so this person is also interested so the person on the top left
 is interested mainly in I think small bounding boxes that are on the bottom
 left and the person here is mostly interested in what's I'm really
 interested what's in the center what's large in the center I want give me large
 things that are in the center right and then this person right here is really
 interested on stuff that's on the right side of the image so you see in order to
 get different sort of a difference in bounding box predictions you train n
 different people to ask different questions of the of the input image and
 this asking of questions is exactly what an attention mechanism is so this
 person right here let's let's take this this person and I'm saying person these
 are vectors these are learned object queries but this person first they will
 simply ask the question what's on what's on the right side and then the the image
 features right getting drawing the image features it will have an attention
 mechanism to this part of the image features and then it will get back some
 signal right and then it will transform that with its own signal up and then it
 will ask maybe again okay now that I know more because you see that person is
 interested in multiple things it's interested in those things and those
 things so at first it will focus on these things but then it says on now I'm
 now I know more right there is there I know I see there is actually something
 on the right side so in the higher layers it can then go back and ask the
 image more questions by sending these Q vectors of the attention mechanism and
 it will get back the V vectors from the image features that correspond to these
 Q things so up and up the layers this person can ask more refined questions
 about what that particular person is interested in okay and since you have
 the different people here that ask different questions you basically learn
 the people in a way such that across the data set they all together they cover
 every possible image pretty well again these people what they're interested in
 initially is not dependent on the picture you simply learn this in a
 global manner all right this is the best way I have of describing it he basically
 learned and people that are each one is interested in different things different
 classes and different regions in the image and each one of these people is
 going to output their best guess of what is where based on what they're interested
 in so that person might say I'm you know I'm the person that's interested kind of
 in the left side of things so I I'm going to output that there is a bird
 right here now these people if this is a transformer right and everything can
 attend to everything they can actually communicate with each other as they
 incorporate information from the image so in each layer they can do both they
 can incorporate information from the image and they can communicate with each
 other and then in the next layer they can do it again and again and again and
 thereby they can sort of they can sort of say well you already got the left side
 I will take the right side you already got the bird class I will take the
 elephant class and so on so you see here how the the architecture of the
 transformer actually is also very conducive to doing this bounding box
 prediction in that these different things can sort of attend to each other
 and therefore communicate with each other all right I hope that sort of
 makes sense now before we get into the experiments I want to list a third
 reason of why the transformer especially the encoders might actually also make a
 giant amount of sense here since you unroll the image into height and width
 and you have to imagine what does the transformer do the transformer as we
 said here has this notion of a tension where from any point in the sequence it
 can gather information from any other point in the sequence and this that's
 usually one of the downsides of the transformers is done via a quadratic
 attention mechanism so if I just list one feature channel go over here if I
 just list one feature channel right here this is height times width of the image
 right this is this is the entire image unrolled in one vector height times
 width and here I unroll it again height times width then this this matrix that I
 can build right here which is called the attention matrix right here it will tell
 me which parts of the sequence attend to which other parts okay so if you have an
 image that has the let's say the number three and you really want to figure out
 whether or not this is a three then the bow up here must communicate with the
 bow down here right they need to share information is it ah there's a bow here
 there's a bow here and there is a spiky thing here that must be a three so you
 want something this is rather at the beginning of the sequence you want this
 to attend first of all it will attend itself so you get fairly high values
 along the diagonal maybe 10 10 10 11 11 12 I saw this olig skit 100 million nine
 nine but it also like this this part here at the beginning of the sequence
 let's say it's here because this is unrolled right needs to attend to the
 end so this needs to attend to the end which we will put an 11 here and
 the other way around doesn't always need to be symmetrical by the way okay but in
 any case this is going to be a h times w squared matrix because everything can
 attend to everything and that's the attention mechanism why do I think this
 is so good for bounding boxes because let's let's imagine you actually have a
 matrix that is like this okay height times width times height times width
 every single point in here actually defines a bounding box because this
 point this point right here in this dimension corresponds to one location in
 the image and on this axis it corresponds to another location now in
 the attention matrix simply means these two points need to communicate but if
 you have two pixels you actually have defined a bounding box right here right
 you you are actually you're defining a bounding box and the the fact that this
 is happening in the exact same matrices could mean that the transformers are
 uniquely well the transformers across sequences of these height times width
 unrolled images are uniquely well conducive to these bounding box
 prediction tasks I'm actually a bit astounded because when I first just read
 the title this immediately popped to my mind I'm like oh yes of course and
 they're going to predict the bounding boxes by simply training so what you
 would do what I thought this was gonna be is out you output an actual matrix
 like this and then you simply each point you can you can simply classify right so
 you can classify here whether whether or not like at in this direction there is a
 bird right and then if you have two points like this for example you and you
 also classify whether in this direction there is a bird right and this naturally
 defines a bounding box or you could like take this matrix and actually just
 classify individual points in this matrix to be the bounding boxes because
 they already define bounding boxes so I just I think these these quadratic
 things are are uniquely I mean someone must have thought of this or if not
 cite the YouTube channel it would be funny first paper ever to actually have
 to cite the YouTube channel but again yeah so transformers seem to be a good
 idea for these kinds of things so how do they do of course they do well they are
 on par with these other much much much more complex architectures these faster
 are CNN models they are apparently much more complex but they are on par with
 this they do however train forever I think they train for like six days on
 eight GPUs is not that much if you compare to like language models on
 hundreds of TPUs but still okay I don't want to go into the numbers of
 experiments but what is pretty cool is that they can now visualize this sort of
 attention and you can see right here that if they look at a particular point
 in the image and visualize the attention it will actually attend to the instance
 itself so it will like these are usually the problems for these detection
 algorithms when things overlap and are partially occluded but you can see right
 here that the attention is on the part of the image that makes the instance in
 the back and the attention here is on the part of this and it doesn't sort of
 overlap into the others so that is one thing that's pretty impressive about
 these architectures the other thing they show is for example it can generalize to
 many many instances so here it has never seen 24 giraffes in one image but yet it
 can absolutely do that and giraffe giraffe giraffe giraffe giraffe and the
 one of the coolest images I find are these here where you can see right here
 again attention visualization and you see that even within the bounding box
 of the front elephant here you see that the attention on this foot of the back
 elephant is is is assigned to this blue bounding box so this is the blue
 basically the blue bounding box person that is attending to that back foot that
 means they they these things really sort of understand or they learn these things
 like occlusion and just hard I have a hard time describing it but you can see
 it visually here right like how it clearly learns that these are two
 instances that are sort of occluding each other but this this this instance can
 actually appear within the bounding box of the other instance and the same goes
 for the zebra here that are partially occluding each other and you can see that
 the attention is correctly like even here that this back foot of this zebra
 is correctly labeled so all in all that is pretty cool and they take it a step
 further and they say well with this architecture we can actually pretty
 easily do pixel wise classification so this is this cocoa stuff and things data
 set where I don't know which one is the stuff and which one is the things I
 think things is the objects and stuff is like sky and mountains and so on and
 so this is a classification task where you actually have to label every single
 pixel so what they do is they simply input this through their detector and
 they detect the instances they take the attention maps of the instances and then
 they scale it up this right here is just a CNN sort of in reverse that scales up
 the image because they have scaled it down as we said they scale it up again
 and then they can simply classify each pixel where each of these you remember
 we had these different people here that it that cared about different things in
 the image each of these people will classify their respective pixels the
 pixels they feel responsible for and then you simply merge all of these
 people's predictions together into this prediction and again this gives pretty
 pretty impressive results I am I mean this is this is fun this looks like it
 sort of works haven't they do quantitative analysis of course but I'm
 just impressed by the examples right here
 alright that was sort of it I really enjoyed reading this papers the
 simplicity is pretty cool they do have not only do they have code in the paper
 to show how ridiculously easy it is to get this to run this is all you need in
 pytorch but they do actually have code and as I understand they also have
 pre-trained models so they have this model zoo right here where they give you
 the pre-trained models so you can play with it and you can even load it from
 torch hub yourself and you can train it yourself they have a collab all is there
 all right again if you enjoyed this video consider leaving a like
 subscribing and I'll see you next time bye bye