CornerNet: Detecting Objects as Paired Keypoints (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper corner top left bottom right corners cv computer vision vision object detection detr bounding box center anchor pooling local cnn convolutions convolutional neural network hourglass skip connection heatmap embedding push pull loss overlap filters channels
Many object detectors focus on locating the center of the object they want to find. However, this leaves them with the secondary problem of determining the specifications of the bounding box, leading to undesirable solutions like anchor boxes. This paper directly detects the top left and the bottom right corners of objects independently, along with descriptors that allows to match the two later and form a complete bounding box. For this, a new pooling method, called corner pooling, is introduced.

OUTLINE:
0:00 - Intro & High-Level Overview
1:40 - Object Detection
2:40 - Pipeline I - Hourglass
4:00 - Heatmap & Embedding Outputs
8:40 - Heatmap Loss
10:55 - Embedding Loss
14:35 - Corner Pooling
20:40 - Experiments

Paper: https://arxiv.org/abs/1808.01244
Code: https://github.com/princeton-vl/CornerNet

Abstract:
We propose CornerNet, a new approach to object detection where we detect an object bounding box as a pair of keypoints, the top-left corner and the bottom-right corner, using a single convolution neural network. By detecting objects as paired keypoints, we eliminate the need for designing a set of anchor boxes commonly used in prior single-stage detectors. In addition to our novel formulation, we introduce corner pooling, a new type of pooling layer that helps the network better localize corners. Experiments show that CornerNet achieves a 42.2% AP on MS COCO, outperforming all existing one-stage detectors.

Authors: Hei Law, Jia Deng

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hello there! Today we're looking at corner net detecting objects as paired key points by Hylah and Jia Ding. So on a high level this paper detects objects in images. Let's say this is an image and here's a chair. You know you have your chair. And the way you detect the chair for this paper is going to be you detect the bottom right and the top left corners of the bounding box of the image. So rather than detecting the middle and then specifying height and width, like we saw in the Facebook DETR paper, you detect the two corners. And this paper goes through what they have to do to get this to work, including a new pooling method called corner pooling. So that's the gist of the paper. As always, if you like content like this, consider subscribing and sharing it out to other people. That would be very helpful. So a commenter actually recommended this paper to me after I made a video on Facebook's DETR object detection pipeline. I said something like, okay, since that paper always would detect the middle of the object and the height and width, couldn't you make something that also that detects the corners here and the corner here and then that would define a bounding box just as well. And in the comments, and thank you very much for that, I, someone made that pointed me to this paper, it's a bit older, as you can see, but it's, I still think it's, it's pretty cool. So we've already seen the the problem, like the problem isn't hard. And it's detecting bounding boxes in images. And in these data set, the problems, the difficult parts are that you sometimes have multiple objects, like here, if two humans, they can be overlapping, they can be of different sizes, that could be like a third human, like small back here, there can be other objects, you don't know how many there are, and so on. So it is it is a fairly complicated problem. But as I already said, the way that corner net here does this is by predicting the locations of the top left and bottom right corner, thereby defining a bounding box. And it does this independently. So there's one network basically, that does the top left and one that does the bottom right, and they are then combined. And at the end, they're sort of refined, I think. So the architecture is pretty simple. First, you put the image through a con net, which is like a feature extractor. So this is the basic part. It was even the basic part of Facebook's DETR pipeline. First, you have some sort of con net. Now they in this case use in this hourglass architecture that described down, down here somewhere. And this basically compresses the image into a smaller resolution. So I would take that image and compress it down to very small resolution, but many, many channels. So it's sort of forced to learn a global semantic representation, and then it up samples the image again, and down samples it again, and it up samples it again through. So at each of these steps, there are many convolutional layers right here. And because that would lose you too much space, like local information, there are skip connections built in between pairs of layers where information can travel without computation, basically. So this is a fairly standard architecture right here. But then after this hourglass CNN, you get to these prediction modules. Now let me switch back to the top drawing. Ultimately, what you want as an output of these prediction modules is two things. So first of all, you want these heat maps, sorry about that. And these heat maps will simply tell you where are the corners. Okay. Now the heat maps, their dimensions are the height of the image. Sorry, the height here, h, come on, and the width of the image. And this here would be the number of classes C. Okay, so you have one channel for each of the classes that you predict. And the heat map will basically be very high at the location and channel where there is a corner of that. So you see have one heat map for the top left corners, and one heat map for the bottom right corners. And then also what you want to predict are these embeddings. Now, simply because you have, you know, I said there can be multiple instances of the same class in the in the same image. So now you have in this case, particular case, you're gonna, even if you predict absolutely correctly, you predict two top left corners and two bottom right corners. Now this isn't particularly hard because there's only one configuration that can possibly be but there could be situations where there are multiple. And that's why you need to somehow match these corners, you have to match, you have to know which ones of those are the same objects. And they do this by a second output in their heads called this embeddings. These embeddings, they're simply vectors. And the only thing that they're asked to do is they're asked to have a large inner product, whenever they belong to the same object, and they are asked to have a small inner product, sorry, when they're when they belong to the different two different objects. So this orange thing here would have a large inner product with this green bottom right corner embedding. Okay, so you train these embeddings, they don't need to mean anything. You simply train them to predict the same thing for the same objects and different things for different objects. So after that, when you match the corners, you can simply go over you can say, ah, this which one of these two right here has the larger inner product, or you can do like some Hungarian matching and maximize the total inner product or something like this. This was quite surprising to me that it works, but it's based on a line of research that is already has already established that this can work. Because ultimately, these things, these two pipelines do not really communicate, right. So I'm going to guess what they learn is sort of a sort of a descriptor of the actual object that's there. Because if both describe the objects that that's there, with their embeddings, their embeddings are going to have a large inner product. And if they describe different objects, then their embeddings are not going to match, right. So even though you train that this objective, I still think that these embeddings would pick up something about the object, something about the visual characteristics of the objects will be very interesting to see whether someone could actually parse out what they what they do, because it's almost impossible otherwise for these things to be learnable. Alright, so that's the that's the goal right here, you want to get these heat maps in these embeddings. And the way you do it is fairly easy architecturally, you have these two prediction modules, one for top left and one for bottom right. And each of them have three outputs, the heat maps, the embeddings. And here the offsets are simply a way for you to deal with the fact that you downsample and by downsampling, you have to round certain pixels to certain locations. And then the offsets, they they compensate for this. But I don't want to focus on these right now. So you simply have these two outputs right here. Now we'll look at corner pooling in a second. But how do you train this? So you can now say, okay, if I have a picture like this, there there is exactly two locations in the class human, where the the top left corner is correct. And that's right here. And that's right here. Okay, so two locations. So I fill I make my matrix, my target matrix with a one here and the one here, and zeros everywhere else. Alright 0000000000. And I train my network to give me this particular thing as an output for these heat for this heat map in the channel human. This this might work, but it is more profitable, let's say, if you allow for some slack. So what they say is, you know, since if I'm anywhere within this orange circle right here with my prediction, my resulting bounding box is still going to overlap fairly well with the ground truth bounding box. And the accuracy measures for these things, I think, are based on how much you overlap with the ground truth bounding boxes. So what they do basically is they, they give, they put a one in the spot where the actual corner is, and then they put like a 0.9 around it 0.9 0.9, and so on, and they kind of flatten out. So this is sort of a Gaussian right here in multiple dimensions. If that drawing makes any sense. And they say, well, you the closer you are, basically, the more reward you get. So you train it to predict in this general location. Now, of course, the size, exact size of this Gaussian has to be dependent on the actual size of the box itself. And they have, they, they regard that and say exactly how they calculate these Gaussians. But for the understanding, it's just important that they do give some slack here in how they compute the loss with respect to the heat map. Now the loss with respect to the to the embeddings is pretty simple, pretty straightforward. So remember these embeddings, you have two embeddings per, you have the top left embedding, that's the ETK, the top embed, the top corner embedding, and you have the bottom right embedding. And what you want is for them to be close together when they describe the same object, right? So this is this push and pull losses. So in the pull loss, what you want to do is you want to minimize the distances of these two things to this thing right here. And this thing is simply, so EK is simply the the mean, so it's ETK plus EBK divided by two, that's simply if your top left corner is here and your bottom right corner is here, and they have embeddings, this one has this embedding, and this one has that embedding, then the mean of the two embeddings, which I guess is whatever this right here. Yeah, that's about the mean. So the location is not important, actually. So it's about the embedding vectors. It's not about where the corners are. The two embedding vectors must be close together, and you model that not directly by making them close to each other, but by making both close to their mean. And that probably saves you some back propagation troubles where you, if you have two moving parts in a loss function, and you optimize both, then you tend to, so you have two things, you want to bring them closer together, they might tend to overshoot or something like this. Okay, so this brings those two closer together. And in the push loss, what you want to do is you want to simply make the mean between the two, remember this is this is the mean, this the mean embedding of this object far away from the mean embedding of any other object in the picture. Okay, so this here is a margin loss, which means that you cap it at some point. So if they're close together, if the embeddings of two different objects are close together, you can see here this quantity will be small, and therefore it will lead to this delta, you give a loss of one, the delta here is one in this case. But as they get further apart, you're more and more happy, and you reduce your loss until you don't give, you don't give any any bonus for them being super far apart, you don't simply don't want them to be closer together than one. All right. In their case, I think they have a the dimension of these vectors is actually one, which basically means they just output the single number, which I find astonishing that that works. Yes, they use embeddings of one dimension. So they just use numbers. Astonishing that it works, but okay. So that's how you train the the embedding output embeddings close together of the same objects of the two corners, and embeddings far apart for different objects. Alright, so we can now predict where the corners are, and we can match them. Now, one center part of this is the corner pooling. And why is the corner pooling necessary? So what's the problem with this sort of approach? The problem, and they have an example right here, the problem when you want to predict a corner of an object is that in a CNN, what CNN is good at is like local neighborhood information, right? So if you have to predict, let's go for the moon, actually, here, let's predict the location of the moon, if I have to predict the location of the moon, and I'm a CNN, and I have this receptive field, I'm like, Oh, yes, it's like in here. And then I have this receptive field. And I'm like, yes, it's in here. And then I zoom in on the corner, not on the moon itself, but on the corner where I need to predict, right? At some point, I like I'm sort of, I'm like, wait, wait, where is it? Because in this particular receptive field of this resolution, I have I have no clue if the moon is close, right? So at the location where the actual bounding box is, I have no local information of the object, because usually objects are not squares, they're sort of round like the moon, or like here, the plane, these corners, they have no local information about where the plane is. And corner pooling is a method to propagate that information along the axis. So what corner in corner pooling, what you would allow the location here in the CNN to do is to not only look at its itself, so its own location, but actually to extend its field of view over to the right, and down to the bottom, it's asked to predict a top left corner. So what you do is you max pool everything from here to this corner detector. So the corner detector will basically be able to detect whenever in either this band right here. So whenever in this band right here, there is the top, like the top of an object, like the top of the moon here, this corner detector can say, ah, that's probably the right height right here for a corner. And it combines this with the information of this side here, where it also says, oh, there is the side of the moon, that's probably the correct, you know, up down. So there's probably a corner right here. Okay, whereas a location right here would get the same signal from the right, or like almost the same signal, plus this signal right here. But in essence, it would also detect the top of the moon, but it would not get the same signal from down here. And therefore it says, ah, even though to the right, I see some the top of an object, I don't see the left of an object to my bottom. So I'm not going to predict a corner right here. Alright, so this corner pooling goes for the top left, and of course, equivalently goes for the bottom right, that can always max pools to up and to the left of itself. And that's exactly what you see here. So in this corner pooling, what you can do is you can propagate signal to the left and to up, and then you add the two informations, and that will give you your output feature. And you can calculate this actually fairly efficiently by doing like, like you do a cumulative sum, you do like a cumulative maximum across the different axes, and then you simply add two arrays. And that's it. So you simply put the corner pooling before you predict the these different outputs right here, the heat maps and the embeddings, which means that this hourglass network is not affected by this. Just the predictors of heat maps and embeddings, they then get the information from this hourglass network into these into these directions. Okay, I think that's a pretty, pretty neat method of solving this. And here they show how you can calculate this. And then the corner pooling is right here, they do add a skip connection here. Because sometimes, if you just aggregate this information, you might, you might actually get confused because so the trouble of course comes when there are multiple, like different objects that have, you know, the same top. And then there's also a person right here that so it gets like a signal, it gets another signal that there is the left side of a person right here. Or maybe, you know, not like this. So it will it will predict like a corner, maybe here, where there is none. So it's, it's, sometimes it is important to have local information still. And that's exactly what this skip connection is supposed to address. I guess the situation up here would be resolved by the different embeddings. But still, so you have that you add and you put another bunch of convolutional layers on top of that, and then you'll get your predictions. And that's it. You mix all the losses. So there is a detection loss from the embeddings, there's the sorry, from the heat maps, there is the pull and push losses for the embeddings. And there's this offset loss that you train to compensate for the down the down sampling errors. And that's it. And they ablate the various things here, basically, they show that they're better than other one shot or one stage predictors. So apparently, there's one stage predictors where you have a single pass through a neural network. And there's two stage predictors where you have multiple or two passes through different neural networks. And they compete in the in the one, one stage neural network category, if so to say. And they show that they get significant improvements with and due to this corner pooling, which is pretty cool to see, because it makes sense. It sort of makes sense, how you would like to think about it like this. And to see that it helps is pretty neat. Yeah, they also investigate how large they have to make these these Gaussians and so on. And these are some qualitative examples, you can see that without the corner pooling, what you'll get is that so the top here and the left and the right are correct are detected correctly. But you can see that probably the network thinks that there is an extension of the object right here, and therefore, doesn't do doesn't do a good job. Because this this position right here, it has no access to sort of it has to use like a long range access, it can't it can't really look in detail at the features here or here. So when it scans up and down the side, where the bottom corner where the bottom break is, it can it can only look at very coarse features, because it has to basically transmit information in the CNN of a higher layer and the higher layer has a higher receptive field, which means it has a lower resolution. So it can't really go and look very in very detailed fashion at this border right here. So it misses it. Okay. Same right here, as you can see, so there are a number of failure cases that they can now solve using this method compared to if they didn't use the corner pooling. They show some also some times where their method fails, for example, here, it matches the top, top left and bottom right corners of two different objects. Because they their embeddings were close enough. And yeah, that's what I'm saying. I'm I'm wondering what these embeddings actually learn, because they are generated independently. So not entirely sure. It's also not exactly what I had in mind when I formulated this idea in the last video. But I'm actually not sure what I had in mind myself, to be honest. But in my mind, it seemed to be like you should be able to train a network. If there is an object right here, you could train a network to predict for any given location, let's say how many pixels to its bottom right, or maybe you want to normalize by the area that's there, are part of a particular object. And then you could use, you could predict each pixel and use like the differences between the differences between the points as as scores for bounding boxes. I don't know if you see what I mean. You could basically tell the you have you'd have one network predict everything to the to the bottom right, and then you'd use the differences. And the transformers would be very good at that because they can sort of have this attention between each pairs of points and so on. I'm not entirely sure, but this might just be crap. Yeah, here's some more examples. This appears to work really nicely. But of course, in the qualitative, qualitative examples, it always works nicely, but they also demonstrated. All right, I found this paper all in all pretty cool, pretty neat. It's a simple idea. It's executed well. I don't have the feeling that there are like too many tricks in here. And they show really that the improvement seems to be due to their their corner pooling method. And that's pretty neat. So if you like this paper, make sure to check it out. And I'll see you next time. Bye bye.
 Hello there! Today we're looking at corner net detecting objects as paired key points by
 Hylah and Jia Ding. So on a high level this paper detects objects in images. Let's say this is an
 image and here's a chair. You know you have your chair. And the way you detect the chair for this
 paper is going to be you detect the bottom right and the top left corners of the bounding box of
 the image. So rather than detecting the middle and then specifying height and width, like we saw in
 the Facebook DETR paper, you detect the two corners. And this paper goes through what they
 have to do to get this to work, including a new pooling method called corner pooling. So that's
 the gist of the paper. As always, if you like content like this, consider subscribing and
 sharing it out to other people. That would be very helpful. So a commenter actually recommended
 this paper to me after I made a video on Facebook's DETR object detection pipeline. I said something
 like, okay, since that paper always would detect the middle of the object and the height and width,
 couldn't you make something that also that detects the corners here and the corner here and then that
 would define a bounding box just as well. And in the comments, and thank you very much for that,
 I, someone made that pointed me to this paper, it's a bit older, as you can see, but it's, I still
 think it's, it's pretty cool. So we've already seen the the problem, like the problem isn't hard. And
 it's detecting bounding boxes in images. And in these data set, the problems, the difficult parts
 are that you sometimes have multiple objects, like here, if two humans, they can be overlapping,
 they can be of different sizes, that could be like a third human, like small back here, there can be
 other objects, you don't know how many there are, and so on. So it is it is a fairly complicated
 problem. But as I already said, the way that corner net here does this is by predicting the
 locations of the top left and bottom right corner, thereby defining a bounding box. And it does this
 independently. So there's one network basically, that does the top left and one that does the bottom
 right, and they are then combined. And at the end, they're sort of refined, I think. So the
 architecture is pretty simple. First, you put the image through a con net, which is like a feature
 extractor. So this is the basic part. It was even the basic part of Facebook's DETR pipeline. First,
 you have some sort of con net. Now they in this case use in this hourglass architecture that
 described down, down here somewhere. And this basically compresses the image into a smaller
 resolution. So I would take that image and compress it down to very small resolution,
 but many, many channels. So it's sort of forced to learn a global semantic representation,
 and then it up samples the image again, and down samples it again, and it up samples it again
 through. So at each of these steps, there are many convolutional layers right here. And because that
 would lose you too much space, like local information, there are skip connections built
 in between pairs of layers where information can travel without computation, basically. So this is
 a fairly standard architecture right here. But then after this hourglass CNN, you get to these
 prediction modules. Now let me switch back to the top drawing. Ultimately, what you want as an output
 of these prediction modules is two things. So first of all, you want these heat maps, sorry about
 that. And these heat maps will simply tell you where are the corners. Okay. Now the heat maps,
 their dimensions are the height of the image. Sorry, the height here, h, come on, and the width
 of the image. And this here would be the number of classes C. Okay, so you have one channel for each
 of the classes that you predict. And the heat map will basically be very high at the location and
 channel where there is a corner of that. So you see have one heat map for the top left corners,
 and one heat map for the bottom right corners. And then also what you want to predict are these
 embeddings. Now, simply because you have, you know, I said there can be multiple instances of the same
 class in the in the same image. So now you have in this case, particular case, you're gonna, even if
 you predict absolutely correctly, you predict two top left corners and two bottom right corners. Now
 this isn't particularly hard because there's only one configuration that can possibly be but there
 could be situations where there are multiple. And that's why you need to somehow match these
 corners, you have to match, you have to know which ones of those are the same objects. And they do
 this by a second output in their heads called this embeddings. These embeddings, they're simply
 vectors. And the only thing that they're asked to do is they're asked to have a large inner product,
 whenever they belong to the same object, and they are asked to have a small inner product, sorry,
 when they're when they belong to the different two different objects. So this orange thing here would
 have a large inner product with this green bottom right corner embedding. Okay, so you train these
 embeddings, they don't need to mean anything. You simply train them to predict the same thing for
 the same objects and different things for different objects. So after that, when you match the corners,
 you can simply go over you can say, ah, this which one of these two right here has the larger inner
 product, or you can do like some Hungarian matching and maximize the total inner product
 or something like this. This was quite surprising to me that it works, but it's based on a line of
 research that is already has already established that this can work. Because ultimately, these
 things, these two pipelines do not really communicate, right. So I'm going to guess what
 they learn is sort of a sort of a descriptor of the actual object that's there. Because if both
 describe the objects that that's there, with their embeddings, their embeddings are going to have a
 large inner product. And if they describe different objects, then their embeddings are not going to
 match, right. So even though you train that this objective, I still think that these embeddings
 would pick up something about the object, something about the visual characteristics of the objects
 will be very interesting to see whether someone could actually parse out what they what they do,
 because it's almost impossible otherwise for these things to be learnable. Alright, so that's the
 that's the goal right here, you want to get these heat maps in these embeddings. And the way you do
 it is fairly easy architecturally, you have these two prediction modules, one for top left and one
 for bottom right. And each of them have three outputs, the heat maps, the embeddings. And here
 the offsets are simply a way for you to deal with the fact that you downsample and by downsampling,
 you have to round certain pixels to certain locations. And then the offsets, they they
 compensate for this. But I don't want to focus on these right now. So you simply have these two
 outputs right here. Now we'll look at corner pooling in a second. But how do you train this? So you
 can now say, okay, if I have a picture like this, there there is exactly two locations in the class
 human, where the the top left corner is correct. And that's right here. And that's right here. Okay,
 so two locations. So I fill I make my matrix, my target matrix with a one here and the one here,
 and zeros everywhere else. Alright 0000000000. And I train my network to give me this particular
 thing as an output for these heat for this heat map in the channel human. This this might work,
 but it is more profitable, let's say, if you allow for some slack. So what they say is, you know,
 since if I'm anywhere within this orange circle right here with my prediction, my resulting bounding
 box is still going to overlap fairly well with the ground truth bounding box. And the accuracy
 measures for these things, I think, are based on how much you overlap with the ground truth bounding
 boxes. So what they do basically is they, they give, they put a one in the spot where the actual
 corner is, and then they put like a 0.9 around it 0.9 0.9, and so on, and they kind of flatten out.
 So this is sort of a Gaussian right here in multiple dimensions. If that drawing makes any
 sense. And they say, well, you the closer you are, basically, the more reward you get. So you train
 it to predict in this general location. Now, of course, the size, exact size of this Gaussian has
 to be dependent on the actual size of the box itself. And they have, they, they regard that and
 say exactly how they calculate these Gaussians. But for the understanding, it's just important
 that they do give some slack here in how they compute the loss with respect to the heat map.
 Now the loss with respect to the to the embeddings is pretty simple, pretty straightforward. So
 remember these embeddings, you have two embeddings per, you have the top left embedding, that's the
 ETK, the top embed, the top corner embedding, and you have the bottom right embedding. And what you
 want is for them to be close together when they describe the same object, right? So this is this
 push and pull losses. So in the pull loss, what you want to do is you want to minimize the distances
 of these two things to this thing right here. And this thing is simply, so EK is simply the the mean,
 so it's ETK plus EBK divided by two, that's simply if your top left corner is here and your bottom
 right corner is here, and they have embeddings, this one has this embedding, and this one has
 that embedding, then the mean of the two embeddings, which I guess is whatever this right here. Yeah,
 that's about the mean. So the location is not important, actually. So it's about the embedding
 vectors. It's not about where the corners are. The two embedding vectors must be close together,
 and you model that not directly by making them close to each other, but by making both close to
 their mean. And that probably saves you some back propagation troubles where you, if you have two
 moving parts in a loss function, and you optimize both, then you tend to, so you have two things,
 you want to bring them closer together, they might tend to overshoot or something like this. Okay,
 so this brings those two closer together. And in the push loss, what you want to do is you want to
 simply make the mean between the two, remember this is this is the mean, this the mean embedding
 of this object far away from the mean embedding of any other object in the picture. Okay, so this
 here is a margin loss, which means that you cap it at some point. So if they're close together,
 if the embeddings of two different objects are close together, you can see here this
 quantity will be small, and therefore it will lead to this delta, you give a loss of one,
 the delta here is one in this case. But as they get further apart, you're more and more happy,
 and you reduce your loss until you don't give, you don't give any any bonus for them being super far
 apart, you don't simply don't want them to be closer together than one. All right. In their case,
 I think they have a the dimension of these vectors is actually one, which basically means they just
 output the single number, which I find astonishing that that works. Yes, they use embeddings of one
 dimension. So they just use numbers. Astonishing that it works, but okay. So that's how you train
 the the embedding output embeddings close together of the same objects of the two corners,
 and embeddings far apart for different objects. Alright, so we can now predict where the corners
 are, and we can match them. Now, one center part of this is the corner pooling. And why is the
 corner pooling necessary? So what's the problem with this sort of approach? The problem, and they
 have an example right here, the problem when you want to predict a corner of an object is that in
 a CNN, what CNN is good at is like local neighborhood information, right? So if you have to
 predict, let's go for the moon, actually, here, let's predict the location of the moon, if I have
 to predict the location of the moon, and I'm a CNN, and I have this receptive field, I'm like,
 Oh, yes, it's like in here. And then I have this receptive field. And I'm like, yes, it's in here.
 And then I zoom in on the corner, not on the moon itself, but on the corner where I need to predict,
 right? At some point, I like I'm sort of, I'm like, wait, wait, where is it? Because in this
 particular receptive field of this resolution, I have I have no clue if the moon is close, right?
 So at the location where the actual bounding box is, I have no local information of the object,
 because usually objects are not squares, they're sort of round like the moon, or like here, the
 plane, these corners, they have no local information about where the plane is. And corner pooling is a
 method to propagate that information along the axis. So what corner in corner pooling, what you
 would allow the location here in the CNN to do is to not only look at its itself, so its own location,
 but actually to extend its field of view over to the right, and down to the bottom, it's asked to
 predict a top left corner. So what you do is you max pool everything from here to this corner
 detector. So the corner detector will basically be able to detect whenever in either this band
 right here. So whenever in this band right here, there is the top, like the top of an object, like
 the top of the moon here, this corner detector can say, ah, that's probably the right height right
 here for a corner. And it combines this with the information of this side here, where it also says,
 oh, there is the side of the moon, that's probably the correct, you know, up down. So there's probably
 a corner right here. Okay, whereas a location right here would get the same signal from the right,
 or like almost the same signal, plus this signal right here. But in essence, it would also detect
 the top of the moon, but it would not get the same signal from down here. And therefore it says,
 ah, even though to the right, I see some the top of an object, I don't see the left of an object to
 my bottom. So I'm not going to predict a corner right here. Alright, so this corner pooling goes
 for the top left, and of course, equivalently goes for the bottom right, that can always max pools to
 up and to the left of itself. And that's exactly what you see here. So in this corner pooling,
 what you can do is you can propagate signal to the left and to up, and then you add the two
 informations, and that will give you your output feature. And you can calculate this actually
 fairly efficiently by doing like, like you do a cumulative sum, you do like a cumulative maximum
 across the different axes, and then you simply add two arrays. And that's it. So you simply put
 the corner pooling before you predict the these different outputs right here, the heat maps and
 the embeddings, which means that this hourglass network is not affected by this. Just the
 predictors of heat maps and embeddings, they then get the information from this hourglass network
 into these into these directions. Okay, I think that's a pretty, pretty neat method of solving
 this. And here they show how you can calculate this. And then the corner pooling is right here,
 they do add a skip connection here. Because sometimes, if you just aggregate this information,
 you might, you might actually get confused because so the trouble of course comes when there are
 multiple, like different objects that have, you know, the same top. And then there's also a person
 right here that so it gets like a signal, it gets another signal that there is the left side of a
 person right here. Or maybe, you know, not like this. So it will it will predict like a corner,
 maybe here, where there is none. So it's, it's, sometimes it is important to have local information
 still. And that's exactly what this skip connection is supposed to address. I guess the situation up
 here would be resolved by the different embeddings. But still, so you have that you add and you put
 another bunch of convolutional layers on top of that, and then you'll get your predictions. And
 that's it. You mix all the losses. So there is a detection loss from the embeddings, there's the
 sorry, from the heat maps, there is the pull and push losses for the embeddings. And there's this
 offset loss that you train to compensate for the down the down sampling errors. And that's it. And
 they ablate the various things here, basically, they show that they're better than other one shot
 or one stage predictors. So apparently, there's one stage predictors where you have a single pass
 through a neural network. And there's two stage predictors where you have multiple or two passes
 through different neural networks. And they compete in the in the one, one stage neural network
 category, if so to say. And they show that they get significant improvements with and due to this
 corner pooling, which is pretty cool to see, because it makes sense. It sort of makes sense,
 how you would like to think about it like this. And to see that it helps is pretty neat. Yeah,
 they also investigate how large they have to make these these Gaussians and so on. And these are
 some qualitative examples, you can see that without the corner pooling, what you'll get is that so the
 top here and the left and the right are correct are detected correctly. But you can see that
 probably the network thinks that there is an extension of the object right here, and therefore,
 doesn't do doesn't do a good job. Because this this position right here, it has no access to sort
 of it has to use like a long range access, it can't it can't really look in detail at the features
 here or here. So when it scans up and down the side, where the bottom corner where the bottom
 break is, it can it can only look at very coarse features, because it has to basically transmit
 information in the CNN of a higher layer and the higher layer has a higher receptive field,
 which means it has a lower resolution. So it can't really go and look very in very detailed fashion
 at this border right here. So it misses it. Okay. Same right here, as you can see, so there are a
 number of failure cases that they can now solve using this method compared to if they didn't use
 the corner pooling. They show some also some times where their method fails, for example, here,
 it matches the top, top left and bottom right corners of two different objects. Because they
 their embeddings were close enough. And yeah, that's what I'm saying. I'm I'm wondering what
 these embeddings actually learn, because they are generated independently. So not entirely sure.
 It's also not exactly what I had in mind when I formulated this idea in the last video. But
 I'm actually not sure what I had in mind myself, to be honest. But in my mind, it seemed to be like
 you should be able to train a network. If there is an object right here, you could train a network
 to predict for any given location, let's say how many pixels to its bottom right, or maybe you want
 to normalize by the area that's there, are part of a particular object. And then you could use,
 you could predict each pixel and use like the differences between the differences between the
 points as as scores for bounding boxes. I don't know if you see what I mean. You could basically
 tell the you have you'd have one network predict everything to the to the bottom right, and then
 you'd use the differences. And the transformers would be very good at that because they can
 sort of have this attention between each pairs of points and so on. I'm not entirely sure,
 but this might just be crap. Yeah, here's some more examples. This appears to work really nicely.
 But of course, in the qualitative, qualitative examples, it always works nicely, but they also
 demonstrated. All right, I found this paper all in all pretty cool, pretty neat. It's a simple idea.
 It's executed well. I don't have the feeling that there are like too many tricks in here.
 And they show really that the improvement seems to be due to their their corner pooling method.
 And that's pretty neat. So if you like this paper, make sure to check it out. And I'll see you next
 time. Bye bye.