[Code] How to use Facebook's DETR object detection algorithm in Python (Full Tutorial)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper facebook fair fb facebook ai object detection coco bounding boxes hungarian matching bipartite cnn transformer attention encoder decoder images vision pixels segmentation classes stuff things attention mechanism squared unrolled overlap threshold rcnn code pytorch colab notebook ipython python torch hub torchvision bounding box image computer vision
Watch my as I struggle my way up the glorious path of using the DETR object detection model in PyTorch.

Original Video on DETR: https://youtu.be/T35ba_VXkMY

Their GitHub repo: https://github.com/facebookresearch/detr
My Colab: https://colab.research.google.com/drive/1Exoc3-A141_h8GKk-B6cJxoidJsgOZOZ?usp=sharing

OUTLINE:
0:00 - Intro
0:45 - TorchHub Model
2:00 - Getting an Image
6:00 - Image to PyTorch Tensor
7:50 - Handling Model Output
15:00 - Draw Bounding Boxes
20:10 - The Dress
22:00 - Rorschach Ink Blots
23:00 - Forcing More Predictions
28:30 - Jackson Pollock Images
32:00 - Elephant Herds

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Howdy ho, how's it going? So today we are going to try out the DETR, the end-to-end object detection with transformers from Facebook AI research. And they have a github repo and they pretty much give you everything like the model, the pre-trained weights and so on. So today we're going to check out how easy it is to get started with that. So in order to do that they have like a colab but we won't look at it too much. I've glanced at it and we'll basically see how far can we go without looking at it too much and how easy is that. So what I've done is I've spun up a colab that I will share at the end and I've imported torch and just loaded the model so you don't have to wait for that to happen. So I've loaded that up and now we have it in the cache. So now we can basically go ahead and load an image into the model and try to detect objects in the image. So first of all this is super easy, right? You simply load this from torch hub. It's kind of like the the tensorflow hub. You simply give the name of the model. You say I want the pre-trained please. Shag-a-boom! You now have a model. So if we look at that model this is going to be this entire entire DETR model right here with all the transformer and ResNet and whatnot. Okay this is almost a bit too much right here. So what we want is an image. So let's go find an image. Where better to find an image than Google? So let's find an image of dogs because dogs is one of the classes in this Coco dataset. This one's nice, right? Okay so we want the image address. We want to load it in here somehow. So let the URL is... Let's make this into some sort of like an input thing where we can paste the URL right here. Okay there we go. So we have this right here and that's the URL. All right no that's not the URL at all. Is it? Cool better. Now we need to load this. For that we're gonna use the requests library. Always a pleasure. Requests, requests. So the way to load a binary file is you can put the URL here and you can say streamed here. I glanced this from the other thing and the raw entry will get you the bytes. No, oh sorry. Get URL streamed. Stream. Yeah so this will get you the sort of the bytes of the image and then use just say image.open and of course we need the image from the pill library, the python image library. So import image. We got that and we can open that image up and with a bit of luck. Yeah yeah. So this model expects I think Coco dataset is 640 by 480 images but they if you can see right here and we're going to take a quick glance at their transforming they resize it to 800 so we're gonna we're gonna steal that part right here. People last time were some found it really funny that I called copy pasting to go serage. So we'll from now on we'll call it just seraging. What we also need are the class labels because that's in defining the Coco dataset right. So these are the class labels. Let's take those and okay so this T here these are torch vision transforms. We're gonna need that so from let's say so if you don't know torch vision it's kind of an addition to PyTorch that just helps you with with images and has a lot of datasets and these transforms they're really helpful because so let's call this image because you can you know resize but they have much more like random cropping and rotating images and so on pretty much everything you need for pre-training and this here is just the standard image net I believe the image net normalization so these are the means and these are the standard deviations so you can see that. So this is what the image net antivirus looks like. So now in terms of pans right here for example What we need are the fx 800 and I believe if you rescale the 640 to 800 you get 600 here. Fairly sure. And then let's display it just because we can. It's a bit squished but we don't care. And let's put that up here so we only need to execute it once. Nice. So from now on it should be a breeze. So what these transforms do is they resize the image. We don't need that anymore. They make it into a tensor and then they normalize by that. So if we run our image through this, because our image right now is this pill image, right? So our image is this pill image but if we run it through the transforms then we'll get a tensor. So that's pretty cool. So the model as it is a deep learning model it expects batches. So we'll unsqueeze that in the first dimension and then we get batches. So shape, let's see, we don't have unskies. No, of course we don't. So this is a one image of three channels of 600 by 800. So this is the y-index coordinates I guess are shifted. Yes, in PyTorch. Cool. So we'll call this our image tensor. Now we just need to put it into the model. So model, we put that in there. And since we don't, let's actually up here put the model in eval mode. I don't know if that's already done but you know you can never be sure enough that the batch norms aren't, but I think it probably doesn't have batch norms. Okay, you're not utilizing the GPU. We'll do that. We'll do that. Thanks. So, how do we use the GPU? We put our model on the GPU. Model equals model.cuda. Yes, yes, yes. I think so. This is gonna work. Okay. We're gonna come back to this later. So we forward our image, of course we also need that on the GPU. And this worked. This worked. This worked. Nice. Okay. And since this is just for evaluation, we should probably go with no grad right here because we don't need this whole gradient stuff if we do that. Okay. I'm dumb. There you go. And nothing happens of course because we need to capture the output somehow. Let's look at that. Output. Wow. Wow. Just wow. So the output is a dictionary, right, because we get back class labels and bounding boxes. So let's look at the pred boxes. Let's look at that tensor. That's a tensor. Very nice. Let's look at its shape. Let's not print giant tensors anymore. Cool. So since this was a batch of one, we should probably go with the zero. And you can see right here, there is a hundred bounding boxes and each one has four numbers. And if you go with the other thing that's in there, the logits, then you'll see that there are also should be a hundred logits and hello, there should be a hundred logits and each one is of size 92 because there are 92 different classes. 92. We'll see about that. Well one is going to be the nothing class, right? By the way, how many classes do we have? We have 91 classes. Okay. Cool. We can deal with that. All right. So what are we going to do next? What we want to do is for each of the logit predictions, we want to find which class it corresponds to. So what we're going to do is we're going to take the argmax of the last dimension. So you can see here, almost all of these things correspond to class 91 and class 91 is not in our classes because our class is only length 91. So that must be the nothing class. So what we can technically do is for logits and boxes in, let's just zip them together. And like this. Okay. Class is oops. Class is the logits argmax. If that's 92 or let's say, if that's larger than the length of our classes, we'll just skip it for now. Okay. So that should work somehow. And if not, then our label should be the class index right here. So let's just see what the detector detects right here. It detects nothing. Why does it detect nothing? That doesn't seem good. What are we doing wrong? We zip together the logits. Oh yeah, of course, we still need the zero with entry. We are cool. So so so so we can delete this. And now finally, beautiful dogs, two dogs detected. Excellent. So now for each of these dogs, we want the bounding box. Okay. So now we somehow need to think of how are we going to draw this on an image. And well, let's, let's actually make a copy of that image, because I don't really trust myself. And then at the end of this, we're just going to display that image, right. Now actually, the reason I make a copy is because in these in this pillow library, you can actually draw on these images. And we're going to use that to draw these bounding boxes. So for that, we need an image draw, if I remember correctly. And I think later, we also want some text. So we need an image font. Yes. All right. So let's draw a bounding box right here, where, so first of all, let's look at that bounding box. Let's call this box box, print box dot shape and break right here. What's happening? Let's not do this right now. So this is a boxes of size four. Now this could be two things. It could be x zero, y zero, x one, y one. So the two corner points are the kind of the boundaries, or it could be x, y width height. Now from the paper, I know that they predict the center and the width and the height. So I'm going to go with that, and I'm just going to guess that it's like x, y, w, h, and not some other way around. If this is a bad guess, then yeah, we'll see. We can just print out one of these boxes. And honestly, no clue that looks reason. Oh, by the way, we should scale that up. Yeah. So these are normalized coordinates, probably between zero and one. So we should scale that up. So we should probably the x coordinates, which is scale by 800 and the y by 600. So let's do it. So first of all, we scale our box by 800 in the x and here is a y and the width is the x direction and this is the y direction. Boom. Okay. We should probably get that on CPU. We'll just hack together a bunch of things right here. Okay, so now this isn't the correct. So we so our x and y and w and h are going to be this box. So now we need to actually draw on the image. We're going to do that. So let's first go x zero x one is x minus w half x plus w half y zero y one is the same for y with h plus h half. Coolio. Now we need an image draw object. So I think draw on this image. So whatever you draw on the draw object will end up on the image. So we can use that to draw a bounding box and let's just quickly look it up. So pill Python draw rectangle, maybe. There we go. Okay, so there's this rectangle. Yeah, there's the rectangle function. And you can see you put in a shape x y here and with height like this. Wait for real, we wouldn't even have to need to transform it. I'm pretty sure you can go x. I thought I remember you could do the different thing as well. But it's called rectangle. Okay, so let's do that. So draw rectangle and we'll go we'll go x zero or we'll go x, y with height. Let's display that down here. Yeah, that looks that looks nothing like we want. But it's you know, it's a start. Maybe actually we need the other thing here. We need x zero, y zero, x one, y one. Yes, yes, doggy. Okay, we still have the break in here. Now we get both dogs. Nice. Okay, let's do I think fill. Yes. Red. And let's go for width five or so. Five seems like a good width. Oh god, five is a terrible width. Oh, it's not fill. I think it's it's outline. Yeah, yeah, yeah. Okay. Okay. Let's still go with five. Cool, we got our dogs. Now we need to put like some snappy text labels. I think there is actually a pill image draw text. I think that exists because I've this font thing. Yeah, exactly. So you need the font thing. Get a font in there. And then Yeah, exactly. You could put a text like this. Okay, so you probably need the x and y coordinates of the text. So let's do that. W dot text. And let's just go with x and y right here, put it right in the middle. And the text is going to be our label, of course. And we want the fill that's now going to be the color of the text. Let's go with white and the font. We're going to load some font right here. Font dot. How are we doing this? True type, true type. Okay. Ah, no, no cheating. Let's just go with regular fonts. It won't look as fancy, but we'll be fine. So where where is our text? You see it? I don't see it. Red, let's make it red. Yes, there we go. Okay, so it wasn't red enough. This should work. Did we just not see it? I'm dumb enough. Cool. So we have two dogs. How easy was that? Actually, we wasted the most time with like bounding boxes and stuff. Absolutely cool. Right. Okay, so now we can have some fun with it. I'm going to scale this down for a bit because you don't need to see the actual code anymore so much so you can see the image more. So we'll go to the images. And the first thing I want to do is the dress. What does this think of the dress? Okay, so we'll copy that. And we'll go into our colab and just paste this right here. But a boom, but a beam sounds nice. And what is wrong? The size of a tensor must match the size of a tensor. We do something wrong. Transform image. Our image is this. Maybe this is like an RGBA image. I think if this is RGBA, we should just convert it to like an RGB. Pretty sure you can do something like this right here. This should work. If it has an alpha channel, then that will remove it. Yes, now it works. Okay, let's see what the model thinks of this. Okay, apparently there's a car and there's a surfboard and there's a person and there's a person. Nice, see? Well we didn't figure out whether the dress was blue or white or gold. It was just a person. Now you could actually like threshold by how sure you are of a given class. But where's the fun in that? So let's go further. Let's do some Rorschach inkblots, because those are always lots and lots of fun. So which one should we go for? This one looks like fun. So we'll put this into here. And it's astonishing, right? This Cocoa data set, it only has these 90 classes. Like it doesn't have anything else. So it's a cake. And this here, what is it? Okay, we'll have to go maybe with blue. What is it? Pop sign. Okay, but so you might think, what if we want more? Like what if we want more predictions? So there is a hack, right? Right now the model can always assign mass to this not a class thing, like right here, this class 91. In order for it to say, I don't think there's anything there. But generally we have 100 predictions, right? So you see where this is going. So yes, let's change it up a bit. And let's go here. Let's first extract these tensors and boxes. Okay, so we have the boxes and this and logits and boxes. Okay, so we got that. What we want to do is basically we want to filter the, we want to basically just remove the last class before we do the argmax. And thereby we want to force the model to make a prediction. It won't be a very good prediction, because of course, this is only the second highest class and it's arguable how much that counts. But still, it'll do something. So this must be done in the logits, right? So we'll look at the logits. And the logits are of shape 100. So we have 100 predictions of 92 classes. Now the first thing we want to do is just remove the last class. So let's go everything here until the last class. Alright, so now we have 91. Actually, let's make it more generic. Whatever is in however many classes are okay. So we don't have this class anymore. So now if we do the softmax over the last thing, we can technically we get 91. But now they're normalized, so they add up to one. So it's kind of a probability distribution. Next, we want to find the max over this, and that will give us a max output. So we don't want to plot all the 100 predictions, because that would just be like squares all over the place, and we'd have no clue what's happening. So this max output right here, what we're trying to find is we're trying to find a, let's say the five best predictions or so the five ones where the model thinks where the model is most confident. It's not really a good metric, but you know. So these are the probability values of all of the 100 predictions. So what we want is like the top K. Okay, so let's go with five. And again, we'll get like a top K output. Let's call that top K. And I think it also has like values and indices. Yes. So now we simply need to filter from the logits and the boxes where these top ones are. So we'll filter the logits. We'll filter the logits by that top K indices, and we'll also filter the I am not very gifted today. Boxes. By the way, I'm using a colab just because it's nice to kind of play around with a model, because if I were to use a file, I'd have to restart and reload the model over and over again. Just not as nice. Right. So now we have the logits and the boxes. And if we do that right now, we get always the top five predictions. How nice is that? And you can see the top five predictions are probably still KKKKKKK. Just to verify that. And we can put its shape. See, this is what I don't like about this stuff. Yes. Okay. So we just have five predictions of 92 things. And we don't want the 92 we've already said. So we just want the 91. Let's actually put that here. Okay. So now we have five by 91. And now it should give us the top five. Ah, there we go. So many cakes and many stop signs. That's fine. That's cool. So the ultimate test right here is going to be. Yes. The human adversarial example. Let's check it out. So we'll put in a Jackson Pollock image and we'll see what the model says. Now we're actually forcing it to make predictions, right? So it can't escape. It will need to do something. Okay, I made another mistake. I would need to copy the image address right here. Like this. That's what happens when you're not an idiot. You get the actual image. So what does the model think of our pretty image? Okay. I can't even read that. So let's make this into white. Bird. Bird. Bird. Okay. Lots of birds in this image. Clearly, clearly lots of birds in this image. Let's try another one. Let's go with this. This one. Yes. Yes. Absolutely. Love it. So we copy image address. And beam boom. More birds. Wow. There's a lot of birds in these Pollock images. Just so many birds. Okay, let's try one last. How about this one? This one is a bit more human friendly, right? Put it in here. Bang. And and. Okay, we get some detections. There's a clock right here. There is a. What's that? House? Horses? Let's print. Let's print the labels. So just so we know what they are. Cake, horse, car, horse and clock. Okay. So I see the clock. Like this here is clearly a clock. Then this rectangle on the right side must be something. Let's put this to read as well. Now that's terrible. White. Back to white. How about back to white? Okay, clock. We got horse right here and house probably. And the entire image is again a cake. Yes. Okay. So as you can see, it is a pretty, pretty good system. But of course, it is only these 90 classes. But it's for now it's a it's pretty cool. And it works pretty well and just the easiness with which you get which which you can get this stuff elephants in Kruger National Park. Just the easiness is astonishing. You can just load it up, kind of have this bit of a notebook and with a bit of like a very few lines of code, you can put something together that detects these bounding boxes. Lots of elephants. And remember, we only have the top five elephants right here. So what happens if we go for more? Where is our top K? So here we can maybe say the top 15 predictions. And as always, if we want to make the model to make its own decision, we can simply revert back and add back the no class label. All right, with that, I hope you like this video. If you did, then maybe tell YouTube that you liked it, share it out. And I will share this notebook in the description for you to find and play around with. All right, thanks for watching. Bye bye.
 Howdy ho, how's it going? So today we are going to try out the DETR, the end-to-end object detection
 with transformers from Facebook AI research. And they have a github repo and they pretty much give
 you everything like the model, the pre-trained weights and so on. So today we're going to check
 out how easy it is to get started with that. So in order to do that they have like a colab but
 we won't look at it too much. I've glanced at it and we'll basically see how far can we go
 without looking at it too much and how easy is that. So what I've done is I've spun up a colab
 that I will share at the end and I've imported torch and just loaded the model so you don't have
 to wait for that to happen. So I've loaded that up and now we have it in the cache. So now we can
 basically go ahead and load an image into the model and try to detect objects in the image. So
 first of all this is super easy, right? You simply load this from torch hub. It's kind of like the
 the tensorflow hub. You simply give the name of the model. You say I want the pre-trained please.
 Shag-a-boom! You now have a model. So if we look at that model this is going to be this entire
 entire DETR model right here with all the transformer and ResNet and whatnot. Okay this
 is almost a bit too much right here. So what we want is an image. So let's go find an image.
 Where better to find an image than Google? So let's find an image of dogs because dogs is one of the
 classes in this Coco dataset. This one's nice, right? Okay so we want the image address. We want
 to load it in here somehow. So let the URL is... Let's make this into some sort of
 like an input thing where we can paste the URL right here. Okay there we go.
 So we have this right here and that's the URL. All right no that's not the URL at all. Is it?
 Cool better. Now we need to load this. For that we're gonna use the requests library.
 Always a pleasure. Requests, requests. So the way to load a binary file is you can
 put the URL here and you can say streamed here. I glanced this from the other thing and the raw
 entry will get you the bytes. No, oh sorry. Get URL streamed. Stream. Yeah so this will get you the
 sort of the bytes of the image and then use just say image.open and of course we need the image from
 the pill library, the python image library. So import image. We got that and we can open that
 image up and with a bit of luck. Yeah yeah. So this model expects I think Coco dataset is
 640 by 480 images but they if you can see right here and we're going to take a quick glance at
 their transforming they resize it to 800 so we're gonna we're gonna steal that part right here.
 People last time were some found it really funny that I called copy pasting to go serage. So
 we'll from now on we'll call it just seraging. What we also need are the class labels because
 that's in defining the Coco dataset right. So these are the class labels. Let's take those
 and okay so this T here these are torch vision transforms. We're gonna need that so from
 let's say
 so if you don't know torch vision it's kind of an addition to PyTorch
 that just helps you with with images and has a lot of datasets and these transforms they're really
 helpful because so let's call this image because you can you know resize but they have much more
 like random cropping and rotating images and so on pretty much everything you need for pre-training
 and this here is just the standard image net I believe the image net normalization so these
 are the means and these are the standard deviations so you can see that. So this is
 what the image net antivirus looks like. So now in terms of pans right here for example
 What we need are the fx
 800 and I believe if you rescale the 640 to 800 you get 600 here.
 Fairly sure.
 And then let's display it just because we can.
 It's a bit squished but we don't care.
 And let's put that up here so we only need to execute it once.
 Nice.
 So from now on it should be a breeze.
 So what these transforms do is they resize the image.
 We don't need that anymore.
 They make it into a tensor and then they normalize by that.
 So if we run our image through this, because our image right now is this pill image, right?
 So our image is this pill image but if we run it through the transforms then we'll get
 a tensor.
 So that's pretty cool.
 So the model as it is a deep learning model it expects batches.
 So we'll unsqueeze that in the first dimension and then we get batches.
 So shape, let's see, we don't have unskies.
 No, of course we don't.
 So this is a one image of three channels of 600 by 800.
 So this is the y-index coordinates I guess are shifted.
 Yes, in PyTorch.
 Cool.
 So we'll call this our image tensor.
 Now we just need to put it into the model.
 So model, we put that in there.
 And since we don't, let's actually up here put the model in eval mode.
 I don't know if that's already done but you know you can never be sure enough that the
 batch norms aren't, but I think it probably doesn't have batch norms.
 Okay, you're not utilizing the GPU.
 We'll do that.
 We'll do that.
 Thanks.
 So, how do we use the GPU?
 We put our model on the GPU.
 Model equals model.cuda.
 Yes, yes, yes.
 I think so.
 This is gonna work.
 Okay.
 We're gonna come back to this later.
 So we forward our image, of course we also need that on the GPU.
 And this worked.
 This worked.
 This worked.
 Nice.
 Okay.
 And since this is just for evaluation, we should probably go with no grad right here
 because we don't need this whole gradient stuff if we do that.
 Okay.
 I'm dumb.
 There you go.
 And nothing happens of course because we need to capture the output somehow.
 Let's look at that.
 Output.
 Wow.
 Wow.
 Just wow.
 So the output is a dictionary, right, because we get back class labels and bounding boxes.
 So let's look at the pred boxes.
 Let's look at that tensor.
 That's a tensor.
 Very nice.
 Let's look at its shape.
 Let's not print giant tensors anymore.
 Cool.
 So since this was a batch of one, we should probably go with the zero.
 And you can see right here, there is a hundred bounding boxes and each one has four numbers.
 And if you go with the other thing that's in there, the logits, then you'll see that
 there are also should be a hundred logits and hello, there should be a hundred logits
 and each one is of size 92 because there are 92 different classes.
 92.
 We'll see about that.
 Well one is going to be the nothing class, right?
 By the way, how many classes do we have?
 We have 91 classes.
 Okay.
 Cool.
 We can deal with that.
 All right.
 So what are we going to do next?
 What we want to do is for each of the logit predictions, we want to find which class it
 corresponds to.
 So what we're going to do is we're going to take the argmax of the last dimension.
 So you can see here, almost all of these things correspond to class 91 and class 91 is not
 in our classes because our class is only length 91.
 So that must be the nothing class.
 So what we can technically do is for logits and boxes in, let's just zip them together.
 And like this.
 Okay.
 Class is oops.
 Class is the logits argmax.
 If that's 92 or let's say, if that's larger than the length of our classes, we'll just
 skip it for now.
 Okay.
 So that should work somehow.
 And if not, then our label should be the class index right here.
 So let's just see what the detector detects right here.
 It detects nothing.
 Why does it detect nothing?
 That doesn't seem good.
 What are we doing wrong?
 We zip together the logits.
 Oh yeah, of course, we still need the zero with entry.
 We are cool.
 So so so so we can delete this.
 And now finally, beautiful dogs, two dogs detected.
 Excellent.
 So now for each of these dogs, we want the bounding box.
 Okay.
 So now we somehow need to think of how are we going to draw this on an image.
 And well, let's, let's actually make a copy of that image, because I don't really trust
 myself.
 And then at the end of this, we're just going to display that image, right.
 Now actually, the reason I make a copy is because in these in this pillow library, you
 can actually draw on these images.
 And we're going to use that to draw these bounding boxes.
 So for that, we need an image draw, if I remember correctly.
 And I think later, we also want some text.
 So we need an image font.
 Yes.
 All right.
 So let's draw a bounding box right here, where, so first of all, let's look at that bounding
 box.
 Let's call this box box, print box dot shape and break right here.
 What's happening?
 Let's not do this right now.
 So this is a boxes of size four.
 Now this could be two things.
 It could be x zero, y zero, x one, y one.
 So the two corner points are the kind of the boundaries, or it could be x, y width height.
 Now from the paper, I know that they predict the center and the width and the height.
 So I'm going to go with that, and I'm just going to guess that it's like x, y, w, h,
 and not some other way around.
 If this is a bad guess, then yeah, we'll see.
 We can just print out one of these boxes.
 And honestly, no clue that looks reason.
 Oh, by the way, we should scale that up.
 Yeah.
 So these are normalized coordinates, probably between zero and one.
 So we should scale that up.
 So we should probably the x coordinates, which is scale by 800 and the y by 600.
 So let's do it.
 So first of all, we scale our box by 800 in the x and here is a y and the width is the
 x direction and this is the y direction.
 Boom.
 Okay.
 We should probably get that on CPU.
 We'll just hack together a bunch of things right here.
 Okay, so now this isn't the correct.
 So we so our x and y and w and h are going to be this box.
 So now we need to actually draw on the image.
 We're going to do that.
 So let's first go x zero x one is x minus w half x plus w half y zero y one is the same
 for y with h plus h half.
 Coolio.
 Now we need an image draw object.
 So I think draw on this image.
 So whatever you draw on the draw object will end up on the image.
 So we can use that to draw a bounding box and let's just quickly look it up.
 So pill Python draw rectangle, maybe.
 There we go.
 Okay, so there's this rectangle.
 Yeah, there's the rectangle function.
 And you can see you put in a shape x y here and with height like this.
 Wait for real, we wouldn't even have to need to transform it.
 I'm pretty sure you can go x.
 I thought I remember you could do the different thing as well.
 But it's called rectangle.
 Okay, so let's do that.
 So draw rectangle and we'll go we'll go x zero or we'll go x, y with height.
 Let's display that down here.
 Yeah, that looks that looks nothing like we want.
 But it's you know, it's a start.
 Maybe actually we need the other thing here.
 We need x zero, y zero, x one, y one.
 Yes, yes, doggy.
 Okay, we still have the break in here.
 Now we get both dogs.
 Nice.
 Okay, let's do I think fill.
 Yes.
 Red.
 And let's go for width five or so.
 Five seems like a good width.
 Oh god, five is a terrible width.
 Oh, it's not fill.
 I think it's it's outline.
 Yeah, yeah, yeah.
 Okay.
 Okay.
 Let's still go with five.
 Cool, we got our dogs.
 Now we need to put like some snappy text labels.
 I think there is actually a pill image draw text.
 I think that exists because I've this font thing.
 Yeah, exactly.
 So you need the font thing.
 Get a font in there.
 And then Yeah, exactly.
 You could put a text like this.
 Okay, so you probably need the x and y coordinates of the text.
 So let's do that.
 W dot text.
 And let's just go with x and y right here, put it right in the middle.
 And the text is going to be our label, of course.
 And we want the fill that's now going to be the color of the text.
 Let's go with white and the font.
 We're going to load some font right here.
 Font dot.
 How are we doing this?
 True type, true type.
 Okay.
 Ah, no, no cheating.
 Let's just go with regular fonts.
 It won't look as fancy, but we'll be fine.
 So where where is our text?
 You see it?
 I don't see it.
 Red, let's make it red.
 Yes, there we go.
 Okay, so it wasn't red enough.
 This should work.
 Did we just not see it?
 I'm dumb enough.
 Cool.
 So we have two dogs.
 How easy was that?
 Actually, we wasted the most time with like bounding boxes and stuff.
 Absolutely cool.
 Right.
 Okay, so now we can have some fun with it.
 I'm going to scale this down for a bit because you don't need to see the actual code anymore
 so much so you can see the image more.
 So we'll go to the images.
 And the first thing I want to do is the dress.
 What does this think of the dress?
 Okay, so we'll copy that.
 And we'll go into our colab and just paste this right here.
 But a boom, but a beam sounds nice.
 And what is wrong?
 The size of a tensor must match the size of a tensor.
 We do something wrong.
 Transform image.
 Our image is this.
 Maybe this is like an RGBA image.
 I think if this is RGBA, we should just convert it to like an RGB.
 Pretty sure you can do something like this right here.
 This should work.
 If it has an alpha channel, then that will remove it.
 Yes, now it works.
 Okay, let's see what the model thinks of this.
 Okay, apparently there's a car and there's a surfboard and there's a person and there's
 a person.
 Nice, see?
 Well we didn't figure out whether the dress was blue or white or gold.
 It was just a person.
 Now you could actually like threshold by how sure you are of a given class.
 But where's the fun in that?
 So let's go further.
 Let's do some Rorschach inkblots, because those are always lots and lots of fun.
 So which one should we go for?
 This one looks like fun.
 So we'll put this into here.
 And it's astonishing, right?
 This Cocoa data set, it only has these 90 classes.
 Like it doesn't have anything else.
 So it's a cake.
 And this here, what is it?
 Okay, we'll have to go maybe with blue.
 What is it?
 Pop sign.
 Okay, but so you might think, what if we want more?
 Like what if we want more predictions?
 So there is a hack, right?
 Right now the model can always assign mass to this not a class thing, like right here,
 this class 91.
 In order for it to say, I don't think there's anything there.
 But generally we have 100 predictions, right?
 So you see where this is going.
 So yes, let's change it up a bit.
 And let's go here.
 Let's first extract these tensors and boxes.
 Okay, so we have the boxes and this and logits and boxes.
 Okay, so we got that.
 What we want to do is basically we want to filter the, we want to basically just remove
 the last class before we do the argmax.
 And thereby we want to force the model to make a prediction.
 It won't be a very good prediction, because of course, this is only the second highest
 class and it's arguable how much that counts.
 But still, it'll do something.
 So this must be done in the logits, right?
 So we'll look at the logits.
 And the logits are of shape 100.
 So we have 100 predictions of 92 classes.
 Now the first thing we want to do is just remove the last class.
 So let's go everything here until the last class.
 Alright, so now we have 91.
 Actually, let's make it more generic.
 Whatever is in however many classes are okay.
 So we don't have this class anymore.
 So now if we do the softmax over the last thing, we can technically we get 91.
 But now they're normalized, so they add up to one.
 So it's kind of a probability distribution.
 Next, we want to find the max over this, and that will give us a max output.
 So we don't want to plot all the 100 predictions, because that would just be like squares all
 over the place, and we'd have no clue what's happening.
 So this max output right here, what we're trying to find is we're trying to find a,
 let's say the five best predictions or so the five ones where the model thinks where
 the model is most confident.
 It's not really a good metric, but you know.
 So these are the probability values of all of the 100 predictions.
 So what we want is like the top K.
 Okay, so let's go with five.
 And again, we'll get like a top K output.
 Let's call that top K.
 And I think it also has like values and indices.
 Yes.
 So now we simply need to filter from the logits and the boxes where these top ones are.
 So we'll filter the logits.
 We'll filter the logits by that top K indices, and we'll also filter the I am not very gifted
 today.
 Boxes.
 By the way, I'm using a colab just because it's nice to kind of play around with a model,
 because if I were to use a file, I'd have to restart and reload the model over and over
 again.
 Just not as nice.
 Right.
 So now we have the logits and the boxes.
 And if we do that right now, we get always the top five predictions.
 How nice is that?
 And you can see the top five predictions are probably still KKKKKKK.
 Just to verify that.
 And we can put its shape.
 See, this is what I don't like about this stuff.
 Yes.
 Okay.
 So we just have five predictions of 92 things.
 And we don't want the 92 we've already said.
 So we just want the 91.
 Let's actually put that here.
 Okay.
 So now we have five by 91.
 And now it should give us the top five.
 Ah, there we go.
 So many cakes and many stop signs.
 That's fine.
 That's cool.
 So the ultimate test right here is going to be.
 Yes.
 The human adversarial example.
 Let's check it out.
 So we'll put in a Jackson Pollock image and we'll see what the model says.
 Now we're actually forcing it to make predictions, right?
 So it can't escape.
 It will need to do something.
 Okay, I made another mistake.
 I would need to copy the image address right here.
 Like this.
 That's what happens when you're not an idiot.
 You get the actual image.
 So what does the model think of our pretty image?
 Okay.
 I can't even read that.
 So let's make this into white.
 Bird.
 Bird.
 Bird.
 Okay.
 Lots of birds in this image.
 Clearly, clearly lots of birds in this image.
 Let's try another one.
 Let's go with this.
 This one.
 Yes.
 Yes.
 Absolutely.
 Love it.
 So we copy image address.
 And beam boom.
 More birds.
 Wow.
 There's a lot of birds in these Pollock images.
 Just so many birds.
 Okay, let's try one last.
 How about this one?
 This one is a bit more human friendly, right?
 Put it in here.
 Bang.
 And and.
 Okay, we get some detections.
 There's a clock right here.
 There is a.
 What's that?
 House?
 Horses?
 Let's print.
 Let's print the labels.
 So just so we know what they are.
 Cake, horse, car, horse and clock.
 Okay.
 So I see the clock.
 Like this here is clearly a clock.
 Then this rectangle on the right side must be something.
 Let's put this to read as well.
 Now that's terrible.
 White.
 Back to white.
 How about back to white?
 Okay, clock.
 We got horse right here and house probably.
 And the entire image is again a cake.
 Yes.
 Okay.
 So as you can see, it is a pretty, pretty good system.
 But of course, it is only these 90 classes.
 But it's for now it's a it's pretty cool.
 And it works pretty well and just the easiness with which you get which which you can get
 this stuff elephants in Kruger National Park.
 Just the easiness is astonishing.
 You can just load it up, kind of have this bit of a notebook and with a bit of like a
 very few lines of code, you can put something together that detects these bounding boxes.
 Lots of elephants.
 And remember, we only have the top five elephants right here.
 So what happens if we go for more?
 Where is our top K?
 So here we can maybe say the top 15 predictions.
 And as always, if we want to make the model to make its own decision, we can simply revert
 back and add back the no class label.
 All right, with that, I hope you like this video.
 If you did, then maybe tell YouTube that you liked it, share it out.
 And I will share this notebook in the description for you to find and play around with.
 All right, thanks for watching.
 Bye bye.