How I Read a Paper: Facebook's DETR (Video Tutorial)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper ml reading papers understanding quickly quick fast ultralearning research facebook detr object detection transformers how to
I retrace my first reading of Facebook AI's DETR paper and explain my process of understanding it.

OUTLINE:
0:00 - Introduction
1:25 - Title
4:10 - Authors
5:55 - Affiliation
7:40 - Abstract
13:50 - Pictures
20:30 - Introduction
22:00 - Related Work
24:00 - Model
30:00 - Experiments
41:50 - Conclusions & Abstract
42:40 - Final Remarks

Original Video about DETR: https://youtu.be/T35ba_VXkMY

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there people! So a lot of you have asked me how I read papers and honestly I don't think there is any super special method to it but you know I thought because people have asked me to make a video on it so I'll make a video on it and I'll try to share my method of reading papers and hopefully this is going to be somewhat of a mini series or a series where I every now and then discuss how I read one of the papers that I make videos about and I'll try to select them such that different things are highlighted. Now I've selected this one right here for really for no particular reason other than I sort of remembered it and I'm going to try to go with you through how I read this and how I encountered this and kind of try to honestly share what I thought at the first time when I read it and I hope this helps some of you. If it does help you and if you like content like this of course feel free to share this out and subscribe. If you haven't seen my original video on this paper it might be worth to go watch it. I'll link it and with that let's dive in. So again this might be not really something new but I'll just go through it. So first thing I do is of course read the title. So the title has three parts end to end object detection with transformers. So what I notice that I do myself is I like through reading a paper it's like read the paper with an open mind. I don't do that. I almost immediately form an opinion and a hypothesis of what's going on. So I see transformers so I know what transformers are. If you don't I've made a video, I've made lots of videos on transformers. Attention is all you need is the base paper for that. So I know what a transformer is. And I know that transformers are usually in NLP. They are usually used in NLP. There are things like other things with transformers but it's usually an NLP model. Then I read object detection and I know object detection is a computer vision task. So immediately this here is sort of a difference and I immediately try to assess what's the new idea in this paper. And in this case it might be applying transformers to object detection but then I also see end to end. And the only reason to put that in a title is because that's the novelty. Because usually in deep learning we're sort of used to systems being end to end. And even if most systems aren't end to end, a lot of people don't. It's like end to end image classification on ImageNet. Thanks. So I was guessing that the reason they put in end to end into the title was because that's actually something that's special about the model. So now I have like two competing hypotheses of why this paper matters. First of all because it does it with transformers and second because it does it end to end. And of course the true fear is that the combination of end to end transformers, all of that, is what makes this model. And I already form like a hypothesis of whether I like this or not. I have to be honest. I have very quick judgment of papers of whether I like them or not and then I sort of catch myself each time and I still try to... So for most papers actually that I have sort of a negative opinion at the beginning where I... Well, negative. There are papers where I think like there is no way this is going to you know work or something like this. I'm actually positively convinced throughout the paper. So for most papers that I read, I'm trying to find the positive things in there. But I do form an opinion pretty quickly usually. Alright, so the second thing. This part right here I don't even see. This is like advertisements on Twitter. I have always had issues with author names. People will come to me and be like, oh have you seen the new Vignoles paper? And I have no clue. And then when they say like, oh that's where they use this character level model to do that. And I'm like, oh that paper. So I do not care who the authors are of a paper. I can't remember the papers by their author names. I've gotten better at it I have to say. But I've always had trouble with this. Now that's not to say that a name doesn't pop out to me. If this would be like a like Joshua Benj or someone like really famous, then of course that would catch my eye. But I also know that you know, Joshua Benjo's paper, Joshua Benjo's lab is huge. So just because a big name is on the paper doesn't mean that the paper is going to be of any good or bad quality. Sometimes the authors give you an indication of what kind of research is there. Like if you see Jeff Klune or Kenneth O Stanley, you know that there's going to be this certain type of learning to explore and kind of a bit more out-of-the-box thinking in their papers, which I really like. But it doesn't immediately give you clue. Maybe if you go by first authors, it's much more indicative if you have already read some of their papers. But most often I just ignore authors and go on. The affiliation sometimes matters in that it's a bit of a vicious cycle. If there's a big name affiliation like Facebook AI, Google AI and so on, these papers also get more exposure in the press and so on. So whenever Google publishes a paper, all of these all these pop-sci magazines like Diverge and This and Lifehacker and Hacker News and whatnot, they like write a blurb about it. So often they get much more scrutinized for these papers. They get much more the public attention, but they also get much more scrutiny, which in turn means that there is a bit more pressure on them to do good experiments. So that biases me a little bit into the direction of believing their experimental evidence more. Now usually this is also backed up by the fact that I am actually convinced by their experiments. Usually these big name papers, often I find myself even without or disregarding the affiliation to be convinced more than of regular papers. My most often issue with papers is that I don't believe the experiments. I make no difference. Even if it's Facebook, my prior is the experiments are crap and I don't believe them and they have to convince me of the opposite. But I can't say that it doesn't affect me, that it's like a big-name affiliation. Okay, so then the second thing is I sometimes I see the paper on archive and I skim the abstract. Sometimes the abstract is informative and sometimes not. So here it's like blah blah blah. A new method that views object detection as a direct set prediction problem. I'm like oh yeah okay. It streamlines the detection, effectively removing the need for many hand-designed components like non-maximum suppression, yada yada yada. The main ingredients called detection transformer, a set-based global loss that forces unique prediction via bipartite matching, and the transformer encoder decoder architecture. So they make it clear here why it matters and that's what I want to get at is sort of what's the new thing in this paper. Most papers are, even though they're all very long and have lots of math and so on, they often have like one or maybe two new core things that they really tell you. Sometimes zero. But a lot of times it's like one thing that they really do and you sort of have to... But they're trying to cloak it often because they need to make their research as impactful as possible, right? But you need to sort of figure out what it is they're doing. Here they make it fairly easy for us in that they say okay. They remove the need for many hand-designed components like non-maximum suppression, which tells me that they are building something that's easier than what came before them. And that already tells me it's not necessarily going to be better. Their argument is more that it's going to be easier, right? There are sort of two kinds of experimental results. The ones where you try to beat what came before you and the ones where you're trying to say look our thing works just as well as this other thing while being more advantageous in some other metric. So I would place this already in the sort of second category. And then they say what are the actual ingredients? It's a set-based global loss that forces unique predictions via bipartite matching. Now I at this point I know what these terms mean but at this point I actually don't have to know what the terms mean. What I need to recognize is that I simply have to go later and figure out what that is. And a transformer-based encoder decoder architecture, okay? So there are two things right here that I remember I need to pay attention to later. There's this loss which seems to be special and there is the transformer architecture which they say, okay, the model basically consists of those two things. And then they have a short description of what it does. Given a fixed small set of learned object queries, there are reasons about the relations of the objects and the global image context to directly output the final set of predicted in parallel. That almost tells me nothing. Yeah, okay, the model reasons. Maybe this in parallel is something but... The model is conceptually simple and does not require specialized library unlike many other modern detectors. This sort of repeats, this enforces my hypothesis that they're going with the hey this is a much easier way of doing things approach. Dettor demonstrates accuracy and runtime performance on par with well-established that further confirms my hypothesis that this is on par, right? The runtime performance on par with the current state of the art. And at the end they say moreover, Dettor can easily be generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and preterm models are available. Okay. Now this last part when I first read it is like, okay, can easily be generalized to produce this panoptic segmentation. I didn't know yet whether this is like a central claim of their paper that it can do this segmentation or whether this is like an added benefit to their paper. Because you can read it in both ways and I'm just ready to find this out in the paper. Now after I've read the abstract and sort of already formed the hypothesis of what's going on. So here I already in my mind I already sort of have a model of how would I do that, right? How would I do that? And then what would I do? So right now what I might be thinking is if I have a transformer over images that directly outputs the predictions in parallel, I'm imagining like an image and the image somehow needs to go into a transformer. So maybe there's like an encoder, like a CNN encoder that gives me image features. And then it's so maybe you sample this down, this image. This is just me hypothesizing what could be going on, right? And then I might be unrolling that, right? This image into a vector of these lower pixels. And then so in my mind what I would do right here without knowing anything more would be to do something like BERT span prediction. So I would have BERT right here and I so for I would input the sequence right here and then to detect an object I would sort of think that maybe the BERT, you know, BERT has an output that is the same length as the input, right? So it's it's very good at sequence tagging and things like this. So maybe how it detects an object is going to be that it sort of like tags the center location in the pixel of an object right here or it tags somehow the corners of the of the bounding box. But then I don't know how this is going to be in parallel. Maybe BERT outputs like a score for each location and then you do some kind of matching right here. So this is my initial hypothesis of what's going on. And then I scroll through and honestly the first thing I do is I go and find the pictures. And no no different in all like since since your first book you read that's what you do. I go and find the pictures because usually if someone proposes anything new that they're gonna try to make a picture of it. Luckily I don't do like super theoretical what not your Bayesian generalization bounds and I don't know. So most often papers I read have some sort of picture and that's very awful to me. I know, I know, but yeah. So I find this picture and here I see okay you have image, you have CNN okay gives you set of image features so so far so good. Then transformer encoder decoder then set of box predictions so all of them come out here and I already read they're in parallel and then bipartite matching loss. So here they I can see they color these in different ways and these color appear to match with these colors right here right in the green here and these they they also this is a very good graphic right from this I can already read that these here go to the no object. A lot of times the graphics aren't very good so this this is what I'm not saying in every paper you can learn by looking at the graphics like sometimes the graphics are terrible and you're like what's going on here I like I don't this this makes no sense. This happens a lot in this paper right here this happens to be very very good explanatory graphics so I'll take advantage of that and I do the same thing in the other papers right but then later when it doesn't match what I read in the text I'll have to you know update my belief and so on but here I see that these go to no object and this goes to no object so I don't know yet that this is the test set at the point where I read this I was sort of confused by this but I recognized that each of these boxes right here is going to be either resulting in a bounding box or in the no object prediction so from that I could conclude that these things here are maybe some sort of a fixed set right but I still thought that you know these that this would actually be the output of these image features so that in this case you'd have like six set of image features and then you'd have like BERT here even though that's not an encoder decoder I still this was still my running hypothesis that somehow you'd map these image features to these boxes right here so and I didn't know what to what to make of this this thing right here so then I went through some more and look for more pictures and there are not sometimes I also kind of glance at the formulas but okay when I ever I see this this is just I mean this is kind of useless like okay cool you minimize the loss thanks this okay didn't really pay attention to that ah new picture cool so this picture is much more informative than the other picture now I believe with the other picture they were trying to showcase this loss how they do the matching and even though I could read a lot from that picture I did not get that part and therefore I felt when I saw this and I just glanced at it I'm like wait what's what's different than up here it seems like the same but okay let's look at this so again we see okay you have set of image features that comes out of the CNN so that conforms with my belief but then this here goes into a transformer encoder and this comes out so immediately I see oh this is not the same as these boxes here right that was my hypothesis that these things here would be the colored boxes so I I say okay obviously that's not what happens this thing here it seems to be sort of the encoded image information then that's somehow fed into here and that then there are these object query things and they seem to correspond to this so I'm a bit more confused right now what I can see is that these then will result in these in these boxes okay so being confused by that I look for more pictures so I go look for more pictures and this here seems to be like of a visualization a lot of these papers have some sort of ablation experiments or so and so on this I just find really cool picture for now I don't know yet what it means this I don't know yet what it means and I go down skip all of this and then back here in the appendix I find this here which I immediately mapped to the previous where this is the end and this is a decoder and I've already read the attention is all you need paper and that that point it clicked in me is like ah this is not a BERT transformer this is one of these transformers that has an encoder in the decoder even though they told me like 50 billion times already I was too stupid until this point so now I know okay okay I see what's going on so the image goes through here and then this goes as a side input like as an attention from the decoder to the encoder like I know in NLP right so in NLP this here would be a source sequence like maybe if you do translation and this here would be a target sequence so now whenever I see a transformer like this and it outputs something at this I I look at it as okay this here is sort of the input that goes as like a side input over here and usually here you have the target sequence but that's not the case right here right you have these these object queries so this is how far I get from the pictures now I go up so I have a sort of I have questions now I have questions and that's when I start reading the paper only now do I start reading the paper after I've looked through all the images form the hypothesis and sort of have questions on how this works and we'll go a bit faster from now on to just not bore you with all the things so the introduction is often very important even though it's called introduction and maybe you know if you read a book like if there's like introduction or prologue or something like this it's often kind of pointless introduction in these research papers is one of the most important points because all of these papers they try basically all of them try to convince a reviewer to accept them and in order to do that they will set up their main points and their main story immediately in the introduction so what you'll usually have is a problem statement which is here like why what's what's wrong right now and then you have like a story of how their paper addresses the issue okay and that's that's here we streamline the training pipeline by viewing object prediction the other yada yada this is often formulates in words what the paper is about and what contribution the paper makes right this is like a this is like a longer abstract the abstract is often very very cryptic very dense this here is often much more informative of what the paper does so for understanding the paper and a high level the introduction is the best place but given that I've already looked at the images and so on I don't actually draw many new much new information from this thing then there's related work and honestly I I skip it like unless I'm the actual reviewer of a paper like when I'm the reviewer of a paper I read the related work but often the related work is just like you first of all you cite a bunch of your friends and then you cite the mandatory papers and then you cite every single person that you think could be a reviewer because or you've actually been rejected from a conference with a reviewer claiming that you're you haven't compared or you haven't cited that or that paper you can pretty much be sure that that's the if if it's not a glaring of omission if it's like a niche paper and you haven't cited it then you're like okay I'm gonna cite it just because the next conference you could be my reviewer again so I'm not I'm not sure that these related work sections they're necessary like if someone wants to write their thesis and they go and read this paper and they want references oftentimes this is a good place but a lot of it is just blah blah blah blah blah okay I know I know disagree with me if you want oh yeah to maybe to reading quality so I tend to at this point I tend to not skim so at first I skim but at this point I tend to read every sentence and read it closely and understand it and when I realized like I'm tired or something I don't just skim the paper I've tried to skim papers and it doesn't doesn't work try to read every sentence understand every sentence and okay if you don't understand it don't stop reading because of that but try to not skim and be like oh yeah yeah yeah okay I gotta go to go to go to go that's is not helpful except related work skip completely cool then a lot of times in this paper now is the the model and this is the section I'm actually interested in right so I read very very closely here and then I find out what their their loss is all about and again I stress read these things and understand them right sometimes it's hard but if you're if you're confused that means you either they've done a bad job or they made a mistake or that you haven't understood something if you can't understand the sentence try to read on maybe it's clarified later and then you know go back but again do not do not like just start a lot of times when I read paper previously like I wouldn't understand something quite well yet and then I would be like oh yeah yeah yeah and then I noticed that I start skipping and skimming more and more because that would you know pop up again and again and I wouldn't understand it again and again and then at the end I would just be kind of glancing at the paper and I don't want to do that right here so I want to read every sentence and understand it okay so here then I find out about the loss and then I if I don't know something here then I'll go and look it up on maybe on Wikipedia or something like this now I don't need to on actually I don't need to understand every single part of it right that's maybe I should correct myself so for example this bounding box loss here they talk about the second part of the max and question Hungarian possible is this box loss that scores bounding boxes unlike many detectors that do box prediction with some initiality yada yada yada they say the most commonly used L1 loss will have different scales for a small so here they basically talk about how they mix the losses they see overall our box losses that defined as this and this now I haven't I don't know what these losses are I just assume there's some bounding box losses so when I it's not true when I say understand everything understand the things that are integral to the story of the paper right how exactly they compute bounding box losses at this point I don't care I just assume that there's some loss that I can back propagate right I what is important is that they do this Hungarian matching thing right as soon as I get that I'm like ah that was this you know this this thing no this thing up here this thing this with the matching thing now I get it now I know there are always the same amount of boxes here and there are always the same amount of labels here and all we need to do is somehow match them and I immediately think why is that relevant oh because when something is already matched to an object some other thing cannot be matched to the same object and that's how we you know prevent the fact that all the things predict the same thing right and so that immediately becomes clear and as I said there is usually like one or two ideas in a paper I don't assume or I don't care what their exact loss function is because I've sort of gotten the idea up here of what the loss is about all right so I hope that's clear under very closely read the things and understand the things that are necessary for the story if you find if you think something is not necessary for the story and then later end up not understanding that maybe come back and you know read it again in any case I would I would rather I would rather skip something and assume it's not necessary if I think so and then come back then trying to understand every everything but the things I do read I try to understand thoroughly okay then there's the architecture okay and that again I read closely and get backbone okay transformer encoder okay and now I understand much more closely decoder okay and here I get now finally I get what this is about decodes and objects in parallel yada yada yada these input embeddings are learned positional encodings that we refer to as object queries and similarly to the encoder we add them to the input at each attention layer so now they name I've already seen these object queries here and the only word I actually need from this sentence are learned the fact that they're positional encodings I just kind of ignore as soon as they say learned I know aha these things here are learned they have actually they're always the same for each of the images they're just overall learned okay so now I feel I understand the entire model and yeah so they then they say auxiliary decoding losses and this sometimes you have to pay attention to like auxiliary auxiliary things because those are the the things that here they say explicitly we found helpful to use auxiliary losses sometimes they they won't say why they did it they'll just say our loss consists of three things and you know if you look at the three things only one of the things is really a part of their story so far and that you should immediately conclude that they've put in the other things because they tried it and it didn't work right so you can also kind of get an estimate of the brittleness and so on of the system in that you see how many unnecessary things are there or how many things are not straightforward how many things are the easiest thing that you would do when you would go about and do what they did okay so then you this concludes this model or method usually this section is called like method or model or something like this and you go to experiments now the main question I have so far or I have maybe I have some more questions about the model itself that I haven't been able to pick up from this section which is not the case here but I simply keep those questions in mind and see whether they are resolved later right so I keep an awareness of what I don't understand but from here on my main issue is are they demonstrating that their story works right so they're here they're they're proposing a loss and a model and in my mind they now need to convince me that that works and that's that's it's not as easy as simply to show me some numbers that they are good at some benchmark they need to show me that they get those numbers because of what they claim so here they claim well okay they propose a new they propose a new architecture so what they need to convince me of is that the architecture itself makes sense right but in other papers when when you propose like and when you say like oh we for example in an LSTM when you build in an attention mechanism and you claim oh we you know the attention mechanism can look back at the source sequence in one step then you need to convince me that that actually happens right so you need to not only do you need to perform well you need to convince me that you perform well because of what you claim your model does right so and that's often difficult and I specifically look out in the experiments for usually the question is like where are they trying to bullshit me right where are they trying to are or are they trying to bullshit me are they trying to cover up the fact that something doesn't work and all the experiments are always in the best light possible of course and you have to keep that in mind but a lot of times you can also already see from the experiments that okay are they doing something weird are they not showing me some obvious experiment or and that's a lot of times I guess is there an easier explanation for why they get the results that they get other than their explanation right and it is it is their job to convince you that their explanation is the correct one for these numbers and especially if there is an easier one that they haven't excluded and then I don't believe the experiments if that's the case right if there is an easier explanation for the effect I'm I'm very skeptical but some papers have an easier job here than other papers so in this paper they basically show results on a on a on a task and since their paper is about hey our pipeline is just easier than other pipelines what they first of all need to do is they just need to like match the numbers of other pipelines and here I see that okay in these results often you have maybe a table or something here you see like this their model other models and their model is the best model in a lot of cases now if the best thing is of course if their model throughout is the best the worst thing is if it's like scattered like this even if their model is the best but in every single benchmark a different configuration of their model is the best that's that's sort of a bad sign unless they can explicitly explain why that is and it's also not that good of a sign if these things are spread out like this like sometimes this baseline is good sometimes their model is better and so on so pay attention to that now in this paper it doesn't matter so much that's actually fine because what they're trying to show is that their model is on par and way easier and they've already made the case in what way it is easier it's easier in terms of architecture if they were to say it's much faster then after that I would expect you know an experiment in speed while these numbers are matched so but since they say it's it's easier I've already seen the architecture I'm convinced of that now that they show okay our numbers match or actually I'm surprised they even outperform a lot of times then I'm quite happy with these experiments so also look for differences between numbers and the spread of numbers now it's not easy to say what if like point one is a big or a small difference that depends on the task but if you know pay attention to these things pay attention to the fact that these results are noisy and oftentimes there is a lot more hyper parameter tuning going into the model of the paper then into the baseline models right you want to make your look your stuff look as good as possible and here is a little bit where the institutional credibility of someone like Facebook comes in in that I tend to believe their results a bit more than other results not mega but a bit more yeah also look at patterns that they don't point out in the text so if there is like a pattern if you see like an interaction between the number of parameters and the score or something like this just try to be on the lookout of that and see if you can spot something that you think or think about whether that makes sense or not in what your hypothesis would be so here we go on and okay then they go into ablations and a lot of a lot of these papers do ablations and I generally appreciate that so here they visualize that the attention mechanism in their model actually refers to different instances right encoder self-attentions for a set of reference points the encoder is able to separate individual instances and you can see that pretty clearly right here where and even here with the overlapping cows and this is the sort of experiment that I would expect that actually convinces me that their architecture does what it says that it does right and something like this where you see like totally overlapping things with the attention of the individual things visualized so telling me like especially this one right here the the foot of the back elephant actually being focused by the attention of the bounding box of the back elephant that's the sort of experiment that convinces me that their claims like that their numbers really come from what they claim it comes from okay so at the end of the experimental section you should always ask yourself have they really convinced me that their story is true right that the improvement or when egg whenever they get an improvement or whatever they get what is is due to the story that they want to sell me or could there be like an easier explanation or does something not fit is like are there are the experiments different than from what you would expect here okay so these are these are my main questions are they are they convincing me of their story it's not do they have state-of-the-art numbers I don't care I don't care even though like sometimes so there is a bit of a catch I I don't care about state of the art numbers now let's say you have a table like this and you have a computer vision model and one of the models is like on the C for 10 data set now if your baseline model has like a 91 92 percent accuracy on C for 10 when I know the state-of-the-art is 96 I don't care right I know like I've done C for 10 I know with like I don't know five six layers CNN you can reach these 91 92 93 percent accuracy and to get to the 96 97 you'd actually be like in the region of a wide resinette and whatnot so it I I know that even though you're a few points behind state-of-the-art I know you know this this is valid still so I don't care but if you were to be like at 80 percent accuracy on C for 10 then I then I get a bit like hmm I like it's pretty easy to get to 90 percent plus with like a standard CNN so there I immediately start to wonder why is there an explanation now this could be like a theoretical paper that says oh we investigate MLPs and that's why we only get that number so that's that would be fine but if something is out of the ordinary like this then I pay attention but never because something isn't like the latest and greatest state-of-the-art that's just dumb okay and also if only evaluate what the paper claims it does right if the paper says we want to show that we are on par with current models then don't be mad if the paper doesn't outperform these models they didn't claim that right so yeah so after these ablations I'm actually pretty happy right here with the results and this right here when I saw this I didn't I didn't expect that but I read the experiment description that these are these different learned object queries and what they do and that gave me an increased understanding of how these object queries actually work right so at that point I still had like a vague I knew that these are learned but reading this and sort of looking at it studying it a bit I was like oh okay then I understood even better what they are so again when I say understand everything in the method section you can still have questions and but you just have to keep it in mind for later and then here I go on and there's this DETR for panoptic segmentation and they here they propose like a new model so I first look at it and I'm like okay they propose a new model they can do stuff like this now this is not object detection and again I'm not sure is this like a is this like a an add-on to the method or is was was this up here just an intermediate step to this and honestly after reading that I still wasn't sure it seems like something in between of course the paper is also a bit longer than other papers it just it seems it's too long for just being a side note but it's too short for being its own thing so that was just a bit weird and I treated is as just like a oh we can also do this with our model but I didn't pay like too much attention to that okay so at the end I you know look at conclusions now the conclusions of a paper are much much often they are not nearly as informative as the introduction the conclusions they all often tend to be very generic and kind of hedging a bit against criticisms saying what would be up for future work which is again hedging against criticism because you simply say well we didn't do this that's future work yes so again I read it but I don't really pay attention to it and then I gloss over the abstract I just would kind of scroll through the abstract if there's something that catches my eye I would look at it and if not then not and then I basically go to the start and whenever I didn't understand something I go back I look at it again and I try to think are all my questions answered and have they sufficiently convinced me that their story is the thing that really has the effect right here and then if I now were to make a video of this I've often found it useful to just put the paper away for a while and it's I usually get the best results when I read the paper the day before and then make a video the day after or if not I'll just you know put it away do something else do some email responding programming going outside eating lunch just some kind of a break between first read or between your first couple of reads and riff just I don't even think about the paper I just kind of it's just in the subconscious it kind of brews right and I happen to think about the paper every now and then but I don't make a conscious effort to be like oh how am I gonna explain this and so on but I just found the the worst videos are the ones where I immediately make the video after reading a paper and I've just discovered that if I kind of take a break and then I look at it again right I look I don't read it fully again but I if I have if I have the feeling I've understood it I don't read it fully again but I just kind of look at it and go again through the story and I think that's even if you you know want to if you want to talk about a paper in a reading group or tell you know explain it to your friends or whatnot this is often very useful just put it away for a while let it mellow and I find that helps a lot okay that was my process of reading this particular paper now we again this this is a high quality paper so it's I find it's a pretty easy read in that I simply need to understand what they did and I'm pretty happy with their experiments I maybe next time I can find an experiment or a paper where I'm initially more skeptical and not as happy with what I find but yeah let me know if you enjoyed this or if you would like to see any other explanation I don't exactly know if this is what you expected from a video like this so let me know maybe I have misunderstood you completely or it's way too long way too detailed or way too undetailed yeah leave me a comment and I'll see you next time bye bye
 Hi there people! So a lot of you have asked me how I read papers and honestly
 I don't think there is any super special method to it but you know I thought
 because people have asked me to make a video on it so I'll make a video on it
 and I'll try to share my method of reading papers and hopefully this is
 going to be somewhat of a mini series or a series where I every now and then
 discuss how I read one of the papers that I make videos about and I'll try to
 select them such that different things are highlighted. Now I've selected this
 one right here for really for no particular reason other than I sort of
 remembered it and I'm going to try to go with you through how I read this and how
 I encountered this and kind of try to honestly share what I thought at the
 first time when I read it and I hope this helps some of you. If it does help
 you and if you like content like this of course feel free to share this out and
 subscribe. If you haven't seen my original video on this paper it might
 be worth to go watch it. I'll link it and with that let's dive in. So again this
 might be not really something new but I'll just go through it. So first
 thing I do is of course read the title. So the title has three parts end to end
 object detection with transformers. So what I notice that I do myself is I like
 through reading a paper it's like read the paper with an open mind. I don't do
 that. I almost immediately form an opinion and a hypothesis of what's going
 on. So I see transformers so I know what transformers are. If you don't I've
 made a video, I've made lots of videos on transformers. Attention is all you need
 is the base paper for that. So I know what a transformer is. And I know
 that transformers are usually in NLP. They are usually used in NLP. There are
 things like other things with transformers but it's usually an NLP
 model. Then I read object detection and I know object detection is a computer
 vision task. So immediately this here is sort of a difference and I immediately
 try to assess what's the new idea in this paper. And in this case it might be
 applying transformers to object
 detection but then I also see end to end. And the only reason to put that in a
 title is because that's the novelty. Because usually in deep learning we're
 sort of used to systems being end to end. And even if most systems
 aren't end to end, a lot of people don't. It's like end to end image classification
 on ImageNet. Thanks. So I was guessing that the reason they put in end to
 end into the title was because that's actually something that's special about
 the model. So now I have like two competing hypotheses of why this paper
 matters. First of all because it does it with transformers and second because it
 does it end to end. And of course the true fear is that the combination of
 end to end transformers, all of that, is what makes this model. And I already
 form like a hypothesis of whether I like this or not. I have to be honest.
 I have very quick judgment of papers of whether I like them or not and then I
 sort of catch myself each time and I still try to... So for most papers
 actually that I have sort of a negative opinion at the beginning where I...
 Well, negative. There are papers where I think like there is no way this is going to
 you know work or something like this. I'm actually positively convinced
 throughout the paper. So for most papers that I read, I'm trying
 to find the positive things in there. But I do form an opinion pretty quickly
 usually. Alright, so the second thing. This part right here I don't even
 see. This is like advertisements on Twitter. I have always had
 issues with author names. People will come to me and be like, oh have you seen
 the new Vignoles paper? And I have no clue. And then when they say like, oh that's
 where they use this character level model to do that. And I'm like, oh that
 paper. So I do not care who the authors are of a paper. I
 can't remember the papers by their author names. I've gotten better at it I
 have to say. But I've always had trouble with this. Now that's not to say that a
 name doesn't pop out to me. If this would be like a like Joshua Benj or
 someone like really famous, then of course that would catch my eye. But I
 also know that you know, Joshua Benjo's paper, Joshua Benjo's lab is huge. So just
 because a big name is on the paper doesn't mean that the paper is going to
 be of any good or bad quality. Sometimes the authors give you an indication of
 what kind of research is there. Like if you see Jeff Klune or Kenneth O
 Stanley, you know that there's going to be this certain type of
 learning to explore and kind of a bit more out-of-the-box thinking in
 their papers, which I really like. But it doesn't immediately give you clue. Maybe
 if you go by first authors, it's much more indicative if you have already read
 some of their papers. But most often I just ignore authors and go on. The
 affiliation sometimes matters in that it's a bit of a vicious cycle. If
 there's a big name affiliation like Facebook AI, Google AI and so on, these
 papers also get more exposure in the press and so on. So whenever
 Google publishes a paper, all of these all these pop-sci magazines like
 Diverge and This and Lifehacker and Hacker News and whatnot, they like
 write a blurb about it. So often they get much more scrutinized for these papers.
 They get much more the public attention, but they also get
 much more scrutiny, which in turn means that there is a bit more pressure on
 them to do good experiments. So that biases me a little bit into the
 direction of believing their experimental evidence more. Now usually
 this is also backed up by the fact that I am actually convinced by their
 experiments. Usually these big name papers, often I find myself
 even without or disregarding the affiliation to be convinced more than of
 regular papers. My most often issue with papers is that I don't believe
 the experiments. I make no difference. Even if it's Facebook, my
 prior is the experiments are crap and I don't believe them and they have to
 convince me of the opposite. But I can't say that it doesn't affect me,
 that it's like a big-name affiliation. Okay, so then the second thing is I
 sometimes I see the paper on archive and I skim the abstract. Sometimes the
 abstract is informative and sometimes not. So here it's like blah blah blah. A
 new method that views object detection as a direct set prediction problem. I'm
 like oh yeah okay. It streamlines the detection, effectively removing the need
 for many hand-designed components like non-maximum suppression, yada yada yada.
 The main ingredients called detection transformer, a set-based global loss that
 forces unique prediction via bipartite matching, and the transformer encoder
 decoder architecture. So they make it clear here why it matters and that's
 what I want to get at is sort of what's the new thing in this
 paper. Most papers are, even though they're all very long and have lots of
 math and so on, they often have like one or maybe two new core things that they
 really tell you. Sometimes zero. But a lot of times it's like one thing that they
 really do and you sort of have to... But they're trying to cloak it often
 because they need to make their research as impactful as possible, right? But you
 need to sort of figure out what it is they're doing. Here they make it fairly
 easy for us in that they say okay. They remove the need for many hand-designed
 components like non-maximum suppression, which tells me that they are building
 something that's easier than what came before them. And that already tells me
 it's not necessarily going to be better. Their argument is more that it's going
 to be easier, right? There are sort of two kinds of experimental results. The
 ones where you try to beat what came before you and the ones where you're
 trying to say look our thing works just as well as this other thing while being
 more advantageous in some other metric. So I would place this already in the
 sort of second category. And then they say what are the actual ingredients? It's
 a set-based global loss that forces unique predictions via bipartite
 matching. Now I at this point I know what these terms mean but at this point I
 actually don't have to know what the terms mean. What I need to recognize is
 that I simply have to go later and figure out what that is. And a
 transformer-based encoder decoder architecture, okay? So there are two
 things right here that I remember I need to pay attention to later. There's this
 loss which seems to be special and there is the transformer architecture which
 they say, okay, the model basically consists of
 those two things. And then they have a short description of what it does. Given
 a fixed small set of learned object queries, there are reasons about the relations
 of the objects and the global image context to directly output the final set
 of predicted in parallel. That almost tells me nothing. Yeah, okay, the model
 reasons. Maybe this in parallel is something but... The model is conceptually
 simple and does not require specialized library unlike many other modern
 detectors. This sort of repeats, this enforces my hypothesis that they're
 going with the hey this is a much easier way of doing things approach. Dettor
 demonstrates accuracy and runtime performance on par with well-established
 that further confirms my hypothesis that this is on par, right? The runtime
 performance on par with the current state of the art. And at the end they say
 moreover, Dettor can easily be generalized to produce panoptic
 segmentation in a unified manner. We show that it significantly outperforms
 competitive baselines. Training code and preterm models are available. Okay. Now
 this last part when I first read it is like, okay, can easily be generalized to
 produce this panoptic segmentation. I didn't know yet whether this is
 like a central claim of their paper that it can do this segmentation or whether
 this is like an added benefit to their paper. Because you can read it in both
 ways and I'm just ready to find this out in the paper. Now after I've read the
 abstract and sort of already formed the hypothesis of what's going on. So here I
 already in my mind I already sort of have a model of how would I do that, right?
 How would I do that? And then what would I do? So right now what I
 might be thinking is if I have a transformer over images that directly
 outputs the predictions in parallel, I'm imagining like an image and the image
 somehow needs to go into a transformer. So maybe there's like an encoder, like a
 CNN encoder that gives me image features. And then it's so maybe you sample
 this down, this image. This is just me hypothesizing what could be going on,
 right? And then I might be unrolling that, right? This image into a vector of these
 lower pixels. And then so in my mind what I would do right here without knowing
 anything more would be to do something like BERT span prediction. So I would
 have BERT right here and I so for I would input the sequence right here and
 then to detect an object I would sort of think that maybe the BERT, you know, BERT
 has an output that is the same length as the input, right? So it's it's very good
 at sequence tagging and things like this. So maybe how it detects an object is
 going to be that it sort of like tags the center location in the pixel of
 an object right here or it tags somehow the corners of the of the bounding box.
 But then I don't know how this is going to be in parallel. Maybe BERT outputs
 like a score for each location and then you do some kind of matching right here.
 So this is my initial hypothesis of what's going on. And then I scroll
 through and honestly the first thing I do is I go and find the pictures. And no
 no different in all like since since your first book you read that's what you do. I
 go and find the pictures because usually if someone proposes anything new that
 they're gonna try to make a picture of it. Luckily I don't do like super
 theoretical what not your Bayesian generalization bounds and I don't know.
 So most often papers I read have some sort of picture and that's very awful to
 me. I know, I know, but yeah. So I find this picture and here I see okay you have
 image, you have CNN okay gives you set of image features so so far so good. Then
 transformer encoder decoder then set of box predictions so all of them come out
 here and I already read they're in parallel and then bipartite matching
 loss. So here they I can see they color these in different ways and these color
 appear to match with these colors right here right in the green here and these
 they they also this is a very good graphic right from this I can already
 read that these here go to the no object. A lot of times the graphics aren't very
 good so this this is what I'm not saying in every paper you can learn by looking
 at the graphics like sometimes the graphics are terrible and you're like
 what's going on here I like I don't this this makes no sense. This happens a lot
 in this paper right here this happens to be very very good explanatory graphics so
 I'll take advantage of that and I do the same thing in the other papers right but
 then later when it doesn't match what I read in the text I'll have to you know
 update my belief and so on but here I see that these go to no object and this
 goes to no object so I don't know yet that this is the test set at the point
 where I read this I was sort of confused by this but I recognized that each of
 these boxes right here is going to be either resulting in a bounding box or in
 the no object prediction so from that I could conclude that these things here are
 maybe some sort of a fixed set right but I still thought that you know these that
 this would actually be the output of these image features so that in this
 case you'd have like six set of image features and then you'd have like BERT
 here even though that's not an encoder decoder I still this was still my
 running hypothesis that somehow you'd map these image features to these boxes
 right here so and I didn't know what to what to make of this this thing right
 here so then I went through some more and look for more pictures and there are
 not sometimes I also kind of glance at the formulas but okay when I ever I see
 this this is just I mean this is kind of useless like okay cool you minimize the
 loss thanks this okay didn't really pay attention to that ah new picture cool so
 this picture is much more informative than the other picture now I believe
 with the other picture they were trying to showcase this loss how they do the
 matching and even though I could read a lot from that picture I did not get that
 part and therefore I felt when I saw this and I just glanced at it I'm like
 wait what's what's different than up here it seems like the same but okay
 let's look at this so again we see okay you have set of image features that
 comes out of the CNN so that conforms with my belief but then this here goes
 into a transformer encoder and this comes out so immediately I see oh this
 is not the same as these boxes here right that was my hypothesis that these
 things here would be the colored boxes so I I say okay obviously that's not what
 happens this thing here it seems to be sort of the encoded image information
 then that's somehow fed into here and that then there are these object query
 things and they seem to correspond to this so I'm a bit more confused right
 now what I can see is that these then will result in these in these boxes okay
 so being confused by that I look for more pictures so I go look for more
 pictures and this here seems to be like of a visualization a lot of these papers
 have some sort of ablation experiments or so and so on this I just find really
 cool picture for now I don't know yet what it means this I don't know yet what
 it means and I go down skip all of this and then back here in the appendix I
 find this here which I immediately mapped to the previous where this is the
 end and this is a decoder and I've already read the attention is all you
 need paper and that that point it clicked in me is like ah this is not a
 BERT transformer this is one of these transformers that has an encoder in the
 decoder even though they told me like 50 billion times already I was too stupid
 until this point so now I know okay okay I see what's going on so the image goes
 through here and then this goes as a side input like as an attention from the
 decoder to the encoder like I know in NLP right so in NLP this here would be a
 source sequence like maybe if you do translation and this here would be a
 target sequence so now whenever I see a transformer like this and it outputs
 something at this I I look at it as okay this here is sort of the input that goes
 as like a side input over here and usually here you have the target
 sequence but that's not the case right here right you have these these object
 queries so this is how far I get from the pictures now I go up so I have a
 sort of I have questions now I have questions and that's when I start
 reading the paper only now do I start reading the paper after I've looked
 through all the images form the hypothesis and sort of have questions on
 how this works and we'll go a bit faster from now on to just not bore you with
 all the things so the introduction is often very important even though it's
 called introduction and maybe you know if you read a book like if there's like
 introduction or prologue or something like this it's often kind of pointless
 introduction in these research papers is one of the most important points
 because all of these papers they try basically all of them try to convince a
 reviewer to accept them and in order to do that they will set up their main
 points and their main story immediately in the introduction so what you'll
 usually have is a problem statement which is here like why what's what's
 wrong right now and then you have like a story of how their paper addresses the
 issue okay and that's that's here we streamline the training pipeline by
 viewing object prediction the other yada yada this is often formulates in words
 what the paper is about and what contribution the paper makes right this
 is like a this is like a longer abstract the abstract is often very very cryptic
 very dense this here is often much more informative of what the paper does so
 for understanding the paper and a high level the introduction is the best place
 but given that I've already looked at the images and so on I don't actually
 draw many new much new information from this thing then there's related work and
 honestly I I skip it like unless I'm the actual reviewer of a paper like when
 I'm the reviewer of a paper I read the related work but often the related work
 is just like you first of all you cite a bunch of your friends and then you cite
 the mandatory papers and then you cite every single person that you think
 could be a reviewer because or you've actually been rejected from a conference
 with a reviewer claiming that you're you haven't compared or you haven't cited
 that or that paper you can pretty much be sure that that's the if if it's not a
 glaring of omission if it's like a niche paper and you haven't cited it then
 you're like okay I'm gonna cite it just because the next conference you could be
 my reviewer again so I'm not I'm not sure that these related work sections
 they're necessary like if someone wants to write their thesis and they go and
 read this paper and they want references oftentimes this is a good place but a
 lot of it is just blah blah blah blah blah okay I know I know disagree with me
 if you want oh yeah to maybe to reading quality so I tend to at this point I
 tend to not skim so at first I skim but at this point I tend to read every
 sentence and read it closely and understand it and when I realized like
 I'm tired or something I don't just skim the paper I've tried to skim papers and
 it doesn't doesn't work try to read every sentence understand every sentence
 and okay if you don't understand it don't stop reading because of that but
 try to not skim and be like oh yeah yeah yeah okay I gotta go to go to go to go
 that's is not helpful except related work skip completely cool then a lot of
 times in this paper now is the the model and this is the section I'm actually
 interested in right so I read very very closely here and then I find out what
 their their loss is all about and again I stress read these things and
 understand them right sometimes it's hard but if you're if you're confused
 that means you either they've done a bad job or they made a mistake or that you
 haven't understood something if you can't understand the sentence try to
 read on maybe it's clarified later and then you know go back but again do not
 do not like just start a lot of times when I read paper previously like I
 wouldn't understand something quite well yet and then I would be like oh yeah
 yeah yeah and then I noticed that I start skipping and skimming more and
 more because that would you know pop up again and again and I wouldn't understand
 it again and again and then at the end I would just be kind of glancing at the
 paper and I don't want to do that right here so I want to read every sentence
 and understand it okay so here then I find out about the loss and then I if I
 don't know something here then I'll go and look it up on maybe on Wikipedia or
 something like this now I don't need to on actually I don't need to understand
 every single part of it right that's maybe I should correct myself so for
 example this bounding box loss here they talk about the second part of the max
 and question Hungarian possible is this box loss that scores bounding boxes
 unlike many detectors that do box prediction with some initiality yada yada
 yada they say the most commonly used L1 loss will have different scales for a
 small so here they basically talk about how they mix the losses they see overall
 our box losses that defined as this and this now I haven't I don't know what
 these losses are I just assume there's some bounding box losses so when I it's
 not true when I say understand everything understand the things that
 are integral to the story of the paper right how exactly they compute bounding
 box losses at this point I don't care I just assume that there's some loss that
 I can back propagate right I what is important is that they do this
 Hungarian matching thing right as soon as I get that I'm like ah that was this
 you know this this thing no this thing up here this thing this with the matching
 thing now I get it now I know there are always the same amount of boxes here and
 there are always the same amount of labels here and all we need to do is
 somehow match them and I immediately think why is that relevant oh because
 when something is already matched to an object some other thing cannot be
 matched to the same object and that's how we you know prevent the fact that
 all the things predict the same thing right and so that immediately becomes
 clear and as I said there is usually like one or two ideas in a paper I don't
 assume or I don't care what their exact loss function is because I've sort of
 gotten the idea up here of what the loss is about all right so I hope that's
 clear under very closely read the things and understand the things that are
 necessary for the story if you find if you think something is not necessary for
 the story and then later end up not understanding that maybe come back and
 you know read it again in any case I would I would rather I would rather skip
 something and assume it's not necessary if I think so and then come back then
 trying to understand every everything but the things I do read I try to
 understand thoroughly okay then there's the architecture okay and that again I
 read closely and get backbone okay transformer encoder okay and now I
 understand much more closely decoder okay and here I get now finally I get
 what this is about decodes and objects in parallel yada yada yada these input
 embeddings are learned positional encodings that we refer to as object
 queries and similarly to the encoder we add them to the input at each attention
 layer so now they name I've already seen these object queries here and the only
 word I actually need from this sentence are learned the fact that they're
 positional encodings I just kind of ignore as soon as they say learned I know
 aha these things here are learned they have actually they're always the same
 for each of the images they're just overall learned okay so now I feel I
 understand the entire model and yeah so they then they say auxiliary decoding
 losses and this sometimes you have to pay attention to like auxiliary auxiliary
 things because those are the the things that here they say explicitly we found
 helpful to use auxiliary losses sometimes they they won't say why they
 did it they'll just say our loss consists of three things and you know if
 you look at the three things only one of the things is really a part of their
 story so far and that you should immediately conclude that they've put in
 the other things because they tried it and it didn't work right so you can also
 kind of get an estimate of the brittleness and so on of the system in
 that you see how many unnecessary things are there or how many things are not
 straightforward how many things are the easiest thing that you would do when you
 would go about and do what they did okay so then you this concludes this model or
 method usually this section is called like method or model or something like
 this and you go to experiments now the main question I have so far or I have
 maybe I have some more questions about the model itself that I haven't been
 able to pick up from this section which is not the case here but I simply keep
 those questions in mind and see whether they are resolved later right so I keep
 an awareness of what I don't understand but from here on my main issue is are
 they demonstrating that their story works right so they're here they're
 they're proposing a loss and a model and in my mind they now need to convince me
 that that works and that's that's it's not as easy as simply to show me some
 numbers that they are good at some benchmark they need to show me that they
 get those numbers because of what they claim so here they claim well okay they
 propose a new they propose a new architecture so what they need to
 convince me of is that the architecture itself makes sense right but in other
 papers when when you propose like and when you say like oh we for example in
 an LSTM when you build in an attention mechanism and you claim oh we you know
 the attention mechanism can look back at the source sequence in one step then you
 need to convince me that that actually happens right so you need to not only do
 you need to perform well you need to convince me that you perform well because
 of what you claim your model does right so and that's often difficult and I
 specifically look out in the experiments for usually the question is like where
 are they trying to bullshit me right where are they trying to are or are they
 trying to bullshit me are they trying to cover up the fact that something doesn't
 work and all the experiments are always in the best light possible of course and
 you have to keep that in mind but a lot of times you can also already see from
 the experiments that okay are they doing something weird are they not showing me
 some obvious experiment or and that's a lot of times I guess is there an easier
 explanation for why they get the results that they get other than their
 explanation right and it is it is their job to convince you that their
 explanation is the correct one for these numbers and especially if there is an
 easier one that they haven't excluded and then I don't believe the experiments
 if that's the case right if there is an easier explanation for the effect I'm
 I'm very skeptical but some papers have an easier job here than other papers so
 in this paper they basically show results on a on a on a task and since
 their paper is about hey our pipeline is just easier than other pipelines what
 they first of all need to do is they just need to like match the numbers of
 other pipelines and here I see that okay in these results often you have maybe a
 table or something here you see like this their model other models and their
 model is the best model in a lot of cases now if the best thing is of course
 if their model throughout is the best the worst thing is if it's like
 scattered like this even if their model is the best but in every single
 benchmark a different configuration of their model is the best that's that's
 sort of a bad sign unless they can explicitly explain why that is and it's
 also not that good of a sign if these things are spread out like this like
 sometimes this baseline is good sometimes their model is better and so on
 so pay attention to that now in this paper it doesn't matter so much that's
 actually fine because what they're trying to show is that their model is on
 par and way easier and they've already made the case in what way it is easier
 it's easier in terms of architecture if they were to say it's much faster then
 after that I would expect you know an experiment in speed while these numbers
 are matched so but since they say it's it's easier I've already seen the
 architecture I'm convinced of that now that they show okay our numbers match
 or actually I'm surprised they even outperform a lot of times then I'm quite
 happy with these experiments so also look for differences between numbers and
 the spread of numbers now it's not easy to say what if like point one is a big
 or a small difference that depends on the task but if you know pay attention
 to these things pay attention to the fact that these results are noisy and
 oftentimes there is a lot more hyper parameter tuning going into the model of
 the paper then into the baseline models right you want to make your look your
 stuff look as good as possible and here is a little bit where the institutional
 credibility of someone like Facebook comes in in that I tend to believe their
 results a bit more than other results not mega but a bit more yeah also look at
 patterns that they don't point out in the text so if there is like a pattern
 if you see like an interaction between the number of parameters and the score
 or something like this just try to be on the lookout of that and see if you can
 spot something that you think or think about whether that makes sense or not in
 what your hypothesis would be so here we go on and okay then they go into
 ablations and a lot of a lot of these papers do ablations and I generally
 appreciate that so here they visualize that the attention mechanism in their
 model actually refers to different instances right encoder self-attentions
 for a set of reference points the encoder is able to separate individual
 instances and you can see that pretty clearly right here where and even here
 with the overlapping cows and this is the sort of experiment that I would
 expect that actually convinces me that their architecture does what it says
 that it does right and something like this where you see like totally
 overlapping things with the attention of the individual things visualized so
 telling me like especially this one right here the the foot of the back
 elephant actually being focused by the attention of the bounding box of the
 back elephant that's the sort of experiment that convinces me that their
 claims like that their numbers really come from what they claim it comes from
 okay so at the end of the experimental section you should always ask yourself
 have they really convinced me that their story is true right that the
 improvement or when egg whenever they get an improvement or whatever they get
 what is is due to the story that they want to sell me or could there be like
 an easier explanation or does something not fit is like are there are the
 experiments different than from what you would expect here okay so these are
 these are my main questions are they are they convincing me of their story it's
 not do they have state-of-the-art numbers I don't care I don't care even
 though like sometimes so there is a bit of a catch I I don't care about state
 of the art numbers now let's say you have a table like this and you have a
 computer vision model and one of the models is like on the C for 10 data set
 now if your baseline model has like a 91 92 percent accuracy on C for 10 when I
 know the state-of-the-art is 96 I don't care right I know like I've done C for 10
 I know with like I don't know five six layers CNN you can reach these 91 92 93
 percent accuracy and to get to the 96 97 you'd actually be like in the region of
 a wide resinette and whatnot so it I I know that even though you're a few
 points behind state-of-the-art I know you know this this is valid still so I
 don't care but if you were to be like at 80 percent accuracy on C for 10 then I
 then I get a bit like hmm I like it's pretty easy to get to 90 percent plus
 with like a standard CNN so there I immediately start to wonder why is there
 an explanation now this could be like a theoretical paper that says oh we
 investigate MLPs and that's why we only get that number so that's that would be
 fine but if something is out of the ordinary like this then I pay attention
 but never because something isn't like the latest and greatest state-of-the-art
 that's just dumb okay and also if only evaluate what the paper claims it does
 right if the paper says we want to show that we are on par with current models
 then don't be mad if the paper doesn't outperform these models they didn't
 claim that right so yeah so after these ablations I'm actually pretty happy
 right here with the results and this right here when I saw this I didn't I
 didn't expect that but I read the experiment description that these are
 these different learned object queries and what they do and that gave me an
 increased understanding of how these object queries actually work right so at
 that point I still had like a vague I knew that these are learned but reading
 this and sort of looking at it studying it a bit I was like oh okay then I
 understood even better what they are so again when I say understand everything
 in the method section you can still have questions and but you just have to keep
 it in mind for later and then here I go on and there's this DETR for panoptic
 segmentation and they here they propose like a new model so I first look at it
 and I'm like okay they propose a new model they can do stuff like this now
 this is not object detection and again I'm not sure is this like a is this like
 a an add-on to the method or is was was this up here just an intermediate step
 to this and honestly after reading that I still wasn't sure it seems like
 something in between of course the paper is also a bit longer than other papers
 it just it seems it's too long for just being a side note but it's too short for
 being its own thing so that was just a bit weird and I treated is as just like
 a oh we can also do this with our model but I didn't pay like too much attention
 to that okay so at the end I you know look at conclusions now the conclusions
 of a paper are much much often they are not nearly as informative as the
 introduction the conclusions they all often tend to be very generic and kind
 of hedging a bit against criticisms saying what would be up for future work
 which is again hedging against criticism because you simply say well we didn't do
 this that's future work yes so again I read it but I don't really pay attention
 to it and then I gloss over the abstract I just would kind of scroll through the
 abstract if there's something that catches my eye I would look at it and if
 not then not and then I basically go to the start and whenever I didn't
 understand something I go back I look at it again and I try to think are all my
 questions answered and have they sufficiently convinced me that their
 story is the thing that really has the effect right here and then if I now were
 to make a video of this I've often found it useful to just put the paper away for
 a while and it's I usually get the best results when I read the paper the day
 before and then make a video the day after or if not I'll just you know put
 it away do something else do some email responding programming going outside
 eating lunch just some kind of a break between first read or between your
 first couple of reads and riff just I don't even think about the paper I just
 kind of it's just in the subconscious it kind of brews right and I happen to
 think about the paper every now and then but I don't make a conscious effort to be
 like oh how am I gonna explain this and so on but I just found the the worst
 videos are the ones where I immediately make the video after reading a paper
 and I've just discovered that if I kind of take a break and then I look at it
 again right I look I don't read it fully again but I if I have if I have the
 feeling I've understood it I don't read it fully again but I just kind of look
 at it and go again through the story and I think that's even if you you know want
 to if you want to talk about a paper in a reading group or tell you know explain
 it to your friends or whatnot this is often very useful just put it away for
 a while let it mellow and I find that helps a lot okay that was my process of
 reading this particular paper now we again this this is a high quality paper
 so it's I find it's a pretty easy read in that I simply need to understand what
 they did and I'm pretty happy with their experiments I maybe next time I can find
 an experiment or a paper where I'm initially more skeptical and not as happy
 with what I find but yeah let me know if you enjoyed this or if you would like to
 see any other explanation I don't exactly know if this is what you expected
 from a video like this so let me know maybe I have misunderstood you
 completely or it's way too long way too detailed or way too undetailed yeah
 leave me a comment and I'll see you next time bye bye