Context R-CNN: Long Term Temporal Context for Per-Camera Object Detection (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper vision cnn convolutional neural network coco object detection region of interest rcnn r-cnn attention attention mechanism google caltech gazelle wildlife wild trap traffic object car bus vehicle lighting time sampling frames memory long-term query
Object detection often does not occur in a vacuum. Static cameras, such as wildlife traps, collect lots of irregularly sampled data over a large time frame and often capture repeating or similar events. This model learns to dynamically incorporate other frames taken by the same camera into its object detection pipeline.

OUTLINE:
0:00 - Intro & Overview
1:10 - Problem Formulation
2:10 - Static Camera Data
6:45 - Architecture Overview
10:00 - Short-Term Memory
15:40 - Long-Term Memory
20:10 - Quantitative Results
22:30 - Qualitative Results
30:10 - False Positives
32:50 - Appendix & Conclusion

Paper: https://arxiv.org/abs/1912.03538

My Video On Attention Is All You Need: https://youtu.be/iDulhoQ2pro

Abstract:
In static monitoring cameras, useful contextual information can stretch far beyond the few seconds typical video understanding models might see: subjects may exhibit similar behavior over multiple days, and background objects remain static. Due to power and storage constraints, sampling frequencies are low, often no faster than one frame per second, and sometimes are irregular due to the use of a motion trigger. In order to perform well in this setting, models must be robust to irregular sampling rates. In this paper we propose a method that leverages temporal context from the unlabeled frames of a novel camera to improve performance at that camera. Specifically, we propose an attention-based approach that allows our model, Context R-CNN, to index into a long term memory bank constructed on a per-camera basis and aggregate contextual features from other frames to boost object detection performance on the current frame.
We apply Context R-CNN to two settings: (1) species detection using camera traps, and (2) vehicle detection in traffic cameras, showing in both settings that Context R-CNN leads to performance gains over strong baselines. Moreover, we show that increasing the contextual time horizon leads to improved results. When applied to camera trap data from the Snapshot Serengeti dataset, Context R-CNN with context from up to a month of images outperforms a single-frame baseline by 17.9% mAP, and outperforms S3D (a 3d convolution based baseline) by 11.2% mAP.

Authors: Sara Beery, Guanhang Wu, Vivek Rathod, Ronny Votel, Jonathan Huang

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we'll look at context R-CNN, long-term temporal context for per-camera object detection by Sarah Beery, Guan Hong Wu, Vivek Rathod, Ronnie Votel, and Jonathan Huang. So, on a high level, this paper tries to do object detection for cameras where the camera is in the same place for a long time. For example, these wild trap cameras or traffic cameras right here. It proposes to do object detection by incorporating data from the images that the camera has seen in the past to help the detection in the current frame. And it does so via an attention mechanism that it runs over a memory of past data. So we're going to take a look at how this is done and how well it works. And yeah, stick around if you want to know. As always, if you enjoy content like this, then consider sharing it out, telling your friends about it. Subscribe if you haven't and tell me what you think in the comments. So, the paper starts off and describes the problem. And the problem is fairly simple. You want to do object detection in images. Object detection is the task of basically, if I give you an image, you should tell me what is on the image and where. So in this case, here you would have to draw me this bounding box and say, this is a deer. On the bottom, you would have to draw bounding boxes. Maybe they have to be rectangular, maybe not. And say, this is a bus. And here is a truck. And here is another truck. And here is a car. And so on. So, there can be many objects in an image. There can be one object. There can be objects of different classes or there can be no objects at all. So this is just object detection. And there have been many papers on this. And specifically, there has been this RCNN. And this is the model that we're going to extend. So the RCNN model or specifically the faster RCNN model that we're going to build on is a model that simply detects these bounding boxes in single images. But now we consider the situation where we have a camera that records images for a long, long time. So in these wild trap cameras, they often sit there for months. And it's not that easy to make use of them because in addition to there being a lot of data, they have motion triggers. So that could be there is no nothing for a long time. And then there's the animal walks in the trap. And then you have a bunch of images, like one per second for 10 seconds. And then you have nothing again for like a day or two days. And then you have 10 images again because another animal walks in or maybe doesn't. And so on and another. So you have irregular sampling frequencies. You have very, very different distance between the frames. All of this makes it very, very not suited for models like temporal convolutions or things like LSTMs because they don't work super well with data like this. Now I know there are formulations where LSTMs can do this, but they don't work very well with these super long contexts and irregular sampling frequencies and so on. So the idea is if we have a frame right here, like this one, and we want to detect what's on it, we should be able to pull information from other frames that the same camera has seen like from this one or from this one or from this one right here. And we should be able to do so in a dynamic way. Now why could that help? If you look at, for example, down here, these images have been taken. They say images were taken on separate days. But you can see this thing right here is in both images. Or a very similar thing. It's probably that bus's regular route. So in order to classify whether or not this here is a bus, it might be very helpful to also look at this picture right here and see, ah, you know, it's about at the same location. It looks the same and also it looks like a bus. So you know, that kind of gives evidence that this could be, this other thing could also be a bus. Then also there are background objects. So sometimes the single frame detectors get confused. It might be labeling this here as a car because just the lighting, the exact lighting in this picture is just off by the correct amount that it is confused. But considering this picture over here, maybe it recognizes here, no, that's not a car. And it can bring over this evidence to that frame and consider, ah, maybe you know this is the same thing. So it's not a car. So this is not the same than simply adding training data. We really consider the fact here that these images, they come from the same camera that is in the same location or maybe, you know, that is filming the same thing. So this, all of this is going to be within the same camera, not just adding, adding IID training data. And with animals as well, like often the same animal has like its regular route or within these 10, these burst of tens, the same animal will kind of walk around a bit and maybe, you know, here it's half occluded, but maybe in a different image you see, ah, here I see the nose. So it helps you make a better prediction. Also animals are often in kind of crowds and that helps if you see that there are other deer around, the probability that this is a deer increases rapidly. So how are we going to do this? What we're going to do is we're going to build an attention mechanism that can do these kinds of look into the past and some, also a little bit of the future as we will see, but mainly we'll look into other images from the same camera in a dynamic way and we'll learn how to address those other images from a memory bank. So the architecture is described right here. Now as you can see, we are still in the business of doing object detection. So what we'll do is we'll sort of hijack a existing object detector and the object detector we're going to hijack is going to be this FRCNN, this faster RCNN object detector. That's an object detector for a single frame problem. So that means you have one image and you're supposed to detect what's on it. It has two stages, as you can see. So stage one, if you have an image and let's say there's some stuff on it, stuff, stuff, stuff, there's stuff. What stage one is supposed to do is it's supposed to extract regions of interest. This could be, okay, all of these are regions of interest. So it simply says, well, there is something, there might be something right here in these regions of interest. And then it describes each of these regions of interest using features. So it extracts these regions of interest and each region of interest gets features assigned to it. So, well, these are, I think these are like seven by seven by 2048 features, but let's just say for the sake of describing it that these are just a vector of features for each region of interest. So each region of interest is going to be associated with one vector of features that this model extracts. Okay. And the next region of interest also has a vector and the next region of interest also has a vector and so on. Stage two then takes each one of these, takes each one of these vectors and assigns a class to it. So this would be deer right here. Okay, so stage one proposes regions of interest along with features. Then stage two takes each of these regions of interest and classifies them basically. And I guess there's many in between stages. Like this is massively simplified. There's non-maximum suppression. There is kind of an alignment stage where you can refine the bounding box and so on. But in essence, these are two stages. And you can see that this system here, it goes in between the two stages. So all of this right here, we shove in between the two stages. So we'll still use the stage one and we'll still use the stage two, but in between in this thing right here, we'll try to sort of pimp these features such that the stage two detector has an easier time classifying. Okay. So now we're going to pimp these features by incorporating in because these features right now, if we just do it vanilla, these are just from the current frame. And we're going to add to them information from other frames of the same camera. And we're going to do it in two different ways. So the first way, as you can see here, the first way is this short term memory. And the second way is the long term memory. Now the two are slightly different. As you can guess, the short term memory is going to be only over a short time period around the current frame. And the long term memory is going to be basically across a very long time horizon into the past. You can see we're trying to classify this blue frame right here, what we call the key frame. So what we'll do is we'll run it through stage one. Cool. So we're going to add two features for each region of interest. And then you can see this goes here and through these residual connections, this goes into stage two over here. So basically, stage two still receives the same input, it receives whatever stage one outputs for the key frame. But we're going to add to that twice. So we're going to add two things as I said. So the short term memory is added right here. Now how do we build the short term memory? We build the short term memory simply by considering all the frames around the key frame. And this you can see right here, the current window around the key frame, which can be like one frame around it or two frames or three frames, just a few frames around the current frame. And this can be fairly helpful. As we said, for example, if the deer moves a bit, the car moves a bit, you know, it gets into a slightly different lighting and so on. This can help us very much to classify the current key frame if we also have features from the surrounding frames. So for each of these surrounding frames, we also run them through the stage one detector to also extract regions of interest. And that all of these features go into this memory short term memory bank right here. There's different strategies, you don't always have to extract all of the regions of interest. You can also extract just the top one and so on, or you can extract the mean, since these are fairly, you know, consistent, the cameras at the same place. There are many ways you can do this. But what you ultimately end up with is a short term memory bank that kind of is so you'll have a bank and you have lots of these feature vectors in here for your region, your regions of interest of the surrounding frames. Now if this here, if this here is your half occluded deer, right, so this is the half occluded deer, and you want to consider information from the surrounding frames, maybe in the next frame, so maybe this is three frames, like 123 and two is the key frame, maybe in the next frame, the deer moves a bit and you see its nose. And that this particular region of interest here is relevant. So how do you know how do you now get from this entire memory, this feature vector that would be helpful? And the answer is you get it through an attention mechanism. You can see that right here, the way the short term memory is added is through this attention block. They describe the attention block right here. It is a fairly standard attention mechanism. So I've done a video on attention is all you need. If you don't know what an attention mechanism is, go check it out. But you can see it's it's very, very standard. So you have these input features, which are the features that come from the key frame. And you have the context features, which are all the features in your memory bank, you encode the input features with into a query using a fully connected layers and the context features into keys. And then you match the queries with the keys and the softmax in order to get a weighting over the context features. And then you aggregate the values from the context features. So this is a standard attention mechanism. What does it mean? It basically means that each of these vectors right here, they will emit a key that kind of describes what kind of information is contained in that vector. The vector over here will emit a query that describes what sort of information it is looking for to in order to describe what's in the region of interest as well as possible. And then you simply match the query with the keys to determine which key fits best to that query and whichever one fits best, let's say this one here, then you take that vector from the memory bank and incorporate it together with your current information that you already have. So that's how you address things that from other frames using an attention mechanism. Okay, now if this were all, you know, we could train this right now. We could train all of this because all of this is differentiable, right? This stage one detector right here is differentiable. It goes here and here, you know, the information, the attention mechanism is differentiable. The stage two detector is differentiable, all differentiable. Cool. We can train this end to end. Now what's the problem? The problem is this long term memory right here. So in this memory, ideally, we would want to fit, let's say an entire day, an entire week or even an entire month of data from one of these cameras. And it's just not feasible that we expand this current window here to an entire month or an entire week for many of those of those cameras, because even though they have a low frame rate and so on, it's still too much in order to then be all differentiable, all backpropagatable and so on. So we can't really backprop in for these long term memory. In essence, what we want to do is exactly the same. We want to build up a memory of all of the regions of interest or maybe selected regions or all of the best regions of interest, whatever heuristic strategy we have of the past, whatever this camera has seen, let's say in the last month or in the current week or something like this, we want to build all of this up and then use an attention mechanism, just the same in order to incorporate it. But we have to come up with these things right here in some other way than a way where we can backprop. So we can't really use this stage one detector right here because this is the one we're training and so we have to backprop through it. Now an easy proposal is to simply use it anyway, but do like a stop gradient on it so we don't backprop through it. That is one way but this the paper decides on a different way. The paper decides that all of the past basically right here, right here and so on, we'll take a pre-trained object detector. So not the one we're training currently, but we'll take a pre-trained one that was pre-trained either on something like Cocoa, which is an object detection data set, or you can pre-train it on Cocoa and then fine tune it on a task you're interested in, in a single frame fashion. For whatever way, we'll take a pre-trained object detector or region of interest extractor and that will give us for each frame in the past will give us also the regions of interest along with the features. And these are the features that we then go and put into the memory bank. Sorry, my tablet just crashed a bit. There we go. Okay, so we'll take a pre-trained extractor right here. That will give us features for regions of interest, we'll put that into the memory bank, and then we will use an attention mechanism to incorporate it. Now the attention mechanism we can train, but we cannot train the extractor for the features. And this is the difference to the short term memory where we can actually train the feature extractor in order to help us with building the memory. Now the memory is simply built without a goal in mind basically. And the attention mechanism basically has to learn that it doesn't work with features that are meant for its task. It works with features that have been originally created for a different tasks and they're not going to change. But as we'll see, this, you know, can be handled. So that's what they do. They incorporate short term and long term memory into their stage two prediction and then the stage two prediction simply takes in all of those features and classifies the class of the object. And that's the architecture of context rcnn. It's rcnn with long and short term context. So they describe very different ways of you know, how they built the memory and so on, how they built the features. I didn't, I kind of glossed over this right now. There's a lot of consideration in building these things and you have to look at the paper how they exactly do this. I'm more interested in the high level architecture and the sort of ideas behind it. So when they do this, they do outperform the, they do outperform the current or the single frame baselines by quite a bit. So this SS and this CCT are these wildlife data sets, whereas this CC, I think this is the city something city cam, these are this street data set. As you can see, they do outperform the single frame baseline by quite a bit. Now interesting, as you can see right here, as they increase the time horizon of this long term memory, so they, they can, they can now choose how much information do they want to put in that long term memory. As they increase the time horizon for one minute, one hour, one day and so on, the performance goes up and up and up, which is a strong indication that these features actually help from the time horizon, because you don't have more parameters. You simply increase the amount of information in the memory bank. And if the performance goes up, you can make a very strong claim that these, this is actually due to the fact that you have more information in that memory bank. Couldn't really guess any other explanation right here. So they, they do, they do investigate different memory strategies. They do a lot of ablations right here, where they also say, okay, what if we only have the short term attention? What if we only have the long term attention? What to only, if we only have self attention, that means attention only into the current frame, but of across regions of interest. That's interesting if you have like a herd of animals and so on, and they all help. But as you can see, the long term attention tends to help the most in this data set. And the short term attention help helps a lot in this data set. If you just compare to the other owner, these are two different metrics, not data sets. Sorry about that. But in essence, it helps the most when you combine the two and that's, you know, that's pretty cool to see. So they do some qualitative results, which I find very interesting. For example, they can visualize what the attention weights of their models are. So here, you always have a very long timeframe, I think an entire month in this, in this memory bank of the long term memory. Now in the top classification, you see the large thing here, the large frame is the one you actually want to classify. And the other frames are the frames where the top attention score is so that the attention weights are the highest. So here in order to classify this, what does the model pay attention to? Or which other frames does the model pay attention to? And you can see right here, they are all spread across the entire month. Here is the timeline. The most attended to pictures are spread across the entire month. And almost all of them actually have that work on in here. So this must be like its regular route. And the model recognizes that and pulls in information from all these other images in order to correctly classify it here. On the other hand, on the next example, this gazelle and tablet crashed right here. It also puts all the weight on top of images of that same gazelle. But you can see maybe that gazelle was only there for this one particular moment. And all the pictures this camera has of it is, you know, in the very few moments that the gazelle was around. You can see they all come here from the same point in time, or very, very close points in time. And you can see that it puts a lot of weight on wherever the gazelle is. So you know, that's a pretty strong indication that it actually learns to pull in the correct information be that from long time horizon, or from a short time horizon if necessary. You can also see right here, they visualize the top attention, where the top attention weights go, in terms of how long the frames where the attention goes to is away from the frame that they're trying to classify. So these graphics are somewhat kind of weird to interpret. This here always means how much is the total time of the buffer. So the memory buffer here contains always pictures from the total from one hour before until one hour after the key frame you want to classify. So this is the frame you want to classify at minute zero. And the memory buffer contains images from 60 minutes before to 60 minutes after. So it's not real time, right? You go back to your through your footage and you try to classify. You can also pull out images from the future. You can see there's most attention is on the current frame, which makes sense. You're trying to classify the current frame and it kind of falls off as you go further and further away. This is across the entire data set. So this is not a specific example, which also makes sense. Probably in most of the time, the relevant information is closer in time rather than farther away. But also you can see that the distribution is pretty spread out. So it makes the model makes use of the entire range of time. And you can see that throughout, even if you have an entire day in the buffer or two days, even if you have entire week before and week after in the buffer, and even if you have an entire month here. And especially if you look at when you have an entire week in the buffer, you can see the periodicity through the days. So that means the model tends to pay attention to images that are from the same time of day. Compared to the current key frame. That's fairly, fairly good indication that the model has actually learned to address these this memory by its content, right? Now night and day isn't super difficult because you can just go on the brightness and so on. But still, it's pretty cool to see that this is actually happening. They do have some failure cases of the single frame model that their model is able to handle up here. And they make a lot of sense. So here you can see that there is an object that's moving out of frame. And the single frame detector wasn't able to recognize this probably because it's moving out of frame. Whereas this new this context rcnn is able to detect it probably because it looked at the frame just before it where the car was somewhere back here and it could correctly classify it. Well, that's well, I just disregard my drawings. Here it managed to recognize this animal in the back, whereas this old model, the single frame model hasn't also probably by looking either at frames next to it or by looking at other frames of herds of animals and realizing that usually when there's two elephants, there's more. Here you can see that the object highly occluded. So we're talking about the object like at the very edge of the frame object poorly lit. This is particularly impressive. And also an example where the animals are often in herds. And if you see one deer, the likelihood that there's other deer is very high in this particular camera. And by aggregating information from different frames, you can see that maybe it's always the same patch of the air that comes by. And here, the single frame detector detects this patch here as a vehicle where it shouldn't. And of course, the new model, the context RCNN is able to recognize that this is present in all of the frames. And in most frames, the single object detector doesn't detect it as a vehicle. And so it can kind of carry over that information. Now you can already see sort of what the downsides might be if the single object detector is like very, very, very sure that this is in a single frame that this is a car. It could carry over that information to the other frame. So even though the single frame detector might have failed in that particular frame, if it fails super hard, it might, you know, shout that to all the other frames basically dominate the memory saying like, look, this is a car, I'm like pretty sure. And it will carry over that information to all of the other frames. And they say in one of these high confidence mistakes, it basically detected the same tree as a giraffe over and over again. What I find particularly interesting is they do look at, so here they have this curve of on the bottom, you have confidence threshold, so how confident the model is. And on the y-axis, you have the number of false positives. And you can see that in the low confidence regime, the context RCNN has lower false positives than the single frame detector. And the green line here is when you only have positive boxes. So when you only include regions of interest where there is an actual object, which in this case is sort of hurtful, you also want the regions of interest where there is nothing because that helps you avoid false positives in other frames. That's why the orange line is below the green line. But strangely here in the high confidence regime, you can see that the single frame model has fewer false positives than the context RCNN. And I like the text that they have to this. In figure 7, we can see that adding empty representations reduces the number of false positives across all confidence threshold compared to the same model with only positive representations. We investigated the 100 highest confidence false positives from context RCNN and found that in almost all of them, in 97 out of 100, the model had correctly found and classified animals that were missed by human annotators. So basically these graphs are even underestimating how good that model is because the model appears to be better than the human annotators of the test set. I find that to be pretty, pretty impressive. And here you can see failure modes where they say, for example, when exploring the confident false positives on the snapshot Serengeti data set, the three out of 100 images, so whatever was not a human failure, where context RCNN erroneously detected an animal, where all of the same tree highly confidently predicted to be a giraffe. So this is a failure mode when the model is highly confident it might spill that over to other frames because we now aggregate the information within the same camera across the frames. To be said, of course, their train test split is such that there's not the same camera in the training data as in the testing data. They have entirely different cameras in the testing data than in the training data, just so there is no information leakage. So that's the model right here, how it works. It's pretty cool. It kind of wedges itself in between any single frame object detector that has these two stages. And it's a pretty neat idea to bring in context from the past or even the future of the same camera. Just a quick glance at the appendix, they have lots of different examples right here. In one example, their camera kind of fell over and they say, well, it still worked. The system was still able to kind of do attention across this failure, this kind of tipping over of the camera. They have more examples right here, which I find pretty impressive, like these super low light things where it correctly detects like the possum. And yeah, I invite you to check out the paper, the code they say should be out soon. And I'll see you next time. Bye bye.
 Hi there, today we'll look at context R-CNN, long-term temporal context for per-camera
 object detection by Sarah Beery, Guan Hong Wu, Vivek Rathod, Ronnie Votel, and Jonathan
 Huang.
 So, on a high level, this paper tries to do object detection for cameras where the camera
 is in the same place for a long time.
 For example, these wild trap cameras or traffic cameras right here.
 It proposes to do object detection by incorporating data from the images that the camera has seen
 in the past to help the detection in the current frame.
 And it does so via an attention mechanism that it runs over a memory of past data.
 So we're going to take a look at how this is done and how well it works.
 And yeah, stick around if you want to know.
 As always, if you enjoy content like this, then consider sharing it out, telling your
 friends about it.
 Subscribe if you haven't and tell me what you think in the comments.
 So, the paper starts off and describes the problem.
 And the problem is fairly simple.
 You want to do object detection in images.
 Object detection is the task of basically, if I give you an image, you should tell me
 what is on the image and where.
 So in this case, here you would have to draw me this bounding box and say, this is a deer.
 On the bottom, you would have to draw bounding boxes.
 Maybe they have to be rectangular, maybe not.
 And say, this is a bus.
 And here is a truck.
 And here is another truck.
 And here is a car.
 And so on.
 So, there can be many objects in an image.
 There can be one object.
 There can be objects of different classes or there can be no objects at all.
 So this is just object detection.
 And there have been many papers on this.
 And specifically, there has been this RCNN.
 And this is the model that we're going to extend.
 So the RCNN model or specifically the faster RCNN model that we're going to build on is
 a model that simply detects these bounding boxes in single images.
 But now we consider the situation where we have a camera that records images for a long,
 long time.
 So in these wild trap cameras, they often sit there for months.
 And it's not that easy to make use of them because in addition to there being a lot of
 data, they have motion triggers.
 So that could be there is no nothing for a long time.
 And then there's the animal walks in the trap.
 And then you have a bunch of images, like one per second for 10 seconds.
 And then you have nothing again for like a day or two days.
 And then you have 10 images again because another animal walks in or maybe doesn't.
 And so on and another.
 So you have irregular sampling frequencies.
 You have very, very different distance between the frames.
 All of this makes it very, very not suited for models like temporal convolutions or things
 like LSTMs because they don't work super well with data like this.
 Now I know there are formulations where LSTMs can do this, but they don't work very well
 with these super long contexts and irregular sampling frequencies and so on.
 So the idea is if we have a frame right here, like this one, and we want to detect what's
 on it, we should be able to pull information from other frames that the same camera has
 seen like from this one or from this one or from this one right here.
 And we should be able to do so in a dynamic way.
 Now why could that help?
 If you look at, for example, down here, these images have been taken.
 They say images were taken on separate days.
 But you can see this thing right here is in both images.
 Or a very similar thing.
 It's probably that bus's regular route.
 So in order to classify whether or not this here is a bus, it might be very helpful to
 also look at this picture right here and see, ah, you know, it's about at the same location.
 It looks the same and also it looks like a bus.
 So you know, that kind of gives evidence that this could be, this other thing could also
 be a bus.
 Then also there are background objects.
 So sometimes the single frame detectors get confused.
 It might be labeling this here as a car because just the lighting, the exact lighting in this
 picture is just off by the correct amount that it is confused.
 But considering this picture over here, maybe it recognizes here, no, that's not a car.
 And it can bring over this evidence to that frame and consider, ah, maybe you know this
 is the same thing.
 So it's not a car.
 So this is not the same than simply adding training data.
 We really consider the fact here that these images, they come from the same camera that
 is in the same location or maybe, you know, that is filming the same thing.
 So this, all of this is going to be within the same camera, not just adding, adding IID
 training data.
 And with animals as well, like often the same animal has like its regular route or within
 these 10, these burst of tens, the same animal will kind of walk around a bit and maybe,
 you know, here it's half occluded, but maybe in a different image you see, ah, here I see
 the nose.
 So it helps you make a better prediction.
 Also animals are often in kind of crowds and that helps if you see that there are other
 deer around, the probability that this is a deer increases rapidly.
 So how are we going to do this?
 What we're going to do is we're going to build an attention mechanism that can do these kinds
 of look into the past and some, also a little bit of the future as we will see, but mainly
 we'll look into other images from the same camera in a dynamic way and we'll learn how
 to address those other images from a memory bank.
 So the architecture is described right here.
 Now as you can see, we are still in the business of doing object detection.
 So what we'll do is we'll sort of hijack a existing object detector and the object detector
 we're going to hijack is going to be this FRCNN, this faster RCNN object detector.
 That's an object detector for a single frame problem.
 So that means you have one image and you're supposed to detect what's on it.
 It has two stages, as you can see.
 So stage one, if you have an image and let's say there's some stuff on it, stuff, stuff,
 stuff, there's stuff.
 What stage one is supposed to do is it's supposed to extract regions of interest.
 This could be, okay, all of these are regions of interest.
 So it simply says, well, there is something, there might be something right here in these
 regions of interest.
 And then it describes each of these regions of interest using features.
 So it extracts these regions of interest and each region of interest gets features assigned
 to it.
 So, well, these are, I think these are like seven by seven by 2048 features, but let's
 just say for the sake of describing it that these are just a vector of features for each
 region of interest.
 So each region of interest is going to be associated with one vector of features that
 this model extracts.
 Okay.
 And the next region of interest also has a vector and the next region of interest also
 has a vector and so on.
 Stage two then takes each one of these, takes each one of these vectors and assigns a class
 to it.
 So this would be deer right here.
 Okay, so stage one proposes regions of interest along with features.
 Then stage two takes each of these regions of interest and classifies them basically.
 And I guess there's many in between stages.
 Like this is massively simplified.
 There's non-maximum suppression.
 There is kind of an alignment stage where you can refine the bounding box and so on.
 But in essence, these are two stages.
 And you can see that this system here, it goes in between the two stages.
 So all of this right here, we shove in between the two stages.
 So we'll still use the stage one and we'll still use the stage two, but in between in
 this thing right here, we'll try to sort of pimp these features such that the stage two
 detector has an easier time classifying.
 Okay.
 So now we're going to pimp these features by incorporating in because these features
 right now, if we just do it vanilla, these are just from the current frame.
 And we're going to add to them information from other frames of the same camera.
 And we're going to do it in two different ways.
 So the first way, as you can see here, the first way is this short term memory.
 And the second way is the long term memory.
 Now the two are slightly different.
 As you can guess, the short term memory is going to be only over a short time period
 around the current frame.
 And the long term memory is going to be basically across a very long time horizon into the past.
 You can see we're trying to classify this blue frame right here, what we call the key
 frame.
 So what we'll do is we'll run it through stage one.
 Cool.
 So we're going to add two features for each region of interest.
 And then you can see this goes here and through these residual connections, this goes into
 stage two over here.
 So basically, stage two still receives the same input, it receives whatever stage one
 outputs for the key frame.
 But we're going to add to that twice.
 So we're going to add two things as I said.
 So the short term memory is added right here.
 Now how do we build the short term memory?
 We build the short term memory simply by considering all the frames around the key frame.
 And this you can see right here, the current window around the key frame, which can be
 like one frame around it or two frames or three frames, just a few frames around the
 current frame.
 And this can be fairly helpful.
 As we said, for example, if the deer moves a bit, the car moves a bit, you know, it gets
 into a slightly different lighting and so on.
 This can help us very much to classify the current key frame if we also have features
 from the surrounding frames.
 So for each of these surrounding frames, we also run them through the stage one detector
 to also extract regions of interest.
 And that all of these features go into this memory short term memory bank right here.
 There's different strategies, you don't always have to extract all of the regions of interest.
 You can also extract just the top one and so on, or you can extract the mean, since
 these are fairly, you know, consistent, the cameras at the same place.
 There are many ways you can do this.
 But what you ultimately end up with is a short term memory bank that kind of is so you'll
 have a bank and you have lots of these feature vectors in here for your region, your regions
 of interest of the surrounding frames.
 Now if this here, if this here is your half occluded deer, right, so this is the half
 occluded deer, and you want to consider information from the surrounding frames, maybe in the
 next frame, so maybe this is three frames, like 123 and two is the key frame, maybe in
 the next frame, the deer moves a bit and you see its nose.
 And that this particular region of interest here is relevant.
 So how do you know how do you now get from this entire memory, this feature vector that
 would be helpful?
 And the answer is you get it through an attention mechanism.
 You can see that right here, the way the short term memory is added is through this attention
 block.
 They describe the attention block right here.
 It is a fairly standard attention mechanism.
 So I've done a video on attention is all you need.
 If you don't know what an attention mechanism is, go check it out.
 But you can see it's it's very, very standard.
 So you have these input features, which are the features that come from the key frame.
 And you have the context features, which are all the features in your memory bank, you
 encode the input features with into a query using a fully connected layers and the context
 features into keys.
 And then you match the queries with the keys and the softmax in order to get a weighting
 over the context features.
 And then you aggregate the values from the context features.
 So this is a standard attention mechanism.
 What does it mean?
 It basically means that each of these vectors right here, they will emit a key that kind
 of describes what kind of information is contained in that vector.
 The vector over here will emit a query that describes what sort of information it is looking
 for to in order to describe what's in the region of interest as well as possible.
 And then you simply match the query with the keys to determine which key fits best to that
 query and whichever one fits best, let's say this one here, then you take that vector from
 the memory bank and incorporate it together with your current information that you already
 have.
 So that's how you address things that from other frames using an attention mechanism.
 Okay, now if this were all, you know, we could train this right now.
 We could train all of this because all of this is differentiable, right?
 This stage one detector right here is differentiable.
 It goes here and here, you know, the information, the attention mechanism is differentiable.
 The stage two detector is differentiable, all differentiable.
 Cool.
 We can train this end to end.
 Now what's the problem?
 The problem is this long term memory right here.
 So in this memory, ideally, we would want to fit, let's say an entire day, an entire
 week or even an entire month of data from one of these cameras.
 And it's just not feasible that we expand this current window here to an entire month
 or an entire week for many of those of those cameras, because even though they have a low
 frame rate and so on, it's still too much in order to then be all differentiable, all
 backpropagatable and so on.
 So we can't really backprop in for these long term memory.
 In essence, what we want to do is exactly the same.
 We want to build up a memory of all of the regions of interest or maybe selected regions
 or all of the best regions of interest, whatever heuristic strategy we have of the past, whatever
 this camera has seen, let's say in the last month or in the current week or something
 like this, we want to build all of this up and then use an attention mechanism, just
 the same in order to incorporate it.
 But we have to come up with these things right here in some other way than a way where we
 can backprop.
 So we can't really use this stage one detector right here because this is the one we're training
 and so we have to backprop through it.
 Now an easy proposal is to simply use it anyway, but do like a stop gradient on it so we don't
 backprop through it.
 That is one way but this the paper decides on a different way.
 The paper decides that all of the past basically right here, right here and so on, we'll take
 a pre-trained object detector.
 So not the one we're training currently, but we'll take a pre-trained one that was pre-trained
 either on something like Cocoa, which is an object detection data set, or you can pre-train
 it on Cocoa and then fine tune it on a task you're interested in, in a single frame fashion.
 For whatever way, we'll take a pre-trained object detector or region of interest extractor
 and that will give us for each frame in the past will give us also the regions of interest
 along with the features.
 And these are the features that we then go and put into the memory bank.
 Sorry, my tablet just crashed a bit.
 There we go.
 Okay, so we'll take a pre-trained extractor right here.
 That will give us features for regions of interest, we'll put that into the memory bank,
 and then we will use an attention mechanism to incorporate it.
 Now the attention mechanism we can train, but we cannot train the extractor for the
 features.
 And this is the difference to the short term memory where we can actually train the feature
 extractor in order to help us with building the memory.
 Now the memory is simply built without a goal in mind basically.
 And the attention mechanism basically has to learn that it doesn't work with features
 that are meant for its task.
 It works with features that have been originally created for a different tasks and they're
 not going to change.
 But as we'll see, this, you know, can be handled.
 So that's what they do.
 They incorporate short term and long term memory into their stage two prediction and
 then the stage two prediction simply takes in all of those features and classifies the
 class of the object.
 And that's the architecture of context rcnn.
 It's rcnn with long and short term context.
 So they describe very different ways of you know, how they built the memory and so on,
 how they built the features.
 I didn't, I kind of glossed over this right now.
 There's a lot of consideration in building these things and you have to look at the paper
 how they exactly do this.
 I'm more interested in the high level architecture and the sort of ideas behind it.
 So when they do this, they do outperform the, they do outperform the current or the single
 frame baselines by quite a bit.
 So this SS and this CCT are these wildlife data sets, whereas this CC, I think this is
 the city something city cam, these are this street data set.
 As you can see, they do outperform the single frame baseline by quite a bit.
 Now interesting, as you can see right here, as they increase the time horizon of this
 long term memory, so they, they can, they can now choose how much information do they
 want to put in that long term memory.
 As they increase the time horizon for one minute, one hour, one day and so on, the performance
 goes up and up and up, which is a strong indication that these features actually help from the
 time horizon, because you don't have more parameters.
 You simply increase the amount of information in the memory bank.
 And if the performance goes up, you can make a very strong claim that these, this is actually
 due to the fact that you have more information in that memory bank.
 Couldn't really guess any other explanation right here.
 So they, they do, they do investigate different memory strategies.
 They do a lot of ablations right here, where they also say, okay, what if we only have
 the short term attention?
 What if we only have the long term attention?
 What to only, if we only have self attention, that means attention only into the current
 frame, but of across regions of interest.
 That's interesting if you have like a herd of animals and so on, and they all help.
 But as you can see, the long term attention tends to help the most in this data set.
 And the short term attention help helps a lot in this data set.
 If you just compare to the other owner, these are two different metrics, not data sets.
 Sorry about that.
 But in essence, it helps the most when you combine the two and that's, you know, that's
 pretty cool to see.
 So they do some qualitative results, which I find very interesting.
 For example, they can visualize what the attention weights of their models are.
 So here, you always have a very long timeframe, I think an entire month in this, in this memory
 bank of the long term memory.
 Now in the top classification, you see the large thing here, the large frame is the one
 you actually want to classify.
 And the other frames are the frames where the top attention score is so that the attention
 weights are the highest.
 So here in order to classify this, what does the model pay attention to?
 Or which other frames does the model pay attention to?
 And you can see right here, they are all spread across the entire month.
 Here is the timeline.
 The most attended to pictures are spread across the entire month.
 And almost all of them actually have that work on in here.
 So this must be like its regular route.
 And the model recognizes that and pulls in information from all these other images in
 order to correctly classify it here.
 On the other hand, on the next example, this gazelle and tablet crashed right here.
 It also puts all the weight on top of images of that same gazelle.
 But you can see maybe that gazelle was only there for this one particular moment.
 And all the pictures this camera has of it is, you know, in the very few moments that
 the gazelle was around.
 You can see they all come here from the same point in time, or very, very close points
 in time.
 And you can see that it puts a lot of weight on wherever the gazelle is.
 So you know, that's a pretty strong indication that it actually learns to pull in the correct
 information be that from long time horizon, or from a short time horizon if necessary.
 You can also see right here, they visualize the top attention, where the top attention
 weights go, in terms of how long the frames where the attention goes to is away from the
 frame that they're trying to classify.
 So these graphics are somewhat kind of weird to interpret.
 This here always means how much is the total time of the buffer.
 So the memory buffer here contains always pictures from the total from one hour before
 until one hour after the key frame you want to classify.
 So this is the frame you want to classify at minute zero.
 And the memory buffer contains images from 60 minutes before to 60 minutes after.
 So it's not real time, right?
 You go back to your through your footage and you try to classify.
 You can also pull out images from the future.
 You can see there's most attention is on the current frame, which makes sense.
 You're trying to classify the current frame and it kind of falls off as you go further
 and further away.
 This is across the entire data set.
 So this is not a specific example, which also makes sense.
 Probably in most of the time, the relevant information is closer in time rather than
 farther away.
 But also you can see that the distribution is pretty spread out.
 So it makes the model makes use of the entire range of time.
 And you can see that throughout, even if you have an entire day in the buffer or two days,
 even if you have entire week before and week after in the buffer, and even if you have
 an entire month here.
 And especially if you look at when you have an entire week in the buffer, you can see
 the periodicity through the days.
 So that means the model tends to pay attention to images that are from the same time of day.
 Compared to the current key frame.
 That's fairly, fairly good indication that the model has actually learned to address
 these this memory by its content, right?
 Now night and day isn't super difficult because you can just go on the brightness and so on.
 But still, it's pretty cool to see that this is actually happening.
 They do have some failure cases of the single frame model that their model is able to handle
 up here.
 And they make a lot of sense.
 So here you can see that there is an object that's moving out of frame.
 And the single frame detector wasn't able to recognize this probably because it's moving
 out of frame.
 Whereas this new this context rcnn is able to detect it probably because it looked at
 the frame just before it where the car was somewhere back here and it could correctly
 classify it.
 Well, that's well, I just disregard my drawings.
 Here it managed to recognize this animal in the back, whereas this old model, the single
 frame model hasn't also probably by looking either at frames next to it or by looking
 at other frames of herds of animals and realizing that usually when there's two elephants, there's
 more.
 Here you can see that the object highly occluded.
 So we're talking about the object like at the very edge of the frame object poorly lit.
 This is particularly impressive.
 And also an example where the animals are often in herds.
 And if you see one deer, the likelihood that there's other deer is very high in this particular
 camera.
 And by aggregating information from different frames, you can see that maybe it's always
 the same patch of the air that comes by.
 And here, the single frame detector detects this patch here as a vehicle where it shouldn't.
 And of course, the new model, the context RCNN is able to recognize that this is present
 in all of the frames.
 And in most frames, the single object detector doesn't detect it as a vehicle.
 And so it can kind of carry over that information.
 Now you can already see sort of what the downsides might be if the single object detector is like
 very, very, very sure that this is in a single frame that this is a car.
 It could carry over that information to the other frame.
 So even though the single frame detector might have failed in that particular frame, if it
 fails super hard, it might, you know, shout that to all the other frames basically dominate
 the memory saying like, look, this is a car, I'm like pretty sure.
 And it will carry over that information to all of the other frames.
 And they say in one of these high confidence mistakes, it basically detected the same tree
 as a giraffe over and over again.
 What I find particularly interesting is they do look at, so here they have this curve of
 on the bottom, you have confidence threshold, so how confident the model is.
 And on the y-axis, you have the number of false positives.
 And you can see that in the low confidence regime, the context RCNN has lower false positives
 than the single frame detector.
 And the green line here is when you only have positive boxes.
 So when you only include regions of interest where there is an actual object, which in
 this case is sort of hurtful, you also want the regions of interest where there is nothing
 because that helps you avoid false positives in other frames.
 That's why the orange line is below the green line.
 But strangely here in the high confidence regime, you can see that the single frame
 model has fewer false positives than the context RCNN.
 And I like the text that they have to this.
 In figure 7, we can see that adding empty representations reduces the number of false
 positives across all confidence threshold compared to the same model with only positive
 representations.
 We investigated the 100 highest confidence false positives from context RCNN and found
 that in almost all of them, in 97 out of 100, the model had correctly found and classified
 animals that were missed by human annotators.
 So basically these graphs are even underestimating how good that model is because the model appears
 to be better than the human annotators of the test set.
 I find that to be pretty, pretty impressive.
 And here you can see failure modes where they say, for example, when exploring the confident
 false positives on the snapshot Serengeti data set, the three out of 100 images, so
 whatever was not a human failure, where context RCNN erroneously detected an animal, where
 all of the same tree highly confidently predicted to be a giraffe.
 So this is a failure mode when the model is highly confident it might spill that over
 to other frames because we now aggregate the information within the same camera across
 the frames.
 To be said, of course, their train test split is such that there's not the same camera in
 the training data as in the testing data.
 They have entirely different cameras in the testing data than in the training data, just
 so there is no information leakage.
 So that's the model right here, how it works.
 It's pretty cool.
 It kind of wedges itself in between any single frame object detector that has these two stages.
 And it's a pretty neat idea to bring in context from the past or even the future of the same
 camera.
 Just a quick glance at the appendix, they have lots of different examples right here.
 In one example, their camera kind of fell over and they say, well, it still worked.
 The system was still able to kind of do attention across this failure, this kind of tipping
 over of the camera.
 They have more examples right here, which I find pretty impressive, like these super
 low light things where it correctly detects like the possum.
 And yeah, I invite you to check out the paper, the code they say should be out soon.
 And I'll see you next time.
 Bye bye.