RepNet: Counting Out Time - Class Agnostic Video Repetition Counting in the Wild (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper vision counting self-similarity temporal frames video repeating lines transformer attention cnn convolutional neural network repetitions periodicity period repeat actions kinetics countix
Counting repeated actions in a video is one of the easiest tasks for humans, yet remains incredibly hard for machines. RepNet achieves state-of-the-art by creating an information bottleneck in the form of a temporal self-similarity matrix, relating video frames to each other in a way that forces the model to surface the information relevant for counting. Along with that, the authors produce a new dataset for evaluating counting models.

OUTLINE:
0:00 - Intro & Overview
2:30 - Problem Statement
5:15 - Output & Loss
6:25 - Per-Frame Embeddings
11:20 - Temporal Self-Similarity Matrix
19:00 - Periodicity Predictor
25:50 - Architecture Recap
27:00 - Synthetic Dataset
30:15 - Countix Dataset
31:10 - Experiments
33:35 - Applications
35:30 - Conclusion & Comments

Paper Website: https://sites.google.com/view/repnet
Colab: https://colab.research.google.com/github/google-research/google-research/blob/master/repnet/repnet_colab.ipynb

Abstract:
We present an approach for estimating the period with which an action is repeated in a video. The crux of the approach lies in constraining the period prediction module to use temporal self-similarity as an intermediate representation bottleneck that allows generalization to unseen repetitions in videos in the wild. We train this model, called RepNet, with a synthetic dataset that is generated from a large unlabeled video collection by sampling short clips of varying lengths and repeating them with different periods and counts. This combination of synthetic data and a powerful yet constrained model, allows us to predict periods in a class-agnostic fashion. Our model substantially exceeds the state of the art performance on existing periodicity (PERTUBE) and repetition counting (QUVA) benchmarks. We also collect a new challenging dataset called Countix (~90 times larger than existing datasets) which captures the challenges of repetition counting in real-world videos.

Authors: Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Check out these videos on the top. Each one kind of contains a repeating action. So on the left you see someone doing jumping jacks in a fairly regular pattern. In the middle it gets a bit more difficult because what you see is a tennis ball bouncing and it bounces faster and faster and faster as time goes on. On the right you see that there is a short intro sequence before the repeating action, the person shoveling the cement, is displayed. So the goal here is to build an AI that can detect that a repeating action is happening and if it detects so that it can count how often this repeating action is happening. You can already see the difficulties here is not only the recognition itself but the fact that the repeating actions can be different and can be of different length and cannot always look the same and so on. So this paper uses these temporal self-similarity matrices that you see at the bottom here to achieve this and we're going to explore how that's happening. So the paper we'll look at is called Counting Out Time Class Agnostic Video Repetition Counting in the Wild by Debidada Dwebidi, Yusuf Aitar, Jonathan Thompson, Pierre Sermonet and Andrew Zizerman of Google Research and DeepMind. So as I already said this paper detects repeating actions and is able to count them and on a high level what they do is they encode the video using convolutional networks then they build these temporal self-similarity matrices between the frames in order to detect the repetitions and they decode that into the predictions using another neural network. This is all trained end to end and they also make a new data set for this task. So that's the high level. If you want to find out how exactly that's done I invite you to stick around because we'll go into the paper. If you like content like this don't forget to share it out, leave a like and tell me what you think of the paper in the comments. I do read the comments so yeah I'm very happy to read what you all think about it. Okay so as we already said they say we present an approach for estimating the period with which an action is repeated in a video and that's actually understating what the problem is here. The problem is many manyfold. As you can see on the right here even if you don't get what this self-similarity matrix is yet the outputs that you want are at the bottom. So what you want is first of all a per frame periodicity prediction. So that means that for each frame you want to know is there even a repeating action happening or not in that particular frame. You can see here at the beginning at the end of the video there is no repeating action and then in the middle there is a repeating action. So that's the first thing you want to know. The second thing is this per frame period length prediction. So for each frame that is part of a repeating action you want to know what is the period length of that repeating action and that can change throughout the video. So you need it per frame and once you have it per frame you can actually count the number of repetitions. So those are two problems already. The third problem that this paper solves is that there is no adequate data set to train a model like this it seems. For example this model right here this QUVA data set I believe has 100 data points and these are meant for testing. So you would build your system somewhere and then you would test it on those 100 data points. But they claim not even those are large and diverse enough for these systems to be evaluated. So they build. We also collect a new challenging data set called Countix which is 90 times larger than existing data sets which captures the challenges of repetition counting in real world videos and that actually consists of a train and test split. So you can also train a system using it. Let's dive into the architecture. I think this paper is a very very very cool example of even though we're in this deep learning paradigm where you just throw neural networks at a problem it's a very cool example that you can still achieve a lot by smartly constructing this because we used to achieve a lot by smartly constructing features and so on. In this case the goal is achieved by smartly constructing the architecture of the neural network itself to give you back a good performance on the particular task right here. Okay so if that tablet lets me actually scroll around let's go to the architecture. So figure two shows the architecture in more detail. So we'll go through it from the beginning to the end. Actually let's go let's go to the end so you know what is supposed to happen. So for each frame in this video what we'll need is a period length and a periodicity. These are two so the bottom is a binary variable and the top is a it's actually a number but it is predicted in kind of binned as a classification task. It doesn't really matter we need two outputs that we can compare with the labels right. In this case the videos are of length 64 so there's 64 frames per video and for each of those frames we want a period length and for each of those frames we want a periodicity binary prediction. And that as I said we compare it with our labels and then we can calculate a loss. So this is the loss the loss is at the end comparing these two labels and then everything else is trained using back propagation on that loss. Okay so now with that in mind let's go to the beginning. So the video is taken and fed through an encoder in order to produce these per frame embeddings. So we want an embedding for each frame that means for each of the 64 frames we want to obtain one vector of length 512 that describes that particular frame in terms that the model can understand. And we do that using an encoder. Now the encoder it has a bunch of parts to it. It's not just a blob as you see right here. So the encoder consists of three things. First of all there's this convolutional feature extractor which is a ResNet 50 architecture. It's simply a convolution and you let the convolutional neural network run on each frame independently. This is simply a feature extractor from images right like you know it from any other image processing task. But of course here we have a video. So it would be nice if the frames knew something about the other frames right. Especially if you think of something like a jumping jack. If you are in this position right here. It doesn't tell you everything about that video frame. If you consider the frame before it and maybe the frame before it is this and maybe the frame after it is that, you can clearly see that the hands or the arms are in an upward motion. So the next step of the encoder tries to integrate that temporal information into the embeddings and that is achieved via a 3D convolution. So once we process each frame individually then we feed it into one layer through a layer of 3D convolution to add local temporal information to the per frame features. So if you don't know what a 3D convolution is, this already drew. So in a 2D convolution what you want to do is you want to have this filter right here which is a 2D filter for each of the channels and you slide it across the image like this. And you can have multiple channels of the input image right here and you can actually do this multiple times which corresponds to multiple output channels of the filters but the actual convolution is happening in two dimensions. So the sliding is across the width and the height of the image. In contrast to that if you have a 3D convolution and you have the same input stack of images and now this we have to you have to pay attention here. This right this stack right here is the individual channels of one image so of one video frame. This stack right here that I'm drawing these are the video frames stacked. Now each of these stacked video frames can have multiple channels resulting from its 2D convolution or even just RGB. So I can't really draw in four dimensions but now we stack the video frames and now our kernel our filter will be also in the direction of the video frame. So I don't know if this is really recognizable if I draw it like this but as you can see the kernel is not only 2D but 3D and now the sliding so if we have actually more than that the sliding is not only done into the direction of height and width but also into the direction of depth so that each of the frames each of the video frames right here can incorporate information from its immediate neighbors in case of a 3x3x3 filter or even more neighbors but here we use a 3x3x3 filter. Okay so that's that's how we obtain these embeddings right here and at the end there is a dimension reduction like a max pooling or something but ultimately what you'll end up with is for each of the 64 video frames you get one vector and that vector mainly describes that particular video frame like if you consider the green one here but also contains information from the video frame before it and from the video frame after it. Okay so that's sort of and and that's that's the so the temporal convolution is not to detect the periodic actions because it's just one frame into the future and into the past it is more like what we said here in order to give you extra information of what's happening in a particular frame because especially for periodicity it's actually important if the arms are going up or down. Cool so then comes the heart of the architecture. The heart of the architecture is this temporal self-similarity matrix. Now what does this do? This is relating the frames to each other and important to note here this is just a single channel image so there is no other ones of these. For the entire video frame sequence you have one 64x64 matrix and all the signal has to go through this matrix right everything from here is only through this matrix there's no residual connections there is no skip connections that's all that's your information bottleneck and by having a bottleneck like this these the authors here force the model to basically do a good job at making this temporal self-similarity happen if it wants to achieve a low loss and that's what I mean by you can achieve a lot by having a smart architecture. This temporal self-similarity matrix is actually not learned this can be computed deterministically from the embeddings right here so what you do is you each row here corresponds to one frame so you take each frame that corresponds to a row and you calculate simply the distance to or the similarity with each of the other frames so this is as you can see it's 64x64 so frame i here is simply compared to each of the other frames j and depending on whether that embedding is very similar to the embedding of frame j this number is going to be high or low. Now there's also a after that you do a softmax across here such that this is going to be like a distribution and not just raw numbers but that's ultimately not that important so what you can see is that the diagonal is very prominent and that makes sense because technically the diagonal is always one right if for example if you use the inner product as a similarity the diagonal should always be one but here we have the softmax so it's not but ultimately we can say that any frame should be very similar to its is going to be very similar to itself so that's why but then you can see right here there's a pattern emerging and that pattern is these diagonal lines in this direction as you can see and what what does that mean they actually have a larger version of this down here so what does the diagonal pattern mean it means that so here the diagonal that's okay frame i is very similar to frame i cool but the other lines what do they mean so if i look at frame i right here it is also very similar to this frame j now this wouldn't be further you know frames are similar but the line means that if i have if i look 10 frames later i plus 10 that's very similar to j plus 10 and if i now look at i plus 20 frame that's very similar to j plus 20 so this is the this is here why the the pattern is emerging of a line because if i go 10 frames into the future it's similar to the other one 10 frames into the future and so on and that means the line indicates that this entire sequence starting from i 20 frames into the future is repeated starting from j actually j is earlier here so but you get the point and if i have a bunch of these lines that means that this subsequence is repeated again and again and again throughout the video so each of these lines is basically one repetition of the sequence from the middle here at some other point in the video and that's pretty fascinating and that's these these self-similarity matrices that's what they're sort of showing now they don't use the inner product here as a self-similarity metric they actually use as you can see right here they use the negative square distance but the effect is the same so negative square distance followed by a row wise softmax operation so you could say hi we are basically done having this self-similarity matrix what we could do let's let's say we could train it we don't worry about how to train it we could simply take each row right here and we plot the intensities across that row and that's maybe you know like something like this and then there's the diagonal is a bit higher okay we could just use a heuristic to detect these bumps here basically calculate the length count the bumps and calculate the period length right that that should be pretty easy with like a simple heuristic but the authors here they want more they want to solve more problems so what are some of the problems we already saw some of the problems namely for example here is the hammer throw so the hammer throw starts out slow and gets faster and faster and faster and you can see this pretty clearly at the lines right here namely if you go through time so you start off here and you go through time you can see that the distance to the next line here is fairly large but you go through time further the distance gets shorter you go through time further the distance gets even shorter so these pattern of lines here that's kind of converging towards that it indicates that this repeating action gets faster and faster and faster this is nice to see here at the bouncy ball example where you can see it starts out pretty slow but it gets faster and faster and here you if you if you have this full thing right here that basically means all the frames are self similar to each other which basically means if you stop the video right that's if you have 10 frames in a row the same thing the ball is just lying on the ground all of these frames will be self similar so there's probably no bouncing happening down here you can see pretty well from the pattern what happens and here in this mixing concrete example that we saw at the beginning you can see that at the beginning at the end there's this intro and outro sequence and only in the middle is there a repeating action and that's indicated by this line pattern is only at in the middle of the videos only between here and here so it's it's going to be pretty difficult to just have a heuristic that reads out these periodic action periodicity and in a true deep learning fashion the the authors here oh sorry maybe you can't see that i've shifted my recording window so maybe sometimes something's out of frame and you have to yell at me if i do that please so i hope you you saw this that you have the ever the speeding up here and here they're visible in the pattern and then here you have the beginning sequence the end sequence that have no repeating pattern and the repeating pattern only merged in the middle so the authors want to do this through of course a deep learning network they want to read out the periodicities not through a heuristic but using a deep network you know respectable that's at the times we live in so what do they do first of all you have to see right here everything that happens from here as i understand it is per frame so they simply take a row of this matrix right here like this red line and that is independently pulled through to the end so there is no interaction happening anymore between the individual frame data the only interaction that happens is a little bit here at the temporal convolutions but the only real interaction between the frames is happening through the self-similarity matrix and again this is the information bottleneck that the authors force the information through everything happening from here no that's actually not right there is this convolution right here but still this is the information bottleneck you have to go through so right here we process this image using a convolution so this is an image right and we can process it using a convolutional neural network so what we do is we have a 64 by 64 image in one channel we simply up sample that not up sample but we expand the channels to 32 channels now as i said it's pretty easy to think we can just go to the end here use a conv net to produce our final 512 by so 512 embeddings we have here again 64 by 64 that we then use to predict the final result but the authors here do something different they do transformer layers in the middle but only per frame so what does it mean so here you up sample to 32 channels and then that means that one of these blocks right here one of these blocks corresponds to one row in the self-similarity matrix which corresponds to one frame and from now on so from now on i want to say what i said before from now on it's all just this one block they are independent of each other okay so you take this one block and you feed it through a transformer to achieve at your final embedding of 512 and it's probably best if we read what they say about it okay so if we're given this self-similarity matrices matrix they consist of row each row is the per frame self-similarity representation and generates two outputs the per frame period length estimation and the per frame binary periodicity classification note that both l and p are vectors and their elements are per frame predictions okay the architecture of the period predictor module can be viewed in figure two note that the predictors share a common architecture and waits until the last classification phase the shared processing pipeline starts with starts with 32 2d convolutional filters of size three by three followed by a transformer layer which uses a multi-headed attention with trainable positional embeddings in the form of a 64 length variable that is learned by training okay it's i guess the transformers learned by training and the positional embeddings are also learned by training that's fairly common we use four heads with 512 dimension in the transformer by the way if you don't know what a transformer is watch the video on attention is all you need i made one it's very popular yeah so with each head being 128 dimensions in size after the shared pipeline we have two classifiers period length classifier and periodicity classifier tau sorry this is fine this is tau each of them consists of two fully connected layers of size 512 so i guess the the the pipeline here is pretty simple the question could be why do they use a transformer and not simply another convolutional network so here they up sample the image as we saw into 32 channels and then they simply want to take one of these one of these blocks here and that corresponds a little bit so we have for one frame right what does it mean we have basically we have 64 by 32 things so the 64 things it's this one frames temporal connection to each other frame given that you know comes from this self-similarity matrix so it kind of relates this frame that we're considering to each of the other frames and each of this each of these entries is a 32 size vector this is sort of a this is you can consider like a sequence of 64 things 64 embeddings so to use a transformer here it's pretty natural if you think of this as like a sequence transformation task i i would guess so the transformer can if there are these peaks right here like we saw right here the transformer can make very good sense of that because of course the attention mechanism from a one peak it can attend to all the other peaks and can sort of relate the different peaks to each other and then determine the periodicity length whereas with a convolutional network i guess that's going to be a lot harder because of the sort of invariance built into the convolution i'm not sure maybe they also it just worked better but that's how i think about it it's that for a given frame you basically have a sequence classification or a set classification task and the attention mechanism allows you to in one single step connect each peak with each other peak or each information with each other information in this sequence all right so at the end you have just fully connected layers again only on a per frame basis and that will give you the output and again you compare this to the label and you back prop through everything everything here is differentiable so all of this is trained to achieve minimum possible loss and because you train everything to achieve minimal possible loss you make this encoder right here which is the crucial part because the encoder is must give you good embeddings which must give you a sensible self-similarity matrix right you train the encoder to encode things that are relevant for the task and that's what makes the whole thing work okay so we've gone through the architecture now the problem right here is the the data set so they also go into how they do inference they can actually do a bunch of things like play the video at different speeds and then look at what each of the predictions so if a double speed it predicts half the period length then you can be more sure and so on so that's pretty cool but they go into another point right here and that's the data set so they produce this countix data set but also on the other hand which is something I also find very cool is they produce a synthetic data set so here they say we train with synthetic repetitions and that can be sort of I didn't know what to think of it at first I was just like huh but then it's pretty cool so if you have a video with these these are the frames of the video right so the video goes in this temporal direction what you can do is simply go here go through these frames and just repeat these frames and repeat them and repeat them and at the end you have these frames right and then you have a data set and if you if you assume that most videos do not naturally contain repeating actions right most videos are just videos they're not videos of something repeating then you can safely assume that these parts here are non repeating so and these parts here are repeating this is one of the labels that you need right the problem with synthetic data set is always to have the labels and also you know how many there are because you can simply count the number of times that you go through it you can even make it faster slower and so on so this synthetic approach is pretty cool and especially the bottom right here because this might be kind of hacky because each time each time you jump from the end of one of those arrows to the beginning right you have kind of a hack in the indie video because you know it's not continuous so what you can do and this is the the bottom here you can do this reversal technique where you go to the end and then you play the frames backwards and then you play the frames forwards again backwards again forwards again and then you go out here and that gives you one continuous motion right if someone if it's simply a video of someone lifting their hand like it starts out down here and it goes here and it goes here and then if you do this technique it would go down again down again up again up again and so on so that's you know i think it's a fairly smart technique honestly now they tried this and it doesn't work super well so what they also have to do is they have to do manual camera motion augmentation so that's so camera motion augmentation it basically means that if you just do a repeating action like this it's sort of i guess it's too monotonic it doesn't really cover real videos with repeating actions so what they do is they kind of simulate a moving camera and you simulate that much like you would do image augmentation so you can rotate the camera over time you can translate it you can scale it differently and through if you do that throughout the video and you change it around how the camera moves then that appears to work fairly well so if they now compare this and their data set they perform pretty well so in their data set they take this kinetics data set and they crowdsource the label and the tasks in the data set they're pretty diverse as you can see right here so you have sports like rope training mountain climbers but you have also things like playing ukulele exercising arms slicing an onion and so on and you can see that the repetition count is fairly diverse as well so from one or two repetitions per video it goes to 50 or so and the period length is also between one and five seconds though as you as i already said you don't have to you don't have to count on that because you can always play the video slower or faster and then determine other periodicities so in their experiment first of all they perform pretty well and they show that if they train on their data set and on the synthetic data set they perform better than if they just train on the synthetic or they just train on their data set they also show pretty clearly that the addition of this temporal self-similarity matrix helps tremendously you can see right here in each of these boxes is the comparison and this obi I think is the off by one error so it kind of forgives you if you're off by one count but otherwise you get a zero if you're wrong and you can see that the self-similarity matrix helps tremendously they also compare with some other architectural choices instead of the transformer I guess yeah so I guess they just take it because it performs pretty well and they do a lot of lot of ablations but what I particularly appreciate is that they do something like this so what they do at the end went once they've trained the architectures they do a 1d PCA protection of the encoder features over time now the encoder features they were 512 dimensional right this is the thing before it goes into the self-similarity matrix so those we said the encoder is the crucial part here because it needs to take the video and encode things that make them accessible to calculating the self-similarity now they do a 1d PCA so a projection into one dimension of these features and you can already see at this one dimensional projection that the periodicity here is clearly clearly visible namely for example right here every time up here is when the legs are up and every time down here is when the legs are down right here so that is very very impressive and that kind of that really shows that the model is doing what you claim what you claim that it's doing like I'm almost more interested in experiments like this than in and in these numbers right here because the numbers could always be because you've just thrown more stuff at it right so they go over a bunch of possible applications of their model so first of all you can do something like as we can see repetition counting from videos you can do periodicity detection those were the things that the model is trained to do but there's also a bunch of things that the model can now implicitly do namely something like change inspection where they say look if someone's chopping this pineapple right here then at the end of each of the repetitions there is something that changed namely the number of slices of pineapple is it bread is it I can't I think it's pineapple okay so the number of slices or pieces right here changes so in essence this could be the base for another model estimating whatever changed or training to recognize numbers of pieces and so on also you can detect the speed so the speed of a repeating action if you perform something slow or fast this model can implicitly do it and this they call cross-period retrieval so if you know when the repetitions are you know that okay maybe the first frame so always on the upswing right here these should all these should all be fairly similar visually right as with respect to the repeating action so you can see that even though this whenever the kid in the swing here is close it looks fairly different in in a purely visual sense in a pixel sense but it is at the same point in the repeating action and that's you know that's that's pretty cool so you can technically retrieve related things even though they visually they don't look similar that much yeah that that's the the kind of applications here are probably many many fold and I also think that so in this measure of intelligence paper by françois choulet he basically claims that this is one of the innate abilities of humans they can count you know they can count things this is something you're basically born with and maybe this thing right here will become sort of a staple staple component for many other things that we build AI on I would not be surprised but maybe it will just fade into history I think it's pretty cool project especially you know the the architectural choice here to pull everything through this self-similarity matrix and the you know just just looking at this matrix already makes you kind of know that this thing works alright this was it from me let me know in the comments what you think about the paper check out the website the website has a lot of video demo examples of what they're doing I think the data set as well and yeah I'll see you next time bye bye
 Hi there! Check out these videos on the top. Each one kind of contains a repeating action.
 So on the left you see someone doing jumping jacks in a fairly regular pattern. In the
 middle it gets a bit more difficult because what you see is a tennis ball bouncing and
 it bounces faster and faster and faster as time goes on. On the right you see that there
 is a short intro sequence before the repeating action, the person shoveling the cement, is
 displayed. So the goal here is to build an AI that can detect that a repeating action
 is happening and if it detects so that it can count how often this repeating action
 is happening. You can already see the difficulties here is not only the recognition itself but
 the fact that the repeating actions can be different and can be of different length and
 cannot always look the same and so on. So this paper uses these temporal self-similarity
 matrices that you see at the bottom here to achieve this and we're going to explore how
 that's happening. So the paper we'll look at is called Counting Out Time Class Agnostic
 Video Repetition Counting in the Wild by Debidada Dwebidi, Yusuf Aitar, Jonathan Thompson, Pierre
 Sermonet and Andrew Zizerman of Google Research and DeepMind. So as I already said this paper
 detects repeating actions and is able to count them and on a high level what they do is they
 encode the video using convolutional networks then they build these temporal self-similarity
 matrices between the frames in order to detect the repetitions and they decode that into
 the predictions using another neural network. This is all trained end to end and they also
 make a new data set for this task. So that's the high level. If you want to find out how
 exactly that's done I invite you to stick around because we'll go into the paper. If
 you like content like this don't forget to share it out, leave a like and tell me what
 you think of the paper in the comments. I do read the comments so yeah I'm very happy
 to read what you all think about it. Okay so as we already said they say we present
 an approach for estimating the period with which an action is repeated in a video and
 that's actually understating what the problem is here. The problem is many manyfold. As
 you can see on the right here even if you don't get what this self-similarity matrix
 is yet the outputs that you want are at the bottom. So what you want is first of all a
 per frame periodicity prediction. So that means that for each frame you want to know
 is there even a repeating action happening or not in that particular frame. You can see
 here at the beginning at the end of the video there is no repeating action and then in the
 middle there is a repeating action. So that's the first thing you want to know. The second
 thing is this per frame period length prediction. So for each frame that is part of a repeating
 action you want to know what is the period length of that repeating action and that can
 change throughout the video. So you need it per frame and once you have it per frame you
 can actually count the number of repetitions. So those are two problems already. The third
 problem that this paper solves is that there is no adequate data set to train a model like
 this it seems. For example this model right here this QUVA data set I believe has 100
 data points and these are meant for testing. So you would build your system somewhere and
 then you would test it on those 100 data points. But they claim not even those are large and
 diverse enough for these systems to be evaluated. So they build. We also collect a new challenging
 data set called Countix which is 90 times larger than existing data sets which captures
 the challenges of repetition counting in real world videos and that actually consists of
 a train and test split. So you can also train a system using it. Let's dive into the architecture.
 I think this paper is a very very very cool example of even though we're in this deep
 learning paradigm where you just throw neural networks at a problem it's a very cool example
 that you can still achieve a lot by smartly constructing this because we used to achieve
 a lot by smartly constructing features and so on. In this case the goal is achieved by
 smartly constructing the architecture of the neural network itself to give you back a good
 performance on the particular task right here. Okay so if that tablet lets me actually scroll
 around let's go to the architecture. So figure two shows the architecture in more detail.
 So we'll go through it from the beginning to the end. Actually let's go let's go to
 the end so you know what is supposed to happen. So for each frame in this video what we'll
 need is a period length and a periodicity. These are two so the bottom is a binary variable
 and the top is a it's actually a number but it is predicted in kind of binned as a classification
 task. It doesn't really matter we need two outputs that we can compare with the labels
 right. In this case the videos are of length 64 so there's 64 frames per video and for
 each of those frames we want a period length and for each of those frames we want a periodicity
 binary prediction. And that as I said we compare it with our labels and then we can calculate
 a loss. So this is the loss the loss is at the end comparing these two labels and then
 everything else is trained using back propagation on that loss. Okay so now with that in mind
 let's go to the beginning. So the video is taken and fed through an encoder in order
 to produce these per frame embeddings. So we want an embedding for each frame that means
 for each of the 64 frames we want to obtain one vector of length 512 that describes that
 particular frame in terms that the model can understand. And we do that using an encoder.
 Now the encoder it has a bunch of parts to it. It's not just a blob as you see right
 here. So the encoder consists of three things. First of all there's this convolutional feature
 extractor which is a ResNet 50 architecture. It's simply a convolution and you let the
 convolutional neural network run on each frame independently. This is simply a feature extractor
 from images right like you know it from any other image processing task. But of course
 here we have a video. So it would be nice if the frames knew something about the other
 frames right. Especially if you think of something like a jumping jack. If you are in this position
 right here. It doesn't tell you everything about that video frame. If you consider the
 frame before it and maybe the frame before it is this and maybe the frame after it is
 that, you can clearly see that the hands or the arms are in an upward motion. So the next
 step of the encoder tries to integrate that temporal information into the embeddings and
 that is achieved via a 3D convolution. So once we process each frame individually then
 we feed it into one layer through a layer of 3D convolution to add local temporal information
 to the per frame features. So if you don't know what a 3D convolution is, this already
 drew. So in a 2D convolution what you want to do is you want to have this filter right
 here which is a 2D filter for each of the channels and you slide it across the image
 like this. And you can have multiple channels of the input image right here and you can
 actually do this multiple times which corresponds to multiple output channels of the filters
 but the actual convolution is happening in two dimensions. So the sliding is across the
 width and the height of the image. In contrast to that if you have a 3D convolution and you
 have the same input stack of images and now this we have to you have to pay attention
 here. This right this stack right here is the individual channels of one image so of
 one video frame. This stack right here that I'm drawing these are the video frames stacked.
 Now each of these stacked video frames can have multiple channels resulting from its
 2D convolution or even just RGB. So I can't really draw in four dimensions but now we
 stack the video frames and now our kernel our filter will be also in the direction of
 the video frame. So I don't know if this is really recognizable if I draw it like this
 but as you can see the kernel is not only 2D but 3D and now the sliding so if we have
 actually more than that the sliding is not only done into the direction of height and
 width but also into the direction of depth so that each of the frames each of the video
 frames right here can incorporate information from its immediate neighbors in case of a
 3x3x3 filter or even more neighbors but here we use a 3x3x3 filter. Okay so that's that's
 how we obtain these embeddings right here and at the end there is a dimension reduction
 like a max pooling or something but ultimately what you'll end up with is for each of the
 64 video frames you get one vector and that vector mainly describes that particular video
 frame like if you consider the green one here but also contains information from the video
 frame before it and from the video frame after it. Okay so that's sort of and and that's
 that's the so the temporal convolution is not to detect the periodic actions because
 it's just one frame into the future and into the past it is more like what we said here
 in order to give you extra information of what's happening in a particular frame because
 especially for periodicity it's actually important if the arms are going up or down. Cool so
 then comes the heart of the architecture. The heart of the architecture is this temporal
 self-similarity matrix. Now what does this do? This is relating the frames to each other
 and important to note here this is just a single channel image so there is no other
 ones of these. For the entire video frame sequence you have one 64x64 matrix and all
 the signal has to go through this matrix right everything from here is only through this
 matrix there's no residual connections there is no skip connections that's all that's
 your information bottleneck and by having a bottleneck like this these the authors here
 force the model to basically do a good job at making this temporal self-similarity happen
 if it wants to achieve a low loss and that's what I mean by you can achieve a lot by having
 a smart architecture. This temporal self-similarity matrix is actually not learned this can be
 computed deterministically from the embeddings right here so what you do is you each row
 here corresponds to one frame so you take each frame that corresponds to a row and you
 calculate simply the distance to or the similarity with each of the other frames so this is as
 you can see it's 64x64 so frame i here is simply compared to each of the other frames
 j and depending on whether that embedding is very similar to the embedding of frame
 j this number is going to be high or low. Now there's also a after that you do a softmax
 across here such that this is going to be like a distribution and not just raw numbers
 but that's ultimately not that important so what you can see is that the diagonal is very
 prominent and that makes sense because technically the diagonal is always one right if for example
 if you use the inner product as a similarity the diagonal should always be one but here
 we have the softmax so it's not but ultimately we can say that any frame should be very similar
 to its is going to be very similar to itself so that's why but then you can see right here
 there's a pattern emerging and that pattern is these diagonal lines in this direction
 as you can see and what what does that mean they actually have a larger version of this
 down here so what does the diagonal pattern mean it means that so here the diagonal that's
 okay frame i is very similar to frame i cool but the other lines what do they mean so if
 i look at frame i right here it is also very similar to this frame j now this wouldn't
 be further you know frames are similar but the line means that if i have if i look 10
 frames later i plus 10 that's very similar to j plus 10 and if i now look at i plus 20
 frame that's very similar to j plus 20 so this is the this is here why the the pattern
 is emerging of a line because if i go 10 frames into the future it's similar to the other
 one 10 frames into the future and so on and that means the line indicates that this entire
 sequence starting from i 20 frames into the future is repeated starting from j actually
 j is earlier here so but you get the point and if i have a bunch of these lines that
 means that this subsequence is repeated again and again and again throughout the video so
 each of these lines is basically one repetition of the sequence from the middle here at some
 other point in the video and that's pretty fascinating and that's these these self-similarity
 matrices that's what they're sort of showing now they don't use the inner product here
 as a self-similarity metric they actually use as you can see right here they use the
 negative square distance but the effect is the same so negative square distance followed
 by a row wise softmax operation so you could say hi we are basically done having this self-similarity
 matrix what we could do let's let's say we could train it we don't worry about how to
 train it we could simply take each row right here and we plot the intensities across that
 row and that's maybe you know like something like this and then there's the diagonal is
 a bit higher okay we could just use a heuristic to detect these bumps here basically calculate
 the length count the bumps and calculate the period length right that that should be pretty
 easy with like a simple heuristic but the authors here they want more they want to solve
 more problems so what are some of the problems we already saw some of the problems namely
 for example here is the hammer throw so the hammer throw starts out slow and gets faster
 and faster and faster and you can see this pretty clearly at the lines right here namely
 if you go through time so you start off here and you go through time you can see that the
 distance to the next line here is fairly large but you go through time further the distance
 gets shorter you go through time further the distance gets even shorter so these pattern
 of lines here that's kind of converging towards that it indicates that this repeating action
 gets faster and faster and faster this is nice to see here at the bouncy ball example
 where you can see it starts out pretty slow but it gets faster and faster and here you
 if you if you have this full thing right here that basically means all the frames are self
 similar to each other which basically means if you stop the video right that's if you
 have 10 frames in a row the same thing the ball is just lying on the ground all of these
 frames will be self similar so there's probably no bouncing happening down here you can see
 pretty well from the pattern what happens and here in this mixing concrete example that
 we saw at the beginning you can see that at the beginning at the end there's this intro
 and outro sequence and only in the middle is there a repeating action and that's indicated
 by this line pattern is only at in the middle of the videos only between here and here so
 it's it's going to be pretty difficult to just have a heuristic that reads out these
 periodic action periodicity and in a true deep learning fashion the the authors here
 oh sorry maybe you can't see that i've shifted my recording window so maybe sometimes something's
 out of frame and you have to yell at me if i do that please so i hope you you saw this
 that you have the ever the speeding up here and here they're visible in the pattern and
 then here you have the beginning sequence the end sequence that have no repeating pattern
 and the repeating pattern only merged in the middle so the authors want to do this through
 of course a deep learning network they want to read out the periodicities not through
 a heuristic but using a deep network you know respectable that's at the times we live in
 so what do they do first of all you have to see right here everything that happens from
 here as i understand it is per frame so they simply take a row of this matrix right here
 like this red line and that is independently pulled through to the end so there is no interaction
 happening anymore between the individual frame data the only interaction that happens is
 a little bit here at the temporal convolutions but the only real interaction between the
 frames is happening through the self-similarity matrix and again this is the information bottleneck
 that the authors force the information through everything happening from here no that's actually
 not right there is this convolution right here but still this is the information bottleneck
 you have to go through so right here we process this image using a convolution so this is
 an image right and we can process it using a convolutional neural network so what we
 do is we have a 64 by 64 image in one channel we simply up sample that not up sample but
 we expand the channels to 32 channels now as i said it's pretty easy to think we can
 just go to the end here use a conv net to produce our final 512 by so 512 embeddings
 we have here again 64 by 64 that we then use to predict the final result but the authors
 here do something different they do transformer layers in the middle but only per frame so
 what does it mean so here you up sample to 32 channels and then that means that one of
 these blocks right here one of these blocks corresponds to one row in the self-similarity
 matrix which corresponds to one frame and from now on so from now on i want to say what
 i said before from now on it's all just this one block they are independent of each other
 okay so you take this one block and you feed it through a transformer to achieve at your
 final embedding of 512 and it's probably best if we read what they say about it okay so
 if we're given this self-similarity matrices matrix they consist of row each row is the
 per frame self-similarity representation and generates two outputs the per frame period
 length estimation and the per frame binary periodicity classification note that both
 l and p are vectors and their elements are per frame predictions okay the architecture
 of the period predictor module can be viewed in figure two note that the predictors share
 a common architecture and waits until the last classification phase the shared processing
 pipeline starts with starts with 32 2d convolutional filters of size three by three followed by
 a transformer layer which uses a multi-headed attention with trainable positional embeddings
 in the form of a 64 length variable that is learned by training okay it's i guess the
 transformers learned by training and the positional embeddings are also learned by training that's
 fairly common we use four heads with 512 dimension in the transformer by the way if you don't
 know what a transformer is watch the video on attention is all you need i made one it's
 very popular yeah so with each head being 128 dimensions in size after the shared pipeline
 we have two classifiers period length classifier and periodicity classifier tau sorry this
 is fine this is tau each of them consists of two fully connected layers of size 512
 so i guess the the the pipeline here is pretty simple the question could be why do they use
 a transformer and not simply another convolutional network so here they up sample the image as
 we saw into 32 channels and then they simply want to take one of these one of these blocks
 here and that corresponds a little bit so we have for one frame right what does it mean
 we have basically we have 64 by 32 things so the 64 things it's this one frames temporal
 connection to each other frame given that you know comes from this self-similarity matrix
 so it kind of relates this frame that we're considering to each of the other frames and
 each of this each of these entries is a 32 size vector this is sort of a this is you
 can consider like a sequence of 64 things 64 embeddings so to use a transformer here
 it's pretty natural if you think of this as like a sequence transformation task i i would
 guess so the transformer can if there are these peaks right here like we saw right here
 the transformer can make very good sense of that because of course the attention mechanism
 from a one peak it can attend to all the other peaks and can sort of relate the different
 peaks to each other and then determine the periodicity length whereas with a convolutional
 network i guess that's going to be a lot harder because of the sort of invariance built into
 the convolution i'm not sure maybe they also it just worked better but that's how i think
 about it it's that for a given frame you basically have a sequence classification or a set classification
 task and the attention mechanism allows you to in one single step connect each peak with
 each other peak or each information with each other information in this sequence all right
 so at the end you have just fully connected layers again only on a per frame basis and
 that will give you the output and again you compare this to the label and you back prop
 through everything everything here is differentiable so all of this is trained to achieve minimum
 possible loss and because you train everything to achieve minimal possible loss you make
 this encoder right here which is the crucial part because the encoder is must give you
 good embeddings which must give you a sensible self-similarity matrix right you train the
 encoder to encode things that are relevant for the task and that's what makes the whole
 thing work okay so we've gone through the architecture now the problem right here is
 the the data set so they also go into how they do inference they can actually do a bunch
 of things like play the video at different speeds and then look at what each of the predictions
 so if a double speed it predicts half the period length then you can be more sure and
 so on so that's pretty cool but they go into another point right here and that's the data
 set so they produce this countix data set but also on the other hand which is something
 I also find very cool is they produce a synthetic data set so here they say we train with synthetic
 repetitions and that can be sort of I didn't know what to think of it at first I was just
 like huh but then it's pretty cool so if you have a video with these these are the frames
 of the video right so the video goes in this temporal direction what you can do is simply
 go here go through these frames and just repeat these frames and repeat them and repeat them
 and at the end you have these frames right and then you have a data set and if you if
 you assume that most videos do not naturally contain repeating actions right most videos
 are just videos they're not videos of something repeating then you can safely assume that
 these parts here are non repeating so and these parts here are repeating this is one
 of the labels that you need right the problem with synthetic data set is always to have
 the labels and also you know how many there are because you can simply count the number
 of times that you go through it you can even make it faster slower and so on so this synthetic
 approach is pretty cool and especially the bottom right here because this might be kind
 of hacky because each time each time you jump from the end of one of those arrows to the
 beginning right you have kind of a hack in the indie video because you know it's not
 continuous so what you can do and this is the the bottom here you can do this reversal
 technique where you go to the end and then you play the frames backwards and then you
 play the frames forwards again backwards again forwards again and then you go out here and
 that gives you one continuous motion right if someone if it's simply a video of someone
 lifting their hand like it starts out down here and it goes here and it goes here and
 then if you do this technique it would go down again down again up again up again and
 so on so that's you know i think it's a fairly smart technique honestly now they tried this
 and it doesn't work super well so what they also have to do is they have to do manual
 camera motion augmentation so that's so camera motion augmentation it basically means that
 if you just do a repeating action like this it's sort of i guess it's too monotonic it
 doesn't really cover real videos with repeating actions so what they do is they kind of simulate
 a moving camera and you simulate that much like you would do image augmentation so you
 can rotate the camera over time you can translate it you can scale it differently and through
 if you do that throughout the video and you change it around how the camera moves then
 that appears to work fairly well so if they now compare this and their data set they perform
 pretty well so in their data set they take this kinetics data set and they crowdsource
 the label and the tasks in the data set they're pretty diverse as you can see right here so
 you have sports like rope training mountain climbers but you have also things like playing
 ukulele exercising arms slicing an onion and so on and you can see that the repetition
 count is fairly diverse as well so from one or two repetitions per video it goes to 50
 or so and the period length is also between one and five seconds though as you as i already
 said you don't have to you don't have to count on that because you can always play the video
 slower or faster and then determine other periodicities so in their experiment first
 of all they perform pretty well and they show that if they train on their data set and on
 the synthetic data set they perform better than if they just train on the synthetic or
 they just train on their data set they also show pretty clearly that the addition of this
 temporal self-similarity matrix helps tremendously you can see right here in each of these boxes
 is the comparison and this obi I think is the off by one error so it kind of forgives
 you if you're off by one count but otherwise you get a zero if you're wrong and you can
 see that the self-similarity matrix helps tremendously they also compare with some other
 architectural choices instead of the transformer I guess yeah so I guess they just take it
 because it performs pretty well and they do a lot of lot of ablations but what I particularly
 appreciate is that they do something like this so what they do at the end went once
 they've trained the architectures they do a 1d PCA protection of the encoder features
 over time now the encoder features they were 512 dimensional right this is the thing before
 it goes into the self-similarity matrix so those we said the encoder is the crucial part
 here because it needs to take the video and encode things that make them accessible to
 calculating the self-similarity now they do a 1d PCA so a projection into one dimension
 of these features and you can already see at this one dimensional projection that the
 periodicity here is clearly clearly visible namely for example right here every time up
 here is when the legs are up and every time down here is when the legs are down right
 here so that is very very impressive and that kind of that really shows that the model is
 doing what you claim what you claim that it's doing like I'm almost more interested in experiments
 like this than in and in these numbers right here because the numbers could always be because
 you've just thrown more stuff at it right so they go over a bunch of possible applications
 of their model so first of all you can do something like as we can see repetition counting
 from videos you can do periodicity detection those were the things that the model is trained
 to do but there's also a bunch of things that the model can now implicitly do namely something
 like change inspection where they say look if someone's chopping this pineapple right
 here then at the end of each of the repetitions there is something that changed namely the
 number of slices of pineapple is it bread is it I can't I think it's pineapple okay
 so the number of slices or pieces right here changes so in essence this could be the base
 for another model estimating whatever changed or training to recognize numbers of pieces
 and so on also you can detect the speed so the speed of a repeating action if you perform
 something slow or fast this model can implicitly do it and this they call cross-period retrieval
 so if you know when the repetitions are you know that okay maybe the first frame so always
 on the upswing right here these should all these should all be fairly similar visually
 right as with respect to the repeating action so you can see that even though this whenever
 the kid in the swing here is close it looks fairly different in in a purely visual sense
 in a pixel sense but it is at the same point in the repeating action and that's you know
 that's that's pretty cool so you can technically retrieve related things even though they visually
 they don't look similar that much yeah that that's the the kind of applications here are
 probably many many fold and I also think that so in this measure of intelligence paper by
 françois choulet he basically claims that this is one of the innate abilities of humans
 they can count you know they can count things this is something you're basically born with
 and maybe this thing right here will become sort of a staple staple component for many
 other things that we build AI on I would not be surprised but maybe it will just fade into
 history I think it's pretty cool project especially you know the the architectural choice here
 to pull everything through this self-similarity matrix and the you know just just looking
 at this matrix already makes you kind of know that this thing works alright this was it
 from me let me know in the comments what you think about the paper check out the website
 the website has a lot of video demo examples of what they're doing I think the data set
 as well and yeah I'll see you next time bye bye