Object-Centric Learning with Slot Attention (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper google ethz vision objects slots attention mechanism gru lstm routing capsules permutation invariant encoder set detr embeddings transformer weight sharing disentanglement render tetris clevr cnn convolutional neural network attention
Visual scenes are often comprised of sets of independent objects. Yet, current vision models make no assumptions about the nature of the pictures they look at. By imposing an objectness prior, this paper a module that is able to recognize permutation-invariant sets of objects from pixels in both supervised and unsupervised settings. It does so by introducing a slot attention module that combines an attention mechanism with dynamic routing.

OUTLINE:
0:00 - Intro & Overview
1:40 - Problem Formulation
4:30 - Slot Attention Architecture
13:30 - Slot Attention Algorithm
21:30 - Iterative Routing Visualization
29:15 - Experiments
36:20 - Inference Time Flexibility
38:35 - Broader Impact Statement
42:05 - Conclusion & Comments

Paper: https://arxiv.org/abs/2006.15055

My Video on Facebook's DETR: https://youtu.be/T35ba_VXkMY
My Video on Attention: https://youtu.be/iDulhoQ2pro
My Video on Capsules: https://youtu.be/nXGHJTtFYRU

Abstract:
Learning object-centric representations of complex scenes is a promising step towards enabling efficient abstract reasoning from low-level perceptual features. Yet, most deep learning approaches learn distributed representations that do not capture the compositional properties of natural scenes. In this paper, we present the Slot Attention module, an architectural component that interfaces with perceptual representations such as the output of a convolutional neural network and produces a set of task-dependent abstract representations which we call slots. These slots are exchangeable and can bind to any object in the input by specializing through a competitive procedure over multiple rounds of attention. We empirically demonstrate that Slot Attention can extract object-centric representations that enable generalization to unseen compositions when trained on unsupervised object discovery and supervised property prediction tasks.

Authors: Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, Thomas Kipf

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we'll look at object-centric learning with slot attention by Francesco Locotello, Thomas Kipf and others of Google Brain, ETH Zurich and MPI. On a high level this paper recognizes scenes of objects from single pixels and it's best I show you a picture of what's going on. So you have scenes like this where there is some sort of an arrangement of objects and there are multiple tasks you can do here. Specifically they consider the task of unsupervised recognition of objects which they call object discovery and supervised classification of objects. The difficulty being that these are sets of objects so there is no ordering to the sets. They do this via a thing they call slot attention that basically is a permutation invariant attention mechanism over these objects in both the supervised and unsupervised domain and they do this in a fashion where they iteratively route the attention in order to make the different slots compete for attention over these objects. So that's the sort of high level. If you are in this field you probably know right now what's going on. If you're not we'll dive into it together so stay tuned. If you like content like this consider sharing it out, leaving a like or tell me what you think about it in the comments. I appreciate any suggestion for making these videos better so people can learn more from it. Alright so the problem I've already described the problem a little bit but let's go a bit deeper here. You have images like this and the images we're considering are going to be images that have some sort of arrangement of objects or what we humans would call objects. In this case you can see there is this gray square, not sorry, this gray cube right here. There is a smaller green cube and then there is a yellow cylinder. Now in the task of object discovery what you're supposed to do is you're simply supposed to say that there is an object right here, there is an object about here and there is an object here. So basically you're supposed to point to the pixels where there are objects and you're supposed to segment the objects from each other. You can see right here that this model, we don't know how it works yet, but it separates the left cube here, the bottom cube here and the top right cylinder right here. In the task of set prediction you're supposed to say what objects there are. So you're supposed to say there is a gray cube right here, a green cube right here and there is a yellow cylinder right there. Actually you don't have to say where they are I guess. There are many different variants of this task but mainly you're supposed to classify them, meaning you have to say there is a gray cube there. I believe in this case it's with coordinates but you can do it without. The difficulty here of course being that these are sets so there is no natural order in it. So if you say there is a green cube and a yellow cylinder it's going to be the same as there is a yellow cylinder and a green cube. So you have to build an architecture that is somehow invariant with respect to the labels. We've seen a lot of the concepts in this video in this paper before. This video is sort of a kind of a mash together of different concepts of other places. So what you'll see is for example this property of the fact that here you see are the labels for these objects. This could be there is a green cube, there is a gray cube and you'll have to come up with an architecture that if here you predict that green cube you consider it correct even though the corresponding label isn't the one for the green cube. And we saw this for example in this DETR architecture by Facebook where they use a matching loss but we'll get into that. Okay so these are the tasks. The tasks are object discovery and set prediction. So how does this paper deal with this? They use this thing called a slot attention module. Now the slot attention module is in essence it's pretty simple. What it does is it has these different slots right here as you can see and it divides the input into features. So you can see there is a CNN encoder because we're working with pixels it's natural that we want to encode these into a CNN. This CNN will probably down sample the image a bit and divide subdivided into this grid right here. So you have a fairly coarse grid. The grid is actually not a bit finer than you see here. This is just for example but you'll have ultimately a number of features so each pixel right here is going to be a feature. Each feature will have not only this one channel as you see here but many many channels of information down here. So the CNN will encode each of these regions in the picture into a feature vector and then you have these slots. So what you'll want to do and we maybe look at this so you'll have the features right here. These are your features and you'll have the slots and the slots let's say there are fewer slots than features. Three slots, four slots as in this case. What you'll want to do is you'll want to assign the features to the slots. So you maybe say okay this feature right here and this feature right here they go to this slot and then these two features go to this slot and then these two go to this and that feature goes to that and that's equivalent to basically subdividing the picture into these slots. Ultimately your goal is going to be to say that these features right here these pixels right here are going maybe into that slot and then these ones right here are going into that slot and these ones here going into that slot and the rest so all of their background is going into that one. You can see that if you have a system like this if you can train it correctly then it becomes pretty easy to classify it right here because you can just take each slot and independently classify it. Because you already know you already have assigned all the pixels where the object appears into that slot you can just super easily predict a class from it. So we're almost at the end so you now predict for each slot a class or a description of the object whatever you want to predict and this is the exact same thing as in this Facebook paper now where for each of these slots we've predicted a bounding box. The question is how do you assign this to the labels and that's pretty easy that there's this thing called the Hungarian matching that basically what you're saying is you want to be as forthcoming as possible right so if you predict a gray cube somewhere and there is a gray cube somewhere here you want to match them you'll say okay I'm going to give you the benefit of the doubt and I'm going to do your model I'm going to assume with the gray cube you meant that gray cube right here and if there is the yellow rectangle and the yellow rectangle somewhere over there you don't incur any penalty as long as you predict the correct things. Now only whenever you predict like a second yellow rectangle so both of these slots now so this slot and this slot for some reason they predict a yellow rectangle this one correctly and this one was assigned this object and it incorrectly predicts a yellow rectangle oh sorry other way around this one incorrectly predicts a yellow rectangle where there is no second yellow rectangle in our label set there's only this maybe this green cube then this will be a mistake because it can't be matched it will be matched to the one where it has the least loss but it will be matched to something that's not a yellow rectangle and therefore that's going to be a mistake so this is how you calculate the loss function with this matching algorithm and you can calculate that matching in a deterministic fashion so you can back propagate through it so you can see if this slot assignment works we'll have a pretty easy time then calculating the classes coming up with a loss the same for the unsupervised object discovery what we'll do is we'll run these things through this slot decoder now this slot decoder is very similar to an a generator in GANs for example it takes a hidden representation as input now the hidden representation here is going to be these these slots and it's going to up sample it into an image if we train the whole if if we have a good slot assignment mechanism we can pretty easily train a decoder like this right with any method you want in this case I believe they use some yeah some sort of up sampling up convolution architecture right here and they use the L2 they minimize the reconstruction error between the end the output image and the input image so it's sort of like a variational autoencoder or just autoencoder objective in this case all right so we know how to encode a picture into hidden representation using a standard convolutional neural network and we know once our slot attention mechanism works we pretty much know how to go from there so the question is what is this slot attention mechanism now what we're supposed to do is we're supposed to again assign each one of the features into a slot and in a very specific fashion so if you think about the pixels right here there can be multiple of these pixels or multiple of the regions multiple features can be assigned to one slot but we'd rather not have the same feature assigned to multiple slots so each slot takes in many features but the features should be this divided between the slots such that only one slot attends to a feature and by me saying attend you probably already know where this is going so if you have the features and you consider the slots right and we just look at a single feature for now what we'll do is we'll have an attention mechanism from the slots going into the features so if you don't know what an attention mechanism is I have this video called attention is all you need where I explained this but briefly the features they will emit something that's called a key which is a vector and then the slots will emit a query which are also vectors and the sir the the information is now routed by agreement of key and query in this case this thing this this feature right here would be routed to this slot now it would be routed to both slots but it wouldn't be routed as much to the bottom slot and we make sure that this happens by using a softmax assignment so if this is like 9 and this is 4 what we'll do is a softmax assignment such that after that so we have a proper distribution which would be something like after the softmax be something like 0.9 and 0.1 right here so you can see that the attention is fairly hard so this is basically it's a differentiable way to assign these things okay so an attention mechanism fulfills the property that we want to basically assign features to the slots in a way that the slots compete for the features as you can see right here if this slot here matches the feature the best it come it out competes the other slot because at the end this has to be normalized to one because of the softmax so this competition is the heart of the slot attention mechanism and this is this is how it works so this is the slot attention module as you can see so you'll take your inputs and they have lots of layer norms in here but disregard the the layer norms so what you'll do is you'll calculate the agreement between the inputs and the slots now you might wonder in a standard attention mechanism you'll have input signal coming from here which is like maybe these are the input signals and then you construct the keys and the queries for the next layer you construct all from that input signal right and also the values by the way you construct everything from that input signal but in this case will have many features and will only have a fixed amount of slots right here so where do these slots come from where do the the signal for the keys come from in the Facebook DETR paper we saw that these are learned embeddings however in this case right here these are not learned the slots are initialized randomly so at the beginning of each thing the slots are initialized randomly you can think of this as an attention mechanism where you have the attention module right here and then at the beginning you simply have randomly initialized positional embedding or randomly initialized slots and then the image is going to be encoded through a CNN right here giving you a bunch of these features and then you'll have cross attention between these features and the slots and that will give you the next layer right here okay all right so you want to calculate the routing between the inputs and the slots and then you want to perform a softmax over the slots which will give you this competitive nature between the slots so all the slots are going to compete for the features to be routed to them and then this is simply the second part of the attention mechanism and so you will have a weighted mean now this is a slightly different from an attention mechanism because in a real attention mechanism you'll have a weighted sum right here here you will have a weighted mean but it's basically such that you can have a different amount of slots and the kind of values will stay the same that's why you do the mean so you weight them up and the values are simply a function of the inputs this is like in a standard attention mechanism then what you'll do you can see that this is now called updates okay so you start with the slots randomly and then you use the slots to route the information you take the inputs and you use that information routing to calculate the updates now you put the updates through a GRU with the state being the previous slots and then you'll add that to the slots either this says optional residual MLP so what you can do is you will have a residual MLP or not this is a fairly complicated thing but if you think of it it is just a transformer so what they describe here sorry the purpose of this GRU here of course is that the GRU is a recurrent unit and you can see right here that they do this multiple times so once you start with the random slots right but then you update the slots and you go you go again okay so you do this first of all you'll have the features and you'll just have random slots and then you do a bit of routing okay okay so now we have a bit of routing cool you update these slots to be the next set of slots and then you take the same features and route them again so you you route them again and this is supposed to be kind of this iterative procedure you might have seen this in capsule networks I've done a video on capsule networks where exactly this type of iterative routing you always have the same routing functions right these the functions for value and key and query are always the same but you do this iteratively many times in a row this is like a transformer with weight sharing it's exactly the same right so you have these slots you initialize them randomly you do your query times keys your soft max times the value right here and this the transformer even has this plus this MLP layer right here like this the transformer has that in there and then you simply do it again so up here you have the next transformer layer but instead of being its own layer you'll copy the so it's it's weight shared it's a transformer with weight sharing between the modules and the inputs they are also copied up here this these side inputs all right it's otherwise it's the it's the same thing except that these aren't produced by an encoder that is also a transformer they're actually produced by a CNN and the weights here are shared the only difference is that in between here they also have like this GRU this GRU thing but they do an ablation on it and it's actually not that important so you could might as well just leave it away bring it brings only very few benefits so this is how I want how I think of this model this is a multi layer a t layer transformer with weight sharing in for the individual layers where the inputs the input positional encoding are randomly initialized each time okay now they they really stress this random initialization because this differs from the DETR paper in that in the DETR paper these things here are learned and the DETR paper we have also this kind of object detection thing and it what will happen when you learn these is that for example this one right here might specialize in objects that are sort of on the top left of the image and this one might specialize in objects that are kind of long and in the middle and so on and this one might specialize to something else now I can't tell you what works better or what not it seems like you can if I were to implement something like this I might want to go with the Facebook one and then just have more right here in this paper they opt for having fewer but because they're fewer if you learn them they become I guess too specialized and you will need to keep them agnostic so you don't want to learn them you simply want to randomly initialize them each time and via via the iterative routing via the weight sharing they will be sort of assigned correctly all right I hope you could follow this yeah if you if you want to anthropomorphize this you could think if each of these slots starts out just randomly and then just by sheer coincidence through this attention mechanism they happen to be assigned a couple of these features now because we train the model to perform well because they're already assigned these features in the next layer they'll basically ask through the query function they'll ask for more of that they'll basically say oh I'm now responsible kind of for the gray pixels give me more of the gray pixels right give me give me more of that and then in the next layer even more of that even more of that and you'll see in the in the investigations into what happens exactly this type of thing happening so if we skip ahead to the experiments where they show what happens through the iterations you can see this right here so the attention the attention maps of these slots you can see that in after the first step you can see that you know it's slot two right here is assigned kind of these both of these objects slot three is already pretty so the first step kind of learns to segment a little bit of the image but not you know too well slot four it also the attention map here is pretty pretty wonky but if you in the next step and this is kind of crucial basically the these slots they specialize there's a slot to realize as well I have a lot of these these blue pixels I'm gonna give me more of those right give me more of those so it gets all the blue pixel well slot four has a lot of these golden pixels says give me more of that of those golden stuff that's also regionally right next to that and since these two compete I'm pretty sure slot two would also ask for more of the golden pixels because it has a lot of golden pixels but it competes with slot four because of the softmax so all of the golden pixels are assigned to slot four and not slot two well all of the blue pixels that slot four surely asks for as well are assigned to slot two in the next iteration so I actually consider iteration one is for you take the randomly initialized slots and you kind of assign them stuff so this is mainly the this is now mainly the transformer layer learning to segment but then step two is where the magic really happens is where the slots they kind of realize what's assigned to them and they ask for more of it and through the competition you'll get this separation into objects right so the whole thing is trained end to end which basically means that these functions get really good at doing this kind of segmentation alright and then in subsequent iterations you can just see this effect multiplying even more and more right but I might even you might even be able to think that you might want to separate step one and the subsequent steps because step one is sort of seems fundamentally different from steps two three four and so on because step one is this kind of assignment process and then the other steps are refinement so if I were to take this model and make it better I would try to have a special like not way sharing between steps the first step and the subsequent steps but what do I know this apparently it works okay you can also look at the reconstruction since their objective is to reconstruct so basically what each slot outputs each slot out if you reconstruct each slot here we these are the different slots each slot is supposed to output a picture of the reconstruction now if we consider that each slot is responsible for an object you might very well say okay this slot here gives me a picture with just the object in it that it's supposed to reconstruct and then this lot here gives me a picture with just the object that it is supposed to reconstruct now how do you know how to combine these pictures especially since they might be overlapping and so on so the way you do it is you actually output four channels so you output R G B and A so a being the alpha channel so each slot also has to decide where the object is that it reconstructs and so each so this this here might be okay everything here is alpha one and including the shadow maybe maybe there's a little shadow and everything else is alpha zero and then the alpha maps you combine also via a softmax to ensure that they sum up to one so you combine the pictures including their alpha maps but that means you can basically reconstruct from the slots where the where the objects are now you'll know you'll notice this is this thing here you'll notice that they often use for example here four different slots because even though the image has three different objects why is that because you need to reconstruct the entire image so you need at least one slot for the background and that's always what you can see right here so if you have the sorry the reconstruction you'll see that slot too with time with iterations it reconstructs this cube slot three reconstructs the ball slot four reconstructs the yellow cylinder and slot one reconstructs the background okay also here if you see the attention masks you see that the slot one will be responsible for the background here the background is significantly darker than in these others though they do say the background doesn't really tend to go to one slot in particular it tends to kind of spread out across all the slots and this might mean more investigation yeah so they have these different tasks right here for example to segment these Tetris blocks here and you can see the segmentations it works pretty pretty well now why does this work so well it's probably because of the data sets so these kinds of data sets they come you know they they're produced by a generator and the generator specifically has these these objects right here and it sort of in it arranges them in an independent fashion the background is really clean right the objects themselves are really clean and geometric and so on and they're they're kind of arranged in a random fashion and then there's a render of that so this is like super super duper clean data set and I guess that has a lot to do with why these methods work so well because they can just assume okay an object is generally you know spatially something some geometric shape that I know it's close together it's pretty independent from its surrounding and it's trained with objects that are almost zero correlated like there is zero correlation between the objects in the training data set so I wouldn't yet apply this much to to real-world problems but is an interesting thought right here so that's the sort of idea behind the paper I hope you got that they do a lot of experiments and here is a bit where my quarrels start so they say that they compare for example with the with these in the unsupervised object discovery experiments they have this data set called clever and this data set has these images with sort of I believe clever six has all up to six different objects now this is already one of the things this is not a specifically quarrel with this model but if your data set has six things they all they give like they give seven slots because they know that the data set has at most six things which means they can always cover all the things now it works when there's less objects but I think the knowledge of how many objects there's gonna be is also a big part of why these models work and why maybe it's not entirely ready yet for the real world but anyway they compare to these two baselines they're called iodine which also employs kind of a recurrent architecture but not with an attention mechanism and Monet and they say yada yada yada this replaces lot of them no that's not it for the Monet iodine DSP and baselines we compare with the published numbers as we use the same experimental setup so they say they use the same experimental setup and that's why they don't re-implement these models but they use the published numbers in their respective papers which is something you can do this is often I guess these machine translation papers and so on they do this just because you know it's a lot to to run these things however here I'm a bit skeptical first yes because it is it is Google so they do have a lot of resources available to technically run these things I've seen at least Monet has an implementation by the author or I've seen one of them the other one also there is an implementation and there's eight authors on this particular paper so I yeah I this would be okay they say as we use the same experimental setup so even in that case if you have the same setup it's more okay but it really depends on you really having the same setup and this is a bit where it kind of falls so for example one example right here is they say we train the moon using the atom optimizer with a learning rate and 60 so on they use a single GPU we further make use of learning rate warm-up to prevent early saturation of the attention mechanism and an exponential decay decay schedule in the learning rate which we found to reduce variance so I've checked these other models and none of them talks about learning rate warm-up and nowhere in their code is there a learning rate warm-up now you might you might argue okay this is specific to this model it might need this but if you look at the results right here for example you see that they don't outperform these other models by too much so you can see right here this is on par this here outperforms this one a little bit but then also the star here means that they have left out one of the denotes that one outlier was excluded from evaluation I guess which is valid if it's a super outlier but in this case I would categorize this model as a different way of doing doing things and not necessarily outperforming the others so this also if you look at the ablations the differences here are miniscule and in these ablations that they show every single thing they do like gives them a little bit of a boost and you just make it kind of across the line to reach state-of-the-art I'd rather have research move in a direction where we just show cool ideas and that they work and that's what this paper does to to be fair what I do have more of a problem with a little bit is this here on clever six we can use a batch size of up to 64 on a single V 100 GPU as opposed to four in this iodine baseline right compared to ion and our model is significantly more efficient in terms of both memory consumption and runtime which is you know something I believe but this characterization that they use a batch size of four and here in this paper they can use a size up to 64 on a single V 100 GPU I've read the iodine paper and the iodine paper says yes they do use a batch size of four on one GPU so they also use one GPU but they say their GPU has a RAM of 12 gigabytes and 12 gigabyte RAM GPUs that points to something like TI either I guess a 1080 or a 20 2080 or something like this this is not a V 100 the V 100 come in 16 or probably Google has the 32 gigabyte version so this is a 32 gigabyte GPU that is significantly better than the TI GPUs these V 100 they cost like five or ten times more than the TI GPUs and to simply say we have also one GPU and we can run up to a batch size of 64 and they can only run a batch size of four it seems I don't know it seems sort of overstating what you can do now maybe I'm wrong maybe they have actually tested this other model and concluded also on their GPUs it can only run to a batch size of in a bat as a batch size of four but I highly doubt it because in their here the paper is cited and in their paper they explicitly named that they use for a batch size of four for their 12 gigabyte GPUs so yeah in this that that just kind of pulls through so there's the there's the miniscule improvements and then there is the ablations of all these tricks where everyone just gives you a little bit and then there is this kind of very very very favorable comparison wordsmithy a bit which gives a bit of a bitter taste to what I think is actually a very very cool method because how why is this method so cool because for example these slots here they are trained to also absorb the background right so you can technically at inference time increase the number of slots even though you've trained with just with a few slots right you can you can increase the number of slots and the model can just handle it and they show it right here in these results here you can this data set has six objects and this data set has ten objects now the model has only been trained ever on six objects and they can just up the number of slots at inference time and it'll work also very well also they can now up the number of iterations since these are all weight shared these iterations right we've looked at it there's there's weight sharing between the iterations there is nothing stopping you from just piling on here because it's weight shared you don't need any more weights you can just refine this iteration and since the iteration themselves are refining these attention masks anyway you might as well at inference time refine them some more they have an ablation where they show that technically like three two or three iterations at training time gives them the best result I guess just because of gradient propagation because more layers means you have to propagate the gradient back more but at inference time you can just up these iterations and as you can see right here you get better and better so this these results are are pretty cool and they respect the property that sets should be permutation invariant and and so on this routing view of the transformer is pretty cool even though it's you can look at it as a transformer with weight sharing or an iterative routing protocol like in capsules so all of this I I find to be very very cool idea and I think that's how we should look at this this paper so before I am too critical of this paper I want to say that I really like the idea and the algorithm here the implementation yeah so that was the paper at last I actually want to look at the broader impact statement just because I've I've complained about brought the need for broader impact statements so I just want to kind of go just read them and just like look how how the different companies how the different institutions how the different people how the community reacts to them crafts them and so on so this one I find particularly interesting let's go through it says the slot detention module allows to learn object centric representation from perceptual input okay as such it is a general module that can be used in a wide range of domains and applications in our paper we only consider artificially generated data set under well controlled settings where slots are expected to specialize to objects however the specializations of our model is implicit and fully driven by the downstream task we remark that as a concrete measure to assess whether the module is specialized in unwanted ways one can visualize the attention masks to understand how the input features are distributed across the slots while more work is required to properly address the usefulness of the attention coefficients in explaining the overall predictions of the network especially if the input features are not human interpretable we argued that they may serve as a step towards more transparent and interpretable predictions this is a I mean it's a fine statement but it's not a broader impact statement right if you followed a bit what the broader impact statement is supposed to be this is not one okay the closest this comes to a broader impact statement is said as such it is a general model that can be used in a wide range of domains and applications and maybe a little bit that you can visualize the attention masks to understand how the input features are distributed but the broader impact statement is supposed to give you a preview of how this might affect society at large while this here just kind of lists properties of the model for the research community and sort of for this for the application of this model as as you know the introspection of the model itself this says nothing about society as such so maybe you know maybe that's I think that the smarter people will turn the broader impact statement into more of an introduction section because that's something you usually put in a conclusion or in an introduction where you say look here are some things our model can do and this is what it might be useful for and this is how you could introspect it and so on and since the broader impact statement especially at NURIPS you were allowed to put the broader impact statement on the main paper so not in the appendix but it wouldn't count towards your page limit it's I guess pretty foreseeable without what people are gonna start to do is simply put more of their paper into the broader impact section kind of cloaked in the veneer of a broader impact statement but I this is this is clearly not what what the broader impact statement was originally supposed to be now I don't know if this is good or bad I just think these authors are you know they're doing I think the a good thing here by simply telling us actually something useful about the model but that's just my opinion I do thank you for being with me here I know this was a bit ranty flip-flopping back and forth between the different things we haven't looked at set prediction at all we've only looked at these kind of masks but I invite you to go through the paper yourself and check it out it's pretty cool and they do describe a lot of things in pretty detail the appendix is very long and has very many ablations and this is something I do appreciate and with that bye bye and see you next time
 Hi there! Today we'll look at object-centric learning with slot attention
 by Francesco Locotello, Thomas Kipf and others of Google Brain, ETH Zurich and
 MPI. On a high level this paper recognizes scenes of objects from single
 pixels and it's best I show you a picture of what's going on. So you have
 scenes like this where there is some sort of an arrangement of objects and
 there are multiple tasks you can do here. Specifically they consider the task of
 unsupervised recognition of objects which they call object discovery and
 supervised classification of objects. The difficulty being that these are sets of
 objects so there is no ordering to the sets. They do this via a thing they call
 slot attention that basically is a permutation invariant attention mechanism
 over these objects in both the supervised and unsupervised domain and
 they do this in a fashion where they iteratively route the attention in order
 to make the different slots compete for attention over these objects. So that's
 the sort of high level. If you are in this field you probably know right now
 what's going on. If you're not we'll dive into it together so stay tuned. If you
 like content like this consider sharing it out, leaving a like or tell me what
 you think about it in the comments. I appreciate any suggestion for making
 these videos better so people can learn more from it.
 Alright so the problem I've already described the problem a little bit but
 let's go a bit deeper here. You have images like this and the images we're
 considering are going to be images that have some sort of arrangement of objects
 or what we humans would call objects. In this case you can see there is this gray
 square, not sorry, this gray cube right here. There is a smaller green cube and
 then there is a yellow cylinder. Now in the task of object discovery what you're
 supposed to do is you're simply supposed to say that there is an object right
 here, there is an object about here and there is an object here. So basically
 you're supposed to point to the pixels where there are objects and you're
 supposed to segment the objects from each other. You can see right here that
 this model, we don't know how it works yet, but it separates the left cube here,
 the bottom cube here and the top right cylinder right here. In the task of set
 prediction you're supposed to say what objects there are. So you're supposed to
 say there is a gray cube right here, a green cube right here and there is a
 yellow cylinder right there. Actually you don't have to say where they are I guess.
 There are many different variants of this task but mainly you're supposed to
 classify them, meaning you have to say there is a gray cube there. I believe in
 this case it's with coordinates but you can do it without. The difficulty here of
 course being that these are sets so there is no natural order in it. So if
 you say there is a green cube and a yellow cylinder it's going to be the
 same as there is a yellow cylinder and a green cube. So you have to build an
 architecture that is somehow invariant with respect to the labels. We've
 seen a lot of the concepts in this video in this paper before. This video is sort
 of a kind of a mash together of different concepts of other places. So
 what you'll see is for example this property of the fact that here you see
 are the labels for these objects. This could be there is a green cube, there is
 a gray cube and you'll have to come up with an architecture that if here you
 predict that green cube you consider it correct even though the corresponding
 label isn't the one for the green cube. And we saw this for example in this DETR
 architecture by Facebook where they use a matching loss but we'll get into that.
 Okay so these are the tasks. The tasks are object discovery and set prediction.
 So how does this paper deal with this? They use this thing called a slot
 attention module. Now the slot attention module is in essence it's pretty simple.
 What it does is it has these different slots right here as you can see and it
 divides the input into features. So you can see there is a CNN encoder because
 we're working with pixels it's natural that we want to encode these into a CNN.
 This CNN will probably down sample the image a bit and divide subdivided into
 this grid right here. So you have a fairly coarse grid. The grid is actually
 not a bit finer than you see here. This is just for example but you'll have
 ultimately a number of features so each pixel right here is going to be a
 feature. Each feature will have not only this one channel as you see here but
 many many channels of information down here. So the CNN will encode each of
 these regions in the picture into a feature vector and then you have these
 slots. So what you'll want to do and we maybe look at this so you'll have the
 features right here. These are your features and you'll have the slots and
 the slots let's say there are fewer slots than features. Three slots,
 four slots as in this case. What you'll want to do is you'll want to
 assign the features to the slots. So you maybe say okay this feature right here
 and this feature right here they go to this slot and then these two features go
 to this slot and then these two go to this and that feature goes to that and
 that's equivalent to basically subdividing the picture into these
 slots. Ultimately your goal is going to be to say that these features right here
 these pixels right here are going maybe into that slot and then these ones
 right here are going into that slot and these ones here going into that slot and
 the rest so all of their background is going into that one. You can see that if
 you have a system like this if you can train it correctly then it becomes pretty
 easy to classify it right here because you can just take each
 slot and independently classify it. Because you already know you already
 have assigned all the pixels where the object appears into that slot you can
 just super easily predict a class from it. So we're almost at the end so you
 now predict for each slot a class or a description of the object whatever you
 want to predict and this is the exact same thing as in this Facebook paper now
 where for each of these slots we've predicted a bounding box. The
 question is how do you assign this to the labels and that's pretty easy that
 there's this thing called the Hungarian matching that basically what you're
 saying is you want to be as forthcoming as possible right so if you predict a
 gray cube somewhere and there is a gray cube somewhere here you want to match
 them you'll say okay I'm going to give you the benefit of the doubt and I'm
 going to do your model I'm going to assume with the gray cube you meant that
 gray cube right here and if there is the yellow rectangle and the yellow
 rectangle somewhere over there you don't incur any penalty as long as you
 predict the correct things. Now only whenever you predict like a second
 yellow rectangle so both of these slots now so this slot and this slot for some
 reason they predict a yellow rectangle this one correctly and this one was
 assigned this object and it incorrectly predicts a yellow rectangle oh sorry
 other way around this one incorrectly predicts a yellow rectangle where there
 is no second yellow rectangle in our label set there's only this maybe this
 green cube then this will be a mistake because it can't be matched it will be
 matched to the one where it has the least loss but it will be matched to
 something that's not a yellow rectangle and therefore that's going to be a
 mistake so this is how you calculate the loss function with this matching
 algorithm and you can calculate that matching in a deterministic fashion so
 you can back propagate through it so you can see if this slot assignment works
 we'll have a pretty easy time then calculating the classes coming up with a
 loss the same for the unsupervised object discovery what we'll do is we'll
 run these things through this slot decoder now this slot decoder is very
 similar to an a generator in GANs for example it takes a hidden representation
 as input now the hidden representation here is going to be these these slots
 and it's going to up sample it into an image if we train the whole if if we
 have a good slot assignment mechanism we can pretty easily train a decoder like
 this right with any method you want in this case I believe they use some yeah
 some sort of up sampling up convolution architecture right here and they use the
 L2 they minimize the reconstruction error between the end the output image
 and the input image so it's sort of like a variational autoencoder or just
 autoencoder objective in this case all right so we know how to encode a picture
 into hidden representation using a standard convolutional neural network
 and we know once our slot attention mechanism works we pretty much know how
 to go from there so the question is what is this slot attention mechanism now
 what we're supposed to do is we're supposed to again assign each one of the
 features into a slot and in a very specific fashion so if you think about
 the pixels right here there can be multiple of these pixels or multiple of
 the regions multiple features can be assigned to one slot but we'd rather not
 have the same feature assigned to multiple slots so each slot takes in
 many features but the features should be this divided between the slots such that
 only one slot attends to a feature and by me saying attend you probably already
 know where this is going so if you have the features and you consider the slots
 right and we just look at a single feature for now what we'll do is we'll
 have an attention mechanism from the slots going into the features so if you
 don't know what an attention mechanism is I have this video called attention is
 all you need where I explained this but briefly the features they will emit
 something that's called a key which is a vector and then the slots will emit a
 query which are also vectors and the sir the the information is now routed by
 agreement of key and query in this case this thing this this feature right here
 would be routed to this slot now it would be routed to both slots but it
 wouldn't be routed as much to the bottom slot and we make sure that this happens
 by using a softmax assignment so if this is like 9 and this is 4 what we'll do is
 a softmax assignment such that after that so we have a proper distribution
 which would be something like after the softmax be something like 0.9 and 0.1
 right here so you can see that the attention is fairly hard so this is
 basically it's a differentiable way to assign these things okay so an attention
 mechanism fulfills the property that we want to basically assign features to the
 slots in a way that the slots compete for the features as you can see right
 here if this slot here matches the feature the best it come it out
 competes the other slot because at the end this has to be normalized to one
 because of the softmax so this competition is the heart of the slot
 attention mechanism and this is this is how it works so this is the slot
 attention module as you can see so you'll take your inputs and they have
 lots of layer norms in here but disregard the the layer norms so what
 you'll do is you'll calculate the agreement between the inputs and the
 slots now you might wonder in a standard attention mechanism you'll have input
 signal coming from here which is like maybe these are the input signals and
 then you construct the keys and the queries for the next layer you
 construct all from that input signal right and also the values by the way you
 construct everything from that input signal but in this case will have many
 features and will only have a fixed amount of slots right here so where do
 these slots come from where do the the signal for the keys come from in the
 Facebook DETR paper we saw that these are learned embeddings however in this
 case right here these are not learned the slots are initialized randomly so at
 the beginning of each thing the slots are initialized randomly you can think
 of this as an attention mechanism where you have the attention module right here
 and then at the beginning you simply have randomly initialized positional
 embedding or randomly initialized slots and then the image is going to be
 encoded through a CNN right here giving you a bunch of these features and then
 you'll have cross attention between these features and the slots and that
 will give you the next layer right here okay
 all right so you want to calculate the routing between the inputs and the
 slots and then you want to perform a softmax over the slots which will give
 you this competitive nature between the slots so all the slots are going to
 compete for the features to be routed to them and then this is
 simply the second part of the attention mechanism and so you will have a weighted
 mean now this is a slightly different from an attention mechanism because in
 a real attention mechanism you'll have a weighted sum right here here you will
 have a weighted mean but it's basically such that you can have a different
 amount of slots and the kind of values will stay the same that's why you do the
 mean so you weight them up and the values are simply a function of the
 inputs this is like in a standard attention mechanism then what you'll do
 you can see that this is now called updates okay so you start with the
 slots randomly and then you use the slots to route the information
 you take the inputs and you use that information routing to calculate the
 updates now you put the updates through a GRU with the state being the previous
 slots and then you'll add that to the slots either this says optional residual
 MLP so what you can do is you will have a residual MLP or not this is a fairly
 complicated thing but if you think of it it is just a transformer so what they
 describe here sorry the purpose of this GRU here of course is that the GRU is a
 recurrent unit and you can see right here that they do this multiple times so
 once you start with the random slots right but then you update the slots and
 you go you go again okay so you do this first of all you'll have the features
 and you'll just have random slots and then you do a bit of routing okay okay
 so now we have a bit of routing cool you update these slots to be the next set of
 slots and then you take the same features and route them again so you you
 route them again and this is supposed to be kind of this iterative procedure you
 might have seen this in capsule networks I've done a video on capsule networks
 where exactly this type of iterative routing you always have the same
 routing functions right these the functions for value and key and query
 are always the same but you do this iteratively many times in a row this is
 like a transformer with weight sharing it's exactly the same right so you have
 these slots you initialize them randomly you do your query times keys your soft
 max times the value right here and this the transformer even has this plus this
 MLP layer right here like this the transformer has that in there and then
 you simply do it again so up here you have the next transformer layer but
 instead of being its own layer you'll copy the so it's it's weight shared it's
 a transformer with weight sharing between the modules and the inputs they
 are also copied up here this these side inputs all right it's otherwise it's the
 it's the same thing except that these aren't produced by an encoder that is
 also a transformer they're actually produced by a CNN and the weights here
 are shared the only difference is that in between here they also have like this
 GRU this GRU thing but they do an ablation on it and it's actually not
 that important so you could might as well just leave it away bring it brings
 only very few benefits so this is how I want how I think of this model this is a
 multi layer a t layer transformer with weight sharing in for the individual
 layers where the inputs the input positional encoding are randomly
 initialized each time okay now they they really stress this random
 initialization because this differs from the DETR paper in that in the DETR paper
 these things here are learned and the DETR paper we have also this kind of
 object detection thing and it what will happen when you learn these is that for
 example this one right here might specialize in objects that are sort of
 on the top left of the image and this one might specialize in objects that are
 kind of long and in the middle and so on and this one might specialize to
 something else now I can't tell you what works better or what not it seems like
 you can if I were to implement something like this I might want to go with the
 Facebook one and then just have more right here in this paper they opt for
 having fewer but because they're fewer if you learn them they become I guess
 too specialized and you will need to keep them agnostic so you don't want to
 learn them you simply want to randomly initialize them each time and via via
 the iterative routing via the weight sharing they will be sort of assigned
 correctly all right I hope you could follow this yeah if you if you want to
 anthropomorphize this you could think if each of these slots starts out just
 randomly and then just by sheer coincidence through this attention
 mechanism they happen to be assigned a couple of these features now because we
 train the model to perform well because they're already assigned these features
 in the next layer they'll basically ask through the query function they'll ask
 for more of that they'll basically say oh I'm now responsible kind of for the
 gray pixels give me more of the gray pixels right give me give me more of
 that and then in the next layer even more of that even more of that and
 you'll see in the in the investigations into what happens exactly this type of
 thing happening so if we skip ahead to the experiments where they show what
 happens through the iterations you can see this right here so the attention
 the attention maps of these slots you can see that in after the first step you
 can see that you know it's slot two right here is assigned kind of these
 both of these objects slot three is already pretty so the first step kind of
 learns to segment a little bit of the image but not you know too well slot
 four it also the attention map here is pretty pretty wonky but if you in the
 next step and this is kind of crucial basically the these slots they
 specialize there's a slot to realize as well I have a lot of these these blue
 pixels I'm gonna give me more of those right give me more of those so it gets
 all the blue pixel well slot four has a lot of these golden pixels says give me
 more of that of those golden stuff that's also regionally right next to
 that and since these two compete I'm pretty sure slot two would also ask for
 more of the golden pixels because it has a lot of golden pixels but it competes
 with slot four because of the softmax so all of the golden pixels are assigned to
 slot four and not slot two well all of the blue pixels that slot four surely
 asks for as well are assigned to slot two in the next iteration so I actually
 consider iteration one is for you take the randomly initialized slots and you
 kind of assign them stuff so this is mainly the this is now mainly the
 transformer layer learning to segment but then step two is where the magic
 really happens is where the slots they kind of realize what's assigned to them
 and they ask for more of it and through the competition you'll get this
 separation into objects right so the whole thing is trained end to end which
 basically means that these functions get really good at doing this kind of
 segmentation alright and then in subsequent iterations you can just see
 this effect multiplying even more and more right but I might even you might
 even be able to think that you might want to separate step one and the
 subsequent steps because step one is sort of seems fundamentally different
 from steps two three four and so on because step one is this kind of
 assignment process and then the other steps are refinement so if I were to
 take this model and make it better I would try to have a special like not
 way sharing between steps the first step and the subsequent steps but what do I
 know this apparently it works okay you can also look at the reconstruction
 since their objective is to reconstruct so basically what each slot outputs each
 slot out if you reconstruct each slot here we these are the different slots
 each slot is supposed to output a picture of the reconstruction now if we
 consider that each slot is responsible for an object you might very well say
 okay this slot here gives me a picture with just the object in it that it's
 supposed to reconstruct and then this lot here gives me a picture with just
 the object that it is supposed to reconstruct now how do you know how to
 combine these pictures especially since they might be overlapping and so on so
 the way you do it is you actually output four channels so you output R G B and A
 so a being the alpha channel so each slot also has to decide where the object
 is that it reconstructs and so each so this this here might be okay everything
 here is alpha one and including the shadow maybe maybe there's a little
 shadow and everything else is alpha zero and then the alpha maps you combine also
 via a softmax to ensure that they sum up to one so you combine the pictures
 including their alpha maps but that means you can basically reconstruct from
 the slots where the where the objects are now you'll know you'll notice this
 is this thing here you'll notice that they often use for example here four
 different slots because even though the image has three different objects why is
 that because you need to reconstruct the entire image so you need at least one
 slot for the background and that's always what you can see right here so if
 you have the sorry the reconstruction you'll see that slot too with time with
 iterations it reconstructs this cube slot three reconstructs the ball slot
 four reconstructs the yellow cylinder and slot one reconstructs the background
 okay also here if you see the attention masks you see that the slot one will be
 responsible for the background here the background is significantly darker than
 in these others though they do say the background doesn't really tend to go to
 one slot in particular it tends to kind of spread out across all the slots and
 this might mean more investigation yeah so they have these different tasks right
 here for example to segment these Tetris blocks here and you can see the
 segmentations it works pretty pretty well now why does this work so well it's
 probably because of the data sets so these kinds of data sets they come you
 know they they're produced by a generator and the generator specifically
 has these these objects right here and it sort of in it arranges them in an
 independent fashion the background is really clean right the objects
 themselves are really clean and geometric and so on and they're they're
 kind of arranged in a random fashion and then there's a render of that so this is
 like super super duper clean data set and I guess that has a lot to do with
 why these methods work so well because they can just assume okay an object is
 generally you know spatially something some geometric shape that I know it's
 close together it's pretty independent from its surrounding and it's trained
 with objects that are almost zero correlated like there is zero
 correlation between the objects in the training data set so I wouldn't yet
 apply this much to to real-world problems but is an interesting thought
 right here so that's the sort of idea behind the paper I hope you got that
 they do a lot of experiments and here is a bit where my quarrels start so they
 say that they compare for example with the with these in the unsupervised
 object discovery experiments they have this data set called clever and this
 data set has these images with sort of I believe clever six has all up to six
 different objects now this is already one of the things this is not a
 specifically quarrel with this model but if your data set has six things they all
 they give like they give seven slots because they know that the data set has
 at most six things which means they can always cover all the things now it works
 when there's less objects but I think the knowledge of how many objects
 there's gonna be is also a big part of why these models work and why maybe it's
 not entirely ready yet for the real world but anyway they compare to these
 two baselines they're called iodine which also employs kind of a recurrent
 architecture but not with an attention mechanism and Monet and they say
 yada yada yada this replaces lot of them no that's not it for the Monet iodine
 DSP and baselines we compare with the published numbers as we use the same
 experimental setup so they say they use the same experimental setup and that's
 why they don't re-implement these models but they use the published numbers in
 their respective papers which is something you can do this is often I
 guess these machine translation papers and so on they do this just because you
 know it's a lot to to run these things however here I'm a bit skeptical first
 yes because it is it is Google so they do have a lot of resources available to
 technically run these things I've seen at least Monet has an implementation by
 the author or I've seen one of them the other one also there is an implementation
 and there's eight authors on this particular paper so I yeah I this would
 be okay they say as we use the same experimental setup so even in that case
 if you have the same setup it's more okay but it really depends on you
 really having the same setup and this is a bit where it kind of falls so for
 example one example right here is they say we train the moon using the atom
 optimizer with a learning rate and 60 so on they use a single GPU we further make
 use of learning rate warm-up to prevent early saturation of the attention
 mechanism and an exponential decay decay schedule in the learning rate which we
 found to reduce variance so I've checked these other models and none of them
 talks about learning rate warm-up and nowhere in their code is there a
 learning rate warm-up now you might you might argue okay this is specific to
 this model it might need this but if you look at the results right here for
 example you see that they don't outperform these other models by too
 much so you can see right here this is on par this here outperforms this one a
 little bit but then also the star here means that they have left out one of the
 denotes that one outlier was excluded from evaluation I guess which is valid
 if it's a super outlier but in this case I would categorize this model as a
 different way of doing doing things and not necessarily outperforming the others
 so this also if you look at the ablations the differences here are
 miniscule and in these ablations that they show every single thing they do
 like gives them a little bit of a boost and you just make it kind of across the
 line to reach state-of-the-art I'd rather have research move in a direction
 where we just show cool ideas and that they work and that's what this paper
 does to to be fair what I do have more of a problem with a little bit is this
 here on clever six we can use a batch size of up to 64 on a single V 100 GPU
 as opposed to four in this iodine baseline right compared to ion and our
 model is significantly more efficient in terms of both memory consumption and
 runtime which is you know something I believe but this characterization that
 they use a batch size of four and here in this paper they can use a size up to
 64 on a single V 100 GPU I've read the iodine paper and the iodine paper says
 yes they do use a batch size of four on one GPU so they also use one GPU but they
 say their GPU has a RAM of 12 gigabytes and 12 gigabyte RAM GPUs that points to
 something like TI either I guess a 1080 or a 20 2080 or something like this this
 is not a V 100 the V 100 come in 16 or probably Google has the 32 gigabyte
 version so this is a 32 gigabyte GPU that is significantly better than the TI
 GPUs these V 100 they cost like five or ten times more than the TI GPUs and to
 simply say we have also one GPU and we can run up to a batch size of 64 and
 they can only run a batch size of four it seems I don't know it seems sort of
 overstating what you can do now maybe I'm wrong maybe they have actually tested
 this other model and concluded also on their GPUs it can only run to a batch
 size of in a bat as a batch size of four but I highly doubt it because in their
 here the paper is cited and in their paper they explicitly named that they
 use for a batch size of four for their 12 gigabyte GPUs so yeah in this that
 that just kind of pulls through so there's the there's the miniscule
 improvements and then there is the ablations of all these tricks where
 everyone just gives you a little bit and then there is this kind of very very
 very favorable comparison wordsmithy a bit which gives a bit of a bitter taste
 to what I think is actually a very very cool method because how why is this
 method so cool because for example these slots here they are trained to also
 absorb the background right so you can technically at inference time increase
 the number of slots even though you've trained with just with a few slots right
 you can you can increase the number of slots and the model can just handle it
 and they show it right here in these results here you can this data set has
 six objects and this data set has ten objects now the model has only been
 trained ever on six objects and they can just up the number of slots at inference
 time and it'll work also very well also they can now up the number of iterations
 since these are all weight shared these iterations right we've looked at it
 there's there's weight sharing between the iterations there is nothing stopping
 you from just piling on here because it's weight shared you don't need any
 more weights you can just refine this iteration and since the iteration
 themselves are refining these attention masks anyway you might as well at
 inference time refine them some more they have an ablation where they show
 that technically like three two or three iterations at training time gives them
 the best result I guess just because of gradient propagation because more layers
 means you have to propagate the gradient back more but at inference time you can
 just up these iterations and as you can see right here you get better and better
 so this these results are are pretty cool and they respect the property that
 sets should be permutation invariant and and so on this routing view of the
 transformer is pretty cool even though it's you can look at it as a
 transformer with weight sharing or an iterative routing protocol like in
 capsules so all of this I I find to be very very cool idea and I think that's
 how we should look at this this paper so before I am too critical of this paper I
 want to say that I really like the idea and the algorithm here the
 implementation yeah so that was the paper at last I actually want to look at
 the broader impact statement just because I've I've complained about
 brought the need for broader impact statements so I just want to kind of go
 just read them and just like look how how the different companies how the
 different institutions how the different people how the community reacts to them
 crafts them and so on so this one I find particularly interesting
 let's go through it says the slot detention module allows to learn object
 centric representation from perceptual input okay as such it is a general module
 that can be used in a wide range of domains and applications in our paper we
 only consider artificially generated data set under well controlled settings
 where slots are expected to specialize to objects however the specializations
 of our model is implicit and fully driven by the downstream task we remark
 that as a concrete measure to assess whether the module is specialized in
 unwanted ways one can visualize the attention masks to understand how the
 input features are distributed across the slots while more work is required to
 properly address the usefulness of the attention coefficients in explaining the
 overall predictions of the network especially if the input features are not
 human interpretable we argued that they may serve as a step towards more
 transparent and interpretable predictions this is a I mean it's a fine
 statement but it's not a broader impact statement right if you followed a bit
 what the broader impact statement is supposed to be this is not one okay the
 closest this comes to a broader impact statement is said as such it is a
 general model that can be used in a wide range of domains and applications and
 maybe a little bit that you can visualize the attention masks to
 understand how the input features are distributed but the broader impact
 statement is supposed to give you a preview of how this might affect society
 at large while this here just kind of lists properties of the model for the
 research community and sort of for this for the application of this model as as
 you know the introspection of the model itself this says nothing about society
 as such so maybe you know maybe that's I think that the smarter people will turn
 the broader impact statement into more of an introduction section because
 that's something you usually put in a conclusion or in an introduction where
 you say look here are some things our model can do and this is what it might
 be useful for and this is how you could introspect it and so on and since the
 broader impact statement especially at NURIPS you were allowed to put the
 broader impact statement on the main paper so not in the appendix but it
 wouldn't count towards your page limit it's I guess pretty foreseeable without
 what people are gonna start to do is simply put more of their paper into the
 broader impact section kind of cloaked in the veneer of a broader impact
 statement but I this is this is clearly not what what the broader impact
 statement was originally supposed to be now I don't know if this is good or bad
 I just think these authors are you know they're doing I think the a good thing
 here by simply telling us actually something useful about the model but
 that's just my opinion I do thank you for being with me here I know this was a
 bit ranty flip-flopping back and forth between the different things we haven't
 looked at set prediction at all we've only looked at these kind of masks but
 I invite you to go through the paper yourself and check it out it's pretty
 cool and they do describe a lot of things in pretty detail the appendix is
 very long and has very many ablations and this is something I do appreciate and
 with that bye bye and see you next time