Dynamic Routing Between Capsules
machine learning deep learning capsules capsule networks google brain hinton jeff hinton geoff hinton routing neural networks convolution convolutional neural networks deep neural networks cnns mnist multimnist disentanglement architecture reconstruction alternative dnn ml ai artificial intelligence brain visual system classifier image nonlinearity entities objects capsule network
Geoff Hinton's next big idea! Capsule Networks are an alternative way of implementing neural networks by dividing each layer into capsules. Each capsule is responsible for detecting the presence and properties of one particular entity in the input sample. This information is then allocated dynamically to higher-level capsules in a novel and unconventional routing scheme. While Capsule Networks are still in their infancy, they are an exciting and promising new direction.

Abstract:
A capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. Active capsules at one level make predictions, via transformation matrices, for the instantiation parameters of higher-level capsules. When multiple predictions agree, a higher level capsule becomes active. We show that a discrimininatively trained, multi-layer capsule system achieves state-of-the-art performance on MNIST and is considerably better than a convolutional net at recognizing highly overlapping digits. To achieve these results we use an iterative routing-by-agreement mechanism: A lower-level capsule prefers to send its output to higher level capsules whose activity vectors have a big scalar product with the prediction coming from the lower-level capsule.

Authors: Sara Sabour, Nicholas Frosst, Geoffrey E Hinton

https://arxiv.org/abs/1710.09829


YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Minds: https://www.minds.com/ykilcher
BitChute: https://www.bitchute.com/channel/10a5ui845DOJ/
 Hi there! Today we're looking at dynamic routing between capsules by Sara Sabour, Nicholas Frost and Jeffrey Hinton of Google Brain. This paper is a bit older but it's made quite the impact at the time and so we'll go through it. I find this pretty hard paper to read and kind of understand because a lot of things are very implicit and hand wavy. So we'll kind of go through it and try to get the best out of it, try to explain what capsules are and what they do and how they stack against current networks. So capsule network in essence is a new type of neural network made of capsules. And here it says a capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. Kind of cryptic but so what they're saying is that in a capsule network, let me try to draw one here actually, in a capsule network you have what's called capsules. Capsules you can imagine as just little blobs of things right? And they're also ordered in layers in this case. Let's actually leave away the second layer. And each of these of these capsules will correspond to an entity in the input. Let's say the input is an image. So somewhere here there is an image right? Then maybe this capsule here will be responsible for detecting is there a wall in the image. And this one will be responsible for detecting is there a roof. This one will be is there a door. And this one will be responsible for detecting is there a lake in the image right? So now each of these each of these capsules can for on one hand can either be high or low. So if you if you imagine now a situation where wall high, roof high, door high, lake low. It means probably the image has a house on it right? But second of all not only can it predict whether or not a given entity is present in an image but the individual capsules are also responsible for encoding the exact way or shape or form that this entity takes. So the wall could have different aspects such as color color green. It could have size tall. It could have orientation. orientation is like I don't know vertical. Cool. Then roof could have angle right? Angle wide. So it's a wide roof or a flat roof right? These are these are kind of attributes of these things that also the capsules would encode. So ultimately what these capsules that they are proposing will output is the roof capsule here for example would output a vector. So the output of the roof capsule is a let me draw a coordinate system is a vector. Now the length of the vector will represent so that the length draw this norm here will represent the probability that the roof is in the image. That there is a roof in an image right? The roof is element of this input image. This is simply the length and the individual coordinates will encode these attributes. So this here for example this axis could be the angle of the roof and this axis could be the color. Let's say just that the angle is like some degree number that can be positive or negative. Maybe a roof can be like this. Right this so this is but in essence this is a flat roof and this is a very narrow angle roof. So you can imagine something like this and then the color could also be maybe parameterized on a one-dimensional. It can have more dimensions than two I just can't draw more. So the depending on where this where this arrow now points the for example this vector here has the same probability that there is a roof in the image like if the output is this but the color will be different. The angle will be the same because they're roughly on the same this axis here but the color of this will encode a different different colored roof. And then if the vector is something like this a very short vector it will encode the same the same angle and color directions. So maybe I shouldn't say the position on the axis it's more like this angle and this this angle that encode the attributes. So the kind of the angular components if you will encode the attributes and the length encodes the probability. So this small vector has the same direction in terms of color and angle of the roof but it's much less probable much less likely. So this if the capsule outputs the little blue vector here it says well if there is a roof it's going to be this color in this angle but I'm really that really don't think there's a roof in this image. Whereas if it outputs the large green one then it says I'm pretty sure that there's a roof and it's going to be this angle and this this this angle and this color. Alright so that's that is what each capsule is supposed to do. Each capsule takes the input and outputs a vector that encodes if the entity that the capsule is responsible for is present in the image A and B what properties this entity has. And then we get to the point where there's the next layer of capsules. So the next layer of capsules takes information that each capsule here takes information from each capsule in the lower layer like like you're used to from your neural network and integrates this information and we'll talk about how this works. It integrates all of this information right all of these are vectors now that come from the lower integrates all of this information and again each capsule in this next layer is responsible for a entity. Now these entities in the higher layers are usually composite entities of the lower layers. So this one here could be responsible for house, this one could be responsible for national park, national park and this one could be responsible for beach or something like this right. And then each of these will integrate all of this information from the lower layers and then come up with their own output vector encoding whether or not a given entity is present in the in the image. Of course the house class will pick up if there is a door a roof and a wall in the image the house classes will pick up on that or that's how it's meant to work house class is meant to pick up on that and then itself output a large vector saying there's probably a house in this in this image. So each of these capsules in by itself is responsible for encoding the presence and attributes of a object or object part or entity or part of entity in the given input data. And of course the last layer here it will simply be your classification layer. So in the last layer you have as many capsules as you have classes in your classification task. So this is mainly for a classification task and then you can classify and you can kind of train the whole system like this. So how exactly this happens we'll see next. Alright so they make kind of analogies to the visual system and so on. We'll jump these you can everyone that does deep learning in some way is trying to to make that. We're rather going to the specifics of how these capsules work and how their specific suggestions for them. Note that they say this is in no way the only implementation of capsules. It's just kind of an example to show how one could do it. Alright so first of all they present their what you might call non-linearity. So their non-linearity what it needs to do is if you look at these capsule networks the outputs here the length of the outputs of these vectors right they're supposed to represent probabilities and as such they they need to be so here it roof this door maybe a vector like this wall maybe a vector like that. So initially we simply specify the output is a vector and in essence these capsules are implemented in much the same way like your classic neural network layer would be implemented. So each of these capsules will be in essence a neural network layer by itself that outputs a vector. There's nothing constraining the length of the vector initially so their non-linearity does constrain the vector to be of maximum length 1 and of minimum length 0. That's this non-linearity here. So S here is the unscaled output of the capsule and you can see here if the length of S gets close to 1 or sorry gets really large then this here becomes irrelevant. This whole term will be 1 and then the length of the final output of V here will be 1. Right so if this is very large then the the length of the scaled output will be 1 however if the if the length is really small of the original output so if this goes towards 0 then this becomes irrelevant this becomes irrelevant this will go towards 0 and the entire length will go towards 0. So this is kind of a nice way to scale these outputs always to be between length 0 and 1. Then next thing is so how this I find I find the the most complicated part right so we'll jump ahead actually to how a capsule's network is implemented and this is the the capsule network they implement so first it's an MNIST classifier you have an MNIST image here and it first goes through a simple convolutional layer that's that's nothing new this is a classic convolutional layer is there's 256 channels it has a 9 by 9 filters and stride 1 so it will output a 20 by 20 time by 256 tensor then each of these so each of the outputs here is sent to each of these capsules and now they're convolutional capsules so that makes it a bit more complicated but don't you know don't worry primarily about them being convolutional capsules the analogy is exactly as in a classic neural network you can implement these capsules as void-feed-forward capsules or as convolutional capsules and maybe also as transformer capsules I don't think anyone's done that all right there's a paper for you the so you'll send you'll send the output of this convolution layer to each capsule and then you have basically just two layer of capsules here the first layer consists of 32 what they call primary caps sorry the these 32 capsules each will output an eight dimensional vector and I'm simplifying here it's it's convolutional but they will just for simplest they will each output an eight dimensional vector right and these are exactly as we said before so each of these will be responsible ultimately for a given entity or part of entity being there like in MNIST this could be is there a little curve on the bottom left side right this might indicate the presence of a six or an eight something like this and then the these capsules here each is they represented as a row so each of these rows here is a capsule and we have ten of these and these are your simply your final classification capsules so each capsule is responsible for indicating the presence or absence of one particular class of digits so this will be of a one of a two of a three of a four and so on of a zero I guess somewhere as well so these are ten capsules and the question is how does information go from a capsule here from the output of a capsule or to any of capsule here and the easy way to do this is simply to say as in a classical neural network the output here simply goes to the input here just you just put it there basically on on unchanged now there is a bit of an issue here with the dimensions but you can simply say well we simply put a weight matrix in to route into the capsules but the idea of these capsules and this paper is to say wait wait these capsules actually we want to make them decide to which capsule in the next layer will they send their input right so the capsules can kind of decide where they want to send their output to like where is this where is the capsule that detects the maybe this one detects is there a line in the right side of the image right indicating maybe a seven or a one this is probably most relevant for the one class and for the seven class so it might decide to route its output there and the idea of how this routing happens is basically the topic of this paper so the the capsules route their output to the appropriate next layers capsules how is this done all right this is done via the what's called the routing mechanism that I find it quite poorly described here so I will simply draw it I will simply try to make it up all right so we have capsules and as I've drawn them before right we have one two three capsules and we maybe have two parent capsules each of these capsules here will output a vector as we said and we'll only do it for this this one sorry vector here so this will output this vector and needs to decide where to here or to here do I send to this output now what it does is there is an iterative procedure that has multiple steps and this is I think this is at least the way I understand I think the important part to understand is that if we forward pass data through this network it actually doesn't go forward in a straight line what it actually does is it goes through a layer and then it does multiple steps in between layers until it has decided where it wants to go in the next layer and then it goes on to the next layer and if there's another capsule layers it does again multiple steps before it goes on so that's that's my take on it and the multiple steps are as follows first I'll send my output vector to to all of the all of the layers like equally all of the parent capsules and so will will everyone else right everyone will send theirs equally to the parent now this isn't just done and this may be here this isn't just done just by sending it but this is actually done by modulation of weight matrices so each thing here if this is capsule I and this is capsule J there is a weight matrix in between W I J that is learned right this is a static weight matrix and each one of these red red arrows you see here has such a weight matrix attached to it so each each line you see here is actually modulated by such a weight matrix so there is an a quadratic number of these weight matrices flying around and this will also then allow you that maybe this vector is eight dimensional but the input vector here is 16 dimensional what we saw before all right so the out the input of capsule J here it will receive let's see what it receives it will receive the output of capsule will the output of capsule 1 V 1 modulated by the let's let's call this yeah let's call this J modulated by 1 J W 1 J and it will also receive this is a set the output of capsule 2 modulated by the weight matrix for sorry weight matrix for capsule 2 and so on now what it does is it adds this these all up into a soft max so sorry let's write this so soft it will add those all up in a soft max weighted fashion so it will actually compute a a weighted average of those now the weights at the beginning are are just one because it gets each from each lower capsule it gets equal amount of this vector but then this will give you an output so this will give you some output let's put this in green this will give you an output that's I don't know how they call it in the paper let's just call it O J right and then what you do is all right you compare how much do each of the individual contributions agree with OJ so you actually compute for each of these you would compute the inner product so you would compute the inner product of W 1 J V 1 with OJ and you would compute the inner product of W 2 J V 2 with OJ all right the inner product and then these inner products here will become the weighting coefficients for the soft max in the next iteration all right so this I mean this this is a bit convoluted but ultimately what you're saying is if you're a capsule here you'll send your output forward you have an output you send it forward right to the other capsule and the other capsule will so this is this is your output and we'll forget about this weight matrix 6 for now this is your up the other capsule will output its own its own output computed from the lower layers now we do an iteration again if your output now aligns with this you will send more of it and these these two that I've drawn here actually align pretty well right so you'll send more of it is more more more right and now maybe the output that next computed output of the same capsule will be even more in that direction because you've contributed more right you'll send more and then you're like in the next iteration wow these two are really equal sorry this should be red here your ears just keeps being the same and then you say well I'm gonna send even more to that one right whereas another capsule that it's whose initial output was basically whose initial output was basically like this it will by itself compute the inner product with the original this original it will send it here right it will compute the inner product with the original output and it will realize well these do not align very much and then it will send less right it will send less to the next step and because it sends less in the next step of course the output will then probably align even less with that vector and then it will send less and less and less so this is called dynamic routing the the idea behind it is kind of that you route by agreement so you will route to the parent capsules that agree with your output and by agreement we mean kind of the inner product is high after modulating by this weight matrix and that sort of so that basically means this weight matrix is responsible for deciding which information is relevant together whenever you have two vectors that align in the same layer then the in the sense of the capsule networks those represent the same kind of information and those will be routed together to the same capsule in terms of the examples we made maybe if a door and a roof is present then these these these weight matrices that connect door and roof to the house class they will transform a high vector in door and roof into aligning vectors for the house class and thereby saying look these two if I look at them through if I look at a door and a roof through the perspective of trying to be a house right then they are in much agreement on the presence of a house so if I am a house right I am a house and I look at a door and I look at a roof through the kind of from the perspective of being a house right this is this is what these weight matrices do they always have a perspective of the parent capsule then these two things they make a lot of sense together and thus I will route them to the same place so they can both contribute to their being a house now from the perspective of a house if I look at a little beach with a tree on it right then that does not that is not the same that does not really is not the same information as a door or a roof so I will not route this to the house in the in the same strength that is sort of the best way I have of explaining it how these capsules work basically the lower entities will always be routed for the relevance of the higher entities that are trying to are trying to combine the lower entities if that wasn't it's not entirely clear to me either yet but it's the best shot I I can give and the routing is here formalized I find it hard to follow the important thing is that there is an inner loop in all of this so there is an like kind of an an inner iteration and this inner iteration is computed in every forward pass and so these routing where the information goes in the next layer that is only the prior probability for that is learned but the actual routing coefficients those are dynamically computed in every forward pass so every forward pass goes it goes information goes through a layer then it goes multiple steps between two layers until it decides exactly what the distribution for the next layer is and then the next layer computes its outputs and that goes again multiple steps between these layers and the next layer so that's the the basic thing to remember there's also some normalization involved the squash is the non-linearity we discussed so what do they actually train now at the end here they have a they have these ten capsules and each capsule will be responsible for recognizing one the presence of one digit in the MNIST data set of course and so what they do is they take the length of these vectors that are output by these capsules these capsules are feed-forward capsules as opposed to the convolutional capsules here so the feed-forward capsules output again a vector the length of this vector is taken and then it's basically trained like you would train a regression problem and the loss here is specified up here so if the if the image actually does contain this if the training label actually has this digit present this T here encodes that so if if K let's say K is 2 right so if K 2 if there is a 2 in the image when we know that because it's a training image then the length of the output of capsule number 2 should be high and this simply encodes that it should be very close to this M plus an M plus here is that I think they said it to 0.9 so they say you should be the length should be as close as possible to 0.9 whereas if the 2 is not present then TK will be 0 then this part will be active so it's only one of these two parts will be active then the length of the vector so of capsule number 2 should be close to this M negative which is 0.1 it's basically a regression problem saying if if there if the given entity is in the image then please make the length as close as possible to 0.9 and if it's not make it as close as possible to 0.1 so this this is a classic say regression loss on the length of the output vectors the the lambda is just a factor to to dampen the contribution for all the negative classes with respect to the one positive class of course per capsule it turns out this is actually not enough so this will be the classification output but it's it seems not enough they don't say it's not enough but they simply say we additionally do the following so they also do is they introduce a reconstruction loss now if this model is trained correctly then these capsules here these last capsules especially this one maybe that's the capsule corresponding to the class of the digit 8 will not only encode if an 8 is there or not as in the length of the vector output but it will also encode the properties of dates it is a 16 dimensional vector so it will encode hopefully things like the stroke width so then it might encode the maybe the rotation of the digit then it might be controlled the tightness of the of the loop so you can have an 8 with very large loops or it can have an 8 sorry this is a smaller rate I can have an 8 with very tight loops so it might you know encode things like this so technically it is it will be possible to reconstruct from this description reconstruct say the width is high the rotation is zero and the tightness is low then maybe I have a wide widely stroked not tight 8 that is not rotated right so it should be possible to reconstruct this and they they do exactly that so they take this last capsule of the class that is the actual training label that's called the reconstruction target and they feed this to a simple feed-forward neural network that at the end you see this is exactly the MNIST size will try to reconstruct the the image so if the image here this image goes in then it goes all through here it will take the class for here feed it through this network reshape it to an image again and hopefully what will come out is again this for here and it will then have an auxiliary auxiliary loss in addition to the loss of this of this classification loss here will auxiliary loss that tries to reconstruct the original image right and that's simply a I believe it's just an L2 reconstruction loss that is that is scaled down that it doesn't dominate so they also train the network basically to reconstruct this and I believe they do this because the length isn't quite enough to make it do what they want it to do thus they by having this reconstruction here they really kind of enforce that the individual capsules the individual dimensions must encode some kind of information about the original image and since the original images in the MNIST data set at least vary by those things by stroke width by rotation by tightness that by this loss will be reflected in the in the reconstruction all right so how are they doing here you see different examples of inputs and then reconstructed outputs and this you know seems pretty good actually so you see here all of these the input image is reconstructed fairly well so the numbers up here in the fall so the right are the failure cases here it the input image is a five labeled in the training data but the network actually classifies it as a three but then if you now you have two choices right this this is the same sample I have two choices for reconstruction either you reconstruct the capsule that is actually the is that you know is the true capsule that should be activated and you reconstruct from that or you reconstruct from the capsule that the network says the it classifies it as so here it mixed up a five four three if you still take the five the capsule and reconstructed you see it actually looks like the original image but it looks much more like a five and if you take the three capsule to reconstruct which is what the network classified this as it's still it looks like the original image but it looks much more like an actual three right it's it's missing the the part up here whereas over here it's it's missing this part here so that the network really seems to kind of learn the different variations of these digits and in an ambiguous case such as this one it you know it can it can actually go either way and it can actually reconstruct the original output in either interpretations once as a three and once as a five it will be interesting to see what the actual lengths of the vector of both of these classes were that were mixed up and here they compare their accuracies so they have a baseline model which I believe is just a CNN where they get a decent kind of error and then the capsule networks they get a lower error and here you see as you add the reconstruction loss and as you add routing more so one step of routing simply means the first step is where you send your output equally to each parent that is as in the classical neural network case but if you introduce three steps of routing then your error drops even lower so they they kind of are on par with baseline CNNs on MNIST here they also explore what their capsules learn so as I said the individual capsules the dimensions should encode kind of properties of the variations of the of the class class samples and here they explore this in the different capsules so they change some dimensions and they run it through their reconstruction networks and indeed they discover that there is like a scale and thickness dimension stroke thickness dimension there's a skew dimension and so on width and translation so that this is pretty remarkable these networks really if you train them in this way they really seem to learn about the entities and about the properties of the entities and that seems to be quite interesting you see that there's everything here stays well within the class that the capsule is assigned to they also yeah this robustness to affine transformations where they improve over the baseline it's kind of an auxiliary experiment the next interesting experiment is what they call the multi MNIST experiment the multi MNIST experiment is done by taking two different MNIST digits and basically just overlapping them so that they have you know shift them slightly but as you see here or here they are overlapped heavily and the task of the network is to figure out which two overlapping digits are in the image and the the network is very very good at doing this the capsule network that is and better than the the baselines because the capsule network simply encodes the presence and properties of a particular instance in the image if you simply take the top two length capsules and then reconstruct those independently then you're you can you can you can basically segment the image and you see this here so the different colorations come from two different reconstructions of the image from two different capsules so green is from one capsule and red from the other capsule so the network correctly identifies that it's a 6 and the zero right and it also correctly identifies not only which pixels belong to the 6 and which belong to 0 but also pixels that belong to both so that's not a not a problem if you use capsule networks as they are are notable to say here they the way they train is is they train the actual reconstruction by only reconstructing one at a time so the kind of the premise of the data set is that you actually have access to the underlying individual digits while training so like the images of the individual digits you don't only have this label here but that's a detail here are some kind of failure cases where it it misclassified or you miss specify the capsules and it's kind of unable use here you see to to assign the digits of the misclassified or the pixels of the misclassified thing it's quite interesting to look at the failure cases but I find it more interesting to look actually the success cases and the kind of ease at which the at which the capsule networks can do this simply by how they're structured alright so then lastly they also experiment on C for 10 and interestingly the C for 10 experiments show that the capsule networks don't perform as well there and as you know C for 10 is a data set that is about the same size as MNIST but it's first of all color and second of all is natural images and so they have quite a bit of clutter it's not black and white black background white digits it's actually there's a sky like on an image there's lots of things going on and right there's my tree and there's stuff here and there's stuff here and the the capsule networks they like to account for things in the image so they like to have a capsule corresponding to everything that's going on here and here and here and here and here if the whole background is black that is not a problem you can account for simply the background but if there's lots of things going on then these capsule networks get they get they get a bit over explanatory they want to explain everything and that degrades the performance now this paper basically says yeah you can have a something like a none of the above category and they found that it helped to introduce that in my opinion that it I think the the the solution will be more towards introduction of a better loss function for this because like such that you don't need kind of to explain the entire thing rather than here we'll hear what you do is you simply explain it by saying it's none of the above but it's incredibly hard to balance that my opinion yeah all right so that is basically the end of this they say they have a discussion here where they compare capsules against other related work but I hope that you kind of got an overview of how this works now and as much as possible and with that that was it for me and thanks for watching bye bye
 Hi there! Today we're looking at dynamic routing between capsules by Sara Sabour,
 Nicholas Frost and Jeffrey Hinton of Google Brain. This paper is a bit older
 but it's made quite the impact at the time and so we'll go through it. I find
 this pretty hard paper to read and kind of understand because a lot of things
 are very implicit and hand wavy. So we'll kind of go through it and try to get the
 best out of it, try to explain what capsules are and what they do and how
 they stack against current networks. So capsule network in essence is a
 new type of neural network made of capsules. And here it says a capsule is a
 group of neurons whose activity vector represents the instantiation
 parameters of a specific type of entity such as an object or an object part. Kind
 of cryptic but so what they're saying is that in a capsule network, let me try to
 draw one here actually, in a capsule network you have what's called capsules.
 Capsules you can imagine as just little blobs of things right? And they're also
 ordered in layers in this case. Let's actually leave away the second layer. And
 each of these of these capsules will correspond to an entity in the input.
 Let's say the input is an image. So somewhere here there is an image right?
 Then maybe this capsule here will be responsible for detecting is there a
 wall in the image. And this one will be responsible for detecting is there a
 roof. This one will be is there a door. And this one will be responsible for
 detecting is there a lake in the image right? So now each of these each of these
 capsules can for on one hand can either be high or low. So if you if you imagine
 now a situation where wall high, roof high, door high, lake low. It means
 probably the image has a house on it right? But second of all not only can it
 predict whether or not a given entity is present in an image but the individual
 capsules are also responsible for encoding the exact way or shape or form
 that this entity takes. So the wall could have different aspects such as color
 color green. It could have size tall. It could have orientation. orientation is
 like I don't know vertical. Cool. Then roof could have angle right? Angle wide.
 So it's a wide roof or a flat roof right? These are these are kind of attributes
 of these things that also the capsules would encode. So ultimately what these
 capsules that they are proposing will output is the roof capsule here for
 example would output a vector. So the output of the roof capsule is a let me
 draw a coordinate system is a vector. Now the length of the vector will
 represent so that the length draw this norm here will represent the probability
 that the roof is in the image. That there is a roof in an image right? The roof is
 element of this input image. This is simply the length and the individual
 coordinates will encode these attributes. So this here for example this axis could
 be the angle of the roof and this axis could be the color. Let's say just that
 the angle is like some degree number that can be positive or negative. Maybe a
 roof can be like this. Right this so this is but in essence this is a flat roof
 and this is a very narrow angle roof. So you can imagine something like this and
 then the color could also be maybe parameterized on a one-dimensional. It
 can have more dimensions than two I just can't draw more. So the depending on
 where this where this arrow now points the for example this vector here has the
 same probability that there is a roof in the image like if the output is this but
 the color will be different. The angle will be the same because they're roughly
 on the same this axis here but the color of this will encode a different
 different colored roof. And then if the vector is something like this a very
 short vector it will encode the same the same angle and color directions. So maybe
 I shouldn't say the position on the axis it's more like this angle and this this
 angle that encode the attributes. So the kind of the angular components if you
 will encode the attributes and the length encodes the probability. So this
 small vector has the same direction in terms of color and angle of the roof but
 it's much less probable much less likely. So this if the capsule outputs the
 little blue vector here it says well if there is a roof it's going to be this
 color in this angle but I'm really that really don't think there's a roof in
 this image. Whereas if it outputs the large green one then it says I'm pretty
 sure that there's a roof and it's going to be this angle and this this this
 angle and this color. Alright so that's that is what each capsule is supposed to
 do. Each capsule takes the input and outputs a vector that encodes if the
 entity that the capsule is responsible for is present in the image A and B
 what properties this entity has. And then we get to the point where there's the
 next layer of capsules. So the next layer of capsules takes information that each
 capsule here takes information from each capsule in the lower layer like like
 you're used to from your neural network and integrates this information and
 we'll talk about how this works. It integrates all of this information right
 all of these are vectors now that come from the lower integrates all of this
 information and again each capsule in this next layer is responsible for a
 entity. Now these entities in the higher layers are usually composite entities of
 the lower layers. So this one here could be responsible for house, this one could
 be responsible for national park, national park and this one could be
 responsible for beach or something like this right. And then each of these will
 integrate all of this information from the lower layers and then come up with
 their own output vector encoding whether or not a given entity is present in the
 in the image. Of course the house class will pick up if there is a door a roof
 and a wall in the image the house classes will pick up on that or that's
 how it's meant to work house class is meant to pick up on that and then itself
 output a large vector saying there's probably a house in this in this image.
 So each of these capsules in by itself is responsible for encoding the presence
 and attributes of a object or object part or entity or part of entity in the
 given input data. And of course the last layer here it will simply be your
 classification layer. So in the last layer you have as many capsules as you
 have classes in your classification task. So this is mainly for a
 classification task and then you can classify and you can kind of train the
 whole system like this. So how exactly this happens we'll see next.
 Alright so they make kind of analogies to the visual system and so on.
 We'll jump these you can everyone that does deep learning in some way is trying
 to to make that. We're rather going to the specifics of how these capsules work
 and how their specific suggestions for them. Note that they say this is in no
 way the only implementation of capsules. It's just kind of an example to show how
 one could do it. Alright so first of all they present their what you might call
 non-linearity. So their non-linearity what it needs to do is if you look at
 these capsule networks the outputs here the length of the outputs of these
 vectors right they're supposed to represent probabilities and as such they
 they need to be so here it roof this door maybe a vector like this wall maybe
 a vector like that. So initially we simply specify the output is a vector
 and in essence these capsules are implemented in much the same way like
 your classic neural network layer would be implemented. So each of these
 capsules will be in essence a neural network layer by itself that outputs a
 vector. There's nothing constraining the length of the vector initially so
 their non-linearity does constrain the vector to be of maximum length 1 and of
 minimum length 0. That's this non-linearity here. So S here is the
 unscaled output of the capsule and you can see here if the length of S gets
 close to 1 or sorry gets really large then this here becomes irrelevant.
 This whole term will be 1 and then the length of the final output of V here
 will be 1. Right so if this is very large then the the length of the scaled
 output will be 1 however if the if the length is really small of the original
 output so if this goes towards 0 then this becomes irrelevant this becomes
 irrelevant this will go towards 0 and the entire length will go towards 0.
 So this is kind of a nice way to scale these outputs always to be between length 0
 and 1. Then next thing is so how this I find I find the the most complicated
 part right so we'll jump ahead actually to how a capsule's network is implemented
 and this is the the capsule network they implement so first it's an MNIST
 classifier you have an MNIST image here and it first goes through a simple
 convolutional layer that's that's nothing new this is a classic
 convolutional layer is there's 256 channels it has a 9 by 9 filters and
 stride 1 so it will output a 20 by 20 time by 256 tensor then each of these
 so each of the outputs here is sent to each of these capsules and now they're
 convolutional capsules so that makes it a bit more complicated but don't you
 know don't worry primarily about them being convolutional capsules the
 analogy is exactly as in a classic neural network you can implement these
 capsules as void-feed-forward capsules or as convolutional capsules and maybe also
 as transformer capsules I don't think anyone's done that all right there's a
 paper for you the so you'll send you'll send the output of this convolution
 layer to each capsule and then you have basically just two layer of capsules
 here the first layer consists of 32 what they call primary caps sorry the these
 32 capsules each will output an eight dimensional vector and I'm simplifying
 here it's it's convolutional but they will just for simplest they will each
 output an eight dimensional vector right and these are exactly as we said before
 so each of these will be responsible ultimately for a given entity or part of
 entity being there like in MNIST this could be is there a little curve on the
 bottom left side right this might indicate the presence of a six or an
 eight something like this and then the these capsules here each is they
 represented as a row so each of these rows here is a capsule and we have ten
 of these and these are your simply your final classification capsules so each
 capsule is responsible for indicating the presence or absence of one particular
 class of digits so this will be of a one of a two of a three of a four and so on
 of a zero I guess somewhere as well so these are ten capsules and the question
 is how does information go from a capsule here from the output of a
 capsule or to any of capsule here and the easy way to do this is simply to say
 as in a classical neural network the output here simply goes to the input
 here just you just put it there basically on on unchanged now there is a
 bit of an issue here with the dimensions but you can simply say well we simply
 put a weight matrix in to route into the capsules but the idea of these capsules
 and this paper is to say wait wait these capsules actually we want to make them
 decide to which capsule in the next layer will they send their input right
 so the capsules can kind of decide where they want to send their output to like
 where is this where is the capsule that detects the maybe this one detects is
 there a line in the right side of the image right indicating maybe a seven or
 a one this is probably most relevant for the one class and for the seven class so
 it might decide to route its output there and the idea of how this routing
 happens is basically the topic of this paper so the the capsules route their
 output to the appropriate next layers capsules how is this done all right this
 is done via the what's called the routing mechanism that I find it quite
 poorly described here so I will simply draw it I will simply try to make it up
 all right so we have capsules and as I've drawn them before right we have one
 two three capsules and we maybe have two parent capsules each of these capsules
 here will output a vector as we said and we'll only do it for this this one sorry
 vector here so this will output this vector and needs to decide where to here
 or to here do I send to this output now what it does is there is an iterative
 procedure that has multiple steps and this is I think this is at least the way
 I understand I think the important part to understand is that if we forward pass
 data through this network it actually doesn't go forward in a straight line
 what it actually does is it goes through a layer and then it does multiple steps
 in between layers until it has decided where it wants to go in the next layer
 and then it goes on to the next layer and if there's another capsule layers it
 does again multiple steps before it goes on so that's that's my take on it and
 the multiple steps are as follows first I'll send my output vector to to all of
 the all of the layers like equally all of the parent capsules and so will will
 everyone else right everyone will send theirs equally to the parent now this
 isn't just done and this may be here this isn't just done just by sending it
 but this is actually done by modulation of weight matrices so each thing here if
 this is capsule I and this is capsule J there is a weight matrix in between W I J
 that is learned right this is a static weight matrix and each one of these red
 red arrows you see here has such a weight matrix attached to it so each
 each line you see here is actually modulated by such a weight matrix so
 there is an a quadratic number of these weight matrices flying around and this
 will also then allow you that maybe this vector is eight dimensional but the
 input vector here is 16 dimensional what we saw before all right so the out the
 input of capsule J here it will receive let's see what it receives it will
 receive the output of capsule will the output of capsule 1 V 1 modulated by the
 let's let's call this yeah let's call this J modulated by 1 J W 1 J and it
 will also receive this is a set the output of capsule 2 modulated by the
 weight matrix for sorry weight matrix for capsule 2 and so on now what it does
 is it adds this these all up into a soft max so sorry let's write this so soft it
 will add those all up in a soft max weighted fashion so it will actually
 compute a a weighted average of those now the weights at the beginning are are
 just one because it gets each from each lower capsule it gets equal amount of
 this vector but then this will give you an output so this will give you some
 output let's put this in green this will give you an output that's I don't know
 how they call it in the paper let's just call it O J right and then what you do
 is all right you compare how much do each of the individual contributions
 agree with OJ so you actually compute for each of these you would compute the
 inner product so you would compute the inner product of W 1 J V 1 with OJ and
 you would compute the inner product of W 2 J V 2 with OJ all right the inner
 product and then these inner products here will become the weighting
 coefficients for the soft max in the next iteration all right so this I mean
 this this is a bit convoluted but ultimately what you're saying is if
 you're a capsule here you'll send your output forward you have an output you
 send it forward right to the other capsule and the other capsule will so
 this is this is your output and we'll forget about this weight matrix 6 for
 now this is your up the other capsule will output its own its own output
 computed from the lower layers now we do an iteration again if your output now
 aligns with this you will send more of it and these these two that I've drawn
 here actually align pretty well right so you'll send more of it is more more
 more right and now maybe the output that next computed output of the same capsule
 will be even more in that direction because you've contributed more right
 you'll send more and then you're like in the next iteration wow these two are
 really equal sorry this should be red here your ears just keeps being the same
 and then you say well I'm gonna send even more to that one right whereas
 another capsule that it's whose initial output was basically whose initial
 output was basically like this it will by itself compute the inner product with
 the original this original it will send it here right it will compute the inner
 product with the original output and it will realize well these do not align
 very much and then it will send less right it will send less to the next step
 and because it sends less in the next step of course the output will then
 probably align even less with that vector and then it will send less and
 less and less so this is called dynamic routing the the idea behind it is kind
 of that you route by agreement so you will route to the parent capsules that
 agree with your output and by agreement we mean kind of the inner product is
 high after modulating by this weight matrix and that sort of so that
 basically means this weight matrix is responsible for deciding which
 information is relevant together whenever you have two vectors that align
 in the same layer then the in the sense of the capsule networks those represent
 the same kind of information and those will be routed together to the same
 capsule in terms of the examples we made maybe if a door and a roof is
 present then these these these weight matrices that connect door and roof to
 the house class they will transform a high vector in door and roof into
 aligning vectors for the house class and thereby saying look these two if I look
 at them through if I look at a door and a roof through the perspective of trying
 to be a house right then they are in much agreement on the presence of a
 house so if I am a house right I am a house and I look at a door and I look at
 a roof through the kind of from the perspective of being a house right this
 is this is what these weight matrices do they always have a perspective of the
 parent capsule then these two things they make a lot of sense together and
 thus I will route them to the same place so they can both contribute to their
 being a house now from the perspective of a house if I look at a little beach
 with a tree on it right then that does not that is not the same that does not
 really is not the same information as a door or a roof so I will not route this
 to the house in the in the same strength that is sort of the best way I have of
 explaining it how these capsules work basically the lower entities will always
 be routed for the relevance of the higher entities that are trying to are
 trying to combine the lower entities if that wasn't it's not entirely clear to
 me either yet but it's the best shot I I can give and the routing is here
 formalized I find it hard to follow the important thing is that there is an
 inner loop in all of this so there is an like kind of an an inner iteration and
 this inner iteration is computed in every forward pass and so these routing
 where the information goes in the next layer that is only the prior probability
 for that is learned but the actual routing coefficients those are
 dynamically computed in every forward pass so every forward pass goes it goes
 information goes through a layer then it goes multiple steps between two layers
 until it decides exactly what the distribution for the next layer is and
 then the next layer computes its outputs and that goes again multiple steps
 between these layers and the next layer so that's the the basic thing to
 remember there's also some normalization involved the squash is the non-linearity
 we discussed so what do they actually train now at the end here they have a
 they have these ten capsules and each capsule will be responsible for
 recognizing one the presence of one digit in the MNIST data set of course
 and so what they do is they take the length of these vectors that are output
 by these capsules these capsules are feed-forward capsules as opposed to the
 convolutional capsules here so the feed-forward capsules output again a
 vector the length of this vector is taken and then it's basically trained
 like you would train a regression problem and the loss here is specified
 up here so if the if the image actually does contain this if the training label
 actually has this digit present this T here encodes that so if if K let's say K
 is 2 right so if K 2 if there is a 2 in the image when we know that because it's
 a training image then the length of the output of capsule number 2 should be
 high and this simply encodes that it should be very close to this M plus an
 M plus here is that I think they said it to 0.9 so they say you should be the
 length should be as close as possible to 0.9 whereas if the 2 is not present then
 TK will be 0 then this part will be active so it's only one of these two
 parts will be active then the length of the vector so of capsule number 2 should
 be close to this M negative which is 0.1 it's basically a regression problem
 saying if if there if the given entity is in the image then please make the
 length as close as possible to 0.9 and if it's not make it as close as possible
 to 0.1 so this this is a classic say regression loss on the length of the
 output vectors the the lambda is just a factor to to dampen the contribution for
 all the negative classes with respect to the one positive class of course per
 capsule it turns out this is actually not enough so this will be the
 classification output but it's it seems not enough they don't say it's not
 enough but they simply say we additionally do the following so they
 also do is they introduce a reconstruction loss now if this model is
 trained correctly then these capsules here these last capsules especially
 this one maybe that's the capsule corresponding to the class of the digit
 8 will not only encode if an 8 is there or not as in the length of the vector
 output but it will also encode the properties of dates it is a 16
 dimensional vector so it will encode hopefully things like the stroke width
 so then it might encode the maybe the rotation of the digit then it might be
 controlled the tightness of the of the loop so you can have an 8 with very
 large loops or it can have an 8 sorry this is a smaller rate I can have an 8
 with very tight loops so it might you know encode things like this so
 technically it is it will be possible to reconstruct from this description
 reconstruct say the width is high the rotation is zero and the tightness is
 low then maybe I have a wide widely stroked not tight 8 that is not rotated
 right so it should be possible to reconstruct this and they they do exactly
 that so they take this last capsule of the class that is the actual training
 label that's called the reconstruction target and they feed this to a simple
 feed-forward neural network that at the end you see this is exactly the MNIST
 size will try to reconstruct the the image so if the image here this image
 goes in then it goes all through here it will take the class for here feed it
 through this network reshape it to an image again and hopefully what will come
 out is again this for here and it will then have an auxiliary auxiliary loss in
 addition to the loss of this of this classification loss here will auxiliary
 loss that tries to reconstruct the original image right and that's simply a
 I believe it's just an L2 reconstruction loss that is that is scaled down that it
 doesn't dominate so they also train the network basically to reconstruct this
 and I believe they do this because the length isn't quite enough to make it do
 what they want it to do thus they by having this reconstruction here they
 really kind of enforce that the individual capsules the individual
 dimensions must encode some kind of information about the original image
 and since the original images in the MNIST data set at least vary by those
 things by stroke width by rotation by tightness that by this loss will be
 reflected in the in the reconstruction all right so how are they doing here you
 see different examples of inputs and then reconstructed outputs and this you
 know seems pretty good actually so you see here all of these the input image is
 reconstructed fairly well so the numbers up here in the fall so the right are the
 failure cases here it the input image is a five labeled in the training data but
 the network actually classifies it as a three but then if you now you have two
 choices right this this is the same sample I have two choices for
 reconstruction either you reconstruct the capsule that is actually the is that
 you know is the true capsule that should be activated and you reconstruct from
 that or you reconstruct from the capsule that the network says the it classifies
 it as so here it mixed up a five four three if you still take the five the
 capsule and reconstructed you see it actually looks like the original image
 but it looks much more like a five and if you take the three capsule to
 reconstruct which is what the network classified this as it's still it looks
 like the original image but it looks much more like an actual three right it's
 it's missing the the part up here whereas over here it's it's missing this
 part here so that the network really seems to kind of learn the different
 variations of these digits and in an ambiguous case such as this one it you
 know it can it can actually go either way and it can actually reconstruct the
 original output in either interpretations once as a three and once
 as a five it will be interesting to see what the actual lengths of the vector of
 both of these classes were that were mixed up and here they compare their
 accuracies so they have a baseline model which I believe is just a CNN
 where they get a decent kind of error and then the capsule networks they get a
 lower error and here you see as you add the reconstruction loss and as you add
 routing more so one step of routing simply means the first step is where you
 send your output equally to each parent that is as in the classical neural
 network case but if you introduce three steps of routing then your error drops
 even lower so they they kind of are on par with baseline CNNs on MNIST here
 they also explore what their capsules learn so as I said the individual capsules
 the dimensions should encode kind of properties of the variations of the of
 the class class samples and here they explore this in the different capsules so
 they change some dimensions and they run it through their reconstruction networks
 and indeed they discover that there is like a scale and thickness dimension
 stroke thickness dimension there's a skew dimension and so on width and
 translation so that this is pretty remarkable these networks really if you
 train them in this way they really seem to learn about the entities and about
 the properties of the entities and that seems to be quite interesting you see
 that there's everything here stays well within the class that the capsule is
 assigned to they also yeah this robustness to affine transformations
 where they improve over the baseline it's kind of an auxiliary experiment the
 next interesting experiment is what they call the multi MNIST experiment the
 multi MNIST experiment is done by taking two different MNIST digits and basically
 just overlapping them so that they have you know shift them slightly but as you
 see here or here they are overlapped heavily and the task of the network is
 to figure out which two overlapping digits are in the image and the the
 network is very very good at doing this the capsule network that is and better
 than the the baselines because the capsule network simply encodes the
 presence and properties of a particular instance in the image if you simply take
 the top two length capsules and then reconstruct those independently then
 you're you can you can you can basically segment the image and you see this here
 so the different colorations come from two different reconstructions of the
 image from two different capsules so green is from one capsule and red from
 the other capsule so the network correctly identifies that it's a 6 and
 the zero right and it also correctly identifies not only which pixels belong
 to the 6 and which belong to 0 but also pixels that belong to both so that's not
 a not a problem if you use capsule networks as they are
 are notable to say here they the way they train is is they train the actual
 reconstruction by only reconstructing one at a time so the kind of the premise
 of the data set is that you actually have access to the underlying individual
 digits while training so like the images of the individual digits you don't
 only have this label here but that's a detail here are some kind of failure
 cases where it it misclassified or you miss specify the capsules and it's kind
 of unable use here you see to to assign the digits of the misclassified or the
 pixels of the misclassified thing it's quite interesting to look at the failure
 cases but I find it more interesting to look actually the success cases and the
 kind of ease at which the at which the capsule networks can do this simply by
 how they're structured alright so then lastly they also experiment on C for 10
 and interestingly the C for 10 experiments show that the capsule
 networks don't perform as well there and as you know C for 10 is a data set that
 is about the same size as MNIST but it's first of all color and second of all is
 natural images and so they have quite a bit of clutter it's not black and white
 black background white digits it's actually there's a sky like on an
 image there's lots of things going on and right there's my tree and there's
 stuff here and there's stuff here and the the capsule networks they like to
 account for things in the image so they like to have a capsule corresponding to
 everything that's going on here and here and here and here and here if the whole
 background is black that is not a problem you can account for simply the
 background but if there's lots of things going on then these capsule networks
 get they get they get a bit over explanatory they want to explain
 everything and that degrades the performance now this paper basically
 says yeah you can have a something like a none of the above category and they
 found that it helped to introduce that in my opinion that it I think the the
 the solution will be more towards introduction of a better loss function
 for this because like such that you don't need kind of to explain the entire
 thing rather than here we'll hear what you do is you simply explain it by
 saying it's none of the above but it's incredibly hard to balance that my
 opinion yeah all right so that is basically the end of this they say they
 have a discussion here where they compare capsules against other related
 work but I hope that you kind of got an overview of how this works now and as
 much as possible and with that that was it for me and thanks for watching bye
 bye