Topographic VAEs learn Equivariant Capsules (Machine Learning Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper vae variational bayesian variational methods variational autoencoder max welling elbo prior student t reparameterization trick log likelihood encoder decoder
#tvae #topographic #equivariant

Variational Autoencoders model the latent space as a set of independent Gaussian random variables, which the decoder maps to a data distribution. However, this independence is not always desired, for example when dealing with video sequences, we know that successive frames are heavily correlated. Thus, any latent space dealing with such data should reflect this in its structure. Topographic VAEs are a framework for defining correlation structures among the latent variables and induce equivariance within the resulting model. This paper shows how such correlation structures can be built by correctly arranging higher-level variables, which are themselves independent Gaussians.

OUTLINE:
0:00 - Intro
1:40 - Architecture Overview
6:30 - Comparison to regular VAEs
8:35 - Generative Mechanism Formulation
11:45 - Non-Gaussian Latent Space
17:30 - Topographic Product of Student-t
21:15 - Introducing Temporal Coherence
24:50 - Topographic VAE
27:50 - Experimental Results
31:15 - Conclusion & Comments

Paper: https://arxiv.org/abs/2109.01394
Code: https://github.com/akandykeller/topographicvae

Abstract:
In this work we seek to bridge the concepts of topographic organization and equivariance in neural networks. To accomplish this, we introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. Furthermore, through topographic organization over time (i.e. temporal coherence), we demonstrate how predefined latent space transformation operators can be encouraged for observed transformed input sequences -- a primitive form of unsupervised learned equivariance. We demonstrate that this model successfully learns sets of approximately equivariant features (i.e. "capsules") directly from sequences and achieves higher likelihood on correspondingly transforming test sequences. Equivariance is verified quantitatively by measuring the approximate commutativity of the inference network and the sequence transformations. Finally, we demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.

Authors: T. Anderson Keller, Max Welling

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there. Today we'll look at topographic VAEs learn equivariant capsules by T. Anderson Keller and Max Welling. On a high level this paper proposes a new type of variational autoencoder where the latent variables aren't independent but are organized in a topographic way. Now what that means we're going to look at that but in essence it means that it can represent transformations in the real world of a certain kind as transformations inside of the latent space of the model. So the whole question is here how do we build a latent space and a model where this naturally happens as we train it. We want the real world to somehow correspond to the latent space in a way such that if the real world moves the latent space moves equivalently or equivariantly. That's where this word is going to come in. So we're going to go through the paper. I have to say I don't understand this fully as well. These variational frameworks they are always kind of I feel kind of math heavy and they take a very different approach than the papers I might be used to. So I'm going to tell you what I think is going on here and if I'm completely wrong this is entirely possible please let me know. Alright let's dive into the paper. This is the first graphic right here that shows kind of an overview over the system. So what do they want to achieve? What they say is we're not going to consider we're going to try to build a generative model like a variational autoencoder but we're not going to consider any kind of data. We're going to consider data essentially frames of a video. So we're going to assume that what we're looking at is kind of a video and the transitions inside the video are sort of continuous sort of monotonic and slow. So here you can see the seven rotates slowly and also changes its color slowly relatively monotonously over this sequence. So what they're going to say is we're gonna our model is going to take this entire sequence one a picture is going to be kind of the focus here so this green one is the focus but we're going to take in this entire sequence right here into the model and we want the model to come up with a latent representation of the focus image. In this case it's going to be, we'll jump a step here, is going to be this thing right here. Let's call that I don't even remember how they call it let's call it like Z hat. Okay this is a latent representation of the focus image and now obviously in a regular variational autoencoder I could now push this again into the decoder and get back the same image and I can do so here as well. However we want something else as well. We also want that if I now transform my latent space in a certain way and this way is going to be this role operation in this paper. If I transform my latent space in this way I want this to correspond to moving forward in this sequence right so I have a sequence as an input and I say well my latent space should be such that if I perform certain operations right here in this case I roll by 10 that that corresponds not to the picture that I have input but to the picture that would be if I were to observe this transition 10 steps into the future. So roll by 10 and roll in this case means you can see here they have two of these what they call you know capsules I think they call them capsules the left one and the right one and the roll simply means that I take every variable latent variable and I simply roll them forward so this is over the latent dimension I just roll them forward by one step I do that 10 times this is as you can see this is arranged in sort of a torus here and a 1d torus so I can just roll this around and also this capsule I can just roll it around 10 times and that hopefully if we train the model correctly should correspond to not the input image but the image that is 10 steps into the future. So that is the goal now we don't want to train a model explicitly to predict 10 steps into the future that would be would be a valid task but it's not what this model wants what this model wants is say can we build a model architecture and the latent space architecture such that this is kind of happens automatically and let's see well you can already see kind of how this latent space comes to be I said this Z hat here is going to be the latent representation you can see that is not the thing that is directly output by the encoder the encoder in this case outputs many things so it outputs a Z variable so the Z hat is what I call kind of Z normalized the Z variable is kind of Z unnormalized so it outputs a Z variable for the focus image but it also outputs these U squared variable or it outputs the U variables which we then square so these U variables right here are output I'm gonna guess this is from this image and this is from this image and this is from this image and also kind of look into the future right here and yeah so I have these U variables and I define sort of a a window a context window around which I look and I also predict them I square them and then I sum them all up but pull the square root right here and I divide so this is why I say kind of a normalized Z is what comes out of this but it's fairly fairly complicated right but this is going to in a way encourage this behavior so let's see why that is and for that I want to just draw back a little bit to like a regular VAE a regular variational autoencoder so if in a regular VAE you have like an image this is encoded decoded and you get back an image right so in a regular VAE what you assume is you assume that the latent space is sort of made up out of these independent latent variables latent random variables they're Gaussian distributed and yeah there I already said they're independent from each other and you you claim if I know the latent variables so essentially if I know the mean and variance of these then you know producing an image is is easy right you can simply train a neural network I input you know which which var I input what values my latent variables are or how the Gaussians are parameterized alternatively I input that and I train the decoder to produce a picture from that that is easy the question is if I have a picture trusty cat right here if I have a picture what are the corresponding latent variables you know what are the values of the latent variables that makes sense right here and of course in a VAE we train the encoder and the decoder jointly such that they cooperatively can construct this latent space like okay how how should how should the latent space look from which the decoder decodes but I just want to turn your attention to the question of the encoders job is essentially to take in an image and produce what values the latent variables are and the latent variables are assumed to be independent from each other and Gaussian distributed now this is where this model right here differs okay so this model says well we're going to assume we have observed and latent variables observed variables X and latent variables T observed are I guess the images or the image sequences and T are the latent variables so this I guess what this would be equivalent to Z hat to what I called Z hat they call team all right so they say will formulate the joint distribution note that in this framework in these variational frameworks I don't it's not my thing either but what you do is you always you propose a mechanism by which the data and by which the variables are generated so you as a designer of the algorithm propose the structure of how the latent variables work together and then you have some small parts in there that you say well these things I don't know I'm gonna let a neural network do these things but essentially you come and you impose a structure upon the world right and you know if you get the structure correct your model will work fine if you don't get the structure correct your model won't work fine but this is a bit of a different way of working than you know saying well I train a conv net to predict so we're going to propose our structure we're going to say the joint distribution of observed and latent variables factorizes into these two it factorizes into this conditional so if I have the latent variables right then what are the images and times the prior across the latent variables now we already seen this distribution it's the first one is listed here again this conditional distribution that's simply your decoder in the VAE framework and that's written here it essentially says well to produce an image I'm going to put T the latent variable into this neural network G right here and that will give me the distribution of my output image so this is your decoder in the VAE now the interesting part and where it differs from a regular VAE is right here where they say well how do our latent how does our latent space look well this is zooming around our latent space isn't a independent Gaussians it's actually this TPOT distribution this topographic product no where where does it I forgot what it I forgot what it's what it's called a topographic product of student T's model the TPOT topographic product of student T that's going to be our distribution and that distribution is going to encourage this topographically organized latent space right so we can ask how does it how does it do that note that the encoder isn't here yet because we've only we've defined we've imposed degenerative process of the data the generative process starts at the latent space I said if I know what the latent variables are I can just ask my decoder to produce an image so this distribution here tells us you know the latent variables are distributed like this and then there we go now obviously what we want is we want our encoder to produce the variables the latent variables but we also want what the encoder produces to follow this distribution right here and that's going to be the sort of difficulty right here because what we know what we can train with back propagation is pretty much Gaussians you know like we can train things where we can apply the reparameterization trick that's stuff we can back prop through stuff we can Gaussians we can sample from efficiently and so on we have closed form solution for the KL divergences in the objectives so essentially what we can do in these variational frameworks is Gaussians not topographic product of student is however here they show okay we can in fact construct a product of student is this is no this is not yet a topographic product is just a product of student is distribution from Gaussians and that is I take one z variable and I take a bunch of u variables and they're all distributed like Gaussians and I square the use I sum them up I average them and then I take the square root and I divide z by dot and this variable right here that's going to be a univariate student t random variable this should be kind of known if you've ever taken statistics or like use the t-test for anything okay and you know this is already quite familiar and I can extend this now to the multi-dimensional case so if t is a multi-dimensional student is random variable composed of independent Z's and use then we can construct t as a vector and that is going to be distributed according to a product of student t's variable and this should connect to what we've seen before right we said that this models organization of the latent space is pretty much of this form that we saw right here we have the z variable divided by the square root of the sum of the squared u variables and now we learn how we can construct the product of student t's latent space given z and u independent Gaussians and that is you know now it should connect for you in deep learning variational frameworks we can work pretty much only with Gaussian random variables in this model we want to work with product of student t random variables and here is the way how we can construct the product of student t random errors from Gaussian random variables so that's why here we the neural networks will output the Z and the u that's what they will output that's those are those are Gaussians or supposed to be Gaussians and then we transform them by dividing them and summing them up in this way to the latent variable that the decoder receives which is this Z hat or t I guess to this is what the decoder receives so we know that if the encoder as output Gaussian random variables the decoder will receive a product of student t random variable now why is the product of student t random variable special in any way because it enables us to what they call here introduce topography in essence and they formulate this a little bit what it does is it it lets it if if some of the use in this some and some of the you in this some are the same which you can see by the indices in this case they are not but if some are shared that means that the two were the two t variables not the two Z the two t so this is one t and this is another t right this is t1 this is t2 lots of t these two variables will no longer be independent they will actually be dependent on each other so this is a way how we can construct latent spaces where some of the variables are actually correlated or in some other way have have higher order correlations with each other meaning that the value of one is not independent from the value of the other one and that is pretty much a basis for what we want for constructing these topographic latent spaces so here they say introducing topography essentially what we're going to do is we're not we're going to define neighborhoods across our u variables and we're going to share the u variables according to these neighborhoods and that's going to make the in the components of t dependent on each other and this sounds complicated but essentially you can imagine instead of having like four latent random variable which are all Gaussians now we have simply one set of z variables and one set of u variables and we're going to consider an entire sequence and not just one one image right so we were going to consider an entire sequence of images like this right here every image produces one z and one u variable and then when we consider an image let's say this is the focus right now we consider its z and we consider a neighborhood of use and that's just going to amount sort of like a convolution like this is maybe a neighborhood of three so we're going to consider this u this u and this u so we're going to construct the z on top of the fraction divided by this thing squared this bubble here squared this bubble here squared square root of top on top of that and that's going to be our t so the t for this image right here that's going to be this whole fraction so when we train the VAE we input the whole sequence we focus on for example this picture we construct its t by looking at its z and its neighborhood of use then we put that t into the decoder the decoder is going to produce an image and then we can apply a loss function between those two okay so that is the loss that's the loss function right the loss function note that the loss function doesn't say you need if you roll ten times then it needs to be the picture that's ten times ahead that is not the case at all we actually don't have the role function in here but even now even once we introduce the role function in the in the latent space we're not going to explicitly train the model to predict the future we're simply going to construct as we did here the latent space such that it such that this naturally happens so how are we going to do this almost the same and here you have they talk about capsules so you can see that they divide this neighborhood structure so the W defines the neighborhood structure you can see here some of the use they are connected and then other ones are connected but these use are not connected with those they kind of talk about capsules essentially it's just that they make some of the variables dependent on each other and some not or or when they do these neighborhood things they just have two sets of variables like to have two sets of Z's and use and they only yeah they construct two T variables and that that's what they call capsules that I don't I don't know why the capsule terminology enters this paper necessarily but you know they they want to draw a connection here so temporal coherence now we get to how do we organize this latent space such that the role operation now also gets in and this is pretty simple it's actually just an extension of this right here so here if you consider these images here as images of a sequence we always said well you need to be connected to sort of your your neighboring variables and now sorry your neighboring you variables as they are right and now we're going to say the same thing but but I'm going to draw the critical path here again so this we have a Z variable right here we have you variables from the neighborhood okay and we're going to take the Z variable on top of the fraction and we're going to take the U variables below the fraction right here like so like so like so now before we do this before we take the U variables here below the fraction we're going to roll the U variables according to their distance from according to their distance from the focus so in this case this would be simply one rollback this would be simply one roll forward so in the language of this paper what this means is that we don't want we we don't want this image or it given a particular position in this image right this position right here if we simply apply the classic neighborhood structure we say we want this position in this image to be correlated with the same position a step back and a step forward now if we construct the role like this what we're saying is no no no no I don't want I want I want this position to be correlated with maybe this position here and this position there like slightly behind and slightly ahead but I'm obviously not going to tell the model what I expect I simply say please this image is one time stack well black this image is one time step back from me please roll the latent space by one and that's going to be your relevant variable and in this case it's please roll the latent space of this thing one forward and that's going to be your relevant latent variable so it's not that we train we we train rolling this t variable here because the t is what finally comes out we're not training this t to roll forward or back and then predict ten steps ahead we're simply saying how you are influenced you as a focus how you are influenced by pictures before and after you you're not simply taking into account their latent variables you want to take into account rolled versions of their latent variables in order for you to reconstruct yourself in the training objective and it turns out at least that's how I understand it right and it turns out so here you can see the whole process we're going to take images we're going to produce mean and variance of late of Gaussian variables for the Z and the u variables so if you had just a VAE it would just be this right here and those will be a layer you're latent variables but not here we produce two sets Z's and use then we're going to construct the t variables I don't know why this is on the bottom here but then we're going to construct the t variables according to this formula W here is the neighborhood structure you define it you and Z are the variables you produced from your encoder or you sampled from what your encoder produced and mu here is also a learnable parameter a learnable mean parameter and then we want to stick this these T's into you're going to stick these T's into this neural network now here it says Z and ZL and UL but essentially this here this here these create T oh here it's here you're going to stick the T into your decoder neural network remember the G how do we get the picture from the latent variable that's the decoder and stick that into the decoder and out you get an image and you train it with the classic elbow the evidence lower bound which says okay what I want is I want to reconstruct the picture accurately right that's this term right here to reconstruct the picture accurately but I also want that my Z well essentially what I want is that my T variables are distributed according to this TPOT distribution I want to enforce that but I can't right I can work with Gaussians so what but what I can do is I can say well the Z variables and the U variables they must be as Gaussian as possible so I penalize the KL divergence between what I produce which is this right here and the Gaussian like a a pure Gaussian this has a closed form I can I can calculate KL divergences from what I produce with Gaussians no problem okay and that's the training loss and I simply average that over the input sequence and there there you go now the evaluation of these things I have to say after reading through the experiments in the evaluations this is this is a paper kind of an idea at least I feel so right correct me if I'm wrong but I feel that this is sort of an idea paper it's like here's an idea it works if we you know specifically construct a data set for it and if we specifically also the experiments are appear to be kind of fiddly like you have to really you know get your parameters right to make this work but if you do then you know the model behaves as you as you expect and so they measure things like is the rolled version of the latent variables really equal to the latent variables a couple of time steps ahead and things like this and they produce these these maps so here is one where the latent space isn't a 1d torus like we looked at so 1d torus is this right so you go around around around sorry and this is a 2d torus so a 2d torus is like a plane and if you leave here you come back here and if you leave here you come back here so if you if you roll this up and then you you have a pipe and if you close the pipe you have like a donut so that's a torus so if they have a topographic space like a torus they and they simply apply that to MNIST the test set sort of looks like this I don't know if you want to read something into this like feel free I'm not sure but in when they go with the sequences so here you see like the sequences I think on top is what they input and then this is the continuation that the model doesn't see on the bottom is what the model produces you can see the model does get to a point where it understands how these sequences go here all right it goes large large and then it kind of flips around to the smallest this is a expected behavior here as well the rotation it model continues the rotation and it turns out even if the model is just trained with they have these these experiments even if the model is just trained with single transformations so either a role sorry either a rotation or a scale transformation or a color change it can generalize to multiple transformations at once as you can see right here colors and rotations can the model can generalize to that fairly fairly well okay I don't want to get too much into the experiments because I'm not sure how important that the numbers here are I'm safe to say if you construct this model and if you apply it to the you know problems where exactly this is needed and if you get the hyper parameters right then this model actually works it's better whereas a regular neural network it could not easily incorporate the concept of these slow changing transitions it would sort of have to learn okay what color comes after red orange okay what color comes after orange yellow okay what color comes after yellow green I guess the other model has to learn that as well but this model it cannot represent the transition in a sequence as sort of as it has to learn it as a parameterized function rather than being able to map it to an internal transformation of the rate of the latent space like the topographic VAE can do okay that was it for me I'm not competent enough to tell you how big of a step this is it feels to me like a little step it might be a giant step I don't know okay it feels to me like it's kind of an idea paper to show something neat that you could do in an idealized case it might be that this is a much bigger deal than than I think I thought it was a cool paper I thought it was a neat idea it's written even though it's I think under you know more high love sorry more more so I'm not as competent at it but I could still make sense of it so if you enjoy this give it a read yeah let me know if you have any comments and that was it bye bye thanks
 Hello there. Today we'll look at topographic VAEs learn equivariant
 capsules by T. Anderson Keller and Max Welling. On a high level this paper
 proposes a new type of variational autoencoder where the latent variables
 aren't independent but are organized in a topographic way. Now what that means
 we're going to look at that but in essence it means that it can
 represent transformations in the real world of a certain kind as transformations
 inside of the latent space of the model. So the whole question is here how do we
 build a latent space and a model where this naturally happens as we train it.
 We want the real world to somehow correspond to the latent space in a way
 such that if the real world moves the latent space moves equivalently or
 equivariantly. That's where this word is going to come in. So we're going to go
 through the paper. I have to say I don't understand this fully as well. These
 variational frameworks they are always kind of I feel kind of math heavy and
 they take a very different approach than the papers I might be used to. So I'm
 going to tell you what I think is going on here and if I'm completely wrong this
 is entirely possible please let me know. Alright let's dive into the paper. This
 is the first graphic right here that shows kind of an overview over the system.
 So what do they want to achieve? What they say is we're not going to consider
 we're going to try to build a generative model like a variational autoencoder but
 we're not going to consider any kind of data. We're going to consider data
 essentially frames of a video. So we're going to assume that what we're
 looking at is kind of a video and the transitions inside the
 video are sort of continuous sort of monotonic and slow. So here you can
 see the seven rotates slowly and also changes its color slowly relatively
 monotonously over this sequence. So what they're going to say is we're gonna our
 model is going to take this entire sequence one a picture is going to be
 kind of the focus here so this green one is the focus but we're going to take in
 this entire sequence right here into the model and we want the model to come up
 with a latent representation of the focus image. In this case it's going to
 be, we'll jump a step here, is going to be this thing right here. Let's call that
 I don't even remember how they call it let's call it like Z hat. Okay this is a
 latent representation of the focus image and now obviously in a regular
 variational autoencoder I could now push this again into the decoder and get back
 the same image and I can do so here as well. However we want something else as
 well. We also want that if I now transform my latent space in a certain
 way and this way is going to be this role operation in this paper. If I
 transform my latent space in this way I want this to correspond to moving
 forward in this sequence right so I have a sequence as an input and I say well my
 latent space should be such that if I perform certain operations right here in
 this case I roll by 10 that that corresponds not to the picture that I
 have input but to the picture that would be if I were to observe this transition
 10 steps into the future. So roll by 10 and roll in this case means you can see
 here they have two of these what they call you know capsules I think they call
 them capsules the left one and the right one and the roll simply means that I
 take every variable latent variable and I simply roll them forward so this is
 over the latent dimension I just roll them forward by one step I do that 10
 times this is as you can see this is arranged in sort of a torus here and a
 1d torus so I can just roll this around and also this capsule I can just roll it
 around 10 times and that hopefully if we train the model correctly should
 correspond to not the input image but the image that is 10 steps into the
 future. So that is the goal now we don't want to train a model explicitly to
 predict 10 steps into the future that would be would be a valid task but it's
 not what this model wants what this model wants is say can we build a model
 architecture and the latent space architecture such that this is kind of
 happens automatically and let's see well you can already see kind of how this
 latent space comes to be I said this Z hat here is going to be the latent
 representation you can see that is not the thing that is directly output by the
 encoder the encoder in this case outputs many things so it outputs a Z variable
 so the Z hat is what I call kind of Z normalized the Z variable is kind of Z
 unnormalized so it outputs a Z variable for the focus image but it also outputs
 these U squared variable or it outputs the U variables which we then square so
 these U variables right here are output I'm gonna guess this is from this image
 and this is from this image and this is from this image and also kind of look
 into the future right here and yeah so I have these U variables and I define sort
 of a a window a context window around which I look and I also predict them I
 square them and then I sum them all up but pull the square root right here and
 I divide so this is why I say kind of a normalized Z is what comes out of this
 but it's fairly fairly complicated right but this is going to in a way
 encourage this behavior so let's see why that is and for that I want to just draw
 back a little bit to like a regular VAE a regular variational autoencoder so if
 in a regular VAE you have like an image this is encoded decoded and you get back
 an image right so in a regular VAE what you assume is you assume that the latent
 space is sort of made up out of these independent latent variables latent
 random variables they're Gaussian distributed and yeah there I already
 said they're independent from each other and you you claim if I know the latent
 variables so essentially if I know the mean and variance of these then you know
 producing an image is is easy right you can simply train a neural network I
 input you know which which var I input what values my latent variables are or
 how the Gaussians are parameterized alternatively I input that and I train
 the decoder to produce a picture from that that is easy the question is if I
 have a picture trusty cat right here if I have a picture what are the
 corresponding latent variables you know what are the values of the latent
 variables that makes sense right here and of course in a VAE we train the
 encoder and the decoder jointly such that they cooperatively can construct
 this latent space like okay how how should how should the latent space look
 from which the decoder decodes but I just want to turn your attention to the
 question of the encoders job is essentially to take in an image and
 produce what values the latent variables are and the latent variables are assumed
 to be independent from each other and Gaussian distributed now this is where
 this model right here differs okay so this model says well we're going to
 assume we have observed and latent variables observed variables X and
 latent variables T observed are I guess the images or the image sequences and T
 are the latent variables so this I guess what this would be equivalent to Z hat
 to what I called Z hat they call team all right so they say will formulate the
 joint distribution note that in this framework in these variational frameworks
 I don't it's not my thing either but what you do is you always you propose a
 mechanism by which the data and by which the variables are generated so you as a
 designer of the algorithm propose the structure of how the latent variables
 work together and then you have some small parts in there that you say well
 these things I don't know I'm gonna let a neural network do these things but
 essentially you come and you impose a structure upon the world right and you
 know if you get the structure correct your model will work fine if you don't
 get the structure correct your model won't work fine but this is a bit of a
 different way of working than you know saying well I train a conv net to
 predict so we're going to propose our structure we're going to say the joint
 distribution of observed and latent variables factorizes into these two it
 factorizes into this conditional so if I have the latent variables right then
 what are the images and times the prior across the latent variables now we
 already seen this distribution it's the first one is listed here again this
 conditional distribution that's simply your decoder in the VAE framework and
 that's written here it essentially says well to produce an image I'm going to
 put T the latent variable into this neural network G right here and that
 will give me the distribution of my output image so this is your decoder in
 the VAE now the interesting part and where it differs from a regular VAE is
 right here where they say well how do our latent how does our latent space
 look well this is zooming around our latent space isn't a independent Gaussians
 it's actually this TPOT distribution this topographic product no where where
 does it I forgot what it I forgot what it's what it's called a topographic
 product of student T's model the TPOT topographic product of student T that's
 going to be our distribution and that distribution is going to encourage this
 topographically organized latent space right so we can ask how does it how does
 it do that note that the encoder isn't here yet because we've only we've
 defined we've imposed degenerative process of the data the generative
 process starts at the latent space I said if I know what the latent variables
 are I can just ask my decoder to produce an image so this distribution here tells
 us you know the latent variables are distributed like this and then there we
 go now obviously what we want is we want our encoder to produce the variables the
 latent variables but we also want what the encoder produces to follow this
 distribution right here and that's going to be the sort of difficulty right here
 because what we know what we can train with back propagation is pretty much
 Gaussians you know like we can train things where we can apply the
 reparameterization trick that's stuff we can back prop through stuff we can
 Gaussians we can sample from efficiently and so on we have closed form solution
 for the KL divergences in the objectives so essentially what we can do in these
 variational frameworks is Gaussians not topographic product of student is
 however here they show okay we can in fact construct a product of student is
 this is no this is not yet a topographic product is just a product of student is
 distribution from Gaussians and that is I take one z variable and I take a bunch
 of u variables and they're all distributed like Gaussians and I square
 the use I sum them up I average them and then I take the square root and I divide
 z by dot and this variable right here that's going to be a univariate student
 t random variable this should be kind of known if you've ever taken statistics
 or like use the t-test for anything okay and you know this is already quite
 familiar and I can extend this now to the multi-dimensional case so if t is a
 multi-dimensional student is random variable composed of independent Z's and
 use then we can construct t as a vector and that is going to be distributed
 according to a product of student t's variable and this should connect to what
 we've seen before right we said that this models organization of the latent
 space is pretty much of this form that we saw right here we have the z variable
 divided by the square root of the sum of the squared u variables and now we learn
 how we can construct the product of student t's latent space given z and u
 independent Gaussians and that is you know now it should connect for you in
 deep learning variational frameworks we can work pretty much only with Gaussian
 random variables in this model we want to work with product of student t random
 variables and here is the way how we can construct the product of student t
 random errors from Gaussian random variables so that's why here we the
 neural networks will output the Z and the u that's what they will output
 that's those are those are Gaussians or supposed to be Gaussians and then we
 transform them by dividing them and summing them up in this way to the latent
 variable that the decoder receives which is this Z hat or t I guess to this is
 what the decoder receives so we know that if the encoder as output Gaussian
 random variables the decoder will receive a product of student t random
 variable now why is the product of student t random variable special in
 any way because it enables us to what they call here introduce topography in
 essence and they formulate this a little bit what it does is it it lets it
 if if some of the use in this some and some of the you in this some are the
 same which you can see by the indices in this case they are not but if some are
 shared that means that the two were the two t variables not the two Z the two t
 so this is one t and this is another t right this is t1 this is t2 lots of t
 these two variables will no longer be independent they will actually be
 dependent on each other so this is a way how we can construct latent spaces
 where some of the variables are actually correlated or in some other way have
 have higher order correlations with each other meaning that the value of one is
 not independent from the value of the other one and that is pretty much a
 basis for what we want for constructing these topographic latent spaces so here
 they say introducing topography essentially what we're going to do is
 we're not we're going to define neighborhoods across our u variables and
 we're going to share the u variables according to these neighborhoods and
 that's going to make the in the components of t dependent on each other
 and this sounds complicated but essentially you can imagine instead of
 having like four latent random variable which are all Gaussians now we have
 simply one set of z variables and one set of u variables and we're going to
 consider an entire sequence and not just one one image right so we were going to
 consider an entire sequence of images like this right here every image
 produces one z and one u variable and then when we consider an image let's say
 this is the focus right now we consider its z and we consider a neighborhood of
 use and that's just going to amount sort of like a convolution like this is maybe
 a neighborhood of three so we're going to consider this u this u and this u so
 we're going to construct the z on top of the fraction divided by this thing
 squared this bubble here squared this bubble here squared square root of top
 on top of that and that's going to be our t so the t for this image right here
 that's going to be this whole fraction so when we train the VAE we input the
 whole sequence we focus on for example this picture we construct its t by
 looking at its z and its neighborhood of use then we put that t into the decoder
 the decoder is going to produce an image and then we can apply a loss function
 between those two okay so that is the loss that's the loss function right the
 loss function note that the loss function doesn't say you need if you
 roll ten times then it needs to be the picture that's ten times ahead that is
 not the case at all we actually don't have the role function in here but even
 now even once we introduce the role function in the in the latent space
 we're not going to explicitly train the model to predict the future we're simply
 going to construct as we did here the latent space such that it such that this
 naturally happens so how are we going to do this almost the same and here you
 have they talk about capsules so you can see that they divide this neighborhood
 structure so the W defines the neighborhood structure you can see here
 some of the use they are connected and then other ones are connected but these
 use are not connected with those they kind of talk about capsules essentially
 it's just that they make some of the variables dependent on each other and
 some not or or when they do these neighborhood things they just have two
 sets of variables like to have two sets of Z's and use and they only yeah they
 construct two T variables and that that's what they call capsules that I
 don't I don't know why the capsule terminology enters this paper
 necessarily but you know they they want to draw a connection here so temporal
 coherence now we get to how do we organize this latent space such that the
 role operation now also gets in and this is pretty simple it's actually just an
 extension of this right here so here if you consider these images here as images
 of a sequence we always said well you need to be connected to sort of your
 your neighboring variables and now sorry your neighboring you variables as they
 are right and now we're going to say the same thing but but I'm going to draw the
 critical path here again so this we have a Z variable right here we have you
 variables from the neighborhood okay and we're going to take the Z variable on
 top of the fraction and we're going to take the U variables below the fraction
 right here like so like so like so now before we do this before we take the U
 variables here below the fraction we're going to roll the U variables
 according to their distance from according to their distance from the
 focus so in this case this would be simply one rollback this would be
 simply one roll forward so in the language of this paper what this means
 is that we don't want we we don't want this image or it given a particular
 position in this image right this position right here if we simply apply
 the classic neighborhood structure we say we want this position in this image
 to be correlated with the same position a step back and a step forward now if we
 construct the role like this what we're saying is no no no no I don't want I
 want I want this position to be correlated with maybe this position here
 and this position there like slightly behind and slightly ahead but I'm
 obviously not going to tell the model what I expect I simply say please this
 image is one time stack well black this image is one time step back from me
 please roll the latent space by one and that's going to be your relevant
 variable and in this case it's please roll the latent space of this thing one
 forward and that's going to be your relevant latent variable so it's not
 that we train we we train rolling this t variable here because the t is what
 finally comes out we're not training this t to roll forward or back and then
 predict ten steps ahead we're simply saying how you are influenced you as a
 focus how you are influenced by pictures before and after you you're not simply
 taking into account their latent variables you want to take into account
 rolled versions of their latent variables in order for you to reconstruct
 yourself in the training objective and it turns out at least that's how I
 understand it right and it turns out so here you can see the whole process we're
 going to take images we're going to produce mean and variance of late of
 Gaussian variables for the Z and the u variables so if you had just a VAE it
 would just be this right here and those will be a layer you're latent variables
 but not here we produce two sets Z's and use then we're going to construct the t
 variables I don't know why this is on the bottom here but then we're going to
 construct the t variables according to this formula W here is the neighborhood
 structure you define it you and Z are the variables you produced from your
 encoder or you sampled from what your encoder produced and mu here is also a
 learnable parameter a learnable mean parameter and then we want to stick
 this these T's into you're going to stick these T's into this neural network
 now here it says Z and ZL and UL but essentially this here this here these
 create T oh here it's here you're going to stick the T into your decoder neural
 network remember the G how do we get the picture from the latent variable that's
 the decoder and stick that into the decoder and out you get an image and you
 train it with the classic elbow the evidence lower bound which says okay what
 I want is I want to reconstruct the picture accurately right that's this
 term right here to reconstruct the picture accurately but I also want that
 my Z well essentially what I want is that my T variables are distributed
 according to this TPOT distribution I want to enforce that but I can't right I
 can work with Gaussians so what but what I can do is I can say well the Z
 variables and the U variables they must be as Gaussian as possible so I penalize
 the KL divergence between what I produce which is this right here and the Gaussian
 like a a pure Gaussian this has a closed form I can I can calculate KL
 divergences from what I produce with Gaussians no problem okay and that's the
 training loss and I simply average that over the input sequence and there there
 you go now the evaluation of these things I have to say after reading
 through the experiments in the evaluations this is this is a paper kind
 of an idea at least I feel so right correct me if I'm wrong but I feel that
 this is sort of an idea paper it's like here's an idea it works if we you know
 specifically construct a data set for it and if we specifically also the
 experiments are appear to be kind of fiddly like you have to really you know
 get your parameters right to make this work but if you do then you know the
 model behaves as you as you expect and so they measure things like is the
 rolled version of the latent variables really equal to the latent variables a
 couple of time steps ahead and things like this and they produce these these
 maps so here is one where the latent space isn't a 1d torus like we looked at
 so 1d torus is this right so you go around around around sorry and this is a
 2d torus so a 2d torus is like a plane and if you leave here you come back
 here and if you leave here you come back here so if you if you roll this up and
 then you you have a pipe and if you close the pipe you have like a donut so
 that's a torus so if they have a topographic space like a torus they and
 they simply apply that to MNIST the test set sort of looks like this I don't know
 if you want to read something into this like feel free I'm not sure but in when
 they go with the sequences so here you see like the sequences I think on top is
 what they input and then this is the continuation that the model doesn't see
 on the bottom is what the model produces you can see the model does get to a
 point where it understands how these sequences go here all right it goes large
 large and then it kind of flips around to the smallest this is a expected
 behavior here as well the rotation it model continues the rotation and it
 turns out even if the model is just trained with they have these these
 experiments even if the model is just trained with single transformations so
 either a role sorry either a rotation or a scale transformation or a color change
 it can generalize to multiple transformations at once as you can see
 right here colors and rotations can the model can generalize to that fairly
 fairly well okay I don't want to get too much into the experiments because I'm
 not sure how important that the numbers here are I'm safe to say if you construct
 this model and if you apply it to the you know problems where exactly this is
 needed and if you get the hyper parameters right then this model
 actually works it's better whereas a regular neural network it could not
 easily incorporate the concept of these slow changing transitions it would sort
 of have to learn okay what color comes after red orange okay what color comes
 after orange yellow okay what color comes after yellow green I guess the
 other model has to learn that as well but this model it cannot represent the
 transition in a sequence as sort of as it has to learn it as a parameterized
 function rather than being able to map it to an internal transformation of the
 rate of the latent space like the topographic VAE can do okay that was it
 for me I'm not competent enough to tell you how big of a step this is it feels
 to me like a little step it might be a giant step I don't know okay it feels to
 me like it's kind of an idea paper to show something neat that you could do in
 an idealized case it might be that this is a much bigger deal than than I think
 I thought it was a cool paper I thought it was a neat idea it's written even
 though it's I think under you know more high love sorry more more so I'm not as
 competent at it but I could still make sense of it so if you enjoy this give
 it a read yeah let me know if you have any comments and that was it bye bye
 thanks