Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
ai deep learning variational autoencoders vae disentanglement representation learning machine learning unsupervised arxiv google google ai mpi eth eth zurich ethz
https://arxiv.org/abs/1811.12359

Abstract:
In recent years, the interest in unsupervised learning of disentangled representations has significantly increased. The key assumption is that real-world data is generated by a few explanatory factors of variation and that these factors can be recovered by unsupervised learning algorithms. A large number of unsupervised learning approaches based on auto-encoding and quantitative evaluation metrics of disentanglement have been proposed; yet, the efficacy of the proposed approaches and utility of proposed notions of disentanglement has not been challenged in prior work. In this paper, we provide a sober look on recent progress in the field and challenge some common assumptions. 
We first theoretically show that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data. Then, we train more than 12000 models covering the six most prominent methods, and evaluate them across six disentanglement metrics in a reproducible large-scale experimental study on seven different data sets. On the positive side, we observe that different methods successfully enforce properties "encouraged" by the corresponding losses. On the negative side, we observe in our study that well-disentangled models seemingly cannot be identified without access to ground-truth labels even if we are allowed to transfer hyperparameters across data sets. Furthermore, increased disentanglement does not seem to lead to a decreased sample complexity of learning for downstream tasks. 
These results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision, investigate concrete benefits of enforcing disentanglement of the learned representations, and consider a reproducible experimental setup covering several data sets.

Authors:
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem
 All right, hello everyone Today we're gonna look at this paper challenging common assumptions in the unsupervised learning of disentangled representations by Francesca Locutello and a bunch of other people at Google AI, ETH Zurich and MPI Full disclaimer, I know these people and I've Talked to them about this work. So just so you know where I'm coming from It's a good paper and it's fairly short to explain. So let's go over it The main thing here is What's called disentanglement? So disentanglement is kind of a property of data in unsupervised learning or not data of your model that you would like to have In unsupervised learning in here, especially in generative models so What they focus on is like Auto encoding here and What that means is I have some data point which could be an image. Let's draw an image here and I compress this usually into a vector and The vector has a couple of dimensions. This is a representation of the Data and from this representation what I can do is I can produce an image again and If I train an autoencoder, I will enforce that my model. So both of these are my model This is called an encoder and this is called a decoder That What they do is that the final image then looks like the original image This is an autoencoder basically a compression algorithm that Tries to find representations such that it can reconstruct the original image again Here we go a little further in that we use what's called variational autoencoders. So All of these all of these experiments here use variants of the variational autoencoder and What a variational autoencoder? Let's skip some here A variational autoencoder is the same thing as an autoencoder except It's a probabilistic framework, so What you do is here? On the bottom you can see an equation that basically is the objective for a VAE and What it does is it says okay, I have an image Let's say this is my image and I use an encoder like in an autoencoder And that gives me an image And that gives me an autoencoder and that gives me a representation Okay but Now I don't use this representation directly to decode but this representation Is simply the parameters from a bunch of distributions Right, so here let's say I have Four four I want four latent factors and the latent factors are basically the latent variables that describe This image so the images could be images of let's say cats and four latent factors could be The color of the fur of the cat the size of the cat the position in the image and the let's say the General lighting of how bright the image is so these could be four latent factors that would explain Best the the image and from that and if the image could be best reconstructed, let's say So the the four latent factors we consider as probability distributions so What our encoder needs to do our encoder needs to produce eight numbers in this case Eight numbers why because for each of these four distributions we want a mean? And a standard deviation So these eight numbers here each one Or each pair of numbers one of them is going to be the mean and the other one is going to be the standard deviation of a distribution and then From these we're going to construct a distribution Like so like okay. Here's the mean here's the standard deviation So the distribution somehow looks like this and then we're going to sample from this distribution. So one sample could be Here one sample could be here one sample could be here here So of course in the middle here, we're going to have more samples. But so the whereas the autoencoder directly uses the encoding to reproduce the image the variational autoencoder the the What the output what the encoder produces here is simply a parameterization for a disk for a distribution and And that distribution then is sampled so we're going to take one sample here So from from each of these so there's going to be multiple of those distributions because we have Eight numbers we are going to produce four distributions in particular So we're going to sample four different numbers. So we're going to sample a new vector with four One two, three four. Well, I didn't have eight at the beginning, but never mind. So here This gives us four numbers, but these are sampled. So these are going to be different every time Even if we feed the same image and from this the decoder Is going to try to reproduce the image and then Again the images the end image and the beginning image are going to be forced to be close to each other But also now since this is a probabilistic framework we also kind of need We need a different loss function for the autoencoder. You can simply penalize how far the images are in let's say l2 norm but here We have two distinct parts to the loss term. So And everything is probabilistic. So let's walk through this here. The first part Of the so we have two parts of the loss term and Here in particular q Is you can see here it takes as an is it is the distribution of z Conditional x and z will always be related representation of the Of the data and x will be the the data itself the data point So q will take the data point and produce z And the z specifically here what's meant is this This thing here This is z Whereas This is this is x And this is also Well, this is x Tilde or something whatever is produced by the decoder So basically what we're gonna do is We're going to punish the kl distance, which is a probabilistic distance measure. We're gonna Measure the distance between the distribution of z under x With the prior over z so p of z here This here Is the prior distribution Prior distribution over z and the prior distribution in va is is often to be taken as a Gaussian so We'll say all right So the our our kind of default assumption on the z variables is that they're that they're gaussians here And We're gonna force basically we're gonna force the encoder to come up with With encodings generally over the data set that are gaussians that are conformal to our prior So here we say specific prior pz I didn't mean to cross that out Right, so this second term enforces the the encoder to produce things that are Gaussian Um, it's specifically with our if our prior is let's say um Zero zero mean unit variance gaussians. It's gonna enforce that the first term here Is different the first term makes the image that has been input to the variational encoder and the image that has been output Close together again. This is a probabilistic Loss so What we're gonna do here is we're gonna take expectations. So the KL distance is also an expectation by the way We're gonna take expectations over Px which is the distribution of the data and also Over Q and Q is again our encoding mechanism Mechanism and we're simply going to punish the Or we're gonna here maximize the the log Probability which is equivalent to minimizing the negative log likelihood Which you might be familiar with of the data given the the z variables so And this is an expectation over q given x so what that means is basically we want the the probability Of this original data point we want Here we output x tilde We We want this to be close to x here. So what we can say is we want the probability that our model outputs x Which has been the original input right given this particular z that it produced to be high As an expectation of q Of z given x So as a bit cryptic, but it means here I input x into q I get out z and when I Have the z what I produce here is what I produce The likelihood that x the original image these are the same is produced should be high So that's a variational autoencoder. I simply encourage the latent representations to be close to my prior which is often Gaussian and I Encourage the output to be similar to the input which I do by Encouraging the likelihood that the output is the input All right, so cool. So what's that have to do with disentanglement disentanglement is property That now I would like to have in my model which is that these These things here Um, or we can also focus on these things here, however, you want to view it or these things here these Latent things that my encoder outputs somehow give me information about the data in a way That's disentangled what that means is I've already I've made an example that's already disentangled where I said, let's let's say we have images of a cat of cats and the fur color is going to be one variable and the color of the eyes of the cat is going to be another one and The position in the image is going to be another one. So these are all fairly independent, right? and so I if I change some Latent factor I can change them pretty much independently. So here this could be the fur color I can change it pretty much independently and cat will just have a different fur and so on What would be non disentangled representations? would be Let's say one encodes the fur of the cat and the other one encodes the Encodes the the species of cat because these are these are highly let's say entangled so the fur color is highly dependent on what species the cat is and It's not really so they kind of you can you can imagine it as these things being correlated, but it's slightly different And there are there's not an agreement on what this entanglement means really we just kind of imagine data is somehow Entangled and we want to kind of pull out these disentangled factors So what they focus on here and the easiest the easiest measure here is the following um, I might want to have some Space All right. So the easiest measure of disentanglement that is come up with here is the following Um, it's an assumption. The assumption is let's say there's data x right We'll call it random variable and we know We know we assume that This data is generated by a bunch of Latent variables z1 z2 z3 Which are? Independent which means that and the technical In this is that the p of z which is all of them can be factorized into p of z i So they are independent Um and these Kind of determine independently the data x now What does that disentanglement of when my model has produced a disentangled representation means I now have a model some model m Which is going to give me a representation of x And the representation as we saw before um Could be these things here, that's the the Representation specifically what these people do is they say okay the mean of the distribution that my encoder gives me That's the representation of x All right, so this gives you a representation of x from which you then might want to you know reconstruct x over here x So then but so the important thing is when is the representation disentangled the representation is disentangled in the easiest sense If the following holds when I change um When I change z i So I introduce a delta to z i to any of these three that means That in the representation of x Which we're just going to say So if there's three dimensions of z we just assume kind of we know that and we also make the representation three-dimensional then exactly one Factor in this is going to change so if I change one factor of the true underlying distribution um Which is independently which all the latent factors are independent then Only one factor in my representation changes. So if that's the case then Kind of I can be fairly sure that i've captured the the true latent structure of the data, right if one if if one of the Of the if I change one of the the z here Let's say I change the z3 and only then uh r3 So I change z3 let's say I have access to the true underlying distribution I ask the the world Ask the world to give me a picture of a cat that where the fur color is different and then I put it I get a data point and then I put it through my model I get a representation and only From the cat that I had before only one of the factors of my representation changes Then I call it disentangled then I can be fairly sure. Okay my representation this dimension of my representation captures the fur color independently of the other factors All right, so that's disentanglement and you notice it requires actually access here to the true distribution Distribution of how the data is generated by the world So this is something you generally don't have but um, it's a technical notion So you can you can certainly postulate it And it's it It's a nice framework and this paper basically proves that Generally learning disentangled representation in that way is impossible um If you don't have some if you don't make some assumptions some a priori assumptions on your data and your model so This is a theorem here and we See here p is any generative model Which admits this factorization Right does that that's what we talked about the true underlying generative process is Is independent in so In its constituents That means there's a bunch of latent variables. They independently from each other produce a data point right X is the data observations Then there exists an infinite family of bijective functions right such that This and this and this and this Okay What that means? is so this thing here basically just means that the um the distributions agree so that the the the overall distributions the let's say the it's not exactly that but the posterior distributions Um, let's say the data looks the same right That what comes out of the process looks the same So there is there is functions that transform the latent distribution into some other distribution, but they look the same in cumulatively All right, and then we have the All right, and then this part here Means you'll see the derivative of fi of u with respect to some Uj which you'll notice i and j are different. Um, this this means that basically the dimensions are Entangled it means that if I take the derivative of one entry In the in the f in the function output and I derive it By another entry then I get a non-zero derivative which means that this Uj influences fi Which basically means that I can produce I can take the z I can transform it in In so z is independent. So it means the i-th dimension has no influence on the j-th dimension Of the of the output and I can transform it into something Where that's no longer the case where the i-th and the j-th dimension very much uh Kind of are entangled or covariate so This means I can take the z that That's kind of everything is independent. I can transform it into something where everything is dependent and they give a nice example here So they say let's say we have Gaussians In two dimensions, so we have one Gaussian here And let me see if I can draw this one Gaussian here Right in two dimensions. They're completely independent um what you'll find is that the kind of distribution overall has Iso lines like this Right, it gives you kind of a hump in the middle two-dimensionally. You can maybe imagine like a bit of a mountain in the middle um All right. So this is what you this is the kind of output distribution If you if you don't know about the underlying factors, you simply see the cumulative distribution, which would be the the big p here um All right. Now we transform this into with f And f is simply a rotation by 45 degrees right, so two new axes this and that and again Our two gaussians are going to be transformed these Right. So these are not these are not disentangled anymore. Well in the in the notion I can't say it like this, but this is easiest to say so these are these are kind of Now that it's rotated in terms of the original coordinate system, which would go like this These very much depend on each other right the jth dimension the if dimension depend on each other because if I sample from one of the gaussians I need now basically two coordinates to describe where it is or Yeah, one isn't just So if I sample from one Gaussian I need both the coordinates but the cumulative distribution or the That is still the same That is still going to look exactly the same so It's again a hump. So it's basically an isometric hump in every direction if I rotate that the It looks exactly the same. This is the p here But now the the if dimension and the jth dimension very much influence each other um, and yeah, interestingly the If you now look at disentanglement if I just have if if I now produce data x here x1 and here I produce data x2 and both go through my model and give me our representation of x1 and the representation of x1 and the representation of x2 I have Without seeing the underlying structure. I have no idea which one of those two It comes from and thereby I have zero chance basically. It's a luck lucky guess um Which one it comes from and there's an infinite family. So I will never find the true underlying distribution here and thereby I will never um I will never be able to satisfy this property that if one of the z changes Then only one of the factors of my representation will change because if I Say, oh, well, obviously this is the case Then i'm going to make a different model and if I say well, this is the case I'm going to make a different model. I don't know which one it is So I have to choose one and it could be the other one. So i'm bound to be wrong in this case 50% of the time, but if it's an infinite family i'm bound to be wrong every time basically, so That's what the theorem basically says I can't Decide on the true underlying distribution. Um, there's an infinite family that Transforms it into it. It transforms every distribution into some other distribution that has basically complete opposite properties of entanglement And I need to choose one and I will never choose the right one because i'm not that lucky And thereby I can't do representation learning that's disentangled All right, so that's the main claim of the paper and um There is a lot of experiments here so what the paper also does is they produce some new Data sets and they test a lot of a lot of architectures basically they say just because it's theoretically impossible It's not impractical because we can actually make these underlying assumptions like we can make some assumptions on the data and then and then we kind of can attempt to do disentanglement learning so they do these data sets and they test different VAE's architectures on it and they basically Um establish where More work should go. So that's that's kind of the rest of the paper I encourage you to look at the rest of the paper I just wanted to give a quick introduction to VAEs and to disentanglement to entangle representation learning I Wasn't technically correct Uh in every detail, but I hope that it's enough and have fun
 All right, hello everyone
 Today we're gonna look at this paper challenging common assumptions in the unsupervised learning of disentangled representations by Francesca Locutello and
 a bunch of other people at Google AI, ETH Zurich and MPI
 Full disclaimer, I know these people and I've
 Talked to them about this work. So just so you know where I'm coming from
 It's a good paper and it's fairly short to explain. So let's go over it
 The main thing here is
 What's called disentanglement? So disentanglement is kind of a property of data in
 unsupervised learning or not data of your model that you would like to have
 In unsupervised learning in here, especially in generative models
 so
 What they focus on is like
 Auto encoding here and
 What that means is I have some data point which could be an image. Let's draw an image here and
 I
 compress this usually into a vector and
 The vector has a couple of dimensions. This is a representation of the
 Data and from this representation what I can do is I can produce an image again and
 If I train an autoencoder, I will enforce that my model. So both of these are my model
 This is called an encoder and this is called a decoder
 That
 What they do is that the final image then looks like the original image
 This is an autoencoder basically a compression algorithm that
 Tries to find representations such that it can reconstruct the original image again
 Here we go a little further in that we use what's called variational autoencoders. So
 All of these all of these experiments here use
 variants of the variational autoencoder and
 What a variational autoencoder?
 Let's skip some here
 A variational autoencoder is the same thing as an autoencoder except
 It's a probabilistic framework, so
 What you do is here?
 On the bottom you can see an equation that basically is the objective for a VAE and
 What it does is it says okay, I have an image
 Let's say this is my image and
 I use an encoder like in an autoencoder
 And that gives me an image
 And that gives me an autoencoder and that gives me a representation
 Okay
 but
 Now I don't use this representation directly to decode but this representation
 Is simply the parameters from a bunch of distributions
 Right, so here let's say I have
 Four four I want four latent factors and the latent factors are basically the latent variables that describe
 This image so the images could be images of let's say cats and four latent factors could be
 The color of the fur of the cat the size of the cat the position in the image and
 the
 let's say the
 General lighting of how bright the image is so these could be four latent factors that would
 explain
 Best the the image and from that and if the image could be best reconstructed, let's say
 So the the four latent factors we consider as probability distributions
 so
 What our encoder needs to do our encoder needs to produce eight numbers in this case
 Eight numbers why because for each of these four distributions we want a mean?
 And a standard deviation
 So these eight numbers here
 each one
 Or each pair of numbers one of them is going to be the mean and the other one is going to be the standard deviation
 of a distribution
 and then
 From these we're going to construct a distribution
 Like so like okay. Here's the mean here's the standard deviation
 So the distribution somehow looks like this and then we're going to sample from this distribution. So one sample could be
 Here one sample could be here one sample could be here here
 So of course in the middle here, we're going to have more samples. But so the whereas the autoencoder directly uses the encoding
 to reproduce the image the variational autoencoder the
 the
 What the output what the encoder produces here is simply a parameterization
 for a disk for a distribution and
 And that distribution then is sampled so we're going to take one sample
 here
 So from from each of these so there's going to be multiple of those distributions because we have
 Eight numbers we are going to produce four distributions
 in particular
 So we're going to sample four different numbers. So we're going to sample a new vector
 with four
 One two, three four. Well, I didn't have eight at the beginning, but never mind. So here
 This gives us four numbers, but these are sampled. So these are going to be different every time
 Even if we feed the same image
 and from this the decoder
 Is going to try to reproduce the image and then
 Again the images the end image and the beginning image are going to be forced to be close to each other
 But also now since this is a probabilistic framework we also kind of need
 We need a different loss function for the autoencoder. You can simply penalize how far the images are in let's say l2 norm
 but here
 We have two distinct parts to the loss term. So
 And everything is probabilistic. So let's walk through this here. The first part
 Of the so we have two parts of the loss term and
 Here in particular q
 Is you can see here it takes as an is it is the distribution of z
 Conditional x and z will always be related representation of the
 Of the data and x will be the the data itself the data point
 So q will take the data point and produce
 z
 And the z specifically here what's meant is
 this
 This thing here
 This is z
 Whereas
 This is this is x
 And this is also
 Well, this is x
 Tilde or something whatever is produced by the decoder
 So basically what we're gonna do is
 We're going to punish the kl distance, which is a probabilistic distance measure. We're gonna
 Measure the distance between the distribution of z under x
 With the prior over z so p of z here
 This here
 Is the prior distribution
 Prior distribution over z and the prior distribution in va is is often to be taken as a
 Gaussian so
 We'll say all right
 So the our our kind of default assumption on the z variables is that they're that they're gaussians here
 And
 We're gonna force basically we're gonna force the encoder to come up with
 With encodings generally over the data set that are gaussians that are conformal to our prior
 So here we say specific prior pz I didn't mean to cross that out
 Right, so this second term enforces the the encoder to produce things that are
 Gaussian
 Um, it's specifically with our if our prior is let's say
 um
 Zero zero mean unit variance gaussians. It's gonna enforce that the first term here
 Is different the first term makes the image that has been input to the variational encoder and the image that has been output
 Close together again. This is a probabilistic
 Loss so
 What we're gonna do here is we're gonna take expectations. So the KL distance is also an expectation by the way
 We're gonna take expectations over
 Px which is the distribution of the data and also
 Over Q and Q is again our encoding
 mechanism
 Mechanism and we're simply going to punish the
 Or we're gonna here maximize the the log
 Probability which is equivalent to minimizing the negative log likelihood
 Which you might be familiar with of the data given the the z variables
 so
 And this is an expectation over q
 given x so what that means is basically we want the
 the
 probability
 Of this original data point we want
 Here we output x tilde
 We
 We want this to be close to x here. So what we can say is we want the probability
 that our model
 outputs x
 Which has been the original input right given this particular z that it produced to be high
 As an expectation of q
 Of z given x
 So as a bit cryptic, but it means here I input x into q I get out z
 and when I
 Have the z what I produce here is what I produce
 The likelihood that x the original image these are the same is produced should be high
 So that's a variational autoencoder. I simply encourage the latent representations to be
 close to my prior which is often Gaussian and I
 Encourage the output to be similar to the input which I do by
 Encouraging the likelihood that the output is the input
 All right, so cool. So what's that have to do with disentanglement disentanglement is property
 That now I would like to have in my model which is that
 these
 These things here
 Um, or we can also focus on these things here, however, you want to view it or these things here
 these
 Latent things that my encoder outputs somehow give me information about the data in a way
 That's disentangled what that means is I've already I've made an example that's already disentangled
 where I said, let's let's say we have images of a cat of cats and
 the fur color is going to be one variable and the color of the eyes of the cat is going to be another one and
 The position in the image is going to be another one. So these are all fairly independent, right?
 and so I
 if I change some
 Latent factor I can change them pretty much independently. So here this could be the fur color
 I can change it pretty much independently and cat will just have a different fur and so on
 What would be non disentangled representations?
 would be
 Let's say one encodes the fur of the cat
 and the other one encodes the
 Encodes the the species of cat because these are these are highly let's say entangled
 so the fur color is highly dependent on what species the cat is and
 It's not really so they kind of you can you can imagine it as these things being correlated, but it's slightly different
 And there are there's not an agreement on what this entanglement means really we just kind of imagine data is somehow
 Entangled and we want to kind of pull out these disentangled factors
 So what they focus on here and the easiest the easiest measure here
 is
 the following um, I might want to have some
 Space
 All right. So the easiest measure of disentanglement that is come up with here is the following
 Um, it's an assumption. The assumption is let's say there's data x
 right
 We'll call it random variable and we know
 We know we assume
 that
 This data is generated
 by a bunch of
 Latent variables z1 z2 z3
 Which are?
 Independent which means that and the technical
 In this is that the p of z which is all of them can be factorized
 into p of z i
 So they are independent
 Um and these
 Kind of determine independently
 the data x
 now
 What does that disentanglement of when my model has produced a disentangled representation means I now have a model some model
 m
 Which is going to give me a representation of x
 And the representation as we saw before
 um
 Could be
 these things here, that's the
 the
 Representation specifically what these people do is they say okay the mean of the distribution that my encoder gives me
 That's the representation of x
 All right, so this gives you a representation of x from which you then might want to you know reconstruct x
 over here
 x
 So then but so the important thing is when is the representation disentangled the representation is disentangled in the easiest sense
 If the following holds when I change
 um
 When I change z i
 So I introduce a delta to z i to any of these three that means
 That in the representation of x
 Which we're just going to say
 So if there's three dimensions of z we just assume kind of we know that and we also make the representation three-dimensional
 then
 exactly one
 Factor in this is going to change so if I change one
 factor of the true underlying distribution
 um
 Which is independently which all the latent factors are independent then
 Only one factor in my representation changes. So if that's the case then
 Kind of I can be fairly sure that i've captured the the true latent structure of the data, right if one if if one of the
 Of the if I change one of the the z here
 Let's say I change the z3
 and only then uh
 r3
 So I change z3 let's say I have access to the true underlying distribution I ask the the world
 Ask the world to give me a picture of a cat that where the fur color is different and then I put it
 I get a data point
 and then I put it through my model I get a representation and
 only
 From the cat that I had before only one of the factors of my representation changes
 Then I call it disentangled then I can be fairly sure. Okay my representation this dimension of my representation captures the fur color
 independently of the other factors
 All right, so that's disentanglement and you notice it requires actually access here to the true
 distribution
 Distribution of how the data is generated by the world
 So this is something you generally don't have but um, it's a technical notion
 So you can you can certainly postulate it
 And it's it
 It's a nice framework and this paper basically proves that
 Generally learning disentangled representation in that way is impossible
 um
 If you don't have some if you don't make some assumptions some a priori assumptions on your data and your model
 so
 This is a theorem here
 and we
 See here p is any generative model
 Which admits this factorization
 Right does that that's what we talked about the true underlying generative process is
 Is independent in so
 In its constituents
 That means there's a bunch of latent variables. They independently from each other produce a data point
 right
 X is the data observations
 Then there exists an infinite family
 of bijective functions
 right such that
 This and this and this and this
 Okay
 What that means?
 is so this thing here
 basically just means that the
 um the distributions agree so that the the the overall distributions the let's say the
 it's not exactly that but the
 posterior distributions
 Um, let's say the data looks the same right
 That what comes out of the process looks the same
 So there is there is functions that transform
 the
 latent distribution into some other distribution, but they
 look the same in
 cumulatively
 All right, and then we have the
 All right, and then this part here
 Means you'll see the derivative of fi of u with respect to
 some
 Uj which you'll notice i and j are different. Um, this
 this means
 that
 basically the dimensions
 are
 Entangled it means that if I
 take the derivative of
 one entry
 In the in the f in the function
 output and I derive it
 By another entry then I get a non-zero derivative which means that this
 Uj
 influences fi
 Which basically means that I can produce I can take the z I can transform it in
 In so z is independent. So it means the i-th dimension has no influence on the j-th dimension
 Of the of the output and I can transform it into something
 Where that's no longer the case where the i-th and the j-th dimension very much
 uh
 Kind of are
 entangled or covariate
 so
 This means I can take the z that
 That's kind of everything is independent. I can transform it into something where everything is dependent and they give a nice example here
 So they say let's say we have
 Gaussians
 In two dimensions, so we have one Gaussian here
 And let me see if I can draw this one Gaussian here
 Right in two dimensions. They're completely independent
 um what you'll find is that the kind of
 distribution overall has
 Iso lines like this
 Right, it gives you kind of a hump in the middle two-dimensionally. You can maybe imagine like a bit of a mountain in the middle
 um
 All right. So this is what you this is the kind of output distribution
 If you if you don't know about the underlying factors, you simply see the cumulative distribution, which would be the the big p here
 um
 All right. Now we transform this into with f
 And f is simply a rotation
 by 45 degrees
 right, so two new axes this
 and that and again
 Our two gaussians are going to be transformed these
 Right. So these are not these are not disentangled anymore. Well in the in the notion
 I can't say it like this, but this is easiest to say so these are these are kind of
 Now that it's rotated in terms of the original coordinate system, which would go like this
 These very much depend on each other right the jth dimension the if dimension depend on each other because if I sample from one of the gaussians
 I need now basically two coordinates to describe
 where it is or
 Yeah, one isn't just
 So if I sample from one Gaussian I need both the coordinates
 but the cumulative distribution or the
 That is still the same
 That is still going to look exactly the same
 so
 It's again a hump. So it's basically an isometric hump in every direction if I rotate that the
 It looks exactly the same. This is the p here
 But now the the if dimension and the jth dimension very much influence each other
 um, and yeah, interestingly the
 If you now look at disentanglement if I just have if if I now produce
 data
 x
 here
 x1 and here I produce data
 x2
 and both
 go through my model
 and give me our representation
 of x1
 and the representation
 of x1 and the representation of x2
 I have
 Without seeing the underlying structure. I have no idea which one of those two
 It comes from and thereby I have zero chance basically. It's a luck lucky guess
 um
 Which one it comes from and there's an infinite family. So I will never find the true underlying
 distribution here and thereby I will never
 um
 I will never be able to satisfy this property that if one of the z changes
 Then only one of the factors of my representation will change because if I
 Say, oh, well, obviously this is the case
 Then i'm going to make a different model and if I say well, this is the case
 I'm going to make a different model. I don't know which one it is
 So I have to choose one and it could be the other one. So i'm bound to be wrong in this case
 50% of the time, but if it's an infinite family i'm bound to be wrong
 every time
 basically, so
 That's what the theorem basically says I can't
 Decide on the true underlying distribution. Um, there's an infinite family that
 Transforms it into it. It transforms every distribution
 into some other distribution that has basically complete opposite properties of entanglement
 And I need to choose one and I will never choose the right one because i'm not that lucky
 And thereby I can't do representation learning that's disentangled
 All right, so that's the main claim of the paper and um
 There is a lot of experiments here
 so what the paper also does is they produce some new
 Data sets and they test a lot of a lot of architectures basically they say just because it's theoretically impossible
 It's not impractical because we can actually make these underlying assumptions
 like we can make some assumptions on the data and then and then
 we kind of
 can
 attempt to do disentanglement learning so they do these
 data sets and they test different VAE's architectures on it and they basically
 Um establish where
 More work should go. So that's that's kind of the rest of the paper
 I encourage you to look at the rest of the paper
 I just wanted to give a quick introduction to VAEs and to disentanglement
 to entangle representation learning
 I
 Wasn't technically correct
 Uh in every detail, but I hope that it's enough and have fun