Set Distribution Networks: a Generative Model for Sets of Images (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper sets images cnn convolutional neural network gan generator encoder discriminator prior mean made latent binary conditional noise distribution probability energy-based energy apple research sdn variational elbo
We've become very good at making generative models for images and classes of images, but not yet of sets of images, especially when the number of sets is unknown and can contain sets that have never been encountered during training. This paper builds a probabilistic framework and a practical implementation of a generative model for sets of images based on variational methods.

OUTLINE:
0:00 - Intro & Overview
1:25 - Problem Statement
8:05 - Architecture Overview
20:05 - Probabilistic Model
33:50 - Likelihood Function
40:30 - Model Architectures
44:20 - Loss Function & Optimization
47:30 - Results
58:45 - Conclusion

Paper: https://arxiv.org/abs/2006.10705

Abstract:
Images with shared characteristics naturally form sets. For example, in a face verification benchmark, images of the same identity form sets. For generative models, the standard way of dealing with sets is to represent each as a one hot vector, and learn a conditional generative model p(x|y). This representation assumes that the number of sets is limited and known, such that the distribution over sets reduces to a simple multinomial distribution. In contrast, we study a more generic problem where the number of sets is large and unknown. We introduce Set Distribution Networks (SDNs), a novel framework that learns to autoencode and freely generate sets. We achieve this by jointly learning a set encoder, set discriminator, set generator, and set prior. We show that SDNs are able to reconstruct image sets that preserve salient attributes of the inputs in our benchmark datasets, and are also able to generate novel objects/identities. We examine the sets generated by SDN with a pre-trained 3D reconstruction network and a face verification network, respectively, as a novel way to evaluate the quality of generated sets of images.

Authors: Shuangfei Zhai, Walter Talbott, Miguel Angel Bautista, Carlos Guestrin, Josh M. Susskind

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we're looking at set distribution networks, a generative model for sets of images by Xuangfei Chai, Walter Tablet, Miguel Angel Bautista, Carlos Gestrin and Josh M. Suskind of Apple. So this paper introduces a generative model for sets and it does so in an energy-based model fashion. It will have an encoder, a decoder in form of a generator, it will have a discriminator and it will have all kinds of math but the end result is a model that can generate sets of images and by sets we mean it can generate different kind of views on the same identity of image and you'll see what that means and it can generate even sets that it has never seen before which makes it different from a class conditional GAN or something like this. So I can't really describe it on a high level in a very concise fashion, you'll just have to stick around and see what's going on right here. So if you like content like this feel also free to share it out and leave it a like, tell me in the comments what you like. This is going to be a fairly math heavy paper and I'll try my best to kind of distill it down to what's happening because ultimately it's not that difficult. Alright so if you have a look at these samples right here these are examples of sets of images. Now without actually caring for top and bottom row they will have some meaning right here. Top row is always a row from the actual data set and the bottom row is the reconstruction of that set. Now you'll see that the images don't really have a correspondence so you see it's the same truck in the top and the bottom row but the orientation here isn't really shared or anything and that's because as we said this is a set network. So what you want to do in this problem setting is you want to take, you want to build a model that can take this set right here from the data set and it can encode it into a latent description that we call z. z simply describes the set as a whole. So z here would be, sorry, would be truck right? It would sort of be the 3D model so not the class truck but the 3D information of the truck without having any information of the different views okay? And then you want to build another model that can generate from this low level representation of the set, can generate the different views okay? Like each one of these by sort of rotating it. So we want to build a model that understands just from the pixels that here we have sets of things that they somehow always share a commonality and in this case they always share their 3D structure right? What they don't share is the view where they rendered from. So our model is supposed to kind of parse the two apart and encode that 3D structure just from the pixels in this z variable and then encode the fact that you can rotate it and look at it from different views into the generative model that then produces the different views okay? And the reason why there is no correspondence between the views is we simply regard these things as sets. So our final objective is simply going to be that the set on top is different views of that particular truck is going to be very similar to the set on the bottom which is also different views of that particular truck and that's what our model is supposed to do. Now you might know something like this from where you can simply say well this this looks like a class conditional GAN right? I simply have the class truck and I you know I feed this to my generator and my discriminator and my encoder and so on and it will produce the same truck and here that and here the bench and so on. The problem I guess becomes more apparent when you go to this different data set. So this is a face data set and again top row being the input and bottom row being the output. Now what is here what is supposed to be preserved you can kind of see in the images is sort of the identity of the person on the photo. Now you have to kind of gloss around your human bias here. You as a human can tell extremely tiny differences between faces and therefore none of the actual identities are going to be preserved right? So this on the top I believe is Ali and on on the bottom here it's not not Ali. So but you'll have to sort of gloss around this and you'll see that what is preserved or what is supposed to be preserved is something like the rough identity of the person on the picture and also a little bit of the image compositions. So you see here in the background you often have sort of these sports backgrounds right where there's kind of a washed out stadium or whatnot and you can see that this is also preserved here. I think it's kind of a glamour glamour shots of this must be some sort of glamour model and you'll see that this as well is preserved. What is different within each set is of course the different views on the same identity right? So you have the same person and you have pictures of them in different of different views, different lighting, different hairstyles and so on. Here you even see you have the black and white image and the set that is produced also contains some black and white images. So this model this is already the trained model here is doing a fairly good job. Here you can see that almost all the pictures have some sort of double like two people with one being sort of half in frame and it leads to a fairly strange thing. I'm gonna guess the model hasn't seen lots of that during training. I like it, particularly like this. This is pretty good. Also here we know that a lot of these face datasets they don't really have bald people all too often so you can see here this results in sort of kind of a weird Richard Branson type picture. In any case it does a fairly good job as you can see right here. And what's the problem with these faces? Can't we just do a class conditional again where we basically say here this is Ali and that's one class in our latent vector and then and so on. What we want to do is we want to train something like this where we can give in a new identity that we haven't seen during training. In fact we want to train something where we don't even know how many sets there are going to be in the end. We simply want to train it in a way where we say look I'm going to give you a set and the set will have images of the same person and you're going to sort of reconstruct that in a way where you output a set of images of that person conserving the identity of the person and the rough style of the picture. So this is really different from a class conditional again as in we don't know how many classes there are and there can be new unseen ones during testing. So how do we go about something like this? And here is where the thing starts. So we're going to dive into a bit of the math here and then into a bit of the reasoning but ultimately what they're going to do is they're going to build three things. They're going to build an encoder, a discriminator, and a generator. So what does the encoder do? And remember our task here is going to be the way we train it is going to be by reconstruction, at least one way we train it. So the encoder is going to take this set of images that we give it, for example here different views of this car, and is going to produce this representation, this set representation. Now what should a property be of the set representation? For example it should be independent of the ordering of these inputs. A set is simply a collection of objects, it's independent of the ordering, and it's also independent of the size in some way. Of course a bigger set gives you more information but the set identity, the fact that this is this particular car, is independent of how many views you have. So what they do is they build and they put each image through an encoder which is a convolutional neural network and I guess that gives them a hidden representation and encoding and embedding for each image and then they have this operation called pool and binarize. And so this does two things, namely first of all it pools, it simply averages these things. So it goes 1 over n C of XI, or C of XI I guess. So this simply averages these encodings right here and this already fulfills our property. So this average is independent of the order of the images and also I can add more and I can add less in expectation the average will result in the same thing. So this here now we have basically lost the information of the exact ordering and so on. This is simply an average of these images. It's a good representation for a set. This is now if we give enough images this will be independent of the particular rendering position and it will only depend on the fact that this is that particular car, if it's trained well of course. And then the second thing is binarize. And here you have to understand what exactly this set latent representation is. How do we encode a set in latent space? And as far as I understand it, what they do is they do the following. Since they don't know how many sets there are, they can't simply do the classic one-hot vector. So what you would do in a class conditional GAN is you would say I have a vector and maybe I have ten classes. So I'll make ten entries right here. Is that ten? I don't know. So if I have C classes I'll make C entries right and I'll put a zero in all of them and a one in where a one where my class is or something like this. So this would be a valid encoding for a class conditional GAN to represent the identity of the class. Here however no we don't know. And also we can't really make this continuous because other if you make this continuous you wouldn't really encode the identity of the set. You would encode more of a continuous latent space and then that becomes kind of different when you have new sets and so on. So what they really want to do is they want to make this representation here be a description of the set itself but not a one-hot. So what do we do? What they do is they do the same thing. They have a vector but not of size C because they don't know C but of some dimensionality D. This can be 10, this can be 4, this can be you know whatever. Just let's say it's 10 again but we don't know how many classes there are. What the model can do is it can encode each class as a binary vector, a binary combination of negative ones and ones. So it can put like a negative one here a one negative one negative one one one negative one and so on. So what does that give you? Now you can encode much more than ten classes. In fact you can with this you can encode two to the ten classes. So it's not that they can encode an unlimited set of set identities but they can encode in this manner they can encode a lot. They can encode this many sets using a representation like this. So this binarize operation it will take this output right here and basically clamp it to either one or negative one. So the set will be encoded by a binary vector like this and then the generator and the discriminator take that information. We'll go over what this architectural choice means but right now this you know see that this is a way to encode a large number of set identities in a low dimensional vector. So there are two things now. There are the discriminator and the generator. So first of all the generator is pretty easy. The generators task is to take this Z that we just saw which is the set identity and to generate different instances of that set. For that it needs this noise here. So if you know a generator from a from a GAN it always kind of needs input noise in order to produce different outputs and that's this thing on the right here. This Z' they simply come from some sort of latent distribution. I think they call this P of psi which is some like a uniform or a Gaussian or something. You just sample some noise right and you combine it with this thing right here which is the set identity. You concatenate it and then each one of these different Z will produce one different view right here. So the generators task is simply take the set identity and combine each with some noise and produce some views of that set. Now the discriminators task right here is going to be to decide it's going to get a set of views of a set of pictures and it's going to have to decide is this set coming from the generator or is this set coming from the data set. Now you can't simply compare images to each other like you would do in a regular GAN because they don't correspond to each other right but what should correspond to each other is this identity this Z identity. So the discriminator is going to take also two inputs. It's going to take this set right here or you know from the data set and it's going to take this right here this Z. Now this Z is going to be the set identity and that you get from the encoder right it's the same as you get right here you get it from the encoder which set you should produce and the same goes here. So the discriminator knows the set discriminator knows for example I'm trying to produce that particular car and it gets a set of images that is supposedly of that particular car needs to decide does it come from the data set or from the generator. So it uses the same encoder pipeline here it's just a like a CNN giving you a latent representation and then it has two tasks the discriminator has two different tasks. First of all this here is the regular GAN path so it there's an there's an MLP there's simply a pipeline that outputs a number and the number is does this come from the data set or does this come from the generator but then there is an additional pipeline that they have found to be vital to train the objective which is a reconstruction pipeline. So this is more like a sort of like an auto encoder pipeline where they have a decoder they try to reconstruct the input set and they then compare it using mean squared error. Now here they try to reconstruct the input set really picture by picture not as a set but picture by picture so that's it's different from the set generator okay and this is this pipeline is just to stabilize the training but it also goes into this the output of the discriminator. So sort of the discriminator is happier the more it can reconstruct the images which seems kind of weird at the beginning but it you know they say it has helped in other GANs I'm not super familiar with GAN literature but it's just a another objective that you can add. So this is going to be the overview right here so if everything works well we should be able to take a set from the data set X right which is going to be you know images different images from the same person we should be able to feed that to the encoder get a latent representation Z for that set that somehow encodes here the identity of the person and now we don't if the encoder works really well we don't have to have seen that person before it will simply somehow encode the identity in that binary vector then we feed that to the generator together with some noise and we'll get out a set of pictures of different views of that same person or the person with a similar identity and pictures with similar kind of picture style and that if our discriminator works well will look very very similar to or will be images really of that same person and our discriminator if we plug that in right here will agree and we plug in the Z right here okay okay that's the overview now the math so they go about this in this sort of probabilistic framework what they say is we have we denote an image set of size n as X and that it comes from the space of sets of images so capital X right here is going to be a set of images as you see here and this here is the space of all sets of images okay so what they want to do is they want to build a probabilistic model of that so a model where you can input a set and it'll tell you how likely is that now you don't you don't actually have to have a number as an output right here what they often do is they start with a formulation like this and what they end up with is simply a model that allows you to sample from this distribution right from which you can estimate the probability but ultimately what we want is a generator that can sample from this okay so how do they build it they decompose this into two parts and this is just a you know a decomposition this is standard decomposition of probability where you'll say okay what's the probability of a set X the probability of a set X is the probability of the latent code of that set times the conditional probability of that set given the latent code so already we we have we ask ourselves what's the probability of X and what we might ask is huh well X you know if I look at X it has these different images of of this it has these different images of that thing whatever is on the image I can first ask what is the probability of that particular thing on the image and then conditioned on that what is the probability that I'll get these particular images right this is this simple decomposition already kind of builds up this model of encoding decoding so we'll go through that step this here is going to be a deterministic function our encoder and this here is going to be a probabilistic function the the decoder and it's probabilistic because every time you call it basically every time you call the generator it's going to give you a different output because you you're going to feed it different noise at the beginning okay so so this is going to be this is going to be our encoder here and this is going to be our decoder so they say here X is a Z is a deterministic function that maps a set X to an element in a discrete space Z okay this is a discrete space as opposed to maybe a regular autoencoder where you have a continuous space so here we want to discrete space and a lot of mathematical problems are going to arise from the fact that this Z is a discrete space and not a and not a continuous space so here P is a prior distribution with the support given by sub C which means all the Z vectors that have some sort of set associated with them which is a subset of all the Z so that basically means that if if we have a given encoder there not all of these binary vectors are going to be filled even you know with if we plug in all the world's faces into our encoder it might not fill all of the binary capabilities that we have so this prayer is only defined on this support of this and here you already kind of see what what kind of mathematical hurdles you have to go through if you do something like this and all the math here is going or most of the math here is going to deal with the fact that we have this discrete thing and so on and a bit of a bit of a little bit of a of a caveat here also is that this here they mentioned this here is a prior you need a prior distribution on your Z variables and this is also not easy so really quickly what does it mean to have a prior distribution on this kind of thing because usually in a regular like auto encoder variational auto encoder right your latent your latent code you'll have a prior on it and that prior can be you know some some continuous thing like a Gaussian and even in a regular GAN as I said you you have your your noise distribution and so on what is a prior on that thing now you can say oh a uniform prior but again we would like to learn this prior such that it matches the data set well now they use a prior from a paper that's called made and a DE and really quickly what it does is it sort of kind of decomposes this thing so what you'll have is a neural network that outputs binary vectors like this and it will sort of output them in a fashion auto regressively so it will output one of them and then condition on that it will output the next condition on that it will output the next and this is such that the probability of this binary vector minus 1 1 1 minus 1 and so on is going to be decomposed into the probability that there's a minus 1 here times the probability that there is a 1 here given that there is a minus 1 here and so on and in different order I don't really want to go into this but just to show you that there is a lot of consideration of mathematical consideration if you really want to go about really want to go about this sort of thing in a formal fashion so they define two things first of all this prior okay it's a prior distribution that they can learn from the data set and then there is this conditional distribution this what you might call a generator right if you're given a Z a one of these binary codes what's the probability of a given set so you're I tell you here it's a Scarlett Johansson what's the probability of these pictures being different views of Scarlett Johansson so that is going to be simply we're going to build this as an energy based model you can you can do this what you'll have to do is you'll have to define an energy and we'll just quickly discuss what that is and then you can build a construct like this where you'll say the probability of a given set is going to be the energy that assigned to that set divided by the energy that I'm going to assign to all of these other sets so this is it's a form of an energy based model you can phrase very many things in terms of these energy based models and young LeCun gave a talk about this at iClear I believe where he gives a lot of different examples of energy based models so I invite you to check this out I've also done a video on some of these energy based models and what you can do with them here it's simply to define this probabilistic model so what we need to do are two things we need to know what is this energy so what is this energy supposed to do this energy is going to be a function that gives you and now I have to think so its energy is going to be a function that gives you a high value if you are unhappy with the input and it gives you a low value if you are happy with the input okay so you see the negative exponential here which basically means if and also the energy is always positive so the best if you are super duper happy with what is what your input is into the function you'll output 0 so if you output 0 here you'll see that e to the exponential function of negative something is going to be quite small and no maybe I have it wrong maybe you output a really high number when you're really happy I'm not sure but it's one of the two so this comes this comes from from a physics from a physics background no no no I'm right so if you if you output if you're not happy you'll output a super high number here which will make this negative exponential be really close to zero and therefore the probability you say if I'm not happy the probability should be close to zero however if you're really happy you'll output a low number right here now the energy always has to be greater or equal to zero but the lower you go the higher this probability is going to be the bottom thing here is simply to normalize the distribution because in a probability distribution you always have to normalize because otherwise it's not a probability and this is what most of these models basically are fighting over how to normalize the distribution and what we're going to do is simply normalize it by sampling which is what most of these things do you can build energy-based models without this which GANs are a variant of that but okay so what we need to do is we need to come up with this energy function that is going to be a high number when we're not happy with the input now what is the input the input is X and Z X and Z what does it mean we're not happy with the input it means that the the image X and you see X here is one of the images of the set that particular image isn't really congruent with the Z with the identity so you either you say what this this isn't really a picture of scarlet Johansson so I'm going to assign this a high value however if you know if this is really any sort of picture of of that person then you're going to assign it a low value and how better to do this than to build a neural network to do that and this is going to be our discriminator so our discriminator is going to take the role of this energy function okay cool now I said you need to normalize and I kind of said it off the cuff and so on but the problem here is again we have this kind of sets and and so on so our probability as you'll notice is the probability of X given Z so we are already given the the identity of the person so what do we need to normalize by we can't simply normalize by all the sets of images in the world like here it's in the integral we need to normalize by all the sets of images that are mapped to that same identity okay so in order to normalize the distribution will basically ask how likely is this set how happy are you with this particular set right here compared to all the sets that could exist that would map to the same identity all right that's why these indicator functions are here and as you can see the part here is simply a normalization where you say I'm just going to produce other I'll consider all the other possible sets of images that map to the same person and I'll simply divide by the energy of those in fact if you do it correctly this particular X is also in that particular set but usually it's going to be a fairly small part but to properly normalize of course you have to consider it as well now this bottom part here as I already said is going to be the main problem of most of these probabilistic methods and as I already said again it's usually approximated by simply sampling a bunch of these sets and not sampling a bunch of these sets and not a by enumerating all the possible sets of images and this sampling is going to make some further problems as we'll see I guess down here so here is what we optimize ultimately we optimize or one of the things we optimize is or are we okay yeah so they say we apply maximum likelihood estimation to estimate the parameters of the thing we just defined where where the negative log likelihood loss for an observed set in the training split is this so this is simply the negative log likelihood the log decomposes into a sum so this is going to be your prior and this is going to be your generator discriminator combo okay it's the generator producing images from that binary code and then the discriminator assigning high or low values to that produced images and also to images from the data set and the thing over here is simply going to be the prior over the z distribution that we briefly discussed now again they have to they have to do some tricks here where they say okay we can get rid of this support by using a normalized distribution over Z which is a bound on that true prior and so on so they're going to replace the the P with a P bar which is over the entire space of Z and and that's going to be a bound but the interesting part I feel more is here where they consider the loss again of this conditional distribution on Z so you'll see the exact same quantity right here but now our loss is going to be the negative log of that now since it's the negative log you can decompose the division into a sum and and this part up here you will see the indicator function here is a bit unnecessary because the the Z here is the Z that we're considering is going to be the Z of that particular set that we're considering and so this equality holds on the top so disregard this and this down here as I said is simply a filter to filter the space of all sets to the ones that correspond to the Z that we have in the energy function so this goes here because it's simply a log of an exponential and the negative signs cancel so you'll end up with this what does it mean you want to minimize this loss right here and part of that is going to be you want to minimize the energy function of these inputs okay you want you want to and this is now the case when it comes to from the data set right so when X and Z come from the data set and E is your discriminator then you want to make the output of the discriminator really small which means that you want to train the discriminator to say I'm really really happy that this particular image comes together with this particular identity encoding now the in the in the if it comes from the generator of course you want to do the exact opposite you want to assign it a high value remember energy low means happy with the input okay and then the normalization down here is as I said the problem so you'll see it stated right here because it's under the the division it's going to get a pick up a negative sign which combines with this negative sign which gives you positive sign right here now this part right here is going to be intractable because it it's not going to be feasible to enumerate all the sets of images it's not even going to be feasible to enumerate all the sets of images that just correspond to that particular identity and in fact it's not even going to be feasible to sample from that because we have no clue right we can't simply generate out of the ether true other pictures of the of a particular person what we can do of course is we can use our we can use our model to produce more images right of that particular identity so what we'll do is we'll replace this distribution with a variational distribution and we'll sample from that now this isn't exactly the same this isn't this log probability anymore and that's why first of all we have a bound here and not an equality this is this is called a variational approximation so we bound this quantity and we can only bound this quantity if we down here introduce the entropy of the variational distribution this is a fairly standard trick in variational approximation methods if you want to look more into this look into kind of VAE explain variational autoencoders explained or anything like this will teach you how how these methods work and in case we replace the distribution with a distribution we can actually produce and what is that distribution look what we're supposed to do is we're supposed to produce a set of image given so sets of images here given particular Z and we can do that that's our generator right so we can use our generator to produce those samples and that's what they say here here we have derived a lower bound by introducing a variational distribution which we parameterized in the form of a generator ok so the generators going to produce that distribution it's going to use this noise production so as you know the generator takes two things it takes the identity encoding and a bit of noise and is going to produce an output set or for each noise is going to produce an output image very cool so that's that's the the kind of math formulation behind this model now they have a model architectures right here but this is all fairly standard except for so for the prior they learned the prior on the z space so you see you have Z being this binary vectors they say we use a standard autoregressive model made with three fully connected layers mainly for its simplicity and robustness and so again maybe you like it took me a while to get what this prior does this prior is supposed to say it's so it's not in again you have the Z vectors always being some sort of from the standard noise but what you can also do is you can learn better noise distribution a better input distribution for your again by basically making again for your input distribution so what you'll do is you'll have a z zero right here and then you'll use a you learn again to learn better input distributions ok and this is what you do here with these with this prior on Z this is more standard in like VAE's than it is in GANs but it exists so encoder say as a necessary option encoder for a set needs to satisfy the permutation variant property we opt to use a simple architecture design where we let this be the average right here so as you can see this is the average and then they they use this binarize operation and the binarize operation here is clamping the values to one or negative one and it is a straight through estimator which means that you will you back prop through it as if you hadn't clamped but you forward prop through it with clamping this is kind of a trick to get through discretization things discriminators job is to assign low energy to observed images and high energy to generated images given a set code Z we use an auto encoder based energy function implementation similar to 25 and here they say we have found that this choice is important as it enables effective learning in early stages of training so that's why they do usually a discriminator would be the energy would be equal to this thing right here which is a small MLP that maps a the input to a real sorry you can't see that that maps the inputs to a real number either high energy I'm not happy low energy I'm very happy here they also include this thing right here which is a decoder so it's kind of a you can maybe think of it as a another little GAN another another another little generator or the the generator part of a VAE or of an autoencoder sorry not a VAE an autoencoder that takes as input the encoding of the particular image and the identity and produces is going to produce something that's close to the output against observe that this is now with respect to a particular image so here we're trying to reconstruct that particular image because we have its input thing right here and we're it's not the same as the generator that is just asked to produce a some view something that corresponds to this particular identity vector okay the generator generates a set conditioned on a set code by sampling and random variables each of which is concatenated with Z and generates an image independently cool so what the losses they they now introduce some margin losses on the things here but basically you can just translate the what we have on top where we formulated negative log likelihood into the losses right here they do have some simplifications for example this to train the prior you what you are to train to train the encoder I think you have to make a bit of an approximation in that the encoder is supposed to match this this Z vector right and that's not differentiable by itself so they have this sort of l1 approximation right here they leave away the entropy from the loss and they have found that to work well they introduced this margin losses right here I I don't want to go into that too much but basically they simply in a way with some approximations they approximate I hear it is the indicator function they approximate is this I was looking for that they they they optimize this log likelihood from above in the way where they always optimize they keep the generator constant and they optimize the rest of the pipeline so the encoder and the discriminator in the prior and then they keep that rest fixed and they encode the generator so what does that do before remember right here we had this this approximation right here where we said you know what comes out of this we were not really optimizing this we're optimizing we're minimizing a lower bound on it right so here's a quantity that we want to minimize but here's a lower bound and we'll just push that lower bound down by optimizing it now that doesn't tell us anything about this thing right here but there is actually more to it so by optimizing the discriminator and the encoder and so on we do minimize this lower bound so that this this loss right here you see this energy function will adjust that whenever we adjust our whenever we adjust our our that particular loss our discriminator will adjust that energy function whenever we adjust our encoder we are going to adjust the part that generates the Z vectors right here so we'll push this down but whenever we optimize our generator that's when we make this gap here smaller okay so we always do two steps first we or first or second in one step we reduce this and in the other step we'll bring these two closer together and as a result of course we hope that it's not just the bottom one going to up down up down up down but we hope that both of them reduce with time because the top one is the one will actually want to reduce that's our actual loss or our log likelihood and that is I guess going to happen in practice so what does this do so as I as we already saw here on top you have a set and you feed that through the encoder feed that through the encoder that gives you a Z identity and then you feed that to the generator and the generator you can ask it you don't have to produce the same amount of images you can produce any amount of images you like they just chose to produce the same amount there's no correspondence but you see it's the same truck and here they manually align these so they just produce a bunch of images on the left is the data set and on the right I guess they just produced like a hundred images and then selected wherever the car looked like the closest to so they ordered them by by hand and that is to show that for example look at the the lighting on the car right here it's it's fairly similar I guess this one has red taillights and the other one hasn't but you can see that the the different views are pretty well captured by the generator and that just from all of these are created from one one binary encoding of this here so this is binary encoded to Z and then all of these different views are created there's no image correspondence so that's pretty cool and another problem you have with sets is how do you evaluate sets you can't you can't go and check for images or image closeness and so on so they have to do some 3d modeling they actually take it now they take these images right here and they have to approximate their 3d shape and then compare that 3d shape with the 3d shape of the original thing in order to just quant quantitatively estimate how well they're doing in the faces the the same thing you input the top row into the encoder and you get back the bottom row we've already looked at that but again to evaluate this they you actually have to go and use some sort of a face detector to recognize is that even is that the same person always and is it so you can evaluate two things you can evaluate are these right here all the same people so you can have a a face detector kind of tell you whether or not these are the same people and the second thing is are these down here the same person as these up here right so those are the the kind of things how you can evaluate this and they've done this and it's a fairly interesting and the results here are not surprising when you look at the images so these are curves curves from this face detector and of course for real images as you can see the this is simply the performance of the face detector so you do get some false positives if you if you want more true positives right so this is a standard curve right here because these face detectors are not perfect so in a given row right here in a given row even if that's from the real data set the face detector would sometimes fail and say no that's not the same person even though from the data set you know it is though the the to match the actual child photo from Ali with his adult photos is even like you can forgive the face detector so that's sort of the the gold standard we're trying to achieve and you can see within the reconstructed sets that that is achieved fairly fairly well so compared to uniform samples this is you know fairly fairly cool fairly close what is less close is this reckon and real and I believe that's when you compare the identity of the real row with the identity of the reconstructed row and that's here so that tells you already that it's the GAN or sorry the model doesn't always preserve the actual identity as seen by a face detector and I don't know what to say except yes that's what you see in the data right also you see that free samples I guess so you can do two things right you can give it a set like a row and encode that into the Z and then you can decode that again and basically reconstruct or you can just sample since you've learned a prior on the Z variable you can simply sample you can simply say give me some new identity maybe that I've never seen before right you have some binary vector and now generator please give me images of that identity and these two here are actually sampled like this and you can see again here it's remarkable that within the same row it's pretty much the the rough identity of the person is conserved right and these are these free samples right here I guess and they they do better than whenever you compare the reconstructed and real but they don't do as well as when you actually input a real data and then reconstruct this so this might be an indication that this prior isn't really working you know all too accurately and I do have my problems with this binary encoding right here because maybe I'm misunderstanding something but if you have these binary vectors as we said here the reason you know the reason why you do one hot encoding in class conditional GANs is you could you could simply say what am I doing a one hot encoding I'll simply say Z equals three for class three and Z equals four for class four like that it should be so easy why am I doing one hot and that's because these models see everything in a linear fashion so if you have class three and then I have class four and then I have class nine the model doesn't see that as three different classes the model sees this as these two are somehow closer together than this right so the reason why we do one hot vectors is that the model cannot do this the model has one independent dimension for each of the classes and whenever that particular dimension is high then it knows that that particular class is activated what this binary encoding here does is sort of it goes back to this thing right here where it says okay there are all of these different categories here it's like you have mini classes and the identity of whatever set you consider is now encoded in these mini classes so that I'm going to guess the first thing here might be something like does that person have a blonde hair and the second thing might be does the image look generally bright or the images image set as a whole look generally bright or dark and and so on so I'm gonna guess these things are encoded here and it'll sort of just end up being kind of a discrete GAN or a discrete autoencoder rather than what they believe but maybe that was their goal all along and I'm misunderstanding right here I just don't think this this binarization is gives you this sort of hoped expressiveness I think there's still a lot of dependence of whether or not a particular thing is on or off okay but enough ranting right here I want to look at the at some more of the samples because I've only shown you the reconstructions what I also find interesting is the free samples so here you can see uncurated shape net samples and on the left so here you can see this effect on from the learned order regressive prior and a uniform prior on the right and here you can see this effect of learning this prior so if I learn the prior it's going to give me back fairly okay objects if I don't learn the prior oh but if I learn the prior you know if I learn the prior really really well that basically means I'm only going to ever produce sets that were in the training data right if I learned like a perfect prior I'll see like wait this you know this particular identity here never shows up so I'm not going to output it and the uniform prior might actually output it and the generator is not going to be trained on that uniform prior so it's just going to give you kind of crap and here in the in the faces you see the same thing now again what I think I don't think that's happening what I think is happening is encoding these kind of micro characteristics not per se identity but it's encoding probably you know hair color what not head shape and so on things like this and in each of these dimensions and that's what is then going to produce so these each row here is an is one sample from that prior on the left is learned which you see is working pretty well in terms of the output and on the right you see it's from the uniform prior now you also see here first of all that approximately identity is preserved but not as much in this uniform prior that's first and second you see that the images are much worse which means that the generator doesn't have as much training on that particular thing because I guess it comes from a prior that it hasn't seen during training alright and here lastly they have reconstructions if you give different number of views so the top row I guess is the input the this row is when you just have four different views so I guess just the first four or something like this input and the bottom one is when you have the full eight views and you can I guess see or even more that this increases with number of views so the the accuracy of this identity increases the more views you input of the set and they have a bunch of other things right here in the appendix I I do invite you to look at this and I hope you sort of saw into a bit how you would go about something like this I I found it quite challenging the math because I'm mainly not used to this kind of variational math but I hope this gives you sort of an impression alright this was it from me tell me what you think and I'll see you next time bye bye
 Hi there, today we're looking at set distribution networks, a generative model
 for sets of images by Xuangfei Chai, Walter Tablet, Miguel Angel Bautista, Carlos
 Gestrin and Josh M. Suskind of Apple. So this paper introduces a generative model
 for sets and it does so in an energy-based model fashion. It will have
 an encoder, a decoder in form of a generator, it will have a discriminator
 and it will have all kinds of math but the end result is a model that can
 generate sets of images and by sets we mean it can generate different kind of
 views on the same identity of image and you'll see what that means and it can
 generate even sets that it has never seen before which makes it different
 from a class conditional GAN or something like this. So I can't really
 describe it on a high level in a very concise fashion, you'll just have to
 stick around and see what's going on right here. So if you like content
 like this feel also free to share it out and leave it a like, tell me in the
 comments what you like. This is going to be a fairly math heavy paper and I'll
 try my best to kind of distill it down to what's happening because ultimately
 it's not that difficult. Alright so if you have a look at these samples right
 here these are examples of sets of images. Now without actually caring for
 top and bottom row they will have some meaning right here. Top row is always a
 row from the actual data set and the bottom row is the reconstruction of that
 set. Now you'll see that the images don't really have a correspondence so
 you see it's the same truck in the top and the bottom row but the orientation
 here isn't really shared or anything and that's because as we said this is a set
 network. So what you want to do in this problem setting is you want to take, you
 want to build a model that can take this set right here from the data set and it
 can encode it into a latent description that we call z. z simply describes the
 set as a whole. So z here would be, sorry, would be truck right? It would sort of be
 the 3D model so not the class truck but the 3D information of the truck
 without having any information of the different views okay? And then you want
 to build another model that can generate from this low level representation of
 the set, can generate the different views okay? Like each one of these by sort of
 rotating it. So we want to build a model that understands just from the pixels
 that here we have sets of things that they somehow always share a commonality
 and in this case they always share their 3D structure right? What they don't share
 is the view where they rendered from. So our model is supposed to kind of parse
 the two apart and encode that 3D structure just from the pixels in this
 z variable and then encode the fact that you can rotate it and look at it from
 different views into the generative model that then produces the different
 views okay? And the reason why there is no correspondence between the views is
 we simply regard these things as sets. So our final objective is simply going
 to be that the set on top is different views of that particular truck is going
 to be very similar to the set on the bottom which is also different views of
 that particular truck and that's what our model is supposed to do. Now you
 might know something like this from where you can simply say well this this
 looks like a class conditional GAN right? I simply have the class truck and I you
 know I feed this to my generator and my discriminator and my encoder and so on
 and it will produce the same truck and here that and here the bench and so on.
 The problem I guess becomes more apparent when you go to this different
 data set. So this is a face data set and again top row being the input and bottom
 row being the output. Now what is here what is supposed to be preserved you can
 kind of see in the images is sort of the identity of the person on the photo. Now
 you have to kind of gloss around your human bias here. You as a human
 can tell extremely tiny differences between faces and therefore none of the
 actual identities are going to be preserved right? So this on the top I
 believe is Ali and on on the bottom here it's not not Ali. So but you'll have to
 sort of gloss around this and you'll see that what is preserved or what is
 supposed to be preserved is something like the rough identity of the person on
 the picture and also a little bit of the image compositions. So you see here in
 the background you often have sort of these sports backgrounds right where
 there's kind of a washed out stadium or whatnot and you can see that this is
 also preserved here. I think it's kind of a glamour glamour shots of this must be
 some sort of glamour model and you'll see that this as well is preserved. What
 is different within each set is of course the different views on the same
 identity right? So you have the same person and you have pictures of them in
 different of different views, different lighting, different hairstyles and so on.
 Here you even see you have the black and white image and the set that is produced
 also contains some black and white images. So this model this is already the
 trained model here is doing a fairly good job. Here you can see that almost
 all the pictures have some sort of double like two people with one being
 sort of half in frame and it leads to a fairly strange thing. I'm gonna
 guess the model hasn't seen lots of that during training. I like it, particularly
 like this. This is pretty good. Also here we know that a lot of these face
 datasets they don't really have bald people all too often so you can see here
 this results in sort of kind of a weird Richard Branson type picture. In any case
 it does a fairly good job as you can see right here. And what's
 the problem with these faces? Can't we just do a class conditional again where
 we basically say here this is Ali and that's one class in our latent
 vector and then and so on. What we want to do is we want to train something like
 this where we can give in a new identity that we haven't seen during
 training. In fact we want to train something where we don't even know how
 many sets there are going to be in the end. We simply want to train
 it in a way where we say look I'm going to give you a set and the set will have
 images of the same person and you're going to sort of reconstruct that in a
 way where you output a set of images of that person conserving the
 identity of the person and the rough style of the picture. So this is really
 different from a class conditional again as in we don't know how many classes
 there are and there can be new unseen ones during testing. So how do we go
 about something like this? And here is where the thing starts. So we're
 going to dive into a bit of the math here and then into a bit of the
 reasoning but ultimately what they're going to do is they're going to build
 three things. They're going to build an encoder, a discriminator, and a
 generator. So what does the encoder do? And remember our task here is going to
 be the way we train it is going to be by reconstruction, at least one way we train
 it. So the encoder is going to take this set of images that we give it, for
 example here different views of this car, and is going to produce this
 representation, this set representation. Now what should a
 property be of the set representation? For example it should be independent of
 the ordering of these inputs. A set is simply a collection of objects, it's
 independent of the ordering, and it's also independent of the size in some way.
 Of course a bigger set gives you more information but the set identity, the
 fact that this is this particular car, is independent of how many views you have.
 So what they do is they build and they put each image through an encoder which
 is a convolutional neural network and I guess that gives them a hidden
 representation and encoding and embedding for each image and then they
 have this operation called pool and binarize. And so this does two
 things, namely first of all it pools, it simply averages these things. So it goes
 1 over n C of XI, or C of XI I guess. So this simply averages these
 encodings right here and this already fulfills our property. So this average is
 independent of the order of the images and also I can add more and I can add
 less in expectation the average will result in the same thing. So this
 here now we have basically lost the information of the exact
 ordering and so on. This is simply an average of these images. It's a good
 representation for a set. This is now if we give enough images this will be
 independent of the particular rendering position and it will only depend on the
 fact that this is that particular car, if it's trained well of course. And then the
 second thing is binarize. And here you have to understand what
 exactly this set latent representation is. How do we encode a set in latent
 space? And as far as I understand it, what they do is they do
 the following. Since they don't know how many sets there are, they can't simply
 do the classic one-hot vector. So what you would do in a class conditional
 GAN is you would say I have a vector and maybe I have ten classes. So I'll make ten
 entries right here. Is that ten? I don't know. So if I have C classes I'll
 make C entries right and I'll put a zero in all of them and a one in where a one
 where my class is or something like this. So this would be a valid encoding for a
 class conditional GAN to represent the identity of the class. Here however no we
 don't know. And also we can't really make this continuous because other if you
 make this continuous you wouldn't really encode the identity of the set.
 You would encode more of a continuous latent space and then
 that becomes kind of different when you have new sets and so on. So what they
 really want to do is they want to make this representation here be a description
 of the set itself but not a one-hot. So what do we do? What they do is they do
 the same thing. They have a vector but not of size C because they don't know C
 but of some dimensionality D. This can be 10, this can be 4, this can be
 you know whatever. Just let's say it's 10 again but we don't know how many
 classes there are. What the model can do is it can encode each class as a binary
 vector, a binary combination of negative ones and ones. So it can put like a
 negative one here a one negative one negative one one one negative one and so
 on. So what does that give you? Now you can encode much more than ten
 classes. In fact you can with this you can encode two to the ten classes.
 So it's not that they can encode an unlimited set of
 set identities but they can encode in this manner they can encode a lot.
 They can encode this many sets using a representation like this. So this
 binarize operation it will take this output right here and basically clamp it
 to either one or negative one. So the set will be encoded by a binary vector like
 this and then the generator and the discriminator take that information.
 We'll go over what this architectural choice means but right
 now this you know see that this is a way to encode a large number of
 set identities in a low dimensional vector. So there are two things now.
 There are the discriminator and the generator. So first of all the generator
 is pretty easy. The generators task is to take this Z that we just saw which is
 the set identity and to generate different instances of that set.
 For that it needs this noise here. So if you know a generator from a from a
 GAN it always kind of needs input noise in order to produce different
 outputs and that's this thing on the right here. This Z' they simply come
 from some sort of latent distribution. I think they call this P of psi which is
 some like a uniform or a Gaussian or something. You just sample some noise
 right and you combine it with this thing right here which is the set identity.
 You concatenate it and then each one of these different Z will produce one
 different view right here. So the generators task is simply take
 the set identity and combine each with some noise and produce some views of
 that set. Now the discriminators task right here is going to be to decide it's
 going to get a set of views of a set of pictures and it's going to have to
 decide is this set coming from the generator or is this set coming from the
 data set. Now you can't simply compare images to each other like you
 would do in a regular GAN because they don't correspond to each other right but
 what should correspond to each other is this identity this Z identity. So the
 discriminator is going to take also two inputs. It's going to take this set right
 here or you know from the data set and it's going to take this right here this
 Z. Now this Z is going to be the set identity and that you get from the
 encoder right it's the same as you get right here you get it from the encoder
 which set you should produce and the same goes here. So the discriminator knows
 the set discriminator knows for example I'm trying to produce that particular
 car and it gets a set of images that is supposedly of that particular car needs
 to decide does it come from the data set or from the generator. So it uses the
 same encoder pipeline here it's just a like a CNN giving you a latent
 representation and then it has two tasks the discriminator has two different
 tasks. First of all this here is the regular GAN path so it there's an
 there's an MLP there's simply a pipeline that outputs a number and the number is
 does this come from the data set or does this come from the generator but then
 there is an additional pipeline that they have found to be vital to train
 the objective which is a reconstruction pipeline. So this is more like a sort of
 like an auto encoder pipeline where they have a decoder they try to reconstruct
 the input set and they then compare it using mean squared error. Now here they
 try to reconstruct the input set really picture by picture not as a set but
 picture by picture so that's it's different from the set generator okay and
 this is this pipeline is just to stabilize the training but it also goes
 into this the output of the discriminator. So sort of the discriminator
 is happier the more it can reconstruct the images which seems
 kind of weird at the beginning but it you know they say it has helped in other
 GANs I'm not super familiar with GAN literature but it's just a another
 objective that you can add. So this is going to be the overview right here so
 if everything works well we should be able to take a set from the data set X
 right which is going to be you know images different images from the same
 person we should be able to feed that to the encoder get a latent representation
 Z for that set that somehow encodes here the identity of the person and now we
 don't if the encoder works really well we don't have to have seen that person
 before it will simply somehow encode the identity in that binary vector then we
 feed that to the generator together with some noise and we'll get out a set of
 pictures of different views of that same person or the person with a similar
 identity and pictures with similar kind of picture style and that if our
 discriminator works well will look very very similar to or will be images really
 of that same person and our discriminator if we plug that in right
 here will agree and we plug in the Z right here
 okay okay that's the overview now the math so they go about this in this sort
 of probabilistic framework what they say is we have we denote an image set of
 size n as X and that it comes from the space of sets of images so capital X
 right here is going to be a set of images as you see here and this here is
 the space of all sets of images okay so what they want to do is they want to
 build a probabilistic model of that so a model where you can input a set and
 it'll tell you how likely is that now you don't you don't actually have to
 have a number as an output right here what they often do is they start with a
 formulation like this and what they end up with is simply a model that allows
 you to sample from this distribution right from which you can estimate the
 probability but ultimately what we want is a generator that can sample from this
 okay so how do they build it they decompose this into two parts and this
 is just a you know a decomposition this is standard decomposition of probability
 where you'll say okay what's the probability of a set X the probability
 of a set X is the probability of the latent code of that set times the
 conditional probability of that set given the latent code so already we we
 have we ask ourselves what's the probability of X and what we might ask
 is huh well X you know if I look at X it has these different images of of this
 it has these different images of that thing whatever is on the image I can
 first ask what is the probability of that particular thing on the image and
 then conditioned on that what is the probability that I'll get these
 particular images right this is this simple decomposition already kind of
 builds up this model of encoding decoding so we'll go through that step
 this here is going to be a deterministic function our encoder and this here is
 going to be a probabilistic function the the decoder and it's probabilistic
 because every time you call it basically every time you call the generator it's
 going to give you a different output because you you're going to feed it
 different noise at the beginning okay so so this is going to be this is going to
 be our encoder here and this is going to be our decoder so they say here X is a
 Z is a deterministic function that maps a set X to an element in a discrete
 space Z okay this is a discrete space as opposed to maybe a regular autoencoder
 where you have a continuous space so here we want to discrete space and a lot
 of mathematical problems are going to arise from the fact that this Z is a
 discrete space and not a and not a continuous space so here P is a prior
 distribution with the support given by sub C which means all the Z vectors that
 have some sort of set associated with them which is a subset of all the Z so
 that basically means that if if we have a given encoder there not all of these
 binary vectors are going to be filled even you know with if we plug in all the
 world's faces into our encoder it might not fill all of the binary capabilities
 that we have so this prayer is only defined on this support of this and here
 you already kind of see what what kind of mathematical hurdles you have to go
 through if you do something like this and all the math here is going or most
 of the math here is going to deal with the fact that we have this discrete
 thing and so on and a bit of a bit of a little bit of a of a caveat here also is
 that this here they mentioned this here is a prior you need a prior
 distribution on your Z variables and this is also not easy so really quickly
 what does it mean to have a prior distribution on this kind of thing because
 usually in a regular like auto encoder variational auto encoder right your
 latent your latent code you'll have a prior on it and that prior can be you
 know some some continuous thing like a Gaussian and even in a regular GAN as I
 said you you have your your noise distribution and so on what is a prior
 on that thing now you can say oh a uniform prior but again we would like to
 learn this prior such that it matches the data set well now they use a prior
 from a paper that's called made and a DE and really quickly what it does is it
 sort of kind of decomposes this thing so what you'll have is a neural network
 that outputs binary vectors like this and it will sort of output them in a
 fashion auto regressively so it will output one of them and then condition on
 that it will output the next condition on that it will output the next and this
 is such that the probability of this binary vector minus 1 1 1 minus 1 and so
 on is going to be decomposed into the probability that there's a minus 1 here
 times the probability that there is a 1 here given that there is a minus 1 here
 and so on and in different order I don't really want to go into this but just to
 show you that there is a lot of consideration of mathematical
 consideration if you really want to go about really want to go about this sort
 of thing in a formal fashion so they define two things first of all this
 prior okay it's a prior distribution that they can learn from the data set
 and then there is this conditional distribution this what you might call a
 generator right if you're given a Z a one of these binary codes what's the
 probability of a given set so you're I tell you here it's a Scarlett Johansson
 what's the probability of these pictures being different views of Scarlett
 Johansson so that is going to be simply we're going to build this as an energy
 based model you can you can do this what you'll have to do is you'll have to
 define an energy and we'll just quickly discuss what that is and then you can
 build a construct like this where you'll say the probability of a given set is
 going to be the energy that assigned to that set divided by the energy that I'm
 going to assign to all of these other sets so this is it's a form of an energy
 based model you can phrase very many things in terms of these energy based
 models and young LeCun gave a talk about this at iClear I believe where he gives
 a lot of different examples of energy based models so I invite you to check
 this out I've also done a video on some of these energy based models and what
 you can do with them here it's simply to define this probabilistic model so what
 we need to do are two things we need to know what is this energy so what is this
 energy supposed to do this energy is going to be a function that gives you
 and now I have to think so its energy is going to be a function that gives you a
 high value if you are unhappy with the input and it gives you a low value if
 you are happy with the input okay so you see the negative exponential here which
 basically means if and also the energy is always positive so the best if you
 are super duper happy with what is what your input is into the function you'll
 output 0 so if you output 0 here you'll see that e to the exponential function
 of negative something is going to be quite small and no maybe I have it wrong
 maybe you output a really high number when you're really happy I'm not sure but
 it's one of the two so this comes this comes from from a physics from a physics
 background no no no I'm right so if you if you output if you're not happy you'll
 output a super high number here which will make this negative exponential be
 really close to zero and therefore the probability you say if I'm not happy the
 probability should be close to zero however if you're really happy you'll
 output a low number right here now the energy always has to be greater or equal
 to zero but the lower you go the higher this probability is going to be the
 bottom thing here is simply to normalize the distribution because in a probability
 distribution you always have to normalize because otherwise it's not a
 probability and this is what most of these models basically are fighting over
 how to normalize the distribution and what we're going to do is simply
 normalize it by sampling which is what most of these things do you can build
 energy-based models without this which GANs are a variant of that but
 okay so what we need to do is we need to come up with this energy function that
 is going to be a high number when we're not happy with the input now what is the
 input the input is X and Z X and Z what does it mean we're not happy with the
 input it means that the the image X and you see X here is one of the images of
 the set that particular image isn't really congruent with the Z with the
 identity so you either you say what this this isn't really a picture of scarlet
 Johansson so I'm going to assign this a high value however if you know if this
 is really any sort of picture of of that person then you're going to assign it a
 low value and how better to do this than to build a neural network to do that and
 this is going to be our discriminator so our discriminator is going to take the
 role of this energy function okay cool now I said you need to normalize and I
 kind of said it off the cuff and so on but the problem here is again we have
 this kind of sets and and so on so our probability as you'll notice is the
 probability of X given Z so we are already given the the identity of the
 person so what do we need to normalize by we can't simply normalize by all the
 sets of images in the world like here it's in the integral we need to
 normalize by all the sets of images that are mapped to that same identity okay so
 in order to normalize the distribution will basically ask how likely is this
 set how happy are you with this particular set right here compared to
 all the sets that could exist that would map to the same identity all right that's
 why these indicator functions are here and as you can see the part here is
 simply a normalization where you say I'm just going to produce other I'll
 consider all the other possible sets of images that map to the same person and
 I'll simply divide by the energy of those in fact if you do it correctly this
 particular X is also in that particular set but usually it's going to be a
 fairly small part but to properly normalize of course you have to
 consider it as well now this bottom part here as I already said is going to be
 the main problem of most of these probabilistic methods and as I already
 said again it's usually approximated by simply sampling a bunch of these sets
 and not sampling a bunch of these sets and not a by enumerating all the
 possible sets of images and this sampling is going to make some further
 problems as we'll see I guess down here so here is what we optimize ultimately
 we optimize or one of the things we optimize is or are we okay yeah so they
 say we apply maximum likelihood estimation to estimate the parameters of
 the thing we just defined where where the negative log likelihood loss for an
 observed set in the training split is this so this is simply the negative log
 likelihood the log decomposes into a sum so this is going to be your prior and
 this is going to be your generator discriminator combo okay it's the
 generator producing images from that binary code and then the discriminator
 assigning high or low values to that produced images and also to images from
 the data set and the thing over here is simply going to be the prior over the z
 distribution that we briefly discussed now again they have to they have to do
 some tricks here where they say okay we can get rid of this support by using a
 normalized distribution over Z which is a bound on that true prior and so on so
 they're going to replace the the P with a P bar which is over the entire space
 of Z and and that's going to be a bound but the interesting part I feel more is
 here where they consider the loss again of this conditional distribution on Z
 so you'll see the exact same quantity right here but now our loss is going to
 be the negative log of that now since it's the negative log you can decompose
 the division into a sum and and this part up here you will see the indicator
 function here is a bit unnecessary because the the Z here is the Z that
 we're considering is going to be the Z of that particular set that we're
 considering and so this equality holds on the top so disregard this and this
 down here as I said is simply a filter to filter the space of all sets to the
 ones that correspond to the Z that we have in the energy function so this goes
 here because it's simply a log of an exponential and the negative signs
 cancel so you'll end up with this what does it mean you want to minimize this
 loss right here and part of that is going to be you want to minimize the
 energy function of these inputs okay you want you want to and this is now the
 case when it comes to from the data set right so when X and Z come from the
 data set and E is your discriminator then you want to make the output of the
 discriminator really small which means that you want to train the
 discriminator to say I'm really really happy that this particular image comes
 together with this particular identity encoding now the in the in the if it
 comes from the generator of course you want to do the exact opposite you want
 to assign it a high value remember energy low means happy with the input
 okay and then the normalization down here is as I said the problem so you'll
 see it stated right here because it's under the the division it's going to
 get a pick up a negative sign which combines with this negative sign which
 gives you positive sign right here now this part right here is going to be
 intractable because it it's not going to be feasible to enumerate all the sets of
 images it's not even going to be feasible to enumerate all the sets of
 images that just correspond to that particular identity and in fact it's not
 even going to be feasible to sample from that because we have no clue right we
 can't simply generate out of the ether true other pictures of the of a
 particular person what we can do of course is we can use our we can use our
 model to produce more images right of that particular identity so what we'll
 do is we'll replace this distribution with a variational distribution and
 we'll sample from that now this isn't exactly the same this isn't this log
 probability anymore and that's why first of all we have a bound here and not an
 equality this is this is called a variational approximation so we bound
 this quantity and we can only bound this quantity if we down here introduce the
 entropy of the variational distribution this is a fairly standard trick in
 variational approximation methods if you want to look more into this look into
 kind of VAE explain variational autoencoders explained or anything like
 this will teach you how how these methods work and in case we replace the
 distribution with a distribution we can actually produce and what is that
 distribution look what we're supposed to do is we're supposed to produce a set
 of image given so sets of images here given particular Z and we can do that
 that's our generator right so we can use our generator to produce those samples
 and that's what they say here here we have derived a lower bound by
 introducing a variational distribution which we parameterized in the form of a
 generator ok so the generators going to produce that distribution it's going to
 use this noise production so as you know the generator takes two things it takes
 the identity encoding and a bit of noise and is going to produce an output
 set or for each noise is going to produce an output image very cool so
 that's that's the the kind of math formulation behind this model now they
 have a model architectures right here but this is all fairly standard except
 for so for the prior they learned the prior on the z space so you see you have
 Z being this binary vectors they say we use a standard autoregressive model made
 with three fully connected layers mainly for its simplicity and robustness and so
 again maybe you like it took me a while to get what this prior does this prior
 is supposed to say it's so it's not in again you have the Z vectors always
 being some sort of from the standard noise but what you can also do is you
 can learn better noise distribution a better input distribution for your again
 by basically making again for your input distribution so what you'll do is you'll
 have a z zero right here and then you'll use a you learn again to learn better
 input distributions ok and this is what you do here with these with this prior on
 Z this is more standard in like VAE's than it is in GANs but it exists so
 encoder say as a necessary option encoder for a set needs to satisfy the
 permutation variant property we opt to use a simple architecture design where
 we let this be the average right here so as you can see this is the average and
 then they they use this binarize operation and the binarize operation
 here is clamping the values to one or negative one and it is a straight
 through estimator which means that you will you back prop through it as if you
 hadn't clamped but you forward prop through it with clamping this is kind of
 a trick to get through discretization things discriminators job is to assign
 low energy to observed images and high energy to generated images given a set
 code Z we use an auto encoder based energy function implementation similar
 to 25 and here they say we have found that this choice is important as it
 enables effective learning in early stages of training so that's why they
 do usually a discriminator would be the energy would be equal to this thing
 right here which is a small MLP that maps a the input to a real sorry you
 can't see that that maps the inputs to a real number either high energy I'm not
 happy low energy I'm very happy here they also include this thing right here
 which is a decoder so it's kind of a you can maybe think of it as a another
 little GAN another another another little generator or the the generator
 part of a VAE or of an autoencoder sorry not a VAE an autoencoder that takes as
 input the encoding of the particular image and the identity and produces is
 going to produce something that's close to the output against observe that this
 is now with respect to a particular image so here we're trying to reconstruct
 that particular image because we have its input thing right here and we're
 it's not the same as the generator that is just asked to produce a some view
 something that corresponds to this particular identity vector okay the
 generator generates a set conditioned on a set code by sampling and random
 variables each of which is concatenated with Z and generates an image
 independently cool so what the losses they they now introduce some margin
 losses on the things here but basically you can just translate the what we have
 on top where we formulated negative log likelihood into the losses right here
 they do have some simplifications for example this to train the prior you what
 you are to train to train the encoder I think you have to make a bit of an
 approximation in that the encoder is supposed to match this this Z vector
 right and that's not differentiable by itself so they have this sort of l1
 approximation right here they leave away the entropy from the loss and they have
 found that to work well they introduced this margin losses right here I I don't
 want to go into that too much but basically they simply in a way with some
 approximations they approximate I hear it is the indicator function they
 approximate is this I was looking for that they they they optimize this log
 likelihood from above in the way where they always optimize they keep the
 generator constant and they optimize the rest of the pipeline so the encoder and
 the discriminator in the prior and then they keep that rest fixed and they encode
 the generator so what does that do before remember right here we had this
 this approximation right here where we said you know what comes out of this we
 were not really optimizing this we're optimizing we're minimizing a lower
 bound on it right so here's a quantity that we want to minimize but here's a
 lower bound and we'll just push that lower bound down by optimizing it now
 that doesn't tell us anything about this thing right here but there is actually
 more to it so by optimizing the discriminator and the encoder and so on
 we do minimize this lower bound so that this this loss right here you see this
 energy function will adjust that whenever we adjust our whenever we adjust our
 our that particular loss our discriminator will adjust that energy
 function whenever we adjust our encoder we are going to adjust the part that
 generates the Z vectors right here so we'll push this down but whenever we
 optimize our generator that's when we make this gap here smaller okay so we
 always do two steps first we or first or second in one step we reduce this and in
 the other step we'll bring these two closer together and as a result of
 course we hope that it's not just the bottom one going to up down up down up
 down but we hope that both of them reduce with time because the top one is
 the one will actually want to reduce that's our actual loss or our log
 likelihood and that is I guess going to happen in practice so what does this do
 so as I as we already saw here on top you have a set and you feed that through
 the encoder feed that through the encoder that gives you a Z identity and
 then you feed that to the generator and the generator you can ask it you don't
 have to produce the same amount of images you can produce any amount of images you
 like they just chose to produce the same amount there's no correspondence but you
 see it's the same truck and here they manually align these so they just
 produce a bunch of images on the left is the data set and on the right I guess
 they just produced like a hundred images and then selected wherever the car
 looked like the closest to so they ordered them by by hand and that is to
 show that for example look at the the lighting on the car right here it's it's
 fairly similar I guess this one has red taillights and the other one hasn't but
 you can see that the the different views are pretty well captured by the
 generator and that just from all of these are created from one one binary
 encoding of this here so this is binary encoded to Z and then all of these
 different views are created there's no image correspondence so that's pretty
 cool and another problem you have with sets is how do you evaluate sets you
 can't you can't go and check for images or image closeness and so on so they
 have to do some 3d modeling they actually take it now they take these
 images right here and they have to approximate their 3d shape and then
 compare that 3d shape with the 3d shape of the original thing in order to just
 quant quantitatively estimate how well they're doing in the faces the the same
 thing you input the top row into the encoder and you get back the bottom row
 we've already looked at that but again to evaluate this they you actually have
 to go and use some sort of a face detector to recognize is that even is
 that the same person always and is it so you can evaluate two things you can
 evaluate are these right here all the same people so you can have a a face
 detector kind of tell you whether or not these are the same people and the
 second thing is are these down here the same person as these up here right so
 those are the the kind of things how you can evaluate this and they've done this
 and it's a fairly interesting and the results here are not surprising when you
 look at the images so these are curves curves from this face detector and of
 course for real images as you can see the this is simply the performance of
 the face detector so you do get some false positives if you if you want more
 true positives right so this is a standard curve right here because these
 face detectors are not perfect so in a given row right here in a given row even
 if that's from the real data set the face detector would sometimes fail and
 say no that's not the same person even though from the data set you know it is
 though the the to match the actual child photo from Ali with his adult photos is
 even like you can forgive the face detector so that's sort of the the gold
 standard we're trying to achieve and you can see within the reconstructed sets
 that that is achieved fairly fairly well so compared to uniform samples this is
 you know fairly fairly cool fairly close what is less close is this reckon and
 real and I believe that's when you compare the identity of the real row
 with the identity of the reconstructed row and that's here so that tells you
 already that it's the GAN or sorry the model doesn't always preserve the actual
 identity as seen by a face detector and I don't know what to say except yes
 that's what you see in the data right also you see that free samples I guess
 so you can do two things right you can give it a set like a row and encode that
 into the Z and then you can decode that again and basically reconstruct or you
 can just sample since you've learned a prior on the Z variable you can simply
 sample you can simply say give me some new identity maybe that I've never seen
 before right you have some binary vector and now generator please give me
 images of that identity and these two here are actually sampled like this and
 you can see again here it's remarkable that within the same row it's pretty
 much the the rough identity of the person is conserved right and these are these
 free samples right here I guess and they they do better than whenever you compare
 the reconstructed and real but they don't do as well as when you actually
 input a real data and then reconstruct this so this might be an indication that
 this prior isn't really working you know all too accurately and I do have my
 problems with this binary encoding right here because maybe I'm
 misunderstanding something but if you have these binary vectors as we said
 here the reason you know the reason why you do one hot encoding in class
 conditional GANs is you could you could simply say what am I doing a one hot
 encoding I'll simply say Z equals three for class three and Z equals four for
 class four like that it should be so easy why am I doing one hot and that's
 because these models see everything in a linear fashion so if you have class
 three and then I have class four and then I have class nine the model doesn't
 see that as three different classes the model sees this as these two are somehow
 closer together than this right so the reason why we do one hot vectors is that
 the model cannot do this the model has one independent dimension for each of
 the classes and whenever that particular dimension is high then it knows that
 that particular class is activated what this binary encoding here does is sort
 of it goes back to this thing right here where it says okay there are all of these
 different categories here it's like you have mini classes and the identity of
 whatever set you consider is now encoded in these mini classes so that I'm going
 to guess the first thing here might be something like does that person have a
 blonde hair and the second thing might be does the image look generally bright
 or the images image set as a whole look generally bright or dark and and so on
 so I'm gonna guess these things are encoded here and it'll sort of just end
 up being kind of a discrete GAN or a discrete autoencoder rather than what
 they believe but maybe that was their goal all along and I'm misunderstanding
 right here I just don't think this this binarization is gives you this sort of
 hoped expressiveness I think there's still a lot of dependence of whether or
 not a particular thing is on or off okay but enough ranting right here I want to
 look at the at some more of the samples because I've only shown you the
 reconstructions what I also find interesting is the free samples so here
 you can see uncurated shape net samples and on the left so here you can see this
 effect on from the learned order regressive prior and a uniform prior on
 the right and here you can see this effect of learning this prior so if I
 learn the prior it's going to give me back fairly okay objects if I don't
 learn the prior oh but if I learn the prior you know if I learn the prior
 really really well that basically means I'm only going to ever produce sets
 that were in the training data right if I learned like a perfect prior I'll see
 like wait this you know this particular identity here never shows up so I'm not
 going to output it and the uniform prior might actually output it and the
 generator is not going to be trained on that uniform prior so it's just going to
 give you kind of crap and here in the in the faces you see the same thing now
 again what I think I don't think that's happening what I think is happening is
 encoding these kind of micro characteristics not per se identity but
 it's encoding probably you know hair color what not head shape and so on
 things like this and in each of these dimensions and that's what is then going
 to produce so these each row here is an is one sample from that prior on the
 left is learned which you see is working pretty well in terms of the output and
 on the right you see it's from the uniform prior now you also see here first
 of all that approximately identity is preserved but not as much in this
 uniform prior that's first and second you see that the images are much worse
 which means that the generator doesn't have as much training on that particular
 thing because I guess it comes from a prior that it hasn't seen during
 training alright and here lastly they have reconstructions if you give
 different number of views so the top row I guess is the input the this row is
 when you just have four different views so I guess just the first four or
 something like this input and the bottom one is when you have the full eight
 views and you can I guess see or even more that this increases with number of
 views so the the accuracy of this identity increases the more views you
 input of the set and they have a bunch of other things right here in the
 appendix I I do invite you to look at this and I hope you sort of saw into a
 bit how you would go about something like this I I found it quite challenging
 the math because I'm mainly not used to this kind of variational math but I hope
 this gives you sort of an impression alright this was it from me tell me
 what you think and I'll see you next time bye bye