TUNIT: Rethinking the Truly Unsupervised Image-to-Image Translation (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper image translation style transfer unsupervised clustering self-supervised cnn convolutional neural networks gan generative adversarial network generator encoder discriminator conditional style pseudo-label augmentation cropping
Image-to-Image translation usually requires corresponding samples or at least domain labels of the dataset. This paper removes that restriction and allows for fully unsupervised image translation of a source image to the style of one or many reference images. This is achieved by jointly training a guiding network that provides style information and pseudo-labels.

OUTLINE:
0:00 - Intro & Overview
1:20 - Unsupervised Image-to-Image Translation
7:05 - Architecture Overview
14:15 - Pseudo-Label Loss
19:30 - Encoder Style Contrastive Loss
25:30 - Adversarial Loss
31:20 - Generator Style Contrastive Loss
35:15 - Image Reconstruction Loss
36:55 - Architecture Recap
39:55 - Full Loss
42:05 - Experiments

Paper: https://arxiv.org/abs/2006.06500
Code: https://github.com/clovaai/tunit

Abstract:
Every recent image-to-image translation model uses either image-level (i.e. input-output pairs) or set-level (i.e. domain labels) supervision at minimum. However, even the set-level supervision can be a serious bottleneck for data collection in practice. In this paper, we tackle image-to-image translation in a fully unsupervised setting, i.e., neither paired images nor domain labels. To this end, we propose the truly unsupervised image-to-image translation method (TUNIT) that simultaneously learns to separate image domains via an information-theoretic approach and generate corresponding images using the estimated domain labels. Experimental results on various datasets show that the proposed method successfully separates domains and translates images across those domains. In addition, our model outperforms existing set-level supervised methods under a semi-supervised setting, where a subset of domain labels is provided. The source code is available at this https URL

Authors: Kyungjune Baek, Yunjey Choi, Youngjung Uh, Jaejun Yoo, Hyunjung Shim

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we'll look at rethinking the truly unsupervised image-to-image translation by Kyungjoon Baek, Yoonjae Choi, Yongjung Woo, Ja-Joon Yoo, and Hyeong-Joon Shim. So in this paper we'll deal with image-to-image translation in an unsupervised fashion. So on a high level they replace the need for domain or really single image label annotations in image-to-image translation by training a guiding network that is able to sort of do a self-clustering of the image domain and therefore that guides the image-to-image translation instead of the previously needed labels. I myself don't know too much about image-to-image translation and style transfer and all of this stuff. This has always been kind of a mystery to me and we'll try to make as much sense as possible out of this paper if you're with me. I might not get everything right but I will give my best of course. As always if you like content like this consider sharing it out and leaving a like and a comment. I do read the comments so I get a good idea of what you have to say about it. Cool so what we're seeing here is an example of image-to-image translation of like a sort of a style transfer. Now what you'll have on the left is a source image. Now the goal is to translate this source image to a different domain while sort of keeping the the features of the image the same. And here is sort of I'm always confused because here it's like we keep the pose of the cat the same okay so we sort of keep the same cat but we want to change its style which means it's breed in this particular case. So on the top you can see that the domain images are they come in these different groups and in fact it's not only those four but the entire data set is split into these different groups and among these different groups you have some sort of a shared style. Now this shared style is what you would like to transfer to the source image. So if you transfer the style of all of these cats right here which all seem to be sort of ginger cats to this instance right here what you'll end up with is a cat okay it was ginger before. Might not be the best example but you you sort of get what I mean is that the thing that you transfer is whatever is common among these domain images okay and that's what I guess explains why the pose of the cat stays the same because it only it is basically taught to keep the image the same except to transfer whatever is common among the images in the domain class and that's image to image transfer or translation. Now until this paper at least that's what the paper claims these image to image translation models they required labels and why is that? That's because you need to know how to build these domains here at the top to get these different style vectors out or you actually would need label annotations for each image for each single image you would need to know which one you need to know which one of the source corresponds to which one of the target so they have a graphic right here where they explain the sort of different the different stages that image to image translation went through historically so first you'd have to have corresponding images one to one where you'd say okay here is an example of a sketch of a shoe and here is the corresponding shoe here is the sketch of another shoe and here is the corresponding shoe and so on and from that you could learn a model that translates from one domain to the other because you have corresponding image level annotations which image corresponds to which so basically which element of the domain a corresponds to which element in the domain b then the next stage of this was when you only need set level annotations and that's sort of what we looked at if you had supervised labels for domains so what you'll say is that there are three domains a b and c and actually let's let's forget c for a moment and just deal with a and b to make it equivalent to the thing on the left now i just know that these things are instances of class a and these things are instances of class b yet i i don't there's no correspondence right there is no this corresponds to this or or something like this so image to image translation is now possible between domains when i just have domain level labels but this is still expensive collecting these labels you know it's is like collecting labels for a supervised data so collecting labels for a supervised data set a human needs to look at each image and then conclude what sort of domain it is their paper introduces the following where you do not have domains anymore you simply have a data set x now this data set your hypothesis is that there are still going to be domains in the data set they can i guess they can be overlapping or not but there are still going to be domains you just don't know what they are so in this case um i guess you could differentiate these people into many many different ways but um in essence you're going to assume that there is some kind of a domain structure you just don't know what it is but if you knew what it was then you could simply apply methods from here to the data set and you'd be done now their paper shows that if you apply something like a self-clustering approach and we've seen these approaches before in the paper about learning to classify images without labels if you have techniques like this you can do like a self-clustering approach on this data set x right here and then you could learn your image to image translation yet this paper shows that if you do that the quality is not as good as if you do both things jointly so what this paper does is it jointly learns to cluster let's say to self-label the images and to make the to do this image to image translation and by doing the tasks jointly they help each other perform better okay that's a general overview so how do they do this they have three different parts to their model there is the encoder or they call this the guiding network there is the generator and there is the discriminator so the generator and the discriminator they are fairly standard GAN generators and discriminators so general adversarial network but they have like a bit of some sort of twists so you can already see from the design from the drawings right here the discriminator is probably the easiest the discriminator gets an image right here it doesn't have to be a generated it is a either a generated image or a real image and it needs to decide you can see right here this means that the input domain is a vector or an image in this case and the output is a number it needs to decide if it's real or fake now in fact it's not as easy because you can see there are these multiple heads right here so this whole thing as I said is built on this kind of pseudo clustering approach there is this pseudo label that comes out of the left side we're going to look at that in a second but in essence you assume that there are multiple classes multiple domains in the data set and the discriminator here has one classification head for each of those classes so from somewhere outside it will get the information oh this is now supposed to be one of those ginger cats right as opposed to one of those black and white cats or one of the brown haired cats no it's one of the ginger cats and then there is a special head on top of the classifier that only classifies fake from real ginger cats okay which is a different classifier from the other domains so the discriminator it's sort of a conditional discriminator conditioned on a label okay from the discriminators point of view it's simply a label conditioned discriminator discriminating between real and false and I think that's yeah how you train the discriminator is you would give an image and you would let this encoder here this guiding network label the image and how we come up with this label again that we'll look in a second but this just gives a label and then you'd for that particular label you'd classify the image into real or false now the fact that there is this shared part right here of course is you could also think of having one discriminator per class but the shared part now gives you some shared features and so on but it's not necessary it's not the the point is that there is a discriminator per class it's class conditional okay so what about the generator I think is I guess is the most complex um what about this this encoding network right here it's e for encoder I guess but they also call it the guiding network so what this does is this is what's this supposed to do is it'll take an image any image and it will output two things one is a label and one is a style code so the label is supposed to be a number between zero and da da da da da k minus one so k minus one so that's supposed to be a class label and how do you know how many classes there are if there are no labels you just guess and your best bet is to slightly over guess so if you expect there to be between 10 and 15 classes maybe put k to 20 okay you don't want to under guess but you can over guess but not by too much of course so you have to have this this estimation of how many classes but then this e it simply comes up with a class label and it also comes up with the style code now these two things are going to go then different pathways in this in this network the label is directly going to the discriminator right the generator does not see the label okay the style code does not go to the discriminator but goes to the generator all right so the two inputs from the encoder they one goes to the discriminator which is the label and one goes to the generator which is the style now the generator lastly it takes a source image and it takes this style code right here now the style code is encapsulating as we said the style of the reference image so the style is supposed to be whatever whatever whatever makes this domain of images the same so the style the way we're going to train this is that the style is going to describe somehow all the images that are from this label the style is going to describe whatever the style is it's very hard to it's very hard to explain if we look at the loss it becomes clearer why the things are how they are so it takes the style code and it takes the source image and it combines them and its task is to output this generated image as you can see in this example the generated image is basically this cat but with the style of the reference image and it outputs an image and the discriminator of course then is tasked with differentiating whether that image is real or fake for the given label over here okay so this is the entire thing and you all train this jointly so you jointly train the encoder to produce these class labels and the styles you train the generator to take in the styles and the source images and output the generated image to fool the discriminator and the discriminator at the same time is trained to differentiate between real and fake images based on the label that the encoder gives very very convoluted and complicated but there are a few things that make it easier first of all as you can see here the pseudo label is detached is argmaxed and detached so the pseudo label really is a number and there is no gradient back propagation along this line okay that makes that makes it a lot easier so the so what we first need is we need a way to train the encoder to come up with suitable class labels even though it doesn't get any back propagation signal into that part of its network so that's where we start with the loss functions the way we're going to do this is we're going to take the following approach we're going to take an image and we're going to take a randomly augmented version so for example a random crop or a horizontal flip and so on so the now we bring in ideas from self-supervision and again if you watch the video on learning to classify images without labels this is one of their main staples these self-supervised approaches really tend to learn representations that allow you to self cluster now in that paper they go further and they do this nearest neighbor thing in this paper they just do sort of the first step of this self clustering which i guess makes it such that you could potentially improve this paper by applying the other paper but who knows so we're going to take an image and we're going to augment it okay so that means we're going to like random crop it or in change its luminance or whatnot so we have two versions of the same image and what we want to maximize we want to maximize the mutual information between not between the images themselves but p is going to be this output of the encoder so x goes into the encoder and the encoder outputs the style and the class label and the class label here so p is going to be the class distribution all right so this is going to be like a histogram or maybe the log it's it's already yes so it's going to be a histogram over classes from which we're going to sample the label c or l or whatnot y hat y but the p is the distribution over output classes so since we don't have a label we can't train the distribution like in a supervised way supervised way so what we'll have to say is we want to maximize the mutual information between the output distribution of the image and the output distribution of the augmented image now that entails the following two quantities there's the entropy of p and there's the conditional entropy of p given p augmented now first of all it means we want to maximize this the entropy of p and that's supposed to be over the entire data set so this is the entropy over the entire data set x what it means is that we want different x's so if there's x1 x2 x3 and so on we want those to have different distributions in labels okay so if if the entropy is really high of the distribution p that means that different images get assigned to different classes some something like this all right if this is low then that would mean all the images basically get assigned to the same class and that's not good we want our classifier since we don't have labels it's a it's basically a cluster we want our cluster to sort of fill the space of possible clusters with the images so that's the first thing we want to maximize the mutual information we need to maximize this entropy and then second we want second since this is a minus here we need to minimize the conditional entropy of p given p augmented what does that mean that means if we know the augmented version of an image its class labeling should be the same as the un-augmented version so that means that if i now take one of these x's to x1 augmented of the do a plus augmented right then that shouldn't really change its class label and this is what these so that should sort of keep the class labeling this is horrible but the idea here is that it's kind of a reverse thinking from supervised learning in supervised learning we have the label like this is class this is class five okay this image is class five and our thinking is this augmentation techniques if i random crop an image or if i change its colorization a little bit the class is not going to change right an airplane with a in front of a blue sky is still an airplane in front of a bit bluer sky so i assume that it'll still have the same label here i don't have the label but what i can require is to say whatever you output for the image it should be the same for the augmented image so these two objectives are enough to give you sort of a rough clustering of the output space maximize the entropy minimize the conditional entropy between two versions of the same image okay that's how we train this pseudo labeling approach so now we have a we have a model that can give a label to each image very cool so how do we train the other parts now there are additional um additional losses here so i'm not sure yeah we'll go over it so this style part is also has to be trained right this encoder outputs a labeling we got that covered and it outputs a style part now the style part if you can see from the graphic it actually goes into let me erase some of that stuff here the style part actually is down here and it feeds into the generator and luckily they write detach here and since they don't write detach anywhere here that means that we do get gradient back propagation from the generator to the style code so that means our our encoder here is trained to help the generator with its task of fooling the discriminator okay but um first of all we're going to forget about that for now what we're going to do is simply look at a loss that they impose on the style they wouldn't they don't have to impose that loss but they have an additional loss on the style codes for the encoder in addition to the fact that there is gradient back propagating from g so the second loss we're going to look at is this style loss the style loss is almost the same so the style loss is a contrastive loss so what you want to do is if you have your data set you have your data set of images and you you know take images out and you train your network on and you train then take the next image or you take batches of image you take train and so on like this right and now you have this image what you want to do for this to work is you want to build up sort of a queue of images that you have already looked at like these images these are going and the queue can be let's say 10 long and you would always throw out the oldest and and in queue and newest so when you're done with this image right here you'll put it into the queue you load your next image and so on so now what does this mean you now always have a queue of other images and it's not important what they are as long as they are others right because now we're going to compare ourselves with others and this is this contrastive loss right here so this style loss is going to be a contrastive loss between this and this now the bottom part this here these are the others these are the other images and what are the individual quantities so s is the style code of the image you're considering right now s plus you could have already guessed it is the style code of the augmented image right so we had our image x let's go again with x1 x2 x3 are different images so we put x1 through the encoder that gives us s the style it also gives us the class label but now we care about this head that gives us the style code and we augment x1 to be x1 plus and we go we put that through the encoder that gives us s plus and now we also put all of these other images remember these are the images that we've looked at previously but the only real importance is that there are other images we put those through here and they get the s minus i in this case three and two so now what will require is that the s the style code of our image is closer to the style code of its augmented version so the same principle again we want we'll say that you know these augmentations they don't really change anything about the style now this argument is a bit more wonky but if you think of you know random crops and random flips don't really change anything about the fur color or so of a of a cat so we want those two to be closer together than s is to any of these other images okay so this is a contrast of loss where you pull together two things that you think should be close and you push apart things that you think should be far away from each other so this style loss basically guarantees that you have a distinct style for each image that is robust to the kind of transformations that you do under augmentation okay specifically this style loss doesn't care about the domain right this is for each image you don't know if these other images are from the same domain or from different domains and that's why the style is basically individual to the image but as we as we're going to see the style does capture something of the domain as well but this loss right here is supposed to be each image has a style right so this is the style code of x this n plus one way classification enables e to utilize not only the similarity of the positive pair but also the dissimilarity of the negative pairs where the negative style codes are stored into a queue using previously sampled images we observe that adding this objective significantly improves unsupervised classification accuracy in animal faces from this to that compared to the previous over clustering approach okay so we have two outputs now and now we go to the adversarial loss so the question is how do we train the generator and the discriminator and the discriminator so they have three different losses for the generator and the discriminator and the most important one of course is this adversarial loss right here so the discriminator simply tries to distinguish is an image real or fake conditioned on a class so in case of a real image and that's this line right here it tries to distinguish is this real or fake based on y and y is x fed to the encoder and the encoder gives you a label all right that's and the label selects the head of the discriminator at the same time that the discriminator is trying to distinguish real from fake so these two lines the generator is trying to fool the discriminator so the upper if you've never seen a GAN loss the upper part here that's the real data and the bottom part here is the fake data now at the same time the discriminator is trying to distinguish real from fake and the generator is trying to make the discrim fool the discriminator so both are of the generator and the discriminator are actually using that loss but the sign in front of it is different okay and since the generator is not involved in the top line you can usually leave that away because there is no backprop path through that and there is no backprop backprop path here because we detach the graph right here so there is no gradient signal going to the encoder so this bottom line what does it mean the generator will take in an image and the style now s tilde comes from x tilde it's x tilde going through the encoder giving you s tilde so this is the reference image right this is you want these this style right here this is the reference image and x is the source image so the generator is supposed to take the source image and basically apply the style from the reference image and generate x i don't even know how to call this x not tilde whatever x fake xf and that's supposed to fool the discriminator now the question is which discriminator right because you need a label for the discriminator the label is conditional with this discriminator is pretty easy because it's simply the label of this image now however as you can see the generator learns to translate x to the target domain while reflecting the style code s tilde so y tilde is going to be the label that comes out of this x so this encoder right here is also going to give us y tilde and that's going to go here all right so recap what we want to put into the discriminator is one time a real image like we do up up here and we get its label from the encoder the encoder gets us a label for each image very cool we'll also take the same image put it through the generator task the generator with transferring the style of another image from here onto it we get the style from the encoder and then the generator is supposed to make an image and we feed that to the discriminator and the discriminator discriminates assuming it comes from class y tilde now you see right here the generator never has access to y tilde okay so the generator is kind of at a disadvantage here the discriminator gets told what kind of image it is in terms of class while the generator because it needs to fool the discriminator it needs to come up with an image of that class but it has no idea of the class it only has the style code so it is forced to learn to sort of it is forced to learn to map a style to associate a style with a particular class and that's what with a particular class and that's how you get the domain into the style that's why the style can capture something like fur color of the different cat breeds because the generator is forced to take the style that the encoder gives and map it to an image of the class y tilde that it also the encoder gives but doesn't tell to the generator okay and in fact there is a more path because you now back propagate the loss to the encoder which means that the encoder will even help the generator it will help the generator make style codes that are very class specific now you can maybe think why why wouldn't you just have one output why doesn't the encoder simply output the label also as the style because that would be the easiest and the reason is because we have different losses on the style and the label okay otherwise that would be a valid tactic so that's cool that's the adversary loss that's the most important loss now there's also additional losses so they they do additional losses that they add on top for the generator they say in order to prevent degenerate situation where the generator ignores the style code and synthesizes a random image in the domain y or in the domain y tilde we impose a style contrastive loss to the generator so now there's still the danger that the degenerator simply produces a valid image right from the data set or even from the domain y tilde though i don't know how it would know why tilde or i've just not seen something i in my mind it doesn't get the y tilde but it could read it from the style but here the danger is to ignore the style i'm slightly i'm slightly confused by this part but maybe looking at the loss will will clear it out so they say we impose a style contrastive loss to the generator now this is almost the same is almost the same as we imposed on the encoder so the generator you can see there is a contrastive loss again where you want to be you want these things to be close and you want these things to be far apart so these s minuses these are going to be the ones from your the style codes of the images from your queue so these are just going to be other images here s tilde that's going to be the style that you get from your reference image so your reference image is going through the encoder and that's going to give you this right here now the question is what is s prime here because in the before we simply had s which was our source image our source image style now what is s prime here s prime is going to be it gets more complicated yes s prime is going to be whoops it's going to be the round trip to the encoder so it's going to be if i generate my image from the source image x and the style s tilde of the reference and then i ask my generate my encoder again what style does this have i get the s prime so it's kind of a round trip right so i i take i take this i ask the encoder what style is it that's s tilde right then i take s tilde go to the generator together with a source image x and that gives me like x fake and then i ask my generator again what style would you assign to the fake image i just produced and then the encoder will tell you i'll give it s fake or s prime in this case and then i compare that s prime with the one i gave before okay so it's sort of a round trip loss of my reference image all right so what does that do if i now and then i ask that s prime be close to s tilde so that means if i generate an image with the style of my reference image the upcoming image should better have the style of the reference image that's all it says so the style of the thing i generate given this style they should better be close and especially closer together than the style of the style with any other image in my queue it makes sense but it's kind of convoluted so you go with your out it's kind of a reconstruction loss except in style space all right and then the last thing is an actual image reconstruction loss so what you'll do is your generator will produce x uh sorry will produce an image from the source image and its own style right here that's important before we input s tilde here so this now is we input the source image and its own style so we go with x we go to the e and we put the style here and we tell the generator if i input the input the source image and its own style then what you give me back better be the source image itself right this is a consistency loss that tells the generator that basically it learns now the generator learns to the generator learns to map to recognize an image with its own style sort of because it doesn't know right it doesn't know that what's coming in here is the style of um it of the image x but now you teach it and i think before this loss you'd have a good chance that uh the styles would just be all over the place they would sort of be consistent but they will not be aligned and with this you force that the style of an image itself if you gent if you put that into the generator it will lead to that image itself okay that's it so this is a this is extremely convoluted right the discriminator is the easiest the discriminator is a class conditional discriminator that gets the label from some mechanism that decides on a label right okay that's the easiest the encoder has two parts the pseudo label which is over here which is trained completely unsupervised detached from everything else in a self clustering approach while the style part here is trained first of all in a contrastive way which makes sense and also in a back propagated way from the generator so the style generation mechanism tries to help the generator okay and that means it's going to leak some information about the label into the style because that helps the generator generator needs to if the generator knows what sort of class it's going to produce it's going to be better okay so you can count on that information being in there but also also because of all the other losses that the generator has and the contrastive loss on the style the style code is going to sort of describe the individual style of an image and but is also going to describe what the style of that class is because it technically needs to contain information about the class and that's why i think this works with the style because there is no inherent notion of like this is the pose of a cat or something like this yeah it still seems like a bit magic to me and then the generator is first of all trained to fool the discriminator given a source image and a style and you can fool the discriminator by producing an image that's so good it looks real and specifically it looks real in the class that the pseudo label has given right so in the class that the encoder has given to it so the generator must somehow come up with an image that's of that class and so it will it will be forced to interpret the style code in terms of that class label which makes the style code the style code and also we have these two additional losses which is the round trip loss to the style space so whatever the generator outputs you should be able to recover the style from it by putting it through the encoder again and then lastly there is a consistency loss where you say if i put an image into a source image and i input its own style again going through the encoder you should give me back the source image itself very complex and all of the generator loss is back propagated through to the encoder so this is the full loss as i said discriminator easy adversarial loss generator adversarial loss plus this style round trip consistency plus the own image round trip consistency encoder gets all of the generator loss all of it so all of this goes here so the encoder fully helps the generator and it is also trained with this mutual information and the style contrastive loss wow that's some losses wow that that's a lot of damage so they do different investigations into their model here and i don't even know if we've missed some of the pictures but ultimately what you can now do is you can do image to image translation either that's the cool thing you can have a reference image for one or what you can do is you can ask your discriminator what kind of domains are there sorry you can ask your encoder what kind of domains are there you've guessed the number of domains so it's maybe 10 or in this case it's uh eight eight eight domains of cats and you can simply divide your data set into these eight domains right one two three four five and so on oh this is 10 okay i can't see anymore so 10 domains and then you can simply calculate for each image you calculate the style vector so the style the style and then you simply take the average one over the number in that in that domain you take the average style vector and that's going to be your target style so you can do image to image translation with a reference image or you can do image to image translation for an entire group of images for example all the images in a given domain and that's how they do these graphs right here now just quickly wait until my tablet decides to show me the paper again thank you all right they do a bunch of investigations into their wholly unholy mixture of losses especially the first concern is couldn't we just train the guiding network like by its own on its own and then after that train this gan thing right that's what we had at the very beginning we said there's this guiding network and it does the clustering and all and couldn't we just train this gan architecture on top of the frozen guiding network and their conclusion is no if we train everything together it works better so on the left you have whenever you train the guiding network by itself and what you're seeing here is the t-sne visualization t-sne is a a down like a non-linear visualization tool of style codes extracted by our guiding network the ground truth domains of all test images is represented in different colors so this is a data set that has labels but you don't you don't provide the labels to this algorithm the algorithm is completely unlabeled but for purposes of investigating we'll visualize the labels with colors and what you'll see here are the t-sne visualizations of the style codes so things that are close together they have similar style codes and the ideal case would be if things that are close together here have the same label and that means the style is sort of representative of the domain okay that's what we want we want the style to capture the domain of an image and ideally not the image itself too much now on the left you see that there is quite a bit of overlap between these quite a bit of wash between the style and the group and on the right if you jointly train the gan together with the guiding network you see that these classes of the style codes which have no reason to cluster are much more clustered and separated and they are separated much more along the lines of the ground truth classes okay so that's pretty cool now i would actually be interested in what happens if you do the separate training with the full pipeline of this learning to classify images without labels thing and their nearest neighbor thing because they've also shown that just purely this self-clustering doesn't work too well but if you then do the nearest neighbor thing on top then that improves the classification significantly so this could potentially help either the separate or the joint training right here and there might be a connection between the joint training and whatever they're doing in any case they also show that then these fid which is a quality metric for gans lower is better that the joint training goes way lower in the fid than the separate training okay that's that's the reason why they built this convoluted thing because it works way better and here they ablate they ablate some of the losses to investigate what's really going on and in this case t-sne visualization of the style space of our guiding network trained on this since this does not have ground truth domain labels each data point is colored with the guiding network's prediction so each color is whatever the guiding network says the classes and the dot is one style each is one style each dot is one style vector and they're projected down to two dimensions you can see pretty clearly that the individual classes the individual clusters of style vectors correspond to different labels of the guiding network which is to be expected but also since they overestimate the number of classes in this case you can see that the even though the class label is different the style network will group the very similar classes together you can see here these are both cheetahs and here are both lions so it'll group them together which is pretty cool and sort of verifies that it recognizes these these different things because you force the guiding network to make 10 classes but the style network is simply continuous so it's cool to see that the style network will make one cluster with styles even though it's different labels and here you can see different samples from these domains just to verify that the guiding network has actually learned to separate things i still find this pretty pretty magical too this is completely unsupervised and it sort of finds these clusters by itself all right they have a bunch of images here as i said this is no longer with one reference images image this is where you take the entire domain so you self-label with your guiding network and then you take the mean vector and that's going to be your target style vector and these are the source images that you transfer and you can see that you know it works pretty well so they always have like one adult animal and one child animal and i guess not or just two different ones here this is particularly cute though i have to show you this fox right here what's going on with that fox like someone help that fox yeah um so we're not at perfection yet as you can see but it's you know that that looks like a pretty pretty cool fox maybe okay where did it go maybe it slipped maybe it's an it's an offshoot of this one on the top left yeah who knows these data sets they have their way and um so this is sort of where you can see the limitations right here um that's not how a baby snow leopard looks you see the limitations here in that all of these animal faces they are still pretty aligned like they're fairly frontal not exactly but they're fairly frontal pictures um they're fairly standardized and so on so we're i don't think we're yet at the level where we can just do you know um fully image to image and you see it especially with faces because we as us humans as us humans are extremely good at you know seeing when there's something wrong with a face but it's still it's still pretty impressive what's possible and i think if the past is of any indication here is summer to winter that actually looks good if the past is of any indication then this technology will be pushed pretty hard and soon we'll be able to do this with a simple smartphone app or something like this so i invite you to check out the paper right here they have lots and lots and lots of examples and t-sneak plots and whatnot in their appendix they have the code online as far as i as i have seen and with that let me know what you think in the comments bye bye
 Hi there! Today we'll look at rethinking the truly unsupervised image-to-image translation
 by Kyungjoon Baek, Yoonjae Choi, Yongjung Woo, Ja-Joon Yoo, and Hyeong-Joon Shim.
 So in this paper we'll deal with image-to-image translation in an unsupervised fashion.
 So on a high level they replace the need for domain or really single image label annotations
 in image-to-image translation by training a guiding network that is able to sort of do a
 self-clustering of the image domain and therefore that guides the image-to-image translation instead
 of the previously needed labels. I myself don't know too much about image-to-image translation
 and style transfer and all of this stuff. This has always been kind of a mystery to me and we'll try
 to make as much sense as possible out of this paper if you're with me. I might not get everything
 right but I will give my best of course. As always if you like content like this consider
 sharing it out and leaving a like and a comment. I do read the comments so I get a good idea of
 what you have to say about it. Cool so what we're seeing here is an example of image-to-image
 translation of like a sort of a style transfer. Now what you'll have on the left is a source image.
 Now the goal is to translate this source image to a different domain while sort of keeping the
 the features of the image the same. And here is sort of I'm always confused because here it's like
 we keep the pose of the cat the same okay so we sort of keep the same cat but we want to change
 its style which means it's breed in this particular case. So on the top you can see that
 the domain images are they come in these different groups and in fact it's not only those four but
 the entire data set is split into these different groups and among these different groups you have
 some sort of a shared style. Now this shared style is what you would like to transfer to the
 source image. So if you transfer the style of all of these cats right here which all seem to be sort
 of ginger cats to this instance right here what you'll end up with is a cat okay it was ginger
 before. Might not be the best example but you you sort of get what I mean is that the thing that
 you transfer is whatever is common among these domain images okay and that's what I guess
 explains why the pose of the cat stays the same because it only it is basically taught to keep
 the image the same except to transfer whatever is common among the images in the domain class
 and that's image to image transfer or translation. Now until this paper at least that's what the
 paper claims these image to image translation models they required labels and why is that?
 That's because you need to know how to build these domains here at the top to get these
 different style vectors out or you actually would need label annotations for each image
 for each single image you would need to know which one you need to know which one of the source
 corresponds to which one of the target so they have a graphic right here where they explain the
 sort of different the different stages that image to image translation went through historically
 so first you'd have to have corresponding images one to one where you'd say okay here is an example
 of a sketch of a shoe and here is the corresponding shoe here is the sketch of another shoe and here
 is the corresponding shoe and so on and from that you could learn a model that translates from one
 domain to the other because you have corresponding image level annotations which image corresponds to
 which so basically which element of the domain a corresponds to which element in the domain b
 then the next stage of this was when you only need set level annotations and that's sort of what we
 looked at if you had supervised labels for domains so what you'll say is that there are three domains
 a b and c and actually let's let's forget c for a moment and just deal with a and b
 to make it equivalent to the thing on the left now i just know that these things are instances
 of class a and these things are instances of class b yet i i don't there's no correspondence
 right there is no this corresponds to this or or something like this so image to image translation
 is now possible between domains when i just have domain level labels but this is still expensive
 collecting these labels you know it's is like collecting labels for a supervised data so
 collecting labels for a supervised data set a human needs to look at each image and then conclude
 what sort of domain it is their paper introduces the following where you do not have domains anymore
 you simply have a data set x now this data set your hypothesis is that there are still going to be
 domains in the data set they can i guess they can be overlapping or not but there are still going to
 be domains you just don't know what they are so in this case um i guess you could differentiate
 these people into many many different ways but um in essence you're going to assume that there is
 some kind of a domain structure you just don't know what it is but if you knew what it was then
 you could simply apply methods from here to the data set and you'd be done now their paper shows
 that if you apply something like a self-clustering approach and we've seen these approaches before in
 the paper about learning to classify images without labels if you have techniques like this you can do
 like a self-clustering approach on this data set x right here and then you could learn your image
 to image translation yet this paper shows that if you do that the quality is not as good as if you
 do both things jointly so what this paper does is it jointly learns to cluster let's say to
 self-label the images and to make the to do this image to image translation and by doing the tasks
 jointly they help each other perform better okay that's a general overview so how do they do this
 they have three different parts to their model there is the encoder or they call this the guiding
 network there is the generator and there is the discriminator so the generator and the discriminator
 they are fairly standard GAN generators and discriminators so general adversarial network
 but they have like a bit of some sort of twists so you can already see from the design from the
 drawings right here the discriminator is probably the easiest the discriminator gets an image right
 here it doesn't have to be a generated it is a either a generated image or a real image and it
 needs to decide you can see right here this means that the input domain is a vector or an image in
 this case and the output is a number it needs to decide if it's real or fake now in fact it's not
 as easy because you can see there are these multiple heads right here so this whole thing
 as I said is built on this kind of pseudo clustering approach there is this pseudo label
 that comes out of the left side we're going to look at that in a second but in essence you assume
 that there are multiple classes multiple domains in the data set and the discriminator here has one
 classification head for each of those classes so from somewhere outside it will get the information
 oh this is now supposed to be one of those ginger cats right as opposed to one of those black and
 white cats or one of the brown haired cats no it's one of the ginger cats and then there is a special
 head on top of the classifier that only classifies fake from real ginger cats okay which is a
 different classifier from the other domains so the discriminator it's sort of a conditional
 discriminator conditioned on a label okay from the discriminators point of view it's simply a
 label conditioned discriminator discriminating between real and false and I think that's yeah
 how you train the discriminator is you would give an image and you would let this encoder here this
 guiding network label the image and how we come up with this label again that we'll look in a second
 but this just gives a label and then you'd for that particular label you'd classify the image
 into real or false now the fact that there is this shared part right here of course is
 you could also think of having one discriminator per class but the shared part now gives you some
 shared features and so on but it's not necessary it's not the the point is that there is a
 discriminator per class it's class conditional okay so what about the generator I think is I
 guess is the most complex um what about this this encoding network right here it's e for encoder I
 guess but they also call it the guiding network so what this does is this is what's this supposed to
 do is it'll take an image any image and it will output two things one is a label and one is a
 style code so the label is supposed to be a number between zero and da da da da da k minus one so
 k minus one so that's supposed to be a class label and how do you know how many classes there are if
 there are no labels you just guess and your best bet is to slightly over guess so if you expect
 there to be between 10 and 15 classes maybe put k to 20 okay you don't want to under guess but you
 can over guess but not by too much of course so you have to have this this estimation of how many
 classes but then this e it simply comes up with a class label and it also comes up with the style
 code now these two things are going to go then different pathways in this in this network the
 label is directly going to the discriminator right the generator does not see the label
 okay the style code does not go to the discriminator but goes to the generator all right so the two
 inputs from the encoder they one goes to the discriminator which is the label and one goes
 to the generator which is the style now the generator lastly it takes a source image and
 it takes this style code right here now the style code is encapsulating as we said the style of the
 reference image so the style is supposed to be whatever whatever whatever makes this domain
 of images the same so the style the way we're going to train this is that the style is going
 to describe somehow all the images that are from this label the style is going to describe whatever
 the style is it's very hard to it's very hard to explain if we look at the loss it becomes clearer
 why the things are how they are so it takes the style code and it takes the source image and it
 combines them and its task is to output this generated image as you can see in this example
 the generated image is basically this cat but with the style of the reference image and it outputs an
 image and the discriminator of course then is tasked with differentiating whether that image
 is real or fake for the given label over here okay so this is the entire thing and you all train
 this jointly so you jointly train the encoder to produce these class labels and the styles you
 train the generator to take in the styles and the source images and output the generated image to
 fool the discriminator and the discriminator at the same time is trained to differentiate between
 real and fake images based on the label that the encoder gives very very convoluted and complicated
 but there are a few things that make it easier first of all as you can see here the pseudo label
 is detached is argmaxed and detached so the pseudo label really is a number and there is no gradient
 back propagation along this line okay that makes that makes it a lot easier so the so what we first
 need is we need a way to train the encoder to come up with suitable class labels even though it
 doesn't get any back propagation signal into that part of its network so that's where we start with
 the loss functions the way we're going to do this is we're going to take the following approach
 we're going to take an image and we're going to take a randomly augmented version so for example
 a random crop or a horizontal flip and so on so the now we bring in ideas from self-supervision
 and again if you watch the video on learning to classify images without labels this is one of
 their main staples these self-supervised approaches really tend to learn representations that allow
 you to self cluster now in that paper they go further and they do this nearest neighbor thing
 in this paper they just do sort of the first step of this self clustering which i guess makes it such
 that you could potentially improve this paper by applying the other paper but who knows so
 we're going to take an image and we're going to augment it okay so that means we're going to like
 random crop it or in change its luminance or whatnot so we have two versions of the same image
 and what we want to maximize we want to maximize the mutual information between not between the
 images themselves but p is going to be this output of the encoder so x goes into the encoder and the
 encoder outputs the style and the class label and the class label here so p is going to be the class
 distribution all right so this is going to be like a histogram or maybe the log it's it's already
 yes so it's going to be a histogram over classes from which we're going to sample the label c or
 l or whatnot y hat y but the p is the distribution over output classes so since we don't have a label
 we can't train the distribution like in a supervised way supervised way so what we'll have to say is we
 want to maximize the mutual information between the output distribution of the image and the
 output distribution of the augmented image now that entails the following two quantities
 there's the entropy of p and there's the conditional entropy of p given p augmented
 now first of all it means we want to maximize this the entropy of p and that's supposed to be over
 the entire data set so this is the entropy over the entire data set x what it means is that we want
 different x's so if there's x1 x2 x3 and so on we want those to have different distributions in labels
 okay so if if the entropy is really high of the distribution p that means that different images
 get assigned to different classes some something like this all right if this is low then that would
 mean all the images basically get assigned to the same class and that's not good we want our
 classifier since we don't have labels it's a it's basically a cluster we want our cluster to sort of
 fill the space of possible clusters with the images so that's the first thing we want to
 maximize the mutual information we need to maximize this entropy and then second we want
 second since this is a minus here we need to minimize the conditional entropy of p given p
 augmented what does that mean that means if we know the augmented version of an image its class
 labeling should be the same as the un-augmented version so that means that if i now take one of
 these x's to x1 augmented of the do a plus augmented right then that shouldn't really
 change its class label and this is what these so that should sort of keep the class labeling
 this is horrible but the idea here is that it's kind of a reverse thinking from supervised learning
 in supervised learning we have the label like this is class this is class five okay this image is
 class five and our thinking is this augmentation techniques if i random crop an image or if i
 change its colorization a little bit the class is not going to change right an airplane with a
 in front of a blue sky is still an airplane in front of a bit bluer sky so i assume that it'll
 still have the same label here i don't have the label but what i can require is to say whatever
 you output for the image it should be the same for the augmented image so these two objectives
 are enough to give you sort of a rough clustering of the output space maximize the entropy minimize
 the conditional entropy between two versions of the same image okay that's how we train this
 pseudo labeling approach so now we have a we have a model that can give a label to each image very
 cool so how do we train the other parts now there are additional um additional losses here so
 i'm not sure yeah we'll go over it so this style part is also has to be trained right this encoder
 outputs a labeling we got that covered and it outputs a style part now the style part if you
 can see from the graphic it actually goes into let me erase some of that stuff here the style part
 actually is down here and it feeds into the generator and luckily they write detach here
 and since they don't write detach anywhere here that means that we do get gradient back propagation
 from the generator to the style code so that means our our encoder here is trained to help the
 generator with its task of fooling the discriminator okay but um first of all we're going to forget
 about that for now what we're going to do is simply look at a loss that they impose on the style
 they wouldn't they don't have to impose that loss but they have an additional loss on the style codes
 for the encoder in addition to the fact that there is gradient back propagating from g so the second
 loss we're going to look at is this style loss the style loss is almost the same so the style loss is
 a contrastive loss so what you want to do is if you have your data set you have your data set of
 images and you you know take images out and you train your network on and you train then take the
 next image or you take batches of image you take train and so on like this right and now you have
 this image what you want to do for this to work is you want to build up sort of a queue of images
 that you have already looked at like these images these are going and the queue can be let's say
 10 long and you would always throw out the oldest and and in queue and newest so when you're done
 with this image right here you'll put it into the queue you load your next image and so on so now
 what does this mean you now always have a queue of other images and it's not important what they
 are as long as they are others right because now we're going to compare ourselves with others and
 this is this contrastive loss right here so this style loss is going to be a contrastive loss
 between this and this now the bottom part this here these are the others these are the other images
 and what are the individual quantities so s is the style code of the image you're considering
 right now s plus you could have already guessed it is the style code of the augmented image right
 so we had our image x let's go again with x1 x2 x3 are different images so we put x1 through the
 encoder that gives us s the style it also gives us the class label but now we care about this head
 that gives us the style code and we augment x1 to be x1 plus and we go we put that through the
 encoder that gives us s plus and now we also put all of these other images remember these are the
 images that we've looked at previously but the only real importance is that there are other images
 we put those through here and they get the s minus i in this case three and two so now what will
 require is that the s the style code of our image is closer to the style code of its augmented
 version so the same principle again we want we'll say that you know these augmentations they don't
 really change anything about the style now this argument is a bit more wonky but if you think of
 you know random crops and random flips don't really change anything about the fur color or so
 of a of a cat so we want those two to be closer together than s is to any of these other images
 okay so this is a contrast of loss where you pull together two things that you think should be close
 and you push apart things that you think should be far away from each other so this style loss
 basically guarantees that you have a distinct style for each image that is robust to the kind
 of transformations that you do under augmentation okay specifically this style loss doesn't care
 about the domain right this is for each image you don't know if these other images are from
 the same domain or from different domains and that's why the style is basically individual
 to the image but as we as we're going to see the style does capture something of the domain as well
 but this loss right here is supposed to be each image has a style right so this is the style code
 of x this n plus one way classification enables e to utilize not only the similarity of the
 positive pair but also the dissimilarity of the negative pairs where the negative style codes are
 stored into a queue using previously sampled images we observe that adding this objective
 significantly improves unsupervised classification accuracy in animal faces from this to that
 compared to the previous over clustering approach okay so we have two outputs now and now
 we go to the adversarial loss so the question is how do we train the generator and the discriminator
 and the discriminator so they have three different losses for the generator and the discriminator and
 the most important one of course is this adversarial loss right here so the discriminator simply tries
 to distinguish is an image real or fake conditioned on a class so in case of a real image and that's
 this line right here it tries to distinguish is this real or fake based on y and y is x fed to the
 encoder and the encoder gives you a label all right that's and the label selects the head of the
 discriminator at the same time that the discriminator is trying to distinguish real from fake so these
 two lines the generator is trying to fool the discriminator so the upper if you've never seen
 a GAN loss the upper part here that's the real data and the bottom part here is the fake data
 now at the same time the discriminator is trying to distinguish real from fake and the generator
 is trying to make the discrim fool the discriminator so both are of the generator and the discriminator
 are actually using that loss but the sign in front of it is different okay and since the generator
 is not involved in the top line you can usually leave that away because there is no backprop path
 through that and there is no backprop backprop path here because we detach the graph right here so
 there is no gradient signal going to the encoder so this bottom line what does it mean the generator
 will take in an image and the style now s tilde comes from x tilde it's x tilde going through the
 encoder giving you s tilde so this is the reference image right this is you want these this style
 right here this is the reference image and x is the source image so the generator is supposed to
 take the source image and basically apply the style from the reference image and generate
 x i don't even know how to call this x not tilde whatever x fake xf and that's supposed to fool
 the discriminator now the question is which discriminator right because you need a label for
 the discriminator the label is conditional with this discriminator is pretty easy because it's
 simply the label of this image now however as you can see the generator learns to translate x to the
 target domain while reflecting the style code s tilde so y tilde is going to be the label
 that comes out of this x so this encoder right here is also going to give us y tilde and that's
 going to go here all right so recap what we want to put into the discriminator is one time a real
 image like we do up up here and we get its label from the encoder the encoder gets us a label for
 each image very cool we'll also take the same image put it through the generator task the
 generator with transferring the style of another image from here onto it we get the style from the
 encoder and then the generator is supposed to make an image and we feed that to the discriminator
 and the discriminator discriminates assuming it comes from class y tilde now you see right here
 the generator never has access to y tilde okay so the generator is kind of at a disadvantage here
 the discriminator gets told what kind of image it is in terms of class while the generator
 because it needs to fool the discriminator it needs to come up with an image of that class
 but it has no idea of the class it only has the style code so it is forced to learn to sort of
 it is forced to learn to map a style to associate a style with a particular class and that's what
 with a particular class and that's how you get the domain into the style that's why the style can
 capture something like fur color of the different cat breeds because the generator is forced to take
 the style that the encoder gives and map it to an image of the class y tilde that it also the
 encoder gives but doesn't tell to the generator okay and in fact there is a more path because you
 now back propagate the loss to the encoder which means that the encoder will even help the generator
 it will help the generator make style codes that are very class specific now you can maybe think
 why why wouldn't you just have one output why doesn't the encoder simply output the label also
 as the style because that would be the easiest and the reason is because we have different losses
 on the style and the label
 okay otherwise that would be a valid tactic so that's cool that's the adversary loss that's the
 most important loss now there's also additional losses so they they do additional losses that
 they add on top for the generator they say in order to prevent degenerate situation where the
 generator ignores the style code and synthesizes a random image in the domain y or in the domain y
 tilde we impose a style contrastive loss to the generator so now there's still the danger that
 the degenerator simply produces a valid image right from the data set or even from the domain
 y tilde though i don't know how it would know why tilde or i've just not seen something
 i in my mind it doesn't get the y tilde
 but it could read it from the style but here the danger is to ignore the style i'm slightly
 i'm slightly confused by this part but maybe looking at the loss will will clear it out
 so they say we impose a style contrastive loss to the generator now this is almost the same
 is almost the same as we imposed on the encoder so the generator you can see there is a
 contrastive loss again where you want to be you want these things to be close and you want these
 things to be far apart so these s minuses these are going to be the ones from your the style codes
 of the images from your queue so these are just going to be other images here s tilde that's
 going to be the style that you get from your reference image so your reference image is going
 through the encoder and that's going to give you this right here now the question is what is s prime
 here because in the before we simply had s which was our source image our source image style
 now what is s prime here s prime is going to be it gets more complicated yes s prime is going to be
 whoops it's going to be the round trip to the encoder so it's going to be if i generate my image
 from the source image x and the style s tilde of the reference and then i ask my generate my
 encoder again what style does this have i get the s prime so it's kind of a round trip right so i
 i take i take this i ask the encoder what style is it that's s tilde right then i take s tilde
 go to the generator together with a source image x and that gives me like x fake and then i ask my
 generator again what style would you assign to the fake image i just produced and then the encoder
 will tell you i'll give it s fake or s prime in this case and then i compare that s prime with
 the one i gave before okay so it's sort of a round trip loss of my reference image
 all right so what does that do if i now and then i ask that s prime be close to s tilde so that
 means if i generate an image with the style of my reference image the upcoming image should better
 have the style of the reference image that's all it says so the style of the thing i generate
 given this style they should better be close and especially closer together than the style of the
 style with any other image in my queue it makes sense but it's kind of convoluted so you go with
 your out it's kind of a reconstruction loss except in style space all right and then the last thing
 is an actual image reconstruction loss so what you'll do is your generator will produce x
 uh sorry will produce an image from the source image and its own style right here that's important
 before we input s tilde here so this now is we input the source image and its own style so we
 go with x we go to the e and we put the style here and we tell the generator if i input the
 input the source image and its own style then what you give me back better be the source image itself
 right this is a consistency loss that tells the generator that basically it learns now the generator
 learns to the generator learns to map to recognize an image with its own style sort of because it
 doesn't know right it doesn't know that what's coming in here is the style of um it of the image
 x but now you teach it and i think before this loss you'd have a good chance that uh the styles
 would just be all over the place they would sort of be consistent but they will not be aligned
 and with this you force that the style of an image itself if you gent if you put that into
 the generator it will lead to that image itself okay that's it so this is a this is extremely
 convoluted right the discriminator is the easiest the discriminator is a class conditional discriminator
 that gets the label from some mechanism that decides on a label right okay that's the easiest
 the encoder has two parts the pseudo label which is over here which is trained completely unsupervised
 detached from everything else in a self clustering approach while the style part here is trained
 first of all in a contrastive way which makes sense and also in a back propagated way from the
 generator so the style generation mechanism tries to help the generator okay and that means it's
 going to leak some information about the label into the style because that helps the generator
 generator needs to if the generator knows what sort of class it's going to produce it's going to be
 better okay so you can count on that information being in there but also also because of all the
 other losses that the generator has and the contrastive loss on the style the style code is
 going to sort of describe the individual style of an image and but is also going to describe what
 the style of that class is because it technically needs to contain information about the class
 and that's why i think this works with the style because there is no inherent notion of like
 this is the pose of a cat or something like this
 yeah it still seems like a bit magic to me and then the generator is first of all trained to
 fool the discriminator given a source image and a style and you can fool the discriminator
 by producing an image that's so good it looks real and specifically it looks real in the class
 that the pseudo label has given right so in the class that the encoder has given to it so the
 generator must somehow come up with an image that's of that class and so it will it will be forced
 to interpret the style code in terms of that class label which makes the style code the style code
 and also we have these two additional losses which is the round trip loss to the style space
 so whatever the generator outputs you should be able to recover the style from it by putting it
 through the encoder again and then lastly there is a consistency loss where you say if i put an
 image into a source image and i input its own style again going through the encoder you should
 give me back the source image itself very complex and all of the generator loss is back propagated
 through to the encoder so this is the full loss as i said discriminator easy adversarial loss
 generator adversarial loss plus this style round trip consistency plus the own image round trip
 consistency encoder gets all of the generator loss all of it so all of this goes here so the
 encoder fully helps the generator and it is also trained with this mutual information and the
 style contrastive loss wow that's some losses wow that that's a lot of damage
 so they do different investigations into their model here and i don't even know if we've
 missed some of the pictures but ultimately what you can now do is you can do image to image
 translation either that's the cool thing you can have a reference image for one or what you can do
 is you can ask your discriminator what kind of domains are there sorry you can ask your
 encoder what kind of domains are there you've guessed the number of domains so it's maybe 10
 or in this case it's uh eight eight eight domains of cats and you can simply divide your data set
 into these eight domains right one two three four five and so on oh this is 10 okay i can't see
 anymore so 10 domains and then you can simply calculate for each image you calculate the style
 vector so the style the style and then you simply take the average one over the number in that in
 that domain you take the average style vector and that's going to be your target style so you can do
 image to image translation with a reference image or you can do image to image translation
 for an entire group of images for example all the images in a given domain and that's how they do
 these graphs right here now just quickly wait until my tablet decides to show me the paper again
 thank you all right they do a bunch of investigations into their wholly unholy mixture
 of losses especially the first concern is couldn't we just train the guiding network
 like by its own on its own and then after that train this gan thing right that's what we had
 at the very beginning we said there's this guiding network and it does the clustering and all
 and couldn't we just train this gan architecture on top of the frozen guiding network and their
 conclusion is no if we train everything together it works better so on the left you have whenever
 you train the guiding network by itself and what you're seeing here is the t-sne visualization
 t-sne is a a down like a non-linear visualization tool of style codes extracted by our guiding
 network the ground truth domains of all test images is represented in different colors so
 this is a data set that has labels but you don't you don't provide the labels to this algorithm
 the algorithm is completely unlabeled but for purposes of investigating we'll visualize the
 labels with colors and what you'll see here are the t-sne visualizations of the style codes so
 things that are close together they have similar style codes and the ideal case would be if things
 that are close together here have the same label and that means the style is sort of representative
 of the domain okay that's what we want we want the style to capture the domain of an image
 and ideally not the image itself too much now on the left you see that there is quite a bit of
 overlap between these quite a bit of wash between the style and the group and on the right if you
 jointly train the gan together with the guiding network you see that these classes of the style
 codes which have no reason to cluster are much more clustered and separated and they are separated
 much more along the lines of the ground truth classes okay so that's pretty cool now i would
 actually be interested in what happens if you do the separate training with the full pipeline of
 this learning to classify images without labels thing and their nearest neighbor thing because
 they've also shown that just purely this self-clustering doesn't work too well but if
 you then do the nearest neighbor thing on top then that improves the classification significantly
 so this could potentially help either the separate or the joint training right here
 and there might be a connection between the joint training and whatever they're doing in any case
 they also show that then these fid which is a quality metric for gans lower is better that the
 joint training goes way lower in the fid than the separate training okay that's that's the reason
 why they built this convoluted thing because it works way better and here they ablate they ablate
 some of the losses to investigate what's really going on and in this case
 t-sne visualization of the style space of our guiding network trained on this since this does
 not have ground truth domain labels each data point is colored with the guiding network's prediction
 so each color is whatever the guiding network says the classes and the dot is one style each
 is one style each dot is one style vector and they're projected down to two dimensions you can
 see pretty clearly that the individual classes the individual clusters of style vectors correspond
 to different labels of the guiding network which is to be expected but also since they overestimate
 the number of classes in this case you can see that the even though the class label is different
 the style network will group the very similar classes together you can see here these are both
 cheetahs and here are both lions so it'll group them together which is pretty cool and sort of
 verifies that it recognizes these these different things because you force the guiding network to
 make 10 classes but the style network is simply continuous so it's cool to see that the style
 network will make one cluster with styles even though it's different labels and here you can
 see different samples from these domains just to verify that the guiding network has actually
 learned to separate things i still find this pretty pretty magical too this is completely
 unsupervised and it sort of finds these clusters by itself all right they have a bunch of images
 here as i said this is no longer with one reference images image this is where you take the entire
 domain so you self-label with your guiding network and then you take the mean vector
 and that's going to be your target style vector and these are the source images that you transfer
 and you can see that you know it works pretty well so they always have like one adult animal
 and one child animal and i guess not or just two different ones here this is particularly cute
 though i have to show you this fox right here what's going on with that fox like someone help that
 fox yeah um so we're not at perfection yet as you can see but it's you know that that looks like a
 pretty pretty cool fox maybe okay where did it go maybe it slipped maybe it's an it's an offshoot
 of this one on the top left yeah who knows these data sets they have their way and um so this is
 sort of where you can see the limitations right here um that's not how a baby snow leopard looks
 you see the limitations here in that all of these animal faces they are still pretty aligned like
 they're fairly frontal not exactly but they're fairly frontal pictures um they're fairly
 standardized and so on so we're i don't think we're yet at the level where we can just do
 you know um fully image to image and you see it especially with faces because we as us humans
 as us humans are extremely good at you know seeing when there's something wrong with a face
 but it's still it's still pretty impressive what's possible and i think if the past is of any
 indication here is summer to winter that actually looks good if the past is of any indication then
 this technology will be pushed pretty hard and soon we'll be able to do this with a simple smartphone
 app or something like this so i invite you to check out the paper right here they have lots
 and lots and lots of examples and t-sneak plots and whatnot in their appendix they have the code
 online as far as i as i have seen and with that let me know what you think in the comments bye bye