SinGAN: Learning a Generative Model from a Single Natural Image
ml ai machine learning artificial ingelligence gan generative image processing deep learning image editing deep dream style transfer convolutional neural networks generative adversarial networks photoshop
With just a single image as an input, this algorithm learns a generative model that matches the input image's patch distribution at multiple scales and resolutions. This enables sampling of extremely realistic looking variations on the original image and much more.

Abstract:
We introduce SinGAN, an unconditional generative model that can be learned from a single natural image. Our model is trained to capture the internal distribution of patches within the image, and is then able to generate high quality, diverse samples that carry the same visual content as the image. SinGAN contains a pyramid of fully convolutional GANs, each responsible for learning the patch distribution at a different scale of the image. This allows generating new samples of arbitrary size and aspect ratio, that have significant variability, yet maintain both the global structure and the fine textures of the training image. In contrast to previous single image GAN schemes, our approach is not limited to texture images, and is not conditional (i.e. it generates samples from noise). User studies confirm that the generated samples are commonly confused to be real images. We illustrate the utility of SinGAN in a wide range of image manipulation tasks.

Authors: Tamar Rott Shaham, Tali Dekel, Tomer Michaeli

https://arxiv.org/abs/1905.01164
https://github.com/tamarott/SinGAN

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we'll look at SINGAN, Learning a Generative Model from a Single Natural Image by Tamar Rott-Schaum, Tali Dekal and Tomer Mikhaili. So this paper, as it says, it's dealing with learning a generative model from just one image. And this kind of needs to be stressed because most generative models, even if they produce single image samples, they're kind of trained on a large image database beforehand to kind of learn what an image is. But this algorithm really starts out clean-slate, right? The algorithm starts out with nothing and then you give it this one single training image. And from that it can then generate all of these things, without ever having seen any other images during training. And the second row is simply a second example where you start clean-slate, input this image and then produce these. And you can see there's quite a bit of variety in the samples you produce from this image. So basically the task is, if you're just given one image, learn something about the distribution. And this paper specifically deals with patch distributions at different scales. So this could be learn about the distribution of these grass to sky here. So learn about the individual birds and so on. And then at lower scales learn about how the border of this grass looks. So the generative model learns that there's always kind of grass at the bottom, where there's just one image at the largest scale. But then at lower scales sometimes the border looks like a sharp corner and sometimes the border is relatively flat, like here. So it can vary up those things and it can make the border different. Also the birds, it kind of learns how the individual birds look and how they're distributed and therefore it can change that. You see there's quite a bit of variety here. You can also change the aspect ratio and you can actually do much more, much weirder things with it. For example, here are some examples of applications. First there is paint to image. So these are different tasks here. So the top row is always the training image. This is the single image you give the algorithm. And then you have a row of input and then this is what the algorithm outputs. So in paint to image you input a training image and you input a, you can do this in MS Paint or something, kind of the way you want the image to look. So what you want the algorithm to do is take the style of this image and put it into the form of that image and it produces this. Looks pretty good. In editing you can tell the algorithm, alright I want this, I want this tower to go lower down, right? I want this house to be more wide. So you'll get an image like this and you can see there are clear kind of contours here and here that are not nice and also the house is, you know, pixel stretched and so on. So this algorithm, this generative algorithm, can produce this image from it which looks much better here around the borders and kind of fills in missing windows to match of course the patch statistics that it sees in this top image, right? You always have to think that all this algorithm sees is the topmost image to learn from. Harmonization is a task where you have an input image and then you like copy paste some object in it and what it does is it will kind of adjust the patch statistics of that object to the surrounding image. And super resolution, finally, finally we get what every single action movie, just the NSA, can do. It's like, ah here is the security camera footage. Zoom in, enhance. Yeah, so I doubt that, you know, hidden number plates here, pixel-ish number plates, all of a sudden can become readable and identifiable but still this is very cool. And lastly you can do animation from this, as you can guess, I guess. It's not a movie. All right, let's look at how they do all of this kind of stuff. All of this is the same model that can be tasked to do these different things through various probing. At its essence it's this multi-scale GAN and the GAN is trained to have a series of generators and a series of discriminators and you always train them one by one. So first you train the lowest resolution and then you keep it fixed and then train the next resolution and so on until you're at the highest resolution. So in each layer, so at the bottom layer, we simply feed in, we simply feed in noise to a generator of a GAN and the generator generates an image. Now you take this image and you take a down sampled version of your training image. Remember you just have one training image. You take a down sampled version of that and you let the discriminator decide which one is real, which one's fake and you train the generator to fool the discriminator as much as possible. Now if you were to do this with the entire image, of course the generator would simply learn to reproduce the original image. So that's no good. So what this paper does more is that the discriminator actually doesn't work on the entire image but just on patches of the image. And that's so that they basically can't memorize the entire image. So the discriminator will pick these patches, these overlapping patches basically. You can imagine it's something like this overlapping patches and it will try to decide for each one is this patch real or is this patch fake? So the generator produces the entire image. This is what the generator produces the entire image but the discriminator can only see the image in patches, in overlapping patches. And that's what makes this paper kind of work. Otherwise they would just remember the single training image because you only have one training image. You kind of need some variety. This is at the lowest scale. Remember you input the noise and the lowest scale in this example is for example 25 by 25 pixel. You scale down your original image here also to 25 by 25 and then you let the discriminator decide. So once you've trained this generator to make very good 25 by 25 pixel images, that in this patch way fool the discriminator. You keep it fixed. For the next stage what you want to do is you always want to go through this layer first. So forget this discriminator now. We've trained this stage. Keep this generator fixed. Input noise, output, whatever the generator produces. Then take this upscale it. For example multiply each side by 2 to 50 by 50 pixels. Input this together with some new noise into the next stage generator. And then the same as before. This generator produces an image. You scale down your original image. You scale it down to now 50 by 50 pixels and you let the discriminator decide again in patches. Since the discriminator patches are always the same size but we scale down the image less and less, the effective patch size of the discriminator becomes much lower. Now this discriminator only sees the image in patches like so. Also the generated image that comes in here. It also sees in these patches and tries to decide are these patches from real or from fake images. You can see that the lowest layer here, this layer, is trained to kind of get the coarse-grained structure of the image. The discriminator will kind of see very large patches. So the generator must match the kind of large-scale structure. These patches won't be very very high resolution because we downscaled the image, but they will be large across the image. So the generator must match the coarse low resolution stuff in the image. But as you go up the layers, up and up the layers, your discriminator sees less and less of the picture at once. It sees less and less of the picture at once. So this discriminator here in the topmost layer can only concentrate on very small patches and therefore this generator will only have to produce things that look real at a very very small scale. So in essence you have this series of generators trained that each one is tasked with basically modeling details at a finer and finer scale until you come to the last final scale. But then each input of each one is the output of the last one. So basically you take whatever the last one has produced and the last one is really good at doing coarser grain things and you add to it your details of this level. And this will in the end give you a very realistic image that matches at every level of resolution, matches the kind of statistics, the patch statistics of this real image. So that's the whole point of this thing. To have this series of generators one after the other, each one adds their own details at its own scale. And this works super well apparently. So each generator is just built like this. It takes some noise and the image of the lower scale, it adds them, sorry for these artifacts, it puts it through five convolutional layers and then simply combines it with the input. And this will produce this image at this scale. That's each layer, it's just five conv layers. And since they're fully convolutional you can actually change the aspect ratio at inference time, you can change the resolution and so on. It seems pretty neat. Of course from experience I can tell you that this probably didn't work at the first try and there is a lot of work even though it seems pretty easy. Keep that in mind. So for training this there are actually two different losses. First of all you have what's called the adversarial loss. And the adversarial loss is your classic GAN loss, where the generator tries to fool the discriminator and the discriminator tries to catch the generator. But then also you have a reconstruction loss. And the reconstruction loss specifically deals at each layer. At each layer you train the generator to reconstruct the original image when you put in a zero noise, except at the lowest layer. But essentially what you want to do is you want to say well when I don't input any noise then please reconstruct the original image. And that seems to be important for the setup to include this noise so that the generative model is basically able to reconstruct the original image as a whole. So these two losses are combined to form the training objective. And again this is not trained on data set. It is trained on a single image. And the productions are pretty cool. So again here are more samples from just the single training images at the left side. And then you have random samples from the single image. You can do things like super resolution, where this picture has been super resoluted to that picture. And I like that they investigate the effects of kind of their setup. So they ask okay what happens if we just have basically two different scales in this scaling setup. Then you see the kind of patch statistics will be very very fine-grained and it won't match any sort of coarse-grained structure. If you have very many scales, the more scales you have better basically. The more different scales you capture. Even more interesting is what if, so at this layer where we have G, G, G, you scale up, scale up, scale up and so on. What you could do is you could not start here, but you say okay scrap this layer. What we actually do is we take the original image and we scale it down and we input that into here instead of inputting the output from the lower layer. So basically you start at let's say the ground truth and that effect is shown here. So if you start at the lowest layer in this particular example you see that sometimes there are weird things. But what you can do is start at a let's say an intermediate layer with the original image and then the variety you get because you kind of keep the coarse-grained structure the same. The variety you get will only be in the right we said there are different layers and but you now eliminate these two layers and replace them with your original image at the scale. So the variety you get will only be from these finer grained lower resolution patches things. So for example as you can see here the zebra samples now differ in how exactly their stripes are manifested. This seems pretty cool. So you have kind of a handle on how fine grained you want your details or your changes to be. They do a bunch of more experiments where you can do a lot of kind of playful things with this thing. There is code available for example here you can see editing again as an example where they compare also with content aware move which I think is implemented in Photoshop and paint harmonization as we saw before. So all of these kind of things are very playful are very cool and I encourage you to check out this paper and the code it seems pretty easy. I have a remark though this again is only learned from a single image and that's the kind of cool part but it should be possible to combine this with some sort of approach over a data set. Like if I have a model that is really good at a single image right producing something that looks like a single image I should be able to combine it with a model that has been learned from a database. It's kind of like a Bayesian approach where you say okay I want to produce the best image so I want to maximize the probability of this image given the other image. But then you can also say aha but that's kind of proportional to j given i times p of i right you know Bayes rule and it seems that this paper is dealing mostly with kind of maximizing the likelihood of the output while you could probably combine it with some sort of prior over natural images and come up with an even better model. Of course then you'd need an actual database of images and training procedure and you need a way to combine these two models. So maybe that's a bit of a challenge. Anyway cool paper check it out bye bye.
 Hi there! Today we'll look at SINGAN, Learning a Generative Model from a Single
 Natural Image by Tamar Rott-Schaum, Tali Dekal and Tomer Mikhaili. So this paper,
 as it says, it's dealing with learning a generative model from just one image. And
 this kind of needs to be stressed because most generative models, even if
 they produce single image samples, they're kind of trained on a large image
 database beforehand to kind of learn what an image is. But this
 algorithm really starts out clean-slate, right? The algorithm starts out with nothing
 and then you give it this one single training image. And from that it can then
 generate all of these things, without ever having seen any other images
 during training. And the second row is simply a second example where you start
 clean-slate, input this image and then produce these. And you can see there's
 quite a bit of variety in the samples you produce from this image. So basically
 the task is, if you're just given one image, learn something about the
 distribution. And this paper specifically deals with patch distributions at
 different scales. So this could be learn about the distribution of these
 grass to sky here. So learn about the individual birds and so on. And then at
 lower scales learn about how the border of this grass looks. So the
 generative model learns that there's always kind of grass at the
 bottom, where there's just one image at the largest scale. But then at lower
 scales sometimes the border looks like a sharp corner and sometimes the
 border is relatively flat, like here. So it can vary up those things and it can
 make the border different. Also the birds, it kind of learns how
 the individual birds look and how they're distributed and therefore it
 can change that. You see there's quite a bit of variety here. You can also change
 the aspect ratio and you can actually do much more, much weirder things with it.
 For example, here are some examples of applications. First there is paint to
 image. So these are different tasks here. So the top row is always the training
 image. This is the single image you give the algorithm. And then you have a row of
 input and then this is what the algorithm outputs. So in paint to image
 you input a training image and you input a, you can do this in MS Paint or
 something, kind of the way you want the image to look. So what you want
 the algorithm to do is take the style of this
 image and put it into the form of that image and it produces this. Looks
 pretty good. In editing you can tell the algorithm, alright I want this, I want
 this tower to go lower down, right? I want this house to be more wide. So you'll get
 an image like this and you can see there are clear kind of contours here and here
 that are not nice and also the house is, you know, pixel stretched and so on. So
 this algorithm, this generative algorithm, can produce this image from it
 which looks much better here around the borders and kind of fills in missing
 windows to match of course the patch statistics that it sees in this top
 image, right? You always have to think that all this algorithm sees is the
 topmost image to learn from. Harmonization is a task where you have
 an input image and then you like copy paste some object in it and what it does
 is it will kind of adjust the patch statistics of that object to the
 surrounding image. And super resolution, finally, finally we get what every single
 action movie, just the NSA, can do. It's like, ah here is the security camera
 footage. Zoom in, enhance. Yeah, so I doubt that, you know, hidden
 number plates here, pixel-ish number plates, all of a sudden can become
 readable and identifiable but still this is very cool. And lastly you can do
 animation from this, as you can guess, I guess. It's not a movie.
 All right, let's look at how they do all of this kind of stuff. All of this is the
 same model that can be tasked to do these different things through various
 probing. At its essence it's this multi-scale GAN and the GAN is trained
 to have a series of generators and a series of discriminators and you always
 train them one by one. So first you train the lowest resolution and then you keep
 it fixed and then train the next resolution and so on until you're at
 the highest resolution. So in each layer, so at the bottom layer, we simply feed in,
 we simply feed in noise to a generator of a GAN and the generator generates
 an image. Now you take this image and you take a down sampled version of
 your training image. Remember you just have one training image. You take a
 down sampled version of that and you let the discriminator decide which one is
 real, which one's fake and you train the generator to fool the discriminator as
 much as possible. Now if you were to do this with the entire image, of course the
 generator would simply learn to reproduce the original image. So that's
 no good. So what this paper does more is that the discriminator
 actually doesn't work on the entire image but just on patches of the image.
 And that's so that they basically can't memorize the
 entire image. So the discriminator will pick these patches, these overlapping
 patches basically. You can imagine it's something like this overlapping patches
 and it will try to decide for each one is this patch real or is this patch fake?
 So the generator produces the entire image. This is what the
 generator produces the entire image but the discriminator can only see the image
 in patches, in overlapping patches. And that's what makes this paper kind of
 work. Otherwise they would just remember the single training image
 because you only have one training image. You kind of need some variety.
 This is at the lowest scale. Remember you input the noise and the lowest
 scale in this example is for example 25 by 25 pixel. You scale down
 your original image here also to 25 by 25 and then you let the discriminator
 decide. So once you've trained this generator to make very good
 25 by 25 pixel images, that in this patch way fool the discriminator. You keep
 it fixed. For the next stage what you want to do is you always want to go
 through this layer first. So forget this discriminator now. We've trained
 this stage. Keep this generator fixed. Input noise, output, whatever the
 generator produces. Then take this upscale it. For example multiply each
 side by 2 to 50 by 50 pixels. Input this together with some new noise into the
 next stage generator. And then the same as before. This generator produces an
 image. You scale down your original image. You scale it down to now 50 by 50
 pixels and you let the discriminator decide again in patches. Since the
 discriminator patches are always the same size but we scale down the image
 less and less, the effective patch size of the discriminator becomes much lower.
 Now this discriminator only sees the image in patches like so. Also the
 generated image that comes in here. It also sees in these
 patches and tries to decide are these patches from real or from fake images.
 You can see that the lowest layer here, this layer, is trained to kind of get the
 coarse-grained structure of the image. The discriminator will
 kind of see very large patches. So the generator must match the kind of
 large-scale structure. These patches won't be very very high resolution
 because we downscaled the image, but they will be large across the image. So the
 generator must match the coarse low resolution stuff in the image. But as you
 go up the layers, up and up the layers, your discriminator sees less and less of
 the picture at once. It sees less and less of the picture at once.
 So this discriminator here in the topmost layer can only concentrate on
 very small patches and therefore this generator will only have to produce
 things that look real at a very very small scale. So in essence you have
 this series of generators trained that each one is tasked with basically
 modeling details at a finer and finer scale until you come to the last final
 scale. But then each input of each one is the output of the last one. So
 basically you take whatever the last one has produced and the last one is really
 good at doing coarser grain things and you add to it your details of this level.
 And this will in the end give you a very realistic image that matches at every
 level of resolution, matches the kind of statistics, the patch statistics of this
 real image. So that's the whole point of this thing. To have
 this series of generators one after the other, each one adds their own details
 at its own scale. And this works super well apparently. So each generator is
 just built like this. It takes some noise and the image of the lower
 scale, it adds them, sorry for these artifacts, it puts it through five
 convolutional layers and then simply combines it with the input. And this
 will produce this image at this scale. That's each layer, it's just five
 conv layers. And since they're fully convolutional you can actually change
 the aspect ratio at inference time, you can change the resolution and so on.
 It seems pretty neat. Of course from experience I can tell you that this
 probably didn't work at the first try and there is a lot of work even though
 it seems pretty easy. Keep that in mind. So for training this there are
 actually two different losses. First of all you have what's called the
 adversarial loss. And the adversarial loss is your classic GAN loss, where
 the generator tries to fool the discriminator and the
 discriminator tries to catch the generator. But then also you have a
 reconstruction loss. And the reconstruction loss specifically deals
 at each layer. At each layer you train the generator to reconstruct the
 original image when you put in a zero noise, except at the lowest layer. But
 essentially what you want to do is you want to say well when I don't input
 any noise then please reconstruct the original image. And that seems to be
 important for the setup to include this noise so that the
 generative model is basically able to reconstruct the original image as a whole.
 So these two losses are combined to form the training objective. And
 again this is not trained on data set. It is trained on a single image.
 And the productions are pretty cool. So again here are more samples from just
 the single training images at the left side. And then you have random samples
 from the single image. You can do things like super resolution, where this picture
 has been super resoluted to that picture. And I like that they investigate the
 effects of kind of their setup. So they ask okay what happens if we just have
 basically two different scales in this scaling setup. Then you see
 the kind of patch statistics will be very very fine-grained and it won't match
 any sort of coarse-grained structure. If you have very many scales, the
 more scales you have better basically. The more different scales you capture.
 Even more interesting is what if, so at this layer where we have G, G, G,
 you scale up, scale up, scale up and so on. What you could do is you could not
 start here, but you say okay scrap this layer. What we actually do is we
 take the original image and we scale it down and we input that into here instead
 of inputting the output from the lower layer. So basically you start at let's
 say the ground truth and that effect is shown here. So if you
 start at the lowest layer in this particular example you see that
 sometimes there are weird things. But what you can do is start at a let's say
 an intermediate layer with the original image and then the variety you get
 because you kind of keep the coarse-grained structure the same. The
 variety you get will only be in the right we said there are different
 layers and but you now eliminate these two layers and replace them with your
 original image at the scale. So the variety you get will only be from these
 finer grained lower resolution patches things. So for example as you can see
 here the zebra samples now differ in how exactly their stripes are manifested.
 This seems pretty cool. So you have kind of a handle on how fine
 grained you want your details or your changes to be. They do a bunch of
 more experiments where you can do a lot of kind of playful things with this
 thing. There is code available for example here you can see editing again
 as an example where they compare also with content aware move which I think is
 implemented in Photoshop and paint harmonization as we saw before. So all of
 these kind of things are very playful are very cool and I encourage you to
 check out this paper and the code it seems pretty easy. I have a remark though
 this again is only learned from a single image and that's the kind of
 cool part but it should be possible to combine this with some sort of approach
 over a data set. Like if I have a model that is really good at a single
 image right producing something that looks like a single image I should be
 able to combine it with a model that has been learned from a database.
 It's kind of like a Bayesian approach where you say okay I want to produce
 the best image so I want to maximize the probability of this image given the
 other image. But then you can also say aha but that's kind of
 proportional to j given i times p of i right you know Bayes rule and it seems
 that this paper is dealing mostly with kind of maximizing the likelihood of the
 output while you could probably combine it with some sort of prior over natural
 images and come up with an even better model. Of course then you'd need an
 actual database of images and training procedure and you need a way to combine
 these two models. So maybe that's a bit of a challenge. Anyway cool paper check
 it out bye bye.