Resolution-robust Large Mask Inpainting with Fourier Convolutions (w/ Author Interview)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper lama inpainting gan adversarial loss function fourier transform fft fast fourier transform fourier convolution fast fourier convolution fourier convolution layer global information generative model periodic strucutre best inpainting ai inpainting first author interview lama inpainting mask filling large mask inpainting remove from picture ai image editing
#lama #inpainting #deeplearning

At the end of the video is an interview with the paper authors!
LaMa is a system that is amazing at removing foreground objects from images, especially when those objects cover a large part of the image itself. LaMa is specifically trained to reconstruct large masked areas and includes global information throughout its forward propagation by using Fourier Convolutions in its layers. This makes it incredibly effective at reconstructing periodic structures with long-range consistency, compared to regular convolutions.

OUTLINE:
0:00 - Intro
0:45 - Sponsor: ClearML
3:30 - Inpainting Examples
5:05 - Live Demo
6:40 - Locality as a weakness of convolutions
10:30 - Using Fourier Transforms for global information
12:55 - Model architecture overview
14:35 - Fourier convolution layer
21:15 - Loss function
24:25 - Mask generation algorithm
25:40 - Experimental results
28:25 - Interview with the authors

Paper: https://arxiv.org/abs/2109.07161
Code: https://github.com/saic-mdal/lama
Online Demo: https://cleanup.pictures/

Sponsor: ClearML
https://clear.ml

Abstract:
Modern image inpainting systems, despite the significant progress, often struggle with large missing areas, complex geometric structures, and high-resolution images. We find that one of the main reasons for that is the lack of an effective receptive field in both the inpainting network and the loss function. To alleviate this issue, we propose a new method called large mask inpainting (LaMa). LaMa is based on i) a new inpainting network architecture that uses fast Fourier convolutions (FFCs), which have the image-wide receptive field; ii) a high receptive field perceptual loss; iii) large training masks, which unlocks the potential of the first two components. Our inpainting network improves the state-of-the-art across a range of datasets and achieves excellent performance even in challenging scenarios, e.g. completion of periodic structures. Our model generalizes surprisingly well to resolutions that are higher than those seen at train time, and achieves this at lower parameter&time costs than the competitive baselines. The code is available at \url{this https URL}.

Authors: Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, Victor Lempitsky

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there, today we're looking at resolution robust large mask in painting with Fourier convolutions also called LAMA by the Samsung AI Center, Samsung Research, EPFL and the Skolkovo Institute of Science and Technology. This is a special paper review because I'm only going to introduce the paper briefly, maybe 15-20 minutes or so and then we're going to talk to the first author of the paper and go a lot more in depth. So if you like, if you like conversations with first authors and the ability for me to ask dumb questions to them, then stay tuned for that. It's going to be in the second half of the video. For the first half though, I first want to demonstrate to you what this model can do. Hey there, this video is sponsored by ClearML. ClearML is an ML Ops stack that is fully open source. It can do experiment tracking, orchestration, deployment, model and features, stores and much more. The self-hosted tier is a first class citizen in ClearML. As I said, it's fully open source, you can look at it, you can audit it, you can extend it, you can run it on your servers. And if you ever come to the point where you need the extra features, you can totally upgrade anytime, they'll gladly take your money. They have a free tier in the cloud, which gets you going pretty far. Now we talked about experiment tracking last time ClearML with two lines of code will track any experiment that you do track the metrics, the outputs, the environments, the dependencies and make everything super duper reproducible. But this time I want to talk about a second part, which is the orchestration engine. So the orchestration engine is responsible for packaging up your experiments, including all dependencies, and then distributing them on your hardware. So that means you can simply submit an experiment to a particular queue and ClearML takes care of running this wherever it's needed. So this is super cool, because it means I can get going on my laptop, run a few experiments there. And as soon as I'm ready, boom, I ship it to the cloud. So here's an example, look at this experiment that has already been run, I got some output, but now I would like to do something different with it. So I click here, I say clone, I give it a meaningful name, like two. And now I've cloned this experiment. And this is kind of a draft experiment right now, it has no results yet. But what I can do, I can go into my configuration, into my hyper parameters, and I can change around the hyper parameters. So I wasn't really happy with the last experiment, I feel a bigger batch size might be needed. So from 128, let's go to 129. Now I'm pretty sure that's going to make all the difference right here. So I save and then I simply click on in queue, I submit it. And now ClearML simply takes care of running that experiment for me. As you might guess, you can have different queues, some for GPU load, some for long running tasks, some high priority, as you're used to from any scheduler. This can also be used in automated fashion, meaning that you can use this for automated hyper parameter search, and you can even do things such as scheduled or triggered tasks. For example, if you want to trigger a training run every day on new incoming data, that's totally doable. Now orchestration is just one part of ClearML. I've shown you experiment tracking last time. And there are many more features to their product. If this sounds interesting to you, if you're an open source fan, go check them out. And thanks so much to ClearML for sponsoring this video. Let's get into it. You can already see it a little bit in figure one right here, the model is able to take a picture, you draw a mask on it. So this is the blue area right here. And the model would auto complete the picture. So the model doesn't see the mask, the model simply sees what is unmasked, then the model is asked to complete that missing area. As you can see, it fills that area in, you know, very, very cleanly. And especially if you look right here, this irregular structure of these door holes, or whatever that is, is preserved even across very large areas. This is very, very cool. This is very difficult to do with these in painting systems. In fact, there is a project website right here, all the code is available. They give this in a little bit more of an animated flair. So you can really see the effect that these models are having. And it's pretty, pretty cool, especially take a look at these repeated structures that are often in the pictures. So these meshes or the lines right here, these tend to be extremely these tend to be especially difficult for in painting models, because in painting models are usually built on convolutional neural networks, and convolutions, notably take into account very local context. Whereas for these patterns, you need to take into account kind of a global context, that's exactly going to be the the message right here. There is an app, there are actually a bunch of apps based on this model. This is a third party app. So this is not by the author. But it is an app built from these models. There are also as I said, code is available. There's like a hugging face space, there is a collab by the author. But this particular app, let's just take a picture right here. It works best on natural images, of course, but we'll just take the channel logo right here. And we'll say we want to erase the pie sign right here. Look how cool that works. What about the paw? Okay, that that is that is kind of disturbing. How about the nose? No, no, no, I don't like that. But it should be able to Yeah, see, so it kind of completes lines, if you cross them out. So this should complete the table, but remove the leg, you can see it's fairly robust, even to use sort of miss specifying bunch of things. So here I draw over the headline, if you saw that, and it remained the head headline remains. So I removed this part, but I crossed into here a bit, you can see the line kind of remains. Now it's got a bit of hair. Yes, kill it with fire. In any case, this is available for you to use if you have more sensible pictures, I'm sure that that will work a little bit better, maybe. There are also different versions of the model. So keep that in mind. And they works also on different resolutions. That's why it's called resolution robust, large mask in painting, which is also very cool. So what is the core idea of this paper, the core idea is going to be these Fourier convolutions right here. And these Fourier convolutions are going to be enabling the model to take into account global context from the very beginning. What is the problem with a convolutional neural network? The problem usually is that in a convolution, if I have a picture, a convolution on a particular point will take into account its local neighborhood, right? And then I sort of slide this over the image right here. And that will give me my representation in the next layer, maybe that's going to be even of the same size. So for a given point right here, I will have a receptive field of the point in the original image, plus some neighborhood. Usually we work with three by three convolutions. So all I'm going to do really is I'm going to look one pixel to the top and one pixel to the bottom, one pixel to the top, one pixel to the left, and one pixel to the right. And that's about it. I'm not going to do any more looking around. So how does a convolutional neural network integrate information across the whole image? And the answer to that is by going for multiple layers. If I simply represent the picture as a set of pixels in one dimension, imagine that the one dimension here is the picture. And I'm going to need a bit more space for that. So as you can see in the first layer, from the first to the second layer, let's say we look at this particular point right here, it's going to be have a receptive field of three. So it's going to look at these pictures, sorry, at these pixels right here. In the next layer, if you can see that the same location is also having a receptive field of three right here. However, since for example, this particular pixel right here also had a receptive field of three, and this particular one also, as you can see, and from layer two on the total receptive field of that so that all the information inflow is going to be from a receptive field of five. Therefore, the more layers we have, the more of information, the more spatial information can be included for a given particular location in the output. But as I said, that takes a lot of layers that takes depth. And especially for these in painting applications, what you want is kind of global information. These masks right here, like these masks, they're pretty big for an in painting application. So they're pretty, pretty wide. And if you can imagine a convolutional layer that looks at a three by three pixel neighborhood, that might be something right here. You know, so you're going to have a whole lot of convolutional kernels that just see the masked pixels, they see nothing of the outside, they simply see a bunch of masked pixels for a whole bunch of layers, right layer two, layer three, until like layer four, there's like there's nothing, no information at all at this position about the outside world about the world beyond the mask. And even then, it's only like this area, we need to go many more layers before we get access to information that is way outside of here. And at that point, it may already be too late. So the Fourier convolutions, they solve that they have the ability at every single layer to look at a global context. And how are they doing this? It's not super expensive. In fact, they're doing this by using of course, Fourier transformations, a Fourier transformation will map a signal to its corresponding frequency domain signal, it is essentially a different way of representing a signal. So if you have a signal, let's say you have like a pure sine wave, you do a Fourier transformation of that entire thing, you can represent that as the components in the Fourier spectrum. And that would simply have like one component at the particular frequency at which this sine wave at which this sine wave is operating. That's the that's not the frequency, that's like one over the frequency right here. But in a more in a more general sense, a Fourier transform will decouple the spatial resolution and give it a transform it into frequency resolution. So if you have a Fourier spectrum, maybe you have a very complicated signal right here, a complicated signal that will give you also a complicated Fourier spectrum, like you have a lot of this, you have like negative this frequency, a lot of this frequency, not too much of this frequency, and so on. If you do a convolution in this domain, you simply convolve across neighbors of the signal. However, if you do a convolution in Fourier domain, you can see you convolve across frequencies, you can evolve across neighboring frequencies, which means that these three things represent three particular sine waves frequencies, maybe the lowest one is like a super long sine wave, the second one is like a bit of a faster sine wave, the third one is even faster sine wave. But what is important is that every single component in the Fourier spectrum represents information about the entirety of the signal. And that is exactly what we want. Whereas the regular convolution is localized in in pixel space, the Fourier convolution is going to be localized in frequency space, but global in pixel space. That is very cool. And of course, Fourier transforms are also one of the things that are extremely fast. It's essentially a linear algebra operation. And there are very fast implementations of discrete Fourier transforms called fast Fourier transforms. That's exactly what they do right here. The whole architecture is going to look like this. There is going to be the image the input image x there's going to be a mask during training that is produced by a mask generation algorithm x is then masked out and the model is tasked to predict the missing pixels that are hidden by the mask. As I said, the model has no access to what's below the mask, I guess that will that would be kind of pointless, right? Yeah. So what we do first, but also this is a fully convolutional architecture that makes it able to essentially transfer to different resolutions, which is another advantage here being a fully convolutional. So what we do is first we downscale a little bit as far as I can tell these images are something like 256 by 256 during training, or it works on crops of 256 by 256, somewhere in that range. But the cool thing is it can generalize to high definition images like 1920 by 1080 or something like this, the same network. So the train the network that's trained on this low, low, quote unquote, low resolution can generate can generalize to very, very high resolution, and it won't lose performance. But we'll see that in the experiments. So first there's down sampling, and then the model is just this is just nine layers. They also have a variant with 18 layers. But the base model is nine layers of this fast Fourier convolution residual block. As you can see, it has a residual connection right here, like a normal resnet, whereas a normal resnet would have two convolution layers right here, we opt for these fast Fourier convolutional layers. Now, they look a bit complicated, but essentially, what we do is we carry two different signals across the entire network, one signal contains local localized information. So one signal is going to operate in the original domain of pixel space and has all that those properties, so it looks at its neighbors and so on. And one signal is going to operate in in more of the global domain. And then in each layer, those two strands of information get to exchange information with each other. So the whole signal is represented as this block here with the two components. But it's essentially just we have like two strands of signal, and then every now and then they get to exchange a bit of information, right, one is the local, the local branch, and one is the global branch of information. So what do we do with the local branch, we have different operations right here. So we have a little conv layer that is in pixel space, actually, we have two of them, right, two conv layers. So we pass this the local signal, this is really just if you just consider this path right here through this one, then ignore ignore this here. If you just go here, this is just like a normal conv net, right, this path here gets information from this side here. It receives it and then there is an addition. So what is that, that is simply this global signal the global signal, also doing a localized convolution in pixel space. So far, there is nothing special if we were to just do this, this would be it would be pointless to have two strands of information, right. But the important thing is that the global strand comes to be in a very special way. So for that, we have to look what information arrives at the global branch right here, because that's the information that's going to be passed in here for the next layer. For that, we see from the local branch, there's a three by three convolution going out over here. So let me draw that in greenish over here. And that is going to be mixed with this global strand of information. And the global strand of information is going through this spectral transform block. The spectral transform block is essentially pretty easy. There is a there's a batch norm, sorry, a convolution batch norm relu block. This is a one by one convolution. This is simply simply simply a linear operator pixel wise, essentially, there's a batch norm, there's a relu for the nonlinearity. And then what we do is we do a fast Fourier transform in 2d. And at the end of the block, we're going to invert that. So fast Fourier transform to operate in Fourier space, and then invert the fast Fourier transform at the end. And inside of it, we're going to do a convolution batch norm relu block right here. So the convolution again, that's a one by one convolution, I believe, followed by batch and followed by relu. So actually even forget what I said about localized convolutions right here, if they just do one by one convolutions, they really operate just on the individual elements of the spectrum by itself, not even, they don't even they don't even consider localized, sorry, neighborhoods of frequencies, they just operate on the individual frequencies, one by one, which is is an option, like one by one convolutions are are a thing. So, you know, pretty cool. This by itself also has residual connection right here, I'm going to guess to make signal flow better or more more stable or some something like this, the observant people might object and say, hey, this thing right here actually outputs complex numbers. So this is in the space of complex numbers. So you'll get vectors with entries like a plus, plus IB. But what we do is simply we take those and we stack them. So we just make like vectors out of them, a and b. So if there is a bunch of numbers, it will just be like a one b one, a one b one, b one, a two, b two, and so on. And we just consider this to be a real vector of double dimensionality, or a real 2d signal of double the dimensionality as before. And that is how we do it. I mean, it's not it's not entirely correct, right. But the model in this way has access to all the relevant information, it can do what it wants with it. Yeah, it can it can learn that half of the dimensions correspond to two phases, or, or whatever, whatever the complex part of this is, it's been a while since since been a while since Fourier transforms. Okay, so these are the exactly so here, that's done, we have, sorry, go back up here to start it, there is first the real FFT, as you can see, that gets you to complex space, then there is complex to real, in which we transform the c channels into two c channels. But now we're in the real numbers. Then there is this value batch norm conv, which retains the signal. And there is real to complex where we go back into complex space. So from reals, 2d to c channels into complex, just c channels, and then we reverse the Fourier transform. And that is a Fourier convolution, as they define it. If we integrate, no, that is the spectral transform block right here, the Fourier transfer, the Fourier convolution is this entire construct right here, as you can see, the spectral transform information then flows in here is combined with some local information that really should be green. And that then goes into this global output and obviously will become the global input to the next layer. So that is how they fuse localized information with global information in every single layer. And that turns out to be pretty, pretty powerful. They do have other improvements right here. And it's it's crazy to see that just how much engineering and how many tricks go into these models to really get them to work. So they also stress that loss function is a really, really important topic right here, because you can't simply reconstruct the original image right here, if you simply tell the model to reconstruct the original image from here, it's going to be bad because if your mask is pretty big, pretty wide, there can be many possible fillings of the mask that makes sense. And since there are many possible ones, if you don't account, if you don't reward the model for getting one of the possible ones without punishing it that it didn't get all the other ones, the model is going to be very confused and is simply going to output the average of all the possible ones, which we don't want we want one of the possible ones. So what we do is we apply a perceptive loss, they call that a perceptive loss. And they explain that over here, what you do is you feed the image, the original image, the original image, this is the real one, and the fake one, and you can already see there's going to be like a discriminator later, right. But you feed them both through a pre trained neural network. And then you compare at intermediate points, or even like at the last latent layer, you compare the two feature maps. So depending on how this network is trained, if that outputs very perceptually salient features, you'll get like a nice loss that doesn't punish you for getting any pixels wrong. But that encourages you to get something that is perceptually similar to what was there in the original image. They also stress that it's really important on how you train this network right here. They suggest to make this network also include global context using either also Fourier convolutions or dilated convolutions. And here you can see that's essentially the formula that means we take the features from the original image and the features from the fake image, and we calculate their distance. And that's going to be the high receptive field perceptual loss. This is not the only thing they do. They also have, as you can see, an adversarial loss. There is also a regularizer on the gradients. So yeah, the final loss you're going to end up with is like a mix of all of these different losses. There's also a discriminator based perceptual loss. And this part right here is by itself, again, a conjunction of two losses. So rest assured, the loss architecture right here is very, very intricate. And I'm going to guess it's taken a lot of experimentation, not only by this paper, but by the whole field here to really come up with nice losses that make your outputs nice. Obviously, there's going to be a bunch of hyper parameters here to tune, which is always fun, but they seem to have done a pretty good job. The last thing they stress, which is important is how you generate masks during training. So during training, you can't just, you know, take your finger and draw on pictures. Like I did, you have to have some heuristic way of generating masks. And I'm not going to go into the detail of how they do it. You can see here compared to this is one of the one of the baselines. And this is one of their heuristics. They have a mix of these large masks and the box masks. So sorry, both are large, but one is called wide masks, which are kind of polygons that they round off the corners, I think, and box masks, which are sort of heuristically generated boxes right here, or stacks of these boxes. And that's, and they mix those two together in order to get the final masking for their images. You can see these are fairly large, like this one here covers more than more than half the image. So these are challenging, challenging tasks. But it is through training with such large masks that you get the models to really learn to fill in it consistently. So what you can see is that in their results, and we're not going to go into all the tape, like they have a lot of tables, a lot of ablations, but red essentially means that it's worse than their model, you can see almost all of the table is red, except some models in some of the benchmarks, for example, in the narrow masks, you will find situations where other models might outperform their model. But as soon as you go to like wide masks, it is no longer, it's no longer really a competition at all. Yeah, so their model seems to be really good. Those white masks, they do a lot of ablations where they switch out different, for example, different convolutions right here, they show what if we switch the Fourier by a dilated convolution, which is also a way to increase the receptive field rapidly or by regular convolution. And again, while there might be some improvement, sometime on narrow masks, as soon as you go to wide masks, the other models degrade pretty quickly, the dilated convolution actually holds up fairly well right here. But one disadvantage of that is that it's very hard to go to higher resolutions, because the higher resolution you go, the dilated convolutions that their receptive fields will also shrink, while the Fourier convolutions receptive fields will always remain essentially global. So here you have some comparison to baselines, you can see of course, they chose these pictures well with kind of the regular structure in the background. But check this out, like this is even this is even their model. But with regular convolutions, and even if they go deeper, doesn't really help. But like this, this is just insane, right? I get it, they pick this picture, but it is like is really good. And you can also see this building how it's completed over here with different methods, and then with their method. And the mask was, you know, fairly, fairly big, as you can see, also the bottom this the mask is huge. Yeah, here they show what happens if you go to higher resolution. So on this rather simpler problem, you can see that a lot of the models do well in the top row, if you just have the kind of a lower resolution. But if you go to really high resolution, a lot of the models struggle while the llama model here still does a big, a good job in their larger model seems to be even better. Yeah, again, lots of ablations, but I'm going to stop right here, and we'll go over to chatting with the first author about this. So I'll see you in a bit. Hello, everyone. I'm here with Roman Suvorov and Elizaveta Logacheva, the authors of the llama paper and llama system as well, I guess I think this is as much a paper as it is an engineering effort. And just because looking at the paper, it already dawns on just how many things are important in this system. And then trying this out myself, it really works like it's snappy, it's really cool. And the results are pretty great, I have to say for a for a learned system. So first, like welcome both of you and big props on big props on the system is very cool. So you've seen you've seen my video, what did strike you? What were they get it wrong? Yeah, first of all, I think that you did a great job in describing the overall paper. And I have almost no, you know, I have almost nothing to no complaints. Yeah, no complaints regarding that. And maybe one point regarding the overall the overall point of the paper. And yeah, as it's seen from the title, Fourier convolution might be stand out a little bit more than other components. But the actually the paper is about that all three components like, like we generate data and how we process images with a neural network and how we optimize this, how what losses do we choose, all these three components are important. And yes, sometimes they can be relatively easily tuned from existing methods and allow to such easy tuning can help to significantly improve the results. So that's that's was the overall point of the paper. Yeah, I had this I had the feeling to you again and again stress that a lot of these things are important, especially the three main components. And you did a lot of ablations to also show that all of these are important. That's why I find it so impressive, right? Because people usually just put which one did you start with first? Did you first have the idea of the Fourier convolutions? Is was that the motivation? No, initially we started when we when we started overall project on the inpainting, we just started with a classic peaks to peaks. So just get clone and pick predict an existing code base from piece to piece. And then we tried to step iteratively identify the most weak points and try to understand what is the reason behind that weakness. And at some stage, we understood that most architectures we tried really lots of different architectures, and we tried existing blocks from other inpainting papers. And we found that almost none of them can handle repetitive patterns. Well, and yes, we started it. When we think about repetitions, the one of the most obvious thing that came in mind is Fourier transform, because it is very natural thing to handle periodic signals. And first we started composing a layer on our own. And then we just googled and found that FFC, which was proposed for recognition tasks. And we thought that it is a great thing to start with and took it and modified it and tuned for that particular task. And yeah, it worked pretty well. So these would be the the Fourier convolutions. Was it already in the form that we see in the paper with the two strands of information like the global and the local? Or did you have to shake things up? No, the right part of this picture reflects the original form of this fast Fourier convolution as it was proposed by the authors. Cool. And did it work out of the box? Yes. But when we tuned that for inpainting, we figured out that the local branch is not really important. And we can handle almost everything with just global branch with that spectral transform. Yeah. So but you still kept the local branch in? Yeah, because it helps for stability, especially in not such large images and large masks. So if we try to push the generalization to high resolution to extreme, and to train on very low resolutions and then infer in very high resolutions, then using only global branch will pay more. But in the real world, some combinations, some combination of these two is more practical. Yeah. So this is it's something I found interesting because you have this point of these large, large masks, or very wide masks and so on. And you stress the importance of your algorithm that produces these different masks. Now when I look at these pictures, it doesn't seem that different, right? If I look at the top row, you know, there's also like some parts of the picture are also occluded relatively big parts, there are kind of some squiggles, they're even relatively wide, right? Why do you have an intuition? Why is the mask generation algorithm so important? Is it important that it's close to what humans do later? Or is it important that it is of a certain shape because of the architecture of the network? Or what's the deal with that? Yeah, as with the architecture, we started with an existing heuristic to draw that masks. And we actually we follow the same algorithm as the one used in Deep Field version two, the first row in that figure. Why masks should be wide? Yeah, because it is important because the width of masks forces the generator to pass the information more far within itself. So if we can cover almost all input image with very thin lines, for example, we can mask out every second row and every second column in the input image. And that would be very something very similar to a super resolution problem. And the percent of the image will be covered by such masks. But the network wouldn't need to pass information far. Yeah, that's why masks are important. And they are more important for fully convolutional architectures, but for a Fourier based they always help as well. And we have a couple of histograms in our supplementary material, which compare actually the first row of that figure with the mask generated by our algorithm. And the difference is pretty huge, actually. It is cool to see that the difference is so big. I think that it was mask that it was point from which we started, actually, because we aimed to inpaint real world examples. And in that examples, masks actually are huge. So we started with big masks in our validation set. And we saw that all other algorithms have fails to fill these large holes. And then we started to think on how we need to change our model that it can incorporate global information. Yeah. Is your algorithm deterministic? Yeah. If I give it the same input and the same mask. And is this correct that the clean up dot pictures app that is really your small model that runs here? No, this is the large model. Oh, this is the big model already. Okay. So here, I've taken this. But what happens? Have you ever tried just masking the whole picture? What's kind of like the default output? That's an interesting... I don't know what will happen. I think something average, a constant color maybe. Let's see. Yeah. All right. Pretty unspectacular. But I guess it's very gray is very high probability, right? Okay. Cool. And then there's the third component is the loss. And I have to say the loss is a monstrosity. There are like 50. So first of all, you have... No, this is the adversarial part of the loss. And then on top of that, you have like the discriminator perceptive loss. I'm going to guess that's the same as the perceptual loss, but in the features of the discriminator. Yeah. So the features which are used to calculate discriminator based perceptual loss are updated throughout the training. This is a pretty commonly used loss in image to image tasks. It helps to stabilize training. So the idea is that the discriminator bases its decisions on features which are perceptually meaningful. So very similar to the perceptive loss that you have up here, right? I think that feature matching or discriminator based perceptual loss helps mostly because it provides a clear signal to the generator. And if in adversarial training, we have to balance discriminator and generator. And if one part is more powerful, the whole thing collapses. And discriminator based perceptual loss helps the generator to catch up when discriminator becomes too powerful. Yeah, that makes sense. For all of these losses, right? And then you have a regularizer on the gradients and you have this wide field perceptive loss and so on. Did you plan this from the beginning? Did you say, you know, here are all the good losses that I know of? Or do you have more losses that you ended up not including? My question is, how, if I'm a if I'm a if I'm a researcher or an engineer trying to come up with such a system, how do I decide which seven losses go into my final loss, right of the 50 possible losses that I could do? Do I try them all? Or are there some some guidelines? Actually, I think all of these losses, except for high perceptual loss are pretty common. And they all are often used in image to image tasks. We need something to force our model to create a realistic picture. So we need discriminator and its loss. We need to reconstruct something that we can reconstruct. So we need some loss for reconstruction and editing losses to restrict it. So we need something that works on features. But we worked a lot on we made a hyperparameter search, of course, and we changed our we work on our perceptual loss form, because we started with this common perceptual loss based on the GG model. But we had a feeling that it can be not perfect because it's because the models that run on classification tasks and models that was trained on classification, they seems to concentrate on texture and not global structure. So we decided to try something else. And then we find these models that run on segmentation tasks. And on data set that is more similar to ours, data set, and we tried it and it worked. So the segmentation task as a training task for the perceptual loss model is sort of a better preconditioner than the classification task? Yeah, because it is natural for the segmentation model to focus more on boundaries of objects instead of their textures. And in case of inpainting, good texture can be learned using only discriminator. Because there is a lot of freedom in how we can generate fine grain textures and there is no need to put some any supervision on that part of. But it's also important that models that are used for segmentation are different. So we compared in our ablation, we compared the same model that was on the training on classification and it works with the same model. Yeah, you have not only do you have a different task with the segmentation, you also include also higher receptive field layers into that model. So the sense, the logic is that if that model also includes global information more, its signal to the model is more accurate. So it's signal to your model will also be more sensitive to that global information. It's a little bit like, you know, in reinforcement learning, people do reward shaping, it seems like you do reward shaping by how you train the different discriminator models that then give your model the sort of the signal to learn. Yeah, I like the sort of the meta idea here. That's pretty cool. Unfortunately, I'm not familiar with reward shaping from reinforcement learning. But our idea here was that basically we have two losses here. The first one is discriminator or the serial and which focus more on fine grain details and the second is perceptual loss, which focuses more on global text, global structures. For the Fourier convolutions, maybe a little bit more conceptual, right? We have this local information in one strand, we have this global information in the other strand. And it's clear that for these large masks, as you show the system works pretty well. What kind of data does your system work not well on? Like what's what would be sort of the worst input that I could give to your system? Like this, this up here is really beautiful, right? What picture could I take such that it is absolute garbage? Yeah, actually, lots of images will be processed bad with our model. I mean, of course, I can give it a picture that is, you know, very dissimilar to the training data set. But let's say I actually had a training data set, what would be the worst domain or the worst kind of picture? Yeah. I think it cannot recreate half of human on something. Yeah, our model focuses mostly on background due to how it was trained. And yeah, it cannot recover foreground objects really well. It cannot do something that requires it to actually know everything about what works and not just take it from picture it sees. Yeah. So is it, is it, do you feel that the model mostly learns how to sort of copy elements from the part it sees to the parts that are masked? Do you think that the learning is mostly teaching the model how to do that? Because it seems the model is very sophisticated in, you know, in Photoshop, you take this stamp tool, right? You say, I'll take a little bit from over here, put it here. Do you think your model is just like a really, really good user of that tool in a sense? Yeah, it seems yes, yes. And in order to be able to create big parts of images from scratch, we need a different kind of model. And we most probably need a kind of capacity within the generator, because without it, it is not possible to create something from nothing. Yeah. Also, our model is quite small, so it cannot really remember everything. Yeah, that is something that I left completely out of my review. I think the fact that your model is compared to the baselines you compare to is a lot smaller, right? It has way less parameters. That is something that's, I think, very cool and enables it to run inside web applications and so on, like on or maybe on a mobile device or... Yeah, I have another question and to the Fourier convolution. So here we have global information and local information, right? As sort of two different things. You mentioned in the paper that other models that have more global information or access to wider information could also work, such as a vision transformer or something like this. My question is, is there an in between between local convolutions and Fourier convolutions? Okay, I mean, there's dilated convolutions. But if I think of a Fourier transform, you transform into a space where locality no longer matters, but frequency matters. And in the original domain, frequency is just kind of doesn't matter, but locality really matters. Is there a transform, are there transforms that we could do that put us in between where, you know, the as I go in the x coordinate, it's a little bit of frequency and a little bit of locality? Like, is there hope that instead of having multiple columns of information, we could sort of choose our space wisely to trade off local and global? Or do you think this is already, you know, local, like a mix with two channels is a good way to go? That's a very good question. Yeah, and I don't know the answer to it. And one thing that comes to my mind is there is a short time Fourier transform, which is often used for music processing, sound processing. And yeah, it kind of combines local convolutions with Fourier transform over. It is roughly can be described as processing the whole signal with a sliding window and transform each sliding window with Fourier transform. Yeah. So it is most obvious combination. If you had to give your intuition why the Fourier convolutions make such a big difference here, of course, like that, we've already discussed Fourier transform kind of loses the locality of the signal and gets global information. But why Fourier transforms? What's kind of good about this particular function that you chose and space that you chose? Surprisingly, if we throw the local branch away, it will still generate something meaningful. So spectral transform doesn't lose that local correlations completely. And I think that this is due to the fact that the generator has spectral transforms and spatial transforms interleaving each other because here we can see that we have cone one by one between two FFT and we have two more convolutions before and after the spectral transform. They are as well one by one. So they don't capture local content directly, but then can combine channels on that particular locations. And yeah, that maybe that can somehow replace traditional convolutions. The fact that these spatial and spectral transforms are interleaved. Yeah. And when we think about generalization to higher resolution, I think spectral transform helps because of the fact that low frequency part of spectrum does not depend on the resolution, on the input resolution that strong. And it is it is almost the same no matter if we have 2056 or sorry 256 or 2000. Yeah. Yeah, that by itself is one of the cool properties again of your paper. The fact that it can scale up to sort of very high resolutions. There are artifacts appearing, but they are not nearly as much as in other other models. Looks pretty cool. Yeah, it doesn't scale up perfectly, but yeah, it's better than fully convolutional architectures. Cool. Yeah. So where do you think I mean, maybe you don't want to disclose necessarily but what is the plan for the future? We don't know where we get throughout research. But yeah, the most obvious thing here is that we can try to improve the way generalized to high resolutions. And the second point is that we are trying to understand why actually it works that because it yeah, it has lots of components. And we conducted an ablation study regarding if validating if each of these components matter, but this is just a surface. And we can go more in depth in that. And we are not satisfied with our laws because it's that huge. There are many components that we need to balance. And we want better laws with just one, just one button, make everything work. And nice. So yeah, I mean, I was almost I was expecting you to say we're not happy with our loss. We want more. We want like more components to make it. But it's I think it's pretty cool that the goal is also to make a system that's kind of as good but simpler. I think they'll make it also much more accessible. Yeah, I think that's a good idea. Yeah. I think they'll make it also much more accessible. Cool. Yeah. Roman, Elisa, sorry, Lisa. Is that correct? Yes. Okay, Lisa and Roman, thank you so much for being here. Was it was a pleasure? Do you have any last criticisms to the video? Or shout outs? No, thank you very much for for the discussion. It was really fun. And thank you for your channel, because it is you make a real good job in helping others to be in time and to catch with the this huge wave of information that we have in the field. Thanks. Thanks. Yeah, thank you. Thank you.
 Hello there, today we're looking at resolution robust large mask in painting with Fourier
 convolutions also called LAMA by the Samsung AI Center, Samsung Research, EPFL and the Skolkovo
 Institute of Science and Technology. This is a special paper review because I'm only going to
 introduce the paper briefly, maybe 15-20 minutes or so and then we're going to talk to the first
 author of the paper and go a lot more in depth. So if you like, if you like conversations with first
 authors and the ability for me to ask dumb questions to them, then stay tuned for that.
 It's going to be in the second half of the video. For the first half though, I first want to
 demonstrate to you what this model can do. Hey there, this video is sponsored by ClearML.
 ClearML is an ML Ops stack that is fully open source. It can do experiment tracking,
 orchestration, deployment, model and features, stores and much more. The self-hosted tier is
 a first class citizen in ClearML. As I said, it's fully open source, you can look at it, you can audit
 it, you can extend it, you can run it on your servers. And if you ever come to the point where
 you need the extra features, you can totally upgrade anytime, they'll gladly take your money.
 They have a free tier in the cloud, which gets you going pretty far. Now we talked about experiment
 tracking last time ClearML with two lines of code will track any experiment that you do track the
 metrics, the outputs, the environments, the dependencies and make everything super duper
 reproducible. But this time I want to talk about a second part, which is the orchestration engine.
 So the orchestration engine is responsible for packaging up your experiments, including all
 dependencies, and then distributing them on your hardware. So that means you can simply submit an
 experiment to a particular queue and ClearML takes care of running this wherever it's needed. So this
 is super cool, because it means I can get going on my laptop, run a few experiments there. And as
 soon as I'm ready, boom, I ship it to the cloud. So here's an example, look at this experiment that
 has already been run, I got some output, but now I would like to do something different with it.
 So I click here, I say clone, I give it a meaningful name, like two. And now I've cloned this experiment.
 And this is kind of a draft experiment right now, it has no results yet. But what I can do, I can go
 into my configuration, into my hyper parameters, and I can change around the hyper parameters. So
 I wasn't really happy with the last experiment, I feel a bigger batch size might be needed. So
 from 128, let's go to 129. Now I'm pretty sure that's going to make all the difference right here. So
 I save and then I simply click on in queue, I submit it. And now ClearML simply takes care of
 running that experiment for me. As you might guess, you can have different queues, some for GPU load,
 some for long running tasks, some high priority, as you're used to from any scheduler. This can
 also be used in automated fashion, meaning that you can use this for automated hyper parameter search,
 and you can even do things such as scheduled or triggered tasks. For example, if you want to
 trigger a training run every day on new incoming data, that's totally doable. Now orchestration is
 just one part of ClearML. I've shown you experiment tracking last time. And there are many more
 features to their product. If this sounds interesting to you, if you're an open source fan,
 go check them out. And thanks so much to ClearML for sponsoring this video. Let's get into it.
 You can already see it a little bit in figure one right here, the model is able to take a picture,
 you draw a mask on it. So this is the blue area right here. And the model would auto complete the
 picture. So the model doesn't see the mask, the model simply sees what is unmasked, then the model
 is asked to complete that missing area. As you can see, it fills that area in, you know, very,
 very cleanly. And especially if you look right here, this irregular structure of these door holes,
 or whatever that is, is preserved even across very large areas. This is very, very cool. This is very
 difficult to do with these in painting systems. In fact, there is a project website right here,
 all the code is available. They give this in a little bit more of an animated flair. So you can
 really see the effect that these models are having. And it's pretty, pretty cool, especially take a
 look at these repeated structures that are often in the pictures. So these meshes or the lines right
 here, these tend to be extremely these tend to be especially difficult for in painting models,
 because in painting models are usually built on convolutional neural networks, and convolutions,
 notably take into account very local context. Whereas for these patterns, you need to take into
 account kind of a global context, that's exactly going to be the the message right here. There is
 an app, there are actually a bunch of apps based on this model. This is a third party app. So this
 is not by the author. But it is an app built from these models. There are also as I said, code is
 available. There's like a hugging face space, there is a collab by the author. But this particular app,
 let's just take a picture right here. It works best on natural images, of course, but we'll just
 take the channel logo right here. And we'll say we want to erase the pie sign right here. Look how
 cool that works. What about the paw? Okay, that that is that is kind of disturbing. How about the
 nose? No, no, no, I don't like that. But it should be able to Yeah, see, so it kind of completes
 lines, if you cross them out. So this should complete the table, but remove the leg, you can see
 it's fairly robust, even to use sort of miss specifying bunch of things. So here I draw over
 the headline, if you saw that, and it remained the head headline remains. So I removed this part,
 but I crossed into here a bit, you can see the line kind of remains. Now it's got a bit of hair.
 Yes, kill it with fire. In any case, this is available for you to use if you have more sensible
 pictures, I'm sure that that will work a little bit better, maybe. There are also different versions
 of the model. So keep that in mind. And they works also on different resolutions. That's why it's
 called resolution robust, large mask in painting, which is also very cool. So what is the core idea
 of this paper, the core idea is going to be these Fourier convolutions right here. And these Fourier
 convolutions are going to be enabling the model to take into account global context from the very
 beginning. What is the problem with a convolutional neural network? The problem usually is that
 in a convolution, if I have a picture, a convolution on a particular point will take into account its
 local neighborhood, right? And then I sort of slide this over the image right here. And that will
 give me my representation in the next layer, maybe that's going to be even of the same size. So for a
 given point right here, I will have a receptive field of the point in the original image, plus
 some neighborhood. Usually we work with three by three convolutions. So all I'm going to do really
 is I'm going to look one pixel to the top and one pixel to the bottom, one pixel to the top,
 one pixel to the left, and one pixel to the right. And that's about it. I'm not going to do any more
 looking around. So how does a convolutional neural network integrate information across the whole
 image? And the answer to that is by going for multiple layers. If I simply represent the picture
 as a set of pixels in one dimension, imagine that the one dimension here is the picture.
 And I'm going to need a bit more space for that. So as you can see in the first layer,
 from the first to the second layer, let's say we look at this particular point right here,
 it's going to be have a receptive field of three. So it's going to look at these pictures, sorry,
 at these pixels right here. In the next layer, if you can see that the same location is also having
 a receptive field of three right here. However, since for example, this particular pixel right
 here also had a receptive field of three, and this particular one also, as you can see, and from layer
 two on the total receptive field of that so that all the information inflow is going to be from a
 receptive field of five. Therefore, the more layers we have, the more of information, the more spatial
 information can be included for a given particular location in the output. But as I said, that takes
 a lot of layers that takes depth. And especially for these in painting applications, what you want
 is kind of global information. These masks right here, like these masks, they're pretty big for an
 in painting application. So they're pretty, pretty wide. And if you can imagine a convolutional
 layer that looks at a three by three pixel neighborhood, that might be something right here.
 You know, so you're going to have a whole lot of convolutional kernels that just see the masked
 pixels, they see nothing of the outside, they simply see a bunch of masked pixels for a whole
 bunch of layers, right layer two, layer three, until like layer four, there's like there's nothing,
 no information at all at this position about the outside world about the world beyond the mask.
 And even then, it's only like this area, we need to go many more layers before we get access to
 information that is way outside of here. And at that point, it may already be too late. So the
 Fourier convolutions, they solve that they have the ability at every single layer to look at a
 global context. And how are they doing this? It's not super expensive. In fact, they're doing this
 by using of course, Fourier transformations, a Fourier transformation will map a signal to its
 corresponding frequency domain signal, it is essentially a different way of representing
 a signal. So if you have a signal, let's say you have like a pure sine wave, you do a Fourier
 transformation of that entire thing, you can represent that as the components in the Fourier
 spectrum. And that would simply have like one component at the particular frequency at which
 this sine wave at which this sine wave is operating. That's the that's not the frequency,
 that's like one over the frequency right here. But in a more in a more general sense, a Fourier
 transform will decouple the spatial resolution and give it a transform it into frequency resolution.
 So if you have a Fourier spectrum, maybe you have a very complicated signal right here,
 a complicated signal that will give you also a complicated Fourier spectrum, like you have a lot
 of this, you have like negative this frequency, a lot of this frequency, not too much of this
 frequency, and so on. If you do a convolution in this domain, you simply convolve across neighbors
 of the signal. However, if you do a convolution in Fourier domain, you can see you convolve across
 frequencies, you can evolve across neighboring frequencies, which means that these three things
 represent three particular sine waves frequencies, maybe the lowest one is like a super long sine
 wave, the second one is like a bit of a faster sine wave, the third one is even faster sine wave.
 But what is important is that every single component in the Fourier spectrum represents
 information about the entirety of the signal. And that is exactly what we want. Whereas the
 regular convolution is localized in in pixel space, the Fourier convolution is going to be localized
 in frequency space, but global in pixel space. That is very cool. And of course, Fourier transforms
 are also one of the things that are extremely fast. It's essentially a linear algebra operation. And
 there are very fast implementations of discrete Fourier transforms called fast Fourier transforms.
 That's exactly what they do right here. The whole architecture is going to look like this. There is
 going to be the image the input image x there's going to be a mask during training that is produced
 by a mask generation algorithm x is then masked out and the model is tasked to predict the missing
 pixels that are hidden by the mask. As I said, the model has no access to what's below the mask,
 I guess that will that would be kind of pointless, right? Yeah. So what we do first, but also this is
 a fully convolutional architecture that makes it able to essentially transfer to different resolutions,
 which is another advantage here being a fully convolutional. So what we do is first we downscale
 a little bit as far as I can tell these images are something like 256 by 256 during training,
 or it works on crops of 256 by 256, somewhere in that range. But the cool thing is it can generalize
 to high definition images like 1920 by 1080 or something like this, the same network. So the
 train the network that's trained on this low, low, quote unquote, low resolution can generate can
 generalize to very, very high resolution, and it won't lose performance. But we'll see that in the
 experiments. So first there's down sampling, and then the model is just this is just nine layers.
 They also have a variant with 18 layers. But the base model is nine layers of this fast Fourier
 convolution residual block. As you can see, it has a residual connection right here, like a normal
 resnet, whereas a normal resnet would have two convolution layers right here, we opt for these
 fast Fourier convolutional layers. Now, they look a bit complicated, but essentially, what we do is
 we carry two different signals across the entire network, one signal contains local localized
 information. So one signal is going to operate in the original domain of pixel space and has all
 that those properties, so it looks at its neighbors and so on. And one signal is going to operate in
 in more of the global domain. And then in each layer, those two strands of information get to
 exchange information with each other. So the whole signal is represented as this block here with the
 two components. But it's essentially just we have like two strands of signal, and then every now and
 then they get to exchange a bit of information, right, one is the local, the local branch, and one
 is the global branch of information. So what do we do with the local branch, we have different
 operations right here. So we have a little conv layer that is in pixel space, actually, we have two
 of them, right, two conv layers. So we pass this the local signal, this is really just if you just
 consider this path right here through this one, then ignore ignore this here. If you just go here,
 this is just like a normal conv net, right, this path here gets information from this side here.
 It receives it and then there is an addition. So what is that, that is simply this global signal
 the global signal, also doing a localized convolution in pixel space. So far, there is nothing
 special if we were to just do this, this would be it would be pointless to have two strands of
 information, right. But the important thing is that the global strand comes to be in a very special
 way. So for that, we have to look what information arrives at the global branch right here, because
 that's the information that's going to be passed in here for the next layer. For that, we see
 from the local branch, there's a three by three convolution going out over here. So let me draw
 that in greenish over here. And that is going to be mixed with this global strand of information.
 And the global strand of information is going through this spectral transform block. The
 spectral transform block is essentially pretty easy. There is a there's a batch norm, sorry,
 a convolution batch norm relu block. This is a one by one convolution. This is simply
 simply simply a linear operator pixel wise, essentially, there's a batch norm, there's a
 relu for the nonlinearity. And then what we do is we do a fast Fourier transform in 2d. And at the
 end of the block, we're going to invert that. So fast Fourier transform to operate in Fourier space,
 and then invert the fast Fourier transform at the end. And inside of it, we're going to do a
 convolution batch norm relu block right here. So the convolution again, that's a one by one
 convolution, I believe, followed by batch and followed by relu. So actually even forget what I
 said about localized convolutions right here, if they just do one by one convolutions, they really
 operate just on the individual elements of the spectrum by itself, not even, they don't even
 they don't even consider localized, sorry, neighborhoods of frequencies, they just operate
 on the individual frequencies, one by one, which is is an option, like one by one convolutions are
 are a thing. So, you know, pretty cool. This by itself also has residual connection right here,
 I'm going to guess to make signal flow better or more more stable or some something like this,
 the observant people might object and say, hey, this thing right here actually outputs complex
 numbers. So this is in the space of complex numbers. So you'll get vectors with entries like a plus,
 plus IB. But what we do is simply we take those and we stack them. So we just make like vectors
 out of them, a and b. So if there is a bunch of numbers, it will just be like a one b one,
 a one b one, b one, a two, b two, and so on. And we just consider this to be a real vector
 of double dimensionality, or a real 2d signal of double the dimensionality as before. And that
 is how we do it. I mean, it's not it's not entirely correct, right. But the model in this way has
 access to all the relevant information, it can do what it wants with it. Yeah, it can it can learn
 that half of the dimensions correspond to two phases, or, or whatever, whatever the complex part
 of this is, it's been a while since since been a while since Fourier transforms. Okay, so these are
 the exactly so here, that's done, we have, sorry, go back up here to start it, there is first the
 real FFT, as you can see, that gets you to complex space, then there is complex to real, in which we
 transform the c channels into two c channels. But now we're in the real numbers. Then there is this
 value batch norm conv, which retains the signal. And there is real to complex where we go back into
 complex space. So from reals, 2d to c channels into complex, just c channels, and then we reverse the
 Fourier transform. And that is a Fourier convolution, as they define it. If we integrate,
 no, that is the spectral transform block right here, the Fourier transfer, the Fourier convolution
 is this entire construct right here, as you can see, the spectral transform information then flows
 in here is combined with some local information that really should be green. And that then goes
 into this global output and obviously will become the global input to the next layer. So that is how
 they fuse localized information with global information in every single layer. And that turns
 out to be pretty, pretty powerful. They do have other improvements right here. And it's it's crazy
 to see that just how much engineering and how many tricks go into these models to really get them to
 work. So they also stress that loss function is a really, really important topic right here, because
 you can't simply reconstruct the original image right here, if you simply tell the model to
 reconstruct the original image from here, it's going to be bad because if your mask is pretty big,
 pretty wide, there can be many possible fillings of the mask that makes sense. And since there are
 many possible ones, if you don't account, if you don't reward the model for getting one of the
 possible ones without punishing it that it didn't get all the other ones, the model is going to be
 very confused and is simply going to output the average of all the possible ones, which we don't
 want we want one of the possible ones. So what we do is we apply a perceptive loss, they call that a
 perceptive loss. And they explain that over here, what you do is you feed the image, the original
 image, the original image, this is the real one, and the fake one, and you can already see there's
 going to be like a discriminator later, right. But you feed them both through a pre trained neural
 network. And then you compare at intermediate points, or even like at the last latent layer,
 you compare the two feature maps. So depending on how this network is trained, if that outputs very
 perceptually salient features, you'll get like a nice loss that doesn't punish you for getting any
 pixels wrong. But that encourages you to get something that is perceptually similar to what
 was there in the original image. They also stress that it's really important on how you train this
 network right here. They suggest to make this network also include global context using either
 also Fourier convolutions or dilated convolutions. And here you can see that's essentially the
 formula that means we take the features from the original image and the features from the fake image,
 and we calculate their distance. And that's going to be the high receptive field perceptual loss.
 This is not the only thing they do. They also have, as you can see, an adversarial loss.
 There is also a regularizer on the gradients. So yeah, the final loss you're going to end up with
 is like a mix of all of these different losses. There's also a discriminator based perceptual loss.
 And this part right here is by itself, again, a conjunction of two losses. So rest assured,
 the loss architecture right here is very, very intricate. And I'm going to guess it's taken a lot
 of experimentation, not only by this paper, but by the whole field here to really come up with nice
 losses that make your outputs nice. Obviously, there's going to be a bunch of hyper parameters
 here to tune, which is always fun, but they seem to have done a pretty good job. The last thing
 they stress, which is important is how you generate masks during training. So during training,
 you can't just, you know, take your finger and draw on pictures. Like I did, you have to have
 some heuristic way of generating masks. And I'm not going to go into the detail of how they do it.
 You can see here compared to this is one of the one of the baselines. And this is one of their
 heuristics. They have a mix of these large masks and the box masks. So sorry, both are large,
 but one is called wide masks, which are kind of polygons that they round off the corners,
 I think, and box masks, which are sort of heuristically generated boxes right here,
 or stacks of these boxes. And that's, and they mix those two together in order to get the final
 masking for their images. You can see these are fairly large, like this one here covers more than
 more than half the image. So these are challenging, challenging tasks. But it is through training with
 such large masks that you get the models to really learn to fill in it consistently. So what you can
 see is that in their results, and we're not going to go into all the tape, like they have a lot of
 tables, a lot of ablations, but red essentially means that it's worse than their model, you can
 see almost all of the table is red, except some models in some of the benchmarks, for example,
 in the narrow masks, you will find situations where other models might outperform their model.
 But as soon as you go to like wide masks, it is no longer, it's no longer really a competition at all.
 Yeah, so their model seems to be really good. Those white masks, they do a lot of ablations
 where they switch out different, for example, different convolutions right here, they show what
 if we switch the Fourier by a dilated convolution, which is also a way to increase the receptive
 field rapidly or by regular convolution. And again, while there might be some improvement,
 sometime on narrow masks, as soon as you go to wide masks, the other models degrade pretty
 quickly, the dilated convolution actually holds up fairly well right here. But one disadvantage of
 that is that it's very hard to go to higher resolutions, because the higher resolution you go,
 the dilated convolutions that their receptive fields will also shrink, while the Fourier
 convolutions receptive fields will always remain essentially global. So here you have some comparison
 to baselines, you can see of course, they chose these pictures well with kind of the regular
 structure in the background. But check this out, like this is even this is even their model. But
 with regular convolutions, and even if they go deeper, doesn't really help. But like this,
 this is just insane, right? I get it, they pick this picture, but it is like is really good.
 And you can also see this building how it's completed over here with different methods,
 and then with their method. And the mask was, you know, fairly, fairly big, as you can see,
 also the bottom this the mask is huge. Yeah, here they show what happens if you go to higher
 resolution. So on this rather simpler problem, you can see that a lot of the models do well in
 the top row, if you just have the kind of a lower resolution. But if you go to really high resolution,
 a lot of the models struggle while the llama model here still does a big, a good job in their larger
 model seems to be even better. Yeah, again, lots of ablations, but I'm going to stop right here,
 and we'll go over to chatting with the first author about this. So I'll see you in a bit.
 Hello, everyone. I'm here with Roman Suvorov and Elizaveta Logacheva, the authors of the llama
 paper and llama system as well, I guess I think this is as much a paper as it is an engineering
 effort. And just because looking at the paper, it already dawns on just how many things are important
 in this system. And then trying this out myself, it really works like it's snappy, it's really cool.
 And the results are pretty great, I have to say for a for a learned system. So first, like welcome
 both of you and big props on big props on the system is very cool. So you've seen you've seen
 my video, what did strike you? What were they get it wrong? Yeah, first of all, I think that you did
 a great job in describing the overall paper. And I have almost no, you know, I have almost nothing to
 no complaints. Yeah, no complaints regarding that. And maybe one point regarding the overall
 the overall point of the paper. And yeah, as it's seen from the title, Fourier convolution might be
 stand out a little bit more than other components. But the actually the paper is about that all three
 components like, like we generate data and how we process images with a neural network and how we
 optimize this, how what losses do we choose, all these three components are important. And yes,
 sometimes they can be relatively easily tuned from existing methods and allow to such easy tuning can
 help to significantly improve the results. So that's that's was the overall point of the paper.
 Yeah, I had this I had the feeling to you again and again stress that a lot of these things are
 important, especially the three main components. And you did a lot of ablations to also show that
 all of these are important. That's why I find it so impressive, right? Because people usually just
 put which one did you start with first? Did you first have the idea of the Fourier convolutions?
 Is was that the motivation? No, initially we started when we when we started overall
 project on the inpainting, we just started with a classic peaks to peaks. So just get clone and
 pick predict an existing code base from piece to piece. And then we tried to step iteratively
 identify the most weak points and try to understand what is the reason behind that weakness. And at
 some stage, we understood that most architectures we tried really lots of different architectures,
 and we tried existing blocks from other inpainting papers. And we found that almost none of them can
 handle repetitive patterns. Well, and yes, we started it. When we think about repetitions,
 the one of the most obvious thing that came in mind is Fourier transform, because it is very
 natural thing to handle periodic signals. And first we started composing a layer on our own.
 And then we just googled and found that FFC, which was proposed for recognition tasks. And we
 thought that it is a great thing to start with and took it and modified it and tuned for
 that particular task. And yeah, it worked pretty well. So these would be the the Fourier
 convolutions. Was it already in the form that we see in the paper with the two strands of information
 like the global and the local? Or did you have to shake things up? No, the right part of this
 picture reflects the original form of this fast Fourier convolution as it was proposed by the
 authors. Cool. And did it work out of the box? Yes. But when we tuned that for inpainting, we
 figured out that the local branch is not really important. And we can handle almost everything
 with just global branch with that spectral transform. Yeah. So but you still kept the
 local branch in? Yeah, because it helps for stability, especially in not such large images
 and large masks. So if we try to push the generalization to high resolution to extreme,
 and to train on very low resolutions and then infer in very high resolutions, then
 using only global branch will pay more. But in the real world, some combinations,
 some combination of these two is more practical. Yeah. So this is it's something I found
 interesting because you have this point of these large, large masks, or very wide masks and so on.
 And you stress the importance of your algorithm that produces these different masks. Now when I
 look at these pictures, it doesn't seem that different, right? If I look at the top row,
 you know, there's also like some parts of the picture are also occluded relatively big parts,
 there are kind of some squiggles, they're even relatively wide, right? Why do you have an
 intuition? Why is the mask generation algorithm so important? Is it important that it's close to what
 humans do later? Or is it important that it is of a certain shape because of the architecture of the
 network? Or what's the deal with that? Yeah, as with the architecture, we started with an
 existing heuristic to draw that masks. And we actually we follow the same algorithm as the
 one used in Deep Field version two, the first row in that figure. Why masks should be wide? Yeah,
 because it is important because the width of masks forces the generator to pass the information more
 far within itself. So if we can cover almost all input image with very thin lines, for example,
 we can mask out every second row and every second column in the input image. And that would be very
 something very similar to a super resolution problem. And the percent of the image will be covered
 by such masks. But the network wouldn't need to pass information far. Yeah, that's why masks are
 important. And they are more important for fully convolutional architectures, but for a Fourier
 based they always help as well. And we have a couple of histograms in our supplementary material,
 which compare actually the first row of that figure with the mask generated by our algorithm.
 And the difference is pretty huge, actually. It is cool to see that the difference is so big.
 I think that it was mask that it was point from which we started, actually, because we
 aimed to inpaint real world examples. And in that examples, masks actually are huge.
 So we started with big masks in our validation set. And we saw that all other algorithms have
 fails to fill these large holes. And then we started to think on how we need to change our
 model that it can incorporate global information. Yeah. Is your algorithm deterministic? Yeah.
 If I give it the same input and the same mask. And is this correct that the clean up dot pictures
 app that is really your small model that runs here? No, this is the large model. Oh, this is
 the big model already. Okay. So here, I've taken this. But what happens? Have you ever tried just
 masking the whole picture? What's kind of like the default output? That's an interesting...
 I don't know what will happen. I think something average, a constant color maybe.
 Let's see. Yeah. All right. Pretty unspectacular. But I guess it's very gray is very high
 probability, right? Okay. Cool. And then there's the third component is the loss. And I have to
 say the loss is a monstrosity. There are like 50. So first of all, you have... No, this is the
 adversarial part of the loss. And then on top of that, you have like the discriminator perceptive
 loss. I'm going to guess that's the same as the perceptual loss, but in the features of the
 discriminator. Yeah. So the features which are used to calculate discriminator based perceptual
 loss are updated throughout the training. This is a pretty commonly used loss in image to image
 tasks. It helps to stabilize training. So the idea is that the discriminator bases its decisions
 on features which are perceptually meaningful. So very similar to the perceptive loss that you have
 up here, right? I think that feature matching or discriminator based perceptual loss helps mostly
 because it provides a clear signal to the generator. And if in adversarial training,
 we have to balance discriminator and generator. And if one part is more powerful, the whole thing
 collapses. And discriminator based perceptual loss helps the generator to catch up when discriminator
 becomes too powerful. Yeah, that makes sense. For all of these losses, right? And then you have a
 regularizer on the gradients and you have this wide field perceptive loss and so on. Did you
 plan this from the beginning? Did you say, you know, here are all the good losses that I know of?
 Or do you have more losses that you ended up not including? My question is, how, if I'm a
 if I'm a if I'm a researcher or an engineer trying to come up with such a system, how do I decide
 which seven losses go into my final loss, right of the 50 possible losses that I could do? Do I
 try them all? Or are there some some guidelines? Actually, I think all of these losses, except for
 high perceptual loss are pretty common. And they all are often used in image to image tasks.
 We need something to force our model to create a realistic picture. So we need discriminator
 and its loss. We need to reconstruct something that we can reconstruct. So we need some
 loss for reconstruction and editing losses to restrict it. So we need something that works on
 features. But we worked a lot on we made a hyperparameter search, of course, and we changed
 our we work on our perceptual loss form, because we started with this common perceptual loss based on
 the GG model. But we had a feeling that it can be not perfect because it's because the models that
 run on classification tasks and models that was trained on classification, they
 seems to concentrate on texture and not global structure. So we decided to try something else.
 And then we find these models that run on segmentation tasks. And on data set that is more
 similar to ours, data set, and we tried it and it worked.
 So the segmentation task as a training task for the perceptual loss model is sort of a better
 preconditioner than the classification task? Yeah, because it is natural for the segmentation
 model to focus more on boundaries of objects instead of their textures. And in case of inpainting,
 good texture can be learned using only discriminator.
 Because there is a lot of freedom in how we can generate fine grain textures and there is no need
 to put some any supervision on that part of. But it's also important that models that are used for
 segmentation are different. So we compared in our ablation, we compared the same model
 that was on the training on classification and it works with the same model.
 Yeah, you have not only do you have a different task with the segmentation, you also include
 also higher receptive field layers into that model. So the sense, the logic is that if that model
 also includes global information more, its signal to the model is more accurate.
 So it's signal to your model will also be more sensitive to that global information. It's a
 little bit like, you know, in reinforcement learning, people do reward shaping, it seems like
 you do reward shaping by how you train the different discriminator models that then
 give your model the sort of the signal to learn. Yeah, I like the sort of the meta idea here.
 That's pretty cool. Unfortunately, I'm not familiar with reward shaping from reinforcement learning.
 But our idea here was that basically we have two losses here. The first one is discriminator or
 the serial and which focus more on fine grain details and the second is perceptual loss, which
 focuses more on global text, global structures. For the Fourier convolutions, maybe a little bit
 more conceptual, right? We have this local information in one strand, we have this global
 information in the other strand. And it's clear that for these large masks, as you show the system
 works pretty well. What kind of data does your system work not well on? Like what's what would
 be sort of the worst input that I could give to your system? Like this, this up here is really
 beautiful, right? What picture could I take such that it is absolute garbage? Yeah, actually,
 lots of images will be processed bad with our model. I mean, of course, I can give it a picture
 that is, you know, very dissimilar to the training data set. But let's say I actually had a training
 data set, what would be the worst domain or the worst kind of picture? Yeah. I think it cannot
 recreate half of human on something. Yeah, our model focuses mostly on background due to how
 it was trained. And yeah, it cannot recover foreground objects really well. It cannot
 do something that requires it to actually know everything about what works and not just take it
 from picture it sees. Yeah. So is it, is it, do you feel that the model mostly learns how to sort of
 copy elements from the part it sees to the parts that are masked? Do you think that the learning
 is mostly teaching the model how to do that? Because it seems the model is very sophisticated
 in, you know, in Photoshop, you take this stamp tool, right? You say, I'll take a little bit from
 over here, put it here. Do you think your model is just like a really, really good user of that tool
 in a sense? Yeah, it seems yes, yes. And in order to be able to create big parts of images from
 scratch, we need a different kind of model. And we most probably need a kind of capacity within
 the generator, because without it, it is not possible to create something from nothing.
 Yeah. Also, our model is quite small, so it cannot really remember everything.
 Yeah, that is something that I left completely out of my review. I think the fact that your model
 is compared to the baselines you compare to is a lot smaller, right? It has way less parameters.
 That is something that's, I think, very cool and enables it to run inside web applications
 and so on, like on or maybe on a mobile device or... Yeah, I have another question and to the
 Fourier convolution. So here we have global information and local information, right? As
 sort of two different things. You mentioned in the paper that other models that have more global
 information or access to wider information could also work, such as a vision transformer
 or something like this. My question is, is there an in between between local convolutions and
 Fourier convolutions? Okay, I mean, there's dilated convolutions. But if I think of a Fourier
 transform, you transform into a space where locality no longer matters, but frequency matters.
 And in the original domain, frequency is just kind of doesn't matter, but locality really matters.
 Is there a transform, are there transforms that we could do that put us in between where, you know,
 the as I go in the x coordinate, it's a little bit of frequency and a little bit of locality?
 Like, is there hope that instead of having multiple columns of information, we could sort of choose
 our space wisely to trade off local and global? Or do you think this is already, you know, local,
 like a mix with two channels is a good way to go? That's a very good question. Yeah, and I don't
 know the answer to it. And one thing that comes to my mind is there is a short time Fourier transform,
 which is often used for music processing, sound processing. And yeah, it kind of combines local
 convolutions with Fourier transform over. It is roughly can be described as processing the whole
 signal with a sliding window and transform each sliding window with Fourier transform. Yeah.
 So it is most obvious combination. If you had to give your intuition why the Fourier convolutions
 make such a big difference here, of course, like that, we've already discussed Fourier transform
 kind of loses the locality of the signal and gets global information. But why Fourier transforms?
 What's kind of good about this particular function that you chose and space that you chose?
 Surprisingly, if we throw the local branch away, it will still generate something meaningful.
 So spectral transform doesn't lose that local correlations completely. And I think that this
 is due to the fact that the generator has spectral transforms and spatial transforms interleaving
 each other because here we can see that we have cone one by one between two FFT and we have two
 more convolutions before and after the spectral transform. They are as well one by one. So they
 don't capture local content directly, but then can combine channels on that particular locations.
 And yeah, that maybe that can somehow replace traditional convolutions. The fact that these
 spatial and spectral transforms are interleaved. Yeah. And when we think about generalization
 to higher resolution, I think spectral transform helps because of the fact that low frequency part
 of spectrum does not depend on the resolution, on the input resolution that strong. And it is
 it is almost the same no matter if we have 2056 or sorry 256 or 2000. Yeah. Yeah, that by itself
 is one of the cool properties again of your paper. The fact that it can scale up to sort of very
 high resolutions. There are artifacts appearing, but they are not nearly as much as in other
 other models. Looks pretty cool. Yeah, it doesn't scale up perfectly, but yeah, it's better than
 fully convolutional architectures. Cool. Yeah. So where do you think I mean, maybe you don't want
 to disclose necessarily but what is the plan for the future? We don't know where we get
 throughout research. But yeah, the most obvious thing here is that we can try to improve the way
 generalized to high resolutions. And the second point is that we are trying to understand why
 actually it works that because it yeah, it has lots of components. And we conducted an ablation
 study regarding if validating if each of these components matter, but this is just a surface.
 And we can go more in depth in that. And we are not satisfied with our laws because it's
 that huge. There are many components that we need to balance. And we want better laws with just one,
 just one button, make everything work. And nice. So yeah, I mean, I was almost I was expecting you
 to say we're not happy with our loss. We want more. We want like more components to make it. But it's
 I think it's pretty cool that the goal is also to make a system that's kind of as good but simpler.
 I think they'll make it also much more accessible. Yeah, I think that's a good idea.
 Yeah. I think they'll make it also much more accessible. Cool. Yeah. Roman, Elisa, sorry, Lisa.
 Is that correct? Yes. Okay, Lisa and Roman, thank you so much for being here.
 Was it was a pleasure? Do you have any last criticisms to the video? Or shout outs? No,
 thank you very much for for the discussion. It was really fun. And thank you for your channel,
 because it is you make a real good job in helping others to be in time and to catch with
 the this huge wave of information that we have in the field. Thanks. Thanks. Yeah, thank you. Thank you.