NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (ML Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper clearml nuwa nüwa visual pretraining pretraining vision models igpt image gpt autoregressive autoregressive image gpt autoregressive image generation nearby self-attention 3dna 3d nearby self-attention transformer transformer for videos deep learning on videos deep learning video generation video manipulation text to image text to video microsoft
#nuwa #microsoft #generative

NÜWA is a unifying architecture that can ingest text, images, and videos and brings all of them into a quantized latent representation to support a multitude of visual generation tasks, such as text-to-image, text-guided video manipulation, or sketch-to-video. This paper details how the encoders for the different modalities are constructed, and how the latent representation is transformed using their novel 3D nearby self-attention layers. Experiments are shown on 8 different visual generation tasks that the model supports.

OUTLINE:
0:00 - Intro & Outline
1:20 - Sponsor: ClearML
3:35 - Tasks & Naming
5:10 - The problem with recurrent image generation
7:35 - Creating a shared latent space w/ Vector Quantization
23:20 - Transforming the latent representation
26:25 - Recap: Self- and Cross-Attention
28:50 - 3D Nearby Self-Attention
41:20 - Pre-Training Objective
46:05 - Experimental Results
50:40 - Conclusion & Comments

Paper: https://arxiv.org/abs/2111.12417
Github: https://github.com/microsoft/NUWA

Sponsor: ClearML
https://clear.ml

Abstract:
This paper presents a unified multimodal pre-trained model called NÜWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate NÜWA on 8 downstream tasks. Compared to several strong baselines, NÜWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, video prediction, etc. Furthermore, it also shows surprisingly good zero-shot capabilities on text-guided image and video manipulation tasks. Project repo is this https URL.

Authors: Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, Nan Duan

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there. Today we'll look at NUA, Visual Synthesis Pre-Training for Neuro-Visual World Creation. This is by researchers of Microsoft Research Asia and Peking University. The paper presents a model that can support a wide variety of image generation tasks such as text to image where you give a piece of text and you get an image. This is a dog with goggles staring at the camera up to something like video manipulation where you want to change the frames of a video according to a piece of text. For example, the car is reversing instead of the car is driving forward. Now you see there's not always text in the loop. Sometimes it's just an image, sometimes it's a sketch, sometimes it's just a video. So all of these kinds of tasks are supported by this model and this paper goes into how the model's architecture is done, specifically how a transformer architecture, essentially an attention mechanism, is able to handle such large data points, essentially contexts not only going to images but beyond images to multiple frames of video. Hey, this video is sponsored by ClearML. ClearML is an MLop stack that is fully open source, it can do experiment tracking, experiment orchestration, deployment, it has model and feature stores, it is a complete package of ML tools. Now what I want to highlight in particular here is this self hosted tier. Self hosted is a first class citizen for ClearML. Everything's open source, therefore, you can look at it, you can audit it, you can extend it however you want and you can host it on your servers. There's also a free tier that is available in the cloud so you can get started with whatever you need in the cloud and then once you need more features you can go to a more professional setup if you don't want a self host. If you love open source then ClearML might be the way to go. It is an end-to-end stack from experimentation all the way to serving, it's vertically integrated, makes your life a whole lot easier and it is appropriate whether you are an individual running experiments or an entire team. Now one of the core pieces of ClearML is of course their experiment tracker. It's super easy to set up, it needs like a single line of code, I guess that's two lines but you know who cares. It integrates with pretty much any tool there is and not only does it record your metrics like you're used to, it also fully grabs all the console output of your experiments, it grabs any artifacts that the run might have produced and most importantly it clearly records not only your hyper parameters but also the other parameters of your environment such as the path and the machine you ran it on and your dependencies. Another really cool feature is that it allows you to compare different experiments for example here it shows you what part of their configuration was different so you're able to pretty quickly figure out what made the difference in any particular run and of course you can grab a bunch of experiments together and then analyze them next to each other. So now there's no excuse anymore for blaming your tools, any fault in your machine learning project will be yours and yours alone if you use ClearML. Isn't that a promise? So I invite you to go over and check it out at clear.ml and thanks so much to ClearML for sponsoring this video and let's get into it. So yeah we'll go into the paper we'll see how they do it. I do find this opening thing right here is a little bit overstated because a lot of these things aren't coming out of the same model but the model is then fine-tuned on different things and I also find the paper is a bit unclear on some of the details and if I understand correctly there is no code yet that we can look at maybe that's going to be released maybe not who knows. To the name Nua is you know there's this Umlaut which we do have in German but I don't believe this is a German inspired name or any sort of Nordic language. I do believe this comes from the symbol in pinyin that also is represented as an Umlaut on the U. It took me like so long to figure out that you have to type a V in pinyin to get that output. I just couldn't spell words like Nü for a long time but now I can so I do believe this is pronounced Nua but correct me if I'm wrong. Also many thanks to Andreas who helped me prepare this paper a little bit, gave me some inputs this is very much appreciated. Follow Andreas on Twitter he also often posts updates for our paper discussions on Discord so very helpful thank you. Alright let's get into it. So this model is something like an image GPT model. If you know image GPT, image GPT is essentially similar like a pixel RNN where you have an image you want to produce the image sort of pixel by pixel left to right top to bottom you produce just one pixel after another after another after another and you learn this how you would learn a language model essentially just pixel by pixel and you can support tasks like completing images where you simply give you everything here you are already set pre-computed and you simply let the model in for these pixels right here or you can support things like image manipulation by simply you have a picture right here and or I'll say that's the cat and you simply cut out part of the image so you cut out this part or something you let the model fill it in so you could do things like in painting or something like this this is supported by image GPT now the problem with something like image GPT is that if you want to have this as sort of a language generation task then your context size is you know if you predict the pixel on the bottom right here the context is like all the pixels in the rest of the image that you've already generated and if you have something like a 200 by 200 image that is for thousand previous pixels now four thousand is just about no is it it's forty thousand sorry sorry about that forty thousand that is definitely outside of the scope of every transformer that we have and beyond that if we now look at video and video is essentially just a stack of images right so you have an image frame the next frame and the next frame if you look at that if you want to produce a single pixel right here not only do you have to take into account all of the pixels of the image that you've generated so far but also all of the pixels of the previous frames that you've generated so far right and that definitely blows the context of any transformer that is infeasible so this model here very much is about how do we make this feasible the answer is going to be a twofold first of all we're going to encode all of the data into a common space that is kind of discrete in latent space and is way less dimensional and the second answer is going to be we're going to use a local attention in order to work in this latent space and finally generate the output so this is an overview over the model I do find it a little bit lacking as a picture but you can see that in general we use these encoders and the encoders they take care of bringing the data whatever the data is into a common representation right here the common representation is going to be a essentially a three-dimensional cube where each element is an embedding vector but we're going to look at that now so how do we encode text our goal is to our goal is going to be to have a latent space to have an encoder for any kind of data and after the encoder the data should be in sort of a latent space and that latent space should be if possible kind of discrete or quantized and we're going to use we're going to use some methods that already exist but for text that's pretty easy for text the encoder is simply it can be the identity function right because if I have a piece of text like a cat whatever if I tokenize that text that is already tokens so right now if we make if we do language modeling or any sort of language processing the first step is tokenizing the text and then associating each token with an embedding vector so this is going to be nice it's going to be a set or a sequence of tokens and that's exactly the representation that we want so for text everything is good we have a sequence of tokens we have a code book usually which is sometimes in case language modeling that's called the embedding matrix that's at the beginning of the model so every every code vector every token is associated with a vector so we can look up that vector in the code book replace the token by the vector and then process the tokens as vector embeddings in the subsequent model we want to do the same with images right we want to get an image and we want to bring it into the latent space as a set of discrete quantized tokens luckily there is a technique how you can do that and that's called the VQ VAE so if I have an image let's say in our cat what I want to do is I want to have an encoder such that it results in a set of latent tokens now a VQ VAE is interesting because what the result is going to be is going to be it's going to be like an image but that image is going to be very low dimensional so here we may have 200 by 200 but over here in this case we have like 3 by 3 and these aren't in fact pixels but they are tokens so these will be vector quantized there will be a code book they call it B and that code book will be vectors for each token and what the encoder does is it essentially reduces the image down to a representation that is 3 by 3 and then every single pixel in that 3 by 3 matrix every single entry right here is going to be clamped to the nearest entry in the code book that's the quantization step if you if you don't know much about this you can look up vector quantized vector quantized anything pretty much but vector quantized VAE is sort of the the main reference right here it's the encoder encodes in a continuous fashion and then there is a discontinuous step a discrete step where we say okay we there is there's latent space and we have this code book vectors here and they're going to live in that latent space as vectors as points in that latent space and if my encoder encodes an image and I take any pixel right here and that pixel might come to be here I don't use the pixel or I don't use this latent token as is I'm going to clamp it to the value directly of that code book vector so all I end up with is a selection of these code book vectors so at each point here there will be one of those code book vectors and I can equivalently say if I like number them this is one two three four I can equivalently say these are essentially tokens so token one might be this might be this might be one this might be two and two three four four four four right and from this I can then have a decoder again that produces back an image and the image of course is now only produced from this latent encoding you might think that is way restrictive but it actually turns out to be very very powerful so instead of using the exact encoding we use the quantized encoding and if our code book is large enough you know you can encode quite a number of things like if you have a thousand tokens you can imagine token one could be you know there's it there's kind of a tree and token two is like a tree stump and token three is like well a tree that is like a has needles like a needle needle like a pine and so on and then your latent description here just kind of roughly outlines the broad shape of the image so not necessarily exactly what's where but just says like you know in the top right there's a bunch of pine trees and in the bottom right there's a road and so on so it's it's a latent tokenized or latent discrete tokenized representation of the image here and you can already see that this is way beneficial because now we're only working in a nine diamond sorry in nine tokens whereas here it's 200 by 200 now we don't have to forget that each of the also each of these tokens obviously is going to be associated with the vectors with a vector so this is not nine dimensional space but it's nine times whatever the vector dimension is that is associated with each token as you know like this is not 200 by 200 it's actually 200 by 200 by 3 since every pixel has a vector of dimension 3 associated to represent color right this VQ VAE is trained as an is if I understand correctly this is the first part where the model that the paper isn't exactly clear what happens right here I'm not sure whether this is trained end to end or whether they train the encoder and decoder here ahead of time because they have different formulations they say like after training this we do that and I'm not sure but essentially they train it like so here is how you obtain the latent representation you send an image that's I through the encoder that's E and then you select the Z the these are the latent vectors sector Z or the now these are the tokens the token indices such that you select the Z according to what's the closest vector from the code book from the code book B so you can see that J are the indices into the code book so the Z will be for for a token I what is Z I will be what entry in the code book vector is closest to that representation that the encoder produced and then the reconstructed image I had is simply going to be and I'll go with my latent representation to the code book I actually get out the vectors the entries of the code book I shove that into the decoder which is G the generator I guess and that gives me the reconstructed image so how am I gonna train this it's easy I want that my produced image is close to the original image right here I also want to train the code book which is B to be close to what my encoder produces so I want the code book to be useful and that means the code book needs to be able to serve just describe the things that the encoder produces right so the code I'm gonna draw the code book closer to the encoders output right here the SG is a stop gradient which means that this part of the loss affects the code book but also we have the symmetric part right here where we're going to teach the encoder to produce things that are better encodable by the code book so here the stop gradient is on the code book which means that this part of the loss affects the encoder it's quite common to split up two losses even though this could be in one loss right since it's symmetric it's quite common to split it up into two parts each one having a stop gradient makes things more stable all right so is this actually yeah probably it's it's just a framework framework specifics right here I don't think s SG is a valid mathematical thing anywhere this really refers to the stop gradient functions in in tensorflow or in pi torch in addition to that they say well the VQ VAE is sort of too strict a little bit so there is an extension called VQ GAN that changes the VQ VAE objective a little bit so they say they add two things right here one is a GAN loss which I'm going to guess is this one right here so you can see they introduce a discriminator that discriminates between real and fake images and I'm going to guess that that here is the loss for the discriminator right because you want the discriminator to recognize real from fake which means you need I and I hat but I don't see I don't see the loss that would be added to the generator because the generators loss I don't think that would necessarily include the true image but I might be wrong because yeah so I mean that the generator would simply not care about the first part right there if even if you included it but you know they introduce a discriminator which we know can help and they also say they introduce a perceptual loss and they simply write this down as we're going to pass both the original image and the generated image through a CNN and then we compare the two this is in contrast to comparing the two images directly as you can see they say that this is to meant to ease the exact constraints between I and I had and focus on high level semantic matching I don't exactly know what these CNNs are if they are trained as well or if they simply take like an off-the-shelf ResNet 50 past the images through and compare the last layers in in order to say well I just want the latent representations to be similar I don't actually want the images to be similar they also don't say whether that replaces this this loss up here or whether that's simply in addition to that loss again we don't know they further they further say that you could do the same thing for videos right you could train like a VQ VAE VQ GAN for videos because after all videos are just a stack here that we saw a stack of of images but they say that didn't work out well so what they do is they simply treat each frame of the video as an image and they pass each frame through this image encoder right here and they simply stack the outputs or they stack the latent representations so that'd be from the first frame then from the second frame from the third frame and so on they stack them like this and that gives you sort of a tensor now keep in mind every single entry right here for example this entry or this entry or this entry every single entry is associated with a vector so this is ultimately and going to end up in a four-dimensional latent tensor that you work with but we can represent it as a three-dimensional tensor of tokens where each token will be an entry in the codebook so how is that a common representation we saw so the text is 1d of tokens or 2d if you consider it as vectors images are 2d as tokens but 3d as vectors and video is 3d as tokens and 4d as vectors how can we make sense of this and we combine all of this by simply introducing a dummy dimensions so if you've ever in like numpy you know you index your vector sorry your vector X with like you know I want everything everything and none right that that's one way you can also use the expand dims or unsqueeze in pytorch or anything like this to make it compatible and essentially use the broadcasting functionality of the frameworks that's essentially what they do here they say you know we have an image we have the latent representation we simply add the placeholder dimension of one since images have no temporal dimension it's just height and width but for videos this one would be I guess not a one so if you can bring them into the same space by using dummy dimensions and broadcasting if necessary so now everything essentially is a 4d latent tensor you can bring in text you can bring in images you can bring in videos the next thing we want to do and again I don't know if these are pre trained the encoder decoder or if these are trained jointly I I don't know the next thing we want to know is okay right now this is simply encoding and then if we ship the representation through the decoder it's right so if we ship it through the encoder and then through the decoder it's going to result in the same image or in a very similar image right so here is going to be like another cat like how does that help us obviously there needs to be something different right we want an image right here I put it through the encoder when I get its latent representation and then we need to do something something with the latent representation get another latent representation then decode that and then we get some sort of a different result right so a different resulting image right here so this is the same for like image completion and so on the question obviously is what happens right here now there is where the sort of the the transform or the attention layers come in until now we've had classic I think these are these are conv nets and so on these encoders decoders like you would be used to if these are images but now what we do is we have essentially a model that transforms the that transforms the latent representation to do meaningful work okay so how is that how is that done they differentiate two things right here they differentiate context which is here on the left broadly which they always or sometimes denote with large C context here and as context they count things like input text or input sketches and the reason it's context is because those things aren't output those things are never given in completely the model will never have to produce them you always input them either you input them or you don't input them but if you do input those things it's conditioning information that the model can look at as a whole right you always enter the full text or the full sketch you never enter like half a sketch the model can't produce sketches the model can only produce images or image frames frames of a video okay so that is the decoder is only images encoders can be for text for images and for sketches so the part over here they would generally call the output y even if like half of it is actual input into the algorithm so here you can see the input is the part of an image and the output is the remaining part of that image or the input is the video frame the output is the future frames right so yeah so that is the output part this should remind you sort of of the original transformer architecture so the sequence to sequence task is you have sort of sequence one and that is always given in full and then you have sequence two that sequence two that maybe maybe you are given not nothing at all or you're sort of given an initial initial token right here or you're given kind of a prefix of what you have to generate and then you have to go on completing sequence two now if you don't have sequence one at all that's a decoder only architecture that's also possible you can condition on nothing but the most general architecture has these two sequences if you remember the original transformer it was exactly like this and then wait let me pull this down a bit and then it had sort of a stack of transfer of attention layers here and a stack of attention layers right here and what you do is within the attention blocks you'd had like self attention where things attend to each other attention here attention attention attention and then inside this block you'd had attention also by with itself but then also you'd had layers where attention would go from the why part so from the output part to the context part so you would let the output right here in a layer collect information from the context by doing what they call cross attention in the original transformer paper I think it's still called cross attention right here both are the same operation both are both are attention operations it's just a matter you always have a queries and keys sorry that's an E keys and values if it's self attention all of these are generated from the same input and if it's not self attention then this for example is generated from the Y input and these two are generated from the context information and that essentially means that Y is requesting information from C so Y is looking is attending to information in C okay same thing here what they have this layer called 3DNA now that's the entire layer name is 3DNA that is 3D nearby self-attention okay so they say this is based on the previous 3D data representation so 3D they essentially mean 4D but 3D tokenized and then each token has a vector as a vector but there the 3D comes in when they do when they discuss how they do their attention by nearby they essentially mean local attention so what they're going to do is they're going to do local attention in this 3D tensor that is I think what I what I could gather so far they formulate this in a general way right here so what you'll do is you'll define this for two tensors X and C and sometimes those are the same and sometimes not so specifically X can be either C in which case it's self-attention or X can be Y in which case it is cross attention from Y to C I guess C could also be Y in which case it is self-attention from Y to Y so yeah I'll just make it a little bit confusing right here in any case it's just a matter of how you compute the how you compute the keys the values and the queries as you can see the queries are the queries are always computed from the entire the queries are always computed from the entire vector or vector tensor X so whatever is producing the query the entire thing is producing the query however for the keys and values what you do is you define a local neighborhood so now we care specifically about how do I produce Y at location ijk you have to imagine we have this 3d representation which is essentially a big cube that cubes elements are these tokens right so this is you can imagine it as a just stack of video frames but in latent space right so in latent space we have this stack of video frames of the latent encodings of the video frames if it's just a single image right you broadcast and so on but in in that case we wonder how from this we need to produce sort of the next layers representation which is also going to be a cube just like it so as much as in an attention layer the input is a sequence of tokens the output is the sequence of tokens as well in this it's the input is a I guess a cube of tokens and the output is again a cube of tokens so how we're going to do that we have and we produce the output for each location we define a neighborhood so if we want to predict this this would be Y at ijk we're going to search ijk over here which is going to be I guess right here okay so this is ijk the same location then we're going to define a local neighborhood around that thing so that could be just it's again going to be a cube like this that is just a little bit bigger and they are using as far as I can tell they're using three by three by three cubes right here so they're going to define a neighborhood and while the queries are generated from sort of the entirety right here of the from the entirety of the tensor the keys and values are only going to be computed from that cube so instead if this is height width and height no this is s let's call that as the temporal dimension and width even though this is already in the latent space it would still be very very expensive to compute self-attention or cross-attention when every single element of the cube attends to every single other element right that's essentially what we'd have to do in an attention layer in text I have a sequence and every sort of every part of the sequence is able to attend to every single other part of the sequence that is not feasible if you have a 3d cube even if it's in a lower dimensional latent space so what I'm going to do is I'm going to say okay if I want to if I want to compute this output right here I can only attend to a local neighborhood around this output here so that's that's that so the queries I can compute once for the whole tensor but then if I so that's I can compute the queries for the whole tensor but if I want to produce a particular location the only place I can attend to is the keys and values of a particular local neighborhood so essentially that piece of the cube here can only look at the local neighborhood around its locations in order to aggregate information that is its local local attention either local cross-attention or local self-attention so we define the neighborhood and produce the query for a particular location I'm not sure if that should be X I JK or not hmm not sure but yeah you can see that the the keys and the values are certainly specific to a location they include this neighborhood right here this n neighborhood the n neighborhood is defined as this set right here which is simply what I just said that that cube and then I compute the softmax simply as and this is I think there's a mistake right here this should be this should definitely be not here this should definitely be here yeah so I'll compute the softmax like I would in the outer product between queries and keys just in that neighborhood and then aggregating the values according to what the softmax of the routing table gives me and that's how I produce this output right here okay so I can do that all in parallel I can essentially produce that next tensor right here of the latent representation and yeah that's that now I just said I produce it all by the way there is a you can see that reduces the complexity from sort of this square to simply every location attending to its local neighborhood so that reduces the complexity by quite a bit so for every location that's this part I have to attend to its local neighborhood that's this part there's also a positional encodings as you can see right here and what we're going to do we're going to first have a stack of layers of self attention for the context like we saw in the original transformer so we're first going to have a stack of L layers right here and after that we're going to have a stack of L layers here and each of those L layers can do either self attention or cross attention but as far as I can tell it's it's kind of different than the original transformer because here you can see the next layer here is produced from the last layers and likewise here if I produce the eye the next layer is produced from the last layers of Y but also from cross attention from the last layer of like to the L layer of C which means that it it only can look at the output layer so the arrows I've drawn here can technically not happen but it always has to look at like the output layer up here I guess that's a way to do it I don't think that's the exact same thing as in the original transformer where you really have as I shown the arrows here it sort of attend to the same height I might also be wrong in this or it's a wrong formula right here that is also completely possible now you can see there is I've masked this there is also this part right here so what we're going to use is we're going to use causal attention so we're only going to attend I said you can do it all at the same time you have to do a causal mask you know like in things like GPT where I produce one token at a time when I produce this token right here I'm only allowed to look at the token that I've already produced and that's the exact same right here in fact we're going to produce this representation we're going to start like at the top left at time step one and we're going to produce the whole image at time step one pixel or not pixel by pixel but element by element in this representation and then we're going to once that is complete that video frame let's say we're going to go to the next step and again do it element by element so this is really a giant autoregressive model now you can with causal attention you can you can train at the same time but during inference you only actually attend to the things in front of you this formula in fact doesn't doesn't exactly I don't is this is this correct because here it says everything needs to be smaller which to me would mean that you know if I'm let's let's just make it for 2d and let's just say it's smaller i smaller j is the question of if I produce this pixel right here technically I should have access to everything up here and the row so far right but with this formula what it would mean is that I have access to only whatever is to the top left of me like this part right here and I don't think that's the case I think this is just sloppy notation right here see ya this denote the generated tokens for now that I don't think is correct to express it like this seems shady it's all it also doesn't tell us exactly in which order the pixels are produced though I think it's first within a time step and then across time steps so yeah that is that is that now let's get to the training objective so I hope you can see that this is one layer of this three DNA and we have L layers here and L I think is 24 in their models we have L layers on for the context and then also L layers of cross and self attention and ultimately we end up up here with the final representation and training we can do in parallel with causal masking but inference we have to do element by element so that's why they praise that their model is reasonably fast but I think it's still like 50 seconds to produce one one image or something like this and that's why so the training objective and here is a little bit where they they yeah where again I I find it to be quite unclear so they say they train it on three tasks and if I understand correctly they train on these three tasks simultaneously so they have three different data sets one is a text to image data set where you can see right here you produce an image and you condition on text okay you and you can see that this lower than T simply means the elements or the tokens lower than T and you go from T equals one until height times width so it's an image so it only has these two dimensions so and you produce I guess pixel by pixel see that that I don't I don't know what what does why mean here if it's really the output why then you know you have that generator here and the generator probably doesn't go pixel by pixel that I don't know maybe it does maybe it actually does in any case you have these three tasks so one is text to image from a data set that does that one is video prediction where you simply input a piece of a video here the C here that is like a no-op so that is the special word none so because you know you still have to input something but if you have no text conditioning you simply input a dummy and then the loss goes over also over the time steps and there is also text to video where you'd input text and video so far and you'd output the rest of the frames so that is yeah again so here probably the loss doesn't necessarily go across all the time steps since part of the video is already given but yeah I guess we'll have to wait for the code to see what really turns out most notably you can see that the conditioning information right here is sometimes it's video right because it's it sometimes video is kind of conditioning implicitly by also already being part of the output but there is no for example sketch conditioning right here it's always either text or nothing and this is pre training so that means everything you see to do with sketch is then fine-tuned so that that was my when I first saw this I thought like oh wow they you know train these jointly everything's joint and then the same model can do all of these tasks and it turns out no actually most of these things are then fine-tuned down the line now they do show that the fun the pre training actually helps quite a bit but you have to understand these are in fact fine-tuned also you can immediately see that something like a video manipulation it's not actually video manipulation like the model doesn't care about that about these frames right here that the car what the car is doing the model doesn't even see this you simply input the first frame and then you let it generate the next frames based on this text right here so it's not necessarily manipulation as much as I give you the beginning of a video and a piece of text and now please predict the video based on the text it's a bit like this here except you already have the first frame if if I understand correctly but I think I think I do there's really no other way I guess I'm not sure maybe they actually into input into maybe they input it into the context right here but I cannot imagine that in any case maybe I completely misunderstand this right here but these are the tasks they give some implementation detail about how the how the latent spaces or you can see that there's a latent space of dimension 1280 yeah the local neighborhood is of size 3 by 3 by 3 or 3 by 3 by 1 for images when there are lonely images and it's the regular attention mechanism if it is text alright so that is it and these the next slides are results experimental results I want to highlight a few so here are things they can do they compare for example with Dalí which is a model that is explicitly trained to produce images from text right whereas this model right here is sort of a multi-purpose model and you can see that in general either the results are comparable or better I mean it's this is at this point is kind of argue arguable you can measure it on certain data sets for example here they they specifically praise this picture right here where they say ah this is very clear and consistent and this other state-of-the-art model is not as not as good I do like some of these outputs right here playing golf on grass the baseline model you can see the baseline model just just screws up though I do think there aren't many days for some tasks there are just no no baselines available because they kind of invented them themselves but you can see that when there is baselines available the baselines usually they either yeah they don't necessarily do so well either so this case this is doesn't really seem to be yeah I guess it's some kind of a human ish thing but this you know looks looks fairly neat and you can see the resolution is also bigger than the resolutions of the competitors that's that's pretty cool you can also as I said this is now fine-tuned right if you actually want the sketch to image or sketch to anything you are going to have to fine-tune it on that data set but if you do you can see that the results are very very cool very accurate this is the input when I guess that green thing here is the vehicle class or even the bus class and yeah the outputs are are pretty convincing honestly so yeah if you if you want you can look at the metrics yourself they have a bunch of more more examples right here as we said specifically things like in painting are doing are quite possible right now so you can say I want to only produce so I want to clamp everything to the original image except this region right here you can give a piece of conditioning text and that together will this so this is newer this is the baseline right here will as you can see fill in the missing pixels in order to also match up with the text because it's been trained on text to image data sets yeah lastly this video manipulation which was one of the sort of appraisals of this paper right here you can see the raw video on top the first row is the divers swimming to the surface that's given to the model so the model is asked to manipulate the video in that way that we're swimming to the bottom or the diver is flying to the sky which surprisingly the model can do as well again I think I think the model simply gets the first frame and then needs to continue the video I don't think the rest of the video has given us conditioning information but I might be wrong right so in if I'm right it would not necessarily be video manipulation but more kind of like video completion conditioned on text but still is pretty cool alright so yeah they have a by the way they have a big appendix they also compare like different local attention mechanisms they have much more output right here yeah some sometimes it's it's very funny but I hope the code is out soon or is already out and I just haven't hadn't found it as a conclusion they say they present newer unified pre-trained model that can generate new or manipulate existing images and videos for eight visual synthesis tasks again caveat here is that only very few only like two or three of those are actually zero shot maybe or resulting from the pre-training for the rest you actually have to fine-tune several contributions are made including a general 3d encoder decoder framework covering text images and videos at the same time that's what we saw is possible by doing this essentially it it's a it's a VQ GAN for images for text it's already in the correct representation and for for videos they simply say well every frame is an image so it's like a general encoder decoder framework covering text images and videos is let's say it's a nice formulation a nearby sparse attention mechanism that considers the nearby characteristic of both spatial and temporal axes that is simply local attention so this nearby sparse attention it simply is local attention they simply do it over the three axes instead of over one axis where local attention was originally presented and third comprehensive experiments on eight synthesis tasks yeah that is that is what they do this our first step towards building an AI platform to enable visual world creation and help content creators yeah I can imagine that like models like these are gonna be pretty powerful for content creators if you can if you can essentially input arbitrary arbitrary modalities and mix them together it's gonna be pretty cool alright so that was a new war let me know what you think and I'll see you next time bye bye
 Hi there. Today we'll look at NUA, Visual Synthesis Pre-Training for Neuro-Visual
 World Creation. This is by researchers of Microsoft Research Asia and Peking
 University. The paper presents a model that can support a wide variety of image
 generation tasks such as text to image where you give a piece of text and you
 get an image. This is a dog with goggles staring at the camera up to something
 like video manipulation where you want to change the frames of a video
 according to a piece of text. For example, the car is reversing instead of the car
 is driving forward. Now you see there's not always text in the loop. Sometimes
 it's just an image, sometimes it's a sketch, sometimes it's just a video. So
 all of these kinds of tasks are supported by this model and this paper
 goes into how the model's architecture is done, specifically how a transformer
 architecture, essentially an attention mechanism, is able to handle such large
 data points, essentially contexts not only going to images but beyond images
 to multiple frames of video. Hey, this video is sponsored by ClearML. ClearML
 is an MLop stack that is fully open source, it can do experiment tracking,
 experiment orchestration, deployment, it has model and feature stores, it is a
 complete package of ML tools. Now what I want to highlight in particular here is
 this self hosted tier. Self hosted is a first class citizen for ClearML.
 Everything's open source, therefore, you can look at it, you can audit it, you
 can extend it however you want and you can host it on your servers. There's
 also a free tier that is available in the cloud so you can get started with
 whatever you need in the cloud and then once you need more features you can go
 to a more professional setup if you don't want a self host. If you love open
 source then ClearML might be the way to go. It is an end-to-end stack from
 experimentation all the way to serving, it's vertically integrated, makes your
 life a whole lot easier and it is appropriate whether you are an
 individual running experiments or an entire team. Now one of the core pieces
 of ClearML is of course their experiment tracker. It's super easy to set up, it
 needs like a single line of code, I guess that's two lines but you know who cares.
 It integrates with pretty much any tool there is and not only does it record
 your metrics like you're used to, it also fully grabs all the console output of
 your experiments, it grabs any artifacts that the run might have produced and
 most importantly it clearly records not only your hyper parameters but also the
 other parameters of your environment such as the path and the machine you ran
 it on and your dependencies. Another really cool feature is that it allows you
 to compare different experiments for example here it shows you what part of
 their configuration was different so you're able to pretty quickly figure out
 what made the difference in any particular run and of course you can
 grab a bunch of experiments together and then analyze them next to each other. So
 now there's no excuse anymore for blaming your tools, any fault in your
 machine learning project will be yours and yours alone if you use ClearML.
 Isn't that a promise? So I invite you to go over and check it out at clear.ml
 and thanks so much to ClearML for sponsoring this video and let's get into it.
 So yeah we'll go into the paper we'll see how they do it. I do find this
 opening thing right here is a little bit overstated because a lot of these things
 aren't coming out of the same model but the model is then fine-tuned on different
 things and I also find the paper is a bit unclear on some of the details and
 if I understand correctly there is no code yet that we can look at maybe
 that's going to be released maybe not who knows. To the name Nua is you know
 there's this Umlaut which we do have in German but I don't believe this is a
 German inspired name or any sort of Nordic language. I do believe
 this comes from the symbol in pinyin that also is represented as an
 Umlaut on the U. It took me like so long to figure out that you have to type a V
 in pinyin to get that output. I just couldn't spell words like Nü for a long
 time but now I can so I do believe this is pronounced Nua but correct me if
 I'm wrong. Also many thanks to Andreas who helped me prepare this paper a
 little bit, gave me some inputs this is very much appreciated. Follow Andreas on
 Twitter he also often posts updates for our paper discussions on Discord so
 very helpful thank you. Alright let's get into it. So this model is something like
 an image GPT model. If you know image GPT, image GPT is essentially similar
 like a pixel RNN where you have an image you want to produce the image sort of
 pixel by pixel left to right top to bottom you produce just one pixel after
 another after another after another and you learn this how you would learn a
 language model essentially just pixel by pixel and you can support tasks like
 completing images where you simply give you everything here you are already set
 pre-computed and you simply let the model in for these pixels right here or
 you can support things like image manipulation by simply you have a
 picture right here and or I'll say that's the cat and you simply cut out
 part of the image so you cut out this part or something you let the model fill
 it in so you could do things like in painting or something like this this is
 supported by image GPT now the problem with something like image GPT is that if
 you want to have this as sort of a language generation task then your
 context size is you know if you predict the pixel on the bottom right here the
 context is like all the pixels in the rest of the image that you've already
 generated and if you have something like a 200 by 200 image that is for
 thousand previous pixels now four thousand is just about no is it it's
 forty thousand sorry sorry about that forty thousand that is definitely
 outside of the scope of every transformer that we have and beyond that
 if we now look at video and video is essentially just a stack of images right
 so you have an image frame the next frame and the next frame if you look at
 that if you want to produce a single pixel right here not only do you have to
 take into account all of the pixels of the image that you've generated so far
 but also all of the pixels of the previous frames that you've generated so
 far right and that definitely blows the context of any transformer that is
 infeasible so this model here very much is about how do we make this feasible
 the answer is going to be a twofold first of all we're going to encode all of
 the data into a common space that is kind of discrete in latent space and is
 way less dimensional and the second answer is going to be we're going to use
 a local attention in order to work in this latent space and finally generate
 the output so this is an overview over the model I do find it a little bit
 lacking as a picture but you can see that in general we use these encoders
 and the encoders they take care of bringing the data whatever the data is
 into a common representation right here the common representation is going to
 be a essentially a three-dimensional cube where each element is an embedding
 vector but we're going to look at that now so how do we encode text our goal is
 to our goal is going to be to have a latent space to have an encoder for any
 kind of data and after the encoder the data should be in sort of a latent space
 and that latent space should be if possible kind of discrete or quantized
 and we're going to use we're going to use some methods that already exist but
 for text that's pretty easy for text the encoder is simply it can be the identity
 function right because if I have a piece of text like a cat whatever if I
 tokenize that text that is already tokens so right now if we make if we do
 language modeling or any sort of language processing the first step is
 tokenizing the text and then associating each token with an embedding vector so
 this is going to be nice it's going to be a set or a sequence of tokens and
 that's exactly the representation that we want so for text everything is good
 we have a sequence of tokens we have a code book usually which is sometimes in
 case language modeling that's called the embedding matrix that's at the beginning
 of the model so every every code vector every token is associated with a vector
 so we can look up that vector in the code book replace the token by the vector
 and then process the tokens as vector embeddings in the subsequent model we
 want to do the same with images right we want to get an image and we want to
 bring it into the latent space as a set of discrete quantized tokens luckily
 there is a technique how you can do that and that's called the VQ VAE so if I have
 an image let's say in our cat what I want to do is I want to have an encoder
 such that it results in a set of latent tokens now a VQ VAE is interesting
 because what the result is going to be is going to be it's going to be like an
 image but that image is going to be very low dimensional so here we may have 200
 by 200 but over here in this case we have like 3 by 3 and these aren't in
 fact pixels but they are tokens so these will be vector quantized there will be a
 code book they call it B and that code book will be vectors for each token and
 what the encoder does is it essentially reduces the image down to a
 representation that is 3 by 3 and then every single pixel in that 3 by 3
 matrix every single entry right here is going to be clamped to the nearest
 entry in the code book that's the quantization step if you if you don't
 know much about this you can look up vector quantized vector quantized
 anything pretty much but vector quantized VAE is sort of the the main
 reference right here it's the encoder encodes in a continuous fashion and then
 there is a discontinuous step a discrete step where we say okay we there is
 there's latent space and we have this code book vectors here and they're
 going to live in that latent space as vectors as points in that latent space
 and if my encoder encodes an image and I take any pixel right here and that
 pixel might come to be here I don't use the pixel or I don't use this latent
 token as is I'm going to clamp it to the value directly of that code book vector
 so all I end up with is a selection of these code book vectors so at each point
 here there will be one of those code book vectors and I can equivalently say
 if I like number them this is one two three four I can equivalently say these
 are essentially tokens so token one might be this might be this might be one
 this might be two and two three four four four four right and from this I can
 then have a decoder again that produces back an image and the image of course is
 now only produced from this latent encoding you might think that is way
 restrictive but it actually turns out to be very very powerful so instead of
 using the exact encoding we use the quantized encoding and if our code book
 is large enough you know you can encode quite a number of things like if you
 have a thousand tokens you can imagine token one could be you know there's it
 there's kind of a tree and token two is like a tree stump and token three is like
 well a tree that is like a has needles like a needle needle like a pine and so
 on and then your latent description here just kind of roughly outlines the broad
 shape of the image so not necessarily exactly what's where but just says like
 you know in the top right there's a bunch of pine trees and in the bottom
 right there's a road and so on so it's it's a latent tokenized or latent
 discrete tokenized representation of the image here and you can already see that
 this is way beneficial because now we're only working in a nine diamond sorry in
 nine tokens whereas here it's 200 by 200 now we don't have to forget that each
 of the also each of these tokens obviously is going to be associated with
 the vectors with a vector so this is not nine dimensional space but it's nine
 times whatever the vector dimension is that is associated with each token as
 you know like this is not 200 by 200 it's actually 200 by 200 by 3 since
 every pixel has a vector of dimension 3 associated to represent color right
 this VQ VAE is trained as an is if I understand correctly this is the first
 part where the model that the paper isn't exactly clear what happens right
 here I'm not sure whether this is trained end to end or whether they train
 the encoder and decoder here ahead of time because they have different
 formulations they say like after training this we do that and I'm not sure but
 essentially they train it like so here is how you obtain the latent
 representation you send an image that's I through the encoder that's E and then
 you select the Z the these are the latent vectors sector Z or the now these
 are the tokens the token indices such that you select the Z according to
 what's the closest vector from the code book from the code book B so you can see
 that J are the indices into the code book so the Z will be for for a token I
 what is Z I will be what entry in the code book vector is closest to that
 representation that the encoder produced and then the reconstructed image I had
 is simply going to be and I'll go with my latent representation to the code
 book I actually get out the vectors the entries of the code book I shove that
 into the decoder which is G the generator I guess and that gives me the
 reconstructed image so how am I gonna train this it's easy I want that my
 produced image is close to the original image right here I also want to train
 the code book which is B to be close to what my encoder produces so I want the
 code book to be useful and that means the code book needs to be able to serve
 just describe the things that the encoder produces right so the code I'm
 gonna draw the code book closer to the encoders output right here the SG is a
 stop gradient which means that this part of the loss affects the code book but
 also we have the symmetric part right here where we're going to teach the
 encoder to produce things that are better encodable by the code book so here
 the stop gradient is on the code book which means that this part of the loss
 affects the encoder it's quite common to split up two losses even though this
 could be in one loss right since it's symmetric it's quite common to split it
 up into two parts each one having a stop gradient makes things more stable all
 right so is this actually yeah probably it's it's just a framework framework
 specifics right here I don't think s SG is a valid mathematical thing anywhere
 this really refers to the stop gradient functions in in tensorflow or in pi
 torch in addition to that they say well the VQ VAE is sort of too strict a
 little bit so there is an extension called VQ GAN that changes the VQ VAE
 objective a little bit so they say they add two things right here one is a GAN
 loss which I'm going to guess is this one right here so you can see they
 introduce a discriminator that discriminates between real and fake
 images and I'm going to guess that that here is the loss for the discriminator
 right because you want the discriminator to recognize real from fake which means
 you need I and I hat but I don't see I don't see the loss that would be added
 to the generator because the generators loss I don't think that would
 necessarily include the true image but I might be wrong because yeah so I mean
 that the generator would simply not care about the first part right there if even
 if you included it but you know they introduce a discriminator which we know
 can help and they also say they introduce a perceptual loss and they
 simply write this down as we're going to pass both the original image and the
 generated image through a CNN and then we compare the two this is in contrast
 to comparing the two images directly as you can see they say that this is to
 meant to ease the exact constraints between I and I had and focus on high
 level semantic matching I don't exactly know what these CNNs are if they are
 trained as well or if they simply take like an off-the-shelf ResNet 50 past
 the images through and compare the last layers in in order to say well I just
 want the latent representations to be similar I don't actually want the images
 to be similar they also don't say whether that replaces this this loss up
 here or whether that's simply in addition to that loss again we don't
 know they further they further say that you could do the same thing for videos
 right you could train like a VQ VAE VQ GAN for videos because after all videos
 are just a stack here that we saw a stack of of images but they say that
 didn't work out well so what they do is they simply treat each frame of the
 video as an image and they pass each frame through this image encoder right
 here and they simply stack the outputs or they stack the latent representations
 so that'd be from the first frame then from the second frame from the third
 frame and so on they stack them like this and that gives you sort of a tensor
 now keep in mind every single entry right here for example this entry or
 this entry or this entry every single entry is associated with a vector so
 this is ultimately and going to end up in a four-dimensional latent tensor that
 you work with but we can represent it as a three-dimensional tensor of tokens
 where each token will be an entry in the codebook so how is that a common
 representation we saw so the text is 1d of tokens or 2d if you consider
 it as vectors images are 2d as tokens but 3d as vectors and video is 3d as
 tokens and 4d as vectors how can we make sense of this and we combine all of this
 by simply introducing a dummy dimensions so if you've ever in like numpy you know
 you index your vector sorry your vector X with like you know I want everything
 everything and none right that that's one way you can also use the expand dims or
 unsqueeze in pytorch or anything like this to make it compatible and
 essentially use the broadcasting functionality of the frameworks that's
 essentially what they do here they say you know we have an image we have
 the latent representation we simply add the placeholder dimension of one since
 images have no temporal dimension it's just height and width but for videos
 this one would be I guess not a one so if you can bring them into the same
 space by using dummy dimensions and broadcasting if necessary so now
 everything essentially is a 4d latent tensor you can bring in text you can
 bring in images you can bring in videos the next thing we want to do and again I
 don't know if these are pre trained the encoder decoder or if these are trained
 jointly I I don't know the next thing we want to know is okay right now this is
 simply encoding and then if we ship the representation through the decoder it's
 right so if we ship it through the encoder and then through the decoder it's
 going to result in the same image or in a very similar image right so here is
 going to be like another cat like how does that help us obviously there needs
 to be something different right we want an image right here I put it through the
 encoder when I get its latent representation and then we need to do
 something something with the latent representation get another latent
 representation then decode that and then we get some sort of a different result
 right so a different resulting image right here so this is the same for like
 image completion and so on the question obviously is what happens right here now
 there is where the sort of the the transform or the attention layers come
 in until now we've had classic I think these are these are conv nets and so on
 these encoders decoders like you would be used to if these are images but now
 what we do is we have essentially a model that transforms the that transforms
 the latent representation to do meaningful work okay so how is that how
 is that done they differentiate two things right here they differentiate
 context which is here on the left broadly which they always or sometimes
 denote with large C context here and as context they count things like input
 text or input sketches and the reason it's context is because those things
 aren't output those things are never given in completely the model will
 never have to produce them you always input them either you input them or you
 don't input them but if you do input those things it's conditioning
 information that the model can look at as a whole right you always enter the
 full text or the full sketch you never enter like half a sketch the model can't
 produce sketches the model can only produce images or image frames frames
 of a video okay so that is the decoder is only images encoders can be for text
 for images and for sketches so the part over here they would generally call the
 output y even if like half of it is actual input into the algorithm so here
 you can see the input is the part of an image and the output is the remaining
 part of that image or the input is the video frame the output is the future
 frames right so yeah so that is the output part this should remind you sort
 of of the original transformer architecture so the sequence to sequence
 task is you have sort of sequence one and that is always given in full and
 then you have sequence two that sequence two that maybe maybe you are given not
 nothing at all or you're sort of given an initial initial token right here or
 you're given kind of a prefix of what you have to generate and then you have
 to go on completing sequence two now if you don't have sequence one at all that's
 a decoder only architecture that's also possible you can condition on nothing
 but the most general architecture has these two sequences if you remember the
 original transformer it was exactly like this and then wait let me pull this down
 a bit and then it had sort of a stack of transfer of attention layers here and a
 stack of attention layers right here and what you do is within the attention
 blocks you'd had like self attention where things attend to each other
 attention here attention attention attention and then inside this block
 you'd had attention also by with itself but then also you'd had layers where
 attention would go from the why part so from the output part to the context part
 so you would let the output right here in a layer collect information from the
 context by doing what they call cross attention in the original transformer
 paper I think it's still called cross attention right here both are the same
 operation both are both are attention operations it's just a matter you always
 have a queries and keys sorry that's an E keys and values if it's self attention
 all of these are generated from the same input and if it's not self attention
 then this for example is generated from the Y input and these two are generated
 from the context information and that essentially means that Y is requesting
 information from C so Y is looking is attending to information in C okay same
 thing here what they have this layer called 3DNA now that's the entire layer
 name is 3DNA that is 3D nearby self-attention okay so they say this is
 based on the previous 3D data representation so 3D they essentially
 mean 4D but 3D tokenized and then each token has a vector as a vector but
 there the 3D comes in when they do when they discuss how they do their
 attention by nearby they essentially mean local attention so what they're
 going to do is they're going to do local attention in this 3D tensor that is I
 think what I what I could gather so far they formulate this in a general way
 right here so what you'll do is you'll define this for two tensors X and C and
 sometimes those are the same and sometimes not so specifically X can be
 either C in which case it's self-attention or X can be Y in which
 case it is cross attention from Y to C I guess C could also be Y in which case
 it is self-attention from Y to Y so yeah I'll just make it a little bit
 confusing right here in any case it's just a matter of how you compute
 the how you compute the keys the values and the queries as you can see the
 queries are the queries are always computed from the entire the queries are
 always computed from the entire vector or vector tensor X so whatever is
 producing the query the entire thing is producing the query however for the keys
 and values what you do is you define a local neighborhood so now we care
 specifically about how do I produce Y at location ijk you have to imagine we
 have this 3d representation which is essentially a big cube that cubes
 elements are these tokens right so this is you can imagine it as a just stack
 of video frames but in latent space right so in latent space we have this
 stack of video frames of the latent encodings of the video frames if it's
 just a single image right you broadcast and so on but in in that case we wonder
 how from this we need to produce sort of the next layers representation which is
 also going to be a cube just like it so as much as in an attention layer the
 input is a sequence of tokens the output is the sequence of tokens as well in
 this it's the input is a I guess a cube of tokens and the output is again a cube
 of tokens so how we're going to do that we have and we produce the output for
 each location we define a neighborhood so if we want to predict this this would
 be Y at ijk we're going to search ijk over here which is going to be I guess
 right here okay so this is ijk the same location then we're going to define a
 local neighborhood around that thing so that could be just it's again going to
 be a cube like this that is just a little bit bigger and they are using as
 far as I can tell they're using three by three by three cubes right here so
 they're going to define a neighborhood and while the queries are generated
 from sort of the entirety right here of the from the entirety of the tensor the
 keys and values are only going to be computed from that cube so instead if
 this is height width and height no this is s let's call that as the temporal
 dimension and width even though this is already in the latent space it would
 still be very very expensive to compute self-attention or cross-attention when
 every single element of the cube attends to every single other element
 right that's essentially what we'd have to do in an attention layer in text I
 have a sequence and every sort of every part of the sequence is able to attend
 to every single other part of the sequence that is not feasible if you
 have a 3d cube even if it's in a lower dimensional latent space so what I'm
 going to do is I'm going to say okay if I want to if I want to compute this
 output right here I can only attend to a local neighborhood around this output
 here so that's that's that so the queries I can compute once for the whole
 tensor but then if I so that's I can compute the queries for the whole tensor
 but if I want to produce a particular location the only place I can attend to
 is the keys and values of a particular local neighborhood so essentially that
 piece of the cube here can only look at the local neighborhood around its
 locations in order to aggregate information that is its local local
 attention either local cross-attention or local self-attention so we define the
 neighborhood and produce the query for a particular location I'm not sure if that
 should be X I JK or not
 hmm not sure but yeah you can see that the the keys and the values are
 certainly specific to a location they include this neighborhood right here
 this n neighborhood the n neighborhood is defined as this set right here which
 is simply what I just said that that cube and then I compute the softmax
 simply as and this is I think there's a mistake right here this should be this
 should definitely be not here this should definitely be here yeah so I'll
 compute the softmax like I would in the outer product between queries and keys
 just in that neighborhood and then aggregating the values according to what
 the softmax of the routing table gives me and that's how I produce this output
 right here okay so I can do that all in parallel I can essentially produce that
 next tensor right here of the latent representation and yeah that's that now
 I just said I produce it all by the way there is a you can see that reduces the
 complexity from sort of this square to simply every location attending to its
 local neighborhood so that reduces the complexity by quite a bit so for every
 location that's this part I have to attend to its local neighborhood that's
 this part there's also a positional encodings as you can see right here and
 what we're going to do we're going to first have a stack of layers of self
 attention for the context like we saw in the original transformer so we're first
 going to have a stack of L layers right here and after that we're going to have
 a stack of L layers here and each of those L layers can do either self
 attention or cross attention but as far as I can tell it's it's kind of different
 than the original transformer because here you can see the next layer here is
 produced from the last layers and likewise here if I produce the eye the
 next layer is produced from the last layers of Y but also from cross
 attention from the last layer of like to the L layer of C which means that it it
 only can look at the output layer so the arrows I've drawn here can technically
 not happen but it always has to look at like the output layer up here I guess
 that's a way to do it I don't think that's the exact same thing as in the
 original transformer where you really have as I shown the arrows here it sort
 of attend to the same height I might also be wrong in this or it's a wrong
 formula right here that is also completely possible now you can see
 there is I've masked this there is also this part right here so what we're going
 to use is we're going to use causal attention so we're only going to attend
 I said you can do it all at the same time you have to do a causal mask you
 know like in things like GPT where I produce one token at a time when I
 produce this token right here I'm only allowed to look at the token that I've
 already produced and that's the exact same right here in fact we're going to
 produce this representation we're going to start like at the top left at time
 step one and we're going to produce the whole image at time step one pixel or
 not pixel by pixel but element by element in this representation and then
 we're going to once that is complete that video frame let's say we're going
 to go to the next step and again do it element by element so this is really a
 giant autoregressive model now you can with causal attention you can you can
 train at the same time but during inference you only actually attend to
 the things in front of you this formula in fact doesn't doesn't exactly I don't
 is this is this correct because here it says everything needs to be smaller
 which to me would mean that you know if I'm let's let's just make it for 2d and
 let's just say it's smaller i smaller j is the question of if I produce this
 pixel right here technically I should have access to everything up here and
 the row so far right but with this formula what it would mean is that I
 have access to only whatever is to the top left of me like this part right here
 and I don't think that's the case I think this is just sloppy notation right
 here see ya this denote the generated tokens for now that I don't think is
 correct to express it like this seems shady it's all it also doesn't tell us
 exactly in which order the pixels are produced though I think it's first
 within a time step and then across time steps so yeah that is that is that now
 let's get to the training objective so I hope you can see that this is one layer
 of this three DNA and we have L layers here and L I think is 24 in their models
 we have L layers on for the context and then also L layers of cross and self
 attention and ultimately we end up up here with the final representation and
 training we can do in parallel with causal masking but inference we have to
 do element by element so that's why they praise that their model is reasonably
 fast but I think it's still like 50 seconds to produce one one image or
 something like this and that's why so the training objective and here is a
 little bit where they they yeah where again I I find it to be quite unclear so
 they say they train it on three tasks and if I understand correctly they
 train on these three tasks simultaneously so they have three
 different data sets one is a text to image data set where you can see right
 here you produce an image and you condition on text okay you and you can
 see that this lower than T simply means the elements or the tokens lower than T
 and you go from T equals one until height times width so it's an image so
 it only has these two dimensions so and you produce I guess pixel by pixel see
 that that I don't I don't know what what does why mean here if it's really the
 output why then you know you have that generator here and the generator
 probably doesn't go pixel by pixel that I don't know maybe it does maybe it
 actually does in any case you have these three tasks so one is text to image from
 a data set that does that one is video prediction where you simply input a
 piece of a video here the C here that is like a no-op so that is the special
 word none so because you know you still have to input something but if you have
 no text conditioning you simply input a dummy and then the loss goes over also
 over the time steps and there is also text to video where you'd input text and
 video so far and you'd output the rest of the frames so that is yeah again so
 here probably the loss doesn't necessarily go across all the time
 steps since part of the video is already given but yeah I guess we'll have to
 wait for the code to see what really turns out most notably you can see that
 the conditioning information right here is sometimes it's video right because
 it's it sometimes video is kind of conditioning implicitly by also already
 being part of the output but there is no for example sketch conditioning right
 here it's always either text or nothing and this is pre training so that means
 everything you see to do with sketch is then fine-tuned so that that was my when
 I first saw this I thought like oh wow they you know train these jointly
 everything's joint and then the same model can do all of these tasks and it
 turns out no actually most of these things are then fine-tuned down the line
 now they do show that the fun the pre training actually helps quite a bit but
 you have to understand these are in fact fine-tuned also you can immediately see
 that something like a video manipulation it's not actually video
 manipulation like the model doesn't care about that about these frames right here
 that the car what the car is doing the model doesn't even see this you simply
 input the first frame and then you let it generate the next frames based on
 this text right here so it's not necessarily manipulation as much as I
 give you the beginning of a video and a piece of text and now please predict the
 video based on the text it's a bit like this here except you already have the
 first frame if if I understand correctly but I think I think I do there's really
 no other way I guess I'm not sure maybe they actually into input into maybe they
 input it into the context right here but I cannot imagine that in any case maybe
 I completely misunderstand this right here but these are the tasks they give
 some implementation detail about how the how the latent spaces or you can see
 that there's a latent space of dimension 1280 yeah the local neighborhood is of
 size 3 by 3 by 3 or 3 by 3 by 1 for images when there are lonely images and
 it's the regular attention mechanism if it is text alright so that is it and
 these the next slides are results experimental results I want to highlight
 a few so here are things they can do they compare for example with Dalí which
 is a model that is explicitly trained to produce images from text right whereas
 this model right here is sort of a multi-purpose model and you can see that
 in general either the results are comparable or better I mean it's this is
 at this point is kind of argue arguable you can measure it on certain data sets
 for example here they they specifically praise this picture right here where
 they say ah this is very clear and consistent and this other state-of-the-art
 model is not as not as good I do like some of these outputs right here playing
 golf on grass the baseline model you can see the baseline model just just screws
 up though I do think there aren't many days for some tasks there are just no
 no baselines available because they kind of invented them themselves but you can
 see that when there is baselines available the baselines usually they
 either yeah they don't necessarily do so well either so this case this is doesn't
 really seem to be yeah I guess it's some kind of a human ish thing but this you
 know looks looks fairly neat and you can see the resolution is also bigger than
 the resolutions of the competitors that's that's pretty cool you can also
 as I said this is now fine-tuned right if you actually want the sketch to image
 or sketch to anything you are going to have to fine-tune it on that data set
 but if you do you can see that the results are very very cool very accurate
 this is the input when I guess that green thing here is the vehicle class or
 even the bus class and yeah the outputs are are pretty convincing honestly so
 yeah if you if you want you can look at the metrics yourself they have a bunch
 of more more examples right here as we said specifically things like in
 painting are doing are quite possible right now so you can say I want to only
 produce so I want to clamp everything to the original image except this region
 right here you can give a piece of conditioning text and that together will
 this so this is newer this is the baseline right here will as you can see
 fill in the missing pixels in order to also match up with the text because it's
 been trained on text to image data sets yeah lastly this video manipulation which
 was one of the sort of appraisals of this paper right here you can see the raw
 video on top the first row is the divers swimming to the surface that's given to
 the model so the model is asked to manipulate the video in that way that
 we're swimming to the bottom or the diver is flying to the sky which
 surprisingly the model can do as well again I think I think the model simply
 gets the first frame and then needs to continue the video I don't think the
 rest of the video has given us conditioning information but I might be
 wrong right so in if I'm right it would not necessarily be video manipulation but
 more kind of like video completion conditioned on text but still is pretty
 cool alright so yeah they have a by the way they have a big appendix they also
 compare like different local attention mechanisms they have much more output
 right here yeah some sometimes it's it's very funny but I hope the code is out
 soon or is already out and I just haven't hadn't found it as a conclusion
 they say they present newer unified pre-trained model that can generate new
 or manipulate existing images and videos for eight visual synthesis tasks again
 caveat here is that only very few only like two or three of those are actually
 zero shot maybe or resulting from the pre-training for the rest you actually
 have to fine-tune several contributions are made including a general 3d encoder
 decoder framework covering text images and videos at the same time that's what
 we saw is possible by doing this essentially it it's a it's a VQ GAN for
 images for text it's already in the correct representation and for for
 videos they simply say well every frame is an image so it's like a general
 encoder decoder framework covering text images and videos is let's say it's a
 nice formulation a nearby sparse attention mechanism that considers the
 nearby characteristic of both spatial and temporal axes that is simply local
 attention so this nearby sparse attention it simply is local attention
 they simply do it over the three axes instead of over one axis where local
 attention was originally presented and third comprehensive experiments on eight
 synthesis tasks yeah that is that is what they do this our first step towards
 building an AI platform to enable visual world creation and help content creators
 yeah I can imagine that like models like these are gonna be pretty powerful for
 content creators if you can if you can essentially input arbitrary arbitrary
 modalities and mix them together it's gonna be pretty cool alright so that was
 a new war let me know what you think and I'll see you next time bye bye