[ML News] Text-to-Image models are taking over! (Imagen, DALL-E 2, Midjourney, CogView 2 & more)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper imagen dalle dalle 2 dall e dall e 2 midjourney midjourney diffusion generative models ai art aiart mlnews ml news kilcher news ml news yannic google imagen cogview cog view cog view 2 dalle mini dalle-mini dalle mega
#mlnews #dalle #imagen 

All things text-to-image models like DALL-E and Imagen!

OUTLINE:
0:00 - Intro
0:30 - Imagen: Google's Text-to-Image Diffusion Model
7:15 - Unified I/O by AllenAI
9:40 - CogView2 is Open-Source
11:05 - Google bans DeepFakes from Colab
13:05 - DALL-E generates real Cosmopolitan cover
15:45 - DALL-E tips & tricks
17:00 - Midjourney moves to Open Beta
17:50 - DALLE-mini is not Crayon
19:00 - Deep Learning Resources

AMENDMENTS:
The Unified-IO paper is here: https://arxiv.org/abs/2206.08916

References:
Imagen: Google's Text-to-Image Diffusion Model
https://imagen.research.google/?utm_source=pocket_mylist
https://arxiv.org/pdf/2205.11487.pdf

Unified I/O by AllenAI
https://unified-io.allenai.org/
https://blog.allenai.org/introducing-ai2s-unified-io-9c0ec7fe1e43

CogView2 is Open-Source
https://github.com/THUDM/CogView2
file:///Users/yk/Downloads/big.1.pdf
https://huggingface.co/spaces/THUDM/CogView2
https://arxiv.org/pdf/2204.14217.pdf

Google bans DeepFakes from Colab
https://www-vice-com.cdn.ampproject.org/c/s/www.vice.com/amp/en/article/v7v4gx/google-bans-deepfakes-from-its-machine-learning-platform?utm_source=pocket_mylist

DALL-E generates real Cosmopolitan cover
https://www.cosmopolitan.com/lifestyle/a40314356/dall-e-2-artificial-intelligence-cover/
https://www.instagram.com/p/CfEwohiJdXW/?hl=en

DALL-E tips & tricks
https://twitter.com/GuyP/status/1544710725708513280?s=09&t=c3NpErPx80INQVeaWkIqIg&utm_source=pocket_mylist
https://twitter.com/GuyP/status/1552681939806691329?s=09&t=LV2ChcukUziXfvfNK-sY0A&utm_source=pocket_mylist
https://twitter.com/GuyP/status/1547234780001042432
https://dallery.gallery/the-dalle-2-prompt-book/

Midjourney moves to Open Beta
https://twitter.com/midjourney?lang=en
https://twitter.com/search?q=%23midjourney&f=image

DALLE-mini is not Crayon
https://www.craiyon.com/

Deep Learning Resources
https://github.com/jacobhilton/deep_learning_curriculum
https://arxiv.org/abs/2206.13446
https://arxiv.org/pdf/2206.13446.pdf

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Google releases imagine an unprecedented text to image model, Cogview 2 improves drastically over Cogview 1 and mid journey moves into open beta. Welcome to ML News. Welcome to ML News. Today, we talk all about text to image models, text and image models, any sort of artistic models that we might have missed and developments over this summer. The first obviously really big one that we've actually missed at the time is imagine imagine is a system by Google, specifically Google Research out of Toronto that is a diffusion model that goes from text to images. Here you can see a bunch of examples. So this is an alien octopus floating through a portal reading a newspaper and this is not some sort of image to image model, the image is created purely from the text, which is crazy. So I hope you see that over the last few years or even months, this quality of text to image models has improved drastically. I think ever since the first Dalí model kind of sparked this push into this area, the rate of progress has been unprecedented. Look at the quality of these things. And also the adherence to text is quite amazing. Now not only is the quality really good, what's also really stunning is the simplicity of these models. We see a continued progression from more complicated systems to actually less complicated systems. So the entire imagine system is just captured in this diagram right here. At the beginning, you have a text that goes into a frozen text encoder. So the text encoder isn't even trained with the model. It's simply used as is from being trained as a pure text model. The text embedding is then fed into a text to image diffusion model. Now diffusion models have gained in popularity in also the last few months competing in quality with autoregressive models. So this is a really cool development where systems like Dalí to use the conglomeration of like latent diffusion and so on. This model simply takes the text embedding feeds it into this diffusion model generates a low resolution 64 by 64 image and then feeds that into super resolution diffusion models. In fact, there are two stages of super resolution. The first one going to 256 by 256 and then the second one going to 1024 by 1024. Now obviously, this is a cool tactic because super resolution models can be trained in a very unsupervised way, you simply take a large image, you sample it down to a smaller image and you train the model to go in the reverse direction. Now while recent progression is definitely in the direction of simplicity and scale, you can't just scale up and be simple and expect that to work. Well, there are actually distinct things you can do to make these models work a lot better. And the imagined paper points out a few of those things. For example, we show that large pre trained frozen text encoders are very effective. And in fact, we show that scaling the pre trained text encoder size is more important than scaling the diffusion model size, which is really interesting because you would think that for an image generation model, the part that actually generates the image is really important, but it's actually the part that pays attention to the text and what's contained in the text that seems to be more benefiting from scale. So the quality and adherence to the prompt that we see in this model is thanks in large part to scaling up the text part of the model. Another thing they also mentioned as being a core contributor to the good quality is what they call a dynamic thresholding diffusion sampler, which enables the use of a very large classifier free guidance weights. Now there are a bunch of technical terms if you haven't followed this literature, essentially in diffusion models, what you do is you have this model that you feed the same image over and over and in each step of that feeding, the image gets a little bit more clear, a little bit more denoise. So you train the model to go from noise to image in sort of a recursive step. Now in each part of that recursion, obviously you generate a new image, you generate each pixel of the image in a given value. Now if you know things about images, you know that usually pixel values go either from zero to 255 or negative one to one or you know, however you specify it, but there is a minimum and maximum value for each pixel. And usually this is only important at the end when you actually want to have the output image, you need to crop it somehow to that range or squeeze it or something like this during the intermediate steps, you have multiple options, you can simply let the system run rampant and have pixel values in whatever like this pixel is 10,334.2 or at each step, you can try to limit it to some range and compress the image. Now both of these options, if you do them in a static way, don't really seem appealing and that's what this paper notices. So they introduce a technique to dynamically threshold to dynamically reduce the range of pixels during the recursive steps in the middle of the diffusion process. In the paper, they describe this in a bit more detail, they say that at each sampling step, they don't just threshold to like a fixed value, but they threshold to a percentile of the absolute pixel values in the image, and then dynamically crop the pictures to that value and then compress that to a range of negative one to one. They say that we find that dynamic thresholding results in significantly better photorealism as well as better image text alignment, especially when using very large guidance weights. So there's another thing if you haven't followed this literature, there is this concept of classifier free guidance, which is a bit of a hack. So the way it works is that this model trains to go from text to image. So every procedure, every generation is conditioned on a piece of text. However, you can do a trick namely during training, you sometimes just leave away the text yet you still try to generate the same image and that teaches the model to just unconditionally generate images without the help of the text. And then at inference time, here's the trick, what you do is you take the text, you take the text encoding and you run two generations in parallel, one of them, you actually feed the text encoding. So that's the real one, the conditioned one. And one of them, you don't feed the text encoding, but the same kind of input noise otherwise, and you let that process run. Now at any intermediate step, now you have a clear diff between what happens if I add the text and what happens if from the same starting point, I simply generate the image without that text. So you have a diff like a vector between the two images. And what you can do now is you can simply scale that up, you can simply say, well, more of that, which presumably leads you into a direction of more conditioning on that text. So people find that this increases the amount by which the model pays attention to the text, naturally. However, that comes with its set of problems. And one of them is more saturated pixels, more pixels out of range and less photorealism because these pixels usually get cropped, the dynamic thresholding helps with that. So I'm sorry, that was a bit of a long winded explanation. However, they do state that this is a core contributor to the quality of their outputs. If you want to learn more, the papers called photorealistic text image diffusion models with deep language understanding. The Allen Institute for AI releases unified IO, which is a general purpose model with what they claim unprecedented breadth that can perform a wide array of visual and linguistic tasks. So the mission here is to cover all kinds of tasks. For example, image generation, region captioning, pose estimation, detection, segmentation, segmentation based generation, you get the idea, there's a lot of tasks that a single model covers. And what does it do? It simply defines encoders and decoders of each of these modalities to a unified token vocabulary. So whether it's images, whether it's text, whether it's anything, their goal is to translate this from and to a unified set of tokens over which they can run our very classic token based NLP autoregressive models. We have a bunch of examples here. So one class of tasks they can handle is image plus text to image. Now image plus text, you might think of descriptions to photographs, but you can do so much more if you simply formulate it correctly. This is very much in the style of something like t five. So for example, if you think of segmentation based generation, the input image isn't a photo but it's the segmentation map and the input text isn't a description but it's kind of like a task description generate an image for this segmentation and then an annotation. So this is part of the problem what the colors mean, the model maps both the image and the text to its latent vocabulary and the output is an image in this case the generated image. Now another class of models is for example, image plus text to text. So for example, the task of region captioning has an image and inside the image there is a bounding box bounding boxes can also naturally be translated to like x and y positions, width and height into a set of redefined tokens and the text describes the tasks to be done. What does the highlighted region describe the output is a piece of text you get the idea the model is sort of trained on all of these tasks and all of these tasks are mapped to a unified language a unified set of tokens and that enables the model to essentially cross learn all of these different things and benefit from the data of all the tasks that might or might not be related. So there is a blog post and the paper isn't out yet but it says it's coming late on 616 which is about one and a half months ago so we're all holding our breaths. CogView 2 is a new model from researchers of Tsinghua University that is also a text to image model. Now CogView 2 is a model that works in English and Chinese it is open there is a hugging face demo available and it focuses mainly on improving performance over the previous system called CogView 1. So the paper that is called faster and better text to image generation via hierarchical transformers goes a lot into detail on how they improve the model since the last iteration and again you can see that the quality and adherence to text of these models is really picking up in steam. So the way that CogView 2 improves in performance and also in quality is by using a sequence of transformations and instead of having fully autoregressive models they have partially bidirectional models. So in multiple stages they train the model to only fill in local parts of the image while attending to all the other image tokens. This allows them to support some degree of bidirectionality while also decoupling some of the generations via local attention so you're able to generate multiple parts of the image at the same time. For example in their super resolution steps as you can see here you can create a lot of the things in parallel which gives a great increase in inference speed. There is a demo on hugging face spaces if you want to play around with it, I'll link it in the description. Motherboard writes Google bans deepfakes from its machine learning platform. So apparently a lot of people have used colabs to generate deepfakes and Google now disallows that use of colab. A lot of people have asked like how are they going to do that? How are they going to inspect the code that you run or something like this? The way I understand it is that as of now it's simply the terms of use of colab prohibit you from running deepfake software. So if you run code like this you'd simply be violating your contract with Google. How and when and how strictly they're actually going to check what code you are running that I think is not described currently. I can imagine that they are going to simply ban the commonly shared colabs that people you know kind of share around to generate deepfakes. A lot of the people who do this kind of stuff they don't really have an idea even of how colabs work or what the code means they simply know how to fill in the stuff and then click play. So that should weed out like a large part of users of this technology. Now while obviously Google has the absolute right to do this, it gets a big gray in what counts as like deepfake software. There are obviously a lot of research projects and even a lot of fun projects that in one way of looking at them would fall under the guise of deepfake software but are completely harmless and there are other projects that might fall under this category depending on how loosely you define it. And the question is essentially how widely is this going to be applied. And as always, I guess we'll just have to wait for precedent cases. My hope is essentially that Google is going to take a quite strict approach to this in that if you try some new method to combine Mickey Mouse and Pikachu, then that doesn't necessarily count as a deepfake but we never know. It's always kind of scary when these companies introduce rules that are essentially up to their own mercy to decide what falls under them and what doesn't but I guess that's the entire tech industry. So yeah. Cosmopolitan has an article about itself, namely about how it designed one of its covers using Dulli. So the cosmopolitan issue is called the AI issue meet the world's first artificially intelligent magazine cover. This is a bit tongue in cheek. Obviously, the cover isn't really intelligent. However, it was created by OpenAI's Dulli 2 system. Now there is a video by the artist who made the cover detailing the entire process on brainstorming meeting with the team, then trying out different prompts getting closer and closer to the final result. And I think this highlights a core notion about these new text to image models. So as you can see here, it's not simply give me a cool Cosmo cover, it is trying and trying modifying the prompts trying again coming up with new ideas brainstorming. It's really kind of like almost like a collaboration between artists and these tools be that in prompt engineering be that in then modifying the image. As you know, Dulli cannot only generate images, it can also modify parts of existing images according to some text stuff. So the prompt that they came up with is a wide angle shot from below of a female astronaut with an athletic feminine body walking with swagger towards camera on Mars in an infinite universe synthwave digital art, it's only missing trending on Artstation, I guess, or Unreal Engine. But yeah, very cool insight. If you want to watch the video, it's Karen x Cheng on Instagram. And one thing that I noticed about this is the fact here, it says, and it only took 20 seconds to make now from the video you just saw, do you have the feeling that this thing only took 20 seconds to make like, no, that is a bit misleading. Obviously, the inference time of Dulli is 20 seconds, but then the entire process of making the cover is days, weeks, months, does not necessarily a replacement for the traditional artists. It's more like a replacement for the Photoshop person. I mean, watch me do this. Okay, right click, copy, give. All right, game is open paste. Cool colors. Saturation, crank that up, yo, bang, and boom, I have made a new magazine cover. If I told you that this magazine cover in its entirety only took 10 seconds to make because it literally took me 10 seconds to perform that sequence of actions, would you think that's an accurate representation of how this picture came to be? Probably not. But let's just forgive Cosmopolitan for the small amount of clickbait here and thank them for bringing the message of how AI can support creativity into the wider world. Speaking of working with Dulli, Guy Parsons on Twitter, that is at GUYP has a big thread on what he calls tips, tricks, games, experiments and combinations for Dulli and just kind of ideas of how you can interact with Dulli. Now this is targeted specifically towards Dulli but obviously this is also going to work for a lot of these other text to image systems as they all have very common bases, very common weaknesses and very common ways of interacting with them. Now he has more threads, for example, this one saying Dulli 2 generates amazing AI images but using these 10 free tools can make them so much better in which he goes into post processing essentially taking the things you get from Dulli and in various ways improving upon them, animating them, making them better, and so on. And on top of that, he also released a free 82 page book, the Dulli prompt book in which he summarizes and elaborates on all of these things in how you can interact with these text to image models in a efficient in a creative and in a more productive way. As I said, the book is available for free and if you are into a career of Dulli prompt engineer in the future, I definitely recommend you read it. Mid Journey has just recently announced that they're now moving to open beta, which essentially means that you can now join without an invite. Now if you are on Twitter, I'm sure you've seen mid journey generations they are super cool. If not, just search for hashtag mid journey on Twitter, and you're going to find like a lot of very amazing generations. This one's called the roots of infinity. Now mid journey is open but it's not free there is like a credit system. However, it is pretty affordable to run a few prompts and with the help of the previous resources you should be able to come up with quite creative prompts in order to test out the system. They also have an elaborate page of instructions and FAQs in order to help you get going and produce the best results possible. I've mentioned this one before, but Dulli mini is now called cry on notice the spelling it's C R A I Y O N. This after opening I was quite displeased with the naming conflict, Dulli mini being sort of very interchangeable with Dulli. So that gave the impression that the two had to do something with one another, which obviously they do as Dulli mini is an open source recreation of the Dulli system. However, Dulli mini has now been rebranded as crayon just to make it clear that it is its own project. Now the name Dulli mini is actually in another way not really descriptive as the system is now powered by the Dulli mega model. So the FAQ says the model used is called Dulli mini specifically the larger version also known as Dulli mega. So if you've used this and you've recently noticed a bit of a bump in performance, that's because the model has been upgraded and it's generally still fun to play around with these things. This is sunrise outdoor weightlifting. And also here you can apply any of the techniques we discussed before the model is also open source so if you don't want to wait for the servers or want to modify it or run it on your own, you can do so. Alright and just two quick helpful resources for this episode. One is the deep learning curriculum by Jacob Hilton, which is a curriculum like a set of resources that where you can learn about deep learning specifically about stuff that Jacob is interested in. This ranges from transformers scaling laws up to optimization, reinforcement learning, interpretability and more. There's also a set of links to other resources. So this in general is pretty helpful if you're kind of into machine learning into deep learning, but some topics you might want to expand your basic knowledge. And the other one is the pen and paper exercises in machine learning by Michael Guttman, which is on archive and is a PDF that goes over various things as it says it's pen and paper exercises. So one chapter for example is factor graphs and message passing. So you get a graphs, you get the factors, and you get an exercise mark the graph with arrows indicating all messages that need to be computed for the computation of P of x one, and there's a solution. So the PDF covers a lot of different areas as you can see right here linear algebra optimization directed graphical models, undirected graphical models, hidden Markov models, model based learning, sampling, and variational inference. Very cool 200 pages of gruesome exercises just for you. Alright, this was it for this week's ML news. I'm well aware that I've in no way covered or exhausted the space of text to image models or artistic models. There are a lot of things out there. I just wanted to give you a bit of an overview what happened in recent weeks. Let me know what you think in the comments. And as always, stay hydrated, and I'll see you next time. Bye bye.
 Google releases imagine an unprecedented text to image model, Cogview 2 improves drastically
 over Cogview 1 and mid journey moves into open beta.
 Welcome to ML News.
 Welcome to ML News.
 Today, we talk all about text to image models, text and image models, any sort of artistic
 models that we might have missed and developments over this summer.
 The first obviously really big one that we've actually missed at the time is imagine imagine
 is a system by Google, specifically Google Research out of Toronto that is a diffusion
 model that goes from text to images.
 Here you can see a bunch of examples.
 So this is an alien octopus floating through a portal reading a newspaper and this is not
 some sort of image to image model, the image is created purely from the text, which is
 crazy.
 So I hope you see that over the last few years or even months, this quality of text to image
 models has improved drastically.
 I think ever since the first Dalí model kind of sparked this push into this area, the rate
 of progress has been unprecedented.
 Look at the quality of these things.
 And also the adherence to text is quite amazing.
 Now not only is the quality really good, what's also really stunning is the simplicity of
 these models.
 We see a continued progression from more complicated systems to actually less complicated systems.
 So the entire imagine system is just captured in this diagram right here.
 At the beginning, you have a text that goes into a frozen text encoder.
 So the text encoder isn't even trained with the model.
 It's simply used as is from being trained as a pure text model.
 The text embedding is then fed into a text to image diffusion model.
 Now diffusion models have gained in popularity in also the last few months competing in quality
 with autoregressive models.
 So this is a really cool development where systems like Dalí to use the conglomeration
 of like latent diffusion and so on.
 This model simply takes the text embedding feeds it into this diffusion model generates
 a low resolution 64 by 64 image and then feeds that into super resolution diffusion models.
 In fact, there are two stages of super resolution.
 The first one going to 256 by 256 and then the second one going to 1024 by 1024.
 Now obviously, this is a cool tactic because super resolution models can be trained in
 a very unsupervised way, you simply take a large image, you sample it down to a smaller
 image and you train the model to go in the reverse direction.
 Now while recent progression is definitely in the direction of simplicity and scale,
 you can't just scale up and be simple and expect that to work.
 Well, there are actually distinct things you can do to make these models work a lot better.
 And the imagined paper points out a few of those things.
 For example, we show that large pre trained frozen text encoders are very effective.
 And in fact, we show that scaling the pre trained text encoder size is more important
 than scaling the diffusion model size, which is really interesting because you would think
 that for an image generation model, the part that actually generates the image is really
 important, but it's actually the part that pays attention to the text and what's contained
 in the text that seems to be more benefiting from scale.
 So the quality and adherence to the prompt that we see in this model is thanks in large
 part to scaling up the text part of the model.
 Another thing they also mentioned as being a core contributor to the good quality is
 what they call a dynamic thresholding diffusion sampler, which enables the use of a very large
 classifier free guidance weights.
 Now there are a bunch of technical terms if you haven't followed this literature, essentially
 in diffusion models, what you do is you have this model that you feed the same image over
 and over and in each step of that feeding, the image gets a little bit more clear, a
 little bit more denoise.
 So you train the model to go from noise to image in sort of a recursive step.
 Now in each part of that recursion, obviously you generate a new image, you generate each
 pixel of the image in a given value.
 Now if you know things about images, you know that usually pixel values go either from zero
 to 255 or negative one to one or you know, however you specify it, but there is a minimum
 and maximum value for each pixel.
 And usually this is only important at the end when you actually want to have the output
 image, you need to crop it somehow to that range or squeeze it or something like this
 during the intermediate steps, you have multiple options, you can simply let the system run
 rampant and have pixel values in whatever like this pixel is 10,334.2 or at each step,
 you can try to limit it to some range and compress the image.
 Now both of these options, if you do them in a static way, don't really seem appealing
 and that's what this paper notices.
 So they introduce a technique to dynamically threshold to dynamically reduce the range
 of pixels during the recursive steps in the middle of the diffusion process.
 In the paper, they describe this in a bit more detail, they say that at each sampling
 step, they don't just threshold to like a fixed value, but they threshold to a percentile
 of the absolute pixel values in the image, and then dynamically crop the pictures to
 that value and then compress that to a range of negative one to one.
 They say that we find that dynamic thresholding results in significantly better photorealism
 as well as better image text alignment, especially when using very large guidance weights.
 So there's another thing if you haven't followed this literature, there is this concept of
 classifier free guidance, which is a bit of a hack.
 So the way it works is that this model trains to go from text to image.
 So every procedure, every generation is conditioned on a piece of text.
 However, you can do a trick namely during training, you sometimes just leave away the
 text yet you still try to generate the same image and that teaches the model to just unconditionally
 generate images without the help of the text.
 And then at inference time, here's the trick, what you do is you take the text, you take
 the text encoding and you run two generations in parallel, one of them, you actually feed
 the text encoding.
 So that's the real one, the conditioned one.
 And one of them, you don't feed the text encoding, but the same kind of input noise otherwise,
 and you let that process run.
 Now at any intermediate step, now you have a clear diff between what happens if I add
 the text and what happens if from the same starting point, I simply generate the image
 without that text.
 So you have a diff like a vector between the two images.
 And what you can do now is you can simply scale that up, you can simply say, well, more
 of that, which presumably leads you into a direction of more conditioning on that text.
 So people find that this increases the amount by which the model pays attention to the text,
 naturally.
 However, that comes with its set of problems.
 And one of them is more saturated pixels, more pixels out of range and less photorealism
 because these pixels usually get cropped, the dynamic thresholding helps with that.
 So I'm sorry, that was a bit of a long winded explanation.
 However, they do state that this is a core contributor to the quality of their outputs.
 If you want to learn more, the papers called photorealistic text image diffusion models
 with deep language understanding.
 The Allen Institute for AI releases unified IO, which is a general purpose model with
 what they claim unprecedented breadth that can perform a wide array of visual and linguistic
 tasks.
 So the mission here is to cover all kinds of tasks.
 For example, image generation, region captioning, pose estimation, detection, segmentation,
 segmentation based generation, you get the idea, there's a lot of tasks that a single
 model covers.
 And what does it do?
 It simply defines encoders and decoders of each of these modalities to a unified token
 vocabulary.
 So whether it's images, whether it's text, whether it's anything, their goal is to translate
 this from and to a unified set of tokens over which they can run our very classic token
 based NLP autoregressive models.
 We have a bunch of examples here.
 So one class of tasks they can handle is image plus text to image.
 Now image plus text, you might think of descriptions to photographs, but you can do so much more
 if you simply formulate it correctly.
 This is very much in the style of something like t five.
 So for example, if you think of segmentation based generation, the input image isn't a
 photo but it's the segmentation map and the input text isn't a description but it's kind
 of like a task description generate an image for this segmentation and then an annotation.
 So this is part of the problem what the colors mean, the model maps both the image and the
 text to its latent vocabulary and the output is an image in this case the generated image.
 Now another class of models is for example, image plus text to text.
 So for example, the task of region captioning has an image and inside the image there is
 a bounding box bounding boxes can also naturally be translated to like x and y positions, width
 and height into a set of redefined tokens and the text describes the tasks to be done.
 What does the highlighted region describe the output is a piece of text you get the
 idea the model is sort of trained on all of these tasks and all of these tasks are mapped
 to a unified language a unified set of tokens and that enables the model to essentially
 cross learn all of these different things and benefit from the data of all the tasks
 that might or might not be related.
 So there is a blog post and the paper isn't out yet but it says it's coming late on 616
 which is about one and a half months ago so we're all holding our breaths.
 CogView 2 is a new model from researchers of Tsinghua University that is also a text
 to image model.
 Now CogView 2 is a model that works in English and Chinese it is open there is a hugging
 face demo available and it focuses mainly on improving performance over the previous
 system called CogView 1.
 So the paper that is called faster and better text to image generation via hierarchical
 transformers goes a lot into detail on how they improve the model since the last iteration
 and again you can see that the quality and adherence to text of these models is really
 picking up in steam.
 So the way that CogView 2 improves in performance and also in quality is by using a sequence
 of transformations and instead of having fully autoregressive models they have partially
 bidirectional models.
 So in multiple stages they train the model to only fill in local parts of the image while
 attending to all the other image tokens.
 This allows them to support some degree of bidirectionality while also decoupling some
 of the generations via local attention so you're able to generate multiple parts of
 the image at the same time.
 For example in their super resolution steps as you can see here you can create a lot of
 the things in parallel which gives a great increase in inference speed.
 There is a demo on hugging face spaces if you want to play around with it, I'll link
 it in the description.
 Motherboard writes Google bans deepfakes from its machine learning platform.
 So apparently a lot of people have used colabs to generate deepfakes and Google now disallows
 that use of colab.
 A lot of people have asked like how are they going to do that?
 How are they going to inspect the code that you run or something like this?
 The way I understand it is that as of now it's simply the terms of use of colab prohibit
 you from running deepfake software.
 So if you run code like this you'd simply be violating your contract with Google.
 How and when and how strictly they're actually going to check what code you are running that
 I think is not described currently.
 I can imagine that they are going to simply ban the commonly shared colabs that people
 you know kind of share around to generate deepfakes.
 A lot of the people who do this kind of stuff they don't really have an idea even of how
 colabs work or what the code means they simply know how to fill in the stuff and then click
 play.
 So that should weed out like a large part of users of this technology.
 Now while obviously Google has the absolute right to do this, it gets a big gray in what
 counts as like deepfake software.
 There are obviously a lot of research projects and even a lot of fun projects that in one
 way of looking at them would fall under the guise of deepfake software but are completely
 harmless and there are other projects that might fall under this category depending on
 how loosely you define it.
 And the question is essentially how widely is this going to be applied.
 And as always, I guess we'll just have to wait for precedent cases.
 My hope is essentially that Google is going to take a quite strict approach to this in
 that if you try some new method to combine Mickey Mouse and Pikachu, then that doesn't
 necessarily count as a deepfake but we never know.
 It's always kind of scary when these companies introduce rules that are essentially up to
 their own mercy to decide what falls under them and what doesn't but I guess that's the
 entire tech industry.
 So yeah.
 Cosmopolitan has an article about itself, namely about how it designed one of its covers using
 Dulli.
 So the cosmopolitan issue is called the AI issue meet the world's first artificially
 intelligent magazine cover.
 This is a bit tongue in cheek.
 Obviously, the cover isn't really intelligent.
 However, it was created by OpenAI's Dulli 2 system.
 Now there is a video by the artist who made the cover detailing the entire process on
 brainstorming meeting with the team, then trying out different prompts getting closer
 and closer to the final result.
 And I think this highlights a core notion about these new text to image models.
 So as you can see here, it's not simply give me a cool Cosmo cover, it is trying and trying
 modifying the prompts trying again coming up with new ideas brainstorming.
 It's really kind of like almost like a collaboration between artists and these tools be that in
 prompt engineering be that in then modifying the image.
 As you know, Dulli cannot only generate images, it can also modify parts of existing images
 according to some text stuff.
 So the prompt that they came up with is a wide angle shot from below of a female astronaut
 with an athletic feminine body walking with swagger towards camera on Mars in an infinite
 universe synthwave digital art, it's only missing trending on Artstation, I guess, or
 Unreal Engine.
 But yeah, very cool insight.
 If you want to watch the video, it's Karen x Cheng on Instagram.
 And one thing that I noticed about this is the fact here, it says, and it only took 20
 seconds to make now from the video you just saw, do you have the feeling that this thing
 only took 20 seconds to make like, no, that is a bit misleading.
 Obviously, the inference time of Dulli is 20 seconds, but then the entire process of
 making the cover is days, weeks, months, does not necessarily a replacement for the traditional
 artists.
 It's more like a replacement for the Photoshop person.
 I mean, watch me do this.
 Okay, right click, copy, give.
 All right, game is open paste.
 Cool colors.
 Saturation, crank that up, yo, bang, and boom, I have made a new magazine cover.
 If I told you that this magazine cover in its entirety only took 10 seconds to make
 because it literally took me 10 seconds to perform that sequence of actions, would you
 think that's an accurate representation of how this picture came to be?
 Probably not.
 But let's just forgive Cosmopolitan for the small amount of clickbait here and thank them
 for bringing the message of how AI can support creativity into the wider world.
 Speaking of working with Dulli, Guy Parsons on Twitter, that is at GUYP has a big thread
 on what he calls tips, tricks, games, experiments and combinations for Dulli and just kind of
 ideas of how you can interact with Dulli.
 Now this is targeted specifically towards Dulli but obviously this is also going to
 work for a lot of these other text to image systems as they all have very common bases,
 very common weaknesses and very common ways of interacting with them.
 Now he has more threads, for example, this one saying Dulli 2 generates amazing AI images
 but using these 10 free tools can make them so much better in which he goes into post
 processing essentially taking the things you get from Dulli and in various ways improving
 upon them, animating them, making them better, and so on.
 And on top of that, he also released a free 82 page book, the Dulli prompt book in which
 he summarizes and elaborates on all of these things in how you can interact with these
 text to image models in a efficient in a creative and in a more productive way.
 As I said, the book is available for free and if you are into a career of Dulli prompt
 engineer in the future, I definitely recommend you read it.
 Mid Journey has just recently announced that they're now moving to open beta, which essentially
 means that you can now join without an invite.
 Now if you are on Twitter, I'm sure you've seen mid journey generations they are super
 cool.
 If not, just search for hashtag mid journey on Twitter, and you're going to find like
 a lot of very amazing generations.
 This one's called the roots of infinity.
 Now mid journey is open but it's not free there is like a credit system.
 However, it is pretty affordable to run a few prompts and with the help of the previous
 resources you should be able to come up with quite creative prompts in order to test out
 the system.
 They also have an elaborate page of instructions and FAQs in order to help you get going and
 produce the best results possible.
 I've mentioned this one before, but Dulli mini is now called cry on notice the spelling
 it's C R A I Y O N.
 This after opening I was quite displeased with the naming conflict, Dulli mini being
 sort of very interchangeable with Dulli.
 So that gave the impression that the two had to do something with one another, which obviously
 they do as Dulli mini is an open source recreation of the Dulli system.
 However, Dulli mini has now been rebranded as crayon just to make it clear that it is
 its own project.
 Now the name Dulli mini is actually in another way not really descriptive as the system is
 now powered by the Dulli mega model.
 So the FAQ says the model used is called Dulli mini specifically the larger version also
 known as Dulli mega.
 So if you've used this and you've recently noticed a bit of a bump in performance, that's
 because the model has been upgraded and it's generally still fun to play around with these
 things.
 This is sunrise outdoor weightlifting.
 And also here you can apply any of the techniques we discussed before the model is also open
 source so if you don't want to wait for the servers or want to modify it or run it on
 your own, you can do so.
 Alright and just two quick helpful resources for this episode.
 One is the deep learning curriculum by Jacob Hilton, which is a curriculum like a set of
 resources that where you can learn about deep learning specifically about stuff that Jacob
 is interested in.
 This ranges from transformers scaling laws up to optimization, reinforcement learning,
 interpretability and more.
 There's also a set of links to other resources.
 So this in general is pretty helpful if you're kind of into machine learning into deep learning,
 but some topics you might want to expand your basic knowledge.
 And the other one is the pen and paper exercises in machine learning by Michael Guttman, which
 is on archive and is a PDF that goes over various things as it says it's pen and paper
 exercises.
 So one chapter for example is factor graphs and message passing.
 So you get a graphs, you get the factors, and you get an exercise mark the graph with
 arrows indicating all messages that need to be computed for the computation of P of x
 one, and there's a solution.
 So the PDF covers a lot of different areas as you can see right here linear algebra optimization
 directed graphical models, undirected graphical models, hidden Markov models, model based learning,
 sampling, and variational inference.
 Very cool 200 pages of gruesome exercises just for you.
 Alright, this was it for this week's ML news.
 I'm well aware that I've in no way covered or exhausted the space of text to image models
 or artistic models.
 There are a lot of things out there.
 I just wanted to give you a bit of an overview what happened in recent weeks.
 Let me know what you think in the comments.
 And as always, stay hydrated, and I'll see you next time.
 Bye bye.