[ML News] Multiplayer Stable Diffusion | OpenAI needs more funding | Text-to-Video models incoming
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper mlnews ml news kilcher news ml news yannic phenaki imagen imagen video phenaki ai phenaki google google ai make a video ai video text to video ai video generator huggingface hugging face what is deep learning deep learning tutorial introduction to deep learning mlinpl ml in pl
#mlnews #ai #mlinpl

Your news from the world of Machine Learning!

OUTLINE:
0:00 - Introduction
1:25 - Stable Diffusion Multiplayer
2:15 - Huggingface: DOI for Models & Datasets
3:10 - OpenAI asks for more funding
4:25 - The Stack: Source Code Dataset
6:30 - Google Vizier Open-Sourced
7:10 - New Models
11:50 - Helpful Things
20:30 - Prompt Databases
22:15 - Lexicap by Karpathy

References:
Stable Diffusion Multiplayer
https://huggingface.co/spaces/huggingface-projects/stable-diffusion-multiplayer?roomid=room-0

Huggingface: DOI for Models & Datasets
https://huggingface.co/blog/introducing-doi

OpenAI asks for more funding
https://www.theinformation.com/articles/openai-valued-at-nearly-20-billion-in-advanced-talks-with-microsoft-for-more-funding
https://www.wsj.com/articles/microsoft-in-advanced-talks-to-increase-investment-in-openai-11666299548

The Stack: Source Code Dataset
https://huggingface.co/datasets/bigcode/the-stack?utm_source=pocket_mylist

Google Vizier Open-Sourced
https://github.com/google/vizier

New Models
https://imagen.research.google/video/
https://phenaki.github.io/
https://makeavideo.studio/?utm_source=pocket_mylist
https://dreamfusion3d.github.io/
https://arxiv.org/pdf/2210.15257.pdf
https://huggingface.co/spaces/PaddlePaddle/ERNIE-ViLG
https://github.com/PaddlePaddle/PaddleHub

Helpful Things
https://thecharlieblake.co.uk/visualising-ml-number-formats
https://griddly.ai/
https://engineering.fb.com/2022/10/18/open-source/ocp-summit-2022-grand-teton/?utm_source=twitter&utm_medium=organic_social&utm_campaign=eng2022h2
https://twitter.com/psuraj28/status/1580640841583902720?utm_source=pocket_mylist
https://huggingface.co/blog/stable_diffusion_jax
https://github.com/Lightning-AI/stable-diffusion-deploy
https://lightning.ai/docs/stable/
https://github.com/CarperAI/trlx
https://github.com/DLR-RM/rl-baselines3-zoo
https://github.com/Sea-Snell/JAXSeq
https://www.reddit.com/r/MachineLearning/comments/xoitw9/p_albumentations_13_is_released_a_python_library/?utm_source=pocket_mylist
https://twitter.com/Warvito/status/1570691960792580096?utm_source=pocket_mylist
https://arxiv.org/abs/2209.07162
https://academictorrents.com/details/63aeb864bbe2115ded0aa0d7d36334c026f0660b
https://huggingface.co/spaces/THUDM/CodeGeeX
https://ai.facebook.com/blog/gpu-inference-engine-nvidia-amd-open-source/?utm_source=twitter&utm_medium=organic_social&utm_campaign=blog
https://github.com/nerfstudio-project/nerfstudio
https://www.nerfacc.com/en/latest/
https://github.com/dstackai/dstack
https://www.reddit.com/r/MachineLearning/comments/yeyxlo/p_openai_whisper_3x_cpu_inference_speedup/?utm_source=pocket_mylist
https://github.com/MiscellaneousStuff/openai-whisper-cpu/issues/1

Prompt Databases
https://huggingface.co/datasets/poloclub/diffusiondb
https://publicprompts.art/
https://visualise.ai/
https://twitter.com/SamuelAlbanie/status/1574111928431026179/photo/1

Lexicap by Karpathy
https://karpathy.ai/lexicap/0139-large.html

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 A lot of text to video models have recently come out, but not only that, a lot of other stuff has happened too, such as multiplayer stable diffusion and OpenAI is looking for even more money from Microsoft. Stay tuned. This is ML News. Hello everyone. As you can see, I'm not in my usual setting. I'm actually currently in Poland. It is the last day of the machine learning in Poland conference. This conference is absolutely glorious. Absolutely fantastic. It was really cool being here. It is over now. I'm going home. But next year, please be here. Or if you're a company that's looking to get rid of some money and sponsor an awesome conference, the ML and PL conference has been organized at least as well as any of the new rips or ICMLs that I've ever been to. And it is very likely that this conference is going to grow and become more notorious in the next few years. There was a great lineup of keynote speakers, tutorials and other content. And I even had the pleasure of joining into a bit of a concert at one of the poster sessions, which was certainly a unique experience. So thanks again to the ML and PL organizers. See you there next year. All right. So stable diffusion is going multiplayer. This is a hugging face space. It's essentially a giant canvas. And you can just come in here and you drag this square somewhere and you give it some kind of a description and it will just kind of fit in what you're doing. All of this is collectively drawn by people. And I'm always afraid because I don't want to destroy something, right? Because all of this is just very, very cool what people come up with. Just another example of something that I would have never thought of. But because stuff is open and release, this is you know, this can be built. So absolutely cool. Give it a try. And maybe this inspires you to build something that is even cooler than this. I don't know what it's going to be. But I'm sure one of you has a great idea right now. Another hugging face news, they introduce DOI, digital object identifiers for data sets and models. DOIs are sort of a standard way in scientific literature of addressing things like addressing papers, addressing artifacts, and now hugging face is introducing these things for their models and data sets on the hub. So on the hub, you're going to see this little box with which you can generate essentially it's a UUID for a model or a data set that is never going to change in the future. Now you can out date it so you can say, well, this one is deprecated. I have a new version of this model, but it is a unique identifier to that model that you have. And this is really good if you want to put it inside a paper so as to make it reproducible. And given that it is a standard, it just incorporates with the whole rest of the scientific ecosystem. So definitely a big plus for anyone who does work in research. The Wall Street Journal writes Microsoft in advance talks to increase investment in open AI. In this article, essentially there isn't much detail, but open AI is apparently asking for more money, more investment. Microsoft has previously invested about a billion dollars into Microsoft. And on top of that, probably really preferential access to Azure in exchange that open AI will provide preferential access to Microsoft for its product. It's funny because here it says last week, Microsoft announced it was integrating Dolly 2 with various products, including Microsoft Design, a new graphic design app, which is cool, and the image creator for search app Bing. Is that their big plan? Is that the one billion dollar investment to get Bing off the ground finally? I'm not sure. Now keep in mind that just because open AI goes and asks for more money, that doesn't mean that they're bankrupt soon. It could also mean that they're planning for an even bigger push startups. And I don't know if open AI can still be considered a startup, but startups often they do take on more money whenever they want to start scaling even more. Now how much open AI wants to scale even more? I don't know. It could also be that they're just out of money and need more. The stack is a data set. It's by the big code project and it's three terabyte of permissively licensed source code. So this data set is fully open, you can download it if you want to train anything like a codex model or something similar. The data set pays specific attention to the licensing of the code that is included in the data set. The code is MIT licensed, Apache licensed, BSD3 licensed, essentially licensed such that you can do whatever you want with it. Now that doesn't get you out of the weeds legally of doing anything and everything because you still have to do things like provide a copyright. Notice if you copy one of these codes verbatim. But the stack not only pays attention to this when they collect this initially, but also as you can see on the hugging face entry in the hugging face hub, there are terms of use for the stack. And one of the terms of use of the stack is that you must always update your own version of the stack to the most recent usable version. And this is because they have essentially a form where you as a source code author can go and request removal of your source code from the stack. So even if you license this under MIT license, they don't want anyone's code who doesn't want to be part of the stack. So you can go and request that your code be removed from the stack, they will then do that update the data set. And by agreeing to these terms, if you download the data set, you essentially agree to always download the newest version and use the newest version of the data set such as to propagate that removal of that code. Now as I understand it, I'm not a lawyer, this is not legal advice. But as I understand it, you are entering into a binding agreement by clicking this checkbox and clicking this button. So think about whether you want that or not. But it is good that another option is out there next to just scraping GitHub, I guess. Google releases Vizier open source Vizier is a black box optimizer that works at scale. So many, many different experiments that need to be hyper parameter optimized. Vizier essentially decides which hyper parameter to try next. So you can run this as a service if you have a lot of parallel workers and you want to run hyper parameter optimizations, they have API's for users. And the user here is essentially someone who wants to do hyper parameter optimization, they have API's for developers, which means that you can put in new optimization algorithms. So if you're a developer of a black box optimization algorithm, you can integrate that with Vizier and they have a benchmarking API. So apparently this thing has been running inside of Google for a while. And now they finally decided to release it open source. So it's certainly tried and tested. All right, now we get into the video models. There have been a few video models. Now they have been released a while back. But I'll just summarize them briefly here. Imagine video is a text to video model, you can see a bunch of samples right here. And they look really, really cool. So this is a video diffusion model. But as far as I understand it is kind of a combination of fully convolutional networks and super resolution networks in order to get this effect. They described this further in a few diagrams on their website. Imagine video uses video unit architecture to capture spatial fidelity and temporal dynamics. Temporal self attention is used in the base video diffusion model, while temporal convolutions are used in the temporal and spatial super resolution models. There is a paper to go along with it if you are interested. Now also from Google research is Fennaky. I'm not exactly sure how to pronounce that. But it is a different text to video model that can produce up to minutes long videos with changing text. So here you can see a prompt that constantly changes. And as it does, the video changes as well. So rather than being a diffusion model, this model compresses video to a tokenized representation and then essentially uses a causal autoregressive language model to continue that tokenized representation. With that they're able to essentially produce unbounded video as the beginning of the video simply drops out of the context. But as long as you feed into the side input more and more text that you want to be produced, you can see that the video keeps changing, keeps adapting and keeps being faithful to the currently in focus part of the prompt. What's interesting is that the training data seems to be mostly text to image with just a few text to video pairs inside of the training data. Now we're not done with the text to video models yet. MetaAI actually released Make a Video, yet another text to video model. And this one is also a bit special because it essentially only produces a single image from text. So this is a essentially text to image model and then an unsupervised video generator from that image. So the text to image model is essentially as we know text to image models, but then the video model is unsupervised. It simply learns from unsupervised video data, how video behaves and is then able to take a single picture, a single frame of that video and make the entire video out of it. The results look really cool. What I think is cool between all of these works is that they all have a different approach for the same problem. The all the results they produce are very cool. And it's going to be interesting to see how this text to video problem will ultimately be like canonically solved, let's say. I don't know, but I'm keeping my eyes open. Now slightly different, but not entirely different is dream fusion. This isn't text to video. This is text to 3D. Now if you think that, you know, is relatively straightforward, then none of these things actually involve 3D training data, at least as far as I can understand it. Rather what they do is they consider the entire scene essentially like a nerve. So what they do is they start with a random 3D scene. So pick your 3D scene, fill a bunch of voxels and don't fill the other voxels. And then you optimize that 3D scene to satisfy text to image models that essentially act as photographs of that scene. So it is a lot like nerve, except that you don't have pictures, but you like optimize for a text to image model rather than optimizing for an actual image. And that is a really cool idea. And it actually seems to work pretty great. Now there's other work still improving text to image diffusion models themselves. Ernie BILG 2.0 is one of them. This is an iteration of the previous model and it is using mixture of denoising experts. I don't want to go too much into this, but you can definitely see right here that the results are breathtaking and very good with a great resolution. Now there is a demo on the hogging face hub. But as far as I understand, this model isn't released, so the demo and the code that they put on GitHub, they simply calls some API where the model is actually stored. This is a neat tool, not directly related to machine learning. But if you've ever wondered what like the difference between a B float 16 and an FP 16 is, I never knew. Charlie Blake has a very cool tool on a blog that essentially shows you the different tradeoffs you can make when you choose a number format. So it shows you for the different numbers, what kind of ranges you can represent with them, where they're good at, where they're not good at. So you can see here clearly the difference between a B float 16 and an FP 16. One can represent a lot of numbers and the other one can represent just very small range of numbers, but to more precision. Gridly JS is a tool that allows you to interact with grid world reinforcement learning environments. So there are a number of cool features right here. You can edit levels directly. You can also try out the levels. You can debug your policies. You can record trajectories. So right now I don't have a trajectory, but what I can do is I can record right here and I can move this thing around here, here, going to the lava and then I die. And you can see the steps I've taken right here. So you can use this to do various kinds of things, debugging, investigating, and so on. If you are into reinforcement learning and you work with grid world, then by all means, check this out. Meta announces their new box, I guess. This is the box. This is an architecture for deep learning, the grand Teton. Essentially they release the architecture open source. So their engineers have sat down and thought long and hard about what it takes for a great machine learning system. Like they're a bit more older VGX boxes. And they essentially tell you, look, we believe that this combination of hardware, this processors, these GPUs connected like this with these power supplies will be a very great base for doing research. Yeah, they're releasing these specs essentially for you to just buy or assemble. I guess whatever you want to do with it. But I can tell you it is relatively hard to decide exactly on every component of the hardware. And it's really great that people who are very competent in this actually think about it and give their suggestions. So if you have a lab or a company and you really want to buy your own hardware, maybe this is a good option for you. Pugging face diffusers from version 0.5.1 on forward supports diffusers in Jax. If you like Jax, if you like stable diffusion, go for it. Muse is an open source stable diffusion production server. Well it is not as much a server as it is sort of like a tutorial on how to bring up a server. This is based on the lightning apps framework, which is open source. And it's kind of an easy way to bring together all the components you need to deploy machine learning things. And this repository is essentially a specification on how to pull up a stable diffusion server. So if you want to deploy stable diffusion yourself, this is probably the fastest and simplest way to do so. TRLX by Carper AI is a library that allows you to do reinforcement learning for text models. So you can see right here, you can give either sort of a reward function or you can give a data set that assigns values to expert demonstrations. And you can train a language model to incorporate that. This is a relatively new domain to do reinforcement learning on text models, but it is cool to have another library to tackle the problem. RLBaselines3zoo is a training framework for stable baselines 3 reinforcement learning agents. Stable baselines is a library that tries to give reference implementations of reinforcement learning algorithms because they're very tricky and they're very hard to get right. So these are good, solid and performant reference implementations. Stable baselines 3 is the third iteration of it. And this repository right here, the zoo contains a number of surrounding things like scripts that make it very easy to interact with it, but also prepared agents and prepared hyper parameter settings that work well in different standard environments. Jaxsec is a library that allows you to train very large language models in Jax. So the cool thing is that with this library, you essentially get things like data parallelism or model parallelism for free. You can just specify them and you can trade them off however you want. This is due to the power and simplicity of Jax. Albuminations, I hope I'm pronouncing that correctly, 1.3 is out and it introduces a bunch of new image augmentations. This is a library for image augmentations. So it's good that they introduce new augmentations that fits very well to the augmentations they already have. There's also a bunch of bug fixes and more. If you're looking for image augmentations in Python, this might be a good library. This is a really cool thing you can do with diffusion models. These people have trained diffusion models of brain images and were able to create new synthetic brain images with a degree of controllability. Now there is a paper on archive if you are interested. You can also download the dataset of 100,000 synthetic brain images. CodeGeeks is a multilingual code generation model. This is as it says, it's essentially something similar like Codex, but it is released. You can actually go and you can download the model and use it yourself. MetaAI releases AI template, which is an inference engine. The goal here is to make inference faster. You get a lot of speed ups over just running standard inference and something like eye torch. So this does two things. First of all, it optimizes your computation graph. If your computation graph contains a lot of like little operations that could be used together into something that's really optimal for a given hardware, or just that can be expressed in a smarter way, then a graph optimizer can do that. And in a second step, there is a compiler to compile all of this to highly performance C++ code that runs on backend hardware such as a GPU that uses CUDA or even an AMD GPU. So if fast inference is a concern to you, this is definitely a thing to check out. Nerve Studio describes itself as a collaboration friendly studio for nerves, but it is more like a collection, an entire collection of software to handle nerves, anything from training, validating, or even experiencing yourself. You can see they have a viewer that allows you to just explore the nerves that you do and make videos from it. But really it covers everything to do with nerves. Now speaking of nerve, Nerf Pack is a pipe torch nerve acceleration toolbox. This gets significant speed ups over simply using nerve code that's out there. For example, vanilla nerve model with eight layer multilayer perceptrons can be trained to better quality in one hour rather than one to two days as in the paper. Dstack, the logo doesn't exactly work on dark background, but Dstack is a library that wants to standardize your ML workflows that you run in the cloud. This is essentially you check your workflows into GitHub and Dstack helps you to run them uniformly anywhere. So in a workflow, you can specify things like your workflow name, obviously, but then it starts, you can say, okay, my provider is bash. So this is essentially a bash script. Now what are the commands? I want to pip install some stuff, I want to run this training script right here, but it also has things like artifacts. And you can also specify things like I want to load data from this S3 bucket over there, I want to run on this cloud over there. So all of this is quite geared towards machine learning. It's certainly not the first workflow engine or the first iteration from, hey, let's check our things into source code. But it is very targeted at running ML workflows in the cloud. Several people have figured out massive speed ups in the OpenAI whisper model. For example, this person here has figured out a 3x speed up on CPU inference, but refers to the GitHub thread where someone else has found an even bigger 3.25x speed up. Again, it's very cool to see what people do when you just give them the model. And lastly, I want to point to a couple of databases for stuff mainly around stable diffusion. So diffusion DB is on the hugging face hub. It's a data set of prompts that have been entered by real users into stable diffusion and the corresponding images that they got out. Public prompts, that's public prompts dot art in your browser is a database of three prompts and three models. These models are mostly trained using dream booth, but if you're looking for inspiration for prompts and what they turn out, then this is maybe a good place to go. Likewise, visualize.ai is a website that goes a little bit more businessy. So it lets you create some free stuff like stable diffusion. But then it also acts like as a bit of a marketplace for these things, such that you could also buy them or sell them. It's cool to see that different business models are trying to spring up around this ecosystem. Ultimately, someone will figure out how to really make money off of this stuff. But you know, it's good to be part of the time when people are just trying stuff and seeing what happens with not only on the research side, but also on the business side. Lastly, Big Science has released prompt source, which is an IDE for natural language prompts. So this is a way to give people a bit more help and a bit more standardization when they use prompts to achieve certain goals, for example, when they use prompts to tackle some of the NLP challenges that are now more and more phrased simply as prompts into these large language models, rather than as data that goes into a specially trained model for that task. So if you find yourself in this situation or a similar one, then prompt source may be for you. Finally, this is a database of all Lex Friedman podcasts transcribed. This is the website of Andre Karpotty. And he used a simple combination of a download script from YouTube combined with OpenAI's whisper to transcribe all of Lex Friedman's podcast episodes. You can go to any one of them, you can click and they are here with time annotations and all is a very simple but very cool project. Thank you, Andre. And I thank all of you for listening. I'll be home again next week. Until then, stay hydrated. Bye bye.
 A lot of text to video models have recently come out, but not only that, a lot of other
 stuff has happened too, such as multiplayer stable diffusion and OpenAI is looking for
 even more money from Microsoft.
 Stay tuned.
 This is ML News.
 Hello everyone.
 As you can see, I'm not in my usual setting.
 I'm actually currently in Poland.
 It is the last day of the machine learning in Poland conference.
 This conference is absolutely glorious.
 Absolutely fantastic.
 It was really cool being here.
 It is over now.
 I'm going home.
 But next year, please be here.
 Or if you're a company that's looking to get rid of some money and sponsor an awesome
 conference, the ML and PL conference has been organized at least as well as any of
 the new rips or ICMLs that I've ever been to.
 And it is very likely that this conference is going to grow and become more notorious
 in the next few years.
 There was a great lineup of keynote speakers, tutorials and other content.
 And I even had the pleasure of joining into a bit of a concert at one of the poster sessions,
 which was certainly a unique experience.
 So thanks again to the ML and PL organizers.
 See you there next year.
 All right.
 So stable diffusion is going multiplayer.
 This is a hugging face space.
 It's essentially a giant canvas.
 And you can just come in here and you drag this square somewhere and you give it some
 kind of a description and it will just kind of fit in what you're doing.
 All of this is collectively drawn by people.
 And I'm always afraid because I don't want to destroy something, right?
 Because all of this is just very, very cool what people come up with.
 Just another example of something that I would have never thought of.
 But because stuff is open and release, this is you know, this can be built.
 So absolutely cool.
 Give it a try.
 And maybe this inspires you to build something that is even cooler than this.
 I don't know what it's going to be.
 But I'm sure one of you has a great idea right now.
 Another hugging face news, they introduce DOI, digital object identifiers for data sets
 and models.
 DOIs are sort of a standard way in scientific literature of addressing things like addressing
 papers, addressing artifacts, and now hugging face is introducing these things for their
 models and data sets on the hub.
 So on the hub, you're going to see this little box with which you can generate essentially
 it's a UUID for a model or a data set that is never going to change in the future.
 Now you can out date it so you can say, well, this one is deprecated.
 I have a new version of this model, but it is a unique identifier to that model that
 you have.
 And this is really good if you want to put it inside a paper so as to make it reproducible.
 And given that it is a standard, it just incorporates with the whole rest of the scientific ecosystem.
 So definitely a big plus for anyone who does work in research.
 The Wall Street Journal writes Microsoft in advance talks to increase investment in open
 AI.
 In this article, essentially there isn't much detail, but open AI is apparently asking for
 more money, more investment.
 Microsoft has previously invested about a billion dollars into Microsoft.
 And on top of that, probably really preferential access to Azure in exchange that open AI will
 provide preferential access to Microsoft for its product.
 It's funny because here it says last week, Microsoft announced it was integrating Dolly
 2 with various products, including Microsoft Design, a new graphic design app, which is
 cool, and the image creator for search app Bing.
 Is that their big plan?
 Is that the one billion dollar investment to get Bing off the ground finally?
 I'm not sure.
 Now keep in mind that just because open AI goes and asks for more money, that doesn't
 mean that they're bankrupt soon.
 It could also mean that they're planning for an even bigger push startups.
 And I don't know if open AI can still be considered a startup, but startups often they do take
 on more money whenever they want to start scaling even more.
 Now how much open AI wants to scale even more?
 I don't know.
 It could also be that they're just out of money and need more.
 The stack is a data set.
 It's by the big code project and it's three terabyte of permissively licensed source code.
 So this data set is fully open, you can download it if you want to train anything like a codex
 model or something similar.
 The data set pays specific attention to the licensing of the code that is included in
 the data set.
 The code is MIT licensed, Apache licensed, BSD3 licensed, essentially licensed such that
 you can do whatever you want with it.
 Now that doesn't get you out of the weeds legally of doing anything and everything because
 you still have to do things like provide a copyright.
 Notice if you copy one of these codes verbatim.
 But the stack not only pays attention to this when they collect this initially, but also
 as you can see on the hugging face entry in the hugging face hub, there are terms of use
 for the stack.
 And one of the terms of use of the stack is that you must always update your own version
 of the stack to the most recent usable version.
 And this is because they have essentially a form where you as a source code author can
 go and request removal of your source code from the stack.
 So even if you license this under MIT license, they don't want anyone's code who doesn't
 want to be part of the stack.
 So you can go and request that your code be removed from the stack, they will then do
 that update the data set.
 And by agreeing to these terms, if you download the data set, you essentially agree to always
 download the newest version and use the newest version of the data set such as to propagate
 that removal of that code.
 Now as I understand it, I'm not a lawyer, this is not legal advice.
 But as I understand it, you are entering into a binding agreement by clicking this checkbox
 and clicking this button.
 So think about whether you want that or not.
 But it is good that another option is out there next to just scraping GitHub, I guess.
 Google releases Vizier open source Vizier is a black box optimizer that works at scale.
 So many, many different experiments that need to be hyper parameter optimized.
 Vizier essentially decides which hyper parameter to try next.
 So you can run this as a service if you have a lot of parallel workers and you want to
 run hyper parameter optimizations, they have API's for users.
 And the user here is essentially someone who wants to do hyper parameter optimization,
 they have API's for developers, which means that you can put in new optimization algorithms.
 So if you're a developer of a black box optimization algorithm, you can integrate that with Vizier
 and they have a benchmarking API.
 So apparently this thing has been running inside of Google for a while.
 And now they finally decided to release it open source.
 So it's certainly tried and tested.
 All right, now we get into the video models.
 There have been a few video models.
 Now they have been released a while back.
 But I'll just summarize them briefly here.
 Imagine video is a text to video model, you can see a bunch of samples right here.
 And they look really, really cool.
 So this is a video diffusion model.
 But as far as I understand it is kind of a combination of fully convolutional networks
 and super resolution networks in order to get this effect.
 They described this further in a few diagrams on their website.
 Imagine video uses video unit architecture to capture spatial fidelity and temporal dynamics.
 Temporal self attention is used in the base video diffusion model, while temporal convolutions
 are used in the temporal and spatial super resolution models.
 There is a paper to go along with it if you are interested.
 Now also from Google research is Fennaky.
 I'm not exactly sure how to pronounce that.
 But it is a different text to video model that can produce up to minutes long videos
 with changing text.
 So here you can see a prompt that constantly changes.
 And as it does, the video changes as well.
 So rather than being a diffusion model, this model compresses video to a tokenized representation
 and then essentially uses a causal autoregressive language model to continue that tokenized
 representation.
 With that they're able to essentially produce unbounded video as the beginning of the video
 simply drops out of the context.
 But as long as you feed into the side input more and more text that you want to be produced,
 you can see that the video keeps changing, keeps adapting and keeps being faithful to
 the currently in focus part of the prompt.
 What's interesting is that the training data seems to be mostly text to image with just
 a few text to video pairs inside of the training data.
 Now we're not done with the text to video models yet.
 MetaAI actually released Make a Video, yet another text to video model.
 And this one is also a bit special because it essentially only produces a single image
 from text.
 So this is a essentially text to image model and then an unsupervised video generator from
 that image.
 So the text to image model is essentially as we know text to image models, but then
 the video model is unsupervised.
 It simply learns from unsupervised video data, how video behaves and is then able to take
 a single picture, a single frame of that video and make the entire video out of it.
 The results look really cool.
 What I think is cool between all of these works is that they all have a different approach
 for the same problem.
 The all the results they produce are very cool.
 And it's going to be interesting to see how this text to video problem will ultimately
 be like canonically solved, let's say.
 I don't know, but I'm keeping my eyes open.
 Now slightly different, but not entirely different is dream fusion.
 This isn't text to video.
 This is text to 3D.
 Now if you think that, you know, is relatively straightforward, then none of these things
 actually involve 3D training data, at least as far as I can understand it.
 Rather what they do is they consider the entire scene essentially like a nerve.
 So what they do is they start with a random 3D scene.
 So pick your 3D scene, fill a bunch of voxels and don't fill the other voxels.
 And then you optimize that 3D scene to satisfy text to image models that essentially act
 as photographs of that scene.
 So it is a lot like nerve, except that you don't have pictures, but you like optimize
 for a text to image model rather than optimizing for an actual image.
 And that is a really cool idea.
 And it actually seems to work pretty great.
 Now there's other work still improving text to image diffusion models themselves.
 Ernie BILG 2.0 is one of them.
 This is an iteration of the previous model and it is using mixture of denoising experts.
 I don't want to go too much into this, but you can definitely see right here that the
 results are breathtaking and very good with a great resolution.
 Now there is a demo on the hogging face hub.
 But as far as I understand, this model isn't released, so the demo and the code that they
 put on GitHub, they simply calls some API where the model is actually stored.
 This is a neat tool, not directly related to machine learning.
 But if you've ever wondered what like the difference between a B float 16 and an FP
 16 is, I never knew.
 Charlie Blake has a very cool tool on a blog that essentially shows you the different tradeoffs
 you can make when you choose a number format.
 So it shows you for the different numbers, what kind of ranges you can represent with
 them, where they're good at, where they're not good at.
 So you can see here clearly the difference between a B float 16 and an FP 16.
 One can represent a lot of numbers and the other one can represent just very small range
 of numbers, but to more precision.
 Gridly JS is a tool that allows you to interact with grid world reinforcement learning environments.
 So there are a number of cool features right here.
 You can edit levels directly.
 You can also try out the levels.
 You can debug your policies.
 You can record trajectories.
 So right now I don't have a trajectory, but what I can do is I can record right here and
 I can move this thing around here, here, going to the lava and then I die.
 And you can see the steps I've taken right here.
 So you can use this to do various kinds of things, debugging, investigating, and so on.
 If you are into reinforcement learning and you work with grid world, then by all means,
 check this out.
 Meta announces their new box, I guess.
 This is the box.
 This is an architecture for deep learning, the grand Teton.
 Essentially they release the architecture open source.
 So their engineers have sat down and thought long and hard about what it takes for a great
 machine learning system.
 Like they're a bit more older VGX boxes.
 And they essentially tell you, look, we believe that this combination of hardware, this processors,
 these GPUs connected like this with these power supplies will be a very great base for
 doing research.
 Yeah, they're releasing these specs essentially for you to just buy or assemble.
 I guess whatever you want to do with it.
 But I can tell you it is relatively hard to decide exactly on every component of the hardware.
 And it's really great that people who are very competent in this actually think about
 it and give their suggestions.
 So if you have a lab or a company and you really want to buy your own hardware, maybe
 this is a good option for you.
 Pugging face diffusers from version 0.5.1 on forward supports diffusers in Jax.
 If you like Jax, if you like stable diffusion, go for it.
 Muse is an open source stable diffusion production server.
 Well it is not as much a server as it is sort of like a tutorial on how to bring up a server.
 This is based on the lightning apps framework, which is open source.
 And it's kind of an easy way to bring together all the components you need to deploy machine
 learning things.
 And this repository is essentially a specification on how to pull up a stable diffusion server.
 So if you want to deploy stable diffusion yourself, this is probably the fastest and
 simplest way to do so.
 TRLX by Carper AI is a library that allows you to do reinforcement learning for text
 models.
 So you can see right here, you can give either sort of a reward function or you can give
 a data set that assigns values to expert demonstrations.
 And you can train a language model to incorporate that.
 This is a relatively new domain to do reinforcement learning on text models, but it is cool to
 have another library to tackle the problem.
 RLBaselines3zoo is a training framework for stable baselines 3 reinforcement learning
 agents.
 Stable baselines is a library that tries to give reference implementations of reinforcement
 learning algorithms because they're very tricky and they're very hard to get right.
 So these are good, solid and performant reference implementations.
 Stable baselines 3 is the third iteration of it.
 And this repository right here, the zoo contains a number of surrounding things like scripts
 that make it very easy to interact with it, but also prepared agents and prepared hyper
 parameter settings that work well in different standard environments.
 Jaxsec is a library that allows you to train very large language models in Jax.
 So the cool thing is that with this library, you essentially get things like data parallelism
 or model parallelism for free.
 You can just specify them and you can trade them off however you want.
 This is due to the power and simplicity of Jax.
 Albuminations, I hope I'm pronouncing that correctly, 1.3 is out and it introduces a
 bunch of new image augmentations.
 This is a library for image augmentations.
 So it's good that they introduce new augmentations that fits very well to the augmentations they
 already have.
 There's also a bunch of bug fixes and more.
 If you're looking for image augmentations in Python, this might be a good library.
 This is a really cool thing you can do with diffusion models.
 These people have trained diffusion models of brain images and were able to create new
 synthetic brain images with a degree of controllability.
 Now there is a paper on archive if you are interested.
 You can also download the dataset of 100,000 synthetic brain images.
 CodeGeeks is a multilingual code generation model.
 This is as it says, it's essentially something similar like Codex, but it is released.
 You can actually go and you can download the model and use it yourself.
 MetaAI releases AI template, which is an inference engine.
 The goal here is to make inference faster.
 You get a lot of speed ups over just running standard inference and something like eye
 torch.
 So this does two things.
 First of all, it optimizes your computation graph.
 If your computation graph contains a lot of like little operations that could be used
 together into something that's really optimal for a given hardware, or just that can be
 expressed in a smarter way, then a graph optimizer can do that.
 And in a second step, there is a compiler to compile all of this to highly performance
 C++ code that runs on backend hardware such as a GPU that uses CUDA or even an AMD GPU.
 So if fast inference is a concern to you, this is definitely a thing to check out.
 Nerve Studio describes itself as a collaboration friendly studio for nerves, but it is more
 like a collection, an entire collection of software to handle nerves, anything from training,
 validating, or even experiencing yourself.
 You can see they have a viewer that allows you to just explore the nerves that you do
 and make videos from it.
 But really it covers everything to do with nerves.
 Now speaking of nerve, Nerf Pack is a pipe torch nerve acceleration toolbox.
 This gets significant speed ups over simply using nerve code that's out there.
 For example, vanilla nerve model with eight layer multilayer perceptrons can be trained
 to better quality in one hour rather than one to two days as in the paper.
 Dstack, the logo doesn't exactly work on dark background, but Dstack is a library that
 wants to standardize your ML workflows that you run in the cloud.
 This is essentially you check your workflows into GitHub and Dstack helps you to run them
 uniformly anywhere.
 So in a workflow, you can specify things like your workflow name, obviously, but then it
 starts, you can say, okay, my provider is bash.
 So this is essentially a bash script.
 Now what are the commands?
 I want to pip install some stuff, I want to run this training script right here, but it
 also has things like artifacts.
 And you can also specify things like I want to load data from this S3 bucket over there,
 I want to run on this cloud over there.
 So all of this is quite geared towards machine learning.
 It's certainly not the first workflow engine or the first iteration from, hey, let's check
 our things into source code.
 But it is very targeted at running ML workflows in the cloud.
 Several people have figured out massive speed ups in the OpenAI whisper model.
 For example, this person here has figured out a 3x speed up on CPU inference, but refers
 to the GitHub thread where someone else has found an even bigger 3.25x speed up.
 Again, it's very cool to see what people do when you just give them the model.
 And lastly, I want to point to a couple of databases for stuff mainly around stable diffusion.
 So diffusion DB is on the hugging face hub.
 It's a data set of prompts that have been entered by real users into stable diffusion
 and the corresponding images that they got out.
 Public prompts, that's public prompts dot art in your browser is a database of three
 prompts and three models.
 These models are mostly trained using dream booth, but if you're looking for inspiration
 for prompts and what they turn out, then this is maybe a good place to go.
 Likewise, visualize.ai is a website that goes a little bit more businessy.
 So it lets you create some free stuff like stable diffusion.
 But then it also acts like as a bit of a marketplace for these things, such that you could also
 buy them or sell them.
 It's cool to see that different business models are trying to spring up around this ecosystem.
 Ultimately, someone will figure out how to really make money off of this stuff.
 But you know, it's good to be part of the time when people are just trying stuff and
 seeing what happens with not only on the research side, but also on the business side.
 Lastly, Big Science has released prompt source, which is an IDE for natural language prompts.
 So this is a way to give people a bit more help and a bit more standardization when they
 use prompts to achieve certain goals, for example, when they use prompts to tackle some
 of the NLP challenges that are now more and more phrased simply as prompts into these
 large language models, rather than as data that goes into a specially trained model for
 that task.
 So if you find yourself in this situation or a similar one, then prompt source may be
 for you.
 Finally, this is a database of all Lex Friedman podcasts transcribed.
 This is the website of Andre Karpotty.
 And he used a simple combination of a download script from YouTube combined with OpenAI's
 whisper to transcribe all of Lex Friedman's podcast episodes.
 You can go to any one of them, you can click and they are here with time annotations and
 all is a very simple but very cool project.
 Thank you, Andre.
 And I thank all of you for listening.
 I'll be home again next week.
 Until then, stay hydrated.
 Bye bye.