[ML News] GPT-3 learns to edit | Google Pathways | Make-A-Scene | CLIP meets GamePhysics | DouBlind
#mlnews #gpt3 #pathways

Your updates on the latest and greatest from the depths of Machine Learning!

Sponsor: Weights & Biases
https://wandb.me/yannic

OUTLINE:
0:00 - Intro
0:15 - Weights & Biases Report about Reports
2:45 - GPT-3 learns to edit
6:30 - Make-A-Scene: Text-to-Image with Human Priors
8:00 - Pathways: Google's new High-Performance ML scheduler
10:45 - DouBlind: Open Peer-Review
12:45 - CLIP meets GamePhysics
14:40 - Residual Quantization pushes Image Generation SOTA
16:15 - Helpful Things

References:
Weights & Biases Report about Reports
https://wandb.ai/wandb/wandb_example/reports/How-many-discoveries-were-lost-because-they-weren-t-written-down---VmlldzoxMjY3MDk5

GPT-3 learns to edit
https://openai.com/blog/gpt-3-edit-insert/?utm_source=pocket_mylist
https://beta.openai.com/playground?model=code-davinci-002

Make-A-Scene: Text-to-Image with Human Priors
https://arxiv.org/pdf/2203.13131.pdf
https://www.youtube.com/watch?v=QLTyqoJJKTo

Pathways: Google's new High-Performance ML scheduler
https://arxiv.org/pdf/2203.12533.pdf

DouBlind: Open Peer-Review
https://doublind.com/#web-intro
https://doublind.com/search?query=kilcher

CLIP meets GamePhysics
https://arxiv.org/pdf/2203.11096.pdf
https://www.reddit.com/r/GamePhysics/comments/9rqabp/red_dead_redemption_2_things_you_find_in_rdr2/
https://asgaardlab.github.io/CLIPxGamePhysics/

Residual Quantization pushes Image Generation SOTA
https://arxiv.org/pdf/2203.01941.pdf
https://github.com/kakaobrain/rq-vae-transformer

Helpful Things
https://github.com/TDAmeritrade/stumpy
https://github.com/linkedin/fasttreeshap
https://github.com/vopani/jaxton
https://twitter.com/mark_riedl/status/1507351959422087173?utm_source=pocket_mylist
https://github.com/eilab-gt/NovGrid
https://developer.nvidia.com/isaac-gym
https://github.com/NVIDIA-Omniverse/IsaacGymEnvs

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 GPT three learns to edit text, text to image generators achieve new heights, and Google finally introduces their pathway system. Welcome to ML News. Quick word from our sponsor weights and biases. If you don't know weights and biases, you should definitely check them out. They are the best when it comes to ML Ops. It's the entire package, they will automatically track your experiments, send everything to the cloud, track your models, your outputs, you can even give them your data sets, they tune your hyper parameters, they make everything shareable with your team and with the wider world is really cool. Today, I want to highlight this report that I found by Scott Condren. So it's a little bit of a showcase what you can do in a one to be report. And what he's showing here is sort of a before picture where people took screenshots of tensor board log plots or even map plot lib plots. Now, he made it a bit pixelish on purpose, but I've definitely seen things like this in papers crazy, but no more with weights and biases reports, you can share your research with the highest quality available. So let's say you've tracked a bunch of experiments and you want to present the best ones, people can check them out interactively, you see right here, I can go I can zoom in, I can click on a run, I can inspect that run in detail, like what were its hyper parameters, how much CPU and RAM did it use, what was the console log output of that run, everything is observable. But not only that, let's say I want to communicate how different hyper parameters affect the final objective. Well, the best way to do this is a plot like this, this shows me all the runs in different hyper parameter configurations on each of these axes and where they end up in the final loss. Again, this is fully interactive. And you as the writer of the report can place it wherever you want. But it's not only about experiments, reports can also include one to be tables and tables are really cool. Tables are like an Excel sheet on steroids. And again, this is fully interactive, I can inspect any cell here. So you can even interactively modify these tables. So I've actually introduced a column in this other person's report that shows me whenever the ground truth label doesn't agree with the model, and I'm able to sort by this and explore wherever the model makes mistakes. This is really neat because it decouples who runs the experiments and the evaluations from who does the analysis on the data. So this is just a small set of features that you can do in reports, and they work especially well within teams or collaborators worldwide. Again, I invite you to check out weights and biases. They've been really great sponsor, go to wannabe.me slash Yannick to let them know I sent you and now let's get into the video. All right, hello, everyone, it's Monday and a new episode of ML news. Wide angle camera, really nice. You see more of me. I don't know if that's a good thing. GPT three gains new editing capabilities. So if you don't know GPT three is a language model by open AI, it's been available through their API, you can go to it, you can ask it to produce text and code. And now they've added a new feature that allows you to actually edit text and code. They have a bunch of demos right here where they write a piece of code and then ask the model to change it in some way, for example, to make the Fibonacci computation use memorization. And then interestingly, to translate it from Python to JavaScript, which is quite impressive. Now, as I said, this doesn't only work for code, it also works for text. And I just thought we give it a try. Alright, so I'm here in the open AI API. And what I can do is I want to go and select the codex edit model, you can see right here you have different modes, there's the complete mode, which gives you the traditional models, there is the insert mode, which gives you the new insert capabilities, and the edit mode again with the edit capabilities. Alright, so let's come up with a simple function. Cool. So now that I have this, I can instruct the model to do all kinds of things. So here in the instructions, I'll say, make a doc string. This is a docs. Well, okay, we might have been oversold a little bit. Let's try. Let's try a bit more generate this functions doc string, this function squares its argument. Excellent. Nice. Add parameter information to the doc string. Nice. All right, we're getting somewhere. Add type hints. Look at that. Here, there's a button uses input. I'm dumb. All right, now let's try this translate to Java script. Boom doc strings been translated functions been translated. Excellent. Yeah, I can definitely see how this is powerful. Let's try another one. Okay, this is a short recursive implementation of a depth first tree search. Now it does have some tricky bits. For example, we're using implicit return value of none in Python, and we're never telling it what the type of node is, we just make it have some properties that are implicitly assumed. So let's see if it gets what this is generate an accurate doc string. Add a doc string to the DFS function. Whoa, whoa, nice. Okay, let's see if it gets the types add type hints. Whoo. Okay, very cool. All right, now the super challenge translate DFS from a recursive to an iterative function. Okay, let's try this. Okay, so this is a super challenge. So an iterative to an iterative algorithm. Yep. That's it. Very, very nice. Okay, there's one thing that I always wanted to do, but it's not in edit mode. Okay, checks if the program holds return not halts program plus I guess the ancient computer scientists would be happy with that answer. Cool. Remember the OpenAI API after a long time of being closed beta waiting list whatnot is now available for access to everyone. So if you want, you can go play with this stuff. There's a new paper out of meta called make a scene scene based text to image generation with human priors. Now this pushes the state of the art in image generation from text. So here are a bunch of examples. For example, the painting of blue elephant or a teddy bear with blue scarves and eyes tilted to its left. Like these are really accurate and really high quality productions. Now there is a bit of a difference between something like this and dali or glide, which is that this takes a number of auxiliary inputs. For example, it can take a segmentation map which you can see here in the middle of the generated images. It can also take reference images from which it will copy over the visual tokens. So there's more information provided to the model. But in return, you get a lot better quality output. Now one cool output of this is the illustration of a story that the author has made and put on YouTube. So the story is called the little red boat. And all the images are illustrated by this model. The little red boat woke up near the shore one day, where are all his friends, he couldn't say he decided to set sail to the open sea to find out where everyone could be. So the story in itself is pretty neat. And I think it gives a nice outlook on the near future we can expect out of these models. Like since I've made my music video, we've come such a long way. And that's not too far back. So the progress in this field is absolutely astounding. So finally, the pathways paper is out, Google has talked about this in a blog post before by Jeff Dean, and we've reported on that. But as of that point, it wasn't really clear what pathways was, I was more under the impression that it is kind of a new model architecture where Google wants to build like these giant models that have multitask components, and you would only update them sparsely and so on. However, this paper right here describes more of like an infrastructure side of things. Now, I don't know, but given that it's called the same, and it is is come out of the same company, I'm pretty sure that you know, this is actually what they meant. Hi, this is Yannick during editing. And Jeff Dean has just posted a tweet that says this paper is about the pathway system that is designed to support the broader pathways vision of creating large scale multitask multiple models with flexible support, yada, yada, yada. So it appears that even though the paper is called exactly the same as the vision, the two are separate things and one is in service of the other. Back to the video. So what is pathways, the best way I can describe it is something like MapReduce for machine learning. So imagine you have all these data centers, and you have all these accelerators around and some are connected with superfast InfiniBand, and some are connected with a network latency, what pathways allows you to do is to super efficiently distribute your computation across any number of devices and in a heterogeneous way. So while we've become pretty good at something like single instruction, multiple data computation, where we simply distribute data to different accelerators, and then run the exact same thing on all of them until we synchronize them again, heterogeneous computation is a little bit more tricky. So if I want something to happen on one part of the data, but then something else on a different part, like that's a problem, especially if the things take different amounts of time, then one is idling and so on pathways is essentially a very, very smart compiler and scheduler to distribute computation across whatever now I'm not knowledgeable enough in hardware and the interconnect between how you trace your functions in your ML programs, how the XLA compiler then figures out how long everything takes and then asynchronously schedules everything in parallel to absolutely optimize your throughput. But this is essentially what's happening right here, I invite you to read the pathways paper, because it is very detailed and gives you a good overview over what's to come in the future. Now, presumably, Google is going to deploy these things in their own data centers, which either means that you can expect faster ML workflows on GCP, maybe the prices will come down, or maybe they'll just make more profit, anything could happen. Doe blind is a social peer review platform. This is a website where anyone can go and review any paper. So this is an open platform, you can make an account, you can search for a paper, you can see what reviews already exist, and you can post your own reviews. And this can happen in a personalized or in an anonymous fashion. Now they've already indexed as far as I can see most of the machine learning papers, but most of them obviously don't have any reviews yet. So I've searched for myself right here. And I agree with the zero out of five star rating, although I think they should have like one like one is generous. But there you see the problems with these types of platforms. Now, while I definitely agree that something like this would be super valuable, with all the problems that come along, you know, anyone can come here and post a review and have bad intentions and smear other people's work and blah, blah, blah. But with all of that, I still think it's a valuable addition. However, this only works if really the whole community decides to make this the hub of things. And I just don't see that happening in the near future anytime soon. Wait, that's a tautology, the near future anytime soon. Like that's the same. All right. So I'm definitely excited to see what happens with these platforms. This is not the only one, but it seems pretty cool. I have not yet seen any incentive here to cash in somehow on this, which makes me a bit more hopeful for this one. But what I'd really like to see is this being connected to something like archive directly so that I don't have to go to this website to get my reviews, but just to review somehow get aggregated from the whole internet to this platform. So when I write something about the paper on Twitter, then it might be aggregated here too. And therefore you don't force the people onto a platform, but you simply grab what's out there about particular papers. Now we've seen previously that something like zeta alpha tries to do this automatically. But there again, that's a different business model. So we'll see what happens in the future. I can't tell but I do welcome good intended efforts to revamp the peer review system. This is an interesting paper clip meets game physics. So this is a pretty simple method to use clip to find bugs in video games. So people often upload buggy footage of video games to Reddit. And I'm sorry that that is that is a bit like, what did you do to that horse? So video game developers might want to structurally search through all of these videos that are played and uploaded from people who find these types of bugs. And this is exactly what this paper does. So they take all of these videos, they index them using clip, and then you're able to search for them. For example, if you search for a person flying in the air in the Grand Theft Auto five database, you'll get all kinds of buggy clips of things that maybe should or maybe shouldn't be happening. Now this is a great help probably to game developers, but it does have a downside. Namely, you can only search for the bugs that you know exist. So this was actually a legitimate person flying in the air, like like I'm pretty sure that's what should happen. But let's say a user comes to you and says, Well, all of a sudden, my character was stuck in the air or stuck in a tree or stuck in a wall. What you could do is you could turn on the search engine. And you could search through all of the footage of all of the people who played this game, whether or not something like this was happening somewhere else. Now the usefulness of this obviously goes beyond video games, you could search any type of image or video footage through that. There are some shortcomings, as I said, you can only search for things that you know. And also right now, this is simply implemented as taking a bunch of frames and then running them through clip and searching across them. So you're not able to necessarily search anything that happens in a temporal fashion. In the video, there's not a true video search, it's more like a frame search. That all being said, pretty cool project, the data set is released, so you can try it out for yourself. Another paper that has caught my attention is auto aggressive image generation using residual quantization by Kakao brain and post tech. This is another paper that pushes the state of the art in image generation from text. So the samples you see here are pretty neat. And they can be generated not only from text, but also conditionally, for example, the top two pictures are conditioned on image net classes, the bottom two pictures are produced from a text prompt. And the core of this paper revolves around a technique called residual quantization. Now, usually, if you do vector quantization, what you want to do is you want to run your image through some sort of a down sampler, some sort of a feature extractor, like a convent or a transformer. And then at the end of that, you quantize it into individual chunks, individual visual tokens, what this model does is as it down samples the image in the feature extractor, it quantizes at each stage, and then it remembers the residual of what it quantized. So it will end up with a multi scale representation essentially of visual token plus whatever is needed to reconstruct the finer grained stage that came before it. So this can retain potentially a lot more information about the fine grain structure of the image and enables these really high quality productions. Now, what's also cool is that the models are available. Specifically, there is a 3.9 billion parameter model available just for you to download. Now, how you're going to run it is a different question, but it is available. All right, let's get into some helpful things for this week. Stumpy is a powerful and scalable library for time series data mining. Fast tree shop is a package that provides algorithm for explainability in tree based algorithms, meaning random forest x g boost, light GBM and so on. Yes, there exists something else than deep learning. Imagine that jackston is a collection of 100 jacks exercises. If you've ever wanted to learn jacks, this might be the place. Nov grid is a variant of mini grid, which allows you to change underlying world dynamics. For example, right here, the fact that the yellow key opens the door is exchanged at test time with the fact that the blue key opens the door. The challenge for the agents is obviously to adjust to these new facts at inference time, which is really hard if you've never trained on them. Isaac Jim is a part of Nvidia's omniverse project. This is an engine to run physics simulations for the purposes of things like reinforcement learning, population based learning, and so on. The main focus here is scale, you can run 1000s of these experiments in parallel, if you have an Nvidia GPU. But still, for the fact that these are physically accurate simulations, it's pretty cool. On GitHub, they also have a repository with a bunch of benchmark environments for Isaac Jim, everything's available to download, check it out. And this was already it for ml news this week, it's been a bit of a slow week, but I hope you still had fun. If you like slow weeks, please subscribe one subscriber equals one pathway at a Google data center. Until then, see you next time.
 GPT three learns to edit text, text to image generators achieve new heights,
 and Google finally introduces their pathway system. Welcome to ML News.
 Quick word from our sponsor weights and biases. If you don't know weights and biases,
 you should definitely check them out. They are the best when it comes to ML Ops. It's the entire
 package, they will automatically track your experiments, send everything to the cloud,
 track your models, your outputs, you can even give them your data sets, they tune your hyper
 parameters, they make everything shareable with your team and with the wider world is really cool.
 Today, I want to highlight this report that I found by Scott Condren. So it's a little bit of
 a showcase what you can do in a one to be report. And what he's showing here is sort of a before
 picture where people took screenshots of tensor board log plots or even map plot lib plots. Now,
 he made it a bit pixelish on purpose, but I've definitely seen things like this in papers crazy,
 but no more with weights and biases reports, you can share your research with the highest quality
 available. So let's say you've tracked a bunch of experiments and you want to present the best ones,
 people can check them out interactively, you see right here, I can go I can zoom in, I can click
 on a run, I can inspect that run in detail, like what were its hyper parameters, how much CPU and
 RAM did it use, what was the console log output of that run, everything is observable. But not only
 that, let's say I want to communicate how different hyper parameters affect the final objective. Well,
 the best way to do this is a plot like this, this shows me all the runs in different hyper parameter
 configurations on each of these axes and where they end up in the final loss. Again, this is
 fully interactive. And you as the writer of the report can place it wherever you want. But it's
 not only about experiments, reports can also include one to be tables and tables are really
 cool. Tables are like an Excel sheet on steroids. And again, this is fully interactive, I can inspect
 any cell here. So you can even interactively modify these tables. So I've actually introduced
 a column in this other person's report that shows me whenever the ground truth label doesn't agree
 with the model, and I'm able to sort by this and explore wherever the model makes mistakes. This
 is really neat because it decouples who runs the experiments and the evaluations from who does the
 analysis on the data. So this is just a small set of features that you can do in reports, and they
 work especially well within teams or collaborators worldwide. Again, I invite you to check out
 weights and biases. They've been really great sponsor, go to wannabe.me slash Yannick to let
 them know I sent you and now let's get into the video. All right, hello, everyone,
 it's Monday and a new episode of ML news. Wide angle camera, really nice. You see more of me.
 I don't know if that's a good thing. GPT three gains new editing capabilities. So if you don't
 know GPT three is a language model by open AI, it's been available through their API, you can go to
 it, you can ask it to produce text and code. And now they've added a new feature that allows you
 to actually edit text and code. They have a bunch of demos right here where they write a piece of
 code and then ask the model to change it in some way, for example, to make the Fibonacci computation
 use memorization. And then interestingly, to translate it from Python to JavaScript,
 which is quite impressive. Now, as I said, this doesn't only work for code, it also works for
 text. And I just thought we give it a try. Alright, so I'm here in the open AI API. And what I can do
 is I want to go and select the codex edit model, you can see right here you have different modes,
 there's the complete mode, which gives you the traditional models, there is the insert mode,
 which gives you the new insert capabilities, and the edit mode again with the edit capabilities.
 Alright, so let's come up with a simple function. Cool. So now that I have this,
 I can instruct the model to do all kinds of things. So here in the instructions, I'll say,
 make a doc string. This is a docs.
 Well, okay, we might have been oversold a little bit. Let's try. Let's try a bit more generate
 this functions doc string, this function squares its argument. Excellent. Nice. Add parameter
 information to the doc string. Nice. All right, we're getting somewhere. Add type hints.
 Look at that. Here, there's a button uses input. I'm dumb. All right, now let's try this translate
 to Java script. Boom doc strings been translated functions been translated. Excellent. Yeah,
 I can definitely see how this is powerful. Let's try another one.
 Okay, this is a short recursive implementation of a depth first tree search. Now it does have some
 tricky bits. For example, we're using implicit return value of none in Python, and we're never
 telling it what the type of node is, we just make it have some properties that are implicitly
 assumed. So let's see if it gets what this is generate an accurate doc string.
 Add a doc string to the DFS function. Whoa, whoa, nice. Okay, let's see if it gets the types add
 type hints. Whoo. Okay, very cool. All right, now the super challenge translate DFS from a recursive
 to an iterative function. Okay, let's try this. Okay, so this is a super challenge.
 So an iterative to an iterative algorithm. Yep. That's it. Very, very nice. Okay,
 there's one thing that I always wanted to do, but it's not in edit mode.
 Okay, checks if the program holds return not halts program plus
 I guess the ancient computer scientists would be happy with that answer. Cool. Remember the OpenAI
 API after a long time of being closed beta waiting list whatnot is now available for access to
 everyone. So if you want, you can go play with this stuff. There's a new paper out of meta called
 make a scene scene based text to image generation with human priors. Now this pushes the state of
 the art in image generation from text. So here are a bunch of examples. For example, the painting of
 blue elephant or a teddy bear with blue scarves and eyes tilted to its left. Like these are really
 accurate and really high quality productions. Now there is a bit of a difference between something
 like this and dali or glide, which is that this takes a number of auxiliary inputs. For example,
 it can take a segmentation map which you can see here in the middle of the generated images. It can
 also take reference images from which it will copy over the visual tokens. So there's more
 information provided to the model. But in return, you get a lot better quality output. Now one cool
 output of this is the illustration of a story that the author has made and put on YouTube. So the
 story is called the little red boat. And all the images are illustrated by this model. The little
 red boat woke up near the shore one day, where are all his friends, he couldn't say he decided to set
 sail to the open sea to find out where everyone could be. So the story in itself is pretty neat.
 And I think it gives a nice outlook on the near future we can expect out of these models. Like
 since I've made my music video, we've come such a long way. And that's not too far back. So the
 progress in this field is absolutely astounding. So finally, the pathways paper is out, Google has
 talked about this in a blog post before by Jeff Dean, and we've reported on that. But as of that
 point, it wasn't really clear what pathways was, I was more under the impression that it is kind of
 a new model architecture where Google wants to build like these giant models that have multitask
 components, and you would only update them sparsely and so on. However, this paper right here describes
 more of like an infrastructure side of things. Now, I don't know, but given that it's called the same,
 and it is is come out of the same company, I'm pretty sure that you know, this is actually what
 they meant. Hi, this is Yannick during editing. And Jeff Dean has just posted a tweet that says
 this paper is about the pathway system that is designed to support the broader pathways vision of
 creating large scale multitask multiple models with flexible support, yada, yada, yada. So it
 appears that even though the paper is called exactly the same as the vision, the two are separate
 things and one is in service of the other. Back to the video. So what is pathways, the best way I
 can describe it is something like MapReduce for machine learning. So imagine you have all these
 data centers, and you have all these accelerators around and some are connected with superfast
 InfiniBand, and some are connected with a network latency, what pathways allows you to do is to
 super efficiently distribute your computation across any number of devices and in a heterogeneous
 way. So while we've become pretty good at something like single instruction, multiple data
 computation, where we simply distribute data to different accelerators, and then run the exact
 same thing on all of them until we synchronize them again, heterogeneous computation is a little
 bit more tricky. So if I want something to happen on one part of the data, but then something else
 on a different part, like that's a problem, especially if the things take different amounts
 of time, then one is idling and so on pathways is essentially a very, very smart compiler and
 scheduler to distribute computation across whatever now I'm not knowledgeable enough in
 hardware and the interconnect between how you trace your functions in your ML programs,
 how the XLA compiler then figures out how long everything takes and then asynchronously schedules
 everything in parallel to absolutely optimize your throughput. But this is essentially what's
 happening right here, I invite you to read the pathways paper, because it is very detailed and
 gives you a good overview over what's to come in the future. Now, presumably, Google is going to
 deploy these things in their own data centers, which either means that you can expect faster
 ML workflows on GCP, maybe the prices will come down, or maybe they'll just make more profit,
 anything could happen. Doe blind is a social peer review platform. This is a website where anyone
 can go and review any paper. So this is an open platform, you can make an account, you can search
 for a paper, you can see what reviews already exist, and you can post your own reviews. And
 this can happen in a personalized or in an anonymous fashion. Now they've already indexed
 as far as I can see most of the machine learning papers, but most of them obviously don't have any
 reviews yet. So I've searched for myself right here. And I agree with the zero out of five star
 rating, although I think they should have like one like one is generous. But there you see the
 problems with these types of platforms. Now, while I definitely agree that something like this would
 be super valuable, with all the problems that come along, you know, anyone can come here and post a
 review and have bad intentions and smear other people's work and blah, blah, blah. But with all
 of that, I still think it's a valuable addition. However, this only works if really the whole
 community decides to make this the hub of things. And I just don't see that happening in the near
 future anytime soon. Wait, that's a tautology, the near future anytime soon. Like that's the same.
 All right. So I'm definitely excited to see what happens with these platforms. This is not the only
 one, but it seems pretty cool. I have not yet seen any incentive here to cash in somehow on this,
 which makes me a bit more hopeful for this one. But what I'd really like to see is this being
 connected to something like archive directly so that I don't have to go to this website to
 get my reviews, but just to review somehow get aggregated from the whole internet to this platform.
 So when I write something about the paper on Twitter, then it might be aggregated here too.
 And therefore you don't force the people onto a platform, but you simply grab what's out there
 about particular papers. Now we've seen previously that something like zeta alpha tries to do this
 automatically. But there again, that's a different business model. So we'll see what happens in the
 future. I can't tell but I do welcome good intended efforts to revamp the peer review system.
 This is an interesting paper clip meets game physics. So this is a pretty simple method to
 use clip to find bugs in video games. So people often upload buggy footage of video games to
 Reddit. And I'm sorry that that is that is a bit like, what did you do to that horse? So video
 game developers might want to structurally search through all of these videos that are played and
 uploaded from people who find these types of bugs. And this is exactly what this paper does. So they
 take all of these videos, they index them using clip, and then you're able to search for them.
 For example, if you search for a person flying in the air in the Grand Theft Auto five database,
 you'll get all kinds of buggy clips of things that maybe should or maybe shouldn't be happening. Now
 this is a great help probably to game developers, but it does have a downside. Namely, you can only
 search for the bugs that you know exist. So this was actually a legitimate person flying in the air,
 like like I'm pretty sure that's what should happen. But let's say a user comes to you and says,
 Well, all of a sudden, my character was stuck in the air or stuck in a tree or stuck in a wall.
 What you could do is you could turn on the search engine. And you could search through all of the
 footage of all of the people who played this game, whether or not something like this was happening
 somewhere else. Now the usefulness of this obviously goes beyond video games, you could search any type
 of image or video footage through that. There are some shortcomings, as I said, you can only search
 for things that you know. And also right now, this is simply implemented as taking a bunch of frames
 and then running them through clip and searching across them. So you're not able to necessarily
 search anything that happens in a temporal fashion. In the video, there's not a true video search,
 it's more like a frame search. That all being said, pretty cool project, the data set is released,
 so you can try it out for yourself. Another paper that has caught my attention is auto aggressive
 image generation using residual quantization by Kakao brain and post tech. This is another paper
 that pushes the state of the art in image generation from text. So the samples you see here
 are pretty neat. And they can be generated not only from text, but also conditionally, for example,
 the top two pictures are conditioned on image net classes, the bottom two pictures are produced from
 a text prompt. And the core of this paper revolves around a technique called residual quantization.
 Now, usually, if you do vector quantization, what you want to do is you want to run your image
 through some sort of a down sampler, some sort of a feature extractor, like a convent or a transformer.
 And then at the end of that, you quantize it into individual chunks, individual visual tokens, what
 this model does is as it down samples the image in the feature extractor, it quantizes at each stage,
 and then it remembers the residual of what it quantized. So it will end up with a multi scale
 representation essentially of visual token plus whatever is needed to reconstruct the finer grained
 stage that came before it. So this can retain potentially a lot more information about the
 fine grain structure of the image and enables these really high quality productions. Now,
 what's also cool is that the models are available. Specifically, there is a 3.9 billion parameter
 model available just for you to download. Now, how you're going to run it is a different question,
 but it is available. All right, let's get into some helpful things for this week. Stumpy is a
 powerful and scalable library for time series data mining. Fast tree shop is a package that provides
 algorithm for explainability in tree based algorithms, meaning random forest x g boost,
 light GBM and so on. Yes, there exists something else than deep learning. Imagine that jackston
 is a collection of 100 jacks exercises. If you've ever wanted to learn jacks, this might be the
 place. Nov grid is a variant of mini grid, which allows you to change underlying world dynamics.
 For example, right here, the fact that the yellow key opens the door is exchanged at test time with
 the fact that the blue key opens the door. The challenge for the agents is obviously to adjust
 to these new facts at inference time, which is really hard if you've never trained on them.
 Isaac Jim is a part of Nvidia's omniverse project. This is an engine to run physics simulations for
 the purposes of things like reinforcement learning, population based learning, and so on.
 The main focus here is scale, you can run 1000s of these experiments in parallel, if you have an
 Nvidia GPU. But still, for the fact that these are physically accurate simulations, it's pretty
 cool. On GitHub, they also have a repository with a bunch of benchmark environments for Isaac Jim,
 everything's available to download, check it out. And this was already it for ml news this week,
 it's been a bit of a slow week, but I hope you still had fun. If you like slow weeks,
 please subscribe one subscriber equals one pathway at a Google data center. Until then,
 see you next time.