[ML News] Microsoft trains 530B model | ConvMixer model fits into single tweet | DeepMind profitable
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper mlnews kilcher news machine learning news microsoft turing nlg convmixer stylegan 3 stylegan v3 billion parameters vqgan gertel ai deepmind alphafold schmidhuber fukuhima neocognitron mosaicml self-driving train china chinese
#mlnews #turingnlg #convmixer

Your latest upates on what's happening in the Machine Learning world.

OUTLINE:
0:00 - Intro
0:16 - Weights & Biases raises on 1B valuation (sponsored)
2:30 - Microsoft trains 530 billion parameter model
5:15 - StyleGAN v3 released
6:45 - A few more examples may be worth billions of parameters
8:30 - ConvMixer fits into a tweet
9:45 - Improved VQGAN
11:25 - William Shatner AI chats about his life
12:35 - Google AI pushes material science
14:10 - Gretel AI raises 50M for privacy protection
16:05 - DeepMind's push into ML for biology
19:00 - Schmidhuber laudates Kunihiko Fukushima for Bower Award
21:30 - Helpful Things
22:25 - Mosaic ML out of stealth mode
23:55 - First German self-driving train
24:45 - Ex-Pentagon Chief: China has already won
26:25 - DeepMind becomes profitable

Sponsor: Weights & Biases
https://wandb.com

References:
Microsoft Trains 530B Parameter Model
https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/

StyleGAN 3 Code Released
https://nvlabs.github.io/stylegan3/
https://github.com/NVlabs/stylegan3
https://colab.research.google.com/github/ouhenio/StyleGAN3-CLIP-notebook/blob/main/StyleGAN3%2BCLIP.ipynb#scrollTo=V_rq-N2m0Tlb

When do labels help?
https://arxiv.org/pdf/2110.04374.pdf

ml_paper.bruh
https://openreview.net/pdf?id=TVHS5Y4dNvM

Improved VQGAN
https://openreview.net/pdf?id=pfNyExj7z2

William Shatner "AI" & Storyfile
https://www.livescience.com/william-shatner-ai-chat?fbclid=IwAR19yapmIotCTL9NIpz1xy2Ayq3H869i7TU34Vm-obxRaCLeX5YMDR_Wl-Y&utm_source=pocket_mylist
https://www.storyfile.com/

GoogleAI Finds Complex Metal Oxides
https://ai.googleblog.com/2021/10/finding-complex-metal-oxides-for.html

GretelAI raises 50M Series B
https://techcrunch.com/2021/10/07/gretel-ai-raises-50m-for-a-platform-that-lets-engineers-build-and-use-synthetic-datasets-to-ensure-the-privacy-of-their-actual-data/
https://gretel.ai/
https://gretel.ai/blog/why-privacy-by-design-matters-more-than-ever

DeepMind's Push in ML for Bio
https://www.biorxiv.org/content/10.1101/2021.10.04.463034v1
https://deepmind.com/blog/article/enformer

Kunihiko Fukushima wins Bower Award: Schmidhuber Congratulates
https://www.fi.edu/laureates/kunihiko-fukushima
https://www.youtube.com/watch?v=ysOw6lNWx2o

Helpful Things
https://github.com/UKPLab/beir#beers-features
https://arxiv.org/pdf/2104.08663.pdf
https://bayesoptbook.com/
https://github.com/nvlabs/imaginaire/
https://github.com/NVlabs/imaginaire/blob/master/projects/gancraft/README.md

MosaicML out of Stealth Mode
https://www.mosaicml.com/
https://www.mosaicml.com/blog/founders-blog
https://app.mosaicml.com/library/imagenet
https://github.com/mosaicml/composer
https://mosaicml-composer.readthedocs-hosted.com/en/stable/

Germany's first self-driving train
https://techxplore.com/news/2021-10-germany-unveils-self-driving.html

Ex-Pentagon Chief: China has already won tech war
https://nypost.com/2021/10/11/pentagon-software-chief-nicolas-chaillan-resigns/

DeepMind becomes profitable
https://bdtechtalks.com/2021/10/07/google-deepmind-2020-earnings/

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Microsoft trains a model that's three times as large as GPT-3. Nvidia releases the third iteration of their style gun model and DeepMind goes hard on ML for biology. Welcome to ML News. You might have already heard this, but Weights and Biases has just raised a Series C round at valuation of 1 billion US dollars and is now officially a unicorn. Congratulations to Weights and Biases, one of the absolute top products in the market. And I'm not just saying this out of the goodness of my heart, they actually pay me to say this. So thank you so much to Weights and Biases for sponsoring this video. Now, how might this benefit you? Imagine Weights and Biases, they get all of this cash right now, they're just going to dump this on you in form of free product. So you can expect the Weights and Biases system to become more powerful, better looking, faster, whatever you want. And for the foreseeable future, it's probably going to be available to you for free as it is right now. Hello. Yeah. Yes. Yes. That's what I said. Okay, I can say that. I mean, are you sure? I mean, forever is kind of a long, like, I'm not sure I can make promises against the nature of the universe. Like, okay. All right. All right. Yes, I'll do it. Okay. All right. So apparently, the products are going to be free forever for personal use and academia. Yes, forever. That's the beauty of startup money. It's spend first and then earn back later. So if you don't know what Weights and Biases is, Weights and Biases is a general suite of tools for machine learning engineers, machine learning researchers, and everyone in the lifecycle of ML products, it can track your experiments, it can save your models and data sets, it can monitor your runs, and it is with you from experiment all the way to deployment. It's usually in the cloud, but it can be on premise. So if you want to take part in that sweet, sweet cash inflow, go to Weights and Biases right now. And again, congratulations to them, they should absolutely pay me more now that they have more. Hello, hello, and welcome everyone to ML news. There's a lot to go through. So let's get going. Microsoft trains Megatron touring NLG 530B. How many words can you accumulate to make a model sound really, really, really big? I guess we're gonna find out with the next iteration. But for this iteration, this is a giant model. Now this is essentially a decoder only language model, much like GPT three, yet it is quite a bit bigger. So this model has 105 layers, it's hidden dimension is over 20,000. And each layer has 128 attention heads. This new model achieves various state of the art results in zero shot NLP tasks. And this blog post details what it can do. And more importantly, how it was trained. So the training relies on this library called deep speed by Microsoft, which is a library to train these large kinds of models split over multiple computers. When I say multiple computers, I don't mean 12 Raspberry Pi's. In fact, this training is powered by 560 DGX A100 servers, that's not 560 GPUs, that's 560 servers, each of which has eight A100 GPUs inside of them. And everything is connected by NVLink and NVSwitch and super duper InfiniBand. So this is an absolute beast. It trained with a batch size of 1920 and achieves about 120 teraflops per second per GPU in throughput. Now the sheer scale of this is absolutely crazy. And it's questionable whether or not humanity really wants to go this route of scaling up in this matter. But I'm glad they did in this case, noteworthy is for example, the fact that they didn't start out with a big batch size. In fact, they started with a batch size of 32 and then gradually increased to the final batch size. Another noteworthy thing is that their training data is based on the pile by Luther AI, which is an open source data set that came out of the efforts of replicating GPT-3, which noteworthy has not released their training data yet. But like GPT-3, the authors here pay close attention to the quality of their data. So even inside the pile, they sample various proportions differently. And they also add some things from common crawl and real news to arrive at their final data set. The article details what kind of scores the model reaches on what kind of zero shot tasks. If you're interested, check it out. I don't know if the model will be accessible or whether this was just an academic exercise or whether Microsoft wants to make money with it. I guess we'll see. Nvidia releases StyleGAN 3. We've covered this paper previously, it was called alias free generative adversarial networks. So not much has changed since then. Notably, you can see the comparison of StyleGAN 2, which had a very hard dependency on the position in the image. So you see the hair texture sort of remains at the point where the image is yet StyleGAN 3 has solved these issues largely, as you can see, the entire objects move around independent of their absolute position. So this gives rise to a lot more maybe controllable, maybe realistic pictures. So what's new is that they have now released the code and the models to go along with this. And people have already tried out a bunch of stuff, including putting these into notebooks together with clip. So thanks to the people involved here and shepherd, Eugenio Herrera and Katherine Krausen. So if you want to try this out, remember StyleGAN 2 is trained on specific data sets. So for example, here I have taken the faces data set, you're able to enter some sort of prompt here for clip. Now I just entered the prompt Eagle because I didn't know what was gonna happen. So here's the start and let's see what happens. Okay. Yep. Yep. All right. But I guess Eagle means I'll just slowly disappear. But people have come up with quite cool stuff here, give it a try and see what happens. Here's an interesting paper by Yuval Kirstein, Patrick Lewis, Sebastian Riedl and Omar Levy called a few more examples maybe worth billions of parameters, they analyze different NLP tasks, and they discover that for some tasks collecting a few labeled examples will in fact increase the performance of the model in a very drastic way compared to something like a zero shot performance. Now this is not the case for all models though, which is the interesting part. So for example, if you take something like open question answering, which is where the model has to recall information or go look for information, then increasing the number of examples doesn't necessarily mean that the model gets better. However, just scaling up the model pre training it on more data that is worth a lot. But if you go to something like extractive question answering, where you don't have to recall anything, in fact, you're given the Wikipedia article usually where the answer is contained somewhere, and all you need to do is find the answer, then a few more labeled examples are actually just as good as scaling the model up to drastic degrees. So the authors hypothesize that in something like open question answering, it's really about how much of pre training you have, which means how much stuff is stored in your weights. Whereas for extractive question answering, it's much more how can you map the question that you're given to specific words in the article, so the model can learn a lot even from very, very simple and few examples. So this might be a thing to consider if you're in an area of NLP, and you may not have a lot of data. And you ask yourself, should I spend the money to get more training examples? Well, I guess it depends on the task. Another interesting paper is something something strike through patches are all you need emoji under review at iClear 2022. So the first question is have paper titles gone too far. So this is an absolute meme paper, but the actual contents are really nice. Essentially, the paper does a hybrid architectures between the vision transformers and the MLP mixers, they hypothesize that at least in part what makes vision transformers good are the fact that they operate on patches and not necessarily the transformer architecture by themselves. So they propose an architecture where you put the image into patches, but then it's just a mix between depth wise convolution and point wise convolution, much like the idea of MLP mixer, where you mix the dimensions and then mix the locations repeatedly. With this, they're able to outperform the other two models. And most importantly, this is to the best of their knowledge, the first model that achieves the elusive goal of having 80% plus image net top one accuracy while also fitting into a tweet. Our field is just memes now. And another paper that piqued my interest vector quantized image modeling with improved VQ GAN. This is an iteration on VQ GAN involving vision transformers, funnily enough, after the last paper, so they go with a two stage approach where in the first stage, they use a transformer encoder and decoder and in between a quantization layer. Now quantization has been really successful in recent months. So it's not surprising that people make strides when introducing quantizations into new places. This then is paired with an autoregressive transformer that takes in the encoded codebook vectors or indices thereof, and essentially learns a language model over these. So you're taking a picture, you encode it into latent space. And then in the latent space, you describe it as a sequence of codebook vectors. And that sequence is essentially a language by itself. And on this language, you can train an autoregressive transformer. So now when you want to sample a new image, you can simply go to your transformer, you can let it sample a sequence of these codebook vectors as they would appear in the data set, you can use the transformer decoder to decode it. And there you get a new image. Now the images of this model look really nice. And that is actually my problem. The images almost look too perfect. They look super smooth. They look absolutely crisp. And just these images right here, they seem so clean that they're not even real anymore. Like I would expect these pictures on the front of like a glossy magazine, a time magazine cover, a National Geographic cover, or something like this, not just pictures taken by some person somewhere. Life Science writes William Shatner AI will chat with you about the Star Trek actors life. Now this article is essentially about a product called story file. The story file looks to be quite a cool product, what they do is they will sit you down and film you and ask you various questions about your life that people may ask. Now you just sit there and you just answer these questions, I guess this is going to take quite a long time. But once you have this compiled, it's sort of like an FAQ about your life. And then what they do is they provide you with this text interface or with a speech interface where you can now ask a question. So what makes this different to a regular FAQ is simply that you ask a question and then it finds the closest match in the FAQ list and gives you that answer as pre recorded. And then there's also one time where Shatner says, I can't make any sense of that. And that's what happens when you answer any other question that it can't map. So how much of this is really AI? Not sure, but it's definitely good that they put AI in quotes when they titled the article. Google AI writes about finding complex metal oxides for technology advancement. This blog post is a pretty cool report about research that has been done in finding new materials. Material science is notoriously difficult because essentially we have no clue what happens if we mix two things together that no one has mixed together before. And given the amount of things there are to mix, most things haven't been mixed before. The authors here developed a new method of using an inkjet printer to essentially print mixtures in various dosages into lines on a piece of, I don't know, cardboard paper, something like this. These are plates and you print out these metal oxide mixtures in lines in various mixtures, components or fractions, then you bake them and then you use optical analysis to try to assess their properties. Now not all properties are accessible via optical analysis, but you can use machine learning to try to suggest to you interesting compounds that you might want to look further at. So out of the giant amount of possible combinatorical possibilities to mix, they have come down to just very few that they needed to test further. So this is very much like drug discovery, where also machine learning is now helping to suggest new compounds that might be interesting to look at. So in the end, they found 51 oxide systems with interesting behavior, only one of them had previously been experimentally validated. So all in all, pretty cool. If you're into material science, give this article definitely a read. Next up, TechCrunch writes Gretel AI raises 50 million US dollars for a platform that lets engineers build and use synthetic data sets to ensure the privacy of their actual data. Gretel AI is a company that focuses on data privacy on how can we make ml work in sensitive settings, how do we not leak private data and so on. So one of their services is they let you abstract your data such that your ml algorithms can still train but they will train on synthetic data that is guaranteed to be privacy protected. Now just conceptually, this is a bit more challenging than it just might seem like any information you pull out of data is potentially related to the privacy of the data where it comes from, even synthetic data, even with various guarantees, as long as information is transmitted, it seems like there might be a risk. But these people are the experts. So I'm not going to claim anything here. And it looks like their tools are useful in a wide variety of applications. Now what I love is their website where they have this demo called accelerate your tasks. And here is the timeline that without Gretel you have to do Oh, no, you have an idea you need to go ask your boss, you need to copy sensitive data. Oh, no, you have to do all these things at once. And then with Gretel, wait, wait, watch that click here. Wow, idea, integrate Gretel instantly synthesize or anonymize data, innovate. In any way, there's a blog post that goes along with the 50 million new funding about why privacy by design matters more than ever. If you're interested, give it a read. And I need to leave. Well, I got kicked up from my other studio. It's not technically my studio, this is going to be resolved pretty soon. You'll see there's going to be a new studio, it's going to be epic. Where were we? Oh, yes, DeepMind has released two new works. One is here on bio archive, and one is a blog post by themselves. So there's a paper to go along with this as well. The first paper is called protein complex prediction with alpha fold multimer. And this is a specifically crafted version of alpha fold to predict the folding of protein complexes. So while the original alpha fold was made to predict how a protein folds from its original chain of amino acids into its final 3d structure, the alpha fold multimer model handles cases where there's not just one chain of amino acids involved, multiple chains will fold up to create what's called a protein complex. And these are notoriously even harder to predict. And these are notoriously even harder to predict than just single protein. So alpha fold multimer contains various improvements that make predicting protein complexes a lot more accurate and improves not only over baselines, but also over the original alpha fold. The second one is called predicting gene expression with AI. And here we move from the land of proteins to the world of genes. So in your cells, you have DNA and DNA is essentially a long strand of information. And from this information, the amino acid chains that make up the proteins are read off and translated and transcribed. Now it is really important to know which parts of the DNA are read and also how often they are read and translated various things on the DNA can influence how different regions are read off. For example, if one part of the DNA is coding for a protein, that region is generally called a gene, then whether or not that gene is actually read off and how much it can be influenced by factors such as how tightly the DNA is wound around proteins called histones. There are also various methyl modifications of the DNA. And lastly, and this might be the most complex thing, there can be what are called promoter and inhibitor systems. And these are the most complex inhibitor sequences that are in front of the gene that influence that gene. And these can be really far away. So imagine a really long text. And whatever is happening in here in the text is influenced by like a single word or two words that come way, way, way before it's like an uber German sentence. So how better to handle this than throw a giant transformer at the problem. And this is what DeepMind did right here with the giant transformer trained on the DNA, they can predict gene expression better than baselines. And this will improve our understanding and prediction of what various modifications to the DNA will do. So if there is some sort of a variance, then gene expressions can be predicted without having to necessarily test it beforehand. Very cool. Give it a read. Kunihiko Fukushima has won the Bauer Award for achievement in science for work on the neocognitron possibly the earliest implementation of what would now be called a convolutional neural network. So Fukushima is pioneering work is being prized with an award and some prize money. And none other than Jürgen Schmidt Huber has publicly released a YouTube video to honor Kuniko Fukushima for this work and for the reception of the award. Now Schmidt Huber actually has opened a YouTube channel as far as I can tell it just for this video or at least that might be the first one. Now is Jürgen going to join the ranks of us ml youtubers it would be amazing. I mean, this is de facto reaction content. So he's already halfway there. Now Schmidt Huber gives a glowing review of the work of Fukushima and what the influences of that work were. And he generally seems to be pretty pleased with Kuniko receiving this award, though about halfway through the speech, he starts to switch from away from work of Fukushima to work of funnily enough, his own labs. Now I think the story arc he had in mind was to sort of give an overview of what Fukushima had done and then set this in relation to what is happening today. But what is happening today is entirely framed in works of Schmidt Huber's lab. Now, of course, he's giving this speech. So fair enough. But with the exception of Dan net, which is a convolutional neural network that is coming from his labs, and a year before Alex net won several competitions in computer vision, the rest of the talk is essentially disconnected from Fukushima's work altogether, talking about LSTMs and how it's one of the most successful papers of all times talking about how transformers were invented in the 90s by his labs, more LSTMs and a brief discussion on Dan net, then going into how highway networks are essentially a precursor to resnets. And at the end, circling back to Fukushima's work. So it's essentially congratulations, his work was awesome. Also, my work is awesome. Also, congratulations, his work is awesome. Now, if you're interested, the entire speech is available on YouTube. And we of course, welcome Juergen to the circle of ml youtubers. Okay, some helpful stuff for this week by year is a benchmark for zero shot evaluation of information retrieval models. This is available on GitHub and it has various data sets and benchmarks for information retrieval. The Bayesian optimization book by Roland Garnett is out online, it will remain free online, but this version is a sort of a pre print and I think comments are very welcome. So if you're into Bayesian optimization or looking to get into it, this is a nice resource. Imaginaire by Nvidia is a pytorch library for GANs that now also includes the famous GAN craft. So if you've always wondered what your Minecraft worlds look like, if they were real places, this might be the place to go. Mosaic is a new ML startup that came out of stealth mode and presents itself as making ML training efficient. Notably, they came up with two products. One is this experiment explorer, which pays special attention to not only your accuracy and your loss curves, but also the cost and the efficiency at which your experiments run. So for a given baseline, you can find out what is the cheapest way to reach the same accuracy, what is the highest quality that you can achieve while keeping the same speed, what if I want the same cost, and so on. The other product is the composer, which is supposedly a library to make training neural networks more reproducible. So you can drop in various extra algorithms such as learning rate schedules or squeeze excite layers and so on. Now, do we really need another neural network library? And how modular is all of this really, I guess we'll see how this develops. To me neural network training is seems to be still intricate enough that libraries are most useful when they give you nice primitives that you can plug together instead of ticking a couple of checkboxes like here, I guess it's going to be pretty hard for them to make all of this work together. On the other hand, it's going to be I guess kind of easy for something like weights and biases to also include a cost measure of training and be a real competitor to mosaic here. So I get it, these people make this their primary mission. But I think it's still going to be a hard fought battle over the ML tooling space. I'm excited to see what happens. Tech Explore writes Germany unveils its first self driving train. Now self driving trains have been used in things like airports and so on. But this is the first self driving train in Germany that runs alongside other trains on the same tracks. So the report here is actually pretty funny in that it says these self driving trains are more punctual and energy efficient than traditional trains, they offer a more reliable service, they transport up to 30% more passengers and significantly improve punctuality and save more than 30% of energy. Now what they're actually saying is that German people suck at running trains. Simply replacing human drivers, coordinators, schedulers and so on with machines makes such a difference. That's on you Germans. That's not on the machines. The New York Post writes Pentagon's first software chief quit because China has already won global tech war pretty strong statement I have to say. So apparently he told the Financial Times there's good reason to be angry at the US for falling behind. We have no competing fighting chance against China in 15 to 20 years. Right now it's a done deal. It's already over in my opinion. He claimed that the US like Beijing should have prioritized artificial intelligence, machine learning and cyber capabilities over traditional military spending like building new fighter jets. Now this is a stance one can take cyber security and cyber warfare are important topics. But the article gets a bit weirder. He attacked Google for not working on AI with the US Defense Department while Chinese companies are obliged to work with Beijing. The US also wasting time debating the ethics of AI while China makes massive investments and issues such concerns he said, well, here is how it works. US companies and governments and military discuss AI ethics to please one particular loud annoying part of the US public mirroring that Chinese companies, government and military also discuss AI ethics to please a very loud part of the US public. I'm not sure how serious we should take these warnings right here. It is of course an interesting question on how much one should balance the very real concerns of AI ethics with the fact that somewhere else in the world, someone might care just a little bit less about that and then overpower you in 1020 years. And lastly, deep mind becomes profitable. So apparently deep mind is now profitable for the first time whilst it has been hemorrhaging money in the past few years. Now the article by tech talks here details how this is exactly happening. Deep mind doesn't have any customers by itself. It's only customer essentially is alphabet. So the parent company is the only customer, which means that deep mind can essentially set any price they want and the customer is going to pay it. So deep mind going into the green might be more an accounting trick than anything else, probably the whole alphabet construct needed to save some taxes. And that was the most optimal way to do it. The article goes into more detail on how hard and expensive it is to really do reinforcement learning in the real world. And also the strategy deep mind pursues where they pay a lot of money to acquire the world's top talent. Now that being said, we have recently more and more seen deep mind venture into solving actual real world problems with things like alpha fold for protein folding prediction and weather now casting, it seems like slowly it might make its way into real markets. Alright, this was it for this week's ML news. Let me know what you think in the comments. I'll see you next time and bye bye.
 Microsoft trains a model that's three times as large as GPT-3. Nvidia releases the third
 iteration of their style gun model and DeepMind goes hard on ML for biology. Welcome to ML News.
 You might have already heard this, but Weights and Biases has just raised a Series C round at
 valuation of 1 billion US dollars and is now officially a unicorn. Congratulations to Weights
 and Biases, one of the absolute top products in the market. And I'm not just saying this out of
 the goodness of my heart, they actually pay me to say this. So thank you so much to Weights and
 Biases for sponsoring this video. Now, how might this benefit you? Imagine Weights and Biases,
 they get all of this cash right now, they're just going to dump this on you in form of free
 product. So you can expect the Weights and Biases system to become more powerful, better looking,
 faster, whatever you want. And for the foreseeable future, it's probably going to be available to
 you for free as it is right now. Hello. Yeah. Yes. Yes. That's what I said.
 Okay, I can say that. I mean, are you sure? I mean, forever is kind of a long, like, I'm not sure I
 can make promises against the nature of the universe. Like, okay. All right. All right.
 Yes, I'll do it. Okay. All right. So apparently, the products are going to be free forever for
 personal use and academia. Yes, forever. That's the beauty of startup money. It's spend first and
 then earn back later. So if you don't know what Weights and Biases is, Weights and Biases is a
 general suite of tools for machine learning engineers, machine learning researchers, and
 everyone in the lifecycle of ML products, it can track your experiments, it can save your models
 and data sets, it can monitor your runs, and it is with you from experiment all the way to deployment.
 It's usually in the cloud, but it can be on premise. So if you want to take part in that sweet,
 sweet cash inflow, go to Weights and Biases right now. And again, congratulations to them,
 they should absolutely pay me more now that they have more.
 Hello, hello, and welcome everyone to ML news. There's a lot to go through. So let's get going.
 Microsoft trains Megatron touring NLG 530B. How many words can you accumulate to make a model
 sound really, really, really big? I guess we're gonna find out with the next iteration. But for
 this iteration, this is a giant model. Now this is essentially a decoder only language model,
 much like GPT three, yet it is quite a bit bigger. So this model has 105 layers, it's hidden dimension
 is over 20,000. And each layer has 128 attention heads. This new model achieves various state of
 the art results in zero shot NLP tasks. And this blog post details what it can do. And more
 importantly, how it was trained. So the training relies on this library called deep speed by
 Microsoft, which is a library to train these large kinds of models split over multiple computers.
 When I say multiple computers, I don't mean 12 Raspberry Pi's. In fact, this training is powered
 by 560 DGX A100 servers, that's not 560 GPUs, that's 560 servers, each of which has eight
 A100 GPUs inside of them. And everything is connected by NVLink and NVSwitch and super duper
 InfiniBand. So this is an absolute beast. It trained with a batch size of 1920 and achieves
 about 120 teraflops per second per GPU in throughput. Now the sheer scale of this is
 absolutely crazy. And it's questionable whether or not humanity really wants to go this route
 of scaling up in this matter. But I'm glad they did in this case, noteworthy is for example,
 the fact that they didn't start out with a big batch size. In fact, they started with a batch
 size of 32 and then gradually increased to the final batch size. Another noteworthy thing is that
 their training data is based on the pile by Luther AI, which is an open source data set that came out
 of the efforts of replicating GPT-3, which noteworthy has not released their training data yet. But like
 GPT-3, the authors here pay close attention to the quality of their data. So even inside the pile,
 they sample various proportions differently. And they also add some things from common crawl and
 real news to arrive at their final data set. The article details what kind of scores the model
 reaches on what kind of zero shot tasks. If you're interested, check it out. I don't know if the model
 will be accessible or whether this was just an academic exercise or whether Microsoft wants to
 make money with it. I guess we'll see. Nvidia releases StyleGAN 3. We've covered this paper
 previously, it was called alias free generative adversarial networks. So not much has changed
 since then. Notably, you can see the comparison of StyleGAN 2, which had a very hard dependency
 on the position in the image. So you see the hair texture sort of remains at the point where the
 image is yet StyleGAN 3 has solved these issues largely, as you can see, the entire objects move
 around independent of their absolute position. So this gives rise to a lot more maybe controllable,
 maybe realistic pictures. So what's new is that they have now released the code and the models
 to go along with this. And people have already tried out a bunch of stuff, including putting
 these into notebooks together with clip. So thanks to the people involved here and shepherd, Eugenio
 Herrera and Katherine Krausen. So if you want to try this out, remember StyleGAN 2 is trained on
 specific data sets. So for example, here I have taken the faces data set, you're able to enter
 some sort of prompt here for clip. Now I just entered the prompt Eagle because I didn't know
 what was gonna happen. So here's the start and let's see what happens. Okay. Yep. Yep. All right.
 But I guess Eagle means I'll just slowly disappear. But people have come up with quite cool stuff here,
 give it a try and see what happens. Here's an interesting paper by Yuval Kirstein, Patrick
 Lewis, Sebastian Riedl and Omar Levy called a few more examples maybe worth billions of parameters,
 they analyze different NLP tasks, and they discover that for some tasks collecting a few labeled
 examples will in fact increase the performance of the model in a very drastic way compared to
 something like a zero shot performance. Now this is not the case for all models though, which is
 the interesting part. So for example, if you take something like open question answering, which is
 where the model has to recall information or go look for information, then increasing the number
 of examples doesn't necessarily mean that the model gets better. However, just scaling up the
 model pre training it on more data that is worth a lot. But if you go to something like extractive
 question answering, where you don't have to recall anything, in fact, you're given the Wikipedia
 article usually where the answer is contained somewhere, and all you need to do is find the
 answer, then a few more labeled examples are actually just as good as scaling the model up
 to drastic degrees. So the authors hypothesize that in something like open question answering,
 it's really about how much of pre training you have, which means how much stuff is stored in
 your weights. Whereas for extractive question answering, it's much more how can you map the
 question that you're given to specific words in the article, so the model can learn a lot even
 from very, very simple and few examples. So this might be a thing to consider if you're in an area
 of NLP, and you may not have a lot of data. And you ask yourself, should I spend the money to get
 more training examples? Well, I guess it depends on the task. Another interesting paper is something
 something strike through patches are all you need emoji under review at iClear 2022. So the first
 question is have paper titles gone too far. So this is an absolute meme paper, but the actual
 contents are really nice. Essentially, the paper does a hybrid architectures between the vision
 transformers and the MLP mixers, they hypothesize that at least in part what makes vision transformers
 good are the fact that they operate on patches and not necessarily the transformer architecture
 by themselves. So they propose an architecture where you put the image into patches, but then
 it's just a mix between depth wise convolution and point wise convolution, much like the idea of MLP
 mixer, where you mix the dimensions and then mix the locations repeatedly. With this, they're able
 to outperform the other two models. And most importantly, this is to the best of their
 knowledge, the first model that achieves the elusive goal of having 80% plus image net top
 one accuracy while also fitting into a tweet. Our field is just memes now. And another paper that
 piqued my interest vector quantized image modeling with improved VQ GAN. This is an iteration on VQ
 GAN involving vision transformers, funnily enough, after the last paper, so they go with a two stage
 approach where in the first stage, they use a transformer encoder and decoder and in between
 a quantization layer. Now quantization has been really successful in recent months. So it's not
 surprising that people make strides when introducing quantizations into new places. This then is paired
 with an autoregressive transformer that takes in the encoded codebook vectors or indices thereof,
 and essentially learns a language model over these. So you're taking a picture, you encode it into
 latent space. And then in the latent space, you describe it as a sequence of codebook vectors.
 And that sequence is essentially a language by itself. And on this language, you can train an
 autoregressive transformer. So now when you want to sample a new image, you can simply go to your
 transformer, you can let it sample a sequence of these codebook vectors as they would appear in the
 data set, you can use the transformer decoder to decode it. And there you get a new image. Now the
 images of this model look really nice. And that is actually my problem. The images almost look too
 perfect. They look super smooth. They look absolutely crisp. And just these images right here,
 they seem so clean that they're not even real anymore. Like I would expect these pictures on
 the front of like a glossy magazine, a time magazine cover, a National Geographic cover,
 or something like this, not just pictures taken by some person somewhere.
 Life Science writes William Shatner AI will chat with you about the Star Trek actors life. Now this
 article is essentially about a product called story file. The story file looks to be quite a
 cool product, what they do is they will sit you down and film you and ask you various questions
 about your life that people may ask. Now you just sit there and you just answer these questions,
 I guess this is going to take quite a long time. But once you have this compiled, it's sort of like
 an FAQ about your life. And then what they do is they provide you with this text interface or with
 a speech interface where you can now ask a question. So what makes this different to a regular FAQ is
 simply that you ask a question and then it finds the closest match in the FAQ list and gives you
 that answer as pre recorded. And then there's also one time where Shatner says, I can't make
 any sense of that. And that's what happens when you answer any other question that it can't map.
 So how much of this is really AI? Not sure, but it's definitely good that they put AI in quotes
 when they titled the article. Google AI writes about finding complex metal oxides for technology
 advancement. This blog post is a pretty cool report about research that has been done in
 finding new materials. Material science is notoriously difficult because essentially we
 have no clue what happens if we mix two things together that no one has mixed together before.
 And given the amount of things there are to mix, most things haven't been mixed before. The authors
 here developed a new method of using an inkjet printer to essentially print mixtures in various
 dosages into lines on a piece of, I don't know, cardboard paper, something like this. These are
 plates and you print out these metal oxide mixtures in lines in various mixtures, components or
 fractions, then you bake them and then you use optical analysis to try to assess their properties.
 Now not all properties are accessible via optical analysis, but you can use machine learning to try
 to suggest to you interesting compounds that you might want to look further at. So out of the giant
 amount of possible combinatorical possibilities to mix, they have come down to just very few that
 they needed to test further. So this is very much like drug discovery, where also machine learning
 is now helping to suggest new compounds that might be interesting to look at. So in the end, they
 found 51 oxide systems with interesting behavior, only one of them had previously been experimentally
 validated. So all in all, pretty cool. If you're into material science, give this article definitely
 a read. Next up, TechCrunch writes Gretel AI raises 50 million US dollars for a platform that lets
 engineers build and use synthetic data sets to ensure the privacy of their actual data. Gretel AI
 is a company that focuses on data privacy on how can we make ml work in sensitive settings,
 how do we not leak private data and so on. So one of their services is they let you abstract your
 data such that your ml algorithms can still train but they will train on synthetic data that is
 guaranteed to be privacy protected. Now just conceptually, this is a bit more challenging
 than it just might seem like any information you pull out of data is potentially related to the
 privacy of the data where it comes from, even synthetic data, even with various guarantees,
 as long as information is transmitted, it seems like there might be a risk. But these people are
 the experts. So I'm not going to claim anything here. And it looks like their tools are useful in
 a wide variety of applications. Now what I love is their website where they have this demo called
 accelerate your tasks. And here is the timeline that without Gretel you have to do Oh, no,
 you have an idea you need to go ask your boss, you need to copy sensitive data. Oh, no, you have to
 do all these things at once. And then with Gretel, wait, wait, watch that click here.
 Wow, idea, integrate Gretel instantly synthesize or anonymize data, innovate.
 In any way, there's a blog post that goes along with the 50 million new funding about why privacy
 by design matters more than ever. If you're interested, give it a read. And I need to leave.
 Well, I got kicked up from my other studio. It's not technically my studio,
 this is going to be resolved pretty soon. You'll see there's going to be a new studio,
 it's going to be epic. Where were we? Oh, yes, DeepMind has released two new works. One is here
 on bio archive, and one is a blog post by themselves. So there's a paper to go along
 with this as well. The first paper is called protein complex prediction with alpha fold
 multimer. And this is a specifically crafted version of alpha fold to predict the folding
 of protein complexes. So while the original alpha fold was made to predict how a protein folds from
 its original chain of amino acids into its final 3d structure, the alpha fold multimer model handles
 cases where there's not just one chain of amino acids involved, multiple chains will fold up to
 create what's called a protein complex. And these are notoriously even harder to predict.
 And these are notoriously even harder to predict than just single protein. So alpha fold multimer
 contains various improvements that make predicting protein complexes a lot more accurate and
 improves not only over baselines, but also over the original alpha fold. The second one is called
 predicting gene expression with AI. And here we move from the land of proteins to the world of
 genes. So in your cells, you have DNA and DNA is essentially a long strand of information. And from
 this information, the amino acid chains that make up the proteins are read off and translated and
 transcribed. Now it is really important to know which parts of the DNA are read and also how
 often they are read and translated various things on the DNA can influence how different regions are
 read off. For example, if one part of the DNA is coding for a protein, that region is generally
 called a gene, then whether or not that gene is actually read off and how much it can be influenced
 by factors such as how tightly the DNA is wound around proteins called histones. There are also
 various methyl modifications of the DNA. And lastly, and this might be the most complex thing,
 there can be what are called promoter and inhibitor systems. And these are the most complex
 inhibitor sequences that are in front of the gene that influence that gene. And these can be really
 far away. So imagine a really long text. And whatever is happening in here in the text is
 influenced by like a single word or two words that come way, way, way before it's like an uber German
 sentence. So how better to handle this than throw a giant transformer at the problem. And this is
 what DeepMind did right here with the giant transformer trained on the DNA, they can predict
 gene expression better than baselines. And this will improve our understanding and prediction of
 what various modifications to the DNA will do. So if there is some sort of a variance, then gene
 expressions can be predicted without having to necessarily test it beforehand. Very cool. Give it
 a read. Kunihiko Fukushima has won the Bauer Award for achievement in science for work on the
 neocognitron possibly the earliest implementation of what would now be called a convolutional neural
 network. So Fukushima is pioneering work is being prized with an award and some prize money. And
 none other than Jürgen Schmidt Huber has publicly released a YouTube video to honor Kuniko Fukushima
 for this work and for the reception of the award. Now Schmidt Huber actually has opened a YouTube
 channel as far as I can tell it just for this video or at least that might be the first one.
 Now is Jürgen going to join the ranks of us ml youtubers it would be amazing. I mean, this is
 de facto reaction content. So he's already halfway there. Now Schmidt Huber gives a glowing review
 of the work of Fukushima and what the influences of that work were. And he generally seems to be
 pretty pleased with Kuniko receiving this award, though about halfway through the speech, he starts
 to switch from away from work of Fukushima to work of funnily enough, his own labs. Now I think the
 story arc he had in mind was to sort of give an overview of what Fukushima had done and then set
 this in relation to what is happening today. But what is happening today is entirely framed in
 works of Schmidt Huber's lab. Now, of course, he's giving this speech. So fair enough. But with the
 exception of Dan net, which is a convolutional neural network that is coming from his labs,
 and a year before Alex net won several competitions in computer vision, the rest of the talk is
 essentially disconnected from Fukushima's work altogether, talking about LSTMs and how it's one
 of the most successful papers of all times talking about how transformers were invented in the 90s
 by his labs, more LSTMs and a brief discussion on Dan net, then going into how highway networks are
 essentially a precursor to resnets. And at the end, circling back to Fukushima's work. So it's
 essentially congratulations, his work was awesome. Also, my work is awesome. Also, congratulations,
 his work is awesome. Now, if you're interested, the entire speech is available on YouTube. And
 we of course, welcome Juergen to the circle of ml youtubers. Okay, some helpful stuff for this week
 by year is a benchmark for zero shot evaluation of information retrieval models. This is available
 on GitHub and it has various data sets and benchmarks for information retrieval. The
 Bayesian optimization book by Roland Garnett is out online, it will remain free online, but this
 version is a sort of a pre print and I think comments are very welcome. So if you're into
 Bayesian optimization or looking to get into it, this is a nice resource. Imaginaire by Nvidia is
 a pytorch library for GANs that now also includes the famous GAN craft. So if you've always wondered
 what your Minecraft worlds look like, if they were real places, this might be the place to go.
 Mosaic is a new ML startup that came out of stealth mode and presents itself as making
 ML training efficient. Notably, they came up with two products. One is this experiment explorer,
 which pays special attention to not only your accuracy and your loss curves, but also the cost
 and the efficiency at which your experiments run. So for a given baseline, you can find out what is
 the cheapest way to reach the same accuracy, what is the highest quality that you can achieve while
 keeping the same speed, what if I want the same cost, and so on. The other product is the composer,
 which is supposedly a library to make training neural networks more reproducible. So you can
 drop in various extra algorithms such as learning rate schedules or squeeze excite layers and so on.
 Now, do we really need another neural network library? And how modular is all of this really,
 I guess we'll see how this develops. To me neural network training is seems to be still intricate
 enough that libraries are most useful when they give you nice primitives that you can plug together
 instead of ticking a couple of checkboxes like here, I guess it's going to be pretty hard for them
 to make all of this work together. On the other hand, it's going to be I guess kind of easy for
 something like weights and biases to also include a cost measure of training and be a real competitor
 to mosaic here. So I get it, these people make this their primary mission. But I think it's still
 going to be a hard fought battle over the ML tooling space. I'm excited to see what happens.
 Tech Explore writes Germany unveils its first self driving train. Now self driving trains have
 been used in things like airports and so on. But this is the first self driving train in Germany
 that runs alongside other trains on the same tracks. So the report here is actually pretty
 funny in that it says these self driving trains are more punctual and energy efficient than
 traditional trains, they offer a more reliable service, they transport up to 30% more passengers
 and significantly improve punctuality and save more than 30% of energy. Now what they're actually
 saying is that German people suck at running trains. Simply replacing human drivers, coordinators,
 schedulers and so on with machines makes such a difference. That's on you Germans. That's not
 on the machines. The New York Post writes Pentagon's first software chief quit because
 China has already won global tech war pretty strong statement I have to say. So apparently
 he told the Financial Times there's good reason to be angry at the US for falling behind. We have
 no competing fighting chance against China in 15 to 20 years. Right now it's a done deal. It's already
 over in my opinion. He claimed that the US like Beijing should have prioritized artificial
 intelligence, machine learning and cyber capabilities over traditional military spending
 like building new fighter jets. Now this is a stance one can take cyber security and cyber
 warfare are important topics. But the article gets a bit weirder. He attacked Google for not working
 on AI with the US Defense Department while Chinese companies are obliged to work with Beijing. The US
 also wasting time debating the ethics of AI while China makes massive investments and issues such
 concerns he said, well, here is how it works. US companies and governments and military discuss AI
 ethics to please one particular loud annoying part of the US public mirroring that Chinese companies,
 government and military also discuss AI ethics to please a very loud part of the US public.
 I'm not sure how serious we should take these warnings right here. It is of course an interesting
 question on how much one should balance the very real concerns of AI ethics with the fact that
 somewhere else in the world, someone might care just a little bit less about that and then overpower
 you in 1020 years. And lastly, deep mind becomes profitable. So apparently deep mind is now
 profitable for the first time whilst it has been hemorrhaging money in the past few years. Now the
 article by tech talks here details how this is exactly happening. Deep mind doesn't have any
 customers by itself. It's only customer essentially is alphabet. So the parent company is the only
 customer, which means that deep mind can essentially set any price they want and the customer is going
 to pay it. So deep mind going into the green might be more an accounting trick than anything else,
 probably the whole alphabet construct needed to save some taxes. And that was the most optimal
 way to do it. The article goes into more detail on how hard and expensive it is to really do
 reinforcement learning in the real world. And also the strategy deep mind pursues where they pay a
 lot of money to acquire the world's top talent. Now that being said, we have recently more and
 more seen deep mind venture into solving actual real world problems with things like alpha fold
 for protein folding prediction and weather now casting, it seems like slowly it might make its
 way into real markets. Alright, this was it for this week's ML news. Let me know what you think
 in the comments. I'll see you next time and bye bye.