[ML News] BLOOM: 176B Open-Source | Chinese Brain-Scale Computer | Meta AI: No Language Left Behind
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper bloom nlp gpt3 gpt 3 gpt-3 eleuther ai eleutherai bigscience bigsciencew big science huggingface hugging face yalm yandex facebook nllb meta ai language meta ai translation machine translation ml news mlnews kilcher news ml news bloom responsible ai rail license ai model license ai license chatbot ai chatbot are chatbots allowed karpathy leaves tesla
#mlnews #bloom #ai 

Today we look at all the recent giant language models in the AI world!

OUTLINE:
0:00 - Intro
0:55 - BLOOM: Open-Source 176B Language Model
5:25 - YALM 100B
5:40 - Chinese Brain-Scale Supercomputer
7:25 - Meta AI Translates over 200 Languages
10:05 - Reproducibility Crisis Workshop
10:55 - AI21 Raises $64M
11:50 - Ian Goodfellow leaves Apple
12:20 - Andrej Karpathy leaves Tesla
12:55 - Wordalle


References:
BLOOM: Open-Source 176B Language Model
https://bigscience.huggingface.co/blog/bloom
https://huggingface.co/spaces/bigscience/license
https://huggingface.co/bigscience/bloom?text=34%2B10%3D44+%0A54%2B20%3D

YALM 100B
https://github.com/yandex/YaLM-100B

Chinese Brain-Scale Supercomputer
https://www.scmp.com/news/china/science/article/3182498/china-supercomputer-achieves-global-first-brain-scale-ai-model?utm_source=pocket_mylist
https://archive.ph/YaoA6#selection-1237.156-1237.246

Meta AI Translates over 200 Languages
https://ai.facebook.com/research/no-language-left-behind/

Reproducibility Crisis Workshop
https://reproducible.cs.princeton.edu/

AI21 Raises $64M
https://techcrunch.com/2022/07/12/openai-rival-ai21-labs-raises-64m-to-ramp-up-its-ai-powered-language-services/?guccounter=1

Ian Goodfellow leaves Apple
https://twitter.com/goodfellow_ian/status/1544638709039091717

Andrey Karpathy leaves Tesla
https://mobile.twitter.com/karpathy/status/1547332300186066944
https://www.businessinsider.com/report-tesla-laid-off-about-200-people-in-autopilot-unit-2022-6?r=US&IR=T

Wordalle
https://huggingface.co/spaces/huggingface-projects/wordalle?utm_source=pocket_mylist

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Bloom finishes training and is now released as the biggest open source language model to date. A new Chinese supercomputer is allegedly able to compute brain scale AI models. And both Ian Goodfellow and Andrej Karpati leave their jobs. Welcome to ML News. Hello and welcome everyone to ML News rather ML old I've been gone for a while. What happened? Yeah, sorry, I was busy getting canceled and all. So but you know, I'm back. So we're going to catch up on everything that happened over the summer. And we're going to do it in different installments. So if your favorite thing is not in the news right now, maybe wait a bit or remind me of it. This installment is all about large models, there have been a plethora of huge models coming out of both companies and research initiatives. Speaking of which big science is a research conglomerate, a workshop, a group of people over 1000 researchers from over 250 countries coming together and trying to replicate something like GPT three not only replicate but go beyond bloom is the result of this effort. It is a 176 billion parameter language model, which is released as fully open source, the model has been developed open source has been trained open source and is now released to the world for everyone to use and research. But not only that other than something like GPT three, we know everything that's going into these models, we know what data is in there. And the data is really cool. The model is explicitly made to be multilingual. In fact, the training data contains over 59 languages, probably even more. Now, 13 of these 59 are programming languages. So the model is also going to be relatively decent at that. But this is a huge step forward for open source research for language research, and especially when it comes to less represented languages in the usual training data. The model was trained with sponsored compute and is available on the hugging face hub to download, you can even enter a little prompt over here, yet they do only accept smaller short prompts for now because the model is rather large. No 54 and 20 is not exactly four, but we'll get there bloom we'll get there. Now one interesting aspect about this model is that it is released under the big science real license, which is the responsible AI license. This license is kind of like a copy left license in the sense that if you create derivative works of this model, like if you fine tune it, you have to release it under the same terms as this license, this license governs the use of the model and essentially says that you cannot use this model for a certain number of things which are listed in the license. So if you look at the license, you have to scroll down a little bit. And if you scroll down more, there's like a huge blank space. And then there's appendix A. And these are the use restriction. Now most of these restrictions are fairly standard. For example, you are not allowed to use the model in any way that violates, you know, state law, international law, federal law, and so on. You're not allowed to use the model for the purpose of exploiting harming or attempt to exploit or harm miners in any way. There's a number of these things. The more interesting ones, which I think are you're not allowed to use the model for fully automated decision making that adversely impacts an individual's legal rights or otherwise creates or modifies a binding enforceable obligation. So binding enforceable obligation will be something like a contract. So you are not allowed to use this model to make automatic contract decisions. I'm not entirely sure what exactly that prohibits. Let's say the authors here intended to prevent something like automated decision making in terms of hiring someone or maybe automated selling of something like insurance, like a person comes, I want to get some insurance, and they just talk to a chat bot and the chat bot, you know, actually makes the contract. I'm not exactly sure how this license would apply here. Like, could I make it such that the chat bot simply makes a suggestion back to the human says like, here is an offer, you know, you can accept it or not? Or does at any point need to be a human in the loop from the side of the model, like for sure, the model can make a contract offer about a piece of insurance, but then maybe an insurance agent will still have to look over that look over the applicant and say, yeah, that's correct. Or that's not correct. I think this is going to be hashed out at some point, which is not now. This is probably not the first time software has released under such restrictions, but probably the first time a big AI model is the other interesting one is you're not allowed to generate or disseminate information or content in any context, for example, posts, articles, tweets, chatbots, or other kinds of automated bots without expressly and intelligibly disclaiming that the text is machine generated. But who would do something like this? I mean, come on. All in all, I think the license is actually fairly permissible. There's a lot of things that you actually can do with a model like this. And that's really cool. And it's available for everyone to research and even build monetizable products on top of it. So let me know what you think in the comments about the model about the licenses and so on. Other big models, YALM 100B as a 100 billion parameter GPT like language model by Yandex, and it can mainly speak English and Russian. Now, if we go not one but three orders of magnitude bigger in terms of models, South China Morning Post writes China supercomputer achieves global first with brain scale AI model. So this apparently and I'm going to say apparently because apparently there are no official statements out yet. There is a new supercomputer in China that has trained a neural network with 174 trillion parameters. That's trillion that is 1000 times bigger than something like GPT three or bloom or any of these biggest models that we have today. Now we've seen trillion parameter models before, but they've usually been sparse in some way and we have no clue over what this model here represents. But as the article says, this does approach the number of synapses in a brain. Now that's not to say that we've replicated the brain, but these models are getting extremely huge. So apparently the scientists said that they had achieved a decent performance from the unprecedented brain scale AI model, whatever that means. They also say the communication between the nodes of the supercomputer is over 23 petabytes per second, with one researcher saying that the machines parallel computing ability mimicked human thinking like eating while watching television that I have to say in all these stages of building a GI. Certainly the last step is going to be an AI that can eat while watching television. I have the feeling there is hardly a greater human achievement than doing those two things at the same time. In fact, it's true, I've never ever seen a robot or a piece of software that can eat while watching television. So if this is true, a GI is almost solved. Meta AI releases a blog post along with a paper under the heading No Language Left Behind, another huge language model, in fact, a translation model that focuses on translating between a plethora, in fact, over 200 languages, and with a particular focus on low resource languages, low resource languages have been a problematic topic for machine translation for a while, because AI models, especially big models that perform really well need lots of data in the question of machine translation, they in fact need aligned data, they need the same text in two different languages to be able to translate between those languages, there are techniques like pivoting, but that still requires you to have like parallel data from both languages to English at some point, this model overcomes this by in fact, using another AI model to automatically align texts of different images. So you can feed in unaligned text and the model will find parts in each of the texts that probably align with each other. This then serves as a base data set to train a translation system. This is really cool. And we've seen this a number of times to in fact, use one model to generate training data for another model. And I strongly believe that we might go beyond this paradigm, this really simple paradigm of, you know, get big data, train one model and done, we've seen a number of configurations, for example, with generative model, we've seen various benefits of having a critic, a model that selects and ranks the outputs of generative models in order to make it better. And in the case with this model right here, and others, we've seen numerous models where first training data is automatically generated by another model. And I think this opens up a possibility if you think of this, if you think not just what can I do with one model, how can I train one model, but think about the models that we already have and think about what you could do to use them to create training data to train other models that we usually wouldn't have enough training data for. This has been thought about, obviously, for a long time, I think a lot of people when they learned about GANs for the first time, they were like, wow, we can create so much training data to train our classifiers. But this is kind of the wrong way around a generative model like a GAN has much more information contained in it than an image classifier, which kind of reduces the space to the number of classes. So it seems like you kind of have to go from models that know less to models that know more what exactly that entails, I think, you know, smart people will have to come up with things like this. But it's really cool to think about. And this is a really cool work. So check it out. Alright, I quickly wanted to mention this workshop here, which is held on July 28. So potentially kind of right now or something like this, depending on when this is released. This is a workshop on the leakage and reproducibility crisis in ML based science, machine learning itself, obviously has a reproducibility problem. But there are also a number of machine learning based papers in other fields such as medicine, chemistry, physics, biology, and whatnot. And these are apparently even worse in terms of reproducibility when they apply machine learning. So this is a workshop focusing on this various pitfalls like no train test split, temporal leakage, and things like pre processing on train and test sets together. Now I have to admit, I'm guilty of this. I've done this before. But if you're interested in topics like this and want to learn more, this workshop is surely a good place to go. TechCrunch writes open AI arrival AI 21 labs raises $64 million to ramp up its AI powered language services yet another startup raising giant amounts of money to build giant models. I'm not exactly sure all this money flowing into this stuff is going to pay off for all of them. I mean, surely not for all of them. Is it going to pay off for a lot of them? I don't know. But I've reported on AI 21 in the past. And I think they have a really interesting approach with their Jurassic X models where they try to compose different tools and make the language model not solve tasks as such but make the language model learn how to use other programs other tools in order to complete its tasks. I think that's a you know, a really cool paradigm to go about things. I'm not sure how it's going to work out for them business wise, but I congratulate them on their funding round. Exciting times. Ian Goodfellow is leaving Apple to join DeepMind has long been rumored articles have been written that he's not happy with the remote working agreements and so on. But he's released a simple tweet and as always take what is rumored by journalists with a grain of salt. Usually, you know, only about 5% of the story of what's going on. In any case, I wish Ian the best of success at DeepMind seems like cool times for him. And very similarly, Andre Karpati is leaving Tesla, he's just recently gone on a sabbatical. And now he's leaving for sure he does not have a place that he's switching to, it seems like he's going to focus on doing things he enjoys and you know, good for Andre. In related news business insider writes Tesla reportedly reportedly again laid off about 200 workers in its autopilot division, very dark rumors actually say that they all are replaced by optimus bots, but that's unconfirmed for now. And the last thing right here, this is word Ali, this is a hugging face space that composes the concept of the popular game word or with Dali. So you get a bunch of images from Dali mini, which is now crayon, and you're supposed to guess the prompt. So this one, every time you refresh, you get a new one. This one, I'm going to take a guess it is Eminem in GTA. E Eminem in GTA. Yeah, yeah. Okay, this first try first try, but it gets harder promise. All right, this was it for ML news slash old slash what happened over the summer slash I'm no longer canceled. I hope you enjoy leave a comment, leave a like share it out, subscribe, all that stuff, please keep hydrated during these warm times and I'll see you next time when we continue.
 Bloom finishes training and is now released as the biggest open source language model to date.
 A new Chinese supercomputer is allegedly able to compute brain scale AI models.
 And both Ian Goodfellow and Andrej Karpati leave their jobs. Welcome to ML News.
 Hello and welcome everyone to ML News rather ML old I've been gone for a while. What happened?
 Yeah, sorry, I was busy getting canceled and all. So but you know, I'm back. So we're going to catch
 up on everything that happened over the summer. And we're going to do it in different installments.
 So if your favorite thing is not in the news right now, maybe wait a bit or remind me of it. This
 installment is all about large models, there have been a plethora of huge models coming out of both
 companies and research initiatives. Speaking of which big science is a research conglomerate,
 a workshop, a group of people over 1000 researchers from over 250 countries coming together and trying
 to replicate something like GPT three not only replicate but go beyond bloom is the result of
 this effort. It is a 176 billion parameter language model, which is released as fully open source,
 the model has been developed open source has been trained open source and is now released to the
 world for everyone to use and research. But not only that other than something like GPT three,
 we know everything that's going into these models, we know what data is in there. And the data is
 really cool. The model is explicitly made to be multilingual. In fact, the training data contains
 over 59 languages, probably even more. Now, 13 of these 59 are programming languages. So the model
 is also going to be relatively decent at that. But this is a huge step forward for open source
 research for language research, and especially when it comes to less represented languages in the
 usual training data. The model was trained with sponsored compute and is available on the hugging
 face hub to download, you can even enter a little prompt over here, yet they do only accept smaller
 short prompts for now because the model is rather large. No 54 and 20 is not exactly four, but we'll
 get there bloom we'll get there. Now one interesting aspect about this model is that it is released
 under the big science real license, which is the responsible AI license. This license is kind of
 like a copy left license in the sense that if you create derivative works of this model, like if you
 fine tune it, you have to release it under the same terms as this license, this license governs the
 use of the model and essentially says that you cannot use this model for a certain number of
 things which are listed in the license. So if you look at the license, you have to scroll down a
 little bit. And if you scroll down more, there's like a huge blank space. And then there's appendix
 A. And these are the use restriction. Now most of these restrictions are fairly standard. For
 example, you are not allowed to use the model in any way that violates, you know, state law,
 international law, federal law, and so on. You're not allowed to use the model for the purpose of
 exploiting harming or attempt to exploit or harm miners in any way. There's a number of these things.
 The more interesting ones, which I think are you're not allowed to use the model for fully
 automated decision making that adversely impacts an individual's legal rights or otherwise creates
 or modifies a binding enforceable obligation. So binding enforceable obligation will be something
 like a contract. So you are not allowed to use this model to make automatic contract decisions.
 I'm not entirely sure what exactly that prohibits. Let's say the authors here intended to prevent
 something like automated decision making in terms of hiring someone or maybe automated selling of
 something like insurance, like a person comes, I want to get some insurance, and they just talk
 to a chat bot and the chat bot, you know, actually makes the contract. I'm not exactly sure how this
 license would apply here. Like, could I make it such that the chat bot simply makes a suggestion
 back to the human says like, here is an offer, you know, you can accept it or not? Or does at any
 point need to be a human in the loop from the side of the model, like for sure, the model can make a
 contract offer about a piece of insurance, but then maybe an insurance agent will still have to
 look over that look over the applicant and say, yeah, that's correct. Or that's not correct.
 I think this is going to be hashed out at some point, which is not now. This is probably not
 the first time software has released under such restrictions, but probably the first time a big
 AI model is the other interesting one is you're not allowed to generate or disseminate information
 or content in any context, for example, posts, articles, tweets, chatbots, or other kinds of
 automated bots without expressly and intelligibly disclaiming that the text is machine generated.
 But who would do something like this? I mean, come on. All in all, I think the license is actually
 fairly permissible. There's a lot of things that you actually can do with a model like this. And
 that's really cool. And it's available for everyone to research and even build monetizable products
 on top of it. So let me know what you think in the comments about the model about the licenses and so
 on. Other big models, YALM 100B as a 100 billion parameter GPT like language model by Yandex,
 and it can mainly speak English and Russian. Now, if we go not one but three orders of magnitude
 bigger in terms of models, South China Morning Post writes China supercomputer achieves global
 first with brain scale AI model. So this apparently and I'm going to say apparently because
 apparently there are no official statements out yet. There is a new supercomputer in China that
 has trained a neural network with 174 trillion parameters. That's trillion that is 1000 times
 bigger than something like GPT three or bloom or any of these biggest models that we have today.
 Now we've seen trillion parameter models before, but they've usually been sparse in some way and
 we have no clue over what this model here represents. But as the article says, this does
 approach the number of synapses in a brain. Now that's not to say that we've replicated the brain,
 but these models are getting extremely huge. So apparently the scientists said that they had
 achieved a decent performance from the unprecedented brain scale AI model, whatever that means. They
 also say the communication between the nodes of the supercomputer is over 23 petabytes per second,
 with one researcher saying that the machines parallel computing ability mimicked human
 thinking like eating while watching television that I have to say in all these stages of building
 a GI. Certainly the last step is going to be an AI that can eat while watching television. I have
 the feeling there is hardly a greater human achievement than doing those two things at the
 same time. In fact, it's true, I've never ever seen a robot or a piece of software that can eat
 while watching television. So if this is true, a GI is almost solved.
 Meta AI releases a blog post along with a paper under the heading No Language Left Behind,
 another huge language model, in fact, a translation model that focuses on translating between a
 plethora, in fact, over 200 languages, and with a particular focus on low resource languages,
 low resource languages have been a problematic topic for machine translation for a while,
 because AI models, especially big models that perform really well need lots of data in the
 question of machine translation, they in fact need aligned data, they need the same text in two
 different languages to be able to translate between those languages, there are techniques
 like pivoting, but that still requires you to have like parallel data from both languages to
 English at some point, this model overcomes this by in fact, using another AI model to automatically
 align texts of different images. So you can feed in unaligned text and the model will find parts
 in each of the texts that probably align with each other. This then serves as a base data set
 to train a translation system. This is really cool. And we've seen this a number of times to
 in fact, use one model to generate training data for another model. And I strongly believe that we
 might go beyond this paradigm, this really simple paradigm of, you know, get big data, train one
 model and done, we've seen a number of configurations, for example, with generative model,
 we've seen various benefits of having a critic, a model that selects and ranks the outputs of
 generative models in order to make it better. And in the case with this model right here,
 and others, we've seen numerous models where first training data is automatically generated
 by another model. And I think this opens up a possibility if you think of this, if you think
 not just what can I do with one model, how can I train one model, but think about the models that
 we already have and think about what you could do to use them to create training data to train
 other models that we usually wouldn't have enough training data for. This has been thought about,
 obviously, for a long time, I think a lot of people when they learned about GANs for the first time,
 they were like, wow, we can create so much training data to train our classifiers. But this is kind of
 the wrong way around a generative model like a GAN has much more information contained in it than
 an image classifier, which kind of reduces the space to the number of classes. So it seems like
 you kind of have to go from models that know less to models that know more what exactly that entails,
 I think, you know, smart people will have to come up with things like this. But it's really cool to
 think about. And this is a really cool work. So check it out. Alright, I quickly wanted to mention
 this workshop here, which is held on July 28. So potentially kind of right now or something like
 this, depending on when this is released. This is a workshop on the leakage and reproducibility crisis
 in ML based science, machine learning itself, obviously has a reproducibility problem. But
 there are also a number of machine learning based papers in other fields such as medicine, chemistry,
 physics, biology, and whatnot. And these are apparently even worse in terms of reproducibility
 when they apply machine learning. So this is a workshop focusing on this various pitfalls like
 no train test split, temporal leakage, and things like pre processing on train and test sets together.
 Now I have to admit, I'm guilty of this. I've done this before. But if you're interested in
 topics like this and want to learn more, this workshop is surely a good place to go.
 TechCrunch writes open AI arrival AI 21 labs raises $64 million to ramp up its AI powered
 language services yet another startup raising giant amounts of money to build giant models.
 I'm not exactly sure all this money flowing into this stuff is going to pay off for all of them.
 I mean, surely not for all of them. Is it going to pay off for a lot of them? I don't know. But
 I've reported on AI 21 in the past. And I think they have a really interesting approach with their
 Jurassic X models where they try to compose different tools and make the language model
 not solve tasks as such but make the language model learn how to use other programs other
 tools in order to complete its tasks. I think that's a you know, a really cool paradigm to
 go about things. I'm not sure how it's going to work out for them business wise, but I congratulate
 them on their funding round. Exciting times. Ian Goodfellow is leaving Apple to join DeepMind
 has long been rumored articles have been written that he's not happy with the remote working
 agreements and so on. But he's released a simple tweet and as always take what is rumored by
 journalists with a grain of salt. Usually, you know, only about 5% of the story of what's going
 on. In any case, I wish Ian the best of success at DeepMind seems like cool times for him. And
 very similarly, Andre Karpati is leaving Tesla, he's just recently gone on a sabbatical. And now
 he's leaving for sure he does not have a place that he's switching to, it seems like he's going
 to focus on doing things he enjoys and you know, good for Andre. In related news business insider
 writes Tesla reportedly reportedly again laid off about 200 workers in its autopilot division,
 very dark rumors actually say that they all are replaced by optimus bots, but that's unconfirmed
 for now. And the last thing right here, this is word Ali, this is a hugging face space that composes
 the concept of the popular game word or with Dali. So you get a bunch of images from Dali mini,
 which is now crayon, and you're supposed to guess the prompt. So this one, every time you refresh,
 you get a new one. This one, I'm going to take a guess it is Eminem in GTA. E Eminem
 in GTA. Yeah, yeah. Okay, this first try first try, but it gets harder promise. All right,
 this was it for ML news slash old slash what happened over the summer slash I'm no longer
 canceled. I hope you enjoy leave a comment, leave a like share it out, subscribe, all that stuff,
 please keep hydrated during these warm times and I'll see you next time when we continue.