Turing-NLG, DeepSpeed and the ZeRO optimizer
deep learning machine learning nlp natural language processing machine translation arxiv attention mechanism attention transformer seq2seq bert long sequence memory gpt-2 Megatron Microsoft distributed parallelism
Microsoft has trained a 17-billion parameter language model that achieves state-of-the-art perplexity. This video takes a look at the ZeRO optimizer that enabled this breakthrough. ZeRO allows you to do model- and data-parallelism without having huge cuts in training speed.

https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/
https://github.com/microsoft/DeepSpeed
https://arxiv.org/abs/1910.02054

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi everyone, today we're going to look at Turing NLGA 17 billion parameter language model by Microsoft. The latest and greatest of language modeling by Microsoft. What is this? It is a language model. A language model is basically a model that learns to produce language, given language. So if you start a sentence it's supposed to finish a sentence. If you start a paragraph it's supposed to finish the paragraph. That's a language model. Ultimately you can make it do different things like answer questions, have a conversation with you, anything to do with understanding language. The special thing about this one is that it's ginormous. So if you look at the scale of language models, so BERT was quite a large thing back in its day. Ye Olde BERT, you can see here it has about 340 million parameters. Now I have to say all of these language models are transformers. This is kind of the state of the art today. So all of these are kind of our transformer based models. Then GPT-2 here, you can see that was the model that was so large it was too dangerous to be released into the world. That stands at 1.5 billion parameters. Megatron LM by Nvidia 8.3 billion and now we are at 17 billion parameters for this language model. And it is a bit better. People just throw more and more and more resources at this language problem. So what you can do with it, you can of course do language modeling. So what happens is you take a bunch of text like all of Wikipedia and all of the internet and all of Reddit and so on and you let the model train on it to understand, to basically produce that sort of language. And then you can measure it, for example it's a perplexity on a validation set. And the Turing NLG is currently state-of-the-art on that. It can also do for example question answering. So you can ask the question and give it a passage about that question and it will then tell you the answer that it deduced from that passage given the question as you can see here. What is more interesting is that a usual QA system will point to the passage. So it will point to the words Tristan Prediman. Whereas with a generative model like this one what you can do is you can make it actually output an answer as a sentence. So it will generate the text Jason Bras was engaged to Tristan Prediman. If you ask a question without giving it a context and just ask it to generate an answer it will do so as well. I don't know if these answers are cherry-picked but they call this zero-shot question answering. So if you ask when did World War II end and it can output World War II ended in 1945. Simply out of regularities it detected in the training data. So I mean that's what I'm kind of wondering. At what point are these models, do they have so many parameters that they simply reproduce the training data? I mean this clearly some article from the training data is about World War II or many are and it simply learned that following a question when did World War II end it needs to answer with the appropriate passage. I'm not sure that is a proper measure of language understanding if you simply can bake more and more of the training data into these many many parameters but I'm not the judge of that here. It can do it very well. So yeah what I'm actually more interested in is this thing is called the zero optimizer that they use to train the model. So the model is just a transformer, it's just a big big transformer model. There is nothing really special about the model except that it is larger than the last model and therefore a bit better. What is interesting is that this would have been pretty impossible to train if it weren't for this zero optimizer of this deep speed library and Microsoft has released this deep speed library. It's compatible for now with PyTorch. You can check this out. I'll put a link into the description and I want to dive into this a bit. There's a paper, it's by Samyam Raj Bandari and all by Microsoft. The paper describes in detail the optimizer but it's not very visual. That's why we're going to the blog post. You can see it gives many speed ups over the previous Megatron LM model that Nvidia just trained using what Nvidia has. Nvidia has machines that are interconnected within the machine with very fast buses between GPUs. But this zero optimizer can now also go over the network and make it pretty fast. Let's explore that a bit. I have the copy this here. We'll look how the zero optimizer works. Usually what you do is if you have multiple GPUs you can do something like this. This is called data parallelism. What you have is a model and the model in this case fits on your GPU. It fits on a single GPU. The blue thing here is the model. I'll actually draw this. The model is a neural network so it has a bunch of layers. Layer, layer, layer, layer. What you want to do is you pass data forward. Here is some loss and then right into the loss function and then backward again. That's basically what you need to do. You need to pass it forward and backward in order to do back propagation training. If this all fits into one box that's completely fine. If this fits into one machine, cool. We can just put many batches of data through batch one, batch two, batch three and so on. Train the model. If you want to do a speed up using this you can do so. If you have lots of data you can do what's called, and I'm always confused, I think this is called data parallelism or is it called model parallelism. In any case what you can do is you can take a second machine or many of those, replicate the model. These two models here are exactly the same. What you do is you take your data and you split it up. You take double the amount of data and you put one batch of data through the top part and you put the other through the bottom part. You do your forward passes on the machines and you do your backward passes. Then what you want to do is you want to sync between the machines what they learned from the data. Each machine has a different set of data points. Each machine calculates its own parameter updates. It learns from the data it has and then they communicate to keep because this here and this here should be the same. It's the same model. They have to keep in sync. This can be usually can be done fairly efficiently especially if these aren't actually two machines but just two GPUs inside of one large machine. If this is a large machine this is GPU 0 and this is GPU 1. This is pretty standard because especially on Nvidia machines they have these whatever I think they call them InfiniBand or so. Nvidia has these connectors that connects the GPUs together really fast. You can keep these in sync but now the problem becomes what if you want to train a model that is larger than this. Let's forget about the data parallelism for now if that is what it's called and just consider a model that is too large. A model that is too large will not fit into a machine. This is a model as a large model. What you want to do is you want to pack some of the model onto your first machine and then take the other part of the model and pack it onto another machine. You separate the model and put it on different machines. If you have a batch of data what you have to do is you pass it pass it pass it forward propagate as you regularly would but then you have an intermediate result. You send that to the next machine and you forward propagate that. At the end here you have a loss. You want to back propagate regularly through this machine. You have an intermediate result of back propagation. Send it over the network and back prop all the way through the model. That's how you can train a model that is too large for one machine if you have multiple machines. The problem here of course is this part. Just as you had to keep in sync the model before, now your communication problem becomes one of... You have to send the intermediate stages to that model and you have to send the intermediate stage of the back propagation back to that part of the model. While this part is working this part is idling away. The network overhead is just very costly. Especially if your model is so large it can't even fit into one of these single boxes. This is very problematic here. It's still doable. But what the zero optimizer does is it does both data and model parallelism. It can train models that are too large for a single machine. It can do data parallelism at the same time. Basically everything is working all the time. There is not much wasted computation. The communication is efficient and so on. It's really a technical achievement. It's not so much a scientific advance. It's really a technical achievement this optimizer. We'll shortly go through. There is a kind of an animation on the website but it's super slow. I think this might be the first time that I will be faster at explaining something than a video. Let's see here. What you do is... Let's just consider these three GPUs. Before that it would all fit on one machine. But now let's say you don't actually have that much memory. You don't have these giant empty blocks here. You just have a bit of that. So you have to split your model. The blue parts here are your model. These are model parameters. The orange part here is memory you need to store gradients. You need as many gradients as you have model parameters. Because you do gradient descent. The green stuff here are what's called optimizer parameters. Now if you just have SGD these would be non-existent. But if you have something like AdaGrad or Atom they have additional parameters for each model parameter that they need to keep track of. So these are stored here. There can be significant overhead. There's also like a floating point 3216 conversion going on here. Don't want to go into that. So you split your model onto these three machines. Let's say that's your entire model. Your model is six blocks wide. You need to forward propagate now through everything. So here is what Xero does. I think it's pretty cool. What we need to do is we have these three different batches of data and we want to forward propagate them all through the model. Through the same model at the same time. As if the model were actually stored on all these machines. Like if all of these machines had the entire model. And we can do a bit of communication. So what we do first is... This one's easy. Data zero through the first two layers here is easy. Because we have them. So bang you go through the first you get an intermediate result here and here. How do we propagate data one through the first layer? We can't send data one here. That would be too expensive. And that's the whole point would be lost. We want to actually compute data one on this GPU at the same time. What we do is before we start we actually communicate these two blocks here to GPU one. We send these parameters around and fill them in here. We send them here and we also send them here. We send the parameters to all the machines. Then we can actually forward prop data one through this and data three through this. So we can do forward prop. After we've communicated all the GPUs can be working. Same with layer two. Layer two simply can send these two here. You can see that these two here to the other machines. Now while it's doing that we've already propagated through the first layer. We've already propagated here and here through the first layer. So we can actually delete these again. We can delete these first layer parameters that we sent around again. So here you see how we can save memory. We don't keep all the model in sync and all the machines. We send whatever we need on the other machines and then once the computation is done they can delete it again. Because there's always one machine, this one here for the middle parameters, that keeps track of the parameters and that can at any point if they're needed send them again. So that's the big kind of catch. You can forward prop now through these two. They're already present. Then you can delete those again on the machines where they're not natively stored. From here you can send those two. Also up here you can send those two and forward prop your model through to the end. That was a mistake. Then each machine calculates its own loss. The backward propagation happens in much the same way. If you follow so far you can already imagine. Now the loss is different because there's a different batch of data going through each machine. There's a different batch of data going through each machine but each machine has computed with the same model due to the communication of the zero optimizer. That's pretty cool. You get the benefits of data parallelism, lots of data on the different machines and you also split up the model across the machines. You don't actually store the model on any of these machines. You only send. From here you send as you need and then you delete again. For the backward propagation, same thing. You calculate gradients. You calculate gradients here and you send the gradients as needed to the other machines. You calculate gradients here and here and you send them to the machine where they're actually needed. This is a weird pen. You send them to that machine. That machine will aggregate all the gradients of all the machines. It will aggregate them and then locally it can compute using these optimizer parameters and so on. It can do all kinds of optimization locally because it has gathered gradients from all the other data. What you end up with, for example, GPU 2 here, for these two layers it has effectively broadcast the layers such that much much more data than it just had itself could run through the layers. It has aggregated gradients from all of that data and now it can use all of these gradients together to make a good update using the optimizer parameters. To make a good update to these model parameters and then in the next iteration it can go ahead and broadcast the model parameters. The new model parameters again. It is able to compute with much more data than it can just fit by itself. It is just doing its part. So Zero and DeepSpeed, Zero is the protocol and DeepSpeed is the actual library. They will do all of this communication and splitting and so on for you over the network in a way that is efficient, in a way that everything runs at the same time and the communication overhead is minimal. You can actually choose which stage you want, so what your trade-off of communication and memory saving will be. This is extremely cool. They say this goes up to whatever 100 billion parameter models if you use... This isn't something for your average Colab user. This is really something for big players. But that being said, I don't think language is solved by simply throwing more parameters at it. I think there's still a bit of a breakthrough ahead yet to come in language understanding with newer model architectures. Alright, that was it for me. Thanks.
 Hi everyone, today we're going to look at Turing NLGA 17 billion parameter
 language model by Microsoft. The latest and greatest of language modeling by
 Microsoft. What is this? It is a language model. A language model is basically a
 model that learns to produce language, given language. So if you start a
 sentence it's supposed to finish a sentence. If you start a paragraph it's
 supposed to finish the paragraph. That's a language model. Ultimately you can make
 it do different things like answer questions, have a conversation with you,
 anything to do with understanding language. The special thing about this
 one is that it's ginormous. So if you look at the scale of language
 models, so BERT was quite a large thing back in its day. Ye Olde BERT, you
 can see here it has about 340 million parameters. Now I have to say all of
 these language models are transformers. This is kind of the state of the art
 today. So all of these are kind of our transformer based models. Then GPT-2
 here, you can see that was the model that was so large it was too dangerous to be
 released into the world. That stands at 1.5 billion parameters. Megatron LM by
 Nvidia 8.3 billion and now we are at 17 billion parameters for this language
 model. And it is a bit better. People just throw more and more and more
 resources at this language problem. So what you can do with it, you can
 of course do language modeling. So what happens is you take a bunch of text like
 all of Wikipedia and all of the internet and all of Reddit and so on and you let
 the model train on it to understand, to basically produce that sort of language.
 And then you can measure it, for example it's a perplexity on a validation set.
 And the Turing NLG is currently state-of-the-art on that. It can also do
 for example question answering. So you can ask the question and give it a
 passage about that question and it will then tell you the answer that it deduced
 from that passage given the question as you can see here. What is more interesting
 is that a usual QA system will point to the passage. So it will point to the
 words Tristan Prediman. Whereas with a generative model like this one what you
 can do is you can make it actually output an answer as a sentence. So it
 will generate the text Jason Bras was engaged to Tristan Prediman.
 If you ask a question without giving it a context and just ask it to generate an
 answer it will do so as well. I don't know if these answers are cherry-picked
 but they call this zero-shot question answering. So if you ask when did World
 War II end and it can output World War II ended in 1945. Simply out of regularities
 it detected in the training data. So I mean that's what I'm kind of wondering.
 At what point are these models, do they have so many parameters that they
 simply reproduce the training data? I mean this clearly some article
 from the training data is about World War II or many are and it simply learned
 that following a question when did World War II end it needs to answer with the
 appropriate passage. I'm not sure that is a proper measure of language
 understanding if you simply can bake more and more of the training data into
 these many many parameters but I'm not the judge of that here. It can do it very
 well. So yeah what I'm actually more interested in is this thing is called the
 zero optimizer that they use to train the model. So the model is just a
 transformer, it's just a big big transformer model. There is nothing really
 special about the model except that it is larger than the last model and
 therefore a bit better. What is interesting is that this would have been
 pretty impossible to train if it weren't for this zero optimizer of this deep
 speed library and Microsoft has released this deep speed library. It's compatible
 for now with PyTorch. You can check this out. I'll put a link into the description
 and I want to dive into this a bit. There's a paper, it's by Samyam Raj
 Bandari and all by Microsoft. The paper describes in detail the optimizer
 but it's not very visual. That's why we're going to the blog post. You can see
 it gives many speed ups over the previous Megatron LM model that
 Nvidia just trained using what Nvidia has. Nvidia has machines that
 are interconnected within the machine with very fast buses
 between GPUs. But this zero optimizer can now also go over the network and make it
 pretty fast. Let's explore that a bit. I have the copy this here. We'll
 look how the zero optimizer works. Usually what you do is if you have
 multiple GPUs you can do something like this. This is called data parallelism.
 What you have is a model and the model in this case fits on your GPU.
 It fits on a single GPU. The blue thing here is the model. I'll actually
 draw this. The model is a neural network so it has a bunch of
 layers. Layer, layer, layer, layer. What you want to do is you pass data
 forward. Here is some loss and then right into the loss function and then backward
 again. That's basically what you need to do. You need to pass it forward and
 backward in order to do back propagation training. If this all fits
 into one box that's completely fine. If this fits into one machine, cool.
 We can just put many batches of data through batch one, batch two, batch three
 and so on. Train the model. If you want to do a speed up using this you can do so.
 If you have lots of data you can do what's called, and I'm always confused, I
 think this is called data parallelism or is it called model parallelism.
 In any case what you can do is you can take a second machine or many of those,
 replicate the model. These two models here are exactly the same.
 What you do is you take your data and you split it up. You take double
 the amount of data and you put one batch of data through the top part and you
 put the other through the bottom part. You do your forward passes on the
 machines and you do your backward passes. Then what you want to do is you want
 to sync between the machines what they learned from the data. Each machine
 has a different set of data points. Each machine calculates its own parameter
 updates. It learns from the data it has and then they communicate to keep
 because this here and this here should be the same. It's the same model.
 They have to keep in sync. This can be usually can be done fairly efficiently
 especially if these aren't actually two machines but just two GPUs inside of one
 large machine. If this is a large machine this is GPU 0 and this is GPU 1.
 This is pretty standard because especially on Nvidia machines they have
 these whatever I think they call them InfiniBand or so.
 Nvidia has these connectors that connects the GPUs together really fast.
 You can keep these in sync but now the problem becomes what if you want to
 train a model that is larger than this. Let's forget about the data parallelism
 for now if that is what it's called and just consider a model that is too large.
 A model that is too large will not fit into a machine. This is a model as a
 large model. What you want to do is you want to pack some of the model onto
 your first machine and then take the other part of the model and pack
 it onto another machine. You separate the model and put it on different
 machines. If you have a batch of data what you have to do is you pass it
 pass it pass it forward propagate as you regularly would but then you have an
 intermediate result. You send that to the next machine and you forward
 propagate that. At the end here you have a loss. You want to back propagate
 regularly through this machine. You have an intermediate result of back
 propagation. Send it over the network and back prop all the way through the model.
 That's how you can train a model that is too large for one machine if you
 have multiple machines. The problem here of course is this part. Just as you had
 to keep in sync the model before, now your communication problem
 becomes one of... You have to send the intermediate stages to that model and
 you have to send the intermediate stage of the back propagation back to that
 part of the model. While this part is working this part is idling away.
 The network overhead is just very costly. Especially if your model is so
 large it can't even fit into one of these single boxes. This is very
 problematic here. It's still doable. But what the zero optimizer does is it does
 both data and model parallelism. It can train models that are too large
 for a single machine. It can do data parallelism at the same time.
 Basically everything is working all the time. There is not much wasted
 computation. The communication is efficient and so on. It's really a
 technical achievement. It's not so much a scientific advance. It's really a
 technical achievement this optimizer. We'll shortly go through. There is a
 kind of an animation on the website but it's super slow. I think
 this might be the first time that I will be faster at explaining something than a
 video. Let's see here. What you do is... Let's just consider these
 three GPUs. Before that it would all fit on one machine. But now let's say you
 don't actually have that much memory. You don't have these giant
 empty blocks here. You just have a bit of that. So you have to split your model.
 The blue parts here are your model. These are model parameters.
 The orange part here is memory you need to store gradients. You need as
 many gradients as you have model parameters. Because you do gradient
 descent. The green stuff here are what's called optimizer parameters. Now if you
 just have SGD these would be non-existent. But if you have something
 like AdaGrad or Atom they have additional parameters for each model
 parameter that they need to keep track of. So these are stored here. There
 can be significant overhead. There's also like a floating point 3216
 conversion going on here. Don't want to go into that. So you split your
 model onto these three machines. Let's say that's your entire model. Your model
 is six blocks wide. You need to forward propagate now through everything.
 So here is what Xero does. I think it's pretty cool. What we need to do
 is we have these three different batches of data and we want to forward
 propagate them all through the model. Through the same model at the same time.
 As if the model were actually stored on all these machines. Like if all of these
 machines had the entire model. And we can do a bit of communication. So what
 we do first is... This one's easy. Data zero through the first two layers
 here is easy. Because we have them. So bang you go through the first
 you get an intermediate result here and here. How do we propagate data one
 through the first layer? We can't send data one here. That would be
 too expensive. And that's the whole point would be lost. We want to
 actually compute data one on this GPU at the same time. What we do is before we
 start we actually communicate these two blocks here to GPU one. We send
 these parameters around and fill them in here. We send them here and we
 also send them here. We send the parameters to all the machines.
 Then we can actually forward prop data one through this and data three through
 this. So we can do forward prop. After we've communicated all the GPUs can be
 working. Same with layer two. Layer two simply can send these
 two here. You can see that these two here to the other machines. Now while
 it's doing that we've already propagated through the first layer.
 We've already propagated here and here through the first layer. So we can
 actually delete these again. We can delete these first layer
 parameters that we sent around again. So here you see how we can save memory.
 We don't keep all the model in sync and all the machines. We send whatever we
 need on the other machines and then once the computation is done they can delete
 it again. Because there's always one machine, this one here for the
 middle parameters, that keeps track of the parameters and that can at any point
 if they're needed send them again. So that's the big kind of catch. You can
 forward prop now through these two. They're already present.
 Then you can delete those again on the machines where they're not natively
 stored. From here you can send those two. Also up here you can send
 those two and forward prop your model through to the end.
 That was a mistake. Then each machine calculates its own loss.
 The backward propagation happens in much the same way.
 If you follow so far you can already imagine.
 Now the loss is different because there's a different batch of data
 going through each machine. There's a different batch of data going
 through each machine but each machine has computed with the same model due to
 the communication of the zero optimizer. That's pretty cool. You get the
 benefits of data parallelism, lots of data on the different machines and you
 also split up the model across the machines. You don't actually store
 the model on any of these machines. You only send.
 From here you send as you need and then you delete again. For the backward
 propagation, same thing. You calculate gradients.
 You calculate gradients here and you send the gradients as needed to the
 other machines. You calculate gradients here and here and you send them to the
 machine where they're actually needed. This is a weird pen. You send them to
 that machine. That machine will aggregate all the gradients of all the machines.
 It will aggregate them and then locally it can compute using
 these optimizer parameters and so on. It can do all kinds of optimization
 locally because it has gathered gradients from all the other data.
 What you end up with, for example, GPU 2 here, for these two layers it has
 effectively broadcast the layers such that much much more data than it just
 had itself could run through the layers. It has aggregated gradients from all of
 that data and now it can use all of these gradients together to make a good
 update using the optimizer parameters. To make a good update to these model
 parameters and then in the next iteration it can go ahead and broadcast
 the model parameters. The new model parameters again. It is able to
 compute with much more data than it can just fit by itself. It is just doing
 its part. So Zero and DeepSpeed, Zero is the protocol and DeepSpeed is the
 actual library. They will do all of this communication and splitting and so on
 for you over the network in a way that is efficient, in a way that everything
 runs at the same time and the communication overhead is minimal. You
 can actually choose which stage you want, so what your trade-off of communication
 and memory saving will be. This is extremely cool. They say this goes up to
 whatever 100 billion parameter models if you use... This isn't something for
 your average Colab user. This is really something for big players.
 But that being said, I don't think language is solved by simply throwing
 more parameters at it. I think there's still a bit of a breakthrough
 ahead yet to come in language understanding with newer model
 architectures. Alright, that was it for me. Thanks.