AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning
ml ai machine learning reinforcement learning deep rl deepmind google starcraft alphastar alphago alphazero value function policy vtrace upgo terran protoss zerg build order strategy pointer network transformer league training league battlenet artificial intelligence bot rl deep reinforcement learning model-free exploiters self-play ficticious self-play rts
DeepMind's new agent to tackle yet another Esport: Starcraft II. This agent uses deep reinforcement learning with a new technique, called League Training, to catapult itself to Grandmaster-level skill at playing this game.

Abstract:
Many real-world applications require artificial agents to compete and coordinate with other agents in complex environments. As a stepping stone to this goal, the domain of StarCraft has emerged as an important challenge for artificial intelligence research, owing to its iconic and enduring status among the most difficult professional esports and its relevance to the real world in terms of its raw complexity and multi-agent challenges. Over the course of a decade and numerous competitions, the strongest agents have simplified important aspects of the game, utilized superhuman capabilities, or employed hand-crafted sub-systems. Despite these advantages, no previous agent has come close to matching the overall skill of top StarCraft players. We chose to address the challenge of StarCraft using general purpose learning methods that are in principle applicable to other complex domains: a multi-agent reinforcement learning algorithm that uses data from both human and agent games within a diverse league of continually adapting strategies and counter-strategies, each represented by deep neural networks. We evaluated our agent, AlphaStar, in the full game of StarCraft II, through a series of online games against human players. AlphaStar was rated at Grandmaster level for all three StarCraft races and above 99.8% of officially ranked human players.

Authors: Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, David Silver

https://www.deepmind.com/blog/article/AlphaStar-Grandmaster-level-in-StarCraft-II-using-multi-agent-reinforcement-learning

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Alright, let's talk about AlphaStar, Grandmaster level in StarCraft 2 using multi-agent reinforcement learning. The corresponding paper looks like this and is by Oriol Vinyals et al. from DeepMind and has been published in the journal of Nature recently. Now let me say this first. Stop publishing in Nature. This is a journal is not open access. It makes its readers pay for getting the article. So actually you can access this article or a public version of it for free but you can't print it, you can't download it unless you pay for it. And this to me, it seems ridiculous because none of this money goes to the authors of the article. None of this money goes to the reviewers. The review quality isn't notably better, at least in the field of computer science. All of this is a publicity stunt by DeepMind because Nature has been kind of impactful in the last decades. It's like, ooh, look at me, I got a big dick I publish in Nature. Nothing more than that. It's like OpenAI saying their model is too dangerous to release to the world. I guess DeepMind might make the same claim about AlphaStar. It's like too dangerous of a StarCraft player. Yeah, so stop this. Publish your research in open access. Nature or journals like these for computer science. It's a remnant of the last century. So go on and join everyone else in distributing knowledge. All right, rant over. Let's jump in into this article. So the article describes how to train a reinforcement learning agent to play the game of StarCraft 2. So StarCraft 2 is this game for everyone who doesn't know. Just very quickly explain the game. StarCraft 2 is a real time strategy game and you're kind of in this top third person view and you control your units and the goal is kind of to move your units around and first of all build up buildings and using those buildings you can then produce more and more diverse units and ultimately you want to kind of produce some sort of army that can go to the opponent and destroy the opponent's base. So you control all of this on a computer using a mouse and a keyboard and StarCraft is notable for being very balanced. So there are three different races you can play. So first are the Terran which are kind of human, human-ish. They have marines and tanks and helicopters I believe and things like this. Then the Protoss are some sort of alien race that are super advanced so they can teleport and have energy shields and things like that. And then last are the Zerg and the Zerg are kind of icky ground dwelling creatures that infect things and spread like a disease. So the interesting thing here is compared to other real-time strategy games is that the three races they play very different. So the game is almost a different game if you play as a different race but they are so well balanced that almost any matchup is kind of a fair game between equally skilled players. So that's makes StarCraft pretty unique. Also pretty unique is the very very high action per minute rates that pro players get. Like they play this insanely fast. So game lasts about 10 to 15 minutes and as I said the goal is to destroy the enemy base. So to train an RL agent to play this is very hard because the action space is very high. You have to target with your mouse part of the screen. You have to look what is on the screen, what can I do. There's this mini map down here. There are things you can do. There are opponents you can target and so on. So all of this is very very very difficult for an RL agent. And at the end, after 10 minutes, you play play play play play and after 10 minutes you either win or you lose. And the RL agent has to figure out which of the actions that I did during those 10 minutes right. Was it this one? Was it this one? Which led to me winning or losing? These are very hard problems for reinforcement learning. And DeepMind has combined almost every trick in the book known so far to RL to achieve this. Now the main contribution I'd say here that is novel is what is called league training and we'll get to that. So first of all, if you don't know what reinforcement learning is, reinforcement learning is basically what I just described. You have an input right, which could be this thing here and you have a set of actions that you can do, which the set of actions here is anywhere you can click right, you can click anywhere on the screen. And you have to do this over and over and over and over again until you either win or you lose. And from that you will see you will at the end receive Yeah, you win or you lose and then you have to kind of learn to play the game. So it's machine learning hardcore because you get minimal information and have to achieve a lot of things from it. So the first thing that DeepMind actually does is it does supervised learning. And we'll get into how exactly the model works later. But first thing DeepMind does is it trains an agent to simply imitate humans, right? So you have human data. And from the human data, you so these are games played by humans, good humans, right? Not not people like me. So these these are games played with humans from a significantly high ELO. And the first thing you extract is this Z here. Now Z is is called a statistics vector. And as I understand it, it's mainly the build order, which means in which order do you build your buildings and units and this is very important in StarCraft. This is a strategic decision where you say, okay, first, I'm going to build three worker units. This is like three workers, worker, worker, worker, and then I'm going to build a house and then I'm going to and so on. So these are major strategic decisions that that you kind of have to make with minutes, minutes ahead of time to plan in advance. And this this is kind of stays constant for the game. So this is extracted and provided to the model as an input. So what is the current strategy basically the current overall strategy? The second thing that is extracted is this is at every time step, the observation that the humans had so the screen that humans see, and also the actions that the human did, right? So the human takes its mouse and clicks somewhere, right? This is supposed to be a mouse pointer and clicks here, right? And then the model, this part here, this is the model. And this is the policy function of the model. So the policy decides what to do, right? Is trained to match the action that the human did. So in essence, first, you train an agent to simply imitate humans. And this you can do by supervised learning, right? This is classic machine learning. Each each step you have this input, which is an image, and you have the strategy you're trying to follow. And from these two, you're simply trying to match the action that the human did, assuming the human made a good decision. So this is how you initialize, right? You don't start from scratch. Now I have to say that even though this name is Alpha star, it has surprisingly little to do with Alpha Go or Alpha Zero that DeepMind has done before. Mainly this is entirely model free reinforcement learning. And goes more into the direction of classic deep RL. And you can see with the human data, you can already get pretty far. So these down here are the leagues of StarCraft. And this this here are percentiles of players. And you see with the supervised training, you can get almost you can get better than 80 85% of human players already. Right? So pretty, pretty impressive already simply by imitating humans. Now so the the the way to to further improve this, and let's actually go first into how the model looks like. So down here, they describe this model. That's it. So the model is supposed to map from input to output. So from the screen that the agent sees, right, and some other things to what the agent is going to do to an action a. If you simply do this at every time step, then you have a game playing agent. So first, the question is, of course, how does this happen? Now the input isn't only the thing that the agencies which is this the mini map and the mini map? I believe that's the mini map or the entire map. Well, it's it's in essence, it is a picture. It is also a list of entities. So the the game engine extracts a list of entities. And these can be inside the screen here and outside the screen for friendly. So the assumption is the agent knows about all of its units and where they are and what their statistics are. So in this entity thing, for each entity, you have a list of what is its health, what is its type, what is its position, does it carry any items and so on all the things you need to know about this entity. This is in this list of entities. And along with that also opponent entities, but only the ones that are on screen. Right. So all of this goes into this list of entities. And then the next features are scalar features. And as I understand it, scalar features are things like what race are you playing currently? What time is it in the game and so on. So these are additional features. And also baseline features. And this is mainly used to train the value network. And if you this is not going to make sense if you know nothing about reinforcement learning. But one main contribution of this paper is or not contribution, but kind of thing that they claim is that for computing the value network, they also use the observations. So all of this of the opponent player, because you know this during training, because you're doing self play, and you don't need this value network during inference. You can actually do this and this improves performance significantly. Alright so that's just for people who know RL very well. Everyone else don't don't worry too much about these things. Alright so these are the inputs, the scalar features, the entity and the minimap. Each one goes through separate encoders. So the minimap goes through a ResNet which is a convolutional network. And the entities go through a transformer which is kind of a thing to, it's appropriate to encode a set of entities right. Scalar features go through a classic feed forward network MLP. All of these get combined here into a deep LSTM that goes over time. Now the deep LSTM is what really makes the strategy because each time step, each time step a screen like this is input into the into the thing. But the agent also needs to remember what did it do last steps two steps ago right. This is important because you don't have full observability. You need to know what did I do in the in the past. And that's where the so if the last step you saw this screen and the step before you saw this screen right then all of this would go through these encoding step into the LSTM right. So the LSTM will encode now over time all of these different steps. And so you can kind of say alright if I have just started building a building I should probably not build the same building again even though I can't see it on the screen right. Because I know that three steps ago I did start building a build build a building. So this is kind of the LSTM is basically where you integrate your strategy over time. So from the LSTM you have to make two predictions. You have to make a prediction of what to do. This is the action and how valuable is your current state and how valuable is your current state. This is called the value network. This is a core component of deep reinforcement learning. These two components one is called the policy which would be everything over here and what is called the value network which is called everything over here. These are the things you need to do actor critic learning and actor critic learning is the current state of the art in deep RL. So deep mind does nothing else here except as I said they use these baseline features for the value network. But if you don't know what a value network is don't worry about it. The important part for playing the game is actually the part over here that called the policy. So first you need to do to decide what action you do and that there are many action types in Starcraft as I already said you can build a building you can move a unit you can actually move the camera that's an action type right because you want to maybe see what's over here or over here or over here. So that's an action you can do and if you have decided on what action you want to do you have to decide when do I do it. So you see the action type once you figured it out it goes into the next neural network and that decides okay when do I do it when do I do this action. So it specifies a delay. Then once you've decided what to do and when to do it it goes into the next neural network and that decides should I put this into the queue of actions because the agent here is limited to a certain number of actions per second and I think it's 22 actions per five seconds or something like this so in order to mimic you know human limitations. So there's a queue of actions to be executed and the agent needs to decide do I really want is this action so important to put it into the queue. Alright if you have decided what to do when to do it whether you would like to do it at all right then you have to you have to say it goes into the next neural network and you have to say alright which units do I want to do it with right if you want to build a building you can have to choose one or many workers to do it. I don't actually know how StarCraft works in this I'm a bit of a noob but you have to you have to select units with which to do the action for most of the thing and there I like the use of a pointer network here so what a pointer network is is a network that can point to its own inputs it's sort of like an attention network but not really in a pointer network if you have a set of inputs like we have here so entity entity entity entity right all these entities and you can see the entity embedding the entity encoder actually has skip connections that go here right so this network directly gets these these entities as input it can then write you then you have a neural network on top of that neural network that the neural network takes all of these things as an input and what the neural network will output is a pointer to one of these things right you can say look I point to this thing right here this is a called a pointer network and yeah as I said it's different from an attention network which might so an attention network is where you get a distribution actually get a distribution in both cases there is a difference but we don't have to really time to go into it here but in essence with a pointer network you can select which of these entities you want to do something with all right now you've decided on which action when whether to cue it with which unit to do it now you have to decide for some actions for example if the action is attack or heal or something this target unit which unit do you want to target or which which location on the map you want to target this is the target point here and you can see again here are skip connections from the entity encoder and from the spatial encoder to these things and while the target unit is an attention network that's this like much like a pointer network you will kind of point to places in lists the target point is a deconvolution or resnet what that means is so you have this spatial encoder here will embed the mini map so there will be a neural network right here actually let's draw the neural network in this color right here it will give you a an embedding of that right and that's what you what you feed into that's what you feed for example into the LSTM but then what you do is you have a deconvolutional network which again produces a mini map but on this mini map there there's not it's not the original mini map but it's kind of a distribution of locations so it said here here do I want to point all right so the that this neural network is responsible for producing this dot on the mini map basically saying okay I know what to do when to do it with which units to do it and so on I want to do it right here on the mini map okay and now you have it right you go from the input which are these things the mini map the entities and so on to what do I want to do where when with which units and so on right this is called a policy and it's extremely complicated every one of these boxes here is a neural network and you can see it's it's very it's a lot to train and they of course they have a lot of resources since they are deep mind but that's the the main thing all right they have a few tricks to train this and we won't go too much into this but one of the tricks is V trace from the Impala paper one of another trick is up go up going policy update and a third trick is TD lambda learning here and all of these are kind of improvements onto classic actor critic reinforcement learning style like a to see your a3c if you are interested then you can you know look into these things so that's how they train it and the question now is what's the protocol for training it we saw okay there is supervised learning cool then there is reinforcement learning all right but you can't just apply and this is in the reinforcement learning this is what we said you get kind of a reward and the reward goes into this TD lambda and V trace and and up going policy update to train the value function and the policy but the special thing that this paper introduces is what's called leak training now in in papers like alpha go or alpha zero what had been done is called self play and self play basically means you have an agent you have an agent right you have this how in a row an agent that's this is supposed to be an artificial intelligence right how to make it artificial okay it has a little hat right a funky hat it's a robot and the robot will play a copy of itself right and the copy it might be slightly different but the it basically these two these two play each other and thereby become better and better and better and you can see this like over time as as the purple one gets better the blue one gets better as well because they they kind of play against each other and when one falls behind right when one falls behind then they simply copy over from the other one they basically copy the other one and then they catch up again right they catch up right and they continue competing so by competing against each other they get better and this is called self play now people have noticed this kind of leads to instabilities because you can get kind of trapped get trapped in cycles like rock paper scissor cycles so what they do is they will actually as they get better so this is the first version right and the second version they are a bit better now so they have bigger hats right and here bigger bigger larger hats right and down here they are even better so they have like ginormous hats but they might have some weaknesses because they only play against each other right so this is the same players but over time what they will do is they will actually play occasionally play old versions of the other player or of themselves right occasionally the new versions will fall back and play old versions or not only the current versions of the agent or old versions of themselves right so this this is called fictitious self play in that you always play the you know not only play the your current kind of opponent or your current self i mean it's the same anyway because you keep copying the weights you also play the old ones and this paper goes a step further and says actually we we do this but we want to prioritize the good ones so for example we know that we know that the current ones are good right but we know that this particular one was also pretty good so far so we are we keep making we keep making these these new ones play against this one more often and this has led to kind of an improvement in these kind of self play algorithms and the real new part of this um alpha star paper is the fact that they do this league training and in the league training they this this is what it looks like but i find this graphic rather confusing i'd rather explain it like something like this all right so there is your current your current strategy and you have a hat right and you do all of the you do all of the all of the i play against myself with the smaller hat thing right i play against past versions of myself fine but then you also do you have what's called exploiters and exploiters an exploiter is a let's call it a triangle hat because it's very evil what it does is it specifically targets only the current good agent right so this this agent right here is tasked with playing old versions of itself and playing the exploiter both at the same time but the exploiter is only tasked with playing this thing so um what it can do is it can specialize in exploiting whatever weaknesses this player has of course the hope is that the this player will become better in response because there's a player trying to exploit it right so every and as this as this player becomes better than this player here is reinitialized and tries to find new weaknesses right so as this as this one continues to learn so the exploiters they are initialized you can see this here so these are called the main agents and you can see they play against each other right one of them they play against each other they play against past versions of themselves so these are past versions of themselves but then there are these main exploiters and the main exploiters they're constantly reinitialized from human data right you can see this here they're reinitialized and they only play against these main players right they don't have to deal with any of the past players or playing against themselves stuff they only try to exploit the main players and thereby the main players get better once they get better than an exploiter they are reinitialized so the exploiters are reinitialized to find new exploits of the main agents the third component is what's called a league exploiter and a league exploiter is the following so the league let's the league exploiter here and its hat is a wavy hat and what the league exploiter does is it plays against past versions of itself and others so it does play against the league exploiter sorry with smaller wavy hat it also plays against this thing by the way the this this here also plays against past versions of this and of everything else you can see here the past version arrows it goes against all past players so this this represents all the past players that ever existed and so does the so does the so here but also against past versions of this of this main exploiter here but the important thing is the current main exploiter doesn't play past versions of its of itself right so this also plays this and this place this and this place this and this also place this so the league exploiter they they do take part in this whole league business like playing against past versions of all the players but it only plays against the main ex against the main exploiters and this is a thing that i find missing here honestly i don't know if i don't understand this but i'm pretty sure i do like these also play these and that's an arrow missing in the in the drawing uh the league exploiters play the main agents but the main difference between the league exploiters and the main agents is the league exploiters they don't play themselves right there is no there's no playing themselves on the league exploiters so the league exploiters what they can do is they can find weaknesses of the entire league and kind of train train the by playing against the main opponents using those found weaknesses you bet that the main ex the main agents will get better against those major weaknesses of the entire league right so the main agents first of all they get better by playing the main exploiters because the main exploiters are mainly trying to exploit the main agents the main agents also get better by playing the league exploiters because the league exploiters find weaknesses of the entire league right so and the main agents they also get better by playing each So that makes these these main agents kind of... You can say they're trained against everything under the sun, against any possible exploit that can be found either in themselves or generally. And thereby they get really good at StarCraft, because they can counter pretty much everything. So this is how league training works and this is what I feel is the main contribution of this paper to the reinforcement learning world. Now they do an ablation study here. You can see where this ends up. So these final agents here, they end up in Grandmaster level StarCraft and beat 99. some percent of human players. So really really good. They do an ablation study of all of the tricks they use. So this is pretty much all tricks they use. And you can see here this includes this league composition. What happens if we only have main agents, then main exploiters, league exploiters, and you can see the elo going up. Then you can see multi-agent learning. How much does this fictitious self play? The fact that we prioritize to strong players and so on. How much does this help? And you again see the elo going up. How much does it help that we use human data? How much does it help that we use these different networks? They have very good ablation studies of how much each of the things help. Here they investigate what if we didn't have a camera interface? So what if we could see the entire game at once and not only the opponents that are within the camera? And what if we didn't need to move the camera? They investigate the off-policy learning corrections that we mentioned and so on. I find this very cool that they do these huge ablation studies to show really how much each of these tricks that they used helps in generating their superior performance. Here you can see how these agents develop. So over training and they have a massive infrastructure and they train for days. You can see this here. But you can see that the the main agents just get better and better and better and better. While the main exploiters of course they stay the same but they kind of keep getting reinitialized. So this main agent is trained to exploit these these sorry these main exploiters trained to exploit these main agents. This one is trying to exploit these ones. They're not by themselves really good agents but they're simply trained to to find and exploit weaknesses of the main agents. Likewise these league exploiters they do get better with the league but they are only concerned with exploiting current and past versions of the league. Also to make the main agents better. So everything is geared towards making these main agents better. And you can see it actually works. They have some analysis of which units these agents build. I'm not too versed in Starcraft to comment on this. But all in all I find this to be a very cool paper and I find it to be described fairly clear what they do. Though they do not release the source code. They release some kind of pseudo code. But the analysis and the ablations are very good. The results are let's say questionable because of course you can't compare machines to humans especially in a game where you have to make quick actions. Even if you limit the actions, they do this here. So they have this monitoring layer which limits the actions and introduces delay and so on. But still if it's not the same as a human who might not always be able to do these 22 actions per five seconds. If something quick happens they may need to have some kind of relaxation phase and so on. But they try with these kind of delays and action limits. They try to model these kind of limitations. I find this as fair as possible. This is what I find kind of problematic. So they own units as I said. The agent can also see the ones that are outside the camera. And that seems kind of shady. Because of course you can you can claim humans can do whatever command groups to also control units outside the camera. But it's not really the case. So that's sort of a distinct advantage that the machine has. But yeah in any case I find it to be very well done. And I hope this made it a bit clearer what the exact contributions are. And with that have a fun time playing against AlphaStar. Bye bye.
 Alright, let's talk about AlphaStar, Grandmaster level in StarCraft 2 using multi-agent reinforcement
 learning.
 The corresponding paper looks like this and is by Oriol Vinyals et al. from DeepMind and
 has been published in the journal of Nature recently.
 Now let me say this first.
 Stop publishing in Nature.
 This is a journal is not open access.
 It makes its readers pay for getting the article.
 So actually you can access this article or a public version of it for free but you can't
 print it, you can't download it unless you pay for it.
 And this to me, it seems ridiculous because none of this money goes to the authors of
 the article.
 None of this money goes to the reviewers.
 The review quality isn't notably better, at least in the field of computer science.
 All of this is a publicity stunt by DeepMind because Nature has been kind of impactful
 in the last decades.
 It's like, ooh, look at me, I got a big dick I publish in Nature.
 Nothing more than that.
 It's like OpenAI saying their model is too dangerous to release to the world.
 I guess DeepMind might make the same claim about AlphaStar.
 It's like too dangerous of a StarCraft player.
 Yeah, so stop this.
 Publish your research in open access.
 Nature or journals like these for computer science.
 It's a remnant of the last century.
 So go on and join everyone else in distributing knowledge.
 All right, rant over.
 Let's jump in into this article.
 So the article describes how to train a reinforcement learning agent to play the game of StarCraft
 2.
 So StarCraft 2 is this game for everyone who doesn't know.
 Just very quickly explain the game.
 StarCraft 2 is a real time strategy game and you're kind of in this top third person view
 and you control your units and the goal is kind of to move your units around and first
 of all build up buildings and using those buildings you can then produce more and more
 diverse units and ultimately you want to kind of produce some sort of army that can go to
 the opponent and destroy the opponent's base.
 So you control all of this on a computer using a mouse and a keyboard and StarCraft is notable
 for being very balanced.
 So there are three different races you can play.
 So first are the Terran which are kind of human, human-ish.
 They have marines and tanks and helicopters I believe and things like this.
 Then the Protoss are some sort of alien race that are super advanced so they can teleport
 and have energy shields and things like that.
 And then last are the Zerg and the Zerg are kind of icky ground dwelling creatures that
 infect things and spread like a disease.
 So the interesting thing here is compared to other real-time strategy games is that
 the three races they play very different.
 So the game is almost a different game if you play as a different race but they are
 so well balanced that almost any matchup is kind of a fair game between equally skilled
 players.
 So that's makes StarCraft pretty unique.
 Also pretty unique is the very very high action per minute rates that pro players get.
 Like they play this insanely fast.
 So game lasts about 10 to 15 minutes and as I said the goal is to destroy the enemy base.
 So to train an RL agent to play this is very hard because the action space is very high.
 You have to target with your mouse part of the screen.
 You have to look what is on the screen, what can I do.
 There's this mini map down here.
 There are things you can do.
 There are opponents you can target and so on.
 So all of this is very very very difficult for an RL agent.
 And at the end, after 10 minutes, you play play play play play and after 10 minutes you
 either win or you lose.
 And the RL agent has to figure out which of the actions that I did during those 10 minutes
 right.
 Was it this one?
 Was it this one?
 Which led to me winning or losing?
 These are very hard problems for reinforcement learning.
 And DeepMind has combined almost every trick in the book known so far to RL to achieve
 this.
 Now the main contribution I'd say here that is novel is what is called league training
 and we'll get to that.
 So first of all, if you don't know what reinforcement learning is, reinforcement learning is basically
 what I just described.
 You have an input right, which could be this thing here and you have a set of actions that
 you can do, which the set of actions here is anywhere you can click right, you can click
 anywhere on the screen.
 And you have to do this over and over and over and over again until you either win or
 you lose.
 And from that you will see you will at the end receive Yeah, you win or you lose and
 then you have to kind of learn to play the game.
 So it's machine learning hardcore because you get minimal information and have to achieve
 a lot of things from it.
 So the first thing that DeepMind actually does is it does supervised learning.
 And we'll get into how exactly the model works later.
 But first thing DeepMind does is it trains an agent to simply imitate humans, right?
 So you have human data.
 And from the human data, you so these are games played by humans, good humans, right?
 Not not people like me.
 So these these are games played with humans from a significantly high ELO.
 And the first thing you extract is this Z here.
 Now Z is is called a statistics vector.
 And as I understand it, it's mainly the build order, which means in which order do you build
 your buildings and units and this is very important in StarCraft.
 This is a strategic decision where you say, okay, first, I'm going to build three worker
 units.
 This is like three workers, worker, worker, worker, and then I'm going to build a house
 and then I'm going to and so on.
 So these are major strategic decisions that that you kind of have to make with minutes,
 minutes ahead of time to plan in advance.
 And this this is kind of stays constant for the game.
 So this is extracted and provided to the model as an input.
 So what is the current strategy basically the current overall strategy?
 The second thing that is extracted is this is at every time step, the observation that
 the humans had so the screen that humans see, and also the actions that the human did, right?
 So the human takes its mouse and clicks somewhere, right?
 This is supposed to be a mouse pointer and clicks here, right?
 And then the model, this part here, this is the model.
 And this is the policy function of the model.
 So the policy decides what to do, right?
 Is trained to match the action that the human did.
 So in essence, first, you train an agent to simply imitate humans.
 And this you can do by supervised learning, right?
 This is classic machine learning.
 Each each step you have this input, which is an image, and you have the strategy you're
 trying to follow.
 And from these two, you're simply trying to match the action that the human did, assuming
 the human made a good decision.
 So this is how you initialize, right?
 You don't start from scratch.
 Now I have to say that even though this name is Alpha star, it has surprisingly little
 to do with Alpha Go or Alpha Zero that DeepMind has done before.
 Mainly this is entirely model free reinforcement learning.
 And goes more into the direction of classic deep RL.
 And you can see with the human data, you can already get pretty far.
 So these down here are the leagues of StarCraft.
 And this this here are percentiles of players.
 And you see with the supervised training, you can get almost you can get better than
 80 85% of human players already.
 Right?
 So pretty, pretty impressive already simply by imitating humans.
 Now so the the the way to to further improve this, and let's actually go first into how
 the model looks like.
 So down here, they describe this model.
 That's it.
 So the model is supposed to map from input to output.
 So from the screen that the agent sees, right, and some other things to what the agent is
 going to do to an action a.
 If you simply do this at every time step, then you have a game playing agent.
 So first, the question is, of course, how does this happen?
 Now the input isn't only the thing that the agencies which is this the mini map and the
 mini map?
 I believe that's the mini map or the entire map.
 Well, it's it's in essence, it is a picture.
 It is also a list of entities.
 So the the game engine extracts a list of entities.
 And these can be inside the screen here and outside the screen for friendly.
 So the assumption is the agent knows about all of its units and where they are and what
 their statistics are.
 So in this entity thing, for each entity, you have a list of what is its health, what
 is its type, what is its position, does it carry any items and so on all the things you
 need to know about this entity.
 This is in this list of entities.
 And along with that also opponent entities, but only the ones that are on screen.
 Right.
 So all of this goes into this list of entities.
 And then the next features are scalar features.
 And as I understand it, scalar features are things like what race are you playing currently?
 What time is it in the game and so on.
 So these are additional features.
 And also baseline features.
 And this is mainly used to train the value network.
 And if you this is not going to make sense if you know nothing about reinforcement learning.
 But one main contribution of this paper is or not contribution, but kind of thing that
 they claim is that for computing the value network, they also use the observations.
 So all of this of the opponent player, because you know this during training, because you're
 doing self play, and you don't need this value network during inference.
 You can actually do this and this improves performance significantly.
 Alright so that's just for people who know RL very well.
 Everyone else don't don't worry too much about these things.
 Alright so these are the inputs, the scalar features, the entity and the minimap.
 Each one goes through separate encoders.
 So the minimap goes through a ResNet which is a convolutional network.
 And the entities go through a transformer which is kind of a thing to, it's appropriate
 to encode a set of entities right.
 Scalar features go through a classic feed forward network MLP.
 All of these get combined here into a deep LSTM that goes over time.
 Now the deep LSTM is what really makes the strategy because each time step, each time
 step a screen like this is input into the into the thing.
 But the agent also needs to remember what did it do last steps two steps ago right.
 This is important because you don't have full observability.
 You need to know what did I do in the in the past.
 And that's where the so if the last step you saw this screen and the step before you saw
 this screen right then all of this would go through these encoding step into the LSTM
 right.
 So the LSTM will encode now over time all of these different steps.
 And so you can kind of say alright if I have just started building a building I should
 probably not build the same building again even though I can't see it on the screen right.
 Because I know that three steps ago I did start building a build build a building.
 So this is kind of the LSTM is basically where you integrate your strategy over time.
 So from the LSTM you have to make two predictions.
 You have to make a prediction of what to do.
 This is the action and how valuable is your current state and how valuable is your current
 state.
 This is called the value network.
 This is a core component of deep reinforcement learning.
 These two components one is called the policy which would be everything over here and what
 is called the value network which is called everything over here.
 These are the things you need to do actor critic learning and actor critic learning
 is the current state of the art in deep RL.
 So deep mind does nothing else here except as I said they use these baseline features
 for the value network.
 But if you don't know what a value network is don't worry about it.
 The important part for playing the game is actually the part over here that called the
 policy.
 So first you need to do to decide what action you do and that there are many action types
 in Starcraft as I already said you can build a building you can move a unit you can actually
 move the camera that's an action type right because you want to maybe see what's over
 here or over here or over here.
 So that's an action you can do and if you have decided on what action you want to do
 you have to decide when do I do it.
 So you see the action type once you figured it out it goes into the next neural network
 and that decides okay when do I do it when do I do this action.
 So it specifies a delay.
 Then once you've decided what to do and when to do it it goes into the next neural network
 and that decides should I put this into the queue of actions because the agent here is
 limited to a certain number of actions per second and I think it's 22 actions per five
 seconds or something like this so in order to mimic you know human limitations.
 So there's a queue of actions to be executed and the agent needs to decide do I really
 want is this action so important to put it into the queue.
 Alright if you have decided what to do when to do it whether you would like to do it at
 all right then you have to you have to say it goes into the next neural network and you
 have to say alright which units do I want to do it with right if you want to build a
 building you can have to choose one or many workers to do it.
 I don't actually know how StarCraft works in this I'm a bit of a noob but you have to
 you have to select units with which to do the action for most of the thing and there
 I like the use of a pointer network here so what a pointer network is is a network that
 can point to its own inputs it's sort of like an attention network but not really in a pointer
 network if you have a set of inputs like we have here so entity entity entity entity
 right all these entities and you can see the entity embedding the entity encoder actually
 has skip connections that go here right so this network directly gets these these entities
 as input it can then write you then you have a neural network on top of that neural network
 that the neural network takes all of these things as an input and what the neural network
 will output is a pointer to one of these things right you can say look I point to this thing
 right here this is a called a pointer network and yeah as I said it's different from an
 attention network which might so an attention network is where you get a distribution actually
 get a distribution in both cases there is a difference but we don't have to really time
 to go into it here but in essence with a pointer network you can select which of these entities
 you want to do something with all right now you've decided on which action when whether
 to cue it with which unit to do it now you have to decide for some actions for example
 if the action is attack or heal or something this target unit which unit do you want to
 target or which which location on the map you want to target this is the target point
 here and you can see again here are skip connections from the entity encoder and from the spatial
 encoder to these things and while the target unit is an attention network that's this like
 much like a pointer network you will kind of point to places in lists the target point
 is a deconvolution or resnet what that means is so you have this spatial encoder here will
 embed the mini map so there will be a neural network right here actually let's draw the
 neural network in this color right here it will give you a an embedding of that right
 and that's what you what you feed into that's what you feed for example into the LSTM but
 then what you do is you have a deconvolutional network which again produces a mini map but
 on this mini map there there's not it's not the original mini map but it's kind of a distribution
 of locations so it said here here do I want to point all right so the that this neural
 network is responsible for producing this dot on the mini map basically saying okay
 I know what to do when to do it with which units to do it and so on I want to do it right
 here on the mini map okay and now you have it right you go from the input which are these
 things the mini map the entities and so on to what do I want to do where when with which
 units and so on right this is called a policy and it's extremely complicated every one of
 these boxes here is a neural network and you can see it's it's very it's a lot to train
 and they of course they have a lot of resources since they are deep mind but that's the the
 main thing all right they have a few tricks to train this and we won't go too much into
 this but one of the tricks is V trace from the Impala paper one of another trick is
 up go up going policy update and a third trick is TD lambda learning here and all of these
 are kind of improvements onto classic actor critic reinforcement learning style like a
 to see your a3c if you are interested then you can you know look into these things so
 that's how they train it and the question now is what's the protocol for training it
 we saw okay there is supervised learning cool then there is reinforcement learning all right
 but you can't just apply and this is in the reinforcement learning this is what we said
 you get kind of a reward and the reward goes into this TD lambda and V trace and and up
 going policy update to train the value function and the policy but the special thing that
 this paper introduces is what's called leak training now in in papers like alpha go or
 alpha zero what had been done is called self play and self play basically means you have
 an agent you have an agent right you have this how in a row an agent that's this is
 supposed to be an artificial intelligence right how to make it artificial okay it has
 a little hat right a funky hat it's a robot and the robot will play a copy of itself right
 and the copy it might be slightly different but the it basically these two these two play
 each other and thereby become better and better and better and you can see this like over
 time as as the purple one gets better the blue one gets better as well because they
 they kind of play against each other and when one falls behind right when one falls behind
 then they simply copy over from the other one they basically copy the other one and
 then they catch up again right they catch up right and they continue competing so by
 competing against each other they get better and this is called self play now people have
 noticed this kind of leads to instabilities because you can get kind of trapped get trapped
 in cycles like rock paper scissor cycles so what they do is they will actually as they
 get better so this is the first version right and the second version they are a bit better
 now so they have bigger hats right and here bigger bigger larger hats right and down here
 they are even better so they have like ginormous hats but they might have some weaknesses because
 they only play against each other right so this is the same players but over time what
 they will do is they will actually play occasionally play old versions of the other player or of
 themselves right occasionally the new versions will fall back and play old versions or not
 only the current versions of the agent or old versions of themselves right so this this
 is called fictitious self play in that you always play the you know not only play the
 your current kind of opponent or your current self i mean it's the same anyway because you
 keep copying the weights you also play the old ones and this paper goes a step further
 and says actually we we do this but we want to prioritize the good ones so for example
 we know that we know that the current ones are good right but we know that this particular
 one was also pretty good so far so we are we keep making we keep making these these
 new ones play against this one more often and this has led to kind of an improvement
 in these kind of self play algorithms and the real new part of this um alpha star paper
 is the fact that they do this league training and in the league training they this this
 is what it looks like but i find this graphic rather confusing i'd rather explain it like
 something like this all right so there is your current your current strategy and you
 have a hat right and you do all of the you do all of the all of the i play against myself
 with the smaller hat thing right i play against past versions of myself fine but then you
 also do you have what's called exploiters and exploiters an exploiter is a let's call
 it a triangle hat because it's very evil what it does is it specifically targets only the
 current good agent right so this this agent right here is tasked with playing old versions
 of itself and playing the exploiter both at the same time but the exploiter is only
 tasked with playing this thing so um what it can do is it can specialize in exploiting
 whatever weaknesses this player has of course the hope is that the this player will become
 better in response because there's a player trying to exploit it right so every and as
 this as this player becomes better than this player here is reinitialized and tries to
 find new weaknesses right so as this as this one continues to learn so the exploiters they
 are initialized you can see this here so these are called the main agents and you can see
 they play against each other right one of them they play against each other they play
 against past versions of themselves so these are past versions of themselves but then there
 are these main exploiters and the main exploiters they're constantly reinitialized from human
 data right you can see this here they're reinitialized and they only play against these main players
 right they don't have to deal with any of the past players or playing against themselves
 stuff they only try to exploit the main players and thereby the main players get better once
 they get better than an exploiter they are reinitialized so the exploiters are reinitialized
 to find new exploits of the main agents the third component is what's called a league
 exploiter and a league exploiter is the following so the league let's the league exploiter here
 and its hat is a wavy hat and what the league exploiter does is it plays against past versions
 of itself and others so it does play against the league exploiter sorry with smaller wavy
 hat it also plays against this thing by the way the this this here also plays against
 past versions of this and of everything else you can see here the past version arrows it
 goes against all past players so this this represents all the past players that ever
 existed and so does the so does the so here but also against past versions of this of
 this main exploiter here but the important thing is the current main exploiter doesn't
 play past versions of its of itself right so this also plays this and this place this
 and this place this and this also place this so the league exploiter they they do take
 part in this whole league business like playing against past versions of all the players but
 it only plays against the main ex against the main exploiters and this is a thing that
 i find missing here honestly i don't know if i don't understand this but i'm pretty sure
 i do like these also play these and that's an arrow missing in the in the drawing uh
 the league exploiters play the main agents but the main difference between the league
 exploiters and the main agents is the league exploiters they don't play themselves right
 there is no there's no playing themselves on the league exploiters so the league exploiters
 what they can do is they can find weaknesses of the entire league and kind of train train
 the by playing against the main opponents using those found weaknesses you bet that
 the main ex the main agents will get better against those major weaknesses of the entire
 league right so the main agents first of all they get better by playing the main exploiters
 because the main exploiters are mainly trying to exploit the main agents the main agents
 also get better by playing the league exploiters because the league exploiters find weaknesses
 of the entire league right so and the main agents they also get better by playing each
 So that makes these these main agents kind of...
 You can say they're trained against everything under the sun,
 against any possible exploit that can be found either in themselves or
 generally. And thereby they get really good at StarCraft,
 because they can counter pretty much everything. So this is how
 league training works and this is what I feel is the main contribution of this
 paper to the reinforcement learning world.
 Now they do an ablation study here. You can see
 where this ends up. So these final agents here,
 they end up in Grandmaster level StarCraft and beat 99.
 some percent of human players. So really really good.
 They do an ablation study of all of the tricks they use.
 So this is pretty much all tricks they use.
 And you can see here this includes this league composition.
 What happens if we only have main agents, then main exploiters, league
 exploiters, and you can see the elo going up.
 Then you can see multi-agent learning. How much does this fictitious
 self play? The fact that we prioritize to strong
 players and so on. How much does this help? And you again see the elo
 going up. How much does it help that we use human data?
 How much does it help that we use these different networks?
 They have very good ablation studies of how much each of the
 things help. Here they investigate what if we didn't have a camera
 interface? So what if we could see the entire game at once and not only
 the opponents that are within the camera?
 And what if we didn't need to move the camera?
 They investigate the off-policy learning corrections that we mentioned
 and so on. I find this very cool that they do these
 huge ablation studies to show really how much each of these tricks that they used
 helps in generating their superior performance.
 Here you can see how these agents develop.
 So over training and they have a massive infrastructure and they train for
 days. You can see this here. But you can see that the
 the main agents just get better and better and better
 and better. While the main exploiters of course
 they stay the same but they kind of keep getting reinitialized.
 So this main agent is trained to exploit these
 these sorry these main exploiters trained to exploit these main agents.
 This one is trying to exploit these ones. They're not by themselves
 really good agents but they're simply trained to
 to find and exploit weaknesses of the main agents.
 Likewise these league exploiters they do get better with the league
 but they are only concerned with exploiting current and past versions of
 the league. Also to make the main agents better.
 So everything is geared towards making these main agents better.
 And you can see it actually works.
 They have some analysis of which units these agents build.
 I'm not too versed in Starcraft to comment on this.
 But all in all I find this to be a very cool paper
 and I find it to be described fairly clear what they do.
 Though they do not release the source code.
 They release some kind of pseudo code. But the analysis and the ablations
 are very good. The results are let's say questionable because of course
 you can't compare
 machines to humans especially in a game where you have to make quick actions.
 Even if you limit the actions, they do this here.
 So they have this monitoring layer which limits the actions and
 introduces delay and so on. But still if it's not the same as a
 human who might not always be able to do these 22
 actions per five seconds. If something quick happens they may
 need to have some kind of relaxation phase and so on.
 But they try with these kind of delays and action limits. They try to
 model these kind of limitations.
 I find this as fair as possible.
 This is what I find kind of problematic. So they own units as I said.
 The agent can also see the ones that are outside the camera.
 And that seems kind of shady. Because of course you can you can claim
 humans can do whatever command groups to also
 control units outside the camera. But it's not really the case.
 So that's sort of a distinct advantage that the machine has.
 But yeah in any case I find it to be very well done.
 And I hope this made it a bit clearer what the exact contributions are.
 And with that have a fun time playing against AlphaStar.
 Bye bye.