What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper google rl deep rl deep reinforcement learning on-policy on policy off policy replay buffer normalization initialization control continuous control deep neural networks agent environment mujoco hyperparameters learning rate optimizer adam entropy regularization grid search
#ai #research #machinelearning

Online Reinforcement Learning is a flourishing field with countless methods for practitioners to choose from. However, each of those methods comes with a plethora of hyperparameter choices. This paper builds a unified framework for five continuous control tasks and investigates in a large-scale study the effects of these choices. As a result, they come up with a set of recommendations for future research and applications.

OUTLINE:
0:00 - Intro & Overview
3:55 - Parameterized Agents
7:00 - Unified Online RL and Parameter Choices
14:10 - Policy Loss
16:40 - Network Architecture
20:25 - Initial Policy
24:20 - Normalization & Clipping
26:30 - Advantage Estimation
28:55 - Training Setup
33:05 - Timestep Handling
34:10 - Optimizers
35:05 - Regularization
36:10 - Conclusion & Comments

Paper: https://arxiv.org/abs/2006.05990

Abstract:
In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress (Engstrom'20). As a step towards filling that gap, we implement over 50 such "choices" in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.

Authors: Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, Olivier Bachem


Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there, today we're looking at what matters in on policy reinforcement learning, a large-scale empirical study by Google Brain. On a high level, this paper investigates five different continuous control tasks and they train agents with all the different choices that you can make basically on these continuous control tasks. The different choices are like network, width and height of the value and policy network, learning rate, type of loss function, regularization constants, and they train all of these agents and they try to parse out what works in general and what doesn't. They have some surprising findings that number seven will surprise you. Yeah, okay, so that's the study on a high level. As always, if you like content like this, consider subscribing and sharing it out. That would be excellent. So they say that on policy reinforcement learning has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state of the art implementations take numerous low and high level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. So the sort of things they mean here are the things that when you read the paper, the algorithm will be sort of described pretty well on the main idea. But then if you look at the code, there's a whole bunch of hacks there. Like on the Atari environment, you have to repeat certain actions. You have to introduce sticky actions. Then the question is, do you have like a random starts or the always start at the exact same time? Therefore, the randomness of the level is not given. Then you whether or not you normalize certain observations. But we've had these things even in supervised learning or NLP, things like this, we've had pre processing. I remember the first ResNet paper that beat ImageNet to a significant degree over the last year's baseline. It was, oh yeah, we have the simple idea of the ResNet. And they have an entire section where they go, oh, and we do this normalization, we do this pre processing, we do this and this and this and this and this. And I mean, there's an argument to be made for all of these choices. But often it's hard to disentangle if the choice choices of these pre processing things or whatever the choices are matters, or if the idea in the paper matters. And it's also very hard to compare different things. So what they're doing here, so I would say this is not only a problem in RL, this is a problem generally. They say as we step towards filling the gap, we implement over 50 such choices in a unified on policy RL framework, allowing us to investigate their impact in a large scale empirical study. So large scale empirical study is basically means grid search over these choices, kind of smart grid search. We train over 250,000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on policy training of RL agents. So as far as I could figure out the code and or and or the checkpoints of these 250,000 agents, or the code of this unified on policy RL framework is not available yet. And I don't know if it's going to be but basically what they're doing is they're building one agent. So in usually you have this agent environment dichotomy right here, you get observation and reward. And here you get you give action, they build one single agent that has a lot of switches that has a lot of like flags that you can say, okay, either do you want this loss or this loss? Cool. Do you want this regularization or this regularization? And if so, by how much right? And so on. So I have this agent with lots and lots and lots of switches over 50 of these choices that they implement right here, and they can basically turn each one on and off. And therefore they can investigate these algorithms. So let's jump over the most surprising finding, which, okay, the most I can tell you the most surprising finding is that the initialized policy initialization scheme matters significantly. Okay, that's what people maybe didn't know. What also matters a lot is the learning rate and things like the discount factor. But I think people in RL were already familiar with that. I find it also interesting what doesn't matter, namely, most things seem to not really matter too much. But there might be other explanations for this. Alright, so they say we consider the setting of on policy reinforcement learning for continuous control. Now this is where I have a bit of a problem right here. Because the title is what matters in on policy reinforcement learning. It's not what matters in on policy reinforcement learning for continuous control. They do say in the abstract here, as you've already seen, in the last sentence that they have continuous control environment, five continuous control environments. But yeah, I get it. We need to make the title a bit click baity. But the title overstates a bit what this paper says. This paper basically says what works in these particular five continuous control environments, right? So they vary a lot of things with respect to the agent, but they keep the environments relatively constant. And it's not five diverse environments. It's five mojo co continuous control environments that are very, very, very similar to each other in terms of their observation in terms of how the world works and so on. So consider this paper as an investigation in what works and doesn't work for these five and possibly for very relatively close environments. So that's that's I think my biggest trouble I have with this paper right here is sort of it overstates what it what it says in the title. But I mean, the investigation itself is done, I feel very, very well. So they say they have a unified on policy learning algorithm, where they research prior work took popular code bases made a list of common Lewis choices and then implemented everything into starting from the seed RL code base. And RL is kind of a framework for distributed or for reinforcement learning in general. And they say whenever we faced we were faced with implementation decisions that required us to take decisions that could not be clearly motivated or had alternative solutions, we further added such decisions as additional choices. So this I feel, if I write research code, this is generally what I do, right? I write my research code, and whenever I come to a place where I'm like, should they use this or this should use this optimizer or this optimizer, I simply make a flag. And then even if it's just one choice for now, right, just make a flag and parameterize everything. And that's, that's, that's the thing here, they parameterize everything. But other than I would do now, then I would sort of sparsely explore the space of these parameters. And they do a more dense, dense observation or dense sampling of this space than it might mean myself would do with limited resources. Of course, being Google, it is possible to do these kinds of things where you investigate all the choices. So they say here difficulty of investigating choices. The primary goal of this paper is to understand how the different choices affect the final performance of an agent and derive recommendations for these choices. There are two key reasons why this is challenging. First, we're mainly interested in insights on choices for good hyper parameters. Yet if all choices are sampled randomly, the performance is very bad and little training progress is made. So that means if you have if you have all of these hyper parameters, then let's let's consider like a three dimensional hyper parameter space, then there are combinations of hyper parameters that are very good right here, maybe here. So there's this this cube in here. That's sort of very good. But the rest aren't really good. So if you just simply sample from anywhere in the space, like here, or here, or here, or here, or here, you will basically never get anything that works, you sort of have to hit the combination correctly. And that's that's a problem in three dimensions, but it's way more a problem in 50 plus dimensions like they have here. So they have to resort to a different strategy. They have to go and basically start out from a good configurations where they say they group these we create groups of choices around thematic groups where we suspect interactions between different choices. For example, we group together all choices related to neural network architecture, we also include the learning rate in all of the groups, as we suspect it may interact with many other choices. And in each experiment, we train a large number of models where we randomly sample the choices within the corresponding group. All other settings for choices not in the group are set to settings of a competitive base configuration that is close to the default PPO versus V2 configuration. Okay, so what they're doing basically is they're saying, now let's, let's consider these. So these groups, you can now think of single dimensions in this space. So or, yeah, so let's consider the space of groups. Let's say you have two different groups. One is the group of network architecture parameters. And the other one is a group of learning behavior like learning rate and training algorithm parameter. What they're saying is they're saying we know of a configuration right here that is good. This is PPO versus two V version two. And now what we're going to do is we're simply going to keep in each experiment, we're going to, if we want to investigate the network architecture, let's say that's this axis, we're going to keep all the other groups the same as this default configuration and only investigate, only basically move this point to the left and to the right. And we're not going to move it up and down, we're going to keep the learning dynamic parameters of the other group or all of the other groups we're going to keep the same and only move it in in the architecture parameter space. Now of course, this is not just one parameter this since they make these groups, this is a multi multi parameter. So at each point here, you can imagine like a little subspace of the inner group and they then sample from these. And that becomes much more feasible, right? So now maybe you have, let's say you have 10 groups of five parameters each, you can densely sample five parameters, like that's sort of possible, you cannot densely sample 50, but you can densely sample five. So what you would do is you would keep the other 45 constant that would correspond to this dimension and all the other dimensions, and you would only vary within the group, which would correspond to this dimension. So now you see that the problem again, of course, is that you're always starting from this point, and you're basically only exploring along the axis of this of this group space, because you always keep one, keep the others constant. And that basically, to me, that means that these experiments are going to be heavily favored in in terms of which of the algorithms is closest to this to this baseline, because if so, if I go with with this particular algorithm, I know that these parameters are the best for this particular algorithm, where if I now use any other algorithm, these parameters might not be the best. And my only my only way of adjusting to that other algorithm is by individually moving here while keeping others constant, so I can basically only improve with it along one of the groups. I hope this makes sort of sense that it feels like this experiment biases the results in favor of whatever is made, whatever choices are made in this baseline. So keep that in mind. Now that being said, PPO, of course, is very popular baseline. So it makes total sense to use that as a as a base to explore from. But it's not like they're doing an actual dense grid sampling of the space. They're doing a sparse sampling in the group space and then a dense sampling within each group. All right, so they let's go into the experiments. The first thing they investigate are the policy losses. Now this is this is a rather important topic. And that basically means how do you train the policy and the choices here are of course PPO, like we saw the proximal policy optimization, but there are also others, namely, for example, policy gradient. You might know that if you learn about reinforcement learning, you will inevitably learn about policy gradients like the first thing you learn next to Q learning. And then V trace is another sort of policy loss. V trace is optimized for distributed reinforcement learning. And they have a bunch of others. And they here they say the goal of this study is to better understand the importance of the policy loss function in the on policy setting considered in this paper was not to provide a general statement that one of the losses is better than the others, as some of them were specifically designed for other settings. Now I, of course, I agree with this with this statement. It's nice that they repeated again right here. So all the results right here are just valid for these environments or environments very similar to these. And you have to keep in mind that the baseline parameters are PPO V2. And they only ever vary one group from these baseline parameters. So that's why in this experiment, for example, it doesn't seem too surprising that the PPO loss, as you can see, outperforms in every single experiment here. Whereas the other losses underperform. So the recommendation is use the PPO policy loss, start with the clipping threshold to 0.25, but also try lower and higher values if possible, because they have found and they have more experiments in the appendix. The appendix is full of these experiments and you can go and look at them. So they but the general recommendation here for them is to use the PPO policy loss if you have these continuous control tasks, and that there is a strong influence of this clipping threshold that is in PPO. Different thing network architecture. And that's basically you have you always have a value network and a policy network. And the question is how many layers how deep and so on. Should you make them? These things here are just MLP since this is continuous control tasks, you don't learn from pixels. As far as I understand it, you learn from the states or the sensors on these robot simulated robots. Now you got this here. They say separate value and policy networks appear to lead to better performance on four out of the five environments. And further regarding network sizes, the optimal width of the policy M of the policy network depends on the complexity of the environment and too low or too high values costs can cause significant drop in performance. But for the value function, there seems to be no downside in using wider networks. Moreover, on some environments, it is beneficial to make the value network wider than the policy one, EGN half cheetah, the best results are achieved with 1632 units per layer, yada yada yada yada. So some there, this is a thing that sort of crystallizes out of this paper, because what you're doing is you have this one policy network and one value network like it's it's this dichotomy where the value network tries to estimate the reward and the policy network tries to maximize the value. So you have you have two learning things here you have this is learned. And this is learned. Now there is a certain degree of interaction as the value network. Of course, the reward is dependent on your policy. So the value network sort of has to take into account the policy when it estimates the reward. But it seems to be that the policy network is the brittler one and therefore more care has to be taken to optimize it, whereas the value network seems to be a bit of more robust to changes. And you've seen this already in that the the the loss choice for the policy seems to be quite important. And here also the network parameters for the policy seem to be the things you have to actually tune per environment, whereas for the value you can pretty much go you can pretty much get any wide network will kind of do. Okay. So they say as for activation functions, we observe that tan H activations perform best and relu perform worst, which is interesting, right, because you would think that in other deep learning tasks relu's have become pretty popular and usually outperform these others, other activation functions. But in this case, no, but this could also be due to other things, because again, they go from these default parameters, which, for example, do not have entropy regularization built in. And if you have a relu where it's basically an unbounded function, whereas the tan H is sort of a more or more bounded function. So that could be, you know, there could be significant interactions here where they have split the groups, and then the choices might be reversed if in the other groups, these parameters were different. But for now, apparently, at tan H activations perform best. The interesting thing here is they say, interestingly, the initial policy appears to have a surprisingly high impact on the training performance. So this is how you initialize the policy network. Again, policy network appears to be the more brittle one and the one that you have to tune more. The key recipe appears to initialize the policy at the beginning of training so that the action distribution is centered around zero, regardless of the observation, and has a rather small standard deviation. This can be achieved by initializing the policy MLP with smaller weights in the last layer. So if you have this policy MLP, it has multiple layers, and then it needs to output an action distribution. So in these continuous control tasks, you basically for each of the joints you have to affect. So you have like a little walker here with four legs and what's that? That's like eight joints or something. So you have to tell this how much force it needs to apply to each of these joints. And as I understand it, that's usually given by the network outputting a mean and a standard deviation. I might be wrong here, but mean and a standard deviation for the distribution of action that's going to be applied here. And then this is sampled from that distribution, the actual force is then sampled. Now they say you should initialize the network such that the mean here is zero across or over your observations. And the way to do that is to simply initialize this last layer here with very small weights. So you and I think their recommendation is to divide to initialize this by 100 times smaller weights than all the other layers. They say other choices appear to be less important. The scale of the last layer initialization matters much less for the value MLP again than for the policy MLP. Apart from the last layer scaling network initialization, it does not matter too much. There appears to be no benefits if the standard deviation of the poly is learned for each state or once globally for all states. For the transformation of policy outputting the standard deviation soft plus and the exponent shape from similar. So most of these choices in their case appear to be relatively similar except the ones that they point out. The recommendation here is initialize the last policy layer with 100 times smaller weights, use soft plus to transform network output into action standard deviation and add a negative offset to its input to decrease the initial standard deviation of actions. Tune this offset is possible. Use tanh as both the activation function if the networks are not too deep right here. This is probably where the relu's would start to shine and transform these samples from the normal distribution to the bounded action space and to transform using a tanh. Use a wide value MLP, no layers shared with the policy, but tune the policy width. It might need to be narrower than the value MLP. Now this here, this no layers shared with the policy, this might be now a result that the policy is quite brittle. So if you can detach the value and the policy that might be of advantage. Which is also surprising right? You would think that these two networks, if they are shared layers, they would learn more about the environment, but apparently not. Then normalization and clipping. So you get a bunch of normalization and clipping techniques, which is for example observation normalization basically means that whatever comes in, you normalize it to a given range. So that's usually you do that for supervised learning. Like if you have if you have MNIST digits, so this is a mostly black image with, okay, can I draw on this with like a small portion of it is white. And what you want, this is usually in the range of zero to 255. So you have zero to 255. What you want to do is you want to normalize that such that it's in the range negative one to one, or alternatively such that its mean is zero and its standard deviation is about one. So people use both things and they tend this alone tends to already boost the performance. So the fact that it's non that this is non negative, and also the fact that this number is somewhat higher than sort of in the zero one range. These are quite important. And they're going to figure out that this is also important right here. So their recommendation is always use observation normalization and check if value function normalization improves performance. So for value function normalization, I believe you would you would normalize the output of the value function. So instead of the value function telling you this is how much worth something is, it simply can tell you sort of that it's more or less worth than something else in a normalized range. Gradient clipping might slightly help but is of secondary importance. Okay, cool. Yeah, so all the other things also don't seem to matter too much like per mini batch advantage normalization and gradient observation clipping. Then advantage estimation. So advantage estimation in reinforcement learning is basically the value network needs to be trained, right? You take a step and a step and a step and a step and in each step you get a reward. And you get you perform many steps. Now the value network sitting right here needs to be trained to predict the total rewards that you can get from here on until the end of the episode. Now usually what you do is you can bootstrap this by sort of a temporal difference thing in that you consider a few steps into the future and then you ask your own value network what it thinks of the rest of the episode. So basically you don't train on the entire rest of the episode, you train on the difference between this and this. And then you can get way more complicated where you actually ask your value network at each step what it thinks and then you go to that value network while integrating this reward but you also go to this value network while integrating these two rewards and so on and then your target becomes sort of a mixture of all of these things. You can get super complex with these different variants and they say we compare the most commonly used advantage estimators n-step, GAE and V-trace and their hyperparameters and their recommendation is use the GAE with lambda equals 0.9. I feel this is not too surprising right here because this n-step is a very basic estimator and the GAE and the V-trace are better and they say the GAE and the V-trace they appear to perform better and they have not found a significant performance difference between the two. So cool. Last thing, no this is second to last thing, almost last thing. Training setup. Now I believe this becomes more important. So they investigate choices related to data collection and mini batch handling. So the number of parallel environments, the number of transitions gathered in each iteration, the number of passes over the data and so on. So this is going to matter quite a bit. The recommendation is to go over experience multiple times. So what you do in these environments is always you have a phase where you collect experience and then you have a phase where you learn from this experience. So you collect experience, you start from here, you collect a bunch of experience, you put all of that experience into a buffer which is like a database and then you have these, what they're called traces. So all of these are now episodes that your agent took. Now all of these episodes consist of many many steps that the agent took. So here is one step, here is one step, here is one step. And each of these steps are going to be one training sample. So each of these steps and also here and here are going to be one training sample. There are multiple problems here. The first and obvious one is if you just leave them in order then you'll have very very correlated mini-batches and that's not good. So you want to kind of shuffle them around in here each time before you go to them. You can go through them multiple times in different order and that works really well. They say you should go over your experience multiple times since that doesn't hurt you and it alleviates you from the necessity to collect more data. The second thing they say is you should shuffle individual transitions before assigning them to mini-batches. Okay we've concluded that. And you should recompute advantages once per data pass. Now what's the point here? Before we talked about you have to you have these advantage estimators which basically means you have to look for each step you have to look ahead a couple of steps decide what the value of this state is or the advantage. And in order to do that as we have seen you kind of look at your own estimation of that future value. So you have this value is dependent on your own estimation of the future value. Now of course if you just do if you can only do this if you have these episode traces if you have these blue episode traces still around you know which step comes after which you cannot do this anymore once this is all in mini-batches and shuffled. So what some people do is they simply compute these things once at the beginning with the value network they have and then they go multiple times over this data and just they shuffle they might shuffle each time but they keep these estimates and that's of course is more and more out of date the more often you go over the data. So what they recommend is you should always go back to this set data set recompute these estimates with your current value network then do the whole shuffling thing again and then do another epoch and then basically come back to here again and recompute the advantages. It makes a lot of sense right but they also find that this actually makes a difference. For faster wall clock time training use many parallel environments and increase the batch size both might hurt the sample complexity but they get you a faster wall clock time which makes sense right if you have more environments then you're going to collect more experience and more different experience and that will speed up your the time that you need for learning. You might collect more samples though so it will also increase your flops. Increase the number of transitions in each iteration if possible. So next thing is time step handling. What do they do? The choices related to the handling of time steps so this is the discount factor frame skip so in these environments you can choose to like ignore intermediate frames how episode termination due to time step limit are handled and their main thing here is that the discount factor is one of the most important hyper parameters and should be tuned per environment and start with a 0.99 discount factor. Drive frame skip if possible there's no need to handle environments step limits in a special way for large step limits. So the discount factor which is also unsurprising right because the discount factor is basically how much you discount future reward and that is inherently dependent on the reward structure of the environment itself so it's really unsurprising that this is a big an important hyper parameter but it's good to note. And then last second okay there's more second to last thing optimizers they investigate different optimizers we investigate two gradient based optimizers adam and rms prop as well as their hyper parameters and their result says you should use adam with momentum though i think they they found that rms prop isn't too much behind that but they say you should tune the learning rate absolutely which is also known in the community right you can't you you if you have a different problem it might require a different learning rate and they find the learning rate to be a important parameter for an important parameter for these problems. So you should tune it but the other parameters of the of these algorithms aren't too much of an influence at least on these particular problems. And then the last thing is regularization so in regularization they try different regularizing methods such as entropy regularization soft constraint entropy should not be lower than some threshold callback libeler divergence between reference distribution and so on and they say we did not find evidence that any of the investigated regularizers helped significantly on our environments with the exception of half cheetoin which all constraints help. So they don't find a particular thing but remember this again this for example here entropy regularization is used in the impala paper which is which in which proposes v trace now they here only have an experiment where they change the loss to v trace without entropy regularization and in this case they turn entropy regularization on with the ppo loss as far as I understand the paper and there you can already see that there is a space that is not explored that is the setting of the original paper that introduced the thing and I think this this if you can remember this study this study like are all GANs created equal they concluded that probably all GANs are created equal especially like Wasserstein GAN isn't too much better than anything else and the author of the Wasserstein GAN paper was furious because they didn't they clearly said in the Wasserstein GAN paper that they atom optimizer doesn't work and they had to use rms prop and then the rms prop was not in that study included so it seems that the limitations of being able to really densely explore these choices is quite it's quite hurtful in in that you can only even though this is a super large-scale study and they train so much right you can only ever make very very very limited very limited sort of conclusions in these things and I would say if you are in these types of problems definitely consider their default settings otherwise what I'd much rather do is to just go to like a piece of code that implements as close as an environment as possible to the one I want and take the hyper parameters from there in the appendix here they describe all of the things that they've tried with the choices of hyper parameters and all of the results and you zoom in on like a random one you already see that the results oftentimes are very diverse very wonky very much like maybe you know this thing isn't so relevant or there's large performance differences that are unclear between the environments so it remains to remains to be seen but the main interpretation here is that you're probably going to have to tune hyper parameters for a while on your own environments all right yeah the appendix is really long and if you want details I invite you to look at it and apart from that I'll see you next time bye bye
 Hello there, today we're looking at what matters in on policy reinforcement learning, a large-scale
 empirical study by Google Brain.
 On a high level, this paper investigates five different continuous control tasks and they
 train agents with all the different choices that you can make basically on these continuous
 control tasks.
 The different choices are like network, width and height of the value and policy network,
 learning rate, type of loss function, regularization constants, and they train all of these agents
 and they try to parse out what works in general and what doesn't.
 They have some surprising findings that number seven will surprise you.
 Yeah, okay, so that's the study on a high level.
 As always, if you like content like this, consider subscribing and sharing it out.
 That would be excellent.
 So they say that on policy reinforcement learning has been successfully applied to many different
 continuous control tasks.
 While RL algorithms are often conceptually simple, their state of the art implementations
 take numerous low and high level design decisions that strongly affect the performance of the
 resulting agents.
 Those choices are usually not extensively discussed in the literature, leading to discrepancy
 between published descriptions of algorithms and their implementations.
 So the sort of things they mean here are the things that when you read the paper, the algorithm
 will be sort of described pretty well on the main idea.
 But then if you look at the code, there's a whole bunch of hacks there.
 Like on the Atari environment, you have to repeat certain actions.
 You have to introduce sticky actions.
 Then the question is, do you have like a random starts or the always start at the exact same
 time?
 Therefore, the randomness of the level is not given.
 Then you whether or not you normalize certain observations.
 But we've had these things even in supervised learning or NLP, things like this, we've had
 pre processing.
 I remember the first ResNet paper that beat ImageNet to a significant degree over the
 last year's baseline.
 It was, oh yeah, we have the simple idea of the ResNet.
 And they have an entire section where they go, oh, and we do this normalization, we do
 this pre processing, we do this and this and this and this and this.
 And I mean, there's an argument to be made for all of these choices.
 But often it's hard to disentangle if the choice choices of these pre processing things
 or whatever the choices are matters, or if the idea in the paper matters.
 And it's also very hard to compare different things.
 So what they're doing here, so I would say this is not only a problem in RL, this is
 a problem generally.
 They say as we step towards filling the gap, we implement over 50 such choices in a unified
 on policy RL framework, allowing us to investigate their impact in a large scale empirical study.
 So large scale empirical study is basically means grid search over these choices, kind
 of smart grid search.
 We train over 250,000 agents in five continuous control environments of different complexity
 and provide insights and practical recommendations for on policy training of RL agents.
 So as far as I could figure out the code and or and or the checkpoints of these 250,000
 agents, or the code of this unified on policy RL framework is not available yet.
 And I don't know if it's going to be but basically what they're doing is they're building one
 agent.
 So in usually you have this agent environment dichotomy right here, you get observation
 and reward.
 And here you get you give action, they build one single agent that has a lot of switches
 that has a lot of like flags that you can say, okay, either do you want this loss or
 this loss?
 Cool.
 Do you want this regularization or this regularization?
 And if so, by how much right?
 And so on.
 So I have this agent with lots and lots and lots of switches over 50 of these choices
 that they implement right here, and they can basically turn each one on and off.
 And therefore they can investigate these algorithms.
 So let's jump over the most surprising finding, which, okay, the most I can tell you the most
 surprising finding is that the initialized policy initialization scheme matters significantly.
 Okay, that's what people maybe didn't know.
 What also matters a lot is the learning rate and things like the discount factor.
 But I think people in RL were already familiar with that.
 I find it also interesting what doesn't matter, namely, most things seem to not really matter
 too much.
 But there might be other explanations for this.
 Alright, so they say we consider the setting of on policy reinforcement learning for continuous
 control.
 Now this is where I have a bit of a problem right here.
 Because the title is what matters in on policy reinforcement learning.
 It's not what matters in on policy reinforcement learning for continuous control.
 They do say in the abstract here, as you've already seen, in the last sentence that they
 have continuous control environment, five continuous control environments.
 But yeah, I get it.
 We need to make the title a bit click baity.
 But the title overstates a bit what this paper says.
 This paper basically says what works in these particular five continuous control environments,
 right?
 So they vary a lot of things with respect to the agent, but they keep the environments
 relatively constant.
 And it's not five diverse environments.
 It's five mojo co continuous control environments that are very, very, very similar to each
 other in terms of their observation in terms of how the world works and so on.
 So consider this paper as an investigation in what works and doesn't work for these five
 and possibly for very relatively close environments.
 So that's that's I think my biggest trouble I have with this paper right here is sort
 of it overstates what it what it says in the title.
 But I mean, the investigation itself is done, I feel very, very well.
 So they say they have a unified on policy learning algorithm, where they research prior
 work took popular code bases made a list of common Lewis choices and then implemented
 everything into starting from the seed RL code base.
 And RL is kind of a framework for distributed or for reinforcement learning in general.
 And they say whenever we faced we were faced with implementation decisions that required
 us to take decisions that could not be clearly motivated or had alternative solutions, we
 further added such decisions as additional choices.
 So this I feel, if I write research code, this is generally what I do, right?
 I write my research code, and whenever I come to a place where I'm like, should they use
 this or this should use this optimizer or this optimizer, I simply make a flag.
 And then even if it's just one choice for now, right, just make a flag and parameterize
 everything.
 And that's, that's, that's the thing here, they parameterize everything.
 But other than I would do now, then I would sort of sparsely explore the space of these
 parameters.
 And they do a more dense, dense observation or dense sampling of this space than it might
 mean myself would do with limited resources.
 Of course, being Google, it is possible to do these kinds of things where you investigate
 all the choices.
 So they say here difficulty of investigating choices.
 The primary goal of this paper is to understand how the different choices affect the final
 performance of an agent and derive recommendations for these choices.
 There are two key reasons why this is challenging.
 First, we're mainly interested in insights on choices for good hyper parameters.
 Yet if all choices are sampled randomly, the performance is very bad and little training
 progress is made.
 So that means if you have if you have all of these hyper parameters, then let's let's
 consider like a three dimensional hyper parameter space, then there are combinations of hyper
 parameters that are very good right here, maybe here.
 So there's this this cube in here.
 That's sort of very good.
 But the rest aren't really good.
 So if you just simply sample from anywhere in the space, like here, or here, or here,
 or here, or here, you will basically never get anything that works, you sort of have
 to hit the combination correctly.
 And that's that's a problem in three dimensions, but it's way more a problem in 50 plus dimensions
 like they have here.
 So they have to resort to a different strategy.
 They have to go and basically start out from a good configurations where they say they
 group these we create groups of choices around thematic groups where we suspect interactions
 between different choices.
 For example, we group together all choices related to neural network architecture, we
 also include the learning rate in all of the groups, as we suspect it may interact with
 many other choices.
 And in each experiment, we train a large number of models where we randomly sample the choices
 within the corresponding group.
 All other settings for choices not in the group are set to settings of a competitive
 base configuration that is close to the default PPO versus V2 configuration.
 Okay, so what they're doing basically is they're saying, now let's, let's consider these.
 So these groups, you can now think of single dimensions in this space.
 So or, yeah, so let's consider the space of groups.
 Let's say you have two different groups.
 One is the group of network architecture parameters.
 And the other one is a group of learning behavior like learning rate and training algorithm
 parameter.
 What they're saying is they're saying we know of a configuration right here that is good.
 This is PPO versus two V version two.
 And now what we're going to do is we're simply going to keep in each experiment, we're going
 to, if we want to investigate the network architecture, let's say that's this axis,
 we're going to keep all the other groups the same as this default configuration and only
 investigate, only basically move this point to the left and to the right.
 And we're not going to move it up and down, we're going to keep the learning dynamic parameters
 of the other group or all of the other groups we're going to keep the same and only move
 it in in the architecture parameter space.
 Now of course, this is not just one parameter this since they make these groups, this is
 a multi multi parameter.
 So at each point here, you can imagine like a little subspace of the inner group and
 they then sample from these.
 And that becomes much more feasible, right?
 So now maybe you have, let's say you have 10 groups of five parameters each, you can
 densely sample five parameters, like that's sort of possible, you cannot densely sample
 50, but you can densely sample five.
 So what you would do is you would keep the other 45 constant that would correspond to
 this dimension and all the other dimensions, and you would only vary within the group,
 which would correspond to this dimension.
 So now you see that the problem again, of course, is that you're always starting from
 this point, and you're basically only exploring along the axis of this of this group space,
 because you always keep one, keep the others constant.
 And that basically, to me, that means that these experiments are going to be heavily
 favored in in terms of which of the algorithms is closest to this to this baseline, because
 if so, if I go with with this particular algorithm, I know that these parameters are the best
 for this particular algorithm, where if I now use any other algorithm, these parameters
 might not be the best.
 And my only my only way of adjusting to that other algorithm is by individually moving
 here while keeping others constant, so I can basically only improve with it along one of
 the groups.
 I hope this makes sort of sense that it feels like this experiment biases the results in
 favor of whatever is made, whatever choices are made in this baseline.
 So keep that in mind.
 Now that being said, PPO, of course, is very popular baseline.
 So it makes total sense to use that as a as a base to explore from.
 But it's not like they're doing an actual dense grid sampling of the space.
 They're doing a sparse sampling in the group space and then a dense sampling within each
 group.
 All right, so they let's go into the experiments.
 The first thing they investigate are the policy losses.
 Now this is this is a rather important topic.
 And that basically means how do you train the policy and the choices here are of course
 PPO, like we saw the proximal policy optimization, but there are also others, namely, for example,
 policy gradient.
 You might know that if you learn about reinforcement learning, you will inevitably learn about
 policy gradients like the first thing you learn next to Q learning.
 And then V trace is another sort of policy loss.
 V trace is optimized for distributed reinforcement learning.
 And they have a bunch of others.
 And they here they say the goal of this study is to better understand the importance of
 the policy loss function in the on policy setting considered in this paper was not to
 provide a general statement that one of the losses is better than the others, as some
 of them were specifically designed for other settings.
 Now I, of course, I agree with this with this statement.
 It's nice that they repeated again right here.
 So all the results right here are just valid for these environments or environments very
 similar to these.
 And you have to keep in mind that the baseline parameters are PPO V2.
 And they only ever vary one group from these baseline parameters.
 So that's why in this experiment, for example, it doesn't seem too surprising that the PPO
 loss, as you can see, outperforms in every single experiment here.
 Whereas the other losses underperform.
 So the recommendation is use the PPO policy loss, start with the clipping threshold to
 0.25, but also try lower and higher values if possible, because they have found and they
 have more experiments in the appendix.
 The appendix is full of these experiments and you can go and look at them.
 So they but the general recommendation here for them is to use the PPO policy loss if
 you have these continuous control tasks, and that there is a strong influence of this clipping
 threshold that is in PPO.
 Different thing network architecture.
 And that's basically you have you always have a value network and a policy network.
 And the question is how many layers how deep and so on.
 Should you make them?
 These things here are just MLP since this is continuous control tasks, you don't learn
 from pixels.
 As far as I understand it, you learn from the states or the sensors on these robot simulated
 robots.
 Now you got this here.
 They say separate value and policy networks appear to lead to better performance on four
 out of the five environments.
 And further regarding network sizes, the optimal width of the policy M of the policy network
 depends on the complexity of the environment and too low or too high values costs can cause
 significant drop in performance.
 But for the value function, there seems to be no downside in using wider networks.
 Moreover, on some environments, it is beneficial to make the value network wider than the policy
 one, EGN half cheetah, the best results are achieved with 1632 units per layer, yada yada
 yada yada.
 So some there, this is a thing that sort of crystallizes out of this paper, because what
 you're doing is you have this one policy network and one value network like it's it's this
 dichotomy where the value network tries to estimate the reward and the policy network
 tries to maximize the value.
 So you have you have two learning things here you have this is learned.
 And this is learned.
 Now there is a certain degree of interaction as the value network.
 Of course, the reward is dependent on your policy.
 So the value network sort of has to take into account the policy when it estimates the reward.
 But it seems to be that the policy network is the brittler one and therefore more care
 has to be taken to optimize it, whereas the value network seems to be a bit of more robust
 to changes.
 And you've seen this already in that the the the loss choice for the policy seems to be
 quite important.
 And here also the network parameters for the policy seem to be the things you have to actually
 tune per environment, whereas for the value you can pretty much go you can pretty much
 get any wide network will kind of do.
 Okay.
 So they say as for activation functions, we observe that tan H activations perform best
 and relu perform worst, which is interesting, right, because you would think that in other
 deep learning tasks relu's have become pretty popular and usually outperform these others,
 other activation functions.
 But in this case, no, but this could also be due to other things, because again, they
 go from these default parameters, which, for example, do not have entropy regularization
 built in.
 And if you have a relu where it's basically an unbounded function, whereas the tan H is
 sort of a more or more bounded function.
 So that could be, you know, there could be significant interactions here where they have
 split the groups, and then the choices might be reversed if in the other groups, these
 parameters were different.
 But for now, apparently, at tan H activations perform best.
 The interesting thing here is they say, interestingly, the initial policy appears to have a surprisingly
 high impact on the training performance.
 So this is how you initialize the policy network.
 Again, policy network appears to be the more brittle one and the one that you have to tune
 more.
 The key recipe appears to initialize the policy at the beginning of training so that the action
 distribution is centered around zero, regardless of the observation, and has a rather small
 standard deviation.
 This can be achieved by initializing the policy MLP with smaller weights in the last layer.
 So if you have this policy MLP, it has multiple layers, and then it needs to output an action
 distribution.
 So in these continuous control tasks, you basically for each of the joints you have
 to affect.
 So you have like a little walker here with four legs and what's that?
 That's like eight joints or something.
 So you have to tell this how much force it needs to apply to each of these joints.
 And as I understand it, that's usually given by the network outputting a mean and a standard
 deviation.
 I might be wrong here, but mean and a standard deviation for the distribution of action that's
 going to be applied here.
 And then this is sampled from that distribution, the actual force is then sampled.
 Now they say you should initialize the network such that the mean here is zero across or
 over your observations.
 And the way to do that is to simply initialize this last layer here with very small weights.
 So you and I think their recommendation is to divide to initialize this by 100 times
 smaller weights than all the other layers.
 They say other choices appear to be less important.
 The scale of the last layer initialization matters much less for the value MLP again
 than for the policy MLP.
 Apart from the last layer scaling network initialization, it does not matter too much.
 There appears to be no benefits if the standard deviation of the poly is learned for each
 state or once globally for all states.
 For the transformation of policy outputting the standard deviation soft plus and the exponent
 shape from similar.
 So most of these choices in their case appear to be relatively similar except the ones that
 they point out.
 The recommendation here is initialize the last policy layer with 100 times smaller weights,
 use soft plus to transform network output into action standard deviation and add a negative
 offset to its input to decrease the initial standard deviation of actions.
 Tune this offset is possible.
 Use tanh as both the activation function if the networks are not too deep right here.
 This is probably where the relu's would start to shine and transform these samples from
 the normal distribution to the bounded action space and to transform using a tanh.
 Use a wide value MLP, no layers shared with the policy, but tune the policy width.
 It might need to be narrower than the value MLP.
 Now this here, this no layers shared with the policy, this might be now a result that
 the policy is quite brittle.
 So if you can detach the value and the policy that might be of advantage.
 Which is also surprising right?
 You would think that these two networks, if they are shared layers, they would learn more
 about the environment, but apparently not.
 Then normalization and clipping.
 So you get a bunch of normalization and clipping techniques, which is for example observation
 normalization basically means that whatever comes in, you normalize it to a given range.
 So that's usually you do that for supervised learning.
 Like if you have if you have MNIST digits, so this is a mostly black image with, okay,
 can I draw on this with like a small portion of it is white.
 And what you want, this is usually in the range of zero to 255.
 So you have zero to 255.
 What you want to do is you want to normalize that such that it's in the range negative
 one to one, or alternatively such that its mean is zero and its standard deviation is
 about one.
 So people use both things and they tend this alone tends to already boost the performance.
 So the fact that it's non that this is non negative, and also the fact that this number
 is somewhat higher than sort of in the zero one range.
 These are quite important.
 And they're going to figure out that this is also important right here.
 So their recommendation is always use observation normalization and check if value function
 normalization improves performance.
 So for value function normalization, I believe you would you would normalize the output of
 the value function.
 So instead of the value function telling you this is how much worth something is, it simply
 can tell you sort of that it's more or less worth than something else in a normalized
 range.
 Gradient clipping might slightly help but is of secondary importance.
 Okay, cool.
 Yeah, so all the other things also don't seem to matter too much like per mini batch advantage
 normalization and gradient observation clipping.
 Then advantage estimation.
 So advantage estimation in reinforcement learning is basically the value network needs to be
 trained, right?
 You take a step and a step and a step and a step and in each step you get a reward.
 And you get you perform many steps.
 Now the value network sitting right here needs to be trained to predict the total rewards
 that you can get from here on until the end of the episode.
 Now usually what you do is you can bootstrap this by sort of a temporal difference thing
 in that you consider a few steps into the future and then you ask your own value network
 what it thinks of the rest of the episode.
 So basically you don't train on the entire rest of the episode, you train on the difference
 between this and this.
 And then you can get way more complicated where you actually ask your value network
 at each step what it thinks and then you go to that value network while integrating this
 reward but you also go to this value network while integrating these two rewards and so
 on and then your target becomes sort of a mixture of all of these things.
 You can get super complex with these different variants and they say we compare the most
 commonly used advantage estimators n-step, GAE and V-trace and their hyperparameters
 and their recommendation is use the GAE with lambda equals 0.9.
 I feel this is not too surprising right here because this n-step is a very basic estimator
 and the GAE and the V-trace are better and they say the GAE and the V-trace they appear
 to perform better and they have not found a significant performance difference between
 the two.
 So cool.
 Last thing, no this is second to last thing, almost last thing.
 Training setup.
 Now I believe this becomes more important.
 So they investigate choices related to data collection and mini batch handling.
 So the number of parallel environments, the number of transitions gathered in each iteration,
 the number of passes over the data and so on.
 So this is going to matter quite a bit.
 The recommendation is to go over experience multiple times.
 So what you do in these environments is always you have a phase where you collect experience
 and then you have a phase where you learn from this experience.
 So you collect experience, you start from here, you collect a bunch of experience, you
 put all of that experience into a buffer which is like a database and then you have these,
 what they're called traces.
 So all of these are now episodes that your agent took.
 Now all of these episodes consist of many many steps that the agent took.
 So here is one step, here is one step, here is one step.
 And each of these steps are going to be one training sample.
 So each of these steps and also here and here are going to be one training sample.
 There are multiple problems here.
 The first and obvious one is if you just leave them in order then you'll have very very correlated
 mini-batches and that's not good.
 So you want to kind of shuffle them around in here each time before you go to them.
 You can go through them multiple times in different order and that works really well.
 They say you should go over your experience multiple times since that doesn't hurt you
 and it alleviates you from the necessity to collect more data.
 The second thing they say is you should shuffle individual transitions before assigning them
 to mini-batches.
 Okay we've concluded that.
 And you should recompute advantages once per data pass.
 Now what's the point here?
 Before we talked about you have to you have these advantage estimators which basically
 means you have to look for each step you have to look ahead a couple of steps decide what
 the value of this state is or the advantage.
 And in order to do that as we have seen you kind of look at your own estimation of that
 future value.
 So you have this value is dependent on your own estimation of the future value.
 Now of course if you just do if you can only do this if you have these episode traces if
 you have these blue episode traces still around you know which step comes after which you
 cannot do this anymore once this is all in mini-batches and shuffled.
 So what some people do is they simply compute these things once at the beginning with the
 value network they have and then they go multiple times over this data and just they shuffle
 they might shuffle each time but they keep these estimates and that's of course is more
 and more out of date the more often you go over the data.
 So what they recommend is you should always go back to this set data set recompute these
 estimates with your current value network then do the whole shuffling thing again and
 then do another epoch and then basically come back to here again and recompute the advantages.
 It makes a lot of sense right but they also find that this actually makes a difference.
 For faster wall clock time training use many parallel environments and increase the batch
 size both might hurt the sample complexity but they get you a faster wall clock time
 which makes sense right if you have more environments then you're going to collect more experience
 and more different experience and that will speed up your the time that you need for learning.
 You might collect more samples though so it will also increase your flops.
 Increase the number of transitions in each iteration if possible.
 So next thing is time step handling.
 What do they do?
 The choices related to the handling of time steps so this is the discount factor frame
 skip so in these environments you can choose to like ignore intermediate frames how episode
 termination due to time step limit are handled and their main thing here is that the discount
 factor is one of the most important hyper parameters and should be tuned per environment
 and start with a 0.99 discount factor.
 Drive frame skip if possible there's no need to handle environments step limits in a special
 way for large step limits.
 So the discount factor which is also unsurprising right because the discount factor is basically
 how much you discount future reward and that is inherently dependent on the reward structure
 of the environment itself so it's really unsurprising that this is a big an important hyper parameter
 but it's good to note.
 And then last second okay there's more second to last thing optimizers they investigate
 different optimizers we investigate two gradient based optimizers adam and rms prop as well
 as their hyper parameters and their result says you should use adam with momentum though
 i think they they found that rms prop isn't too much behind that but they say you should
 tune the learning rate absolutely which is also known in the community right you can't
 you you if you have a different problem it might require a different learning rate and
 they find the learning rate to be a important parameter for an important parameter for these
 problems.
 So you should tune it but the other parameters of the of these algorithms aren't too much
 of an influence at least on these particular problems.
 And then the last thing is regularization so in regularization they try different regularizing
 methods such as entropy regularization soft constraint entropy should not be lower than
 some threshold callback libeler divergence between reference distribution and so on and
 they say we did not find evidence that any of the investigated regularizers helped significantly
 on our environments with the exception of half cheetoin which all constraints help.
 So they don't find a particular thing but remember this again this for example here
 entropy regularization is used in the impala paper which is which in which proposes v trace
 now they here only have an experiment where they change the loss to v trace without entropy
 regularization and in this case they turn entropy regularization on with the ppo loss
 as far as I understand the paper and there you can already see that there is a space
 that is not explored that is the setting of the original paper that introduced the thing
 and I think this this if you can remember this study this study like are all GANs created
 equal they concluded that probably all GANs are created equal especially like Wasserstein
 GAN isn't too much better than anything else and the author of the Wasserstein GAN paper
 was furious because they didn't they clearly said in the Wasserstein GAN paper that they
 atom optimizer doesn't work and they had to use rms prop and then the rms prop was not
 in that study included so it seems that the limitations of being able to really densely
 explore these choices is quite it's quite hurtful in in that you can only even though
 this is a super large-scale study and they train so much right you can only ever make
 very very very limited very limited sort of conclusions in these things and I would say
 if you are in these types of problems definitely consider their default settings otherwise
 what I'd much rather do is to just go to like a piece of code that implements as close as
 an environment as possible to the one I want and take the hyper parameters from there in
 the appendix here they describe all of the things that they've tried with the choices
 of hyper parameters and all of the results and you zoom in on like a random one you already
 see that the results oftentimes are very diverse very wonky very much like maybe you know this
 thing isn't so relevant or there's large performance differences that are unclear between the environments
 so it remains to remains to be seen but the main interpretation here is that you're probably
 going to have to tune hyper parameters for a while on your own environments all right
 yeah the appendix is really long and if you want details I invite you to look at it and
 apart from that I'll see you next time bye bye