Fast reinforcement learning with generalized policy updates (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper rl deep rl q learning deep reinforcement learning q learning machine learning deep q learning successor features deep mind zero shot environment agent task linear regression reward mila neural network reinforcement learning value function state value function state value
#ai #research #reinforcementlearning

Reinforcement Learning is a powerful tool, but it is also incredibly data-hungry. Given a new task, an RL agent has to learn a good policy entirely from scratch. This paper proposes a new framework that allows an agent to carry over knowledge from previous tasks into solving new tasks, even deriving zero-shot policies that perform well on completely new reward functions.

OUTLINE:
0:00 - Intro & Overview
1:25 - Problem Statement
6:25 - Q-Learning Primer
11:40 - Multiple Rewards, Multiple Policies
14:25 - Example Environment
17:35 - Tasks as Linear Mixtures of Features
24:15 - Successor Features
28:00 - Zero-Shot Policy for New Tasks
35:30 - Results on New Task W3
37:00 - Inferring the Task via Regression
39:20 - The Influence of the Given Policies
48:40 - Learning the Feature Functions
50:30 - More Complicated Tasks
51:40 - Life-Long Learning, Comments & Conclusion

Paper: https://www.pnas.org/content/early/2020/08/13/1907370117

My Video on Successor Features: https://youtu.be/KXEEqcwXn8w

Abstract:
The combination of reinforcement learning with deep learning is a promising approach to tackle important sequential decision-making problems that are currently intractable. One obstacle to overcome is the amount of data needed by learning systems of this type. In this article, we propose to address this issue through a divide-and-conquer approach. We argue that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel. By associating each task with a reward function, this problem decomposition can be seamlessly accommodated within the standard reinforcement-learning formalism. The specific way we do so is through a generalization of two fundamental operations in reinforcement learning: policy improvement and policy evaluation. The generalized version of these operations allow one to leverage the solution of some tasks to speed up the solution of others. If the reward function of a task can be well approximated as a linear combination of the reward functions of tasks previously solved, we can reduce a reinforcement-learning problem to a simpler linear regression. When this is not the case, the agent can still exploit the task solutions by using them to interact with and learn about the environment. Both strategies considerably reduce the amount of data needed to solve a reinforcement-learning problem.

Authors:
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we're looking at fast reinforcement learning with generalized policy updates by Andre Boreto, Charbo Ho, Diana Borsa, David Silver and Doyna Precu. So on high level this paper proposes a framework for reinforcement learning where you have many tasks at the same time. And they propose framework where they learn many policies at the same time that can or cannot correspond to these tasks. And then their argument is that if you now have a new task that you haven't seen before, you can easily construct a solution to that task from your old policies, basically mixing what you learned about your old tasks. And it's a pretty general framework and we're going to look at it. In my opinion, it's pretty cool for certain settings. However, I think it kind of breaks down the more general you go, which I guess is expected of such a framework. But as you can see, it's kind of math heavy, but we'll get into the examples and what it's potentially useful for. Alright, so that was it on a high level. If you like content like this, don't hesitate to subscribe to the channel and share it out, leave a like and tell me in the comments what you think. I'm still reading all of them. So I will see it. Cool, let's dive in. So they say the combination of reinforcement learning with deep learning is promising approach to tackle important sequential decision making problems that are currently intractable. Well, they're taking they're talking about, you know, things like mostly these game playing AI is like go and things like this. So where this combination of deep learning with reinforcement learning has really shined or shun, whatever one obstacle to overcome is the amount of data needed by learning systems of this type. So again, if you look at these systems like AlphaGo, they need a simulator and they need to collect enormous amounts of data, even more so with systems like the Dota AI, the OpenAI 5 Dota or StarCraft playing Alpha Star. I think it's Alpha Star. They need so many simulations in order to learn about the tasks because they always start from scratch. In this article, they say, we propose to address this issue through a divide and conquer approach. We argue that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel by associating each track task with a reward function. This problem decomposition can be seamlessly accommodated within the standard reinforcement learning formalism. Okay, so what are they saying right here? They are basically saying that if you have a task, let's say you want to get what's the from here to here. And that's very complicated. Let's make it complicated. Super duper complicated. You can basically subdivide that task into multiple subtasks, right? So here is like left turn, right turn, go straight, left turn, go straight, right turn and so on. And each of these subtasks, you can see the two right turns here might share a lot of common information. There could also be tasks that are at the same time, like you need to go forward and jump, can be decomposed into going forward and to jump. Now, they're saying is if each of these tasks now has its separate reward function in the environment, like for some reason, the environment tells you this, by the way, is task, task one, and you're going to get a positive reward if you do a right turn. And this down here is task two, the left turn task, and you're going to get a positive reward if for that task. So the entire task state can be decomposed into a vector. So in our case here, we have maybe a vector with three elements. Okay, the three elements correspond to turn right, go straight, and turn left. And now you're this this right here is your reward vector. So we're no longer talking this framework, we're no longer talking about just a reward. We're talking about a reward vector. Now each of these tasks is going to give you its own individual reward. So let's say you're here and you're actually turning right. This is going to give you a reward of one for this task, but reward of zero for the other task. So the environment will somehow tell you which tasks you you get reward for. Now there is a notion where you can map this back to a single number. And that is the second thing they introduce here. So the second thing they introduce here is this thing they call W. So W is going to be a mixing vector. W is going to be a vector. I will call W right here. This is the reward vector W is going to be the vector that tells you your final reward. So here we're going to do an inner product. So we're going to transpose this and multiply by W. And W mixes these rewards and comes up with your final reward right here. So this this is maybe the reward vector. This is the reward number. How are we going to call this reward number? So in this case, W would have to look something like this. Let's say this is an example. So the task right here would be to only do right turns. Now this is not a really nice example. We're going to see some nicer examples later on. But you can see that now the environment is specified as a vector of rewards. And you can create a specific tasks like turning right simply by adjusting how you mix these different things by this vector W. And this is going to be the key ingredient here. So they discuss your general reinforcement learning, reinforcement learning lingo. And I think we've gone through this a number of times just very, very quickly. In reinforcement learning, you're given these transitions, you are in a state, you take an action. And that leads you to get a reward R prime and you get into a state S prime in the next state. They say the reward is given by the reward function. So the reward is purely a function of where you are and what you do and where you get to. So for most reinforcement learning problems, you can actually kind of forget about this part right here. Because well, it is kind of important, but you could most reinforcement learning problems, the reward is simply a matter of where you are and what you do. And this can be a random variable, there can be randomness. But maybe it's easier if you for now think about the reward simply as a function of these two things. So what you want to discover is a policy pi, where you input, you input where you are, and the output is going to be what you should you do in that situation. Okay, that is a policy. And associated with each policy is this thing called a Q function. So you can see right here, the Q function of a policy is going to be a function of where you are and what you do. And this is a bit confusing, but it basically means that you are in state s. So you are here. And you have, let's say three options, action one, action two, action three to do. Now the Q function tells you the Q function, this is s, and the A's are the numbers. Okay, so let's say we plug in the state s and for A, we plug in number two, what it will tell you is if I am in state s, and I perform action number two, then how valuable is that for me, and value is defined by all the reward that I'm going to pick up from now until the end of time, or the end of the episode, it depends. But let's say until the end of time. So how much reward am I going to pick up from now until the end of time is a bit of a vague, not a vague question, but a difficult question. I can tell you how much I could estimate how much reward I'm going to pick up in the next step because I know what action I'm doing, I'm performing action number two. But what happens after that? Who knows? So that's where this policy right here comes in. This policy right here says, so the full definition of the Q function is if I'm in state s, and I perform action A right now, and after that, I follow policy pi, what is my reward going to be? Well, now it's well defined. So right now you do action A, and after that, you do whatever action the policy tells you in that specific situation. So that's the Q function. And you can pretty easily see that if you have a Q function, right, if you have an accurate Q function, you can get a good policy by simply always going with the action that gives you the highest Q value, because it's because of a recurrence relationship called the Bellman equation. This thing right here. So your Q function basically decomposes into the reward in the next step, as we said, plus whatever happens after that. And whatever happens after that is just by the nature of how the things are defined is going to be the Q function of whatever the policy is telling you. So you can get a pretty good policy by always doing whatever action your Q function tells you is best. This step of calculating the Q function is called a policy evaluation. And this paper here is going to generalize these notions. Sorry, so this is a policy evaluation. And then the act of selecting an action is going to be a policy improvement. These are just names. Okay, but we need to know them because the paper introduces two new things. I'm going to where do I highlight policy evaluation? I don't know, but here they say this is the policy improvement. Okay, here, policy evaluation, policy improvement. These are the two steps. So the first step is calculate the Q function. The second step is to select an action. And you can see how these things interlock, namely, we can calculate the Q function of a given policy, and we can improve that policy by selecting whatever action is best for the Q function. This paper generalizes this and you can see that there is a little R right here. So the R is just a specific way to reference the reward function used right here. Okay, and you can see it here as well. Now usually we have one policy and one reward. And so what we do is we improve the policy. And that leads us to better evaluate the Q function for a given reward function. And that leads us to improve the policy. Now this paper is going to transform this into the following. We have many policies. So we have policy one, policy two, and so on until policy P. And we also have many reward functions. Reward one, reward two, reward three, and so on until reward, let's call that R. So we have many different tasks right here. And we have many policies. Now in essence, they don't need to have some anything to do with each other for the theory of this paper. But I can simplify this a bit of how they see the world. So let's say you have an agent and the agent has been trained on simply that first task right here and has been trained using classic Q learning, reinforcement learning, whatnot. And that results in this particular policy. And then the agent just from scratch, you restart it again, you run reinforcement learning just on reward number two, and obtained policy number two, and so on. So you do this for all these rewards individually. Okay, so you give the agent a new task, and you ask it to learn a policy for that task. Now you're in a situation where if you have a new task, so R new, the question is, do you again need to train a new policy? And the answer for this paper is no. Because we have all these policies, we don't need to train a new, we can simply mix and match these policies that we already know to obtain a good solution for the new task. So how does the paper do it? It does it. It does it in the following. It defines the successor features. Maybe it's maybe it's better if we first go to an example. So the example they give here is the following. Otherwise, this I guess this might sound just a bit too abstract. Okay, so you have this world here, the agent is the thing here in yellow, and it can just move so its actions are move left, up, right down this this is one step. Okay, in the environment, there are two different objects. One object is a triangle and one object is a square. So there are a number of tasks we can define right now in this thing. So we define tasks according to a reward function. So the reward, let's say the reward one is going to be one, if, if it picks up a square, sorry, the square, and zero else. Just if it picks up a square on any given step, we give it a reward of one. We don't care about the blue triangles. Okay. And then reward two is going to be the opposite. It's going to be one, not the opposite, but one if it picks up a triangle, and zero else. So you can see the good policies right here. So pi one is a good policy for reward one, because it just goes and collects these red things, doesn't care about the blue things, just goes and collects them. Pi two, it goes and collects the blue things, doesn't care about the red things. Okay. So let's imagine that you have run reinforcement learning twice, once for reward one, and once for reward two. And now you have two policies. Okay. So you have two policies, this will lead to pi one, this will lead to pi two. And now I give you the third task. Now the third task is a bit special. It's one, if you pick up a square, and it's, it's zero else, except it's negative one, if you pick up a blue thing. So the order of these is kind of wrong, but it just for visual representation. Okay, so now you're asked to pick up the red things, but avoid the blue things. Okay, pick up as many red things as you can avoid the blue things. And again, as we said, the question is, do you now have to run reinforcement learning again in this agent with your simulator using like Q learning or something like this, from the start, or can you come up with a solution just given these two policies that will perform well on the, on this new task? Okay. And we're going to see how they do it. So what they do is they use successor features. So these successor features, I've done a video about successor features, and I'll link to that. You can look at that. But essentially, essentially, the successor features are defined like this. And for that, we need to know what this thing is right here. They simply call this a feature function. Okay, it's very, it's very ambiguous term. A feature function is a function that takes in a transition. So state action, next state, and maps it to a high dimensional vector. Note, this is almost the same as a reward function, except the reward function simply maps it to a number. Now this is mapped to a higher dimensional thing. Again, I want to, I kind of want to leave out the next state right here just to make things easier on you. So a feature here can be many, many things, but the structure of the features is going to be such that the reward function is going to be this feature times this w vector. So it was a bit, a bit not correct before when I said the reward is now a vector, the reward of a particular task w can be seen as the inner product between the features and the task vector. So w specifies the task and the features, well, they specify the features in our case, it can be, it can be fairly simple, namely, yes, I was, I was definitely wrong at the beginning. So the feature functions right here is which object do you pick up? Okay, so we define the feature function as one zero, if you pick up a square and we define the feature function as zero one, if you pick up a triangle. And now you can, and we define it as, we define it as zero zero, if you pick up nothing. And now you can fairly easily see that the reward of each task can be simply calculated by mixing the features accordingly. Okay, so reward one is going to be simply the feature times a one zero, which is the w vector. So I can specify a task by giving the appropriate w vector. And now you can see that if this is my reward function, my agent can go out into the world if it collects a square, it is going to be rewarded right here. If it collects a triangle, even though the features indicate that it collected a triangle, it doesn't care about it because the w is zero right here. If I now want to give it the new task, the same is true for r2, if I now want to give it a new task r3, right? And you remember the reward function right there, I can achieve that reward function by simply multiplying the same features, the exact same feature functions by this vector right here. Okay. Remember there is a slight difference between the reward function and the feature function in this particular example. The idea of the paper is that the feature function can be rich in in expressivity and you know, tell you all sorts of things about your current state and the reward function is just a number, right? And then the reward is specified by simply linearly mixing these features. So the structure imposed by the paper here is that there are such a thing as a feature, and any task can be described by mixing these same features. That's the issue right here. So the features are going to be constant across tasks. Whereas the w defines the task. Alright, so the the goal here is that if you have learned many, many things during your tasks, what you want to do is you want to learn this feature representation that is the same across all tasks. And then you want to simply have the w specify how to mix these features to get the reward. Now, of course, this is a very strict, very, very definition, not not a lot of things will fall into this unless you make the features like exponentially big, of course. However, they do discuss whenever a task doesn't fall into that. So I hope you're with me so far. This is the first kind of restriction we impose on our worlds that we can tackle with this framework, namely that all of our worlds have all of our tasks in this world have to be a linear mix of the same features. If that's given, then our then we can derive policies for tasks that we have never seen. We can derive good policies by doing zero learning, simply by specifying the task, we can have a good policy for that task from the policies we've already learned for the other tasks. Okay, so the reward three is now simply this. And yeah, notice it's not the same as the reward function, because the reward function had one if you pick up the square negative one, if you pick up the triangle and zero else. So the zero, we don't have to specify here because it's not part of our features. Right, so you can see that the reward function is given simply by that. And we can now, as I said, derive a good policy for this reward by looking at the other policies, even though none of these policies has ever learned to avoid anything. So it makes it defines these successor features right here. So the successor features is much like the Q function, you can see the signature is almost the same. So as a Q function tells you how much reward you're going to get if you do the action a and then follow policy pi, the successor features almost the same thing. However, it doesn't tell you what rewards you're going to get. It tells you which features you're going to get and which features by that we mean the sum of future features. Now you can see this sum, this a little bit this it, of course, it comes from the fact of the linearity up here. So it's not really an additional restriction, but simply to clarify what this means for your environment, your environment has to be able to be looked at in terms of these features and these features, they need to be cumulative. Again, that comes from the fact that it's linear, but to see. So a feature like I want an even number of steps or something like this would be terrible because and they're going into things like this later, but it would be terrible because here we have the sum. And as soon as you if you have a feature that is very high, if you have an even number of steps then or if you have a feature that counts the steps, you will never be able to to do well because if you have a feature that counts the steps, it simply counts up and up and up and up, depending on how many steps you do. And your reward can never be specified in terms of a mix of these features. And therefore your successor features are going to be useless. But in our case, where it's where feature one is pick up is how many of the sorry, I have to rephrase our feature one is whether or not you pick up a square. Therefore if we sum it up, our successor feature one is going to be the number of this is this is a pound sign, the number of squares that you pick up. Okay. Similarly, our feature two is whether or not you pick up a triangle in a particular step. So our successor feature number two is going to be the number of triangles that you pick up over time. I can see that the successor features is kind of the analogous of your Q function, but it is not in terms of a single number, the reward. It is going to be in terms of these features, which is an entire vector. And because we've constructed this in a linear way, you can also pretty clearly see that the Q function is inherently related to the successor features. You can obtain the Q function by simply multiplying the successor features by your task vector W. Now, a lot of you might be wondering where does this W come from? And in our initial case, we're just going to frame everything as being given, right? So we're given this W, we're defining everything from our godlike perspective for now. So don't think all of this is learned by now. Yeah. All right, so how can you now derive this magical new policy? So let's say we have this policy one and we have the policy two, and you have these features that you've learned constantly over both tasks. In fact, it's given, right? This pi function, we give it, we impose it that the feature one is whether you pick up a red square feature two is whether you pick up a blue square. Then we know that the reward functions can be achieved by doing the W. So this here, your W is going to be one zero, and your W here is going to be zero one. And now we want a good policy for task three. And we know we can achieve this by the one negative one W. How can we derive a good policy? And this is this algorithm, this general policy evaluation, a general policy improvement. So it assumes that you, as we said, you have many, many different, many different policy. So here you can see policy one, where's policy two, here's policy two, and so on. It assumes that you have many different features and therefore many different successor features. In fact, you have a vector of them, right? So here you can see feature one, feature two, and so on. And it also assumes that you're in a current state and you have many actions at your disposal right now. Action one, action two, and so on. So this is all the past. You've already defined your features, you have learned these policies, and now you're given a new W, W new. In our case, it's this one negative one. We want the best action. So we are in state S, and we are given this W. We want the best action. Now here is a method where we can simply calculate the best action in terms, by not reinforcement learning at all in this new task. So by structuring things like this here. So what does it really say here? This thing says we are going to evaluate all of these different cells of this tensor right here. So we're going to determine what is the successor feature number two for policy pi one in state S if I right now do a two. This is very abstract. So let's say you're here and action two is actually going to the right. So you're here. Oh, this was yellow. It doesn't matter. So this is action one. This is action two. So action two is you go to the right. You can see that this will let you pick up a triangle. Now here that's action three and so on. Okay. So what's this number going to be? So we are in state S as we said, and we do action two. So action two is going to pick up a triangle, the triangle, the picking up of a triangle means that our pi for this step, or sorry, our five for the step is going to be 01. Okay. So our successor features, this is not the features itself. This is the successor features, the successor features decompose into the next step plus all the next steps that we can follow. Okay, so all the steps that will come. So what are these features going to be is it's going to be the sum over that plus everything that follows. And I can take a little bit of a guess here, which means that this number, so we only care about feature two right here, this feature, feature two, this number is going to be one for the next step, because we are going to pick up a triangle if we do action two. But then after that, we're going to follow policy one. And policy one has been trained to pick up the red squares and not care about triangles. So I'm going to guess that every now and then it will kind of step over a triangle, but it won't fall, it won't, you know, explicitly go look for them. So let's say the episode was 10 more steps, but the board has like 100 squares. So and it has like three triangles on it. So let's say that's like three tenths in expectation. Okay, so this is going to be this is going to be the number that we're looking for. We're doing this for every single one of these cells. Okay, this this thing is going to do for every single one of these cells. And this is very similar to evaluating Q functions, except we're evaluating an entire vector right here. That's the difference to simply learning many Q functions. So if you were to evaluate only a Q function, then you would only have this first matrix, this first block right here. Okay, but you have feature one, feature two, and so on. So you calculate everything in terms of these features. And then by linearity, you can mix it with that vector. So in our case, this is going to be the one negative one, which will give you the Q functions, right? From what we've seen before, you obtain a Q function by simply mixing your successor features with your with this task vector. And if you have a Q function, you can pretty easily determine which action you should take. Now you have here a Q function with respect to every single policy, but you can simply take the max, right? So the max across all of this will determine will determine so you take the max across all the policies, which will give you the Q function for a particular action over all policies that you consider, and then you can simply take the argmax of that and determine the action you should take. Okay, so it's a pretty big evaluation. But if you do this, that means you don't have to do reinforcement learning on this task. It simply determines which action right now is the best given everything that I know from these old policies about the task. And that's not going to be like the optimal policy, per se, but it's going to be one policy that's pretty, pretty good. And you can actually prove some things across that. So they do this right here. And you can see that here is what Q learning does on this new task of picking up the squares and avoiding the triangles Q learning takes a while to get there. However, if you do what they are suggesting, and you know, you give the W, you can supply the W almost from the beginning, you see right here almost from the beginning, it is at a high reward. Now Q learning surpasses it eventually. But it's pretty impressive that without doing any learning, you are immediately good. Right. Now the caveat here, of course, is that they already need these policy pi one and pi two given to the algorithm. And that comes from previous reinforcement learning trials. And they say that they give these trials as many steps as Q learning uses. So they give them this these amounts of steps on these other tasks. So the comparison here is a bit shaky, if you ask me. But the point made is that if you have a new task right now, you can obtain very good solutions. And you don't have to do anything. Okay. And these solutions can be the basis for new reinforcement learning, right? You could start Q learning off right here and then get here much faster potentially and so on. So the next objective right here is that now we have defined the tasks and we had we know what these features are. And we know how to mix these features as imposers of the task. So what happens if we only have the reward function, we specify the task only in terms of the reward function, but we're kind of looking at the features and we're like, agent, please figure out yourself how to apply these features in order to make the reward high. And that's what this thing is right here. This GP and GPI with regress W. So you don't no longer tell it what the W is. It needs to infer it through reinforcement learning, right? And it's not really reinforcement learning. But what it does, where is it? Yeah, it's simply because all of this is linear and this thing here is given. So always remember this thing here is given. And these are the rewards that you obtain. You can simply do a regression to figure out the W of the task. Now that's going to take some time. But as you can see right here, it is going to take a lot less time than than doing Q learning from scratch. Absolutely because you have good features. So this is some this is this gets closer and closer to transfer learning, right? If you imagine that this right here is your pre trained neural network, and you simply learn the last layer of it. You freeze this you do transfer learning fine tune the last layer here we are. So it gets closer and closer and you'll see this trend right here. So it's pretty cool what you can do. But basically, I think it's a lot of math around a framework. And the more and more you relax the kind of impositions that they need for their framework, the more it gets back to simply, well, we do reinforcement learning, at least in my estimation. So before we look at that, this here is a pretty, pretty cool experiment, where they they look at how the how the different tasks can be achieved, if you give different policies. So you'll have noticed that we have always given these two, two tasks 10 and 01. These were our tasks that we trained on. And then one negative one is task we evaluated on. Okay. And you might object and say, wait a minute, these these two tasks, you know, they're pretty good as let's say, pre training tasks, because and it's basically the standard basis, right? And any other tasks can be mixed from those. So these are orthogonal vectors in this vector space. So you're being pretty generous to the system. What happens if we're not as generous? So that's what they do here. So they have different policies, and they evaluate how much you can learn with these different policies. So the way you have to read this diagram is right here, it's going to be the one zero axis as they will they label it right here. And this is going to be the 01 axis. And this is evaluation. So every direction on this circle defines a task, for example, this task right here, as you can see, is going to define the task of picking up both the squares and the triangles, right? Whatever you pick up, you get a reward. However, the task down here is going to be please pick up the squares, but avoid the triangles at all cost. Okay. And now they're going to look what happens if we supply different policies to choose from, remember, we're in this situation, we're getting in this situation where we give everything, and we give initial policies, we give the task vector. And now it's about deriving a good policy just from looking at the old policy. So no learning. As a baseline, you have Q learning, which into a given direction, tells you basically how long Q learning takes or how far Q learning gets with a given amount of steps indicated by this one, two, three, four, and so on. Yeah, you see, I think this is this in how far Q learning gets with these amounts of steps is the dotted lines right here. So Q learning gets this far with 10 to the, I don't know, four, and then this far, 10 to the five and so on. So these are comparisons. You can see that on the outside, Q learning is going to beat this, these methods. But our hope is going to be that of course, if we have this zero shot generalization, it's much better than running Q learning for really long if we get close to it. So the green thing is what we've already seen. Policies one and two will give you a fairly good extent right here. So what does it mean? It means it can solve pretty much everything from here, here, this task, this task, this task. It kind of falls off once we go down here. So once we go to the avoid section, it sort of falls off because it has never learned to avoid. Now, still, we can, of course, do the avoidance by simply imposing a negative collection. But negative collecting and avoiding aren't exactly the same thing in these environments, right? Because avoiding can also be going really close to something but not hitting it while collecting. It's not the inverse of collecting. The inverse of collecting would be like run away as far as far as possible. So we can expect that we've only ever learned to collect, we're not going to be super good at avoiding. Then the other extreme is when we give policies three and four. I haven't told you but you can see it right here. Policy three is explicitly to collect one and avoid the other, while policy four is the opposite right here. Avoid the squares, collect the triangles. And now this policy, this policy is, should be pretty good on all of the tasks in between. As you can see, it has the biggest extent right here. And that also makes sense. By the way, there's nothing down here because the task of avoiding both things doesn't really make sense because you can just stay where you are because there are also these squares where there's nothing. But you can see that the mixture of those is quite potent. So already we can see even though these span a basis, in fact an orthogonal basis as much as these, because of the nature of the features that we define for the task, they are not equivalent in mixing after. So we can be more generous, we can also be less generous if we only provide policy five. And policy five is simply to pick up, to pick up both objects. Then we're going to have a pretty hard time when it comes to avoiding things. So you can see it can do fairly well picking up the various things in a positive manner. But as soon as we cross this line into the like this horizontal line into where it's about avoiding a particular object, it's not it's not the choices of actions we have from policy five aren't going to be super good at that. And they do another they do another thing right here. So that the left thing is where they say, it's important which policies we provide. And the right thing, they want to say something like, it's important. So they want to say, if we provide more policies, that can be advantageous, because we basically have more options to choose from. So now they start off with policy four, and policy four is simply avoid the squares, collect the triangle, you can see it performs fairly well over here, where it's all about avoiding the squares and collecting the triangles as soon as you get into, you know, collecting, or even here the opposite directions, it's pretty bad, right? That's the red thing. And now they add policy two to policy four. So policy two is going to be also to collect the triangles, but to just neglect the squares. And that will also do a bit better. Why does it do better? Because it's better at collecting, because this policy here also needs to avoid. And this policy here doesn't care. So in the regimes where it's better to not care than to avoid, adding this policy, adding these options is going to be good. And you can see that there's a general expansion here as we add more policies. However, I want to point out that, for example, here this black thing, which should be technically superior to the blue thing, because it contains, as you can see here, all the policies that the blue thing contains plus another policy. I don't know if my vision, but I'm pretty sure here the black thing is inside the blue thing. So that means there can also be a disadvantage to adding more policies right here, because maybe you have too much to choose from. And so right here, what we say is we add a policy that is all about collecting the squares. And it is performing, it is actually decreasing the perform. The addition of this is decreasing the performance on tasks where you have to avoid the squares, which I'm not sure if that makes sense. Again, the opposite of collecting isn't avoiding, but I'm just pointing this out. And this isn't really mentioned in the paper. The paper simply says, see, we add policies, therefore we are getting better. I'm not. I don't agree with this, given these results, or maybe the plotting is bad. All right. So they say, okay, more policies better, which I disagree with. They also say, oh, we can, as much as we can regress the W, right, we regress W, we figure out the task, we can even learn the successor features. We can, not the successor features, the pi functions that lead to the successor features. And you can see, if you do it with the true W, you're really good at the beginning. If you do it with a regress W, we can see that before. You can, you, so this is the small version of this plot right here. This is like this section, I think. Yeah. You know, you improve. However, we can also learn this pi function. We can also learn the features. If we're not given the features, maybe we can learn the features. And they say, well, we can do this with, but also by regression. So here, what we can do is we can find the function that minimizes the function and the W along with it that minimizes this error right here. Okay. So you're finding the function and the W that, that matches this error. And this now really is like learning a neural network. I mean, you know, so I get, I get it. You have the I here and the W doesn't depend on the I and so on. But you're getting more and more back to actually simply learning nonlinear functions, mixing them linearly right here. And I think that's going to be kind of the crux of this method. The fact that the more complicated your problems are, the less you are going to be able to do this kind of stuff. And they even go as far as to say, well, what if like before we, the reward is actually something like whether or not you have collected an even number of triangles or squares. Then they say, well, you can simply not have a single W, but you can find a function W. And now the policy is a function of the function of W and you can do potentially the same regression problem. But as you can see, it gets so now you this right here is going to be a function of state. And so you can see that more and more, it simply goes back to basically Q learning again. The only difference here is that you have this intermediate features, but I think you can simply view this, let's say as a hidden layer in a neural network. I get it. Some are held constant across sums and so on. But you know, I like the method in terms of, you know, in terms of the analysis. So if you are given all this stuff, it seems pretty cool that you can derive new policies. It's implication for lifelong learning. They say, look here, you have a bunch of tasks in your database that you've already learned on your agent is going out into the world. It faces a new task. It can use this thing. It can use this thing to obtain a new good policy for that task. It can then use reinforcement learning, or L to refine that policy. And then it can simply save that policy into the database. So it keeps expanding and expanding this thing. So it keeps adding rows and rows and rows right here of new policies that it's learned over the course of its life. So once it's facing a new task, it can just kind of draw from its experience and derive a good initial solution. However, the actual analysis only works, I feel, in quite limited circumstances. And if you want to relax these limited circumstances, then you need to basically regress and regress and regress away from their setup. And I'm not sure. I'm not sure where this is going to go. If this is going to be a general framework for people. It seems like it because it's pretty easy. But then also it seems like most of the world doesn't really fall into this category. In fact, this divide and conquer approach, I'm not sure, but from divide and conquer, I almost imagine something like you subdivide and subdivide and subdivide until you are at some kind of basic task. They still only go for single tasks like this. Here the tasks are somehow in sequence. And I'm not. I think we should really think about hierarchical RL. Now this can be a good first step right here. But most hierarchical RL, even the ones that specify themselves as fully hierarchical, we can do many layers, they rarely go above two layers or three, like one meta layer and one actual layer like this one right here. They rarely go further. Maybe they go two layers, but that's about it. I've seen very little in actual hierarchical or divide and conquer reinforcement learning just because it's so hard to train. All in all, cool paper. And if you want to get into the math a little bit, I think it's pretty easy math. Once you kind of set your goals on what it's actually meant to achieve. If you just read from the beginning, all these reinforcement learning papers, it seems a bit like, why? Why are we doing this? Right? Why do we define this, we define that, we define this? And you're a bit like, yeah, but why? So often it pays in these papers to go at the end to the examples and then come back to the theory, knowing what they want to achieve. All right, that was it for me. Long rant. I'll see you next time. Bye
 Hi there, today we're looking at fast reinforcement learning with generalized policy updates by
 Andre Boreto, Charbo Ho, Diana Borsa, David Silver and Doyna Precu.
 So on high level this paper proposes a framework for reinforcement learning where you have
 many tasks at the same time.
 And they propose framework where they learn many policies at the same time that can or
 cannot correspond to these tasks.
 And then their argument is that if you now have a new task that you haven't seen before,
 you can easily construct a solution to that task from your old policies, basically mixing
 what you learned about your old tasks.
 And it's a pretty general framework and we're going to look at it.
 In my opinion, it's pretty cool for certain settings.
 However, I think it kind of breaks down the more general you go, which I guess is expected
 of such a framework.
 But as you can see, it's kind of math heavy, but we'll get into the examples and what it's
 potentially useful for.
 Alright, so that was it on a high level.
 If you like content like this, don't hesitate to subscribe to the channel and share it out,
 leave a like and tell me in the comments what you think.
 I'm still reading all of them.
 So I will see it.
 Cool, let's dive in.
 So they say the combination of reinforcement learning with deep learning is promising approach
 to tackle important sequential decision making problems that are currently intractable.
 Well, they're taking they're talking about, you know, things like mostly these game playing
 AI is like go and things like this.
 So where this combination of deep learning with reinforcement learning has really shined
 or shun, whatever one obstacle to overcome is the amount of data needed by learning systems
 of this type.
 So again, if you look at these systems like AlphaGo, they need a simulator and they need
 to collect enormous amounts of data, even more so with systems like the Dota AI, the
 OpenAI 5 Dota or StarCraft playing Alpha Star.
 I think it's Alpha Star.
 They need so many simulations in order to learn about the tasks because they always
 start from scratch.
 In this article, they say, we propose to address this issue through a divide and conquer approach.
 We argue that complex decision problems can be naturally decomposed into multiple tasks
 that unfold in sequence or in parallel by associating each track task with a reward
 function.
 This problem decomposition can be seamlessly accommodated within the standard reinforcement
 learning formalism.
 Okay, so what are they saying right here?
 They are basically saying that if you have a task, let's say you want to get what's the
 from here to here.
 And that's very complicated.
 Let's make it complicated.
 Super duper complicated.
 You can basically subdivide that task into multiple subtasks, right?
 So here is like left turn, right turn, go straight, left turn, go straight, right turn
 and so on.
 And each of these subtasks, you can see the two right turns here might share a lot of
 common information.
 There could also be tasks that are at the same time, like you need to go forward and
 jump, can be decomposed into going forward and to jump.
 Now, they're saying is if each of these tasks now has its separate reward function in the
 environment, like for some reason, the environment tells you this, by the way, is task, task
 one, and you're going to get a positive reward if you do a right turn.
 And this down here is task two, the left turn task, and you're going to get a positive reward
 if for that task.
 So the entire task state can be decomposed into a vector.
 So in our case here, we have maybe a vector with three elements.
 Okay, the three elements correspond to turn right, go straight, and turn left.
 And now you're this this right here is your reward vector.
 So we're no longer talking this framework, we're no longer talking about just a reward.
 We're talking about a reward vector.
 Now each of these tasks is going to give you its own individual reward.
 So let's say you're here and you're actually turning right.
 This is going to give you a reward of one for this task, but reward of zero for the
 other task.
 So the environment will somehow tell you which tasks you you get reward for.
 Now there is a notion where you can map this back to a single number.
 And that is the second thing they introduce here.
 So the second thing they introduce here is this thing they call W. So W is going to be
 a mixing vector.
 W is going to be a vector.
 I will call W right here.
 This is the reward vector W is going to be the vector that tells you your final reward.
 So here we're going to do an inner product.
 So we're going to transpose this and multiply by W. And W mixes these rewards and comes
 up with your final reward right here.
 So this this is maybe the reward vector.
 This is the reward number.
 How are we going to call this reward number?
 So in this case, W would have to look something like this.
 Let's say this is an example.
 So the task right here would be to only do right turns.
 Now this is not a really nice example.
 We're going to see some nicer examples later on.
 But you can see that now the environment is specified as a vector of rewards.
 And you can create a specific tasks like turning right simply by adjusting how you mix these
 different things by this vector W. And this is going to be the key ingredient here.
 So they discuss your general reinforcement learning, reinforcement learning lingo.
 And I think we've gone through this a number of times just very, very quickly.
 In reinforcement learning, you're given these transitions, you are in a state, you take
 an action.
 And that leads you to get a reward R prime and you get into a state S prime in the next
 state.
 They say the reward is given by the reward function.
 So the reward is purely a function of where you are and what you do and where you get
 to.
 So for most reinforcement learning problems, you can actually kind of forget about this
 part right here.
 Because well, it is kind of important, but you could most reinforcement learning problems,
 the reward is simply a matter of where you are and what you do.
 And this can be a random variable, there can be randomness.
 But maybe it's easier if you for now think about the reward simply as a function of these
 two things.
 So what you want to discover is a policy pi, where you input, you input where you are,
 and the output is going to be what you should you do in that situation.
 Okay, that is a policy.
 And associated with each policy is this thing called a Q function.
 So you can see right here, the Q function of a policy is going to be a function of where
 you are and what you do.
 And this is a bit confusing, but it basically means that you are in state s.
 So you are here.
 And you have, let's say three options, action one, action two, action three to do.
 Now the Q function tells you the Q function, this is s, and the A's are the numbers.
 Okay, so let's say we plug in the state s and for A, we plug in number two, what it
 will tell you is if I am in state s, and I perform action number two, then how valuable
 is that for me, and value is defined by all the reward that I'm going to pick up from
 now until the end of time, or the end of the episode, it depends.
 But let's say until the end of time.
 So how much reward am I going to pick up from now until the end of time is a bit of a vague,
 not a vague question, but a difficult question.
 I can tell you how much I could estimate how much reward I'm going to pick up in the next
 step because I know what action I'm doing, I'm performing action number two.
 But what happens after that?
 Who knows?
 So that's where this policy right here comes in.
 This policy right here says, so the full definition of the Q function is if I'm in state s, and
 I perform action A right now, and after that, I follow policy pi, what is my reward going
 to be?
 Well, now it's well defined.
 So right now you do action A, and after that, you do whatever action the policy tells you
 in that specific situation.
 So that's the Q function.
 And you can pretty easily see that if you have a Q function, right, if you have an accurate
 Q function, you can get a good policy by simply always going with the action that gives you
 the highest Q value, because it's because of a recurrence relationship called the Bellman
 equation.
 This thing right here.
 So your Q function basically decomposes into the reward in the next step, as we said, plus
 whatever happens after that.
 And whatever happens after that is just by the nature of how the things are defined is
 going to be the Q function of whatever the policy is telling you.
 So you can get a pretty good policy by always doing whatever action your Q function tells
 you is best.
 This step of calculating the Q function is called a policy evaluation.
 And this paper here is going to generalize these notions.
 Sorry, so this is a policy evaluation.
 And then the act of selecting an action is going to be a policy improvement.
 These are just names.
 Okay, but we need to know them because the paper introduces two new things.
 I'm going to where do I highlight policy evaluation?
 I don't know, but here they say this is the policy improvement.
 Okay, here, policy evaluation, policy improvement.
 These are the two steps.
 So the first step is calculate the Q function.
 The second step is to select an action.
 And you can see how these things interlock, namely, we can calculate the Q function of
 a given policy, and we can improve that policy by selecting whatever action is best for the
 Q function.
 This paper generalizes this and you can see that there is a little R right here.
 So the R is just a specific way to reference the reward function used right here.
 Okay, and you can see it here as well.
 Now usually we have one policy and one reward.
 And so what we do is we improve the policy.
 And that leads us to better evaluate the Q function for a given reward function.
 And that leads us to improve the policy.
 Now this paper is going to transform this into the following.
 We have many policies.
 So we have policy one, policy two, and so on until policy P.
 And we also have many reward functions.
 Reward one, reward two, reward three, and so on until reward, let's call that R.
 So we have many different tasks right here.
 And we have many policies.
 Now in essence, they don't need to have some anything to do with each other for the theory
 of this paper.
 But I can simplify this a bit of how they see the world.
 So let's say you have an agent and the agent has been trained on simply that first task
 right here and has been trained using classic Q learning, reinforcement learning, whatnot.
 And that results in this particular policy.
 And then the agent just from scratch, you restart it again, you run reinforcement learning
 just on reward number two, and obtained policy number two, and so on.
 So you do this for all these rewards individually.
 Okay, so you give the agent a new task, and you ask it to learn a policy for that task.
 Now you're in a situation where if you have a new task, so R new, the question is, do
 you again need to train a new policy?
 And the answer for this paper is no.
 Because we have all these policies, we don't need to train a new, we can simply mix and
 match these policies that we already know to obtain a good solution for the new task.
 So how does the paper do it?
 It does it.
 It does it in the following.
 It defines the successor features.
 Maybe it's maybe it's better if we first go to an example.
 So the example they give here is the following.
 Otherwise, this I guess this might sound just a bit too abstract.
 Okay, so you have this world here, the agent is the thing here in yellow, and it can just
 move so its actions are move left, up, right down this this is one step.
 Okay, in the environment, there are two different objects.
 One object is a triangle and one object is a square.
 So there are a number of tasks we can define right now in this thing.
 So we define tasks according to a reward function.
 So the reward, let's say the reward one is going to be one, if, if it picks up a square,
 sorry, the square, and zero else.
 Just if it picks up a square on any given step, we give it a reward of one.
 We don't care about the blue triangles.
 Okay.
 And then reward two is going to be the opposite.
 It's going to be one, not the opposite, but one if it picks up a triangle, and zero else.
 So you can see the good policies right here.
 So pi one is a good policy for reward one, because it just goes and collects these red
 things, doesn't care about the blue things, just goes and collects them.
 Pi two, it goes and collects the blue things, doesn't care about the red things.
 Okay.
 So let's imagine that you have run reinforcement learning twice, once for reward one, and once
 for reward two.
 And now you have two policies.
 Okay.
 So you have two policies, this will lead to pi one, this will lead to pi two.
 And now I give you the third task.
 Now the third task is a bit special.
 It's one, if you pick up a square, and it's, it's zero else, except it's negative one,
 if you pick up a blue thing.
 So the order of these is kind of wrong, but it just for visual representation.
 Okay, so now you're asked to pick up the red things, but avoid the blue things.
 Okay, pick up as many red things as you can avoid the blue things.
 And again, as we said, the question is, do you now have to run reinforcement learning
 again in this agent with your simulator using like Q learning or something like this, from
 the start, or can you come up with a solution just given these two policies that will perform
 well on the, on this new task?
 Okay.
 And we're going to see how they do it.
 So what they do is they use successor features.
 So these successor features, I've done a video about successor features, and I'll link to
 that.
 You can look at that.
 But essentially, essentially, the successor features are defined like this.
 And for that, we need to know what this thing is right here.
 They simply call this a feature function.
 Okay, it's very, it's very ambiguous term.
 A feature function is a function that takes in a transition.
 So state action, next state, and maps it to a high dimensional vector.
 Note, this is almost the same as a reward function, except the reward function simply
 maps it to a number.
 Now this is mapped to a higher dimensional thing.
 Again, I want to, I kind of want to leave out the next state right here just to make
 things easier on you.
 So a feature here can be many, many things, but the structure of the features is going
 to be such that the reward function is going to be this feature times this w vector.
 So it was a bit, a bit not correct before when I said the reward is now a vector, the
 reward of a particular task w can be seen as the inner product between the features
 and the task vector.
 So w specifies the task and the features, well, they specify the features in our case,
 it can be, it can be fairly simple, namely, yes, I was, I was definitely wrong at the
 beginning.
 So the feature functions right here is which object do you pick up?
 Okay, so we define the feature function as one zero, if you pick up a square and we define
 the feature function as zero one, if you pick up a triangle.
 And now you can, and we define it as, we define it as zero zero, if you pick up nothing.
 And now you can fairly easily see that the reward of each task can be simply calculated
 by mixing the features accordingly.
 Okay, so reward one is going to be simply the feature times a one zero, which is the
 w vector.
 So I can specify a task by giving the appropriate w vector.
 And now you can see that if this is my reward function, my agent can go out into the world
 if it collects a square, it is going to be rewarded right here.
 If it collects a triangle, even though the features indicate that it collected a triangle,
 it doesn't care about it because the w is zero right here.
 If I now want to give it the new task, the same is true for r2, if I now want to give
 it a new task r3, right?
 And you remember the reward function right there, I can achieve that reward function
 by simply multiplying the same features, the exact same feature functions by this vector
 right here.
 Okay.
 Remember there is a slight difference between the reward function and the feature function
 in this particular example.
 The idea of the paper is that the feature function can be rich in in expressivity and
 you know, tell you all sorts of things about your current state and the reward function
 is just a number, right?
 And then the reward is specified by simply linearly mixing these features.
 So the structure imposed by the paper here is that there are such a thing as a feature,
 and any task can be described by mixing these same features.
 That's the issue right here.
 So the features are going to be constant across tasks.
 Whereas the w defines the task.
 Alright, so the the goal here is that if you have learned many, many things during your
 tasks, what you want to do is you want to learn this feature representation that is
 the same across all tasks.
 And then you want to simply have the w specify how to mix these features to get the reward.
 Now, of course, this is a very strict, very, very definition, not not a lot of things will
 fall into this unless you make the features like exponentially big, of course.
 However, they do discuss whenever a task doesn't fall into that.
 So I hope you're with me so far.
 This is the first kind of restriction we impose on our worlds that we can tackle with this
 framework, namely that all of our worlds have all of our tasks in this world have to be
 a linear mix of the same features.
 If that's given, then our then we can derive policies for tasks that we have never seen.
 We can derive good policies by doing zero learning, simply by specifying the task, we
 can have a good policy for that task from the policies we've already learned for the
 other tasks.
 Okay, so the reward three is now simply this.
 And yeah, notice it's not the same as the reward function, because the reward function
 had one if you pick up the square negative one, if you pick up the triangle and zero
 else.
 So the zero, we don't have to specify here because it's not part of our features.
 Right, so you can see that the reward function is given simply by that.
 And we can now, as I said, derive a good policy for this reward by looking at the other policies,
 even though none of these policies has ever learned to avoid anything.
 So it makes it defines these successor features right here.
 So the successor features is much like the Q function, you can see the signature is almost
 the same.
 So as a Q function tells you how much reward you're going to get if you do the action a
 and then follow policy pi, the successor features almost the same thing.
 However, it doesn't tell you what rewards you're going to get.
 It tells you which features you're going to get and which features by that we mean the
 sum of future features.
 Now you can see this sum, this a little bit this it, of course, it comes from the fact
 of the linearity up here.
 So it's not really an additional restriction, but simply to clarify what this means for
 your environment, your environment has to be able to be looked at in terms of these
 features and these features, they need to be cumulative.
 Again, that comes from the fact that it's linear, but to see.
 So a feature like I want an even number of steps or something like this would be terrible
 because and they're going into things like this later, but it would be terrible because
 here we have the sum.
 And as soon as you if you have a feature that is very high, if you have an even number of
 steps then or if you have a feature that counts the steps, you will never be able to to do
 well because if you have a feature that counts the steps, it simply counts up and up and
 up and up, depending on how many steps you do.
 And your reward can never be specified in terms of a mix of these features.
 And therefore your successor features are going to be useless.
 But in our case, where it's where feature one is pick up is how many of the sorry, I
 have to rephrase our feature one is whether or not you pick up a square.
 Therefore if we sum it up, our successor feature one is going to be the number of this is this
 is a pound sign, the number of squares that you pick up.
 Okay.
 Similarly, our feature two is whether or not you pick up a triangle in a particular step.
 So our successor feature number two is going to be the number of triangles that you pick
 up over time.
 I can see that the successor features is kind of the analogous of your Q function, but it
 is not in terms of a single number, the reward.
 It is going to be in terms of these features, which is an entire vector.
 And because we've constructed this in a linear way, you can also pretty clearly see that
 the Q function is inherently related to the successor features.
 You can obtain the Q function by simply multiplying the successor features by your task vector
 W.
 Now, a lot of you might be wondering where does this W come from?
 And in our initial case, we're just going to frame everything as being given, right?
 So we're given this W, we're defining everything from our godlike perspective for now.
 So don't think all of this is learned by now.
 Yeah.
 All right, so how can you now derive this magical new policy?
 So let's say we have this policy one and we have the policy two, and you have these features
 that you've learned constantly over both tasks.
 In fact, it's given, right?
 This pi function, we give it, we impose it that the feature one is whether you pick up
 a red square feature two is whether you pick up a blue square.
 Then we know that the reward functions can be achieved by doing the W. So this here,
 your W is going to be one zero, and your W here is going to be zero one.
 And now we want a good policy for task three.
 And we know we can achieve this by the one negative one W. How can we derive a good policy?
 And this is this algorithm, this general policy evaluation, a general policy improvement.
 So it assumes that you, as we said, you have many, many different, many different policy.
 So here you can see policy one, where's policy two, here's policy two, and so on.
 It assumes that you have many different features and therefore many different successor features.
 In fact, you have a vector of them, right?
 So here you can see feature one, feature two, and so on.
 And it also assumes that you're in a current state and you have many actions at your disposal
 right now.
 Action one, action two, and so on.
 So this is all the past.
 You've already defined your features, you have learned these policies, and now you're
 given a new W, W new.
 In our case, it's this one negative one.
 We want the best action.
 So we are in state S, and we are given this W. We want the best action.
 Now here is a method where we can simply calculate the best action in terms, by not reinforcement
 learning at all in this new task.
 So by structuring things like this here.
 So what does it really say here?
 This thing says we are going to evaluate all of these different cells of this tensor right
 here.
 So we're going to determine what is the successor feature number two for policy pi one in state
 S if I right now do a two.
 This is very abstract.
 So let's say you're here and action two is actually going to the right.
 So you're here.
 Oh, this was yellow.
 It doesn't matter.
 So this is action one.
 This is action two.
 So action two is you go to the right.
 You can see that this will let you pick up a triangle.
 Now here that's action three and so on.
 Okay.
 So what's this number going to be?
 So we are in state S as we said, and we do action two.
 So action two is going to pick up a triangle, the triangle, the picking up of a triangle
 means that our pi for this step, or sorry, our five for the step is going to be 01.
 Okay.
 So our successor features, this is not the features itself.
 This is the successor features, the successor features decompose into the next step plus
 all the next steps that we can follow.
 Okay, so all the steps that will come.
 So what are these features going to be is it's going to be the sum over that plus everything
 that follows.
 And I can take a little bit of a guess here, which means that this number, so we only care
 about feature two right here, this feature, feature two, this number is going to be one
 for the next step, because we are going to pick up a triangle if we do action two.
 But then after that, we're going to follow policy one.
 And policy one has been trained to pick up the red squares and not care about triangles.
 So I'm going to guess that every now and then it will kind of step over a triangle, but
 it won't fall, it won't, you know, explicitly go look for them.
 So let's say the episode was 10 more steps, but the board has like 100 squares.
 So and it has like three triangles on it.
 So let's say that's like three tenths in expectation.
 Okay, so this is going to be this is going to be the number that we're looking for.
 We're doing this for every single one of these cells.
 Okay, this this thing is going to do for every single one of these cells.
 And this is very similar to evaluating Q functions, except we're evaluating an entire vector right
 here.
 That's the difference to simply learning many Q functions.
 So if you were to evaluate only a Q function, then you would only have this first matrix,
 this first block right here.
 Okay, but you have feature one, feature two, and so on.
 So you calculate everything in terms of these features.
 And then by linearity, you can mix it with that vector.
 So in our case, this is going to be the one negative one, which will give you the Q functions,
 right?
 From what we've seen before, you obtain a Q function by simply mixing your successor
 features with your with this task vector.
 And if you have a Q function, you can pretty easily determine which action you should take.
 Now you have here a Q function with respect to every single policy, but you can simply
 take the max, right?
 So the max across all of this will determine will determine so you take the max across
 all the policies, which will give you the Q function for a particular action over all
 policies that you consider, and then you can simply take the argmax of that and determine
 the action you should take.
 Okay, so it's a pretty big evaluation.
 But if you do this, that means you don't have to do reinforcement learning on this task.
 It simply determines which action right now is the best given everything that I know from
 these old policies about the task.
 And that's not going to be like the optimal policy, per se, but it's going to be one policy
 that's pretty, pretty good.
 And you can actually prove some things across that.
 So they do this right here.
 And you can see that here is what Q learning does on this new task of picking up the squares
 and avoiding the triangles Q learning takes a while to get there.
 However, if you do what they are suggesting, and you know, you give the W, you can supply
 the W almost from the beginning, you see right here almost from the beginning, it is at a
 high reward.
 Now Q learning surpasses it eventually.
 But it's pretty impressive that without doing any learning, you are immediately good.
 Right.
 Now the caveat here, of course, is that they already need these policy pi one and pi two
 given to the algorithm.
 And that comes from previous reinforcement learning trials.
 And they say that they give these trials as many steps as Q learning uses.
 So they give them this these amounts of steps on these other tasks.
 So the comparison here is a bit shaky, if you ask me.
 But the point made is that if you have a new task right now, you can obtain very good solutions.
 And you don't have to do anything.
 Okay.
 And these solutions can be the basis for new reinforcement learning, right?
 You could start Q learning off right here and then get here much faster potentially
 and so on.
 So the next objective right here is that now we have defined the tasks and we had we know
 what these features are.
 And we know how to mix these features as imposers of the task.
 So what happens if we only have the reward function, we specify the task only in terms
 of the reward function, but we're kind of looking at the features and we're like, agent,
 please figure out yourself how to apply these features in order to make the reward high.
 And that's what this thing is right here.
 This GP and GPI with regress W. So you don't no longer tell it what the W is.
 It needs to infer it through reinforcement learning, right?
 And it's not really reinforcement learning.
 But what it does, where is it?
 Yeah, it's simply because all of this is linear and this thing here is given.
 So always remember this thing here is given.
 And these are the rewards that you obtain.
 You can simply do a regression to figure out the W of the task.
 Now that's going to take some time.
 But as you can see right here, it is going to take a lot less time than than doing Q
 learning from scratch.
 Absolutely because you have good features.
 So this is some this is this gets closer and closer to transfer learning, right?
 If you imagine that this right here is your pre trained neural network, and you simply
 learn the last layer of it.
 You freeze this you do transfer learning fine tune the last layer here we are.
 So it gets closer and closer and you'll see this trend right here.
 So it's pretty cool what you can do.
 But basically, I think it's a lot of math around a framework.
 And the more and more you relax the kind of impositions that they need for their framework,
 the more it gets back to simply, well, we do reinforcement learning, at least in my
 estimation.
 So before we look at that, this here is a pretty, pretty cool experiment, where they
 they look at how the how the different tasks can be achieved, if you give different policies.
 So you'll have noticed that we have always given these two, two tasks 10 and 01.
 These were our tasks that we trained on. And then one negative one is task we evaluated
 on.
 Okay.
 And you might object and say, wait a minute, these these two tasks, you know, they're pretty
 good as let's say, pre training tasks, because and it's basically the standard basis, right?
 And any other tasks can be mixed from those.
 So these are orthogonal vectors in this vector space.
 So you're being pretty generous to the system.
 What happens if we're not as generous? So that's what they do here.
 So they have different policies, and they evaluate how much you can learn with these
 different policies.
 So the way you have to read this diagram is right here, it's going to be the one zero
 axis as they will they label it right here.
 And this is going to be the 01 axis. And this is evaluation.
 So every direction on this circle defines a task, for example, this task right here,
 as you can see, is going to define the task of picking up both the squares and the triangles,
 right?
 Whatever you pick up, you get a reward.
 However, the task down here is going to be please pick up the squares, but avoid the
 triangles at all cost.
 Okay.
 And now they're going to look what happens if we supply different policies to choose
 from, remember, we're in this situation, we're getting in this situation where we give everything,
 and we give initial policies, we give the task vector.
 And now it's about deriving a good policy just from looking at the old policy.
 So no learning.
 As a baseline, you have Q learning, which into a given direction, tells you basically
 how long Q learning takes or how far Q learning gets with a given amount of steps indicated
 by this one, two, three, four, and so on.
 Yeah, you see, I think this is this in how far Q learning gets with these amounts of
 steps is the dotted lines right here.
 So Q learning gets this far with 10 to the, I don't know, four, and then this far, 10
 to the five and so on.
 So these are comparisons.
 You can see that on the outside, Q learning is going to beat this, these methods.
 But our hope is going to be that of course, if we have this zero shot generalization,
 it's much better than running Q learning for really long if we get close to it.
 So the green thing is what we've already seen.
 Policies one and two will give you a fairly good extent right here.
 So what does it mean?
 It means it can solve pretty much everything from here, here, this task, this task, this
 task.
 It kind of falls off once we go down here.
 So once we go to the avoid section, it sort of falls off because it has never learned
 to avoid.
 Now, still, we can, of course, do the avoidance by simply imposing a negative collection.
 But negative collecting and avoiding aren't exactly the same thing in these environments,
 right?
 Because avoiding can also be going really close to something but not hitting it while
 collecting.
 It's not the inverse of collecting.
 The inverse of collecting would be like run away as far as far as possible.
 So we can expect that we've only ever learned to collect, we're not going to be super good
 at avoiding.
 Then the other extreme is when we give policies three and four.
 I haven't told you but you can see it right here.
 Policy three is explicitly to collect one and avoid the other, while policy four is
 the opposite right here.
 Avoid the squares, collect the triangles.
 And now this policy, this policy is, should be pretty good on all of the tasks in between.
 As you can see, it has the biggest extent right here.
 And that also makes sense.
 By the way, there's nothing down here because the task of avoiding both things doesn't really
 make sense because you can just stay where you are because there are also these squares
 where there's nothing.
 But you can see that the mixture of those is quite potent.
 So already we can see even though these span a basis, in fact an orthogonal basis as much
 as these, because of the nature of the features that we define for the task, they are not
 equivalent in mixing after.
 So we can be more generous, we can also be less generous if we only provide policy five.
 And policy five is simply to pick up, to pick up both objects.
 Then we're going to have a pretty hard time when it comes to avoiding things.
 So you can see it can do fairly well picking up the various things in a positive manner.
 But as soon as we cross this line into the like this horizontal line into where it's
 about avoiding a particular object, it's not it's not the choices of actions we have from
 policy five aren't going to be super good at that.
 And they do another they do another thing right here.
 So that the left thing is where they say, it's important which policies we provide.
 And the right thing, they want to say something like, it's important.
 So they want to say, if we provide more policies, that can be advantageous, because we basically
 have more options to choose from.
 So now they start off with policy four, and policy four is simply avoid the squares, collect
 the triangle, you can see it performs fairly well over here, where it's all about avoiding
 the squares and collecting the triangles as soon as you get into, you know, collecting,
 or even here the opposite directions, it's pretty bad, right?
 That's the red thing.
 And now they add policy two to policy four.
 So policy two is going to be also to collect the triangles, but to just neglect the squares.
 And that will also do a bit better.
 Why does it do better?
 Because it's better at collecting, because this policy here also needs to avoid.
 And this policy here doesn't care.
 So in the regimes where it's better to not care than to avoid, adding this policy, adding
 these options is going to be good.
 And you can see that there's a general expansion here as we add more policies.
 However, I want to point out that, for example, here this black thing, which should be technically
 superior to the blue thing, because it contains, as you can see here, all the policies that
 the blue thing contains plus another policy.
 I don't know if my vision, but I'm pretty sure here the black thing is inside the blue
 thing.
 So that means there can also be a disadvantage to adding more policies right here, because
 maybe you have too much to choose from.
 And so right here, what we say is we add a policy that is all about collecting the squares.
 And it is performing, it is actually decreasing the perform.
 The addition of this is decreasing the performance on tasks where you have to avoid the squares,
 which I'm not sure if that makes sense.
 Again, the opposite of collecting isn't avoiding, but I'm just pointing this out.
 And this isn't really mentioned in the paper.
 The paper simply says, see, we add policies, therefore we are getting better.
 I'm not.
 I don't agree with this, given these results, or maybe the plotting is bad.
 All right.
 So they say, okay, more policies better, which I disagree with.
 They also say, oh, we can, as much as we can regress the W, right, we regress W, we figure
 out the task, we can even learn the successor features.
 We can, not the successor features, the pi functions that lead to the successor features.
 And you can see, if you do it with the true W, you're really good at the beginning.
 If you do it with a regress W, we can see that before.
 You can, you, so this is the small version of this plot right here.
 This is like this section, I think.
 Yeah.
 You know, you improve.
 However, we can also learn this pi function.
 We can also learn the features.
 If we're not given the features, maybe we can learn the features.
 And they say, well, we can do this with, but also by regression.
 So here, what we can do is we can find the function that minimizes the function and the
 W along with it that minimizes this error right here.
 Okay.
 So you're finding the function and the W that, that matches this error.
 And this now really is like learning a neural network.
 I mean, you know, so I get, I get it.
 You have the I here and the W doesn't depend on the I and so on.
 But you're getting more and more back to actually simply learning nonlinear functions, mixing
 them linearly right here.
 And I think that's going to be kind of the crux of this method.
 The fact that the more complicated your problems are, the less you are going to be able to
 do this kind of stuff.
 And they even go as far as to say, well, what if like before we, the reward is actually
 something like whether or not you have collected an even number of triangles or squares.
 Then they say, well, you can simply not have a single W, but you can find a function W.
 And now the policy is a function of the function of W and you can do potentially the same regression
 problem.
 But as you can see, it gets so now you this right here is going to be a function of state.
 And so you can see that more and more, it simply goes back to basically Q learning again.
 The only difference here is that you have this intermediate features, but I think you
 can simply view this, let's say as a hidden layer in a neural network.
 I get it.
 Some are held constant across sums and so on.
 But you know, I like the method in terms of, you know, in terms of the analysis.
 So if you are given all this stuff, it seems pretty cool that you can derive new policies.
 It's implication for lifelong learning.
 They say, look here, you have a bunch of tasks in your database that you've already learned
 on your agent is going out into the world.
 It faces a new task.
 It can use this thing.
 It can use this thing to obtain a new good policy for that task.
 It can then use reinforcement learning, or L to refine that policy.
 And then it can simply save that policy into the database.
 So it keeps expanding and expanding this thing.
 So it keeps adding rows and rows and rows right here of new policies that it's learned
 over the course of its life.
 So once it's facing a new task, it can just kind of draw from its experience and derive
 a good initial solution.
 However, the actual analysis only works, I feel, in quite limited circumstances.
 And if you want to relax these limited circumstances, then you need to basically regress and regress
 and regress away from their setup.
 And I'm not sure.
 I'm not sure where this is going to go.
 If this is going to be a general framework for people.
 It seems like it because it's pretty easy.
 But then also it seems like most of the world doesn't really fall into this category.
 In fact, this divide and conquer approach, I'm not sure, but from divide and conquer,
 I almost imagine something like you subdivide and subdivide and subdivide until you are
 at some kind of basic task.
 They still only go for single tasks like this.
 Here the tasks are somehow in sequence.
 And I'm not.
 I think we should really think about hierarchical RL.
 Now this can be a good first step right here.
 But most hierarchical RL, even the ones that specify themselves as fully hierarchical,
 we can do many layers, they rarely go above two layers or three, like one meta layer and
 one actual layer like this one right here.
 They rarely go further.
 Maybe they go two layers, but that's about it.
 I've seen very little in actual hierarchical or divide and conquer reinforcement learning
 just because it's so hard to train.
 All in all, cool paper.
 And if you want to get into the math a little bit, I think it's pretty easy math.
 Once you kind of set your goals on what it's actually meant to achieve.
 If you just read from the beginning, all these reinforcement learning papers, it seems a
 bit like, why?
 Why are we doing this?
 Right?
 Why do we define this, we define that, we define this?
 And you're a bit like, yeah, but why?
 So often it pays in these papers to go at the end to the examples and then come back
 to the theory, knowing what they want to achieve.
 All right, that was it for me.
 Long rant.
 I'll see you next time.
 Bye