Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
rl reinforcement learning ai artificial intelligence udrl schmidhuber policy value reward
Schmidhuber thinking outside the box! Upside-Down RL turns RL on its head and constructs a behavior function that uses the desired reward as an input. The new paradigm shows surprising performance compared to classic RL algorithms.

Abstract:
We transform reinforcement learning (RL) into a form of supervised learning (SL) by turning traditional RL on its head, calling this Upside Down RL (UDRL). Standard RL predicts rewards, while UDRL instead uses rewards as task-defining inputs, together with representations of time horizons and other computable functions of historic and desired future data. UDRL learns to interpret these input observations as commands, mapping them to actions (or action probabilities) through SL on past (possibly accidental) experience. UDRL generalizes to achieve high rewards or other goals, through input commands such as: get lots of reward within at most so much time! A separate paper [61] on first experiments with UDRL shows that even a pilot version of UDRL can outperform traditional baseline algorithms on certain challenging RL problems. We also introduce a related simple but general approach for teaching a robot to imitate humans. First videotape humans imitating the robot's current behaviors, then let the robot learn through SL to map the videos (as input commands) to these behaviors, then let it generalize and imitate videos of humans executing previously unknown behavior. This Imitate-Imitator concept may actually explain why biological evolution has resulted in parents who imitate the babbling of their babies.

Author: Juergen Schmidhuber

https://arxiv.org/abs/1912.02875
https://arxiv.org/abs/1912.02877
 He did it! Crazy son of a bitch did it again! What am I talking about? Jürgen Schmidhuber reinforcement learning upside down! New paper just dropped on the verge of the NeurIPS conference being presented at a workshop here. Presenting upside down reinforcement learning. I am pumped for this one, can you tell? It says we transform reinforcement learning into a form of supervised learning by turning traditional RL on its head. Calling this RL-lar. What do we call this? We'll just call it lar. Upside down reinforcement learning. And so this is upside down. Never mind. Okay, let's just check out how it works. So I'm going to give a brief overview before we go into this paper. Alright, so let's say you have a reinforcement learning problem. Let's say an Atari game for example. And in an Atari game you usually have a screen, right? And let's just say you're playing this marine commander. So there's water here, right? And there might be a bunch of... Here's your boat, right? There's a boat, a little boat. There might be a bunch of opponents right here. Fishy fish opponents, fishy fish opponents and so on. And there are a bunch of gold coins like here. That's a big gold coin, right? And you're kind of supposed to, I think you're supposed to like go get air. You have some air meter over here. Whatever. So there's this Atari game, right? You're supposed to get the reward which is this maybe this coin here. And stay alive as long as possible and so on. So this is a classic reinforcement learning problem. And there are various techniques for this. We've looked at a couple of them. And what upside down reinforcement learning does is basically what you do is you want to transform this input to a new representation. Which basically, well, if I can, maybe I can... Let me get this correctly. So then there's this over here and then there's a little fishy, a little fishy here. And there's a coin right here. So what you want to do is basically turn this input on its head like upside down. And so this way is kind of up or down or whatever in this new representation. And if you actually learn on this new representation with pretty the same techniques, it works much better than the classic RL setting. And this is not only for like these Atari games. Like this appears to hold throughout the RL space. So in robotics, like if you have a robot or whatever, this is a robot. It has a square head, as you can tell. You know, it's supposed to like open a door. You've seen this DARPA challenge. This doesn't work, right? But if you just transform this and actually turn the robot upside down, the robot will be able to open the door just fine. And even like if you have a chessboard and there's like a bunch of pieces on it. The problem in this case is you have to simulate this chessboard. And if you turn this around now, basically all the pieces will fall off. So what you need to do is you need to have a simulator that encodes a magnetic chessboard such that the pieces don't fall off. So it's a bit of programming effort. But if you do that... All right, I'm kidding. This is a new paradigm for RL, but it's unfortunately not as good. Someone should try the magnetic chessboard simulator. Upside down RL is a new paradigm for RL where basically the kind of notion of inputs and outputs of the RL algorithm are switched around a bit. So basic ideas here is that you have an RL algorithm that is also fed with a bunch of commands. So in classic RL what you'll have... Let's actually go back to this Atari game here, right? In classic RL, an RL algorithm will get the Atari game as a screen as an input and is asked from this to predict a bunch of outputs. So in classic Atari, these are eight actions. I'm going to draw three here, like go to the left, go to the right, or press the button for shoot, right? These are the actions you have and the algorithm is tasked. And there are different versions of this. In policy methods, policy gradient methods, typically the algorithm is tasked with outputting a distribution over these actions. In other methods like value learning, Q learning, the algorithm is tasked with assigning each of these actions a value. So in this situation, going to the left will be worth three in the future. Going to the right will be worth negative one and shooting will be worth zero. So you might want to go with this action here. Now in upside-down reinforcement learning, we've had observation going into the model and the model coming up with the value estimation of the different actions. In upside-down reinforcement learning, you'll have the observation and something else going into the model and the model coming up with an action. And this something else is the key. What you input here is your desire, your future desire. And in this paper, they call it a command. So you'll have a command as an input together with the observations. You basically say, here's my state and I would like to achieve, let's say five reward in the next five reward in the next two time steps, right? Make this happen. Right. This is this is your command going into the model and the model will then try to find actions such that in the next two time steps, you'll get five reward. You can easily see a model that learns this will actually be able to, you know, do various things, including doing the classic RL things like get as much reward as possible in given or in the shortest amount of time, but can also do much more. And in the general sense, the difference is how this is trained now. This model, when you train it, as you can see, you don't it's not trained with in my having in mind kind of only to get the maximum reward. It is trained to be much more a general kind of understanding of the world. I mean, learning what do I need to do to achieve a variety of goals? Specifically, what you want to do to train this is the following. Say you have a method of of moving in the world and collecting traces, right? So you go from state, state one, state two, state three. You go with like your action one, action two. Let's draw action three. The state four. And in each of these, you get a you get rewards, right? Reward one reward to reward three. Now, this in classic RL, this will kind of give you one training example, right? So this is if you consider this to be an episode, this will give you one training example to to run this sequence of actions. Upside down RL, you can actually consider this as many, many training examples. And here's what I mean. So if you, for example, start at state one, you can say, aha, within one time step, one one time step, I go to state two and I have achieved our one rewards by doing action a one. Right. So this now can be an input to your model. Your model could learn if you get as an observation, remember the previous thing as an observation, you get s one as a command. You get I want to achieve in one time step. Are one reward. Right. And you train this goes into the model and the model is trained to say a one given if I am in s one and I do a one, I will achieve that. Right. So you train the model to give a one as an output. And this is valid because in the past you've observed going from s one using a one to a state where you get this this kind of reward in this kind of time. But you can also so you can do all of these single steps. They will all provide individual training examples to your model. Right. But then also you can consider a two step thing. So you can say I'm in state s one and I go I go in two time steps. I have achieved our one plus our two reward by doing actions a one then a two. Right. And a two I'm going to do in parents here because what you want to do is you want to always always consider the action that comes right after where you are now. So again your training sample let me draw this up here. Maybe your training sample would be the following. I am in state s one. This would be my observation. My command would be I would like to achieve in two time steps reward r one plus r two reward. Right. This reward this both goes into the model. Right. You tell the model please given the state s one achieve this reward in this time. And the model is supposed to output a one saying ha in the past I was in this state and I did achieve this goal by using that. So the model is supposed to learn to achieve different goals. Right. So now you can not only train from good episodes right. You can train for any episode any episode usually in classic or you kind of want to focus on the good episodes because you want to maximize your reward. But here you can tell the model hey if you've done something particularly stupid let's say here in s three you done something the a three was particularly stupid gave you. So r three here was really bad reward like a negative five billion trillion. And you can actually train the model to recognize this can be a hey look if you are in s three and within one time step you want to achieve negative five billion billion billion trillion. Reward you all you have to do is action a three right. And then the cool thing now is if you are at evaluation time you actually want the big reward what you'll do is you simply plug in a different command simply in one time step still I'm in state s three in one time step. I want to achieve actually three reward not negative a lot right. And the model will have learned that a three will lead to a situation where you get a lot of negative reward. So the model will be like I'm for sure not going to do a three right. I'm going to do something else here because I have learned to map a three to like this really low reward. So in essence this has connections to things like hindsight experience replay and kind of universal value function where you kind of learn to go from any state to any other state in this. But none of these do none of these have this kind of command what Schmidhuber calls command here as an input to the model. And I think actually this is this is really positive to input this because usually in universal value functions what you would say is let's consider a simple grid world right. Whatever your agent is here and you need to you need to reach a goal that's down here. But you might not be able to learn it because it's super sparse reward and so on. But what you can do is you can learn to reach this position and this position and this position from various positions like go here go from here to here. You can learn to go from here to here. And you know in essence you would like it eventually to generalize to all the fields. So you basically learn to go from any position to any other position with your agent with these universal value or universal policy functions having sub goals. But they during that phase where they learn to go from anything to anything they don't they don't necessarily include this reward thing as a as an input. It's more like kind of either a sub goal or like the usual value function will simply approximate the reward. Whereas whereas in this technique we actually have a policy learning we actually output a an action value. Also hindsight experience replay what hindsight experience replay would do in the same situation right. You're here. We might do a videos on this in the future. You're here and you try right. And your agent actually it ends up here right ends up right here. What you can do is you can simply say oh well actually this this was my goal all along and then simply train train your model as if as if this thing here was your goal all along. And not this thing here and treat it as kind of a positive reward for this. At least that's how I understand it. Right. And both of these things are quite different than here where we have this command as input and I do I do like it. So I think this this is very much the basic things here. This it is extra extrapolated to kind of noisy inputs and noisy environments and so on. But this is the basic the basic gist of it. So here you see your you what you will learn is to map all and all is your representation of your input. So the screen for example or the chessboard. And I think also kind of the last action and there were you get in this step plus your horizon and desire. So in how much time you would like to achieve how much reward and then you can also get input some extra goals that you have. And so you can see basically any any episode that you've run in the past will give you a valid training example for this. Your model will simply learn to match the previous experience with the goals that were achieved in the previous experience. So there is lots of lots of generalizations here like how exactly these things are represented. This this time horizon can be a high dimensional object. The desire can be as I understand it somewhat a dimensional object. The extra commands can be like conditionals on these two things. It gets very complicated, but I want to jump ahead to a different paper where so this paper is basically just describing the algorithm. And then the next paper is doing experiments with this. Let's scroll past here. All right. So this paper training agents using up that down reinforcement learning released on the same day, but different authors that have used also made who was also here but have used this to implement a variant of this. And here you see again what I was trying to to explain. So in traditional RL, this especially here Q learning, you'll have this function which gets an observation as input and then Q learning especially. So you also get the action as an input and you're supposed to say for the given observation this particular action has this expected value as a return. Right. That's what I explained at the beginning. That's kind of value based reinforcement learning. Whereas the behavior function here, which would be upside down reinforcement learning gets the observation and a command and will map that to an action. And here again is what we've gone over. This is a bit of a different thing. So this agent has apparently run two different episodes. One point it did this sequence of actions and at the other point from the same starting state it did this sequence of action and you can see here on the right all the training samples we can we can derive from this. So we can say from state s 0 right. If I want to return in one time step, I have experienced this in the past right to return in one time step. All I have to do is take action a one. But if I want one return in one time step, I have to take action a two and you teach your behavior function to learn these things to learn to output these actions with these things here as inputs. And then what you hope of course is that this will generalize that it will learn to generalize that you can say now give me more reward than I have ever seen before right. And it will kind of learn which things correspond to lower reward, which things correspond to higher award and will be able to extrapolate which things will correspond to even higher report reward. Sorry. So they have two algorithms and this is kind of this is reminiscent of the old of the old RL kind of world where you do kind of one algorithm is continuously learning from the experience gathered by another algorithm. So you have one set of algorithms and this even in modern RL this this this is how it's done right. You have two different boxes right. Actually you have probably one box learning the model like this is I'm going to represent this here learner right. And the learner distributes the model to many many machines interacting with the simulators and these machines all they do is run episodes with the learned model and they will send back their experience here. And then the learner can learn from it and then at the end send it again. So so. All right here we go. So in each step what we do in order to to generate a new episode we don't always want to want to kind of execute one given policy. What we do is we sample from the end of the replay buffer and the replay buffer is sorted by returns right. So the highest return episodes are on top. So we want to sample the highest return episodes then we want to say maybe some of them are 10 steps long maybe some of them are five steps long and so on. So we set the horizon to be the mean of the length of these right and we set the desired return how much return should be achieved in this time to be the unit to sample from the uniform distribution between M and M plus S and M is the mean and S is the standard deviation of the selected episode. So so what this means is is like here is a bunch of episodes from the start at the same time. Here's a bunch of episodes that I ran right from here is time zero and then time goes on that I ran that had really high returns right. Now I'm going to take the mean time that these episodes ran like this. This is maybe five time steps. So in five time I want to achieve now how much reward now you look at all the rewards that were achieved. This is maybe a distribution that has some mean here like so and then you say I want to achieve a reward between here and one standard deviation higher than here. So right and this this would be the reward you want to achieve. So what you do is you kind of push your learned model to just go a bit beyond what it has seen so far is basically say look I you can do this but you can just do a bit more in the same amount of time. Please do this and you hope the model has learned to kind of generalize to do this. And if so you will execute these episodes and then these episodes will go back to the learner right. I'll go back to the learner here and the learner will learn from them and hopefully then you can like generalize even more and then you can say I now know how to achieve this bit more reward. Now I can if I run the episode I will achieve even more reward. I can push the model even further right. So at eval time you can always ask the model to produce as much reward as possible in the given time. And of course every episode sent back here is not only one training example as we saw but many many training examples can be derived from these models even beyond what's in what's in this paper. All right. So I think this was a good first shot at describing this algorithm. I hope you get the gist of it. I enjoy this a bit of a criticism for me would be it's still kind of doesn't it. So it doesn't touch the exploration dilemma. So it again deals with kind of incremental incrementally getting better whereas I feel this can easily get stuck in some minimum where it's not possible to do this incremental generalization of the model where you really need a new approach. And that's why games like Montezuma's Revenge are solved using algorithms like Go Explore and not any of the classic algorithms. That being said they have experiments where they show that especially in sparse reward environments they do better than classic or algorithms. So if you for example here take the lunar lander where A to C beats upside down RL and I guess you didn't get Matt Ploidlip to do the upside down. Well the in other in other environments upside down RL clearly beats the classic algorithms. And what I like here is they took a lunar lander and which basically at every time step you get a reward in lunar lander and they hypothesized. Okay this is really good for these classic algorithms that do reward maximization instead of kind of learning this general behavior function. And what they did is they modified the game such that all the reward is given at the end of the episode. And then you see that upside down RL will actually outperform here the classic things where it's exactly the same game you just get the reward at the end. So upside down RL kind of learns the structure of the world learns that you get this reward at the end after such and such many time steps. So you can it will learn please get me zero reward in 50 time steps like no problem. But please get me a thousand rewards in a hundred time steps. No problem. I just go to the end of the episode right. Whereas these pure reward maximization techniques they don't they somehow have a harder time to do that. I like this investigation. I like the thinking outside the box. The Schmidhuber ism of the paper. It's just all great. It's a great time to be alive and check this out and I'll see you. Bye bye.
 He did it! Crazy son of a bitch did it again!
 What am I talking about? Jürgen Schmidhuber reinforcement learning upside down!
 New paper just dropped on the verge of the NeurIPS conference being presented at a workshop here.
 Presenting upside down reinforcement learning. I am pumped for this one, can you tell?
 It says we transform reinforcement learning into a form of supervised learning by turning traditional RL on its head.
 Calling this RL-lar. What do we call this? We'll just call it lar.
 Upside down reinforcement learning.
 And so this is upside down. Never mind.
 Okay, let's just check out how it works.
 So I'm going to give a brief overview before we go into this paper.
 Alright, so let's say you have a reinforcement learning problem. Let's say an Atari game for example.
 And in an Atari game you usually have a screen, right?
 And let's just say you're playing this marine commander. So there's water here, right?
 And there might be a bunch of... Here's your boat, right?
 There's a boat, a little boat. There might be a bunch of opponents right here.
 Fishy fish opponents, fishy fish opponents and so on.
 And there are a bunch of gold coins like here. That's a big gold coin, right?
 And you're kind of supposed to, I think you're supposed to like go get air.
 You have some air meter over here. Whatever.
 So there's this Atari game, right?
 You're supposed to get the reward which is this maybe this coin here.
 And stay alive as long as possible and so on.
 So this is a classic reinforcement learning problem.
 And there are various techniques for this. We've looked at a couple of them.
 And what upside down reinforcement learning does is basically what you do is you want to transform this input to a new representation.
 Which basically, well, if I can, maybe I can... Let me get this correctly.
 So then there's this over here and then there's a little fishy, a little fishy here.
 And there's a coin right here.
 So what you want to do is basically turn this input on its head like upside down.
 And so this way is kind of up or down or whatever in this new representation.
 And if you actually learn on this new representation with pretty the same techniques,
 it works much better than the classic RL setting.
 And this is not only for like these Atari games.
 Like this appears to hold throughout the RL space.
 So in robotics, like if you have a robot or whatever, this is a robot.
 It has a square head, as you can tell.
 You know, it's supposed to like open a door.
 You've seen this DARPA challenge. This doesn't work, right?
 But if you just transform this and actually turn the robot upside down,
 the robot will be able to open the door just fine.
 And even like if you have a chessboard and there's like a bunch of pieces on it.
 The problem in this case is you have to simulate this chessboard.
 And if you turn this around now, basically all the pieces will fall off.
 So what you need to do is you need to have a simulator that encodes a magnetic chessboard
 such that the pieces don't fall off.
 So it's a bit of programming effort. But if you do that...
 All right, I'm kidding.
 This is a new paradigm for RL, but it's unfortunately not as good.
 Someone should try the magnetic chessboard simulator.
 Upside down RL is a new paradigm for RL where basically the kind of notion of inputs
 and outputs of the RL algorithm are switched around a bit.
 So basic ideas here is that you have an RL algorithm that is also fed with a bunch of commands.
 So in classic RL what you'll have...
 Let's actually go back to this Atari game here, right?
 In classic RL, an RL algorithm will get the Atari game as a screen as an input
 and is asked from this to predict a bunch of outputs.
 So in classic Atari, these are eight actions.
 I'm going to draw three here, like go to the left, go to the right, or press the button for shoot, right?
 These are the actions you have and the algorithm is tasked.
 And there are different versions of this.
 In policy methods, policy gradient methods, typically the algorithm is tasked
 with outputting a distribution over these actions.
 In other methods like value learning, Q learning, the algorithm is tasked with assigning each of these actions a value.
 So in this situation, going to the left will be worth three in the future.
 Going to the right will be worth negative one and shooting will be worth zero.
 So you might want to go with this action here.
 Now in upside-down reinforcement learning, we've had observation going into the model
 and the model coming up with the value estimation of the different actions.
 In upside-down reinforcement learning, you'll have the observation and something else going into the model
 and the model coming up with an action.
 And this something else is the key.
 What you input here is your desire, your future desire.
 And in this paper, they call it a command.
 So you'll have a command as an input together with the observations.
 You basically say, here's my state and I would like to achieve,
 let's say five reward in the next five reward in the next two time steps, right?
 Make this happen.
 Right. This is this is your command going into the model and the model will then try to find actions
 such that in the next two time steps, you'll get five reward.
 You can easily see a model that learns this will actually be able to, you know, do various things,
 including doing the classic RL things like get as much reward as possible in given
 or in the shortest amount of time, but can also do much more.
 And in the general sense, the difference is how this is trained now.
 This model, when you train it, as you can see, you don't it's not trained with in my having in mind
 kind of only to get the maximum reward.
 It is trained to be much more a general kind of understanding of the world.
 I mean, learning what do I need to do to achieve a variety of goals?
 Specifically, what you want to do to train this is the following.
 Say you have a method of of moving in the world and collecting traces, right?
 So you go from state, state one, state two, state three.
 You go with like your action one, action two.
 Let's draw action three.
 The state four.
 And in each of these, you get a you get rewards, right?
 Reward one reward to reward three.
 Now, this in classic RL, this will kind of give you one training example, right?
 So this is if you consider this to be an episode, this will give you one training example to to run this sequence of actions.
 Upside down RL, you can actually consider this as many, many training examples.
 And here's what I mean.
 So if you, for example, start at state one, you can say, aha, within one time step, one one time step,
 I go to state two and I have achieved our one rewards by doing action a one.
 Right.
 So this now can be an input to your model.
 Your model could learn if you get as an observation, remember the previous thing as an observation, you get s one as a command.
 You get I want to achieve in one time step.
 Are one reward.
 Right. And you train this goes into the model and the model is trained to say a one given if I am in s one
 and I do a one, I will achieve that.
 Right. So you train the model to give a one as an output.
 And this is valid because in the past you've observed going from s one using a one to a state where you get this this kind of reward in this kind of time.
 But you can also so you can do all of these single steps.
 They will all provide individual training examples to your model.
 Right. But then also you can consider a two step thing.
 So you can say I'm in state s one and I go I go in two time steps.
 I have achieved our one plus our two reward by doing actions a one then a two.
 Right. And a two I'm going to do in parents here because what you want to do is you want to always always consider the action that comes right after where you are now.
 So again your training sample let me draw this up here.
 Maybe your training sample would be the following.
 I am in state s one.
 This would be my observation.
 My command would be I would like to achieve in two time steps reward r one plus r two reward.
 Right. This reward this both goes into the model.
 Right. You tell the model please given the state s one achieve this reward in this time.
 And the model is supposed to output a one saying ha in the past I was in this state and I did achieve this goal by using that.
 So the model is supposed to learn to achieve different goals.
 Right. So now you can not only train from good episodes right.
 You can train for any episode any episode usually in classic or you kind of want to focus on the good episodes because you want to maximize your reward.
 But here you can tell the model hey if you've done something particularly stupid let's say here in s three you done something the a three was particularly stupid gave you.
 So r three here was really bad reward like a negative five billion trillion.
 And you can actually train the model to recognize this can be a hey look if you are in s three and within one time step you want to achieve negative five billion billion billion trillion.
 Reward you all you have to do is action a three right.
 And then the cool thing now is if you are at evaluation time you actually want the big reward what you'll do is you simply plug in a different command simply in one time step still I'm in state s three in one time step.
 I want to achieve actually three reward not negative a lot right.
 And the model will have learned that a three will lead to a situation where you get a lot of negative reward.
 So the model will be like I'm for sure not going to do a three right.
 I'm going to do something else here because I have learned to map a three to like this really low reward.
 So in essence this has connections to things like hindsight experience replay and kind of universal value function where you kind of learn to go from any state to any other state in this.
 But none of these do none of these have this kind of command what Schmidhuber calls command here as an input to the model.
 And I think actually this is this is really positive to input this because usually in universal value functions what you would say is let's consider a simple grid world right.
 Whatever your agent is here and you need to you need to reach a goal that's down here.
 But you might not be able to learn it because it's super sparse reward and so on.
 But what you can do is you can learn to reach this position and this position and this position from various positions like go here go from here to here.
 You can learn to go from here to here.
 And you know in essence you would like it eventually to generalize to all the fields.
 So you basically learn to go from any position to any other position with your agent with these universal value or universal policy functions having sub goals.
 But they during that phase where they learn to go from anything to anything they don't they don't necessarily include this reward thing as a as an input.
 It's more like kind of either a sub goal or like the usual value function will simply approximate the reward.
 Whereas whereas in this technique we actually have a policy learning we actually output a an action value.
 Also hindsight experience replay what hindsight experience replay would do in the same situation right.
 You're here.
 We might do a videos on this in the future.
 You're here and you try right.
 And your agent actually it ends up here right ends up right here.
 What you can do is you can simply say oh well actually this this was my goal all along and then simply train train your model as if as if this thing here was your goal all along.
 And not this thing here and treat it as kind of a positive reward for this.
 At least that's how I understand it.
 Right.
 And both of these things are quite different than here where we have this command as input and I do I do like it.
 So I think this this is very much the basic things here.
 This it is extra extrapolated to kind of noisy inputs and noisy environments and so on.
 But this is the basic the basic gist of it.
 So here you see your you what you will learn is to map all and all is your representation of your input.
 So the screen for example or the chessboard.
 And I think also kind of the last action and there were you get in this step plus your horizon and desire.
 So in how much time you would like to achieve how much reward and then you can also get input some extra goals that you have.
 And so you can see basically any any episode that you've run in the past will give you a valid training example for this.
 Your model will simply learn to match the previous experience with the goals that were achieved in the previous experience.
 So there is lots of lots of generalizations here like how exactly these things are represented.
 This this time horizon can be a high dimensional object.
 The desire can be as I understand it somewhat a dimensional object.
 The extra commands can be like conditionals on these two things.
 It gets very complicated, but I want to jump ahead to a different paper where so this paper is basically just describing the algorithm.
 And then the next paper is doing experiments with this.
 Let's scroll past here.
 All right.
 So this paper training agents using up that down reinforcement learning released on the same day,
 but different authors that have used also made who was also here but have used this to implement a variant of this.
 And here you see again what I was trying to to explain.
 So in traditional RL, this especially here Q learning, you'll have this function which gets an observation as input and then Q learning especially.
 So you also get the action as an input and you're supposed to say for the given observation this particular action has this expected value as a return.
 Right.
 That's what I explained at the beginning.
 That's kind of value based reinforcement learning.
 Whereas the behavior function here, which would be upside down reinforcement learning gets the observation and a command and will map that to an action.
 And here again is what we've gone over.
 This is a bit of a different thing.
 So this agent has apparently run two different episodes.
 One point it did this sequence of actions and at the other point from the same starting state it did this sequence of action and you can see here on the right all the training samples we can we can derive from this.
 So we can say from state s 0 right.
 If I want to return in one time step, I have experienced this in the past right to return in one time step.
 All I have to do is take action a one.
 But if I want one return in one time step, I have to take action a two and you teach your behavior function to learn these things to learn to output these actions with these things here as inputs.
 And then what you hope of course is that this will generalize that it will learn to generalize that you can say now give me more reward than I have ever seen before right.
 And it will kind of learn which things correspond to lower reward, which things correspond to higher award and will be able to extrapolate which things will correspond to even higher report reward.
 Sorry.
 So they have two algorithms and this is kind of this is reminiscent of the old of the old RL kind of world where you do kind of one algorithm is continuously learning from the experience gathered by another algorithm.
 So you have one set of algorithms and this even in modern RL this this this is how it's done right.
 You have two different boxes right.
 Actually you have probably one box learning the model like this is I'm going to represent this here learner right.
 And the learner distributes the model to many many machines interacting with the simulators and these machines all they do is run episodes with the learned model and they will send back their experience here.
 And then the learner can learn from it and then at the end send it again.
 So so.
 All right here we go.
 So in each step what we do in order to to generate a new episode we don't always want to want to kind of execute one given policy.
 What we do is we sample from the end of the replay buffer and the replay buffer is sorted by returns right.
 So the highest return episodes are on top.
 So we want to sample the highest return episodes then we want to say maybe some of them are 10 steps long maybe some of them are five steps long and so on.
 So we set the horizon to be the mean of the length of these right and we set the desired return how much return should be achieved in this time to be the unit to sample from the uniform distribution between M and M plus S and M is the mean and S is the standard deviation of the selected episode.
 So so what this means is is like here is a bunch of episodes from the start at the same time.
 Here's a bunch of episodes that I ran right from here is time zero and then time goes on that I ran that had really high returns right.
 Now I'm going to take the mean time that these episodes ran like this.
 This is maybe five time steps. So in five time I want to achieve now how much reward now you look at all the rewards that were achieved.
 This is maybe a distribution that has some mean here like so and then you say I want to achieve a reward between here and one standard deviation higher than here.
 So right and this this would be the reward you want to achieve. So what you do is you kind of push your learned model to just go a bit beyond what it has seen so far is basically say look I you can do this but you can just do a bit more in the same amount of time.
 Please do this and you hope the model has learned to kind of generalize to do this.
 And if so you will execute these episodes and then these episodes will go back to the learner right.
 I'll go back to the learner here and the learner will learn from them and hopefully then you can like generalize even more and then you can say I now know how to achieve this bit more reward.
 Now I can if I run the episode I will achieve even more reward.
 I can push the model even further right. So at eval time you can always ask the model to produce as much reward as possible in the given time.
 And of course every episode sent back here is not only one training example as we saw but many many training examples can be derived from these models even beyond what's in what's in this paper.
 All right. So I think this was a good first shot at describing this algorithm. I hope you get the gist of it.
 I enjoy this a bit of a criticism for me would be it's still kind of doesn't it.
 So it doesn't touch the exploration dilemma.
 So it again deals with kind of incremental incrementally getting better whereas I feel this can easily get stuck in some minimum where it's not possible to do this incremental generalization of the model where you really need a new approach.
 And that's why games like Montezuma's Revenge are solved using algorithms like Go Explore and not any of the classic algorithms.
 That being said they have experiments where they show that especially in sparse reward environments they do better than classic or algorithms.
 So if you for example here take the lunar lander where A to C beats upside down RL and I guess you didn't get Matt Ploidlip to do the upside down.
 Well the in other in other environments upside down RL clearly beats the classic algorithms.
 And what I like here is they took a lunar lander and which basically at every time step you get a reward in lunar lander and they hypothesized.
 Okay this is really good for these classic algorithms that do reward maximization instead of kind of learning this general behavior function.
 And what they did is they modified the game such that all the reward is given at the end of the episode.
 And then you see that upside down RL will actually outperform here the classic things where it's exactly the same game you just get the reward at the end.
 So upside down RL kind of learns the structure of the world learns that you get this reward at the end after such and such many time steps.
 So you can it will learn please get me zero reward in 50 time steps like no problem.
 But please get me a thousand rewards in a hundred time steps.
 No problem. I just go to the end of the episode right.
 Whereas these pure reward maximization techniques they don't they somehow have a harder time to do that.
 I like this investigation. I like the thinking outside the box.
 The Schmidhuber ism of the paper. It's just all great.
 It's a great time to be alive and check this out and I'll see you. Bye bye.