Dream to Control: Learning Behaviors by Latent Imagination
deep learning machine learning arxiv google rnn recurrent reinforcement learning deep reinforcement learning imagination latent space world model control deepmind deep mind
Dreamer is a new RL agent by DeepMind that learns a continuous control task through forward-imagination in latent space.

https://arxiv.org/abs/1912.01603
Videos: https://dreamrl.github.io/

Abstract:
Learned world models summarize an agent's experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks from images purely by latent imagination. We efficiently learn behaviors by propagating analytic gradients of learned state values back through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.

Authors: Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at Dream to Control Learning Behaviors by Latent Imagination by Dani Jarhofner, Timothy Lillikrup, Jimmy Baa and Mohamed Nerozi. This is a reinforcement learning paper that iterates on a series of previous papers where the goal is to learn a policy. In this case they want to learn policies for these kind of continuous control tasks of these physics-based robots, these hopper or walker types of tasks where you have to control these joints in order to move forward. The goal is that you have multiple observations as you do in reinforcement learning and from each observation you need to somehow come up with an action of what to do. Then that will give you the next observation as well as a reward. If your goal is to move this spider, maybe the reward is proportional to how far you move. So your goal is to collect the maximum reward, which would mean you have to move the spider as far as possible simply by doing the correct actions. The goal of this paper now is to do this by learning to plan ahead in this latent space. As you can see here, the way they do it is they take the observation and they feed it through an encoder. You can think of this as maybe a convolutional neural network or something. Anything that can work, that can take an image as an input and give you a hidden representation. This here is the hidden representation. From this hidden representation you can determine what the next action is going to be. Then you get a new observation and then again you can feed that along with the last hidden state into a new hidden state. Previous models do this a lot. You encode your observation and you have a recurrent neural network that incorporates all of the observations into a hidden state along with the actions you take. Then you always decide on a next action to do. What does this model do differently? This model wants to do this all in hidden space. This model wants to say I am here, I have this observation. Now my encoder tells me that this is going to give me this hidden state. Now what it wants to do is it wants to take in the action that it's doing and without seeing the next observation, it wants to predict it. It wants to say if I am here and I do this action, what might the action be? The action might be to put the joystick to the right. It will learn the hidden state corresponding to the spider being a bit more to the right. This is a bit more to the right than it is right now. It will need to do so a number of time steps into the future and it will learn from its own imagination. It will imagine into the future how the hidden states look and then it will learn from that instead of having to really do the actions in the real world. We've already looked at a number of papers including something like mu0 or I2A or something like this. This now is slightly different. You can see what's different here. What is different is in mu0 we used this latent model in order to plan ahead, like in order to do our decision tree planning ahead and so on. This model doesn't do this. This model still wants to come up with a single policy where you encode your state. On the right is the final result. You encode your state, it gets you to a hidden representation and then from that you determine what your actions going to be and you have your next state and so on. The final goal is simply going to be a policy like a single shot policy without any Monte Carlo tree expansion and so on. What it wants to do is it wants to learn this policy not by interacting in the real world like here on the left but actually by interacting only in the dream world right here. The crucial part if you want to learn from your dreams is to make sure that your dreams are an accurate representation of the real world. We already saw this in a paper called World Models by Jürgen Schmidhuber. In that paper what they did was they first collected experience, like this one, and then they learned from the one observation to predict the next ones or to predict the next hidden states. They did so by basically moving in the world at random. They have this little spider thingy and they just do random movements. They randomly move around and thus they collect these trajectories and then they learn from the random trajectories. The difference that this paper does is it does these steps iteratively. It will not learn from random policy but it will actually first start out learning this random, learning a good policy for its environment model, then acting going back and using that policy in order to learn a better environment model and then again learn using the better environment model in order to learn a better policy. If this wasn't clear enough we'll jump to the algorithm. The algorithm isn't actually too complicated. As I said I think it's a relatively minor iteration on previous research but it appears to work and it works in these kind of continuous control tasks. You see you have three models here that you need to learn and that's what you see over here. There is representation, transition and reward and you'll see they all have the same parameters. That gives you an indication that these things are a single model. Now what is the model representation, transition and reward? This is the thing on the left here. In this part of the algorithm you assume that you have a policy. You already know what action you do or you can even assume that you have some experience. You have your agent is running with a given policy and you simply collect that and now you're trying to learn. Let me scratch all of this. What do you have given? Given is the observation sequence and the actions you took and the rewards you got. That's also given. Each action gives you reward. These things are given, provided to you and now what do you want to learn? You want to learn a representation and a transition and let's say a reward. You also want to predict the next reward. This thing, this thing. As we already said you can do this by encoding the state using for example a CNN and then using an LSTM in order to incorporate this over time. What you learn is the transition from one hidden state to the next hidden state and you also learn how the observation goes into the hidden state. Thirdly you learn that if I'm in this hidden state and I take this particular action I will get this reward in the future. You can learn this from just a set of pre-computed or from a set of experience that you have in your let's say your replay buffer. This is one model and you learn this here in this first step in this called dynamics learning section. You see while not converged, you do dynamics learning, you draw data sequences from your experience, then you compute the model states. These are the hidden states and then you update this parameter theta using representation learning. They don't really specify what representation learning is but they do give examples of what you can do. I think their point is whatever you need to do in order to learn this representation. One example is actually drawn here. One example is you can learn a model that reconstructs the next state or actually sorry reconstructs the same state. You can learn a model that predicts. If you give the observation as an input it goes through the hidden state. You can learn a decoder that reconstructs that observation. This is usually done in things like variational auto encoders in order to produce generative models. This part here would be the generator and that would be kind of the thing of interest if you are doing a variational auto encoder. Of course here our quantity of interest is this encoder model because we want a good representation of the state. It comes down to the same thing. If you can learn a model that learns to accurately reconstruct the observation then your representation here in the middle is probably an informative one. Because you learn the same model across multiple observations that means it can accurately encode what makes one observation different from another one. This is how you learn the theta parameters. The other models here are the action and the value parameters. This is here in the step called behavior learning. In the behavior learning what they say is imagine trajectories from each of the states that you have. What you're going to do is from each of the observations here you're going to obtain the hidden states. From each of the hidden states here, here is an observation from its hidden state, you're going to use the model that you learned here through the LSTM. This is terrible. Through the LSTM you're going to use that model to imagine future trajectories of hidden states. You have given, or now is the observation here, and the hidden state. You're going to imagine future hidden states, you're also going to imagine future rewards. You are going to use your policy in order to determine which actions you're going to take. The ultimate goal here is to learn a good policy, so a policy that will give you better rewards in the future. This is regular reinforcement learning, except that the difference is in regular reinforcement learning I have my observation, I encode it and then I determine what action I want to take. Then I feed that action back into the environment, which would give me the next observation. Then I'd use that to determine, maybe in conjunction with the last hidden state, the next action. In this thing, since we learned a dynamics model of the hidden states, we can simply determine the action and then simply compute what the probable next hidden state is going to be. Then use that to determine an action again and so on. There's no need to go through the environment, which means potentially we can learn much faster without having to expensively interact with the environment. That allows us to basically... Also these models here, they might be quite large, so our backprop now only needs to happen through this path basically, if we want to, or through this path here, in case we have discrete actions. That will be the dynamics learning. As you can see, we predict the rewards and the values and compute value estimates. Then we update these parameters. What we have is here a value function. The value function is dependent on this psi here. This we update using a gradient of its output minus the true value. This here is an estimate of the value. As you know, a value function is supposed to tell you the complete future reward given a state. It's important for us that we have a function that can estimate that, because of course then we can take actions. If we can make this function go high and this is an accurate function, that means we get a lot of reward in the future. It's important to learn this function. Here you can see we adjust it into the direction of matching this quantity better. We'll get to this quantity in a second. You can also see we update this parameter, which is the action model. Here you see that the action model depends on this. This is our policy. This thing here determines which action we take. We update it into the direction. This is a gradient with respect to this value function. We train the policy to maximize the value, which is all the future rewards that we get. Of course we can do this because we can now back propagate through all of these time steps. We have this transition model. We can back propagate through all of this, which is pretty cool. I think in my opinion the workhorse of this paper might be this quantity here. How exactly do you compute the value of a state? Especially in these continuous control tasks you sometimes have a lot of steps. These trajectories might be pretty long and they might be longer than what you can back propagate here reasonably from time step to time step. Even an LSTM might only be able to back prop through a couple of dozen or maybe a few hundred steps in time. Maybe you have longer trajectories here. I think this value estimate here is a main component of extending that range. They say this is according to equation 6 and this is what it does. This is my opinion that this here is the workhorse of the method. It's a three-step process actually. It's pretty heavy. You see this is the quantity they estimate with the value function. It is set between an average over... H is the time horizon that you're looking for. It is set between these two things across the sum over the time horizon. Now each of those things again here is a sum over this tau here, which is this tau and H minus 1. H here is the minimum of tau plus K and tau plus horizon. This quantity looks K steps into the future. For each step to the horizon we look K steps into the future. For each step we look into the future we sum again across these quantities here. These quantities here, what is that? It's a mixture of the reward you get in that particular step plus your own your estimate of the value function at the at the horizon step discounted by that. So it's a pretty... Imagine you have like a time number of steps that you took and each time you get a reward. This is a very complicated way of going into the future, summing up the rewards, going more steps, summing up the rewards again in different fashion and then mixing these individual quantities. So this one, this one, this one that you got from accumulating all of these in a weird fashion. That allows you to look way beyond. Especially you see here your estimate of the value function will actually include your own value function that again probably looks into the future. So what you accumulate from the last step in your time horizon already includes information from all the future steps because you take your own value estimate into account. This is I think it's very convoluted but again I think this complicated value estimate allows you to have a better value estimate far into the future. They do show some kind of samples here of what they can do. I haven't found any videos of it unfortunately but it appears to work pretty well. They have a discussion of different representation learning methods and different experiments and ablations and so on. So I invite you to look at this paper and I hope this was somewhat clear. Bye bye.
 Hi there! Today we're looking at Dream to Control Learning Behaviors by Latent
 Imagination by Dani Jarhofner, Timothy Lillikrup, Jimmy Baa and
 Mohamed Nerozi. This is a reinforcement learning paper that iterates on a
 series of previous papers where the goal is to learn a policy. In this
 case they want to learn policies for these kind of continuous control tasks
 of these physics-based robots, these hopper or walker types of tasks where
 you have to control these joints in order to move forward. The
 goal is that you have multiple observations as you do in reinforcement
 learning and from each observation you need to somehow come up with an action
 of what to do. Then that will give you the next observation as well as a
 reward. If your goal is to move this spider, maybe the reward is
 proportional to how far you move. So your goal is to collect the maximum reward,
 which would mean you have to move the spider as far as possible simply by
 doing the correct actions. The goal of this paper now is to do this by
 learning to plan ahead in this latent space. As you can see
 here, the way they do it is they take the observation and they feed it through an
 encoder. You can think of this as maybe a convolutional neural network or
 something. Anything that can work, that can take an image as an input and give
 you a hidden representation. This here is the hidden representation. From
 this hidden representation you can determine what the next action is going
 to be. Then you get a new observation and then again you can feed that along
 with the last hidden state into a new hidden state. Previous
 models do this a lot. You encode your observation and you have a
 recurrent neural network that incorporates all of the observations
 into a hidden state along with the actions you take. Then you always
 decide on a next action to do. What does this model do differently? This model
 wants to do this all in hidden space. This model wants to say
 I am here, I have this observation. Now my encoder tells me that this is going to
 give me this hidden state. Now what it wants to do is it wants to take in the
 action that it's doing and without seeing the next observation, it wants to
 predict it. It wants to say if I am here and I do this action, what
 might the action be? The action might be to put the joystick to the right. It will
 learn the hidden state corresponding to the spider being a bit more to the right.
 This is a bit more to the right than it is right now. It will need to
 do so a number of time steps into the future and it will learn from
 its own imagination. It will imagine into the future how the hidden
 states look and then it will learn from that instead of having to really do the
 actions in the real world. We've already looked at a number of papers
 including something like mu0 or I2A or something like this. This now is
 slightly different. You can see what's different here.
 What is different is in mu0 we used this latent model in order to
 plan ahead, like in order to do our decision tree planning ahead and so on.
 This model doesn't do this. This model still wants to come up with a single
 policy where you encode your state. On the right is the final result.
 You encode your state, it gets you to a hidden representation and then from that
 you determine what your actions going to be and you have your next state and so on.
 The final goal is simply going to be a policy like a single shot policy
 without any Monte Carlo tree expansion and so on. What it wants to do is it
 wants to learn this policy not by interacting in the real world like here
 on the left but actually by interacting only in the dream world right here.
 The crucial part if you want to learn from your dreams is to make sure
 that your dreams are an accurate representation of the real world.
 We already saw this in a paper called World Models by Jürgen Schmidhuber.
 In that paper what they did was they first collected experience,
 like this one, and then they learned from the one observation
 to predict the next ones or to predict the next hidden states.
 They did so by basically moving in the world at random. They have this
 little spider thingy and they just do random movements. They randomly
 move around and thus they collect these trajectories and then they learn from
 the random trajectories. The difference that this paper does is it does these
 steps iteratively. It will not learn from random policy but it will
 actually first start out learning this random, learning a good policy for its
 environment model, then acting going back and using that policy in order to learn
 a better environment model and then again learn using the better environment
 model in order to learn a better policy. If this wasn't clear enough we'll jump
 to the algorithm. The algorithm isn't actually too complicated. As I said
 I think it's a relatively minor iteration on previous research but it
 appears to work and it works in these kind of continuous control tasks.
 You see you have three models here that you need to learn and that's what you see
 over here. There is representation, transition and reward and you'll see
 they all have the same parameters. That gives you an indication that these
 things are a single model. Now what is the model representation,
 transition and reward? This is the thing on the left here.
 In this part of the algorithm you assume that you have a policy. You
 already know what action you do or you can even assume that you have some
 experience. You have your agent is running with a given policy and you
 simply collect that and now you're trying to learn. Let me scratch all of
 this. What do you have given? Given is the observation sequence and the actions
 you took and the rewards you got. That's also given. Each action gives
 you reward. These things are given, provided to you and now what do
 you want to learn? You want to learn a representation and a transition and
 let's say a reward. You also want to predict the next reward. This thing,
 this thing. As we already said you can do this by encoding the state using
 for example a CNN and then using an LSTM in order to incorporate this over time.
 What you learn is the transition from one hidden state to the next hidden
 state and you also learn how the observation goes into the hidden state.
 Thirdly you learn that if I'm in this hidden state and I take this particular
 action I will get this reward in the future. You can learn this from
 just a set of pre-computed or from a set of experience that you have in your
 let's say your replay buffer. This is one model and you learn this here
 in this first step in this called dynamics learning section. You see
 while not converged, you do dynamics learning, you draw data sequences from
 your experience, then you compute the model states. These are the hidden
 states and then you update this parameter theta using representation
 learning. They don't really specify what representation learning is but they
 do give examples of what you can do. I think their point is whatever you need
 to do in order to learn this representation. One example is
 actually drawn here. One example is you can learn a model that reconstructs the
 next state or actually sorry reconstructs the same state. You can learn a
 model that predicts. If you give the observation as an input it goes
 through the hidden state. You can learn a decoder that reconstructs that
 observation. This is usually done in things like variational auto encoders in
 order to produce generative models. This part here would be the
 generator and that would be kind of the thing of interest if you are doing a
 variational auto encoder. Of course here our quantity of interest is this
 encoder model because we want a good representation of the state.
 It comes down to the same thing. If you can learn a model that learns to
 accurately reconstruct the observation then your representation here in the
 middle is probably an informative one. Because you learn the same model
 across multiple observations that means it can accurately encode what makes one
 observation different from another one. This is how you learn the
 theta parameters. The other models here are the action and the value
 parameters. This is here in the step called behavior learning. In the
 behavior learning what they say is imagine trajectories from each of the
 states that you have. What you're going to do is from each of the observations
 here you're going to obtain the hidden states. From each
 of the hidden states here, here is an observation from its hidden state,
 you're going to use the model that you learned here through the LSTM.
 This is terrible. Through the LSTM you're going to use that model to imagine future
 trajectories of hidden states. You have given, or now is the
 observation here, and the hidden state. You're going to imagine future hidden
 states, you're also going to imagine future rewards. You are going to use
 your policy in order to determine which actions you're
 going to take. The ultimate goal here is to learn a good policy, so a
 policy that will give you better rewards in the future. This is
 regular reinforcement learning, except that the difference is in regular
 reinforcement learning I have my observation, I encode it and then I
 determine what action I want to take. Then I feed that action back into the
 environment, which would give me the next observation. Then I'd use that to
 determine, maybe in conjunction with the last hidden state, the next action.
 In this thing, since we learned a dynamics model of the hidden states, we can simply
 determine the action and then simply compute what the probable next hidden
 state is going to be. Then use that to determine an action again and so on.
 There's no need to go through the environment, which means potentially we
 can learn much faster without having to expensively interact with the
 environment. That allows us to basically... Also these models here, they might be
 quite large, so our backprop now only needs to happen through this path
 basically, if we want to, or through this path here, in case we have
 discrete actions. That will be the dynamics learning.
 As you can see, we predict the rewards and the values and
 compute value estimates. Then we update these parameters. What we have
 is here a value function. The value function is dependent on this psi here.
 This we update using a gradient of its output minus the true value.
 This here is an estimate of the value. As you know, a value function is
 supposed to tell you the complete future reward given a state.
 It's important for us that we have a function that can estimate that, because of
 course then we can take actions. If we can make this function go high and this
 is an accurate function, that means we get a lot of reward in the future.
 It's important to learn this function. Here you can see we adjust it into the
 direction of matching this quantity better. We'll get to this quantity in a
 second. You can also see we update this parameter, which is the action model.
 Here you see that the action model depends on this. This is our policy.
 This thing here determines which action we take. We update it into the
 direction. This is a gradient with respect to this value function.
 We train the policy to maximize the value, which is all the future rewards that we get.
 Of course we can do this because we can now back propagate through all of these
 time steps. We have this transition model. We can back
 propagate through all of this, which is pretty cool. I think in my opinion the
 workhorse of this paper might be this quantity here.
 How exactly do you compute the value of a state? Especially in these continuous
 control tasks you sometimes have a lot of steps. These trajectories
 might be pretty long and they might be longer than what you can back propagate
 here reasonably from time step to time step. Even an LSTM might only be
 able to back prop through a couple of dozen or maybe a few hundred steps in
 time. Maybe you have longer trajectories here. I think this
 value estimate here is a main component of extending that range. They say this
 is according to equation 6 and this is what it does. This is my
 opinion that this here is the workhorse of the method. It's a
 three-step process actually. It's pretty heavy. You see this is the
 quantity they estimate with the value function. It is set between an
 average over... H is the time horizon that you're looking for. It is
 set between these two things across the sum over the time horizon. Now each of
 those things again here is a sum over this tau here, which is this
 tau and H minus 1. H here is the minimum of tau plus K and tau plus horizon.
 This quantity looks K steps into the future. For each
 step to the horizon we look K steps into the future. For each step we
 look into the future we sum again across these quantities here. These
 quantities here, what is that? It's a mixture of the reward you get in that
 particular step plus your own your estimate of the value function at the
 at the horizon step discounted by that. So it's a pretty... Imagine you have
 like a time number of steps that you took and each time you get a reward.
 This is a very complicated way of going into the future,
 summing up the rewards, going more steps, summing up the rewards again in different
 fashion and then mixing these individual quantities. So this one, this
 one, this one that you got from accumulating all of these in a weird
 fashion. That allows you to look way beyond. Especially you see here your
 estimate of the value function will actually include your own value function
 that again probably looks into the future. So what you accumulate from the
 last step in your time horizon already includes information from all the future
 steps because you take your own value estimate into account. This is I think
 it's very convoluted but again I think this complicated value
 estimate allows you to have a better value estimate far into the future.
 They do show some kind of samples here of what they can do. I haven't found any
 videos of it unfortunately but it appears to work pretty well. They have a
 discussion of different representation learning methods and different
 experiments and ablations and so on. So I invite you to look at this paper and I
 hope this was somewhat clear. Bye bye.