Curiosity-driven Exploration by Self-supervised Prediction
https://arxiv.org/abs/1705.05363

Authors: Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell

Abstract:
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward, where curiosity allows for far fewer interactions with the environment to reach the goal; 2) exploration with no extrinsic reward, where curiosity pushes the agent to explore more efficiently; and 3) generalization to unseen scenarios (e.g. new levels of the same game) where the knowledge gained from earlier experience helps the agent explore new places much faster than starting from scratch.
 Hi there! Today we're going to look at this paper, Curiosity-Driven Exploration by Self-Supervised Prediction. It's a relatively short idea, so it shouldn't take too long. So the fundamental idea of the paper is to tackle the reward sparseness problem reinforcement learning. For example, if you have a Super Mario game like here, and there's a number of ways you can think of the reward, but one way you could formulate it is that you simply get kind of a plus one reward when you finish the game, or the level. Let's say you finish the level, you get plus one. If you die or don't make it in time, you get negative one. I think there's no way to not make it in... Oh yeah, there's actually a time limit. So the... The problem here is that your algorithm kind of needs to learn to make things now such that it gets to the end of the level, but the reward is only at the end of the level. So basically step by step it has no signal to go on because the reward is always zero, and it kind of needs to learn these long range dependencies. And that's notoriously hard in reinforcement learning to step by step learn actions that kind of maximize some very long term goal. So you can also think of a game of chess where your reward is going to be whether you win or lose at the end, but step by step it's kind of this... The reward is 50ish steps away. So you have no way of kind of step by step optimizing your actions in a meaningful manner. So there are many ways to get around this. One way that people have done is what's called reward shaping. And reward shaping is you're trying to introduce additional rewards kind of as a designer of the algorithm that you know are kind of good or helping to solve the problem or at least correlated with the reward you're going to get at the end. So in Mario this could be like the further right you go, the more reward you get. You get kind of an additional reward if you go right. Coincidentally I think in real Mario this also gives you points, but our situation is that the reward is just going to be at the end. You could also say like if you kill the... Or if you stomp the goombas, one goomba you stomp, that actually gives you also a bit of reward. In chess you could say like the more pieces you have, that gives you a bit of reward if you have more pieces than your opponent, if your opponent loses pieces. You don't and you also get a bit of reward if you get more territory on the board and so on. So these are all things that we know kind of correlate with the end reward. Like because in Mario for example the end of the level is actually on the right. But of course it's not perfect because sometimes there are situations where you kind of have to go back, go around something or go over something and not immediately go to the right. As well as in chess there are good sacrifices that you can make. So these kind of additional rewards they help, but they're not perfect. And the biggest problem with them is they're very domain specific. So a developer of the algorithm you basically have to know the domain like Super Mario and you have to know the goal is on the right. So you have to construct your reward in order to kind of reflect this. And this is very domain specific. Basically you have to do it for every domain again and again and again. In chess you have to know something about chess to play and so on. So one way around this, and this paper proposes one method to do this, is to introduce an additional reward not based on the domain specifically, but based on what they call this curiosity. And it's specifically curiosity by self supervised prediction. So what does that mean? The idea is not new in that people have kind of done this before. If we go for example down here. So here is this kind of doom environment and what you could say is in my agent I have kind of a little module that's going to predict the future. So like if I'm here then I will basically choose an action, my agent will choose an action, like move forward, like press the forward key and then I will predict how that's going to look. And of course we know this is kind of a 3D environment so this is probably going to be this part of the screen is going to be the full screen because you're now closer and so on the perspective changes a little bit. But basically this should be a learned neural network that predicts the future from the state now and the action now. And basically you can train this in a supervised fashion because you will perform some actions, you will collect some data about this so you can learn a network that is going to predict one step into the future basically, how the environment will look. And then, and this is by no means kind of a new idea to introduce rewards based on this type of learning how the environment acts. We've seen this in like the A3C paper, the original one where the additional reward is something like pixel control where they consider like okay this pixel here, how much can I control it by my action, like how does my action influence it, can I predict this and so on. And to learn how to control the pixels on the screen by your actions and to give a reward based on that so that's been around this idea. And what this paper here does specifically is they say well I'm going to predict the future and if I am wrong about the prediction then that gives me a reward and that's the curiosity part. Basically it means like if I have a good model of what's going to happen in the future and then I predict the future and then I'm wrong it means something new has happened, something special, something that I hadn't expected. And therefore if the goal is to get the algorithm to explore by itself which is what you need to do when you don't have a reward, right? When you don't have a reward what you want your algorithm to do is simply to go around and explore. And in a sense they're saying okay the way to do this is to go by curiosity which means is to go to actively seek out environments that you wouldn't expect basically. So whenever you don't expect something that means it's something new, that means you haven't had this experience before, right? And that means that it's kind of a new state to explore. That you have not seen this before so kind of in absence of any reward you might as well go where you haven't been before and that's kind of the essence. So they outline a number of problems that you might have with this approach. They give the example, let's first actually go to what the model actually looks like. So that's here. You can see this is kind of what they call an intrinsic curiosity module. So you have a state here, you're in a state, you have your policy and your policy gives you an action. And the action goes to the environment and the environment gives you the next state and also what's called the reward. They call here E is the extrinsic reward that you get from the environment. But they also combine this with what's called an intrinsic reward that you get from here that you get from the curiosity module. And that's what we've discussed. It kind of tries to assess how new is the state that I'm going to be in. How surprising it is for me. So the thing is that I'm going to first describe the model how you would build it and how that gets you into problems and then how to fix it. So how you would build this is to have this what's called this forward model. So the forward model takes the action and the current state and it kind of predicts the next state that's in here. Don't worry about the phi hat right now. It predicts the next state and then you compare this to the actual next state. You subtract, you just subtract the next state and then you get the next state. You subtract, you just look at the difference between what you predict the next state is going to be and what the next state really is. And that gives you the intrinsic reward. The more different these are, the higher the reward. That's what we've discussed. How much different is it from what I've expected. So how does that get you into problems? And the authors give a very good illustrative example of say you are in an environment. Let's actually go over here. You are in an environment and you have your screen. And here is kind of a road that you need to maybe walk after. And here are some leaves in the wind. I'm very bad at drawing leaves so imagine these are leaves and there's wind right? Like winds coming from here and kind of shaking up these leaves and so on. So if you simply try to predict this entire screen as your forward model, what's going to happen is you will never be able to predict how these leaves are going to move because there basically you can't influence them. You can predict a bit from the current state but the action you take has no influence on how these leaves are going to move because they are influenced by the wind. And the wind is kind of this random-ish process that you can't control. So the authors say because of this your algorithm is always going to find these leaves basically interesting, curious, be curious about it because it can't predict them. And we've seen that the reward that they model to give an addition is based on how well you cannot predict a certain state. And they say okay if we do like this then these random things that we can't influence will always be surprising and therefore we will always be curious about them and therefore we will always kind of look at the leaves and be amazed and get reward after reward because we can't predict them. That's not the goal. So what they're arguing is that why are these leaves not important for curiosity? Because we can't influence them with our actions. Like we can influence where we go on this road because we can kind of move and the road is kind of static, not governed by these random processes. But the leaves we would like to discard them. We can't influence them. And therefore what they say is what we need is an encoder that takes a state and I'm going to try to delete this annotation. So we need an encoder here features that takes a state and it outputs features of the state. And then our forward model isn't fed with the state, it's fed with the features of the state and is not going to output the next state. So we need an encoder that takes a state and is fed with the features of the state and is not going to output the next state as such but the features of the next state. It predicts the features and then we're going to compare that with the features of the true next state and that's what we compare. So how does this encoder, these features need to look? And they're saying well these features should kind of only consider things about the state that are actually dependent on our actions. And they have a very interesting way of achieving to train such an encoder, such a feature producing function in that they say it's going to be a neural network that we train by training this so called inverse model. So we take this encoder and we train this inverse model on top of it and the inverse model takes the features of the last state and the new state and is trying to predict this action, this action right here. So this is this action, the action we took to get from the old state to the new state. So this inverse model is trained to predict what action was taken to get from the old state to the new state. And by training the encoder with this inverse model, like training this end to end, you will make the encoder such that it only considers things that are actually relevant to predicting this action. So in the leaves example it would discard the leaves. It will discard anything that you can't influence with your action and therefore it will only retain features that are dependent on your action. I think that's quite an interesting way to get rid of the irrelevant information that they don't want. And then they can use this encoder to train this forward model and to essentially get information from the old model and to essentially get this intrinsic reward. So I find this idea quite interesting and as I said the idea of intrinsic reward and curiosity to go for exploration is not new, but I think this kind of approach and I'm sure it's been around in some variants, but I've just stumbled across this and this is quite interesting. So we're going to take a look, and you can go about the math yourself, but they do these kind of experiments and they corrupt, as you can see, part of the screen with noise here and they of course show like, okay, since the noise is not dependent on our action, our features do actually discard this noise, only focus on the part that we can actually influence by our actions. So that's, I think, all in all pretty interesting. They show, of course, that their algorithm then outperforms the kind of baseline of A3C on these sparse reward tasks and the sparser here you can see like the left is like dense reward and then sparse reward and then very sparse reward and at some point you see the A3C simply doesn't do it anymore. But what's also interesting is here you have the ICM in pixels, which kind of means pixel-based curiosity, so where we don't have this encoder, where we simply try to predict the pixels of the environment and that works if you have like this kind of sparse reward thing, but if you want to, if you have the very sparse reward, that also fails and you actually need this encoder that discards what's not relevant for predicting the actions. Yeah, so you can take a look at the rest of the paper yourself. I find it quite interesting. They analyze how their agent explore these mazes and things and they have more experiments on like benchmark tasks. So have a look at it and I'll see you next time.
 Hi there! Today we're going to look at this paper, Curiosity-Driven Exploration by Self-Supervised
 Prediction. It's a relatively short idea, so it shouldn't take too long. So the fundamental
 idea of the paper is to tackle the reward sparseness problem reinforcement learning.
 For example, if you have a Super Mario game like here, and there's a number of ways you
 can think of the reward, but one way you could formulate it is that you simply get kind of
 a plus one reward when you finish the game, or the level. Let's say you finish the level,
 you get plus one. If you die or don't make it in time, you get negative one. I think
 there's no way to not make it in... Oh yeah, there's actually a time limit. So the...
 The problem here is that your algorithm kind of needs to learn to make things now such
 that it gets to the end of the level, but the reward is only at the end of the level.
 So basically step by step it has no signal to go on because the reward is always zero,
 and it kind of needs to learn these long range dependencies. And that's notoriously hard
 in reinforcement learning to step by step learn actions that kind of maximize some very
 long term goal. So you can also think of a game of chess where your reward is going to
 be whether you win or lose at the end, but step by step it's kind of this... The reward
 is 50ish steps away. So you have no way of kind of step by step optimizing your actions
 in a meaningful manner. So there are many ways to get around this. One way that people
 have done is what's called reward shaping. And reward shaping is you're trying to introduce
 additional rewards kind of as a designer of the algorithm that you know are kind of good
 or helping to solve the problem or at least correlated with the reward you're going to
 get at the end. So in Mario this could be like the further right you go, the more reward
 you get. You get kind of an additional reward if you go right. Coincidentally I think in
 real Mario this also gives you points, but our situation is that the reward is just going
 to be at the end. You could also say like if you kill the... Or if you stomp the goombas,
 one goomba you stomp, that actually gives you also a bit of reward. In chess you could
 say like the more pieces you have, that gives you a bit of reward if you have more pieces
 than your opponent, if your opponent loses pieces. You don't and you also get a bit of
 reward if you get more territory on the board and so on. So these are all things that we
 know kind of correlate with the end reward. Like because in Mario for example the end
 of the level is actually on the right. But of course it's not perfect because sometimes
 there are situations where you kind of have to go back, go around something or go over
 something and not immediately go to the right. As well as in chess there are good sacrifices
 that you can make. So these kind of additional rewards they help, but they're not perfect.
 And the biggest problem with them is they're very domain specific. So a developer of the
 algorithm you basically have to know the domain like Super Mario and you have to know the
 goal is on the right. So you have to construct your reward in order to kind of reflect this.
 And this is very domain specific. Basically you have to do it for every domain again and
 again and again. In chess you have to know something about chess to play and so on. So
 one way around this, and this paper proposes one method to do this, is to introduce an
 additional reward not based on the domain specifically, but based on what they call
 this curiosity. And it's specifically curiosity by self supervised prediction. So what does
 that mean? The idea is not new in that people have kind of done this before. If we go for
 example down here. So here is this kind of doom environment and what you could say is
 in my agent I have kind of a little module that's going to predict the future. So like
 if I'm here then I will basically choose an action, my agent will choose an action, like
 move forward, like press the forward key and then I will predict how that's going to look.
 And of course we know this is kind of a 3D environment so this is probably going to be
 this part of the screen is going to be the full screen because you're now closer and
 so on the perspective changes a little bit. But basically this should be a learned neural
 network that predicts the future from the state now and the action now. And basically
 you can train this in a supervised fashion because you will perform some actions, you
 will collect some data about this so you can learn a network that is going to predict one
 step into the future basically, how the environment will look. And then, and this is by no means
 kind of a new idea to introduce rewards based on this type of learning how the environment
 acts. We've seen this in like the A3C paper, the original one where the additional reward
 is something like pixel control where they consider like okay this pixel here, how much
 can I control it by my action, like how does my action influence it, can I predict this
 and so on. And to learn how to control the pixels on the screen by your actions and to
 give a reward based on that so that's been around this idea. And what this paper here
 does specifically is they say well I'm going to predict the future and if I am wrong about
 the prediction then that gives me a reward and that's the curiosity part. Basically it
 means like if I have a good model of what's going to happen in the future and then I predict
 the future and then I'm wrong it means something new has happened, something special, something
 that I hadn't expected. And therefore if the goal is to get the algorithm to explore by
 itself which is what you need to do when you don't have a reward, right? When you don't
 have a reward what you want your algorithm to do is simply to go around and explore.
 And in a sense they're saying okay the way to do this is to go by curiosity which means
 is to go to actively seek out environments that you wouldn't expect basically. So whenever
 you don't expect something that means it's something new, that means you haven't had
 this experience before, right? And that means that it's kind of a new state to explore.
 That you have not seen this before so kind of in absence of any reward you might as well
 go where you haven't been before and that's kind of the essence. So they outline a number
 of problems that you might have with this approach. They give the example, let's first
 actually go to what the model actually looks like. So that's here. You can see this is
 kind of what they call an intrinsic curiosity module. So you have a state here, you're in
 a state, you have your policy and your policy gives you an action. And the action goes to
 the environment and the environment gives you the next state and also what's called
 the reward. They call here E is the extrinsic reward that you get from the environment.
 But they also combine this with what's called an intrinsic reward that you get from here
 that you get from the curiosity module. And that's what we've discussed. It kind of tries
 to assess how new is the state that I'm going to be in. How surprising it is for me. So
 the thing is that I'm going to first describe the model how you would build it and how that
 gets you into problems and then how to fix it. So how you would build this is to have
 this what's called this forward model. So the forward model takes the action and the
 current state and it kind of predicts the next state that's in here. Don't worry about
 the phi hat right now. It predicts the next state and then you compare this to the actual
 next state. You subtract, you just subtract the next state and then you get the next state.
 You subtract, you just look at the difference between what you predict the next state is
 going to be and what the next state really is. And that gives you the intrinsic reward.
 The more different these are, the higher the reward. That's what we've discussed. How much
 different is it from what I've expected. So how does that get you into problems? And the
 authors give a very good illustrative example of say you are in an environment. Let's actually
 go over here. You are in an environment and you have your screen. And here is kind of
 a road that you need to maybe walk after. And here are some leaves in the wind. I'm
 very bad at drawing leaves so imagine these are leaves and there's wind right? Like winds
 coming from here and kind of shaking up these leaves and so on. So if you simply try to
 predict this entire screen as your forward model, what's going to happen is you will
 never be able to predict how these leaves are going to move because there basically
 you can't influence them. You can predict a bit from the current state but the action
 you take has no influence on how these leaves are going to move because they are influenced
 by the wind. And the wind is kind of this random-ish process that you can't control.
 So the authors say because of this your algorithm is always going to find these leaves basically
 interesting, curious, be curious about it because it can't predict them. And we've
 seen that the reward that they model to give an addition is based on how well you cannot
 predict a certain state. And they say okay if we do like this then these random things
 that we can't influence will always be surprising and therefore we will always be curious about
 them and therefore we will always kind of look at the leaves and be amazed and get reward
 after reward because we can't predict them. That's not the goal. So what they're arguing
 is that why are these leaves not important for curiosity? Because we can't influence
 them with our actions. Like we can influence where we go on this road because we can kind
 of move and the road is kind of static, not governed by these random processes. But the
 leaves we would like to discard them. We can't influence them. And therefore what they say
 is what we need is an encoder that takes a state and I'm going to try to delete this
 annotation. So we need an encoder here features that takes a state and it outputs features
 of the state. And then our forward model isn't fed with the state, it's fed with the features
 of the state and is not going to output the next state. So we need an encoder that takes
 a state and is fed with the features of the state and is not going to output the next
 state as such but the features of the next state. It predicts the features and then we're
 going to compare that with the features of the true next state and that's what we compare.
 So how does this encoder, these features need to look? And they're saying well these features
 should kind of only consider things about the state that are actually dependent on our
 actions. And they have a very interesting way of achieving to train such an encoder,
 such a feature producing function in that they say it's going to be a neural network
 that we train by training this so called inverse model. So we take this encoder and we train
 this inverse model on top of it and the inverse model takes the features of the last state
 and the new state and is trying to predict this action, this action right here. So this
 is this action, the action we took to get from the old state to the new state. So this
 inverse model is trained to predict what action was taken to get from the old state to the
 new state. And by training the encoder with this inverse model, like training this end
 to end, you will make the encoder such that it only considers things that are actually
 relevant to predicting this action. So in the leaves example it would discard the leaves.
 It will discard anything that you can't influence with your action and therefore it will only
 retain features that are dependent on your action. I think that's quite an interesting
 way to get rid of the irrelevant information that they don't want. And then they can use
 this encoder to train this forward model and to essentially get information from the old
 model and to essentially get this intrinsic reward. So I find this idea quite interesting
 and as I said the idea of intrinsic reward and curiosity to go for exploration is not
 new, but I think this kind of approach and I'm sure it's been around in some variants,
 but I've just stumbled across this and this is quite interesting. So we're going to take
 a look, and you can go about the math yourself, but they do these kind of experiments and
 they corrupt, as you can see, part of the screen with noise here and they of course
 show like, okay, since the noise is not dependent on our action, our features do actually discard
 this noise, only focus on the part that we can actually influence by our actions. So
 that's, I think, all in all pretty interesting. They show, of course, that their algorithm
 then outperforms the kind of baseline of A3C on these sparse reward tasks and the sparser
 here you can see like the left is like dense reward and then sparse reward and then very
 sparse reward and at some point you see the A3C simply doesn't do it anymore. But what's
 also interesting is here you have the ICM in pixels, which kind of means pixel-based
 curiosity, so where we don't have this encoder, where we simply try to predict the pixels
 of the environment and that works if you have like this kind of sparse reward thing, but
 if you want to, if you have the very sparse reward, that also fails and you actually need
 this encoder that discards what's not relevant for predicting the actions. Yeah, so you can
 take a look at the rest of the paper yourself. I find it quite interesting. They analyze
 how their agent explore these mazes and things and they have more experiments on like benchmark
 tasks. So have a look at it and I'll see you next time.