Reinforcement Learning with Unsupervised Auxiliary Tasks
machine learning artificial intelligence ai deep learning unsupervised learning research academia paper review agents tasks
https://arxiv.org/abs/1611.05397

Abstract:
Deep reinforcement learning agents have achieved state-of-the-art results by directly maximising cumulative reward. However, environments contain a much wider variety of possible training signals. In this paper, we introduce an agent that also maximises many other pseudo-reward functions simultaneously by reinforcement learning. All of these tasks share a common representation that, like unsupervised learning, continues to develop in the absence of extrinsic rewards. We also introduce a novel mechanism for focusing this representation upon extrinsic rewards, so that learning can rapidly adapt to the most relevant aspects of the actual task. Our agent significantly outperforms the previous state-of-the-art on Atari, averaging 880\% expert human performance, and a challenging suite of first-person, three-dimensional \emph{Labyrinth} tasks leading to a mean speedup in learning of 10× and averaging 87\% expert human performance on Labyrinth.

Authors:
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, Koray Kavukcuoglu
 Hi there, today we're looking at reinforcement learning with unsupervised auxiliary tasks by Google. So in this paper the authors consider a reinforcement learning task and I can show you what it looks like. It looks like this kind of a maze or this is an example that they give where you have to navigate the maze, it's 3D and you have to navigate from pixel inputs, you have to collect apples and reach the goal and this gives you rewards. So on the left you can see what the agent is actually seeing, on the right you can see it from a top down view. The problem is of course that the input is very, or the reward is very sparse, meaning that you have to navigate a lot of maze before you even get a single point. So reinforcement learning has a big trouble with this because it relies on constant reward to notice what actions are good and what actions are bad. So what the authors propose is in addition to the regular loss that you would have, so your reward which is this thing, you would also have an additional set of auxiliary tasks and here C goes over the auxiliary control tasks that you specify. Each of those has a reward and you're also trying to maximize these each with some kind of a weight here. And the thing is that the parameters that you maximize over control all of the different tasks so they are partly shared between the tasks. So what you're hoping is that by kind of learning to do one thing you also learn to do another thing. So the difference between this and let's say, you might have, so we've seen kind of work of it like this before where you do it more like an autoencoder setting. So for example you can't, the agent sees the input on the left here and it kind of tries to predict what the next input will be, what the next frame will be. The thought behind this is if you can accurately predict what the next frame will be maybe learn something useful about the environment. In this work it's different because now we couple a reward to these tasks and I can show you here what the authors propose as additional rewards. Sorry, they're further on top. Let me go there. Basically they consider here these two auxiliary control tasks. So pixel changes which means that the agent actually tries to actively change pixels. So it gets a reward for changing the pixels in the input. So it tries to maximize this. It needs to learn what do I need to do to maximize my pixel changes and probably that will be moving around. So it will learn to kind of move around, not move against the wall because if it moves against the wall the pixels won't change. So it will kind of learn to move along the, like how a regular human agent would also move not into a wall, not like into a dead end or something such that the pixels always change. Of course it's not perfect. You can also change your pixels quite a bit by simply spinning around in a circle. But this is one auxiliary tasks that they augment the agent with. The other one is network features. So it's kind of a meta learning here. You actually reward the agent for changing its own internal activations. So the hope is that it kind of learns about something by itself. How can I activate my internal neural network units? And it gets rewarded for that. So it might want to activate a lot of them and want to learn how they're activated. So this kind of self-interspection, you also hope that it kind of leads to a network that does more sophisticated tasks or that by nature of trying to get most pixel changes and the most network feature activations that you also learn something useful for the actual task. So these are the two tasks they propose. In addition, they also do, and they have a drawing of this over here. They also do a lot of other things, namely on the top left, you can kind of see here we have a database agent. This is an A3C agent, meaning that it's an actor critic. So you learn a policy and you learn a value network. We might go over this in a future video. So just consider this a standard reinforcement learning agent. You feed its experience into a replay buffer. And out of the replay buffer, you do many things. So for one, you try to learn these auxiliary tasks. Note that these are shared parameters between all of these networks. That's why the auxiliary tasks actually help. But you also try to better learn your value function. They call this off policy learning because you kind of pause the base agent training for a while and then you train the value function some more, just because that helps. You also try a reward prediction from here. And the way they do it, as they explain, is kind of in a skewed sampling way. So out of all the situations you can be in, the agent will have a reward very, very few times. So what they do is they simply sample out of the replay buffer, out of all the experiences they've had so far, they sample more frequently the experiences where they have actually gotten a reward. That way the hope is, of course, the agent, if you look at the experience here where you actually get an apple, then the agent might learn a lot faster, oh, there's some kind of apple there and I move towards it to get a reward. So that's the hope that you instantly recognize high reward situations and kind of are not so interested in non-reward situations. Of course, it does introduce biased near sampling and you might decide for yourself if that's good or bad. But here it seems to work. So they have a lot of experiments in this task, this labyrinth task, and they, of course, as is with research, they reach state of the art, they're much better than anything else. No, I mean they don't boast this much. So it's actually fair comparisons. The criticisms, so they also evaluate on Atari games, the criticisms that I have are twofold. First of all, the choice of auxiliary tasks is, of course, completely up to the implementer, which means that I have to decide as an implementer of this algorithm what my auxiliary task will be. And here, pixel changes and network features, they seem like fairly general tasks that you could apply to a lot of these kind of problems, but it always kind of comes down to how much knowledge about the task would you like to code into the actor. And here, I mean, you can see it makes sense to get at least the pixel changes as an auxiliary task, but it's questionable how much of kind of domain knowledge this already encodes. So the fact, the choice of these are certainly something that you have to decide as a human. And I think these are good choices. So they're not too domain specific, but also they do correspond to like some kind of visual moving around game task. And the other kind of criticisms, not really criticisms, it's just a remark, is that they do a lot of things. So their paper is about the auxiliary tasks, but they also then do these skewed sampling and the off-policy value learning and so on. And of course, you can kind of argue, yeah, this is all done in other reinforcement learning tasks. That's why it's a fair comparison. I guess it's a philosophical question. If you want to reach state of the art, of course, you have to first of all, get a better method here. This will be the auxiliary tasks. This is the new idea. And then implement all the tricks that the other people have discovered, which is good because you kind of reach the highest performance you can get. But also the problem is you make it harder to compare, you make it harder to see where the improvement is coming from. Have you simply chosen better hyperparameters for the reward predictions of things? Is there an interaction maybe between the auxiliary tasks and the skewed sampling part? All of these kind of things wash out and it's not really clear where the improvement is coming from. On the other hand, if you simply take a basic, basic, basic algorithm, like just A3C here on the top left, and you augment it with nothing but these auxiliary tasks on the bottom left, and then you see an improvement, you can be relatively sure it's due to your new idea. But of course, you won't reach any state of the art numbers because everyone that does A3C also does these tricks. No question here. I'm standing more on the side of not doing the tricks or maybe doing both. Yeah, but decide for yourself and have a nice day.
 Hi there, today we're looking at reinforcement learning with unsupervised auxiliary tasks
 by Google.
 So in this paper the authors consider a reinforcement learning task and I can show you what it looks
 like.
 It looks like this kind of a maze or this is an example that they give where you have
 to navigate the maze, it's 3D and you have to navigate from pixel inputs, you have to
 collect apples and reach the goal and this gives you rewards.
 So on the left you can see what the agent is actually seeing, on the right you can see
 it from a top down view.
 The problem is of course that the input is very, or the reward is very sparse, meaning
 that you have to navigate a lot of maze before you even get a single point.
 So reinforcement learning has a big trouble with this because it relies on constant reward
 to notice what actions are good and what actions are bad.
 So what the authors propose is in addition to the regular loss that you would have, so
 your reward which is this thing, you would also have an additional set of auxiliary tasks
 and here C goes over the auxiliary control tasks that you specify.
 Each of those has a reward and you're also trying to maximize these each with some kind
 of a weight here.
 And the thing is that the parameters that you maximize over control all of the different
 tasks so they are partly shared between the tasks.
 So what you're hoping is that by kind of learning to do one thing you also learn to do another
 thing.
 So the difference between this and let's say, you might have, so we've seen kind of work
 of it like this before where you do it more like an autoencoder setting.
 So for example you can't, the agent sees the input on the left here and it kind of tries
 to predict what the next input will be, what the next frame will be.
 The thought behind this is if you can accurately predict what the next frame will be maybe
 learn something useful about the environment.
 In this work it's different because now we couple a reward to these tasks and I can show
 you here what the authors propose as additional rewards.
 Sorry, they're further on top.
 Let me go there.
 Basically they consider here these two auxiliary control tasks.
 So pixel changes which means that the agent actually tries to actively change pixels.
 So it gets a reward for changing the pixels in the input.
 So it tries to maximize this.
 It needs to learn what do I need to do to maximize my pixel changes and probably that
 will be moving around.
 So it will learn to kind of move around, not move against the wall because if it moves
 against the wall the pixels won't change.
 So it will kind of learn to move along the, like how a regular human agent would also
 move not into a wall, not like into a dead end or something such that the pixels always
 change.
 Of course it's not perfect.
 You can also change your pixels quite a bit by simply spinning around in a circle.
 But this is one auxiliary tasks that they augment the agent with.
 The other one is network features.
 So it's kind of a meta learning here.
 You actually reward the agent for changing its own internal activations.
 So the hope is that it kind of learns about something by itself.
 How can I activate my internal neural network units?
 And it gets rewarded for that.
 So it might want to activate a lot of them and want to learn how they're activated.
 So this kind of self-interspection, you also hope that it kind of leads to a network that
 does more sophisticated tasks or that by nature of trying to get most pixel changes and the
 most network feature activations that you also learn something useful for the actual
 task.
 So these are the two tasks they propose.
 In addition, they also do, and they have a drawing of this over here.
 They also do a lot of other things, namely on the top left, you can kind of see here
 we have a database agent.
 This is an A3C agent, meaning that it's an actor critic.
 So you learn a policy and you learn a value network.
 We might go over this in a future video.
 So just consider this a standard reinforcement learning agent.
 You feed its experience into a replay buffer.
 And out of the replay buffer, you do many things.
 So for one, you try to learn these auxiliary tasks.
 Note that these are shared parameters between all of these networks.
 That's why the auxiliary tasks actually help.
 But you also try to better learn your value function.
 They call this off policy learning because you kind of pause the base agent training
 for a while and then you train the value function some more, just because that helps.
 You also try a reward prediction from here.
 And the way they do it, as they explain, is kind of in a skewed sampling way.
 So out of all the situations you can be in, the agent will have a reward very, very few
 times.
 So what they do is they simply sample out of the replay buffer, out of all the experiences
 they've had so far, they sample more frequently the experiences where they have actually gotten
 a reward.
 That way the hope is, of course, the agent, if you look at the experience here where you
 actually get an apple, then the agent might learn a lot faster, oh, there's some kind
 of apple there and I move towards it to get a reward.
 So that's the hope that you instantly recognize high reward situations and kind of are not
 so interested in non-reward situations.
 Of course, it does introduce biased near sampling and you might decide for yourself if that's
 good or bad.
 But here it seems to work.
 So they have a lot of experiments in this task, this labyrinth task, and they, of course,
 as is with research, they reach state of the art, they're much better than anything else.
 No, I mean they don't boast this much.
 So it's actually fair comparisons.
 The criticisms, so they also evaluate on Atari games, the criticisms that I have are twofold.
 First of all, the choice of auxiliary tasks is, of course, completely up to the implementer,
 which means that I have to decide as an implementer of this algorithm what my auxiliary task will
 be.
 And here, pixel changes and network features, they seem like fairly general tasks that you
 could apply to a lot of these kind of problems, but it always kind of comes down to how much
 knowledge about the task would you like to code into the actor.
 And here, I mean, you can see it makes sense to get at least the pixel changes as an auxiliary
 task, but it's questionable how much of kind of domain knowledge this already encodes.
 So the fact, the choice of these are certainly something that you have to decide as a human.
 And I think these are good choices.
 So they're not too domain specific, but also they do correspond to like some kind of visual
 moving around game task.
 And the other kind of criticisms, not really criticisms, it's just a remark, is that they
 do a lot of things.
 So their paper is about the auxiliary tasks, but they also then do these skewed sampling
 and the off-policy value learning and so on.
 And of course, you can kind of argue, yeah, this is all done in other reinforcement learning
 tasks.
 That's why it's a fair comparison.
 I guess it's a philosophical question.
 If you want to reach state of the art, of course, you have to first of all, get a better
 method here.
 This will be the auxiliary tasks.
 This is the new idea.
 And then implement all the tricks that the other people have discovered, which is good
 because you kind of reach the highest performance you can get.
 But also the problem is you make it harder to compare, you make it harder to see where
 the improvement is coming from.
 Have you simply chosen better hyperparameters for the reward predictions of things?
 Is there an interaction maybe between the auxiliary tasks and the skewed sampling part?
 All of these kind of things wash out and it's not really clear where the improvement is
 coming from.
 On the other hand, if you simply take a basic, basic, basic algorithm, like just A3C here
 on the top left, and you augment it with nothing but these auxiliary tasks on the bottom left,
 and then you see an improvement, you can be relatively sure it's due to your new idea.
 But of course, you won't reach any state of the art numbers because everyone that does
 A3C also does these tricks.
 No question here.
 I'm standing more on the side of not doing the tricks or maybe doing both.
 Yeah, but decide for yourself and have a nice day.