[Classic] Playing Atari with Deep Reinforcement Learning (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper dqn deep q learning deep q networks q learning qlearning rl drl deep rl deep reinforcement learning deepmind david silver atari pong breakout space invaders agent cnn convolutional neural network bellman
#ai #dqn #deepmind

After the initial success of deep neural networks, especially convolutional neural networks on supervised image processing tasks, this paper was the first to demonstrate their applicability to reinforcement learning. Deep Q Networks learn from pixel input to play seven different Atari games and outperform baselines that require hand-crafted features. This paper kicked off the entire field of deep reinforcement learning and positioned DeepMind as one of the leading AI companies in the world.

OUTLINE:
0:00 - Intro & Overview
2:50 - Arcade Learning Environment
4:25 - Deep Reinforcement Learning
9:20 - Deep Q-Learning
26:30 - Experience Replay
32:25 - Network Architecture
33:50 - Experiments
37:45 - Conclusion

Paper: https://arxiv.org/abs/1312.5602

Abstract:
We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.

Authors: Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar (preferred to Patreon): https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there. Today we'll look at playing Atari with deep reinforcement learning by Vladimir Mn. et al. of DeepMind. So this is another one of our series of impactful past papers. This paper right here kicked off an entire revolution in reinforcement learning. Specifically, it sort of started the deep reinforcement learning hype. Before that, reinforcement learning was kind of this weird field of Markov decision processes and so on. Now, I know there were successes and all and stuff was happening. But this really made a lot of waves because it brought the power of deep neural networks to reinforcement learning. And with a pretty simple application of convolutional networks, managed to solve these reinforcement learning games where previous algorithms really couldn't either or were heavily reliant on hand engineered features. So we'll take a look here and what people did back then, what was the state of the art and what they are telling us about that. Kind of set it in relation to today. Alright, if you do like papers like this, commentary like this, share it out, leave a like and tell me in the comments what you think. So let's dive in. They say we present the first deep learning model to successfully learn control policies directly from high dimensional sensory input using reinforcement learning. The model is a convolutional neural network trained with a variant of Q learning whose input is raw pixels and whose output is a value value function estimating future rewards. We apply our method to seven Atari 2600 games from the arcade learning environment with no adjustment of the architecture or learning algorithm. We find that it outperforms all approaches on six of the games and surpasses a human expert on three of them. So there's a lot packed into this. First of all, I wanted to recognize the absolute LaTeX savagery right here. Yeah, you know, it's just just something. I'm not OCD about that kind of stuff. I think we should ditch LaTeX honestly. But so there's a lot of information packed in this abstract right here. So they say this is the first deep learning model to learn control policies. So that's the task of these reinforcement learning algorithms directly from high dimensional sensory input using using reinforcement learning. So what do they mean this are called learning environment if you don't know what it is, it's basically these old games right here, you can kind of emulate them and run them. And the inputs are always sort of the same. So you have one joystick, I believe. So you have kind of this joystick and it can go into various directions like left, right, up, down, and then also the intermediate directions. And then you have a I think also a button that you can push. And that gives you a total of some somewhere around 16 or 20 actions in each of the games. So the good thing about this environment is that the actions are always the same. But of course, they mean different things in different games. So the games here, you know, for example, pong, or this breakout, these games are kind of really low pixel games, as you can see, and they come in form of an image, right. So this is an image, this is like 180 pixels, and this is like 150 pixels. And the task here is to learn a policy, which means which buttons and directions you need to push, depending on the observation right here, these pixels, and achieve the maximum amount of reward. So reward is given in each game differently as well. For example, in this pong game, the reward is every time you score kind of a goal against your opponent, in breakout, you get a reward every time you manage to hit or kill a one of these blocks and so on. So reward is different, but your objective is always to maximize the reward. In a formal framework, you have an agent and an environment. And the environment would always give you an observation, which in this case, the observation is one of these images. And the agent will give back an action. So the action in this case, would be the which button to press or which direction to move the joystick into. And then the environment would get back give back a reward. So the reward is could be, you know, it could be this, you scored a goal. So that's zero most of the time. And sometimes it's one or, or nine. Or it could be like how long you're alive and so on. This is very, very, very variable. So the difficulty of reinforcement learning very often is that these episodes can go for a while. So this whole process here will repeat over time. And the this can go on for hundreds of steps or 1000s of steps until you're done, you know, playing a game like this, like this right here. And the reward can be very sparse. So you might only get a reward at the very end of the game. Sometimes, most often in these games, you get one in between, but still there can be multiple time steps where you don't have a reward. And your task is to figure out which of the actions were the good ones. This is known as the credit assignment problem. And to do the credit assignment problem just from pixels alone, that was unheard of at the time this paper came out. That's why they say we are the first deep learning model to successfully learn the directly from high dimensional sensory inputs. Okay, so the power of deep learning, they argue is that a deep neural network, a convolutional neural network can extract these high level features by itself. However, at this time, people only knew that it could do so for supervised learning, basically for every input of an image, you had a label. And that's how you train these convolutional neural network. Here, it's very different. Here, you will get maybe 1000 of those images. And you'll simply say, well, you got a score of 1100. And somehow you need to figure out which ones of these were the ones that gave you the good score. And how to generalize that. So there are various difficulties here to apply convolutional neural networks to this problem. And they have detail right here, how they did it. So they say the model is a convolutional neural network, which you know, have been demonstrated. So this is after things like Alex net, though before resnet trained with a variant of Q learning, whose input is raw pixels, and whose output is a value function estimating the future. So we'll get into Q learning, Q learning is a reinforcement learning algorithm that has been around, you know, for a long time, but just not combined with deep neural networks. And yeah, then they say, the last cool thing here is that they apply them to seven games with no adjustment of the architecture or learning algorithm. So that they apply the same algorithm, the same hyper parameters to all of the seven games. And the model learns all of the seven games, not as a single model, but as seven different models. However, they all have the same hyper parameters. So they don't need to, to tune them, which is an additional benefit, like this would have been a cool paper, even if they had to sort of tune the algorithm to each of the seven games. But you know, they didn't, which makes it even more impressive is kind of their point here that these that this reinforcement, deep reinforcement learning can be some sort of a general learning mechanism. Of course, you know, later, there has been like giant amount of development since this, and people have come up with all kinds of giant architectures and whatever corrections of policy corrections seem to real continuous control. This is, this is most of it is a derivation of this work right here. And this work, it reads surprisingly simple, I have to say, and it's almost like they had little idea what problems were to be tackled later in RL, because it kind of reads like, you know, we have this thing and you can learn general things. So yeah, that was, we're still not done with reinforcement learning. I guess we we've just started. Okay, so in a bit of more formal setting, what you will have in reinforcement learning are these rewards, right? So at each time step, you perform an action, you're in a state. So the we said you get back an observation, and we call the observation the state. Now, these two things aren't a sorry, these two things aren't entirely the same thing. So the observation is what you get back directly from the environment. And then the state, it can be something more like if you remember something from the last observation, that can be part of your state, the state is basically what you base your decision on. And the observation is the pure thing you get from the environment. Now, in this case, they do some processing, but essentially, we'll, we'll regard them as the same thing. So the state is what you see of the environment. Then in each of these steps, you perform an action, we'll call that this a thing right here. And in each step, you also get a reward for the last action that you took. And the reward is going to be lowercase r. Now, what you want to do formally is you want to maximize the the reward that you get in time t. Sorry, you want to, that's the reward you get at time t prime, you what you want to do, if you play an episode, you're here, and you perform an action, you go here, you perform an action, you go here, you perform an action, you go here, and then your episode is done. For each action, you'll get a reward reward one reward, two reward three, what you want to do is you want to maximize the total the sum of all of these rewards. So over the course of your episode, you want to collect as much reward as you can. There is a discount factor right here, which is sort of saying that rewards that are very far in the future, they're not as important as rewards right now. However, you can set this to one if you want. So you see you want to maximize the future, the sum of future rewards, which is this thing right here. Okay. So how do you how do you do this? There are two main methods in reinforcement learning. The the first one is called a policy gradient method. And very briefly, a policy, we'll call it pi takes in a state s, and it gives you back an action. Okay. And I mean, this is the same for for Q learning, but in a policy gradient method, which is not this paper, but it's like a little bit easier to understand, I believe, we'll simply say, well, we'll simply train a neural network to do that, right. And there is this, there's this policy gradient trick where you can back propagate even though the reward isn't back propagatable, and so on. You can simply say we'll learn a neural network to give us the action that's best, right. So we'll have a neural network, the state goes in neural network, and then we'll just have as many outputs as there are actions like action one, action two, action three, action four, and we'll treat it as a classification problem, right. So you simply train the network to pick the action that is best in this case, and you are, regardless of how you know which action is best right now. In Q learning, you do something else, namely, you train this thing called the Q function. The Q function is a function that takes in a state and an action, and it gives you what the reward is going to be in the future if you are in this state and perform this action, okay. So you are in a given state, right, and you have three actions at your proposal, okay. You have action one, action two, and action three, and you are in state S. What you would do is you would call the Q function, if you had a perfect Q function, it could give you the reward. You would call the Q function three times. You would call the Q function first with A1, say what's Q of S and A1, and the Q function would maybe say that's seven, and here you say what's the Q function of S and A2, and it would maybe say that's four, and here the same thing for A3, and it maybe say that's one. Then you would know, aha, if I take action one, my reward from not only the reward for this step, but my reward from here on until the end of the episode is going to be seven. That is if you had a perfect Q function. Now the Q function is always of course conditioned on a policy right here, so there's what it basically says if I take action A1 right now, and after that I follow policy pi, then I'm going to get the reward of seven. It's a bit of a multi-layered reasoning approach, but ultimately you don't have to worry much about this being conditioned on a policy. Ultimately the Q function says if you take this action right now, what will the reward be for the entire rest of the episode? So if you had a perfect Q function, you could simply ask it about all the actions as we did here, and then pick the action with the highest number. Then you're guaranteed, because there could be a situation where your reward in a single step is going to be very high here, like a hundred, and here is zero, zero. You would be tempted to take that action right here, but after that it's just going to be zero, zero, zero, so your total reward is going to be 100. But here, even though it's zero now, it could be that after that it's 50, and then 40, and then 2000, and so on. So your total reward is going to be much, much more. If you were simply to train a function that tells you what's the reward in the next step, then you would lose, because that function would not be able to look ahead sufficiently. What we're trying to do with the Q function is we're trying to predict to train a function that will tell us not only what's the reward in the next step, but what is the reward in all the steps to come from here. Of course, conditioned on all the decisions we make in the future, but that's this policy pi right here. I hope it's somewhat clear what a Q function is. Interestingly, we can take the same network architecture for this. So what you would do naively is you would build a neural network where you say, okay, a Q function takes a state and an action, so I'll put those into a neural network, and then out comes this estimation of the reward, which we usually call Q. So the value Q is this estimation of the future reward if you take this action in this state. Now, the disadvantage here is that we have to call this neural network once for each action in every state that we're in. So that's like, if there's 10 actions, that's like 10 forward passes. What we could do is we could simply take the same neural network we had for our initial or very initial policy method, and we use that and we simply input state and we'll train it to output the Q for action one and the Q sorry for the state s and action one, the Q for the state s and action two, and so on. So there is going to be this kind of shared encoder. And then that's, it's basically going to encode the state into a latent space and then classify for each of the actions how valuable this particular action would be in that state. So the this here is called a deep Q network. Okay, it's a network that takes in a state and gives you back the Q value. Now the problem right here is we, you know, here we said if we had a perfect Q function, a Q function that was always right, then the problem would be solved because we could just ask the Q function what to do. Of course, we don't have a perfect Q function, we need to train it. So how do we train a Q function? And the answer is surprisingly simple. So what you want to do is you want you are in this you're in this state. And you want to estimate right what your Q value is you want to train your Q network, what you can do is you can simply play an episode according to the Q function you have, and you'll maybe play this episode right here, right? Like you go here and you collect all of this reward. So this entire thing now goes into your data set. And then you have a sample, you know, I was here in this state s I took action one, and I got in total 2090 as a reward. So that is going to be your labeled sample, right? Your labeled sample is going to be s, I was in s, I did a one. And now I have I then got 2090 reward. Cool. And into the next episode, so you're going on playing, and you maybe go down here, and then you get a next training example, I was in state s, I so you keep restarting the episode, so you can get into the same state multiple times, I performed a three, and I got only 100 reward. So that's another training sample. So these training samples right here, you can use to train your Q function. This is called online reinforcement learning, you play the game at the same time as you train your neural network. And you use that improved neural network to play more games. And with time, there is this well known, there is a there are theorems around Q learning that say if you do that iteratively, then your Q function will converge to the optimal Q functions under some assumptions, which of course not given if this is a deep neural network, but you know, who cares? Yeah, so formally, your Q function, as you can see right here, is going to be there is this Bellman recurrence kind of recurrence property of the Q function. So if I am if I am in a state s, and I'm wondering, what is my Q of my state s and my and an action a. And I said with respect to a policy, which the star policy is going to be the policy where we always select the highest Q function. So we'll basically say, we're in state s, we select action a, and after that, we'll just always select whatever the highest scoring action is right like right now, action a might not be the highest scoring, but we'll take a right now. And after that, the highest scoring, that's Q star, it's a Q function conditioned on the policy where after we perform the first action, which is a will take always the best one according to the Q function. Right, that's right here. So we're in state s, we perform a, and s prime is going to be the state that we are going to. So we're in s, we perform a, we get to s prime. So in s prime, which is a function of your environment, we're always going to take the maximum action, and r is going to be the reward of the next step. So you can see this recurrence equation right here that Q star can be framed in terms of Q star. So the Q star of this state is going to depend on the Q star of the next state. And you can use that fact and you can, you know, prove that pretty, we've already done it, basically, you can use that fact to now train your neural network. So your neural network loss function is going to be the following. It's going to say, look, this here is the Q function for state s and action a, that's my, and this is my neural network telling me how much that's worth. And this is the label, right? So here you have to think in terms of back classic supervised learning, this here is going to be your F of X, and this here is going to be your Y, and we'll take the squared loss between the two, except your input X is going to be which state am I in and which action am I taking, and your label is going to be bootstrapped by your own Q function. So your label is going to be the reward you got. Remember, this comes from a replay buffer, we already played that game. And we already know what happened after we performed this action, right? And what happened is we got this reward, and we got into this state. So we can simply ask our own Q function again, what's the best action to take in this state? And what reward would we get? And then we have our label, right? So our label Y is going to be, yeah, I was I was I was pretty confused when I learned this the first time. So I'm going to assume some of you are confused as well. So your Q function is supposed to tell you what's going to be the reward from here until the end of the episode, okay? That you can decompose in the reward that you get from this very next action plus the sum from then, so t plus one, until the end of the episode, okay? So t prime, so that's t prime equals t plus one. All right, so pretty simple. The total reward from now until the end, you can decompose in the reward now plus the reward after that until the end. Now, this here, we know we've played the episode, we know what happened. This here, we can simply ask our Q function again, because we also know what state we got into. And this, as you can see, is very much this, but just one step later. So we can simply ask our own Q function, which might be imperfect, right? But it's certainly a good guess. We say, okay, this reward from now should be equal to the reward we got plus whatever reward we get later. And yes, you might be astounded by the fact that we are using our own neural network, though be with the parameters one time step ago, in order to produce our label. But that is exactly what these Q learning theorems are about. They basically say under some assumptions, if you do this, and you iterate, then this will converge to the optimal Q function. So as you can see right here, this is the this is the gradient of the loss. It's astounding that back then, they still wrote down the gradient of the loss, like almost no one does this. Now, you just say, put this into TensorFlow and go. Yeah, so they make some remarks here, namely that this algorithm is model free, right? There's no model of the environment, you simply learn a function that for each state tells you the Q value for each action. That's, that's all everything, everything that all the logic needs to be within the neural network itself. So that's pretty cool. And they say it's also off policy, it learns about the greedy strategy while following a behavior distribution that ensures adequate exploration of the state space. So while while training, they do this epsilon greedy strategy that follows the greedy strategy, which is where you always take the maximum with one minus epsilon selects a random action with probability epsilon. So while you do your experience, you follow your Q function, you always ask the Q function, what's the best thing to do right here. But you know, that's, that gets you into too much of exploitation. So in epsilon amount of time, you want to do a bit of exploration and just take a random action. Alright, so that's basically the algorithm. So the algorithm is right here. And they have some tricks to get it to work. And the biggest trick they got it to work is the so called replay buffer, this experience replay, because what happens if you play a game of Atari, right, of pong, specifically, then, you know, you have this and you're here and your opponent is here and the ball is here. And then the next frame, you are here again, your opponent might be a bit up, and the ball is here. Okay, and so on. So these samples here, they are all very, very correlated, right, the ones after another, especially if you now build mini batch, let's say, or mini batch sizes to this mini batch has almost no variability in it. So if you've had something like batch norm or whatnot, this, this will be like terrible, because these data samples are correlated. And we in supervised learning, we make a big, pretty big deal out of, you know, shuffling our data set and all of the data points being ID and so on. So what they say is, rather than using the data samples, as we collect them, we put them into a big, big buffer, a big replay buffer. And from that replay buffer, we basically sample at random. Okay, so that means that, you know, some samples can be used multiple times, other samples can be never sampled, because there is a fixed size, and the new ones will always kick out the oldest ones. So some samples might not be used, some samples might be used twice or three times, we can also learn, you know, four times as fast as we sample, and then every sample on average will be used four times. So this, this experience replay proved very, very important for this algorithm to work. That's why they say deep Q learning with experience replay. So they have this replay memory D right here to capacity n. And you initialize your Q function with random weights as you do with a neural network. And then you play these episodes for each episode, you start out with s one, the state one, and you do pre processing. So in pre process, they have some more tricks where they downscale the image, they concatenate four images in a row, because sometimes in Atari get these flicker things. And also, if you concatenate four things in a row, you, for example, can tell it in which direction the ball is moving, and so on. So give a little bit of history. So one sample technically would be four frames, they also do sticky actions, and so on all of these things that you can find today in these emulators that are almost default now, like sticky actions, they invented right here. So for the time steps within the episode, we want to we've probability epsilon select a random action. Otherwise, just ask your Q function, what should I do right here? Give me the best action in this particular state, then you would execute that action and observe a reward and the next state. So the next image right here, you would set the next state to this transition. Okay, so in the state, there can be more, as I said, there can be more than the image like the previous state, and the action you took, but right here, I believe it's like purely the current last four frames. And then you store that transition in the replay buffer. After that, you sample a random mini batch of transitions from the replay buffer. So here, you can see this here is where we de correlate the inputs, because if we simply were to use our last transition for learning, then we would run into a problem. But right here, we sample from that replay buffer. So this is going to be your input, this is going to be your X for your supervised learning of the deep neural network, what's going to be your sorry, without the reward, of course, what's going to be your y, your y, if you're at the end of the episode, it's simply the reward that you got, because there's no more reward coming. However, if you're not at the end, it's the reward that you got from this last step, plus all of the reward that you're going to get in the future. Now, you aren't in the future yet, but you can ask yourself, you can ask your Q function, what that reward is most likely going to be. If your Q function gets better, and this estimate gets better, and your labels get better, then your Q function gets better, and so on in a big circle. And then you perform a gradient descent step on this L2 loss between the label and your prediction. Note that there, if you are in a deep learning framework, there is like a stop gradient on this label right here. So the back propagation only happens with respect to this right here, which makes sense, right? So this is your X, this is your input, and f of X is usually what we back propagate into. Okay, there's no notion yet of like a second Q network and so on, which proved very valuable in the future of this paper. This paper simply applied kind of the most basic version of this, and they simply got it to work. They just got deep neural networks to work with reinforcement learning. And yeah, there's a big chance that this was due to this experience replay, which I believe they did not invent. I mean, this has, of course, been around before, but they were the ones to realize and combine and do that. It's also pretty interesting, the neural network that they actually used was like super duper small. The input to the neural network consists of 84 by 84 by 4 image produced by this. So this is the pre-processing. The first hidden layer convolves 16 8 by 8 filters with stride 4 with the input image and applies the rectifier non-linearity. So the ReLU. The second hidden layer convolves 32 4 by 4 filters with stride 2, again followed by rectifier non-linearity. The final layer hidden layer is fully connected and consists of 256 rectifier units. The output layer is a fully connected linear layer with single output for each valid action. Number of valid actions is vary between 4 and 18 on the games we considered. Okay, as you can see that neural network is pretty small, it's two conv layers. And as was in fashion back then, you had like big filters. So you know big filters from like Alex net. Big filters, but fewer than today. So today, the trend is more like deeper layers, more filters, but they are not as big. They're like three by three filters today only. Yeah, pretty interesting how they did it back then. Interesting also no max pooling and so on. So pretty cool. And here they go into experiments. So they show that their average reward in these games is kind of noisy, but it improves over time, especially also if you look at the average queue of the max action, it continuously goes up during training. So this is really a successful training, especially this investigative experiment they did right here, where you can see one example of how the queue function, what the queue function says. Remember, the queue function gives us the whatever the future reward is going to be. Okay. And here we always look at the max action. So in this first frame, you can see this enemy had just appeared. And you can see that from here to here, there's a spike in the queue value because you can shoot enemies. And that gives you reward. The A this is already so the enemy isn't shot yet by the simple appearance of the enemy, the queue function also like already jumps in value, because it anticipates a future reward, right, then the the agent shoots. And you can see here the shot is about to land at the enemy. And that's when we're here. So this now the queue function is very sure that in the future, there's going to be a high reward. But then once the once the enemy is shot, then there is no more enemy to be shot. And the queue function drops drastically, because it doesn't see a future reward as being as likely as at the beginning when there was this new enemy to be shot. So that's, you know, pretty interesting. And you can see pretty directly that there is a correlation between what's happening in the game and this learned queue function. If you compare this to other methods, and they really say that these other methods, most of them have some kind of very special feature engineered, like, so their method just takes RGB, but the other methods recognize that, oh, in these Atari games, most of the time, you know, there are unique colors for the things. So you know, the enemies are all like green, like, and they make unique channels for those green enemies, or they even have handcrafted object detectors, and tell the algorithm where these objects are. So the comparison really isn't fair. Yet, the DQN outperform these others like almost everywhere. And they also evaluated against a against a human. And I don't actually know they just say an expert human. I have no idea. Maybe just put David Silver in front of computers like, okay, David, here you go. And you can you can, like what happened in Pong? Like, come on, David. But you can see there, there were still problems where the humans were vastly superior. And they mainly attribute this to the difficulty of the problem. And it could also be because for example, in breakout, there's this this kind of the most famous example, where the agent kind of figured out this strategy of shooting the ball, shooting like a hole into this wall that you have to break, and then shooting the ball up here. So the ball bounces up and down. And basically, you win. From then on, you just watch the ball go. And the agent does nothing anymore. So this deep Q networks figured out that strategy, and you need to pull it off very precisely, which of course, the the computer can do very well. So it sometimes achieves these super high scores by pulling something off precisely. But in games where they say where you have to plan ahead for longer, it it kind of fails. And we know that this long planning was about to be a problem for years to come. And it's still not solved. So still, go explore is highly controversial that can solve these kind of long exploration games. And those are still games, right? So we are basically not we are very much further than they were in this paper. But also, we are basically no nowhere yet. Yeah, if I'm if I'm allowed to say that. So I enjoyed reading this paper, I this is it's very it's very well written. If you somehow know how to think about reinforcement learning, like this, this Q function, what the Q function means, and why you would learn it in this way. I find this is not super well described, this kind of requires a bit of a knowledge of not of RL, but just of how to think of RL. But apart from this, everything else is written incredibly well, easy, straightforward. And you know, this was just a nice work of its time. And I appreciate it for that. Alright, I'll see you next time. And I appreciate your time too. Bye.
 Hi there. Today we'll look at playing Atari with deep reinforcement learning by Vladimir
 Mn. et al. of DeepMind. So this is another one of our series of impactful past papers.
 This paper right here kicked off an entire revolution in reinforcement learning. Specifically,
 it sort of started the deep reinforcement learning hype. Before that, reinforcement
 learning was kind of this weird field of Markov decision processes and so on. Now, I know there
 were successes and all and stuff was happening. But this really made a lot of waves because it
 brought the power of deep neural networks to reinforcement learning. And with a pretty simple
 application of convolutional networks, managed to solve these reinforcement learning games where
 previous algorithms really couldn't either or were heavily reliant on hand engineered features.
 So we'll take a look here and what people did back then, what was the state of the art and
 what they are telling us about that. Kind of set it in relation to today. Alright, if you do like
 papers like this, commentary like this, share it out, leave a like and tell me in the comments what
 you think. So let's dive in. They say we present the first deep learning model to successfully learn
 control policies directly from high dimensional sensory input using reinforcement learning. The
 model is a convolutional neural network trained with a variant of Q learning whose input is raw
 pixels and whose output is a value value function estimating future rewards. We apply our method to
 seven Atari 2600 games from the arcade learning environment with no adjustment of the architecture
 or learning algorithm. We find that it outperforms all approaches on six of the games and surpasses a
 human expert on three of them. So there's a lot packed into this. First of all, I wanted to
 recognize the absolute LaTeX savagery right here. Yeah, you know, it's just just something. I'm not
 OCD about that kind of stuff. I think we should ditch LaTeX honestly. But so there's a lot of
 information packed in this abstract right here. So they say this is the first deep learning model
 to learn control policies. So that's the task of these reinforcement learning algorithms directly
 from high dimensional sensory input using using reinforcement learning. So what do they mean this
 are called learning environment if you don't know what it is, it's basically these old games right
 here, you can kind of emulate them and run them. And the inputs are always sort of the same. So you
 have one joystick, I believe. So you have kind of this joystick and it can go into various directions
 like left, right, up, down, and then also the intermediate directions. And then you have a I
 think also a button that you can push. And that gives you a total of some somewhere around 16 or
 20 actions in each of the games. So the good thing about this environment is that the actions are
 always the same. But of course, they mean different things in different games. So the games here,
 you know, for example, pong, or this breakout, these games are kind of really low pixel games,
 as you can see, and they come in form of an image, right. So this is an image, this is like 180
 pixels, and this is like 150 pixels. And the task here is to learn a policy, which means which
 buttons and directions you need to push, depending on the observation right here, these pixels,
 and achieve the maximum amount of reward. So reward is given in each game differently as well. For
 example, in this pong game, the reward is every time you score kind of a goal against your opponent,
 in breakout, you get a reward every time you manage to hit or kill a one of these blocks and
 so on. So reward is different, but your objective is always to maximize the reward. In a formal
 framework, you have an agent and an environment. And the environment would always give you an
 observation, which in this case, the observation is one of these images. And the agent will give
 back an action. So the action in this case, would be the which button to press or which direction to
 move the joystick into. And then the environment would get back give back a reward. So the reward
 is could be, you know, it could be this, you scored a goal. So that's zero most of the time. And
 sometimes it's one or, or nine. Or it could be like how long you're alive and so on. This is very,
 very, very variable. So the difficulty of reinforcement learning very often is that these
 episodes can go for a while. So this whole process here will repeat over time. And the this can go
 on for hundreds of steps or 1000s of steps until you're done, you know, playing a game like this,
 like this right here. And the reward can be very sparse. So you might only get a reward at the very
 end of the game. Sometimes, most often in these games, you get one in between, but still there
 can be multiple time steps where you don't have a reward. And your task is to figure out which of
 the actions were the good ones. This is known as the credit assignment problem. And to do the credit
 assignment problem just from pixels alone, that was unheard of at the time this paper came out.
 That's why they say we are the first deep learning model to successfully learn the directly from high
 dimensional sensory inputs. Okay, so the power of deep learning, they argue is that a deep neural
 network, a convolutional neural network can extract these high level features by itself. However,
 at this time, people only knew that it could do so for supervised learning, basically for every
 input of an image, you had a label. And that's how you train these convolutional neural network.
 Here, it's very different. Here, you will get maybe 1000 of those images. And you'll simply say,
 well, you got a score of 1100. And somehow you need to figure out which ones of these were the
 ones that gave you the good score. And how to generalize that. So there are various difficulties
 here to apply convolutional neural networks to this problem. And they have detail right here,
 how they did it. So they say the model is a convolutional neural network, which you know,
 have been demonstrated. So this is after things like Alex net, though before resnet trained with
 a variant of Q learning, whose input is raw pixels, and whose output is a value function
 estimating the future. So we'll get into Q learning, Q learning is a reinforcement learning
 algorithm that has been around, you know, for a long time, but just not combined with deep neural
 networks. And yeah, then they say, the last cool thing here is that they apply them to seven games
 with no adjustment of the architecture or learning algorithm. So that they apply the same algorithm,
 the same hyper parameters to all of the seven games. And the model learns all of the seven
 games, not as a single model, but as seven different models. However, they all have the
 same hyper parameters. So they don't need to, to tune them, which is an additional benefit,
 like this would have been a cool paper, even if they had to sort of tune the algorithm to each
 of the seven games. But you know, they didn't, which makes it even more impressive is kind of
 their point here that these that this reinforcement, deep reinforcement learning can be some sort of a
 general learning mechanism. Of course, you know, later, there has been like giant amount of
 development since this, and people have come up with all kinds of giant architectures and whatever
 corrections of policy corrections seem to real continuous control. This is, this is most of it
 is a derivation of this work right here. And this work, it reads surprisingly simple, I have to say,
 and it's almost like they had little idea what problems were to be tackled later in RL, because
 it kind of reads like, you know, we have this thing and you can learn general things. So yeah,
 that was, we're still not done with reinforcement learning. I guess we we've just started. Okay,
 so in a bit of more formal setting, what you will have in reinforcement learning are these rewards,
 right? So at each time step, you perform an action, you're in a state. So the we said you
 get back an observation, and we call the observation the state. Now, these two things aren't a sorry,
 these two things aren't entirely the same thing. So the observation is what you get back directly
 from the environment. And then the state, it can be something more like if you remember something
 from the last observation, that can be part of your state, the state is basically what you base
 your decision on. And the observation is the pure thing you get from the environment. Now,
 in this case, they do some processing, but essentially, we'll, we'll regard them as the
 same thing. So the state is what you see of the environment. Then in each of these steps,
 you perform an action, we'll call that this a thing right here. And in each step, you also
 get a reward for the last action that you took. And the reward is going to be lowercase r. Now,
 what you want to do formally is you want to maximize the the reward that you get in time t.
 Sorry, you want to, that's the reward you get at time t prime, you what you want to do, if you play
 an episode, you're here, and you perform an action, you go here, you perform an action,
 you go here, you perform an action, you go here, and then your episode is done. For each action,
 you'll get a reward reward one reward, two reward three, what you want to do is you want to maximize
 the total the sum of all of these rewards. So over the course of your episode, you want to collect
 as much reward as you can. There is a discount factor right here, which is sort of saying that
 rewards that are very far in the future, they're not as important as rewards right now. However,
 you can set this to one if you want. So you see you want to maximize the future, the sum of future
 rewards, which is this thing right here. Okay. So how do you how do you do this? There are two main
 methods in reinforcement learning. The the first one is called a policy gradient method. And very
 briefly, a policy, we'll call it pi takes in a state s, and it gives you back an action. Okay.
 And I mean, this is the same for for Q learning, but in a policy gradient method, which is not
 this paper, but it's like a little bit easier to understand, I believe, we'll simply say, well,
 we'll simply train a neural network to do that, right. And there is this, there's this policy
 gradient trick where you can back propagate even though the reward isn't back propagatable, and so
 on. You can simply say we'll learn a neural network to give us the action that's best, right. So we'll
 have a neural network, the state goes in neural network, and then we'll just have as many outputs
 as there are actions like action one, action two, action three, action four, and we'll treat it as a
 classification problem, right. So you simply train the network to pick the action that is best in
 this case, and you are, regardless of how you know which action is best right now. In Q learning,
 you do something else, namely, you train this thing called the Q function. The Q function is
 a function that takes in a state and an action, and it gives you what the reward is going to be in
 the future if you are in this state and perform this action, okay. So you are in a given state,
 right, and you have three actions at your proposal, okay. You have action one, action two,
 and action three, and you are in state S. What you would do is you would call the Q function,
 if you had a perfect Q function, it could give you the reward. You would call the Q function three
 times. You would call the Q function first with A1, say what's Q of S and A1, and the Q function
 would maybe say that's seven, and here you say what's the Q function of S and A2, and it would
 maybe say that's four, and here the same thing for A3, and it maybe say that's one. Then you would
 know, aha, if I take action one, my reward from not only the reward for this step, but my reward
 from here on until the end of the episode is going to be seven. That is if you had a perfect Q function.
 Now the Q function is always of course conditioned on a policy right here, so there's what it basically
 says if I take action A1 right now, and after that I follow policy pi, then I'm going to get the reward
 of seven. It's a bit of a multi-layered reasoning approach, but ultimately you don't have to
 worry much about this being conditioned on a policy. Ultimately the Q function says if you take
 this action right now, what will the reward be for the entire rest of the episode? So if you had a
 perfect Q function, you could simply ask it about all the actions as we did here, and then pick the
 action with the highest number. Then you're guaranteed, because there could be a situation
 where your reward in a single step is going to be very high here, like a hundred, and here is zero,
 zero. You would be tempted to take that action right here, but after that it's just going to be
 zero, zero, zero, so your total reward is going to be 100. But here, even though it's zero now, it
 could be that after that it's 50, and then 40, and then 2000, and so on. So your total reward is going
 to be much, much more. If you were simply to train a function that tells you what's the reward in the
 next step, then you would lose, because that function would not be able to look ahead sufficiently.
 What we're trying to do with the Q function is we're trying to predict to train a function that
 will tell us not only what's the reward in the next step, but what is the reward in all the steps
 to come from here. Of course, conditioned on all the decisions we make in the future, but that's
 this policy pi right here. I hope it's somewhat clear what a Q function is. Interestingly, we can
 take the same network architecture for this. So what you would do naively is you would build a
 neural network where you say, okay, a Q function takes a state and an action, so I'll put those
 into a neural network, and then out comes this estimation of the reward, which we usually call
 Q. So the value Q is this estimation of the future reward if you take this action in this state. Now,
 the disadvantage here is that we have to call this neural network once for each action in every state
 that we're in. So that's like, if there's 10 actions, that's like 10 forward passes. What we
 could do is we could simply take the same neural network we had for our initial or very initial
 policy method, and we use that and we simply input state and we'll train it to output the Q for
 action one and the Q sorry for the state s and action one, the Q for the state s and action two,
 and so on. So there is going to be this kind of shared encoder. And then that's, it's basically
 going to encode the state into a latent space and then classify for each of the actions how
 valuable this particular action would be in that state. So the this here is called a deep Q network.
 Okay, it's a network that takes in a state and gives you back the Q value. Now the problem right
 here is we, you know, here we said if we had a perfect Q function, a Q function that was always
 right, then the problem would be solved because we could just ask the Q function what to do. Of
 course, we don't have a perfect Q function, we need to train it. So how do we train a Q function?
 And the answer is surprisingly simple. So what you want to do is you want you are in this you're in
 this state. And you want to estimate right what your Q value is you want to train your Q network,
 what you can do is you can simply play an episode according to the Q function you have, and you'll
 maybe play this episode right here, right? Like you go here and you collect all of this reward. So
 this entire thing now goes into your data set. And then you have a sample, you know, I was here in
 this state s I took action one, and I got in total 2090 as a reward. So that is going to be your
 labeled sample, right? Your labeled sample is going to be s, I was in s, I did a one. And now I
 have I then got 2090 reward. Cool. And into the next episode, so you're going on playing, and you
 maybe go down here, and then you get a next training example, I was in state s, I so you keep
 restarting the episode, so you can get into the same state multiple times, I performed a three,
 and I got only 100 reward. So that's another training sample. So these training samples right
 here, you can use to train your Q function. This is called online reinforcement learning, you play
 the game at the same time as you train your neural network. And you use that improved neural network
 to play more games. And with time, there is this well known, there is a there are theorems around
 Q learning that say if you do that iteratively, then your Q function will converge to the optimal
 Q functions under some assumptions, which of course not given if this is a deep neural network,
 but you know, who cares? Yeah, so formally, your Q function, as you can see right here, is going to
 be there is this Bellman recurrence kind of recurrence property of the Q function. So if I am if I
 am in a state s, and I'm wondering, what is my Q of my state s and my and an action a. And I said
 with respect to a policy, which the star policy is going to be the policy where we always select
 the highest Q function. So we'll basically say, we're in state s, we select action a, and after
 that, we'll just always select whatever the highest scoring action is right like right now, action a
 might not be the highest scoring, but we'll take a right now. And after that, the highest scoring,
 that's Q star, it's a Q function conditioned on the policy where after we perform the first action,
 which is a will take always the best one according to the Q function. Right, that's right here. So
 we're in state s, we perform a, and s prime is going to be the state that we are going to. So
 we're in s, we perform a, we get to s prime. So in s prime, which is a function of your environment,
 we're always going to take the maximum action, and r is going to be the reward of the next step. So
 you can see this recurrence equation right here that Q star can be framed in terms of Q star. So
 the Q star of this state is going to depend on the Q star of the next state. And you can use that
 fact and you can, you know, prove that pretty, we've already done it, basically, you can use
 that fact to now train your neural network. So your neural network loss function is going to be the
 following. It's going to say, look, this here is the Q function for state s and action a, that's my,
 and this is my neural network telling me how much that's worth. And this is the label, right? So here
 you have to think in terms of back classic supervised learning, this here is going to be your F of X,
 and this here is going to be your Y, and we'll take the squared loss between the two, except your
 input X is going to be which state am I in and which action am I taking, and your label is going
 to be bootstrapped by your own Q function. So your label is going to be the reward you got. Remember,
 this comes from a replay buffer, we already played that game. And we already know what happened after
 we performed this action, right? And what happened is we got this reward, and we got into this state.
 So we can simply ask our own Q function again, what's the best action to take in this state? And
 what reward would we get? And then we have our label, right? So our label Y is going to be, yeah,
 I was I was I was pretty confused when I learned this the first time. So I'm going to assume some
 of you are confused as well. So your Q function is supposed to tell you what's going to be the
 reward from here until the end of the episode, okay? That you can decompose in the reward that
 you get from this very next action plus the sum from then, so t plus one, until the end of the
 episode, okay? So t prime, so that's t prime equals t plus one. All right, so pretty simple. The total
 reward from now until the end, you can decompose in the reward now plus the reward after that until
 the end. Now, this here, we know we've played the episode, we know what happened. This here, we can
 simply ask our Q function again, because we also know what state we got into. And this, as you can
 see, is very much this, but just one step later. So we can simply ask our own Q function, which might
 be imperfect, right? But it's certainly a good guess. We say, okay, this reward from now should
 be equal to the reward we got plus whatever reward we get later. And yes, you might be astounded by
 the fact that we are using our own neural network, though be with the parameters one time step ago,
 in order to produce our label. But that is exactly what these Q learning theorems are about. They
 basically say under some assumptions, if you do this, and you iterate, then this will converge to
 the optimal Q function. So as you can see right here, this is the this is the gradient of the loss.
 It's astounding that back then, they still wrote down the gradient of the loss, like almost no one
 does this. Now, you just say, put this into TensorFlow and go. Yeah, so they make some remarks
 here, namely that this algorithm is model free, right? There's no model of the environment, you
 simply learn a function that for each state tells you the Q value for each action. That's, that's
 all everything, everything that all the logic needs to be within the neural network itself. So that's
 pretty cool. And they say it's also off policy, it learns about the greedy strategy while following
 a behavior distribution that ensures adequate exploration of the state space. So while while
 training, they do this epsilon greedy strategy that follows the greedy strategy, which is where
 you always take the maximum with one minus epsilon selects a random action with probability epsilon.
 So while you do your experience, you follow your Q function, you always ask the Q function, what's
 the best thing to do right here. But you know, that's, that gets you into too much of exploitation.
 So in epsilon amount of time, you want to do a bit of exploration and just take a random action.
 Alright, so that's basically the algorithm. So the algorithm is right here. And they have some tricks
 to get it to work. And the biggest trick they got it to work is the so called replay buffer, this
 experience replay, because what happens if you play a game of Atari, right, of pong, specifically,
 then, you know, you have this and you're here and your opponent is here and the ball is here. And
 then the next frame, you are here again, your opponent might be a bit up, and the ball is here.
 Okay, and so on. So these samples here, they are all very, very correlated, right, the ones after
 another, especially if you now build mini batch, let's say, or mini batch sizes to this mini batch
 has almost no variability in it. So if you've had something like batch norm or whatnot, this, this
 will be like terrible, because these data samples are correlated. And we in supervised learning,
 we make a big, pretty big deal out of, you know, shuffling our data set and all of the data points
 being ID and so on. So what they say is, rather than using the data samples, as we collect them,
 we put them into a big, big buffer, a big replay buffer. And from that replay buffer, we basically
 sample at random. Okay, so that means that, you know, some samples can be used multiple times,
 other samples can be never sampled, because there is a fixed size, and the new ones will always kick
 out the oldest ones. So some samples might not be used, some samples might be used twice or three
 times, we can also learn, you know, four times as fast as we sample, and then every sample on
 average will be used four times. So this, this experience replay proved very, very important for
 this algorithm to work. That's why they say deep Q learning with experience replay. So they have
 this replay memory D right here to capacity n. And you initialize your Q function with random
 weights as you do with a neural network. And then you play these episodes for each episode,
 you start out with s one, the state one, and you do pre processing. So in pre process, they have
 some more tricks where they downscale the image, they concatenate four images in a row, because
 sometimes in Atari get these flicker things. And also, if you concatenate four things in a row,
 you, for example, can tell it in which direction the ball is moving, and so on. So give a little
 bit of history. So one sample technically would be four frames, they also do sticky actions, and so
 on all of these things that you can find today in these emulators that are almost default now,
 like sticky actions, they invented right here. So for the time steps within the episode, we want to
 we've probability epsilon select a random action. Otherwise, just ask your Q function, what should I
 do right here? Give me the best action in this particular state, then you would execute that
 action and observe a reward and the next state. So the next image right here, you would set the
 next state to this transition. Okay, so in the state, there can be more, as I said, there can be
 more than the image like the previous state, and the action you took, but right here, I believe it's
 like purely the current last four frames. And then you store that transition in the replay buffer.
 After that, you sample a random mini batch of transitions from the replay buffer. So here,
 you can see this here is where we de correlate the inputs, because if we simply were to use our last
 transition for learning, then we would run into a problem. But right here, we sample from that
 replay buffer. So this is going to be your input, this is going to be your X for your supervised
 learning of the deep neural network, what's going to be your sorry, without the reward, of course,
 what's going to be your y, your y, if you're at the end of the episode, it's simply the reward
 that you got, because there's no more reward coming. However, if you're not at the end,
 it's the reward that you got from this last step, plus all of the reward that you're going to get
 in the future. Now, you aren't in the future yet, but you can ask yourself, you can ask your Q
 function, what that reward is most likely going to be. If your Q function gets better, and this
 estimate gets better, and your labels get better, then your Q function gets better, and so on in a
 big circle. And then you perform a gradient descent step on this L2 loss between the label and your
 prediction. Note that there, if you are in a deep learning framework, there is like a stop gradient
 on this label right here. So the back propagation only happens with respect to this right here,
 which makes sense, right? So this is your X, this is your input, and f of X is usually what we back
 propagate into. Okay, there's no notion yet of like a second Q network and so on, which proved very
 valuable in the future of this paper. This paper simply applied kind of the most basic version of
 this, and they simply got it to work. They just got deep neural networks to work with reinforcement
 learning. And yeah, there's a big chance that this was due to this experience replay, which I believe
 they did not invent. I mean, this has, of course, been around before, but they were the ones to
 realize and combine and do that. It's also pretty interesting, the neural network that they actually
 used was like super duper small. The input to the neural network consists of 84 by 84 by 4 image
 produced by this. So this is the pre-processing. The first hidden layer convolves 16 8 by 8 filters
 with stride 4 with the input image and applies the rectifier non-linearity. So the ReLU. The
 second hidden layer convolves 32 4 by 4 filters with stride 2, again followed by rectifier
 non-linearity. The final layer hidden layer is fully connected and consists of 256 rectifier
 units. The output layer is a fully connected linear layer with single output for each valid
 action. Number of valid actions is vary between 4 and 18 on the games we considered. Okay, as you
 can see that neural network is pretty small, it's two conv layers. And as was in fashion back then,
 you had like big filters. So you know big filters from like Alex net. Big filters,
 but fewer than today. So today, the trend is more like deeper layers, more filters,
 but they are not as big. They're like three by three filters today only. Yeah, pretty interesting
 how they did it back then. Interesting also no max pooling and so on. So pretty cool. And here
 they go into experiments. So they show that their average reward in these games is kind of noisy,
 but it improves over time, especially also if you look at the average queue of the max action,
 it continuously goes up during training. So this is really a successful training, especially this
 investigative experiment they did right here, where you can see one example of how the queue
 function, what the queue function says. Remember, the queue function gives us the whatever the future
 reward is going to be. Okay. And here we always look at the max action. So in this first frame,
 you can see this enemy had just appeared. And you can see that from here to here, there's a spike in
 the queue value because you can shoot enemies. And that gives you reward. The A this is already so
 the enemy isn't shot yet by the simple appearance of the enemy, the queue function also like already
 jumps in value, because it anticipates a future reward, right, then the the agent shoots. And
 you can see here the shot is about to land at the enemy. And that's when we're here. So this now the
 queue function is very sure that in the future, there's going to be a high reward. But then once
 the once the enemy is shot, then there is no more enemy to be shot. And the queue function drops
 drastically, because it doesn't see a future reward as being as likely as at the beginning when there
 was this new enemy to be shot. So that's, you know, pretty interesting. And you can see pretty
 directly that there is a correlation between what's happening in the game and this learned queue
 function. If you compare this to other methods, and they really say that these other methods,
 most of them have some kind of very special feature engineered, like, so their method just takes RGB,
 but the other methods recognize that, oh, in these Atari games, most of the time, you know,
 there are unique colors for the things. So you know, the enemies are all like green, like, and they
 make unique channels for those green enemies, or they even have handcrafted object detectors,
 and tell the algorithm where these objects are. So the comparison really isn't fair. Yet,
 the DQN outperform these others like almost everywhere. And they also evaluated against a
 against a human. And I don't actually know they just say an expert human. I have no idea. Maybe
 just put David Silver in front of computers like, okay, David, here you go. And you can you can,
 like what happened in Pong? Like, come on, David. But you can see there, there were still problems
 where the humans were vastly superior. And they mainly attribute this to the difficulty of the
 problem. And it could also be because for example, in breakout, there's this this kind of the most
 famous example, where the agent kind of figured out this strategy of shooting the ball, shooting
 like a hole into this wall that you have to break, and then shooting the ball up here. So the ball
 bounces up and down. And basically, you win. From then on, you just watch the ball go. And the agent
 does nothing anymore. So this deep Q networks figured out that strategy, and you need to pull
 it off very precisely, which of course, the the computer can do very well. So it sometimes achieves
 these super high scores by pulling something off precisely. But in games where they say where you
 have to plan ahead for longer, it it kind of fails. And we know that this long planning was about to
 be a problem for years to come. And it's still not solved. So still, go explore is highly controversial
 that can solve these kind of long exploration games. And those are still games, right? So we are
 basically not we are very much further than they were in this paper. But also, we are basically no
 nowhere yet. Yeah, if I'm if I'm allowed to say that. So I enjoyed reading this paper, I
 this is it's very it's very well written. If you somehow know how to think about reinforcement
 learning, like this, this Q function, what the Q function means, and why you would learn it in this
 way. I find this is not super well described, this kind of requires a bit of a knowledge of not of
 RL, but just of how to think of RL. But apart from this, everything else is written incredibly
 well, easy, straightforward. And you know, this was just a nice work of its time. And I appreciate
 it for that. Alright, I'll see you next time. And I appreciate your time too. Bye.