Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
deep learning machine learning reinforcement learning vector to go vtg continuous control robot concurrent deep rl deep neural networks berkeley google grasping qlearning
Classic RL "stops" the world whenever the Agent computes a new action. This paper considers a more realistic scenario where the agent is thinking about the next action to take while still performing the last action. This results in a fascinating way of reformulating Q-learning in continuous time, then introducing concurrency and finally going back to discrete time.

https://arxiv.org/abs/2004.06089

Abstract:
We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must "think while moving".

Authors: Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine, Julian Ibarz, Karol Hausman, Alexander Herzog

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there. So if you look at these two robots, the left one labeled blocking, the right one labeled concurrent, the blocking robot, as you can see, always has these little pauses in its movement where it does nothing and then it kind of continues with its motion, while the one on the right is one continuous motion that it does. So the reasoning here is that the robot has a camera and the camera takes some time to register what's going on. And then also the robot has a computer inside and the computer also takes some time to decide what to do based on what the camera saw. And while all of this is happening, the robot on the left just freezes. So it performs an action and then it freezes because it takes time to register a state and compute a new action, whereas the robot on the right, it takes the same amount of time to do these things. It also takes a time to register the state and to compute an action, but it does that as it is executing the last action. So it does this in parallel and then it executes the action. Once it has computed a new action, it executes that new action right on top of the old action. And that gives this one big fluid motion. So this requires a new formulation of reinforcement learning. And that's what this paper does. Thinking while moving deep reinforcement learning with concurrent control by people from Google Brain, UC Berkeley and X. So they have a nice diagram here in the supplementary material to show you what is going on in their framework. So in classic reinforcement learning right here, in classic reinforcement learning, you have this dichotomy between agent and environment, right? So the agent and the environment. Now, the agent is supposed to kind of act in the environment in the following manner. The environment will send an observation to the agent. The observation in this case is the picture of the camera. So it sends an observation and the agent will think what to do with the observation, which is called a policy. The policy pie will take an observation and then output an action of what to do and will send back the action to that environment. So in classic RL, you assume that this part here is kind of freezes time. So the environment will output an observation and as and the process of registering the observation of computing the action and of sending the action back is happens in zero time. Of course, it doesn't happen in zero time, but in our reinforcement learning problems, for example, the OpenAI Gym, the environment just stops until it gets the next action. And then it performs the action, right? It performs the action in the environment and by that the environment changes and time happens. And then it stops again as we think of the next action, right? So this is we usually call this one step in the in the kind of classic formulation of RL. The only point that time happens is when the action is executed. No time happens when the state is registered or when the action is computed. And that's what you see here on the left. So in blue, you have the state registration. This is, for example, the camera. The camera has some time in order to register and store the image that it has taken, right? Maybe post process it a little bit. So that's what the camera does. But in our classic formulation, as you can see here, if this is time, it happens instantaneously all at the same time. And also, this is the policy. This is thinking what to do, right? This is your evaluation of your neural network. If this is a neural network, I'm drawing a small neural network here. This happens instantaneously in these formulations and only as the action is executed, time happens, right? And then until here, time freezes again and only once the action is determined, time happens again. In the new formulation now, as you've already seen, what we have here is this kind of continuous framework where, let's say you're here, it actually takes time for the camera to post process the image. It takes more time for you to think about what to do. And then once you decide on an action, that action is going to happen, right? But we can say, for example, at this point, you tell the camera to take a new picture of the state, right? But that takes time. And while that's happening, the old action is still ongoing, right? So you don't even have to say the action is still ongoing, but the world is still moving, right? The world is still changing while you think, while you post process and while you evaluate your policy. The world is thinking and only after some time, right? After this lag time here, have you decided on a new action? And then you can break that old action and kind of perform your new action. And all of this is happening in time. So this is the new framework. Now you see the problem here. The problem is that you base your decisions on the state and time T on time, sorry, time H here. You base your decisions on a state as it was at that time, right? That's what you use to think right here. That's what you store and think about. But you perform. So you perform the action at this point in time. So there is a considerable difference here because the world has now changed. So you see the problem. The action you perform is based on an old knowledge of the world. And you have basically no way of making the action dependent on the current state of the world, because that would require you to capture the current state and that takes time. And in that time, the world has already shifted again. So the agent is kind of required to think ahead about the action that it is currently performing and how the world changes according to that. So this new formulation of reinforcement learning formulates this in a formal way. It formulates it in a formal way and will quickly go through that. Yes, so they go into the very basics here. We'll quickly, quickly go through them. So they introduce these quantities like the policy pi, the transition distribution, the reward and the Q and value function. Now we'll just quickly go over these. So you have the agent. Sorry about that. You have the agent and you have the environment. Hello. So if you think of the agent and the environment, the environment has this transition function. The transition function, it takes it says, OK, I'm in this state and the agent does this action. And here is the probability distribution over the next state. So it says that your little spaceship is here and the meteors are here. And then you push the button. If you push the button for shoot, then you'll be in the same place. The meteors will still be here, but you'll have a little shot coming out of your spaceship. That's what the environment does. Right. So you give it a state and action and it will give you the next state. It will also give you a reward. Right. The reward in the same thing, reward here will be a second output here that tells you either, let's say, negative one if you die or zero if nothing happens or plus one if you shoot a meteor. That's the reward. So this you can think of as the real world. So these two quantities are in the real world in the environment. So that's how you model the environment. Then the agent has these quantities called the policy. So what pi does is much like the transition, but pi takes in a state and gives you an action. Right. So this is now the agent deciding this is thinking. The policy takes in a state and gives an action. And this contains various forms, but it's just a function for now. The agent also has a Q and V function, and these are quite, quite similar. So the Q function, what the Q function will do if if you were in a state and you have several options of what to do, right, you have action one, action two and action three. You're in state S. The Q function of S and A one superscript pi would tell you the following. It would tell you what's my expected reward if I'm in state S. That's here. And perform action A. So A one. So if I now take this path and after this path, I follow the policy pi, right, the policy pi for each of the following. So it's like right now I take action A one. I don't care about my policy. But after that, I follow the policy pi. What is my expected reward going to be until the end of the episode? That's the Q function. And the value function here, very similar, but it only cares about the state. It says if I'm in state S and I just follow the policy pi, even in the first step, right, I just follow this policy pi. What is my expected reward going to be over the course of the episode? That is the Q and the value functions. You can see why Q learning is popular. If you have a good Q function and the Q and the value function, these are the things that you actually want to learn, right? If you have a good Q function, you can simply always plug in every action into your Q function and then simply take the maximum, the action that has the maximum Q value, because that will give you the best reward if your policy pi, right, is kind of self-referential. If your policy is to always take the maximum Q value, then taking the maximum Q value with the policy, given that you take the maximum Q value, will be optimal. All right, this was very convoluted, but all right, so let's start off with modeling the environment in this continuous framework. So instead of having the next state be determined by the current state in action, in the continuous framework, they do this via differential equation. So the DS is how does the environment change? This is the change in the environment that is determined by two functions, F and G. So F is your classic environment function. It takes in a state and an action at time t, right? These are now functions and it will output how the state changes. And the G here is, this is a Wiener process, is to introduce stochasticity, as I understand it, because in the classic formulation, the transition model gives you a probability up here, a probability distribution. So this Wiener process is responsible for introducing that probabilistic nature into this differential equation. But ultimately, it simply tells you how does the state change depending on my state, current state and action that I perform. So the reward function now is also pretty simple. The tau here is a trajectory and the trajectory is simply the state and action over time. So if I integrate from time zero to infinity or to the end of the episode, my reward function at each point in time, right? So I go through my episode and I get high reward, not so high and so on. So the integral under this curve will be my total reward, just like we sum up the reward of individual steps in the discrete case. In the continuous case, you can think of each infinitesimal time step giving you a tiny bit of reward. So the entire reward is just an integral. Then we go on the value function for a given state at time t. So think about what this is. The value function for a state means what reward can I expect starting in this particular state and then following policy pi until the end of the episode. And that here is the expectation over all trajectories that come from my policy of the reward in that trajectory. So I can, you know, if I'm here, my policy now is also a distribution. It can go multiple trajectories, right? And I want to have the expected value of the reward. So each one of these has a reward, the expected value of the reward over all trajectories starting from state s t. And again, here you say that that is the integral over the now here I have a bit of a problem because here they say t equals zero going from here and here. But here the t is already here. So I believe this should be this should be t equals t prime and then t prime t prime t prime up here. And t minus t prime or something like this. In any case, I think it should actually start from this state here and not from time zero. But I might be missing something. I'm not the biggest integrator in the world. So, you know, all right, then you have the Q function. Now think of it what the Q function is in the discrete case, the Q function tells you if I'm in state s and perform action a, what is my expected reward going to be? I have to introduce some different things here. They say if I'm in state s and I act action a at time t until time h right now, you have to say how long you're going to perform the action for until you perform the next action. Right. So h is your your lag time here until you perform the next action. So this now I actually agree with this formulation with the integral here. So this is going to be the integral from time t to time t plus h. That's how long you perform the action. Your reward of performing that action, right. Given the state plus the value function at the end of that. So you're here. You're in s t and you perform action a right. And then this is your state at time t plus h and then you're here. And from there on, you could perform many, many, many actions. But in the original notion of the Q function, the Q function tells you if I'm here and I perform this action and after that, I act according to policy pi. What is my what is my expected reward? And there's a classic recurrence relation in reinforcement learning where you can say the Q function in s t given to a is the reward that I get from performing a in state s plus the value function at state s at the next state. Because the value function is exactly the reward that you would get by following policy pi in that next state. And the Q function means I perform a now and after that I perform pi. So this is the continuous analog. That's why you have this part here where you perform the action for each time. After each time you just go after go with your policy and that will be the value function. So this is the continuous formulation of the of the problem. Right. And now they can introduce these these lagging times. So in their diagram up here, they define these notions. So you have your state s t right here. Then after this time, you capture the new state. Right. So after that time, you capture the new state and decide on an action and then you perform it for each time. Is that correct? Until here. So the the the I minus one of action is performed at this time and the I action is performed at this time. No, that makes no sense. So let's read it. So this is when you capture the state and you need to time to perform to think. Right. This is thinking. And then you perform this action at that time. This is the lag time now. And you perform this action. You want to know you want to know if I perform this action until this time here, what is what is happening? So this is the new Q function takes into account this thing. It tells you if I'm in state s and I think this is thinking leads me to here. This is the old action, right? This is the old action that's still happening while I observe this state. Right. So it means if I do this right now and after thinking, I do this. Right. So I'm at state. I'm at time t. And this is still happening. And then after I think thinking leads me here t plus t a s. I perform this new action. I'm out of colors. I perform this new action at that point until time H. What's my Q function? So my Q function is going to be the integral time t where I start observing the state and start thinking until t plus t a s. That's when I still perform the old action. Right. So this is going to be the reward in the state given the old action. And then at that time, I switch over to the new action. Right. So at that time until time H, now I perform the new action. So this entire part here, this part until here is taking the place of this first part here in the Q function of this first part. Right. So because before it was simply executing one action, we didn't have this concurrency yet. So executing the action. And after that, it's going to be the value function. And now it's executing two actions. First, execute the old action. Then once you're done thinking, execute the new action. And then it's the value function from there on. I hope this is clear. It wasn't clear to me until just now as well. All right. So they define the Monte Carlo estimator where you can do this with just samples of the trajectories instead of expectations. And then they define the Bellman operator, the Bellman backup operator. Now, the Bellman backup operator is an important quantity in value based reinforcement learning because the Bellman backup operator is basically what I talked about before. It tells you that if your policy is to always select the maximum, the action with the maximum Q value, right? That's what's down here. After you do this action, then the policy you arrive at and you can give certain optimality guarantees. But in essence, this is so-called a contraction. So if you always do that and you calculate your Q function that way, it will mean that in the contraction is defined as if you have an operator. If you have two things that are X1 and X2 that are some apart from each other, then after you apply the operator, this T here, X1 minus T X2, they will be closer together, which basically means that the Q to Q functions of the individual states will be closer together. And you'll converge to a single Q function. So given enough time and enough data, you'll converge on one Q function. There's one fixed point Q function that you'll converge to. And you can show under assumption in classic RL that this is going to be the optimal Q function, the true, let's say, Q function. So they first prove this and then they prove a now they go back to discrete time. So now they were in continuous time. They go back to discrete time, but now they have a discrete time formulation with this lag here. And also they prove that that Bellman operator is a contraction. So the contraction part basically means that if you perform Q learning, you're going to arrive at a solution. That's what this means to be contraction. But now, obviously, that solution in classic RL is going to be the optimal Q function. But here, I actually don't know. All right, so they try this out and they introduce one last important concept here, what they call vector to go, which basically means that at the point where they start thinking, where is a good thing to show this at the point where they start thinking, they give a they give the last action with. So at this point right here, where they capture the state, they also sort of the state contains a information about what part of the action that you started here is still outstanding. So maybe your action was and they illustrate this down here. Maybe your action was to move your robot arm from down here to up here. That was your planned action at this point in time. Now, if you are at step, if you perform the action here and here you start capturing the next state, then you would also give this particular vector here to the to the to the agent. So not only will you tell it, hey, by the way, my last action was a T minus one, as you would need in the Q value. You will also say, and this much is outstanding. This is much is where as I still have to do that much. So basically you're saying I wanted to move my arm right here and I still have to do this part of the action. Now you can see why the algorithm is able to learn much better given that information, because otherwise it has it would have to basically infer that vector from kind of differencing the action minus the what probably happened in the meantime. So they test this out. And what results is the robot videos you've seen before where they say they can recover the original Q learning in this continuous framework. So here on the left side, you have blocking actions and it says when it says yes here, it is kind of the old old framework. You see the grasp success at like 92 percent, where as if you go to non blocking actions, but do none of the none of the concurrent information, the grasp success suffers. But you can recover the grasp success if you if you give these concurrent information like introduce time step penalty and you give this vector to go and the information about the previous action. And you can also see that the episode duration here is much lower when you go for the continuous actions than when you are in the old framework, naturally, because you don't need to pause. Right. In this. So this is the simulated robotics and the real world robotic grasping results. You see kind of similar results in that if you do have blocking actions, your grasp success is higher than if you don't. But your duration of your of your policy is cut in half. So maybe this is a trade off worth considering. I think this is a is pretty cool framework. And I think there's going to be a lot of work still outstanding here. And I invite you to check out the paper and look at their videos and their ablation studies of what's important and what not. And with that, bye bye.
 Hi there. So if you look at these two robots, the left one labeled blocking, the right one labeled concurrent,
 the blocking robot, as you can see, always has these little pauses in its movement where it does nothing
 and then it kind of continues with its motion, while the one on the right is one continuous motion that it does.
 So the reasoning here is that the robot has a camera and the camera takes some time to register what's going on.
 And then also the robot has a computer inside and the computer also takes some time to decide what to do based on what the camera saw.
 And while all of this is happening, the robot on the left just freezes. So it performs an action and then it freezes
 because it takes time to register a state and compute a new action, whereas the robot on the right, it takes the same amount of time to do these things.
 It also takes a time to register the state and to compute an action, but it does that as it is executing the last action.
 So it does this in parallel and then it executes the action. Once it has computed a new action, it executes that new action right on top of the old action.
 And that gives this one big fluid motion. So this requires a new formulation of reinforcement learning.
 And that's what this paper does. Thinking while moving deep reinforcement learning with concurrent control by people from Google Brain, UC Berkeley and X.
 So they have a nice diagram here in the supplementary material to show you what is going on in their framework.
 So in classic reinforcement learning right here, in classic reinforcement learning, you have this dichotomy between agent and environment, right?
 So the agent and the environment. Now, the agent is supposed to kind of act in the environment in the following manner.
 The environment will send an observation to the agent. The observation in this case is the picture of the camera.
 So it sends an observation and the agent will think what to do with the observation, which is called a policy.
 The policy pie will take an observation and then output an action of what to do and will send back the action to that environment.
 So in classic RL, you assume that this part here is kind of freezes time.
 So the environment will output an observation and as and the process of registering the observation of computing the action and of sending the action back is happens in zero time.
 Of course, it doesn't happen in zero time, but in our reinforcement learning problems, for example, the OpenAI Gym, the environment just stops until it gets the next action.
 And then it performs the action, right? It performs the action in the environment and by that the environment changes and time happens.
 And then it stops again as we think of the next action, right? So this is we usually call this one step in the in the kind of classic formulation of RL.
 The only point that time happens is when the action is executed. No time happens when the state is registered or when the action is computed.
 And that's what you see here on the left. So in blue, you have the state registration. This is, for example, the camera.
 The camera has some time in order to register and store the image that it has taken, right? Maybe post process it a little bit.
 So that's what the camera does. But in our classic formulation, as you can see here, if this is time, it happens instantaneously all at the same time.
 And also, this is the policy. This is thinking what to do, right? This is your evaluation of your neural network.
 If this is a neural network, I'm drawing a small neural network here. This happens instantaneously in these formulations and only as the action is executed, time happens, right?
 And then until here, time freezes again and only once the action is determined, time happens again.
 In the new formulation now, as you've already seen, what we have here is this kind of continuous framework where, let's say you're here,
 it actually takes time for the camera to post process the image. It takes more time for you to think about what to do.
 And then once you decide on an action, that action is going to happen, right?
 But we can say, for example, at this point, you tell the camera to take a new picture of the state, right?
 But that takes time. And while that's happening, the old action is still ongoing, right?
 So you don't even have to say the action is still ongoing, but the world is still moving, right?
 The world is still changing while you think, while you post process and while you evaluate your policy.
 The world is thinking and only after some time, right?
 After this lag time here, have you decided on a new action?
 And then you can break that old action and kind of perform your new action.
 And all of this is happening in time. So this is the new framework.
 Now you see the problem here. The problem is that you base your decisions on the state and time T on time, sorry, time H here.
 You base your decisions on a state as it was at that time, right?
 That's what you use to think right here. That's what you store and think about.
 But you perform. So you perform the action at this point in time.
 So there is a considerable difference here because the world has now changed.
 So you see the problem. The action you perform is based on an old knowledge of the world.
 And you have basically no way of making the action dependent on the current state of the world,
 because that would require you to capture the current state and that takes time.
 And in that time, the world has already shifted again.
 So the agent is kind of required to think ahead about the action that it is currently performing and how the world changes according to that.
 So this new formulation of reinforcement learning formulates this in a formal way.
 It formulates it in a formal way and will quickly go through that.
 Yes, so they go into the very basics here. We'll quickly, quickly go through them.
 So they introduce these quantities like the policy pi, the transition distribution, the reward and the Q and value function.
 Now we'll just quickly go over these. So you have the agent. Sorry about that.
 You have the agent and you have the environment. Hello.
 So if you think of the agent and the environment, the environment has this transition function.
 The transition function, it takes it says, OK, I'm in this state and the agent does this action.
 And here is the probability distribution over the next state.
 So it says that your little spaceship is here and the meteors are here.
 And then you push the button. If you push the button for shoot, then you'll be in the same place.
 The meteors will still be here, but you'll have a little shot coming out of your spaceship.
 That's what the environment does. Right. So you give it a state and action and it will give you the next state.
 It will also give you a reward. Right.
 The reward in the same thing, reward here will be a second output here that tells you either, let's say, negative one if you die or zero if nothing happens or plus one if you shoot a meteor.
 That's the reward. So this you can think of as the real world. So these two quantities are in the real world in the environment.
 So that's how you model the environment. Then the agent has these quantities called the policy.
 So what pi does is much like the transition, but pi takes in a state and gives you an action. Right. So this is now the agent deciding this is thinking.
 The policy takes in a state and gives an action. And this contains various forms, but it's just a function for now.
 The agent also has a Q and V function, and these are quite, quite similar.
 So the Q function, what the Q function will do if if you were in a state and you have several options of what to do, right, you have action one, action two and action three.
 You're in state S. The Q function of S and A one superscript pi would tell you the following.
 It would tell you what's my expected reward if I'm in state S. That's here. And perform action A.
 So A one. So if I now take this path and after this path, I follow the policy pi, right, the policy pi for each of the following.
 So it's like right now I take action A one. I don't care about my policy. But after that, I follow the policy pi.
 What is my expected reward going to be until the end of the episode? That's the Q function.
 And the value function here, very similar, but it only cares about the state.
 It says if I'm in state S and I just follow the policy pi, even in the first step, right, I just follow this policy pi.
 What is my expected reward going to be over the course of the episode? That is the Q and the value functions.
 You can see why Q learning is popular. If you have a good Q function and the Q and the value function, these are the things that you actually want to learn, right?
 If you have a good Q function, you can simply always plug in every action into your Q function and then simply take the maximum, the action that has the maximum Q value,
 because that will give you the best reward if your policy pi, right, is kind of self-referential.
 If your policy is to always take the maximum Q value, then taking the maximum Q value with the policy, given that you take the maximum Q value, will be optimal.
 All right, this was very convoluted, but all right, so let's start off with modeling the environment in this continuous framework.
 So instead of having the next state be determined by the current state in action, in the continuous framework, they do this via differential equation.
 So the DS is how does the environment change? This is the change in the environment that is determined by two functions, F and G.
 So F is your classic environment function. It takes in a state and an action at time t, right?
 These are now functions and it will output how the state changes.
 And the G here is, this is a Wiener process, is to introduce stochasticity, as I understand it, because in the classic formulation, the transition model gives you a probability up here, a probability distribution.
 So this Wiener process is responsible for introducing that probabilistic nature into this differential equation.
 But ultimately, it simply tells you how does the state change depending on my state, current state and action that I perform.
 So the reward function now is also pretty simple.
 The tau here is a trajectory and the trajectory is simply the state and action over time.
 So if I integrate from time zero to infinity or to the end of the episode, my reward function at each point in time, right?
 So I go through my episode and I get high reward, not so high and so on.
 So the integral under this curve will be my total reward, just like we sum up the reward of individual steps in the discrete case.
 In the continuous case, you can think of each infinitesimal time step giving you a tiny bit of reward.
 So the entire reward is just an integral.
 Then we go on the value function for a given state at time t.
 So think about what this is. The value function for a state means what reward can I expect starting in this particular state and then following policy pi until the end of the episode.
 And that here is the expectation over all trajectories that come from my policy of the reward in that trajectory.
 So I can, you know, if I'm here, my policy now is also a distribution.
 It can go multiple trajectories, right? And I want to have the expected value of the reward.
 So each one of these has a reward, the expected value of the reward over all trajectories starting from state s t.
 And again, here you say that that is the integral over the now here I have a bit of a problem because here they say t equals zero going from here and here.
 But here the t is already here. So I believe this should be this should be t equals t prime and then t prime t prime t prime up here.
 And t minus t prime or something like this. In any case, I think it should actually start from this state here and not from time zero.
 But I might be missing something. I'm not the biggest integrator in the world.
 So, you know, all right, then you have the Q function.
 Now think of it what the Q function is in the discrete case, the Q function tells you if I'm in state s and perform action a, what is my expected reward going to be?
 I have to introduce some different things here. They say if I'm in state s and I act action a at time t until time h right now, you have to say how long you're going to perform the action for
 until you perform the next action. Right. So h is your your lag time here until you perform the next action.
 So this now I actually agree with this formulation with the integral here.
 So this is going to be the integral from time t to time t plus h.
 That's how long you perform the action. Your reward of performing that action, right. Given the state plus the value function at the end of that.
 So you're here. You're in s t and you perform action a right.
 And then this is your state at time t plus h and then you're here. And from there on, you could perform many, many, many actions.
 But in the original notion of the Q function, the Q function tells you if I'm here and I perform this action and after that, I act according to policy pi.
 What is my what is my expected reward? And there's a classic recurrence relation in reinforcement learning where you can say the Q function in s t given to a is the reward that I get from performing a in state s plus the value function at state s at the next state.
 Because the value function is exactly the reward that you would get by following policy pi in that next state.
 And the Q function means I perform a now and after that I perform pi.
 So this is the continuous analog.
 That's why you have this part here where you perform the action for each time.
 After each time you just go after go with your policy and that will be the value function.
 So this is the continuous formulation of the of the problem.
 Right. And now they can introduce these these lagging times.
 So in their diagram up here, they define these notions.
 So you have your state s t right here.
 Then after this time, you capture the new state.
 Right. So after that time, you capture the new state and decide on an action and then you perform it for each time.
 Is that correct?
 Until here.
 So the the the I minus one of action is performed at this time and the I action is performed at this time.
 No, that makes no sense.
 So let's read it.
 So this is when you capture the state and you need to time to perform to think.
 Right. This is thinking.
 And then you perform this action at that time.
 This is the lag time now.
 And you perform this action.
 You want to know you want to know if I perform this action until this time here, what is what is happening?
 So this is the new Q function takes into account this thing.
 It tells you if I'm in state s and I think this is thinking leads me to here.
 This is the old action, right?
 This is the old action that's still happening while I observe this state.
 Right. So it means if I do this right now and after thinking, I do this.
 Right.
 So I'm at state. I'm at time t.
 And this is still happening.
 And then after I think thinking leads me here t plus t a s.
 I perform this new action.
 I'm out of colors.
 I perform this new action at that point until time H.
 What's my Q function?
 So my Q function is going to be the integral time t where I start observing the state and start thinking until t plus t a s.
 That's when I still perform the old action.
 Right. So this is going to be the reward in the state given the old action.
 And then at that time, I switch over to the new action.
 Right.
 So at that time until time H, now I perform the new action.
 So this entire part here, this part until here is taking the place of this first part here in the Q function of this first part.
 Right. So because before it was simply executing one action,
 we didn't have this concurrency yet.
 So executing the action.
 And after that, it's going to be the value function.
 And now it's executing two actions.
 First, execute the old action.
 Then once you're done thinking, execute the new action.
 And then it's the value function from there on.
 I hope this is clear.
 It wasn't clear to me until just now as well.
 All right.
 So they define the Monte Carlo estimator where you can do this with just samples of the trajectories instead of expectations.
 And then they define the Bellman operator, the Bellman backup operator.
 Now, the Bellman backup operator is an important quantity in value based reinforcement learning because the Bellman backup operator is basically what I talked about before.
 It tells you that if your policy is to always select the maximum, the action with the maximum Q value, right?
 That's what's down here.
 After you do this action, then the policy you arrive at and you can give certain optimality guarantees.
 But in essence, this is so-called a contraction.
 So if you always do that and you calculate your Q function that way, it will mean that in the contraction is defined as if you have an operator.
 If you have two things that are X1 and X2 that are some apart from each other, then after you apply the operator, this T here,
 X1 minus T X2, they will be closer together, which basically means that the Q to Q functions of the individual states will be closer together.
 And you'll converge to a single Q function.
 So given enough time and enough data, you'll converge on one Q function.
 There's one fixed point Q function that you'll converge to.
 And you can show under assumption in classic RL that this is going to be the optimal Q function, the true, let's say, Q function.
 So they first prove this and then they prove a now they go back to discrete time.
 So now they were in continuous time.
 They go back to discrete time, but now they have a discrete time formulation with this lag here.
 And also they prove that that Bellman operator is a contraction.
 So the contraction part basically means that if you perform Q learning, you're going to arrive at a solution.
 That's what this means to be contraction.
 But now, obviously, that solution in classic RL is going to be the optimal Q function.
 But here, I actually don't know.
 All right, so they try this out and they introduce one last important concept here, what they call vector to go,
 which basically means that at the point where they start thinking,
 where is a good thing to show this at the point where they start thinking, they give a they give the last action with.
 So at this point right here, where they capture the state,
 they also sort of the state contains a information about what part of the action that you started here is still outstanding.
 So maybe your action was and they illustrate this down here.
 Maybe your action was to move your robot arm from down here to up here.
 That was your planned action at this point in time.
 Now, if you are at step, if you perform the action here and here you start capturing the next state,
 then you would also give this particular vector here to the to the to the agent.
 So not only will you tell it, hey, by the way, my last action was a T minus one, as you would need in the Q value.
 You will also say, and this much is outstanding.
 This is much is where as I still have to do that much.
 So basically you're saying I wanted to move my arm right here and I still have to do this part of the action.
 Now you can see why the algorithm is able to learn much better given that information,
 because otherwise it has it would have to basically infer that vector from kind of differencing the action minus the what probably happened in the meantime.
 So they test this out.
 And what results is the robot videos you've seen before where they say they can recover the original
 Q learning in this continuous framework.
 So here on the left side, you have blocking actions and it says when it says yes here,
 it is kind of the old old framework.
 You see the grasp success at like 92 percent, where as if you go to non blocking actions,
 but do none of the none of the concurrent information, the grasp success suffers.
 But you can recover the grasp success if you if you give these concurrent information like introduce time step penalty and you give this vector to go and the information about the previous action.
 And you can also see that the episode duration here is much lower when you go for the continuous actions than when you are in the old framework,
 naturally, because you don't need to pause.
 Right.
 In this.
 So this is the simulated robotics and the real world robotic grasping results.
 You see kind of similar results in that if you do have blocking actions, your grasp success is higher than if you don't.
 But your duration of your of your policy is cut in half.
 So maybe this is a trade off worth considering.
 I think this is a is pretty cool framework.
 And I think there's going to be a lot of work still outstanding here.
 And I invite you to check out the paper and look at their videos and their ablation studies of what's important and what not.
 And with that, bye bye.