Regularizing Trajectory Optimization with Denoising Autoencoders (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper rl reinforcement learning model predictive control dae denoising autoencoders trajectory trajectory optimization planning adversarial attack errors open loop closed loop joint probability derivative gaussian experience learned model world model model predictive mpc
Can you plan with a learned model of the world? Yes, but there's a catch: The better your planning algorithm is, the more the errors of your world model will hurt you! This paper solves this problem by regularizing the planning algorithm to stay in high probability regions, given its experience.

https://arxiv.org/abs/1903.11981

Interview w/ Harri: https://youtu.be/HnZDmxYnpg4

Abstract:
Trajectory optimization using a learned model of the environment is one of the core elements of model-based reinforcement learning. This procedure often suffers from exploiting inaccuracies of the learned model. We propose to regularize trajectory optimization by means of a denoising autoencoder that is trained on the same trajectories as the model of the environment. We show that the proposed regularization leads to improved planning with both gradient-based and gradient-free optimizers. We also demonstrate that using regularized trajectory optimization leads to rapid initial learning in a set of popular motor control tasks, which suggests that the proposed approach can be a useful tool for improving sample efficiency.

Authors: Rinu Boney, Norman Di Palo, Mathias Berglund, Alexander Ilin, Juho Kannala, Antti Rasmus, Harri Valpola

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we're looking at regularizing trajectory optimization with denoising autoencoders by Renu Bonet, Norman DiPaolo and others of various places, but a lot of the people are from Curious AI. And we actually had a discussion with Hari, who is the CEO of Curious AI. And this was on our Machine Learning Street Talk podcast. So this is another YouTube channel for those of you who don't know, where every week or so we try to have either an interesting discussion or a guest, like sort of an interview or talk about comment on the talk. So if it is not out yet, I'll link to it as soon as it comes out. But if you're watching this video later, make sure to check out our conversation with Hari because it was absolutely fantastic. And in general, if you like videos like this, consider subscribing, liking, sharing if you're still here at the end and liked it. Okay, so this paper on a high level deals with model based reinforcement learning. Model based reinforcement learning means that you are using a model of the world to do reinforcement learning. So in essence, if you have your reinforcement learning setup where you are an agent, and you have to interact with the world, you have to do so in many steps in like a round trip fashion. So you put an action you act and the world gives you back an observation. And you have to act in the world over and over and over such that you will be able to maximize your reward. Now what is model based reinforcement learning? Model based reinforcement learning basically means that the agent here has internally a model of the world. So it sort of understands how the world works. Situations where you have a accurate model of the world are things like chess. So in chess, the rules are very clear, you know how the world's going to behave if you perform a certain action. But in real world applications, it's very, very hard to actually make a model. So people usually rely on learned models. So what does it mean you basically learn a neural network that tries to predict how the world is going to act. So this here is going to be a deep neural network that you learn from what you see in the world. Now trajectory optimization basically means that you are now you now have this world model, and you use it to look ahead, as I said, so you are in the state like here, and you can do let's say three different actions. And you use your world model here, world. And you see, you think how's the world going to react if I do either of those three things, and then you get into three different states. And then again, you after each one, you consider three actions, three actions here, three actions here, and so on. So ultimately, you're going to kind of have an overview over a planning horizon, which here we call H, you kind of look ahead a couple of steps, or there are various ways of doing this. But ultimately, you will basically find that this this path here is really good. So I think I'm going to take this as a first action. So trajectory optimization considers finding the best green path here in this tree of possibilities that your your world model gives you. Okay. Now, what does what do these people say? They say this procedure often suffers from exploiting inaccuracies of the learned model. What does that mean? That basically means that if I have a world model, and it is not accurate, then it is basically, basically the thing that tries to find the best green path here, the optimizer is sort of trying to find the the best path against this world model. Now, if that world model is inaccurate, that can lead to devastating consequences. So what do we mean by this? I'll give you an example. If you have a room, right, and the room is, let's take our classic room like this. And you are here and you would like to go here. And so you're a reinforcement learning agent, you do some exploration, right, you explore a bit here, the next episode, you might go here, and you might go here, and so on. And over time, in this framework, you're going to build a model of the world. So at the beginning, we won't tell you how these rooms look, you have to discover it by yourself. So maybe at the beginning, we only tell you, there's these four walls, the rest you have to figure out. So on and on, you're going to fill in your blanks, you do your first explorations, and you've had there's a bit of a wall here, right. And there might be some wall here, I crashed into that, right, you're going into here, you crash into a wall, you saw there's a wall here and here, there's a wall here, you go maybe here, oh, there's no wall. So you go further, there's no wall anywhere here, you crash here, okay, we already knew there's a wall. Maybe you crash here. All right, so right now, you have, okay, you go here, you have a model of the world in this situation, where there's a wall here, a wall down. And if you now try to do trajectory optimization, remember, you have to go from here to here. If you try to do trajectory optimization, what is it going to turn out? It's going to turn out like, look, there you go. That works just fine. And that's not because you're so good at planning. I mean, you are good at planning, but because your model is inaccurate here, because it has never seen this, your entire training distribution that you trained the world model on, only explored the area over here. All right, so you see how the more efficient this planning algorithm is, like the blue arrow, the thing that finds the blue arrow, the more efficient that is, the more consequential it is when your learned world model has mistakes, because it will exactly exploit these mistakes in order to get the shortest path possible or the highest reward in that case. And this, they call this like almost an adversarial attack on the world model, which is a pretty good way of framing it. They propose actually to solve this problem. They say we propose to regularize trajectory optimization by means of a denoising autoencoder that is trained on the same trajectories as the model of the environment. We show that the proposed regularization leads to improve planning with both gradient based and gradient free optimizers. We also demonstrate that using regularized trajectory optimization leads to rapid initial learning in a set of popular motor control tasks, which suggests that the proposed approach can be useful tool for improving sample efficiency. So in essence, what do they do? They basically say, okay, we want to regularize this using a denoising autoencoder. And I think it's best if we if we look at the at the math for doing this. So the math here starts off as follows, saying you want to learn a world model. This is F here. F is the world model. It takes in a state and an action and it gives you the next state or an approximation to it. And the parameters here indicate that this is some sort of function that you learn like a deep neural network. You can do this in fully or partially observed environments. Now when you plan, what you want to do is you say I have a planning horizon H, right? Then I have a reward function, and the reward function is going to give me a reward for each state action pair. So if I'm in a state and I do a certain action, I'm going to get some reward. This could be you have reached the target or this could be you know how much money you've collected or whatnot. So you're going to look at horizon H, you're going to look H steps into the future, and you want to maximize the sum of all the rewards here. So in the limit, this reduces to simply like, for example, reaching the target in our rooms case if H is infinite. But you can consider a lower planning horizon. So you want to find the action sequence that maximizes this reward in the future. And now this reward relies on your environment model. So here's the algorithm. First you collect some data, okay, that's how you start off. That train the dynamics model, the world model, using the data you've already collected. Then for each time step T, you want to optimize this trajectory. So you want to find the best next action sequence and take the first action, implement the first action and get the new observation. And do you do this in a loop until the end and at the end you say add this data to D. So that's what you do. You use your world model to get the best action sequence. That's how you optimize the trajectory. And then at the end of the episode, you've done an episode, right? You went somewhere, you put all of this into your training data to make the world model better. Something to note here is that the world model will only learn about things that you have done, right? So there is kind of an interaction effect. That's the green area here. The world model only knows the paths, the world model only can accurately estimate the world where you have been. And that's going to turn out to be the entire problem because these blue arrow finder can now go away from that. That's explained here. Potential inaccuracies of the trained model cause substantial difficulties for the planning process. Rather than optimizing what really happens, planning can easily end up exploiting the weaknesses of the predictive model. Planning is effectively an adversarial attack against the agent's own forward model. This results in a wide gap between expectations based on the model and what actually happens. And they have this example here where it's like an industrial control process. And what you have to imagine, there's like some sort of a container here with a liquid in it. And there are two pipes that lead to this container, pipe one and pipe two. And there are valves here. So there's this valve right here and there's this valve right here. So these are valve one and valve two. And there is also an output pipe right here and that's another valve right here. So you can control these three valves, the two inputs and one output. And you have to somehow optimize the reaction in here. So this is a chemical reaction made up out of the two liquids that flow in here. And you have to somehow optimize a property of that. And that's highly nonlinear and has maybe like time shifts. So when you open a valve, it's going to take a while and then it's very nonlinear. And then you are not supposed to break the pressure limit. So you have to also outflow some stuff. And if you just do this with a learned model, it looks like this. So first of all, here is a classic controller. People have been doing this stuff in industry and they basically build controllers for it. And you can do that and that works out really okay-ish. As you can see right here, this is the product rate, what you're supposed to optimize. And you see some sort of a smooth... You're supposed to actually bring it to this dashed line right here. And this is some sort of smooth thing, right? And you're supposed to, I guess, bring the pressure here and the A in purge. I don't know what these quantities are, but you're supposed to bring them to the dashed line and it's very nonlinear and very time dependent. So that works. And you see here kind of the smoothness by which the variables are manipulated. Now, if you just learn a world model and then do this trajectory optimization, basically this is some sort of a planning-based reinforcement learning with a world model. You see right here, it works, but it's super jittery. The pressure spikes here and apparently this here is a pressure limit. So it spikes the pressure limit. And you can see that the manipulated variables are up and down and up and down and up and down because at each step, it basically completely overestimates its potential reward. With things like, wow, this is really good, but all it does is find a weakness in the model and not a really good action per se. Now with their method to already take it away, you can see that now the control task super smoothly and very quickly converges to these optimal things. And you can see that the variables being manipulated are also rather smoothly manipulated. And that's an indication that the model is accurately estimating their rewards. Okay, so how do they do it? Via what they call trajectory, via regularization of trajectory optimization. So in essence, what do we want to regularize here? There are many things we could do to solve this, but the way this paper goes is they say we not only do we want the most return, we also want a high log probability of our taken path. So this here, as you can see, this is observation action and so on, observation action. So this is the future. This right here is the future. So this sequence here is what is going to give me the reward right here. So G is also dependent on these things, but it's not said explicitly here. So G is dependent on your plan. Maybe let's not call this the future. This is the plan. This is the plan you came up with. So this is directly going to influence G and G is the reward you're going to get under your model. But also you want the log probability of the plan itself to be high. Now there, I think there is a bit, there is something missing here and that is conditioned on your training distribution right here. And I think that's a actually rather crucial part. Now that's, that's the KL thing. So this is conditioned on your training. So what you want is you want the plan to be basically in your training distribution. So you, you want what you, you want your plan that you're going to execute. If that is actually part of your training data set, then you know, I have already executed this once before and it's reasonable to assume that therefore my world model has learned from this experience and is going to give me an accurate reward. If we go back to our rooms example, then up here somewhere, if we go back to our rooms example, right, you see that anywhere in the green area where I have already explored the world model is fairly good, right? It's going to give me accurate reflection of the world. But as soon as it go outside the green area, it is not. And inside the green area is basically where my training data is. Now if I in the future actually take a path here, crash into a wall right here, right? You saw in the algorithm at the end of an episode, I'm going to add my trajectory to the training data for the world model. So this green part here expands to include that. And now if I go here again, if my plan goes there again, now I can trust the world model. But also now it has it is actually correct because it has a wall here. So you see that the regularization basically you not only do I want the biggest reward possible under my world model, I also want that the plan that I'm about to execute is has a high probability under my training distribution. Okay. And the way we do this is by denoising auto encoders. We want the log probability here to be high and you do this via a denoising auto encoder. What's a denoising auto encoder? A denoising auto encoder is basically so if you have, for example, an image and the image is of a trusty cat whiskers and a denoising auto encoder is an unsupervised method where you have it's basically an auto encoder. So there is a bunch of layers compressing to a hidden representation, then uncompressing it again. Okay. And at the end, you want to output the same as at the beginning. So it's basically an auto encoder. But the special part about the denoising auto encoder is that first, you take your input and you know, you put some noise on it. So that could mean could mean anything here. But here, what they do is they do they make some Gaussian noise on it. Now, I can't really draw Gaussian noise here, but it would be kind of convolved with Gaussian Gaussian noise. So I'm just going to add some noise like this. So noise, noise, noise. So there's some noise, you see, and then you feed that. That's now what you feed in here. And the algorithm is supposed to reconstruct this, this original image. So the algorithm is basically supposed to take away the noise, it doesn't see the original image, but it's supposed to produce it. And you do this with your training data. What does that mean? Ultimately, for our trajectory optimization, it means that if I have a trajectory that I did before, and it maybe goes here, right? What I can do is I can make a noisy version of it, which would be the black one right here. So I put some noise on it, some noise. Right, it's kind of the same, but okay. And the denoising autoencoder is supposed to give me back the red one. This will simply give me some sort of a probabilistic model of my training distribution. So they go through the math here and show that these denoising autoencoders actually naturally output this log probability, sorry, the gradient of the log probability. Because optimal denoising theory says that for zero mean and Gaussian noise, the optimal denoising function, the optimal denoising function for zero mean Gaussian corruption is this thing right here. So it is, if you give me X and you tell me X has been corrupted by zero mean Gaussian noise of size sigma n, then the best, and you simply tell me, give me back the original image, the best thing I can do is to take what you gave me and add this gradient of the log probability of X if I can, if I have a model of the log probability. So that's the best thing I can do. And that's the best denoising function. And now you have to think a bit in reverse. If we train a denoising autoencoder, that is going to approximate this best function that there is. So we know that the best possible denoising function is this, we train a denoising autoencoder, which in the optimal case is going to converge to the best denoising function. So if we then reformulate and we do denoising autoencoder of X minus or X tilde minus X tilde, that is divided by the standard deviation. Sorry, the variance. That is going to give us this quantity right here, the gradient of the log probability. And the gradient of the log probability of X is exactly what we need to run gradient descent on our function. So here is our function again, G plus this regularization. Now they don't regularize over the entire future, but over these windows. But in essence it's G plus the log probability of your plan. If you take the gradient of that, of course you take the gradient of the sum. So it's the gradient of G plus the gradient of the log probability with respect to the actions. And here simple application of the chain rule will tell you that you have to propagate through the input, through the X. And you need this quantity. The gradient of the log probability with respect to its inputs. Now as we just saw, the optimal denoising autoencoder is going to output that thing. So if we train a denoising autoencoder and we suppose it reaches a good accuracy, then we can obtain this quantity basically for free. And that's the entire trick here. So in essence, what does it mean? In essence what it means is that if we are in our room again, and we have our partial model of the world, let's say we have this model, because we are here and all we've ever explored is these things right here. Now when I go and do my trajectory optimization, and my trajectory optimization wants to go here, I simply say, no, I don't know that, I haven't seen that yet. You can only plan basically within the space where we have already been. So you can plan like here. So here now there is of course, there is going to be some exploration, so some probability that you can go away a bit, but not too much. So in this case, it would result in the planning only to happen in spaces where we've actually been. So it might go here, and then here, because okay, here we haven't been anywhere. But then that would lead me to take the first step in this direction, and not in this direction. And if I take my first step in this first direction, then of course, I'm going to be already a bit on the correct path right here. Because if I take the first step into this direction, then after that, I'm going to have to, if once I crash here, I'm going to have to correct really hard. And that's exactly what's going to give you this super jittery control. Whereas if you only plan where you've already been, you won't, the probability that you're going to have to do like a 180 is going to be much, much lower. Okay. That seems like that's about it. Let's look at the experiments. So they're experiments. Basically I actually want to go down here to this industry, sorry, not the industrial control process, but to the mojo co experiments. So these are kind of continuous control tasks. You might have seen it. There's some like one is a, a, the ant here is basically this 3d and is like a blob and it has I think four legs and each leg has two joints. And it just needs to walk as far as possible or reach some sort of goal. And the half cheetah is like a 2d thing where I think it's something like this. It also has these two legs and it's supposed to walk forward and not fall over. And you can put force basically on each of the, of the joints here. So you see that their baselines are Gaussian processes. And this pets thing is a previous baseline to do, to also do model based control with a learned model. And here they, there's is the main, their main one is the red one. And as you can see that it goes much faster. Well it basically outperforms the rest in these high, in these more complicated tasks. And then card pole or something like this is, is lower dimensional, easier tasks. And you can see that at least it does not hurt. Now they make, they say here something they don't, they don't show in the plots. They say that if you let this run for a while, then basically the, their method doesn't make any improvement anymore. Whereas the baseline methods will sort of at some point surpass it. And the reason that is, and I'm not sure if it's on this exact task, but they mentioned that which it's, it's I respect so far is because they say since we only plan where we know, where did I draw it? Since we only plan where we know, we basically do much less exploration than others. We kind of stick to what we know when we plan. So inherently we do less exploration and in our conversation with Hari, he basically said this, this is intended. And the base, the intention is that you want to do your planning where you know, and then explicitly add a component that does exploration. So you have control over, so you can basically say, huh, I, I've never been here sort of. Now you would be in an exploration phase, you would explicitly go there rather than intermingle your planning with your exploration and basically rely on your planning to screw up and you're exploring. Because if your plan, if you're planning never screws up, then you won't explore either, right? Then you will always reach your goal or your planning will always be correct. And these other methods that don't have this explicitly, they explore every time their planning screws up and you don't want that. You want your planning to be as good as possible. And they do that by sticking to what they know. And then they the next step, which is not in this paper would be to add an explicit exploration policy to reach areas they've never reached before. Okay, so that's the reason why they don't ultimately reach the best accuracy, but they do reach a the initial accuracy much faster than the other tasks, because they plan better. They have a long discussion here of what still problems are like local minima or the planning horizon problem, open loop versus closed loop compounding errors in planning. But I'm going to leave this out for now. And I thank you for being here. I very much invite you to check out the paper for more details. It's pretty cool, pretty easy to read, actually. It's very written very well. And with that, see you next time. Bye bye.
 Hi there, today we're looking at regularizing trajectory optimization with denoising autoencoders
 by Renu Bonet, Norman DiPaolo and others of various places, but a lot of the people are
 from Curious AI.
 And we actually had a discussion with Hari, who is the CEO of Curious AI.
 And this was on our Machine Learning Street Talk podcast.
 So this is another YouTube channel for those of you who don't know, where every week or
 so we try to have either an interesting discussion or a guest, like sort of an interview or talk
 about comment on the talk.
 So if it is not out yet, I'll link to it as soon as it comes out.
 But if you're watching this video later, make sure to check out our conversation with Hari
 because it was absolutely fantastic.
 And in general, if you like videos like this, consider subscribing, liking, sharing if you're
 still here at the end and liked it.
 Okay, so this paper on a high level deals with model based reinforcement learning.
 Model based reinforcement learning means that you are using a model of the world to do reinforcement
 learning.
 So in essence, if you have your reinforcement learning setup where you are an agent, and
 you have to interact with the world, you have to do so in many steps in like a round trip
 fashion.
 So you put an action you act and the world gives you back an observation.
 And you have to act in the world over and over and over such that you will be able to
 maximize your reward.
 Now what is model based reinforcement learning?
 Model based reinforcement learning basically means that the agent here has internally
 a model of the world.
 So it sort of understands how the world works.
 Situations where you have a accurate model of the world are things like chess.
 So in chess, the rules are very clear, you know how the world's going to behave if you
 perform a certain action.
 But in real world applications, it's very, very hard to actually make a model.
 So people usually rely on learned models.
 So what does it mean you basically learn a neural network that tries to predict how the
 world is going to act.
 So this here is going to be a deep neural network that you learn from what you see in
 the world.
 Now trajectory optimization basically means that you are now you now have this world model,
 and you use it to look ahead, as I said, so you are in the state like here, and you can
 do let's say three different actions.
 And you use your world model here, world.
 And you see, you think how's the world going to react if I do either of those three things,
 and then you get into three different states.
 And then again, you after each one, you consider three actions, three actions here, three actions
 here, and so on.
 So ultimately, you're going to kind of have an overview over a planning horizon, which
 here we call H, you kind of look ahead a couple of steps, or there are various ways of doing
 this.
 But ultimately, you will basically find that this this path here is really good.
 So I think I'm going to take this as a first action.
 So trajectory optimization considers finding the best green path here in this tree of possibilities
 that your your world model gives you.
 Okay.
 Now, what does what do these people say?
 They say this procedure often suffers from exploiting inaccuracies of the learned model.
 What does that mean?
 That basically means that if I have a world model, and it is not accurate, then it is
 basically, basically the thing that tries to find the best green path here, the optimizer
 is sort of trying to find the the best path against this world model.
 Now, if that world model is inaccurate, that can lead to devastating consequences.
 So what do we mean by this?
 I'll give you an example.
 If you have a room, right, and the room is, let's take our classic room like this.
 And you are here and you would like to go here.
 And so you're a reinforcement learning agent, you do some exploration, right, you explore
 a bit here, the next episode, you might go here, and you might go here, and so on.
 And over time, in this framework, you're going to build a model of the world.
 So at the beginning, we won't tell you how these rooms look, you have to discover it
 by yourself.
 So maybe at the beginning, we only tell you, there's these four walls, the rest you have
 to figure out.
 So on and on, you're going to fill in your blanks, you do your first explorations, and
 you've had there's a bit of a wall here, right.
 And there might be some wall here, I crashed into that, right, you're going into here,
 you crash into a wall, you saw there's a wall here and here, there's a wall here, you go
 maybe here, oh, there's no wall.
 So you go further, there's no wall anywhere here, you crash here, okay, we already knew
 there's a wall.
 Maybe you crash here.
 All right, so right now, you have, okay, you go here, you have a model of the world in
 this situation, where there's a wall here, a wall down.
 And if you now try to do trajectory optimization, remember, you have to go from here to here.
 If you try to do trajectory optimization, what is it going to turn out?
 It's going to turn out like, look, there you go.
 That works just fine.
 And that's not because you're so good at planning.
 I mean, you are good at planning, but because your model is inaccurate here, because it
 has never seen this, your entire training distribution that you trained the world model
 on, only explored the area over here.
 All right, so you see how the more efficient this planning algorithm is, like the blue
 arrow, the thing that finds the blue arrow, the more efficient that is, the more consequential
 it is when your learned world model has mistakes, because it will exactly exploit these mistakes
 in order to get the shortest path possible or the highest reward in that case.
 And this, they call this like almost an adversarial attack on the world model, which is a pretty
 good way of framing it.
 They propose actually to solve this problem.
 They say we propose to regularize trajectory optimization by means of a denoising autoencoder
 that is trained on the same trajectories as the model of the environment.
 We show that the proposed regularization leads to improve planning with both gradient based
 and gradient free optimizers.
 We also demonstrate that using regularized trajectory optimization leads to rapid initial
 learning in a set of popular motor control tasks, which suggests that the proposed approach
 can be useful tool for improving sample efficiency.
 So in essence, what do they do?
 They basically say, okay, we want to regularize this using a denoising autoencoder.
 And I think it's best if we if we look at the at the math for doing this.
 So the math here starts off as follows, saying you want to learn a world model.
 This is F here.
 F is the world model.
 It takes in a state and an action and it gives you the next state or an approximation to
 it.
 And the parameters here indicate that this is some sort of function that you learn like
 a deep neural network.
 You can do this in fully or partially observed environments.
 Now when you plan, what you want to do is you say I have a planning horizon H, right?
 Then I have a reward function, and the reward function is going to give me a reward for
 each state action pair.
 So if I'm in a state and I do a certain action, I'm going to get some reward.
 This could be you have reached the target or this could be you know how much money you've
 collected or whatnot.
 So you're going to look at horizon H, you're going to look H steps into the future, and
 you want to maximize the sum of all the rewards here.
 So in the limit, this reduces to simply like, for example, reaching the target in our rooms
 case if H is infinite.
 But you can consider a lower planning horizon.
 So you want to find the action sequence that maximizes this reward in the future.
 And now this reward relies on your environment model.
 So here's the algorithm.
 First you collect some data, okay, that's how you start off.
 That train the dynamics model, the world model, using the data you've already collected.
 Then for each time step T, you want to optimize this trajectory.
 So you want to find the best next action sequence and take the first action, implement the first
 action and get the new observation.
 And do you do this in a loop until the end and at the end you say add this data to D.
 So that's what you do.
 You use your world model to get the best action sequence.
 That's how you optimize the trajectory.
 And then at the end of the episode, you've done an episode, right?
 You went somewhere, you put all of this into your training data to make the world model
 better.
 Something to note here is that the world model will only learn about things that you have
 done, right?
 So there is kind of an interaction effect.
 That's the green area here.
 The world model only knows the paths, the world model only can accurately estimate the
 world where you have been.
 And that's going to turn out to be the entire problem because these blue arrow finder can
 now go away from that.
 That's explained here.
 Potential inaccuracies of the trained model cause substantial difficulties for the planning
 process.
 Rather than optimizing what really happens, planning can easily end up exploiting the
 weaknesses of the predictive model.
 Planning is effectively an adversarial attack against the agent's own forward model.
 This results in a wide gap between expectations based on the model and what actually happens.
 And they have this example here where it's like an industrial control process.
 And what you have to imagine, there's like some sort of a container here with a liquid
 in it.
 And there are two pipes that lead to this container, pipe one and pipe two.
 And there are valves here.
 So there's this valve right here and there's this valve right here.
 So these are valve one and valve two.
 And there is also an output pipe right here and that's another valve right here.
 So you can control these three valves, the two inputs and one output.
 And you have to somehow optimize the reaction in here.
 So this is a chemical reaction made up out of the two liquids that flow in here.
 And you have to somehow optimize a property of that.
 And that's highly nonlinear and has maybe like time shifts.
 So when you open a valve, it's going to take a while and then it's very nonlinear.
 And then you are not supposed to break the pressure limit.
 So you have to also outflow some stuff.
 And if you just do this with a learned model, it looks like this.
 So first of all, here is a classic controller.
 People have been doing this stuff in industry and they basically build controllers for it.
 And you can do that and that works out really okay-ish.
 As you can see right here, this is the product rate, what you're supposed to optimize.
 And you see some sort of a smooth...
 You're supposed to actually bring it to this dashed line right here.
 And this is some sort of smooth thing, right?
 And you're supposed to, I guess, bring the pressure here and the A in purge.
 I don't know what these quantities are, but you're supposed to bring them to the dashed
 line and it's very nonlinear and very time dependent.
 So that works.
 And you see here kind of the smoothness by which the variables are manipulated.
 Now, if you just learn a world model and then do this trajectory optimization, basically
 this is some sort of a planning-based reinforcement learning with a world model.
 You see right here, it works, but it's super jittery.
 The pressure spikes here and apparently this here is a pressure limit.
 So it spikes the pressure limit.
 And you can see that the manipulated variables are up and down and up and down and up and
 down because at each step, it basically completely overestimates its potential reward.
 With things like, wow, this is really good, but all it does is find a weakness in the
 model and not a really good action per se.
 Now with their method to already take it away, you can see that now the control task super
 smoothly and very quickly converges to these optimal things.
 And you can see that the variables being manipulated are also rather smoothly manipulated.
 And that's an indication that the model is accurately estimating their rewards.
 Okay, so how do they do it?
 Via what they call trajectory, via regularization of trajectory optimization.
 So in essence, what do we want to regularize here?
 There are many things we could do to solve this, but the way this paper goes is they
 say we not only do we want the most return, we also want a high log probability of our
 taken path.
 So this here, as you can see, this is observation action and so on, observation action.
 So this is the future.
 This right here is the future.
 So this sequence here is what is going to give me the reward right here.
 So G is also dependent on these things, but it's not said explicitly here.
 So G is dependent on your plan.
 Maybe let's not call this the future.
 This is the plan.
 This is the plan you came up with.
 So this is directly going to influence G and G is the reward you're going to get under
 your model.
 But also you want the log probability of the plan itself to be high.
 Now there, I think there is a bit, there is something missing here and that is conditioned
 on your training distribution right here.
 And I think that's a actually rather crucial part.
 Now that's, that's the KL thing.
 So this is conditioned on your training.
 So what you want is you want the plan to be basically in your training distribution.
 So you, you want what you, you want your plan that you're going to execute.
 If that is actually part of your training data set, then you know, I have already executed
 this once before and it's reasonable to assume that therefore my world model has learned
 from this experience and is going to give me an accurate reward.
 If we go back to our rooms example, then up here somewhere, if we go back to our rooms
 example, right, you see that anywhere in the green area where I have already explored the
 world model is fairly good, right?
 It's going to give me accurate reflection of the world.
 But as soon as it go outside the green area, it is not.
 And inside the green area is basically where my training data is.
 Now if I in the future actually take a path here, crash into a wall right here, right?
 You saw in the algorithm at the end of an episode, I'm going to add my trajectory to
 the training data for the world model.
 So this green part here expands to include that.
 And now if I go here again, if my plan goes there again, now I can trust the world model.
 But also now it has it is actually correct because it has a wall here.
 So you see that the regularization basically you not only do I want the biggest reward
 possible under my world model, I also want that the plan that I'm about to execute is
 has a high probability under my training distribution.
 Okay.
 And the way we do this is by denoising auto encoders.
 We want the log probability here to be high and you do this via a denoising auto encoder.
 What's a denoising auto encoder?
 A denoising auto encoder is basically so if you have, for example, an image and the image
 is of a trusty cat whiskers and a denoising auto encoder is an unsupervised method where
 you have it's basically an auto encoder.
 So there is a bunch of layers compressing to a hidden representation, then uncompressing
 it again.
 Okay.
 And at the end, you want to output the same as at the beginning.
 So it's basically an auto encoder.
 But the special part about the denoising auto encoder is that first, you take your input
 and you know, you put some noise on it.
 So that could mean could mean anything here.
 But here, what they do is they do they make some Gaussian noise on it.
 Now, I can't really draw Gaussian noise here, but it would be kind of convolved with Gaussian
 Gaussian noise.
 So I'm just going to add some noise like this.
 So noise, noise, noise.
 So there's some noise, you see, and then you feed that.
 That's now what you feed in here.
 And the algorithm is supposed to reconstruct this, this original image.
 So the algorithm is basically supposed to take away the noise, it doesn't see the original
 image, but it's supposed to produce it.
 And you do this with your training data.
 What does that mean?
 Ultimately, for our trajectory optimization, it means that if I have a trajectory that
 I did before, and it maybe goes here, right?
 What I can do is I can make a noisy version of it, which would be the black one right
 here.
 So I put some noise on it, some noise.
 Right, it's kind of the same, but okay.
 And the denoising autoencoder is supposed to give me back the red one.
 This will simply give me some sort of a probabilistic model of my training distribution.
 So they go through the math here and show that these denoising autoencoders actually
 naturally output this log probability, sorry, the gradient of the log probability.
 Because optimal denoising theory says that for zero mean and Gaussian noise, the optimal
 denoising function, the optimal denoising function for zero mean Gaussian corruption
 is this thing right here.
 So it is, if you give me X and you tell me X has been corrupted by zero mean Gaussian
 noise of size sigma n, then the best, and you simply tell me, give me back the original
 image, the best thing I can do is to take what you gave me and add this gradient of
 the log probability of X if I can, if I have a model of the log probability.
 So that's the best thing I can do.
 And that's the best denoising function.
 And now you have to think a bit in reverse.
 If we train a denoising autoencoder, that is going to approximate this best function
 that there is.
 So we know that the best possible denoising function is this, we train a denoising autoencoder,
 which in the optimal case is going to converge to the best denoising function.
 So if we then reformulate and we do denoising autoencoder of X minus or X tilde minus X
 tilde, that is divided by the standard deviation.
 Sorry, the variance.
 That is going to give us this quantity right here, the gradient of the log probability.
 And the gradient of the log probability of X is exactly what we need to run gradient
 descent on our function.
 So here is our function again, G plus this regularization.
 Now they don't regularize over the entire future, but over these windows.
 But in essence it's G plus the log probability of your plan.
 If you take the gradient of that, of course you take the gradient of the sum.
 So it's the gradient of G plus the gradient of the log probability with respect to the
 actions.
 And here simple application of the chain rule will tell you that you have to propagate through
 the input, through the X.
 And you need this quantity.
 The gradient of the log probability with respect to its inputs.
 Now as we just saw, the optimal denoising autoencoder is going to output that thing.
 So if we train a denoising autoencoder and we suppose it reaches a good accuracy, then
 we can obtain this quantity basically for free.
 And that's the entire trick here.
 So in essence, what does it mean?
 In essence what it means is that if we are in our room again, and we have our partial
 model of the world, let's say we have this model, because we are here and all we've ever
 explored is these things right here.
 Now when I go and do my trajectory optimization, and my trajectory optimization wants to go
 here, I simply say, no, I don't know that, I haven't seen that yet.
 You can only plan basically within the space where we have already been.
 So you can plan like here.
 So here now there is of course, there is going to be some exploration, so some probability
 that you can go away a bit, but not too much.
 So in this case, it would result in the planning only to happen in spaces where we've actually
 been.
 So it might go here, and then here, because okay, here we haven't been anywhere.
 But then that would lead me to take the first step in this direction, and not in this direction.
 And if I take my first step in this first direction, then of course, I'm going to be
 already a bit on the correct path right here.
 Because if I take the first step into this direction, then after that, I'm going to have
 to, if once I crash here, I'm going to have to correct really hard.
 And that's exactly what's going to give you this super jittery control.
 Whereas if you only plan where you've already been, you won't, the probability that you're
 going to have to do like a 180 is going to be much, much lower.
 Okay.
 That seems like that's about it.
 Let's look at the experiments.
 So they're experiments.
 Basically I actually want to go down here to this industry, sorry, not the industrial
 control process, but to the mojo co experiments.
 So these are kind of continuous control tasks.
 You might have seen it.
 There's some like one is a, a, the ant here is basically this 3d and is like a blob and
 it has I think four legs and each leg has two joints.
 And it just needs to walk as far as possible or reach some sort of goal.
 And the half cheetah is like a 2d thing where I think it's something like this.
 It also has these two legs and it's supposed to walk forward and not fall over.
 And you can put force basically on each of the, of the joints here.
 So you see that their baselines are Gaussian processes.
 And this pets thing is a previous baseline to do, to also do model based control with
 a learned model.
 And here they, there's is the main, their main one is the red one.
 And as you can see that it goes much faster.
 Well it basically outperforms the rest in these high, in these more complicated tasks.
 And then card pole or something like this is, is lower dimensional, easier tasks.
 And you can see that at least it does not hurt.
 Now they make, they say here something they don't, they don't show in the plots.
 They say that if you let this run for a while, then basically the, their method doesn't make
 any improvement anymore.
 Whereas the baseline methods will sort of at some point surpass it.
 And the reason that is, and I'm not sure if it's on this exact task, but they mentioned
 that which it's, it's I respect so far is because they say since we only plan where
 we know, where did I draw it?
 Since we only plan where we know, we basically do much less exploration than others.
 We kind of stick to what we know when we plan.
 So inherently we do less exploration and in our conversation with Hari, he basically said
 this, this is intended.
 And the base, the intention is that you want to do your planning where you know, and then
 explicitly add a component that does exploration.
 So you have control over, so you can basically say, huh, I, I've never been here sort of.
 Now you would be in an exploration phase, you would explicitly go there rather than
 intermingle your planning with your exploration and basically rely on your planning to screw
 up and you're exploring.
 Because if your plan, if you're planning never screws up, then you won't explore either,
 right?
 Then you will always reach your goal or your planning will always be correct.
 And these other methods that don't have this explicitly, they explore every time their
 planning screws up and you don't want that.
 You want your planning to be as good as possible.
 And they do that by sticking to what they know.
 And then they the next step, which is not in this paper would be to add an explicit
 exploration policy to reach areas they've never reached before.
 Okay, so that's the reason why they don't ultimately reach the best accuracy, but they
 do reach a the initial accuracy much faster than the other tasks, because they plan better.
 They have a long discussion here of what still problems are like local minima or the planning
 horizon problem, open loop versus closed loop compounding errors in planning.
 But I'm going to leave this out for now.
 And I thank you for being here.
 I very much invite you to check out the paper for more details.
 It's pretty cool, pretty easy to read, actually.
 It's very written very well.
 And with that, see you next time.
 Bye bye.