AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper what is deep learning deep learning tutorial introduction to deep learning reinforcement learning imitation learning uc berkeley sergey levine sergey levine reinforcement learning pieter abbeel pieter abbeel reinforcement learning walk and punch learning from demonstration amp adversarial motion priors physics based reinforcement learning 3d reinforcement learning
#reiforcementlearning #gan #imitationlearning

Learning from demonstrations is a fascinating topic, but what if the demonstrations are not exactly the behaviors we want to learn? Can we adhere to a dataset of demonstrations and still achieve a specified goal? This paper uses GANs to combine goal-achieving reinforcement learning with imitation learning and learns to perform well at a given task while doing so in the style of a given presented dataset. The resulting behaviors include many realistic-looking transitions between the demonstrated movements.

OUTLINE:
0:00 - Intro & Overview
1:25 - Problem Statement
6:10 - Reward Signals
8:15 - Motion Prior from GAN
14:10 - Algorithm Overview
20:15 - Reward Engineering & Experimental Results
30:40 - Conclusion & Comments

Paper: https://arxiv.org/abs/2104.02180
Main Video: https://www.youtube.com/watch?v=wySUxZN_KbM
Supplementary Video: https://www.youtube.com/watch?v=O6fBSMxThR4

Abstract:
Synthesizing graceful and life-like behaviors for physically simulated characters has been a fundamental challenge in computer animation. Data-driven methods that leverage motion tracking are a prominent class of techniques for producing high fidelity motions for a wide range of behaviors. However, the effectiveness of these tracking-based methods often hinges on carefully designed objective functions, and when applied to large and diverse motion datasets, these methods require significant additional machinery to select the appropriate motion for the character to track in a given scenario. In this work, we propose to obviate the need to manually design imitation objectives and mechanisms for motion selection by utilizing a fully automated approach based on adversarial imitation learning. High-level task objectives that the character should perform can be specified by relatively simple reward functions, while the low-level style of the character's behaviors can be specified by a dataset of unstructured motion clips, without any explicit clip selection or sequencing. These motion clips are used to train an adversarial motion prior, which specifies style-rewards for training the character through reinforcement learning (RL). The adversarial RL procedure automatically selects which motion to perform, dynamically interpolating and generalizing from the dataset. Our system produces high-quality motions that are comparable to those achieved by state-of-the-art tracking-based techniques, while also being able to easily accommodate large datasets of unstructured motion clips. Composition of disparate skills emerges automatically from the motion prior, without requiring a high-level motion planner or other task-specific annotations of the motion clips. We demonstrate the effectiveness of our framework on a diverse cast of complex simulated characters and a challenging suite of motor control tasks.

Authors: Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, Angjoo Kanazawa

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hey, yo, where's my money? Well get me my money. Alright we're gonna get into this video in a second. Today we're going to look at AMP, Adversarial Motion Priors for Stylized Physics-Based Character Control by Xuebin Peng, Tsema, Pieter Abil, Sergei Levine and Angchu Kanazawa. And this paper is in the domain of control and reinforcement learning, but it's with a little bit of a twist. So on the high level, this paper trains an agent, a physical agent, as you can see here, to perform some sort of goal in the case on the right, it's walking up to a target and punching the target. But to do so in a certain style, and the style is provided by an expert data set or a demonstration data set. So the technique that the paper presents mixes two things, it mixes goal achieving reinforcement learning, and it also mixes adherence to a given style. And the adherence to a given style, that's going to be the adversarial part right here because that's learned in an adversarial way. The mixture of the two at the end looks pretty, pretty cool. So the setup right here is a setup of goal achieving and imitation learning as we have already outlined. And the way it works is the following, there is going to be a task and the task can be, you have to reach a goal, the task can be you have to punch something, you have to overcome some obstacles, and then reach a goal. Any anything like this is a task. So the goals are fairly high level and they are given, obviously by a reward function. So you place the agent in an environment and there is a reward function. By the way, the agent here is as we already also said, is this sort of physical agent that is going to have some sort of a 3d structure. There is going to be joints that it can move. There's a joint here and one here usually. So and there's a head. The agent is this physical thing and it's in a physics simulation and each one of these joints, it can move kind of independently, sometimes free as a as a ball, sometimes it's restricted. It's modeled very much like a human. There are other I believe other models such as a T Rex, which of course work differently. But you have this agent and the agent is supposed to reach a goal like somewhere over here, there's a little flag, there's a goal. And the way the agent can interact with the world is by putting force on any of these joints. So it can move these joints in pretty specified ways. And that constitutes the actions. So the agent will observe the state and the state here is given mostly by it can observe how all the joints are currently the velocity of the of the joints or of the of the individual parts of itself in relation to itself. So it can sort of feel itself. And it also knows in which direction and generally how far away the target that it needs to reach is. So that's the observation space, the action spaces, it can affect these joints. And the reward function is often modeled in accordance with the goal. So the reward function for walking to some goal might simply be you get reward if you are closer to the goal. Okay, so this encourages the agent to go over there. So we work with quite dense rewards right here. Because I guess the fundamental problems of reinforcement learning aren't exactly the point here. The point here is, can you teach these things to achieve a goal while maintaining a certain style? Now, this is the the task and the environment. In addition to that, you do get a data set. And the data set is demonstrations of a certain nature. So this is not necessarily demonstrations of how to reach the goal. It can be any sort of demonstrations. So usually when people do sort of imitation learning or learning from demonstrations, there is a bit there are some requirements. If you want to do pure learning from demonstration, of course, the demonstrations need to be how to achieve the goal. And that we don't we don't have that here. In other cases, you do need the sort of policy or the action of whoever performed the data set. So don't need that here. Our goal is simply going to be we have to reach the task while while sort of adhering to the data set in a way. And this way, we're going to define in a second. So the data set you can imagine, I think there is a good demonstration down here, you can imagine the data set to give you sort of the style of movement. So in one data set, you can have running movements and walking movements. And in another data set, you could have these movements that were just the these actors walk like zombies. And the goal here is to combine the style of the data set with reaching the goal. Okay, so the combination would look like a zombie walking to the goal, which adheres to the zombie walk in the data set, and the goal in specified by the task. Okay, naturally, you're, you're going to model this as two different reward signals. So there's the reward signals of how much you reach the goal. And there is the reward signal of how well you adhere to the style in the data set. The reward goal right here is modeled by classic reinforcement learning. So this is very much very, very classic. Where do we have it? So you would simply train, I don't even think it's it says here, it's update G and D, yada, yada, yada. So this is a policy gradient method reinforcement learning, which means that you do have a policy function, which takes in a state and maybe a history, and it will give you an it will give you an action. And with that, you also train a value function that takes a state and will give you a value for that state. Now, the value function is purely for training the agent, because you do you do advantage estimation with this value function, but essentially, this is a standard policy gradient method that you train this part is lower part of the this lower part of the thing on sorry, you actually train the whole thing on this reward. But the bottom part you can imagine is it a reward comes from reaching a goal. The top part gives also gives you a reward. Okay. And yes, I want to reiterate, both of these rewards are used to train the policy and the value in a policy gradient fashion. So both rewards ultimately are in this standard advantage estimation reinforcement learning setting. However, the top reward is calculated differently than simply do you reach the goal, the top reward is a measure of how close you are in style to the data set. And that's given by this motion prior. And the motion prior is given by a GAN by a generative adversarial network. And I'm trying to, to find the formula here. I think this here is the the best description of it, though it's just a formula. So a generative adversarial model, I'm pretty sure you're you're all aware, there is a data set right here, there is a generator right here, the generator gets some random noise as an input, it outputs a sample x from the data set, you get a sample x prime or a mini batch. And then both of these, or these either of these goes into the discriminator model. And the discriminator has to decide for any sample, is it real? Or is it fake? So the way this generative adversarial network approaches the problem of specifying which motions are real and which ones are not, is by looking at transitions. So the data set here is not images or so like you're used to in a regular GAN, but the data set is transitions. What does that mean? So in every situation, your humanoid or whatnot is here, and the goal is over here. And this is one state, this is s. And then the agent takes an action, okay, the action could be please lift one leg. And how does that evolve? So the new agent would be kind of here, shifting the weight a little bit and lifting one leg. Okay, so this would be one action, which would lead to a new state s prime. So you have three quantities, you have the state, you have the action that the agent took, and you have the new state s prime. Now you could parameterize the transition either using state and action, or state and next state. The paper here does state and next state for the reason that in the data set, in the data set that you get right here, you do not have the action available, you can probably guess it, but you do have the state and the next state. This data set can come from anywhere it can come from human demonstration, it can come from key frames made by a 3d artist, or maybe another agent that has already solved the problem. Therefore, you don't always have the actions available. So a transition is going to be specified by a state and a next state. And the transitions from the data set are transitions that you observe in the real world. So these are state next state pairs that you observe in the real world. And the generator, the generator essentially outputs state next state pairs. Now this generator isn't a generator in a like in a classic adversarial network. But this here is generated by your policy interacting with the environment, right? So here's your policy, it interacts with the environment. And the environment gives you the state and in the next step, it gives you the next state, right? So by interacting with your environment, you do get state next state pairs, these are essentially your generated pairs. And the discriminator is trained to discriminate between whether or not a transition is from the real data set, or whether it has been generated by your agent. Now of course, this whole system isn't backpropagatable. And that's why you do train it using reinforcement learning. So the reward, the usual backpropagation signal that you would have in a generator right here, you can't do that. That's why you simply take the output here, the loss of the discriminator as a reward for the for the policy right here. So in this case, the policy using policy gradient is trying to fool the discriminator into thinking it into it thinking that the transitions that it generates come from a real data set. While the discriminator at the same time is always trained to differentiate between the true data set and the transitions that the policy generates. Alright, so that gives you a reward signal for the policy. And the other reward signal comes simply from the environment as we've already stated. So these two rewards are then combined with each other and used to train the policy, the discriminator itself, as we already seen is trained. So this thing here is actually the discriminator, this more motion prior is trained one hand from the data set. And on the other hand, from the from the policy generating actions and generating transitions through the environment. Alright, I hope that is a bit clear right here. So there are many components to this, but two are important, the policy, which tries to at the same time reach a goal and fool the discriminator. Those are two rewards, there are two rewards are combined. And on the other hand, the discriminator itself simply gets transitions from the data set and gets transitions from the policy environment interaction and tries to train itself to pull the two apart. So it's a it's a classic two player game. And yeah, that that is what you're used to from a GAN. Alright, and that's essentially it for this thing. Here is the algorithm we generally initialize everything there is a replay buffer like in a classic reinforcement learning which stabilizes training quite a bit. I also mentioned the value function which is used for the advantage estimates of policy gradient. So you for M steps, you collect trajectories using the policy you already have, then you feed the transitions to the discriminator right here. Now this here is a feature function of the state. So you only they have special feature functions, which make the this problem easier. There's a lot of expert knowledge going into how you build the features, how you represent the environment and so on. So it's not quite trivial, but I don't I don't want to go too much into that. You do calculate the style reward according to equation seven, equation seven is simply the discriminator. It's not the discriminator loss. So the discriminator loss is actually is this thing right here. They do use a square loss for the discriminator instead of a classic GAN loss. So the classic GAN loss would be this thing up here, where it's log D minus log one minus D. Yet they use this square loss that they found to work a lot better or least square loss. You can see the discriminator is trained to be close to one if the data comes from the real data set, which is capital M here. And it's trained to be negative one when it comes from the policy. So nothing stops the discriminator from spitting out any number like 15 or three. It's just trained in a least squares fashion to go to these numbers, which gives you a better gradient. So for these continuous control problems, often you have to go to least squares objectives, because which number is being output is often quite important rather than just a classification. And even here where it is actually a classification loss, right, which is surprising, but cool. And then the reward, you know, given a transition is calculated as so this is clipped at zero. So this is also between zero and one, as you can see here, if the discriminator says one, the reward is the highest, the reward is actually one. And when is the discriminator one, the discriminator is one if it thinks that the reward, sorry, that the transition comes from the real data set. So if the policy manages to produce a transition that the discriminator things comes from the real data set, it gets maximum reward. Okay. And if it also reaches the goal, it gets maximum reward from that part of the reward signal too. So the general encouragement that we give the policy is you should reach the goal in a matter that's consistent with the data set. So it should probably pick out things that do both, right, it could try to, it could try to switch between the two modes like, okay, let's do a little bit of data set, let's do a little bit of goal reaching, but it's probably better if it actually picks things from the data set or behaviors from the data set that also reach the goal in a matter consistent with the reward with the task reward. So the algorithm just to finish it goes on. And it says, okay, so this is the style reward. The true reward is given by a mixture, a weighted mixture between the style and the task reward and the weights you have to specify. And then we simply store these, this trajectory in our replay buffer. And then we use the replay buffer to update the discriminator. And we also use the replay buffer to update the value function and the trajectory according to policy gradient. They point out a few things that are important right here to their algorithm. One of them they find very important is this gradient penalty. So GAN training can be a bit unstable. And these gradient penalties, they are a way to stabilize this training. And they found that simply penalizing the norm of the gradient as it comes out of the discriminator is stabilizing the training right here. So this is one thing they've helped. This is one thing that they claim is helping them a lot to actually converge. And this tells you a little bit that it's still quite finicky. They talk a lot about the representation of the actions right here, the policy here in network architecture, the policy and value and discriminator functions. They are very simple multi-layer perceptron. So you can see like the mean of the policy function is specified by a fully connected network with two hidden layers consisting of 1024 and 512. Relu, Relu, consisting of Relu. Okay, I guess that's a fully connected layer with a Relu non-linearity followed by linear output. So the networks aren't super complicated right here. What's more complicated is the training procedure, the loss, the regularization constants and the reward engineering. So there is a lot of reward engineering happening right here. And that's what you find in the appendix. So the reward, for example, for going and punching something is threefold. So if you are far away, it's one reward. If you're close, it's a different reward. And if that target has been hit, it's a different reward, right? I guess the top line makes sense, but the others are sort of reward shaping the behavioral one. So you can see the agent to kind of approach the target fast, but then kind of slow down. And also, you know, if you look at something like dribbling, where there's a ball involved, there is a lot of reward shaping going on. Even in in target location, there is a lot of reward shaping going on, where you sort of encourage the agent to have certain velocities and so on. So this is important because of the experimental results that they show. And that's where we go back to the video. Where's the video? Right here. So keep in mind, their point is you're able to reach a goal in the style of the data set. So this is the simplest task they have. It's called target heading, and the goal is simply to walk or to go in a given direction at a certain speed. And the example clips they have are displayed on the right. So the example clips are of someone walking and of someone running. Yet there is not really a transition in the data set from walking to running. And the agent learns to this transition by itself. So their point is always, look, we have kind of simple things in the data set, we have the individual parts in the data set that the agent should do. But we never have the combination of all the things. And to kind of stitch these parts together, that's the powerful thing about this method, which is pretty cool. So here, you can see at the top right, there is a target speed. And all of these three agents are trained agents. And in the same manner, right, and they're all told to reach that given target speed. However, the agent on the left only has been provided with a data set of people just walking. The agent in the middle, the same, but it has only received a data set of just agents running. So no walking. And on the right, this agent has received a data set of agents walking and running. So you can see that as the target speed changes, the like if it's fast, the walker is not able to keep up when it's slow, the runner is not able to slow down. However, the agent that has the full data set available can not only match the speed and change its style according to the speed, it can it also learns the transitions from one to the other. And this these transitions are not in the data set itself. Okay, so the cool part about this method is it can sort of stitch together the appropriate behaviors from the data set. Even if you don't provide these specifically to solve the task. The Yeah, this is the t rex. I think this is just to show that you don't have use motion capture, but you can use it. You can learn from a provided data set of keyframe animation. And you can also see the there is nothing in the data set about reaching a goal. There's just kind of demonstrations of the t rex walking. And the method is able to adapt this walking style in concordance with reaching a goal. So you can see that the turning is much like the turning in the example clips. Whereas if you've ever seen things like this without without the the examples, these policies that these things come up with are quite weird. So here's a failure case. And so the difference between this method and other methods is other methods, such as this motion tracking in the middle, what they try to do is they try to match a given behavior from the data set as closely as possible. So this it's called motion tracking. Now there is some sophistication to it more than I'm saying right here. But essentially, you have a front flip on the left. And then the motion tracking algorithm tries to learn a policy such that the behavior is followed as closely as possible. Now, again, this is really good when you have the exact demonstration available from what you want to do. It's not so good if you if what you have available as demonstrations is not isn't really what you want to do is just sort of some demonstrations. But there are failure cases, of course, if you want to copy exactly. So if you want to do a front flip, and by the way, the reward function here is how closely you match the motion from the reference motion. So that's the reward function. However, motion tracking does more than that motion tracking really tries to track the motion itself. While this method here would only get the reward of tracking the motion. And you can see it doesn't manage to to actually learn it more like doesn't try it tries to not fail. So it reaches the same end position and that's sort of good enough for it. So there is a Yeah, there is a trade off right here. It's probably also given by how much you weigh the different components. So here you have a data set of agents walking and agents waving. And then what you want to do is you want to have a agent that walks in a direction while they wave the arm or why they they lift the arm or something. So at the left, you can see if you only have a data set, if you only have a data set of the waving agents, it's really struggling moving forward, right that the walking it learns it has no demonstration of walking. So that's a struggle. If you only have the walking demonstration in the middle, then it doesn't really track the arm movement where it should even though there is a reward for it, right? Only Yeah, on the right, I mean, this is somewhat somewhat, but it is kind of able to to interpolate. So if you if you want to check out this video, there is another one that actually explains the paper in a short form. This is from from SIGGRAPH. Go check it out. They do have more sophisticated behaviors. So on the bottom here, you can, for example, see the obstacle run, leap and roll. So the data set contains demonstrations from all of those things, but not the things in conjunction with each other. In this here, at least what they describe in the text in this, this right here, what they have in the data set is demonstrations of walking and demonstrations of getting up from the ground. And whenever so the agent learns that whenever it falls over right here, that it can get up faster if it kind of does this rolling motion right here. So this was nowhere in the data set, but because the agent wants to go to a get up state, both because that will go it that will make it go towards a goal. And also because that matches behavior in the data set, it will learn this rolling motion as it falls down in order to get up again. So that is that's pretty cool. Also in this strike and punch example, the data set apparently only contains agents walking or agents punching, it never contains agents walking, and then punching. So the transition that you saw at the beginning is a learned behavior that wasn't in the data set. So that's, I think it's a it's a pretty cool application of and a combination of two things of adversarial learning and of of learning sorry, not from demonstration because that's adversarial learning of learning to reach a goal. And it's a good Yeah, it's a good demonstration of how you can combine the two they have a lot of ablations where they sort of show that the impact of the data set makes a big difference. I mean, you've seen this in the demonstrations. But also here you can see that again in a graphical form. So the locomotion data set contains both demonstrations of walking and running, while the walk or the run data set only contains demonstrations of either and the here is the target speed versus the average speed that the agent does. Now if you only have a walking data set, the agent no matter the target speeds, the agent will always kind of stick to walking. And if you have the running data set, it can run faster up here. But if you want it to slow down, it can't really run slower than you require. Only when the data set contains both things, can it transition between the two and actually match the running or walking. So what do we think of this? My opinion is it's probably it's very cool. It's a good way of sort of bringing demonstrations into the picture without manually tracking the demonstrations or copying exactly. So you just give some suggestions to the algorithm of what it could do. And you do that in form of a data set, which is something that I like, because it's not as invasive as telling the agent, you know, you need to match the joint movements and so on of the of the demonstration. This enables demonstrations to come in that are of a much broader range, not necessarily reach the goal, not necessarily even have a goal in mind. So that's cool. On the other hand, I think it's pretty finicky because you have to strike the trade off parameter between the two rewards quite cleanly, or clearly for your goal. Because we've already seen right at some point, the agent won't reach the goal anymore. If if this reward here, if the reward of the style is too high, we already saw this if you have a data set of just running, the agent will simply neglect the goal, it won't go slower than, you know, the kind of the slowest run or demonstration or a little bit slower than that, it just won't change its policy because it needs to match the data set. And the this balance seems to be quite, quite a important hyper parameter. And that also makes the provided data set here quite an important thing to to have available. So which data set you provide is also quite important. And lastly, the tasks themselves or the reward of the goal directed task nature, or in this paper, extremely engineered. And that's what I want to come back here lastly to so what they tout, for example, in this walk and punch thing, they say, oh, when the agent is far away, it runs towards the target. But if it's close, it only it slows down. And then when it's really close, it punches the target. And it sort of learns to combine these different skills. But and which is cool, right, because the transition wasn't in the data set. But a big part of it combining these skills is because in the reward, you make the reward different, whether the agent is far away, or whether it's near, you can see that right here. So these things are reward shaped to a high degree to encourage these kinds of transitions to happen, which I think is not really practical in a lot of settings. So it's still to be seen how much this is of practical value in other reinforcement learning tasks where you don't have that available. And also in other reinforcement learning tasks, where maybe the reward is more sparse, and how that affects this thing, because essentially, if the reward is much more sparse and irregular, now you have a problem because now the style signal is much more prominent. And that's not necessarily solved by simply reweighing the style signal. So I'm excited to see what comes out of this line of work. Next, it's a pretty cool line, as I already said, it's a good application of GANs in a different field than images. And with that, let me know what you think in the comments. I'll see you next time. Bye bye.
 Hey, yo, where's my money?
 Well get me my money.
 Alright we're gonna get into this video in a second.
 Today we're going to look at AMP, Adversarial Motion Priors for Stylized Physics-Based Character
 Control by Xuebin Peng, Tsema, Pieter Abil, Sergei Levine and Angchu Kanazawa.
 And this paper is in the domain of control and reinforcement learning, but it's with
 a little bit of a twist.
 So on the high level, this paper trains an agent, a physical agent, as you can see here,
 to perform some sort of goal in the case on the right, it's walking up to a target and
 punching the target.
 But to do so in a certain style, and the style is provided by an expert data set or a demonstration
 data set.
 So the technique that the paper presents mixes two things, it mixes goal achieving reinforcement
 learning, and it also mixes adherence to a given style.
 And the adherence to a given style, that's going to be the adversarial part right here
 because that's learned in an adversarial way.
 The mixture of the two at the end looks pretty, pretty cool.
 So the setup right here is a setup of goal achieving and imitation learning as we have
 already outlined.
 And the way it works is the following, there is going to be a task and the task can be,
 you have to reach a goal, the task can be you have to punch something, you have to overcome
 some obstacles, and then reach a goal.
 Any anything like this is a task.
 So the goals are fairly high level and they are given, obviously by a reward function.
 So you place the agent in an environment and there is a reward function.
 By the way, the agent here is as we already also said, is this sort of physical agent
 that is going to have some sort of a 3d structure.
 There is going to be joints that it can move.
 There's a joint here and one here usually.
 So and there's a head.
 The agent is this physical thing and it's in a physics simulation and each one of these
 joints, it can move kind of independently, sometimes free as a as a ball, sometimes it's
 restricted.
 It's modeled very much like a human.
 There are other I believe other models such as a T Rex, which of course work differently.
 But you have this agent and the agent is supposed to reach a goal like somewhere over here,
 there's a little flag, there's a goal.
 And the way the agent can interact with the world is by putting force on any of these
 joints.
 So it can move these joints in pretty specified ways.
 And that constitutes the actions.
 So the agent will observe the state and the state here is given mostly by it can observe
 how all the joints are currently the velocity of the of the joints or of the of the individual
 parts of itself in relation to itself.
 So it can sort of feel itself.
 And it also knows in which direction and generally how far away the target that it needs to reach
 is.
 So that's the observation space, the action spaces, it can affect these joints.
 And the reward function is often modeled in accordance with the goal.
 So the reward function for walking to some goal might simply be you get reward if you
 are closer to the goal.
 Okay, so this encourages the agent to go over there.
 So we work with quite dense rewards right here.
 Because I guess the fundamental problems of reinforcement learning aren't exactly the
 point here.
 The point here is, can you teach these things to achieve a goal while maintaining a certain
 style?
 Now, this is the the task and the environment.
 In addition to that, you do get a data set.
 And the data set is demonstrations of a certain nature.
 So this is not necessarily demonstrations of how to reach the goal.
 It can be any sort of demonstrations.
 So usually when people do sort of imitation learning or learning from demonstrations,
 there is a bit there are some requirements.
 If you want to do pure learning from demonstration, of course, the demonstrations need to be how
 to achieve the goal.
 And that we don't we don't have that here.
 In other cases, you do need the sort of policy or the action of whoever performed the data
 set.
 So don't need that here.
 Our goal is simply going to be we have to reach the task while while sort of adhering
 to the data set in a way.
 And this way, we're going to define in a second.
 So the data set you can imagine, I think there is a good demonstration down here, you can
 imagine the data set to give you sort of the style of movement.
 So in one data set, you can have running movements and walking movements.
 And in another data set, you could have these movements that were just the these actors
 walk like zombies.
 And the goal here is to combine the style of the data set with reaching the goal.
 Okay, so the combination would look like a zombie walking to the goal, which adheres
 to the zombie walk in the data set, and the goal in specified by the task.
 Okay, naturally, you're, you're going to model this as two different reward signals.
 So there's the reward signals of how much you reach the goal.
 And there is the reward signal of how well you adhere to the style in the data set.
 The reward goal right here is modeled by classic reinforcement learning.
 So this is very much very, very classic.
 Where do we have it?
 So you would simply train, I don't even think it's it says here, it's update G and D, yada,
 yada, yada.
 So this is a policy gradient method reinforcement learning, which means that you do have a policy
 function, which takes in a state and maybe a history, and it will give you an it will
 give you an action.
 And with that, you also train a value function that takes a state and will give you a value
 for that state.
 Now, the value function is purely for training the agent, because you do you do advantage
 estimation with this value function, but essentially, this is a standard policy gradient method
 that you train this part is lower part of the this lower part of the thing on sorry,
 you actually train the whole thing on this reward.
 But the bottom part you can imagine is it a reward comes from reaching a goal.
 The top part gives also gives you a reward.
 Okay.
 And yes, I want to reiterate, both of these rewards are used to train the policy and the
 value in a policy gradient fashion.
 So both rewards ultimately are in this standard advantage estimation reinforcement learning
 setting.
 However, the top reward is calculated differently than simply do you reach the goal, the top
 reward is a measure of how close you are in style to the data set.
 And that's given by this motion prior.
 And the motion prior is given by a GAN by a generative adversarial network.
 And I'm trying to, to find the formula here.
 I think this here is the the best description of it, though it's just a formula.
 So a generative adversarial model, I'm pretty sure you're you're all aware, there is a data
 set right here, there is a generator right here, the generator gets some random noise
 as an input, it outputs a sample x from the data set, you get a sample x prime or a mini
 batch.
 And then both of these, or these either of these goes into the discriminator model.
 And the discriminator has to decide for any sample, is it real?
 Or is it fake?
 So the way this generative adversarial network approaches the problem of specifying which
 motions are real and which ones are not, is by looking at transitions.
 So the data set here is not images or so like you're used to in a regular GAN, but the data
 set is transitions.
 What does that mean?
 So in every situation, your humanoid or whatnot is here, and the goal is over here.
 And this is one state, this is s.
 And then the agent takes an action, okay, the action could be please lift one leg.
 And how does that evolve?
 So the new agent would be kind of here, shifting the weight a little bit and lifting one leg.
 Okay, so this would be one action, which would lead to a new state s prime.
 So you have three quantities, you have the state, you have the action that the agent
 took, and you have the new state s prime.
 Now you could parameterize the transition either using state and action, or state and
 next state.
 The paper here does state and next state for the reason that in the data set, in the data
 set that you get right here, you do not have the action available, you can probably guess
 it, but you do have the state and the next state.
 This data set can come from anywhere it can come from human demonstration, it can come
 from key frames made by a 3d artist, or maybe another agent that has already solved the
 problem.
 Therefore, you don't always have the actions available.
 So a transition is going to be specified by a state and a next state.
 And the transitions from the data set are transitions that you observe in the real world.
 So these are state next state pairs that you observe in the real world.
 And the generator, the generator essentially outputs state next state pairs.
 Now this generator isn't a generator in a like in a classic adversarial network.
 But this here is generated by your policy interacting with the environment, right?
 So here's your policy, it interacts with the environment.
 And the environment gives you the state and in the next step, it gives you the next state,
 right?
 So by interacting with your environment, you do get state next state pairs, these are essentially
 your generated pairs.
 And the discriminator is trained to discriminate between whether or not a transition is from
 the real data set, or whether it has been generated by your agent.
 Now of course, this whole system isn't backpropagatable.
 And that's why you do train it using reinforcement learning.
 So the reward, the usual backpropagation signal that you would have in a generator right here,
 you can't do that.
 That's why you simply take the output here, the loss of the discriminator as a reward
 for the for the policy right here.
 So in this case, the policy using policy gradient is trying to fool the discriminator into thinking
 it into it thinking that the transitions that it generates come from a real data set.
 While the discriminator at the same time is always trained to differentiate between the
 true data set and the transitions that the policy generates.
 Alright, so that gives you a reward signal for the policy.
 And the other reward signal comes simply from the environment as we've already stated.
 So these two rewards are then combined with each other and used to train the policy, the
 discriminator itself, as we already seen is trained.
 So this thing here is actually the discriminator, this more motion prior is trained one hand
 from the data set.
 And on the other hand, from the from the policy generating actions and generating transitions
 through the environment.
 Alright, I hope that is a bit clear right here.
 So there are many components to this, but two are important, the policy, which tries
 to at the same time reach a goal and fool the discriminator.
 Those are two rewards, there are two rewards are combined.
 And on the other hand, the discriminator itself simply gets transitions from the data set
 and gets transitions from the policy environment interaction and tries to train itself to pull
 the two apart.
 So it's a it's a classic two player game.
 And yeah, that that is what you're used to from a GAN.
 Alright, and that's essentially it for this thing.
 Here is the algorithm we generally initialize everything there is a replay buffer like in
 a classic reinforcement learning which stabilizes training quite a bit.
 I also mentioned the value function which is used for the advantage estimates of policy
 gradient.
 So you for M steps, you collect trajectories using the policy you already have, then you
 feed the transitions to the discriminator right here.
 Now this here is a feature function of the state.
 So you only they have special feature functions, which make the this problem easier.
 There's a lot of expert knowledge going into how you build the features, how you represent
 the environment and so on.
 So it's not quite trivial, but I don't I don't want to go too much into that.
 You do calculate the style reward according to equation seven, equation seven is simply
 the discriminator.
 It's not the discriminator loss.
 So the discriminator loss is actually is this thing right here.
 They do use a square loss for the discriminator instead of a classic GAN loss.
 So the classic GAN loss would be this thing up here, where it's log D minus log one minus
 D. Yet they use this square loss that they found to work a lot better or least square
 loss.
 You can see the discriminator is trained to be close to one if the data comes from the
 real data set, which is capital M here.
 And it's trained to be negative one when it comes from the policy.
 So nothing stops the discriminator from spitting out any number like 15 or three.
 It's just trained in a least squares fashion to go to these numbers, which gives you a
 better gradient.
 So for these continuous control problems, often you have to go to least squares objectives,
 because which number is being output is often quite important rather than just a classification.
 And even here where it is actually a classification loss, right, which is surprising, but cool.
 And then the reward, you know, given a transition is calculated as so this is clipped at zero.
 So this is also between zero and one, as you can see here, if the discriminator says one,
 the reward is the highest, the reward is actually one.
 And when is the discriminator one, the discriminator is one if it thinks that the reward, sorry,
 that the transition comes from the real data set.
 So if the policy manages to produce a transition that the discriminator things comes from the
 real data set, it gets maximum reward.
 Okay.
 And if it also reaches the goal, it gets maximum reward from that part of the reward signal
 too.
 So the general encouragement that we give the policy is you should reach the goal in
 a matter that's consistent with the data set.
 So it should probably pick out things that do both, right, it could try to, it could
 try to switch between the two modes like, okay, let's do a little bit of data set, let's
 do a little bit of goal reaching, but it's probably better if it actually picks things
 from the data set or behaviors from the data set that also reach the goal in a matter consistent
 with the reward with the task reward.
 So the algorithm just to finish it goes on.
 And it says, okay, so this is the style reward.
 The true reward is given by a mixture, a weighted mixture between the style and the task reward
 and the weights you have to specify.
 And then we simply store these, this trajectory in our replay buffer.
 And then we use the replay buffer to update the discriminator.
 And we also use the replay buffer to update the value function and the trajectory according
 to policy gradient.
 They point out a few things that are important right here to their algorithm.
 One of them they find very important is this gradient penalty.
 So GAN training can be a bit unstable.
 And these gradient penalties, they are a way to stabilize this training.
 And they found that simply penalizing the norm of the gradient as it comes out of the
 discriminator is stabilizing the training right here.
 So this is one thing they've helped.
 This is one thing that they claim is helping them a lot to actually converge.
 And this tells you a little bit that it's still quite finicky.
 They talk a lot about the representation of the actions right here, the policy here in
 network architecture, the policy and value and discriminator functions.
 They are very simple multi-layer perceptron.
 So you can see like the mean of the policy function is specified by a fully connected
 network with two hidden layers consisting of 1024 and 512.
 Relu, Relu, consisting of Relu.
 Okay, I guess that's a fully connected layer with a Relu non-linearity followed by linear
 output.
 So the networks aren't super complicated right here.
 What's more complicated is the training procedure, the loss, the regularization constants and
 the reward engineering.
 So there is a lot of reward engineering happening right here.
 And that's what you find in the appendix.
 So the reward, for example, for going and punching something is threefold.
 So if you are far away, it's one reward.
 If you're close, it's a different reward.
 And if that target has been hit, it's a different reward, right?
 I guess the top line makes sense, but the others are sort of reward shaping the behavioral
 one.
 So you can see the agent to kind of approach the target fast, but then kind of slow down.
 And also, you know, if you look at something like dribbling, where there's a ball involved,
 there is a lot of reward shaping going on.
 Even in in target location, there is a lot of reward shaping going on, where you sort
 of encourage the agent to have certain velocities and so on.
 So this is important because of the experimental results that they show.
 And that's where we go back to the video.
 Where's the video?
 Right here.
 So keep in mind, their point is you're able to reach a goal in the style of the data set.
 So this is the simplest task they have.
 It's called target heading, and the goal is simply to walk or to go in a given direction
 at a certain speed.
 And the example clips they have are displayed on the right.
 So the example clips are of someone walking and of someone running.
 Yet there is not really a transition in the data set from walking to running.
 And the agent learns to this transition by itself.
 So their point is always, look, we have kind of simple things in the data set, we have
 the individual parts in the data set that the agent should do.
 But we never have the combination of all the things.
 And to kind of stitch these parts together, that's the powerful thing about this method,
 which is pretty cool.
 So here, you can see at the top right, there is a target speed.
 And all of these three agents are trained agents.
 And in the same manner, right, and they're all told to reach that given target speed.
 However, the agent on the left only has been provided with a data set of people just walking.
 The agent in the middle, the same, but it has only received a data set of just agents
 running.
 So no walking.
 And on the right, this agent has received a data set of agents walking and running.
 So you can see that as the target speed changes, the like if it's fast, the walker is not able
 to keep up when it's slow, the runner is not able to slow down.
 However, the agent that has the full data set available can not only match the speed
 and change its style according to the speed, it can it also learns the transitions from
 one to the other.
 And this these transitions are not in the data set itself.
 Okay, so the cool part about this method is it can sort of stitch together the appropriate
 behaviors from the data set.
 Even if you don't provide these specifically to solve the task.
 The Yeah, this is the t rex.
 I think this is just to show that you don't have use motion capture, but you can use it.
 You can learn from a provided data set of keyframe animation.
 And you can also see the there is nothing in the data set about reaching a goal.
 There's just kind of demonstrations of the t rex walking.
 And the method is able to adapt this walking style in concordance with reaching a goal.
 So you can see that the turning is much like the turning in the example clips.
 Whereas if you've ever seen things like this without without the the examples, these policies
 that these things come up with are quite weird.
 So here's a failure case.
 And so the difference between this method and other methods is other methods, such as
 this motion tracking in the middle, what they try to do is they try to match a given behavior
 from the data set as closely as possible.
 So this it's called motion tracking.
 Now there is some sophistication to it more than I'm saying right here.
 But essentially, you have a front flip on the left.
 And then the motion tracking algorithm tries to learn a policy such that the behavior is
 followed as closely as possible.
 Now, again, this is really good when you have the exact demonstration available from what
 you want to do.
 It's not so good if you if what you have available as demonstrations is not isn't really what
 you want to do is just sort of some demonstrations.
 But there are failure cases, of course, if you want to copy exactly.
 So if you want to do a front flip, and by the way, the reward function here is how closely
 you match the motion from the reference motion.
 So that's the reward function.
 However, motion tracking does more than that motion tracking really tries to track the
 motion itself.
 While this method here would only get the reward of tracking the motion.
 And you can see it doesn't manage to to actually learn it more like doesn't try it tries to
 not fail.
 So it reaches the same end position and that's sort of good enough for it.
 So there is a Yeah, there is a trade off right here.
 It's probably also given by how much you weigh the different components.
 So here you have a data set of agents walking and agents waving.
 And then what you want to do is you want to have a agent that walks in a direction while
 they wave the arm or why they they lift the arm or something.
 So at the left, you can see if you only have a data set, if you only have a data set of
 the waving agents, it's really struggling moving forward, right that the walking it
 learns it has no demonstration of walking.
 So that's a struggle.
 If you only have the walking demonstration in the middle, then it doesn't really track
 the arm movement where it should even though there is a reward for it, right?
 Only Yeah, on the right, I mean, this is somewhat somewhat, but it is kind of able to to interpolate.
 So if you if you want to check out this video, there is another one that actually explains
 the paper in a short form.
 This is from from SIGGRAPH.
 Go check it out.
 They do have more sophisticated behaviors.
 So on the bottom here, you can, for example, see the obstacle run, leap and roll.
 So the data set contains demonstrations from all of those things, but not the things in
 conjunction with each other.
 In this here, at least what they describe in the text in this, this right here, what
 they have in the data set is demonstrations of walking and demonstrations of getting up
 from the ground.
 And whenever so the agent learns that whenever it falls over right here, that it can get
 up faster if it kind of does this rolling motion right here.
 So this was nowhere in the data set, but because the agent wants to go to a get up state, both
 because that will go it that will make it go towards a goal.
 And also because that matches behavior in the data set, it will learn this rolling motion
 as it falls down in order to get up again.
 So that is that's pretty cool.
 Also in this strike and punch example, the data set apparently only contains agents walking
 or agents punching, it never contains agents walking, and then punching.
 So the transition that you saw at the beginning is a learned behavior that wasn't in the data
 set.
 So that's, I think it's a it's a pretty cool application of and a combination of two things
 of adversarial learning and of of learning sorry, not from demonstration because that's
 adversarial learning of learning to reach a goal.
 And it's a good Yeah, it's a good demonstration of how you can combine the two they have a
 lot of ablations where they sort of show that the impact of the data set makes a big difference.
 I mean, you've seen this in the demonstrations.
 But also here you can see that again in a graphical form.
 So the locomotion data set contains both demonstrations of walking and running, while the walk or
 the run data set only contains demonstrations of either and the here is the target speed
 versus the average speed that the agent does.
 Now if you only have a walking data set, the agent no matter the target speeds, the agent
 will always kind of stick to walking.
 And if you have the running data set, it can run faster up here.
 But if you want it to slow down, it can't really run slower than you require.
 Only when the data set contains both things, can it transition between the two and actually
 match the running or walking.
 So what do we think of this?
 My opinion is it's probably it's very cool.
 It's a good way of sort of bringing demonstrations into the picture without manually tracking
 the demonstrations or copying exactly.
 So you just give some suggestions to the algorithm of what it could do.
 And you do that in form of a data set, which is something that I like, because it's not
 as invasive as telling the agent, you know, you need to match the joint movements and
 so on of the of the demonstration.
 This enables demonstrations to come in that are of a much broader range, not necessarily
 reach the goal, not necessarily even have a goal in mind.
 So that's cool.
 On the other hand, I think it's pretty finicky because you have to strike the trade off parameter
 between the two rewards quite cleanly, or clearly for your goal.
 Because we've already seen right at some point, the agent won't reach the goal anymore.
 If if this reward here, if the reward of the style is too high, we already saw this if
 you have a data set of just running, the agent will simply neglect the goal, it won't go
 slower than, you know, the kind of the slowest run or demonstration or a little bit slower
 than that, it just won't change its policy because it needs to match the data set.
 And the this balance seems to be quite, quite a important hyper parameter.
 And that also makes the provided data set here quite an important thing to to have available.
 So which data set you provide is also quite important.
 And lastly, the tasks themselves or the reward of the goal directed task nature, or in this
 paper, extremely engineered.
 And that's what I want to come back here lastly to so what they tout, for example, in this
 walk and punch thing, they say, oh, when the agent is far away, it runs towards the
 target.
 But if it's close, it only it slows down.
 And then when it's really close, it punches the target.
 And it sort of learns to combine these different skills.
 But and which is cool, right, because the transition wasn't in the data set.
 But a big part of it combining these skills is because in the reward, you make the reward
 different, whether the agent is far away, or whether it's near, you can see that right
 here.
 So these things are reward shaped to a high degree to encourage these kinds of transitions
 to happen, which I think is not really practical in a lot of settings.
 So it's still to be seen how much this is of practical value in other reinforcement
 learning tasks where you don't have that available.
 And also in other reinforcement learning tasks, where maybe the reward is more sparse, and
 how that affects this thing, because essentially, if the reward is much more sparse and irregular,
 now you have a problem because now the style signal is much more prominent.
 And that's not necessarily solved by simply reweighing the style signal.
 So I'm excited to see what comes out of this line of work.
 Next, it's a pretty cool line, as I already said, it's a good application of GANs in a
 different field than images.
 And with that, let me know what you think in the comments.
 I'll see you next time.
 Bye bye.