Meta-Learning through Hebbian Plasticity in Random Networks (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper hebbian vision car ant quadruped neuroplasticity fire together wire together reinforcement learning deep rl deep reinforcement learning policy network policy gradient evolutionary methods evolution step population correlation gradient episode random adaptive reconfigure damage injury agent
#ai #neuroscience #rl

Reinforcement Learning is a powerful tool, but it lacks biological plausibility because it learns a fixed policy network. Animals use neuroplasticity to reconfigure their policies on the fly and quickly adapt to new situations. This paper uses Hebbian Learning, a biologically inspired technique, to have agents adapt random networks to high-performing solutions as an episode is progressing, leading to agents that can reconfigure themselves in response to new observations.

OUTLINE:
0:00 - Intro & Overview
2:30 - Reinforcement Learning vs Hebbian Plasticity
9:00 - Episodes in Hebbian Learning
10:00 - Hebbian Plasticity Rules
18:10 - Quadruped Experiment Results
21:20 - Evolutionary Learning of Hebbian Plasticity
29:10 - More Experimental Results
34:50 - Conclusions
35:30 - Broader Impact Statement

Videos: https://twitter.com/risi1979/status/1280544779630186499
Paper: https://arxiv.org/abs/2007.02686

Abstract:
Lifelong learning and adaptability are two defining aspects of biological agents. Modern reinforcement learning (RL) approaches have shown significant progress in solving complex tasks, however once training is concluded, the found solutions are typically static and incapable of adapting to new information or perturbations. While it is still not completely understood how biological brains learn and adapt so efficiently from experience, it is believed that synaptic plasticity plays a prominent role in this process. Inspired by this biological mechanism, we propose a search method that, instead of optimizing the weight parameters of neural networks directly, only searches for synapse-specific Hebbian learning rules that allow the network to continuously self-organize its weights during the lifetime of the agent. We demonstrate our approach on several reinforcement learning tasks with different sensory modalities and more than 450K trainable plasticity parameters. We find that starting from completely random weights, the discovered Hebbian rules enable an agent to navigate a dynamical 2D-pixel environment; likewise they allow a simulated 3D quadrupedal robot to learn how to walk while adapting to different morphological damage in the absence of any explicit reward or error signal.

Authors: Elias Najarro, Sebastian Risi

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, take a look at the following problem on the left right here. So you have this quadruped and the goal is to have it walk forward or in any direction as far as possible. Now, usually this is the domain of sort of reinforcement learning. So you have inputs, which is the sensors of the joints of the quadruped and you have outputs, which is how much force you want to put on each of the legs and you have to somehow learn a policy to make it walk forward. Reinforcement learning does that by sort of trial and error using an environment to learn the policy directly. However, this paper does something different. What it does is it learns a policy that is adaptive hearing training, which basically means that at the beginning of each episode, the policy is initialized randomly and by policy here, we mean a policy network, policy neural network, which you can see at the bottom. So that's initialized randomly and then during the episode, depending on the input, this network is changed and adapted in order to achieve high performance. So even at test time, the network is started randomly and then adapted during the episode. So this paper deals with this problem and tries to implement this sort of more biologically plausible way of learning a policy, adapting to the environment and achieve ultimately good performance in this task. And it has some nice property, namely that it can deal with these things, as you can see here, front right leg damage, front left leg damage, but we'll get to that later. But just so you know what's coming. So the paper is called Meta learning through Hebbian plasticity in random networks by Elias Naharo and Sebastian Rizzi. So we'll go through the paper, what it does, what evolutionary methods are really briefly, which they use, what Hebbian plasticity is the difference to classic reinforcement learning. And then we'll look at the experiments and that's going to be it. If you like content like this, as always, don't hesitate to subscribe and share it out. And tell me what you think in the comments. I still read all the comments. So I am very interested in what you think about works like this and about the video itself. Okay, so they say lifelong learning and adaptability are two defining aspects of biological agents. Modern reinforcement learning approaches have shown significant progress in solving complex tasks. However, once training is concluded, the found solutions are typically static and incapable of adapting to new information or perturbations. So they contrast the two things here. Reinforcement learning, as you know, is very powerful in these domains. But its goal is to learn a policy and then that policy is fixed and it's specific to that particular problem. However, biological agents, you know, humans, animals and so on, they are able to adapt usually very, very quickly. They give some sort of examples right here, like if a if an animal is born, it almost immediately knows how to walk. So even if it has some sort of injury, even if it has some sort of disability, usually the animal can walk pretty much instantly. And that means it sort of adapts to the body that it is in sort of reconfigures itself on the fly. And that's what we're going to explore here. So this isn't going to outcompete RL anytime soon. It's just a different way and a biologically more plausible way in order to do that. So again, they say, we still don't know completely how biological brains learn and adapt so efficiently from experience. It is believed that synaptic plasticity plays a prominent role in this process. And that's why they are using these Hebbian learning rules in order to configure the network. So let's contrast the two things for a second. In reinforcement learning, what you have is a policy network. Now the policy network is a neural network that maps sensory inputs to actions. Okay, so you have the observation goes in, and outcomes in action. This is your policy network. Now, during training in reinforcement learning, what you do is you have some sort of environment, okay, this is the environment. And you play this back and forth game with the environment. And you try to improve this policy network right here as best as you can in order to achieve a high reward. Then during testing, so this is train, then during testing, you freeze, you freeze this network right here. So you freeze the network. And then you simply play that game and you see how well it does. Okay, so this gives you some sort of reward. And that's going to be your testing reward. And you know, that can be generalization, it can be two different environments, and so on. But the crucial part is that you in train, you learn, and then you freeze during test. In this, in this particular paper right here, they do something different. So let's call that the Hebbian plasticity world. In the Hebbian plasticity world, again, you have your environment, and you play this game. But you play the game in episodes. And at the beginning of each episode, you initialize this using some sort of distribution here, a normal distribution, you initialize the network, and then you learn, you adapt. During the episode, you adapt the network to have good performance. Okay, so this thing right here, these are the Hebbian rules. So you update the network during the episode. And then at the end of the episode, you go back, you initialize the network, again, you start a new episode, and you again adapt that randomly initialized network. So what's actually learned here isn't the weight of the network. What's learned during training is these rules that transform any randomly initialized network into a high performing network. Now, of course, you might just object and say, Hey, wait a minute, I can just basically hard code the, you know, the optimal weights here into these Hebbian rules. Like my rules can simply, you know, not care about the input and simply output whatever good weights there are. And ultimately, that would lead back to RL. But as you will be able to see in the experiments, they also have some videos provided that I invite you to watch, you can really see that the network reconfigures itself. First of all, at the beginning, it reconfigures itself to a good state. But then also, as the episode is progressing, it continuously reconfigures itself, depending on the input. So this is the real power of these Hebbian rules in that during the episode, the network can continuously reconfigure itself in order to achieve higher reward. So it's not just that I can go from the random initialization to a good performing policy, I can adapt that policy depending on what the input is. So at test time in this Hebbian world, what we're going to do is again, we are going to freeze the learning rules. So you have to kind of rethink, we're going to freeze the Hebbian rules, but still, we're going to randomly initialize our policy in each episode. And then we're going to change that during the episode, okay, and then that's ultimately going to give us our reward. So the thing that's learned is just something different. Here, you learn the weights directly in the RL setting. And then the Hebbian plasticity setting, you learn the rules to update the weights dynamically depending on the input. This is a form of meta learning, right? It's not exactly but it is a form of meta learning. So let's see what those Hebbian rules are. And you can as again, you can see this right here during training. So this is one episode. And it always starts with these random networks at the beginning. And then you can see as you progress, there is structure emerging. And again, I linked to the videos. And you can see that during the episode, even this is changing, and this is especially visible on their other example that they have here, like this, this car example. So in this car example, during the video, you'll see that there's a curve like this. And then as imagine you're a driver, like there is a kind of a left curve coming and you adjust your mental state, let's say, to say, okay, I don't know what's around the curve, I need to be ready to break and so on. And then there is a straight piece coming and you'll be like, well, I see everything, you know, I can focus on different things that you can reconfigure your state in order to adapt to the the observation. And that's exactly what you'll see in that video is that the weights are continuously updating, not so much in these quarter pads to which we'll get later. So these Hebbian rules, what do they look like? These are biologically inspired rules. And they say the following. So this here is the delta W I J. And our perspective of policy networks is going to be that this is a neural network, as we said, and we'll just pick up one layer right here. And there is going to be weights right here, you know, weights from all to all these are going to be fully connected networks, and like this, and there's going to be neuron I somewhere here and neuron J somewhere here. Okay, so neuron I and neuron J are going to have a connection together, this thing right here. And there's going this, the question is going to be how do we update that weight from one time step to the next? Remembering the weights here are changed in each time step, each time step during the episode, we update the weights. So how are they going to be updated? Let's contrast this first to classic reinforcement learning. So in classic reinforcement learning, we would keep these weights the same during the entire episode. And then at the end of the episode, right, we keep those the same. And at the end of the episode, we'll get a reward. And then we'll go back, we'll look back and say, how do we need to change the weights such that in the next episode, the reward will be higher. And in again, in classic reinforcement learning, for example, in policy gradient methods, you will actually calculate a gradient with respect to these weights right here. Actually, let's let's go into that later when we contrast evolutionary methods. So the important part right here is that we change the weights in each time step. So how do we change the weights? Of course, we don't have access to the reward, right, in order to change the weights, the reward is going to come into play when we change the rules to change the weights. But during the episode, we don't have the reward. At least we assume we only get kind of the reward at the end. So we need a different method. And the method is going to be the following right here. The important things in this formula are going to be so how do we change the weights that's dependent on two quantities that appear during each time step, oh, I and oh, j. And these are going to be the outputs of neuron i and neuron j. So how do we change the connection that's going to be dependent on the output of neuron i, which is here called the pre synaptic output, and the output of neuron j, which is going to be the post synaptic output. The rule, the kind of mantra here is the fire together wire together means that if two neurons are active at the same time regularly, then they probably should be connected together because they already correlate. And you can see right here that there is a term in this formula that is oh, i times oh, j. So this here is the correlation between or the covariance, or just the product, if we're exact between these two neurons. And if they are both active regularly, then this quantity is going to be high. And if they're both not active regularly that or if one is active and the other one isn't that quantity is going to be low. And the a parameter here specifies how the weights are updated in response to this. So the a, b, c, d, and eta parameters right here are these are the learned parameters, these are going to be your learned rules to update the weights. So these change once after once per learning step was a once per. So after the episode is done, you're going to change these capital constants right here, including the eta, which is the learning rate. These things right here, these are per step. So this is each step gives you a different oh, i and oh, j. And then you'll adjust the weight based on that, you will see that these constants here, they are per weight. So for each weight in this neural network, we learn a separate rule of how to update that particular weight. So the algorithm can, it can basically decide for a particular way to can decide, well, if these two things fire together often, I want to update my weight very heavily in response to that. Okay, so if the a is very high, that means the connection responds very thoroughly to when the two neurons fire together. That is not the same as to say that connection should always be very strong, it's dependent on the input. So only when this quantity is high, should the network or should the weight be updated, and the a parameter modulates how well it's updated or how how strongly it's up, it can also be negative, it can be zero, basically meaning that, you know, it doesn't matter if they fire together, I don't want to update the weight, this particular weight in response to that. So you can see that you can learn these rules that can adapt to different inputs, because all of the changes the delta here is dependent on the inputs. So on the correlation, but also on the different inputs themselves. And then there is also a constant right here. Okay, this, as you can see, it's a linear function of the inputs of the OI and OJ and their product. So I hope this is clear that these Hebbian rules, you learn ABCD and ETA, and that gives rise to an adaptive network that can change and reconfigure itself over the course of an episode, depending on the inputs. And one of the things right here, and we'll get to how you actually learn the rules itself in a second. But one of the things right here is very visible, as I said in this first experiment, where it reconfigures itself continuously, but also in this experiment with this quadruped right here. So this quadruped, usually, it's you know, you simply walk in the direction that's your reward and RL is perfectly fine at this as well. However, this is a bit of a has a bit of a trick to it. Namely, you are always in one of three situations, either you have an undamaged quadruped, or it's kind of left leg, front left leg is damaged, or its front right leg is damaged. Okay, and you don't tell the you simply sample these situations uniformly, and you don't tell the algorithm which situation it is in. Now, if you look at if you compare two methods, one where you directly learn the weights, you learn a fixed policy to solve, you know, this is one task, right, this is one task. And all of these three things appear with equal probability. So you have to learn one policy to make all of this work. If you learn the weights directly, and you don't have a power, like there's no doubt that like a powerful RL approach could deal with this task. But if in this case, if you just put a standard weight learner with the same number of the same size of policy as the Hebbian they compare to, if you put a weight learner on it, it will not be able to solve this task satisfactorily, what it will do is it will say, well, I need one set of rules that make me walk as far as possible as often as possible. So if you can see at the table, I'm already showing you the results right here. The table right here, if you have these static weights, you can see that it's performing pretty well in two out of three situations, right. So it what it basically does, it says, okay, here is what where there's damage, what it does is it says, I'm going to learn to walk with my left leg using my left front leg. That means when I have no damage or damage to the right front leg, I'm just fine. And I'm just going to take the hit basically, where I have damage to the left front leg, because I'm it's just going to suck. So they solved they solve this like walk more than 100 steps. So that doesn't it, since it can only learn a fixed policy, it basically discards the case where there's damage to the left front leg, it takes that hit in order to be better in the other two methods, you can see it's outperforming the Hebbian rule in the other two methods. But this shows you kind of the difference and the power that these Hebbian rules or these generally neuroplasticity might have because the Hebbian one is perfectly capable of at least in part adapting to the different situations. Now you can see that is not symmetric. Also the Hebbian rules they learn to know there's 860 and there's 440 of a thing that should actually be symmetric, we do expect a drop when there's damage, but it's not symmetric, which means that also the Hebbian rules they kind of randomly focus on one over the other, but at least they're able in some degree to adapt to both. And that's because it depending on the input, you know, it has a rule in there that basically says, well, if the if the back left leg and the front light right leg, you know, if they fire together, then I want to, if they if they fire together, the sensors that show me that they're moving, if they fire together, I'm going to wire them together, because that's how I walk, you know, front, right, back, left, and then the other way around. And if that's not the case, I'm not going to wire them together. So that would be the situation where we have damage. Instead, if they are not wired together, I'm going to, you can do this in the next layer of the neural network, wire these other two things together, you know, if if the first thing is not the case, I'm going to wire these other two things together to make up for that loss. And there you can see there is kind of this logic built into the network. Now, again, I know you can do this with learning a fixed policy, you can achieve the same effects. The point here is just to show that given kind of a same size networks and so on, that you that there might be there might be like a qualitative difference in certain situations. Again, by no means this is meant to outcompete RL or anything like this. Okay, so we'll we went there. Now, how are these rules actually learned? And there we have to again make a distinction that is completely separate from the Hebbian non-Hebbian way. Okay, so the Hebbian non-Hebbian distinction was, do we learn the weights of the policy network directly? Or do we learn the rules to update the weights? Now the question is, whatever we learn, how do we learn it? And again, we have to draw the distinction this time between, I'm going to say classic R, even though the terminology is not really correct, classic RL and evolutionary methods. Okay, so in classic RL, what I would do is I would use my weights in order to obtain a reward. And then I would update my weights. So my delta W would be proportional to the gradient of W of the reward. Okay, so in the classic RL, especially in this is a policy gradient method right now, so I use my policy, my weights to get the reward, and then I would calculate a gradient. And you know, usually the reward isn't differentiable. So you have this reinforced trick in order to pull the reward out. And you can read all of this up if you look at policy gradient, the basic policy gradient methods. But this here tells me I need a gradient, usually this is going to be the reward times the gradient of my FW of my input. So what this means is, what this means is that if my reward is high, then I just want to know, what do I need to do to make more of what I just did? Okay, and the great hint ensures that for every single weight in your neural network, you know what to do. So the gradient means that I have an exact handle on how do I need to change this weight? How do I need to change that weight? How do I need to change this weight in order if the reward is high, and because of this multiplication here, I want to make more of what I just did. And the gradient tells me how. If the reward is low, on the other hand, I want to make less of what I just did. But also the gradient tells me how that can be achieved. I simply go into the other direction than I would if the reward is high. In evolutionary methods, we don't have, we don't do this gradient calculation. Now there can be advantages to not doing gradient calculation. Sometimes backpropagation simply isn't possible. Even if it is possible, and this is maybe the case where we're now, what we need to learn in our case is these rules to update the rules. And imagine you have an episode and that's kind of episode, you have step, step, step, step, and in each step, these rules are applied, right? In each of these steps, the rules are applied. And at the end, you get a reward. So what you need to do is to back propagate that reward through all the steps and then through all the rules. Okay. And that might be just computationally not feasible or the rules, the rules right here are pretty, pretty easy, but the rules might not be differentiable. You actually have the same problem in general in classic RL as well. But you know, you can cut off time steps and so on. There are various hacks. In any case, there can be advantages to not having that gradient and evolutionary methods are a way to do that. In evolutionary method, usually you are don't train one agent, you train a population of agents. So you have a bunch of these neural network agents in here. And the way you update the neural network agent is you simply let them run, you know, you let them run your app, the episode. So this is your W, one of them, you let them run the episode, they get a reward. And then you can do multiple things. So this depends on the evolutionary method. So you can either pick out the best performing agent, or you can update each agent according to some rule. The goal here is simply to basically, you always want to take your weights, you want to add some noise to them. And you want to see, does it get better or worse? If it gets better, good. If it gets worse, not good. Okay, the difference is without the gradient, you don't have a handle on how do you need to change each individual weight. All you can do is basically random walk and observe what happens. And if the random walk is, you know, turns out to be good, you go more into that direction of that random walk. So it's sort of a sort of a poor man's gradient method in these evolutionary methods. Again, completely independent of what we learn, you can use the evolutionary method to learn the fixed weights. And that's what actually what happens in the table I've shown you below. Or you can use the evolutionary method to learn the Hebbian update rules. As well, you can use RL to learn the fixed weight or the update rules. In this paper, they use evolutionary methods to learn the Hebbian update rules. And they compare mostly with using evolutionary methods to learn the fixed weights. Okay, the exact evolutionary step they use right here is the following. So HT here is going to be the thing that you learn. Now as compared to W being the network weights, H is going to be the Hebbian weights, since we learn the Hebbian weights. So how they'll update each agent is going to be they'll take the Hebbian weights. And this this here is how you update, right? This is your delta H. How do you update the Hebbian weights? Well, what you do is you you perform in random perturbations. So I take my weights and I add noise. I just add noise. Okay, so I I'm here. And I just make a bunch of versions of it. And then I observe how well are these versions doing? So how well are my random perturbations doing? This is going to be the fitness fi right here is going to be the fitness. And then I'm just going to perform a weighted average. So this is my weighted average of these new solutions. Okay, so if this solution here did pretty well, and this solution did pretty poorly, I want to walk, you know, in this direction. And then again, I do the same thing here from here, I do a bunch of perturbations. And maybe this one did pretty well. And this one did pretty poorly, I want to walk in this direction, and so on. Okay, so that's how you you'll change the you'll change weights or rules or whatever you want in an evolutionary method. As you know, it's pretty easy. It's easier than reinforcement learning, no back prop, no nothing. Basically a black box optimizer. There are more complicated evolutionary methods, but no, we don't go into those here right now. Okay, so again, I've already shown you these results. Now I said these static weights are also with evolutionary method, they also report what you would get with like a RL approach, like PPO, you would get kind of the same thing as they get as they get here. So, oh, sorry, this is not the same as the table. Yeah, I was confused for a second. This here is for the car environment. Okay, this is this vision based environment. So with their method, they get like an 870 rewards with the heavy and based approach. With the static weight, but still evolutionary method, they get a much lower reward. In fact, the heavy and based approach is about the same as you get here with an RL algorithm. And as we said, the RL algorithm more complicated. And if you use like a state of the art RL algorithm, not just PPO, you get a bit of a better performance, but not that much if you look at the actual numbers. So, you know, pretty cool to see that, again, this is not outperforming anything. This is simply showing that you can do that. They do a number of experiments where they go in the episode and they kind of change stuff in the episode. And one cool thing here is that they go and you know, this is an episode. So at the episode, you start with a random network each time in this heavy and setting. And then pretty quickly, the rules adapt for a high performing right. So it starts to walk, it reconfigures itself and starts to walk. The reward here again, it doesn't have access to that, but we can measure it, of course. And then at this step A right here, they simply go to the weights and zero them out. So they just delete these weights right here. And only 10 time steps later, it has reconfigured itself as you can see right here in order to walk again. So 10 time steps later, reconfigures itself, reconfigures itself. And after a short while right here, it's back to its kind of original performance, as you can see. So that's, I'd say that's fairly impressive in this very short amount of time able to recover from such an intervention. If you do this, of course, if you do this to your policy network, that's statically learned, it's going to be garbage. But I guess the fair comparison would be to delete the heavy and rules themselves. And you know, so it's not like it's not like this can adapt to new situations, or something like this, this is still learned for particular environments, right. But the point here is that you learn the rules. And this is kind of a study on neuroplasticity. Now, my question actually would be why this diagonal pattern appears. And I have not seen like a clear explanation. Especially is this anti diagonal pattern, it's not so much here in the output layer, right, this is the output layer, there are 21 actions or so. And this one is this, this dimension. So not that much there. But there seems to be this rule. And this is not the case at the beginning, right, you saw the beginning you saw at the beginning, it was pretty random matrix. So why? Why? Yeah, here, pretty random. And then there's this diagonal pattern, I don't know why. If you know, let me know. I mean, it's anti diagonal, maybe it is actually diagonal and the forward, the fully connected layer is just defined as something like WT times x. And but maybe this also depends on the random initialization. But there is no inherent way why particular neuron would, you know, care about sending information to like the same height of neuron on the other side. Or is there? I don't know I'm so is this a property of the evolutionary or the learning rules? It seems not because the learning rules don't depend on the position. I'm genuinely confused about this. And maybe, you know, maybe they've written it somewhere, and I've just overlooked it, though. I, they do reference it, they say, oh, there's this diagonal pattern appearing, but I don't think they ever say why it is diagonal. Okay, I might just be I might just be real dumb. Yeah. So they also, you know, they do some more experiments, they show, for example, that if you just have random Hebbian coefficients, then your algorithm just jumps around kind of in in weight space around the zero point. However, if you actually learn these Hebbian coefficients, as they do, you have like this clear attractor here. And you have these kind of oscillating curves. When, you know, when when you do that, and you can see here in the different situations where things are damaged, and so on. So all in all, I think it's a pretty interesting study. And I think this neuroplasticity is, it's a different way, you know, it's unclear to say if it will ever deliver the performance that RL delivers, but certainly there are situations where such plasticity is desired. And if we can also combine this with greater generalization performance, then, you know, we have agents that can quickly kind of reconfigure. And a lot of work by these these kind of open ended learning community also plays into this roles, all in all pretty, pretty cool, non standard way of doing things. Last thing, the broader impact statement. Every now and then we'll look at a broader impact statement, since these are new, just to get kind of an overview of what they look like. So they say the ethical and mutual societal consequences of this work are hard to predict, but likely similar to other work dealing with more adaptive agent and robots. In particular, by giving the robots the ability to still function when injured could make it easier from them being deployed in areas that have both a positive and negative impact on society. Okay, well, again, this it's not really giving robots the ability to still function when they're injured. I first I thought first I thought, okay, they train it when it's fully functioning, but then they damage it during test time. But as I understand it, as I understand the paper, they already train it with the damaged versions, they just don't tell the algorithm in which version it is right now. So it's not the same as being able to work when injured unless you've specifically trained for it. In this case, again, I could be wrong about this. Yeah. In the very long term robots that can adapt could help in industrial automation or help to care for the elderly. On the other hand, more adaptive robots could also be more easily used for military applications. The approach presented in this paper is far from being deployed in these areas, but it's important to discuss its potential long term consequences early on. Now, okay, so let's evaluate the broader impact statement. Let's, well, the first check to do is always to simply replace whatever their method is with the word technology. Okay, so let's do that. In the very long term, technology could help in industrial automation or help to care for the elderly. Check. On the other hand, technology could also be more easily used for military application. Check. Technology is far from being deployed in these areas. Okay, I guess some technology isn't, but advanced technology. Yeah. So again, the rule for broader impact statements seem to be you take whatever your method is and you go up until you find, you know, you're basically at technology or something equivalent, because no one actually I've never seen a broader impact statement that writes about the actual thing in the paper, they always go up like one layer or two. And then it basically regresses to technology, even even though very few papers actually would be able to discuss their particular thing, but you know, and that and then in terms of guidelines on broader impact statement, this one is missing, there's there's always this, the holy trifecta. So the holy trifecta is you go like a, you know, like you're a, you're a Catholic, you go with your finger to your head, chest, left and right. And you say technology good, technology bad, technology biased. Okay, so you want to write a broader impact statement, go up the layers, technology, good bad bias, and we're missing the bias here. So that's, you know, I'm just following what these guidelines to broader impact statements are. I don't make the rules. I'm sorry that the heavy ins make the rules apparently. I'm not heavy in. Okay, I've I hope you've enjoyed this paper and this video. Let me know what you think. Check out the videos that they have. I'll link them. And with that, I wish you a pleasant day. Bye bye.
 Hi there, take a look at the following problem on the left right here. So you have this quadruped
 and the goal is to have it walk forward or in any direction as far as possible. Now,
 usually this is the domain of sort of reinforcement learning. So you have inputs, which is the
 sensors of the joints of the quadruped and you have outputs, which is how much force
 you want to put on each of the legs and you have to somehow learn a policy to make it
 walk forward. Reinforcement learning does that by sort of trial and error using an environment
 to learn the policy directly. However, this paper does something different. What it does
 is it learns a policy that is adaptive hearing training, which basically means that at the
 beginning of each episode, the policy is initialized randomly and by policy here, we mean a policy
 network, policy neural network, which you can see at the bottom. So that's initialized
 randomly and then during the episode, depending on the input, this network is changed and
 adapted in order to achieve high performance. So even at test time, the network is started
 randomly and then adapted during the episode. So this paper deals with this problem and
 tries to implement this sort of more biologically plausible way of learning a policy, adapting
 to the environment and achieve ultimately good performance in this task. And it has
 some nice property, namely that it can deal with these things, as you can see here, front
 right leg damage, front left leg damage, but we'll get to that later. But just so you know
 what's coming. So the paper is called Meta learning through Hebbian plasticity in random
 networks by Elias Naharo and Sebastian Rizzi. So we'll go through the paper, what it does,
 what evolutionary methods are really briefly, which they use, what Hebbian plasticity is
 the difference to classic reinforcement learning. And then we'll look at the experiments and
 that's going to be it. If you like content like this, as always, don't hesitate to subscribe
 and share it out. And tell me what you think in the comments. I still read all the comments.
 So I am very interested in what you think about works like this and about the video
 itself. Okay, so they say lifelong learning and adaptability are two defining aspects
 of biological agents. Modern reinforcement learning approaches have shown significant
 progress in solving complex tasks. However, once training is concluded, the found solutions
 are typically static and incapable of adapting to new information or perturbations. So they
 contrast the two things here. Reinforcement learning, as you know, is very powerful in
 these domains. But its goal is to learn a policy and then that policy is fixed and it's
 specific to that particular problem. However, biological agents, you know, humans, animals
 and so on, they are able to adapt usually very, very quickly. They give some sort of
 examples right here, like if a if an animal is born, it almost immediately knows how to
 walk. So even if it has some sort of injury, even if it has some sort of disability, usually
 the animal can walk pretty much instantly. And that means it sort of adapts to the body
 that it is in sort of reconfigures itself on the fly. And that's what we're going to
 explore here. So this isn't going to outcompete RL anytime soon. It's just a different way
 and a biologically more plausible way in order to do that. So again, they say, we still
 don't know completely how biological brains learn and adapt so efficiently from experience.
 It is believed that synaptic plasticity plays a prominent role in this process. And that's
 why they are using these Hebbian learning rules in order to configure the network. So
 let's contrast the two things for a second. In reinforcement learning, what you have is
 a policy network. Now the policy network is a neural network that maps sensory inputs
 to actions. Okay, so you have the observation goes in, and outcomes in action. This is your
 policy network. Now, during training in reinforcement learning, what you do is you have some sort
 of environment, okay, this is the environment. And you play this back and forth game with
 the environment. And you try to improve this policy network right here as best as you can
 in order to achieve a high reward. Then during testing, so this is train, then during testing,
 you freeze, you freeze this network right here. So you freeze the network. And then
 you simply play that game and you see how well it does. Okay, so this gives you some
 sort of reward. And that's going to be your testing reward. And you know, that can be
 generalization, it can be two different environments, and so on. But the crucial part is that you
 in train, you learn, and then you freeze during test. In this, in this particular paper right
 here, they do something different. So let's call that the Hebbian plasticity world. In
 the Hebbian plasticity world, again, you have your environment, and you play this game.
 But you play the game in episodes. And at the beginning of each episode, you initialize
 this using some sort of distribution here, a normal distribution, you initialize the
 network, and then you learn, you adapt. During the episode, you adapt the network to have
 good performance. Okay, so this thing right here, these are the Hebbian rules. So you
 update the network during the episode. And then at the end of the episode, you go back,
 you initialize the network, again, you start a new episode, and you again adapt that randomly
 initialized network. So what's actually learned here isn't the weight of the network. What's
 learned during training is these rules that transform any randomly initialized network
 into a high performing network. Now, of course, you might just object and say, Hey, wait a
 minute, I can just basically hard code the, you know, the optimal weights here into these
 Hebbian rules. Like my rules can simply, you know, not care about the input and simply
 output whatever good weights there are. And ultimately, that would lead back to RL. But
 as you will be able to see in the experiments, they also have some videos provided that I
 invite you to watch, you can really see that the network reconfigures itself. First of
 all, at the beginning, it reconfigures itself to a good state. But then also, as the episode
 is progressing, it continuously reconfigures itself, depending on the input. So this is
 the real power of these Hebbian rules in that during the episode, the network can continuously
 reconfigure itself in order to achieve higher reward. So it's not just that I can go from
 the random initialization to a good performing policy, I can adapt that policy depending
 on what the input is. So at test time in this Hebbian world, what we're going to do is again,
 we are going to freeze the learning rules. So you have to kind of rethink, we're going
 to freeze the Hebbian rules, but still, we're going to randomly initialize our policy in
 each episode. And then we're going to change that during the episode, okay, and then that's
 ultimately going to give us our reward. So the thing that's learned is just something
 different. Here, you learn the weights directly in the RL setting. And then the Hebbian plasticity
 setting, you learn the rules to update the weights dynamically depending on the input.
 This is a form of meta learning, right? It's not exactly but it is a form of meta learning.
 So let's see what those Hebbian rules are. And you can as again, you can see this right
 here during training. So this is one episode. And it always starts with these random networks
 at the beginning. And then you can see as you progress, there is structure emerging.
 And again, I linked to the videos. And you can see that during the episode, even this
 is changing, and this is especially visible on their other example that they have here,
 like this, this car example. So in this car example, during the video, you'll see that
 there's a curve like this. And then as imagine you're a driver, like there is a kind of a
 left curve coming and you adjust your mental state, let's say, to say, okay, I don't know
 what's around the curve, I need to be ready to break and so on. And then there is a straight
 piece coming and you'll be like, well, I see everything, you know, I can focus on different
 things that you can reconfigure your state in order to adapt to the the observation.
 And that's exactly what you'll see in that video is that the weights are continuously
 updating, not so much in these quarter pads to which we'll get later. So these Hebbian
 rules, what do they look like? These are biologically inspired rules. And they say the following.
 So this here is the delta W I J. And our perspective of policy networks is going to be that this
 is a neural network, as we said, and we'll just pick up one layer right here. And there
 is going to be weights right here, you know, weights from all to all these are going to
 be fully connected networks, and like this, and there's going to be neuron I somewhere
 here and neuron J somewhere here. Okay, so neuron I and neuron J are going to have a
 connection together, this thing right here. And there's going this, the question is going
 to be how do we update that weight from one time step to the next? Remembering the weights
 here are changed in each time step, each time step during the episode, we update the weights.
 So how are they going to be updated? Let's contrast this first to classic reinforcement
 learning. So in classic reinforcement learning, we would keep these weights the same during
 the entire episode. And then at the end of the episode, right, we keep those the same.
 And at the end of the episode, we'll get a reward. And then we'll go back, we'll look
 back and say, how do we need to change the weights such that in the next episode, the
 reward will be higher. And in again, in classic reinforcement learning, for example, in policy
 gradient methods, you will actually calculate a gradient with respect to these weights right
 here. Actually, let's let's go into that later when we contrast evolutionary methods. So
 the important part right here is that we change the weights in each time step. So how do we
 change the weights? Of course, we don't have access to the reward, right, in order to change
 the weights, the reward is going to come into play when we change the rules to change the
 weights. But during the episode, we don't have the reward. At least we assume we only
 get kind of the reward at the end. So we need a different method. And the method is going
 to be the following right here. The important things in this formula are going to be so
 how do we change the weights that's dependent on two quantities that appear during each
 time step, oh, I and oh, j. And these are going to be the outputs of neuron i and neuron
 j. So how do we change the connection that's going to be dependent on the output of neuron
 i, which is here called the pre synaptic output, and the output of neuron j, which is going
 to be the post synaptic output. The rule, the kind of mantra here is the fire together
 wire together means that if two neurons are active at the same time regularly, then they
 probably should be connected together because they already correlate. And you can see right
 here that there is a term in this formula that is oh, i times oh, j. So this here is
 the correlation between or the covariance, or just the product, if we're exact between
 these two neurons. And if they are both active regularly, then this quantity is going to
 be high. And if they're both not active regularly that or if one is active and the other one
 isn't that quantity is going to be low. And the a parameter here specifies how the weights
 are updated in response to this. So the a, b, c, d, and eta parameters right here are
 these are the learned parameters, these are going to be your learned rules to update the
 weights. So these change once after once per learning step was a once per. So after the
 episode is done, you're going to change these capital constants right here, including the
 eta, which is the learning rate. These things right here, these are per step. So this is
 each step gives you a different oh, i and oh, j. And then you'll adjust the weight based
 on that, you will see that these constants here, they are per weight. So for each weight
 in this neural network, we learn a separate rule of how to update that particular weight.
 So the algorithm can, it can basically decide for a particular way to can decide, well,
 if these two things fire together often, I want to update my weight very heavily in response
 to that. Okay, so if the a is very high, that means the connection responds very thoroughly
 to when the two neurons fire together. That is not the same as to say that connection
 should always be very strong, it's dependent on the input. So only when this quantity is
 high, should the network or should the weight be updated, and the a parameter modulates
 how well it's updated or how how strongly it's up, it can also be negative, it can be
 zero, basically meaning that, you know, it doesn't matter if they fire together, I don't
 want to update the weight, this particular weight in response to that. So you can see
 that you can learn these rules that can adapt to different inputs, because all of the changes
 the delta here is dependent on the inputs. So on the correlation, but also on the different
 inputs themselves. And then there is also a constant right here. Okay, this, as you
 can see, it's a linear function of the inputs of the OI and OJ and their product. So I hope
 this is clear that these Hebbian rules, you learn ABCD and ETA, and that gives rise to
 an adaptive network that can change and reconfigure itself over the course of an episode, depending
 on the inputs. And one of the things right here, and we'll get to how you actually learn
 the rules itself in a second. But one of the things right here is very visible, as I said
 in this first experiment, where it reconfigures itself continuously, but also in this experiment
 with this quadruped right here. So this quadruped, usually, it's you know, you simply walk in
 the direction that's your reward and RL is perfectly fine at this as well. However, this
 is a bit of a has a bit of a trick to it. Namely, you are always in one of three situations,
 either you have an undamaged quadruped, or it's kind of left leg, front left leg is damaged,
 or its front right leg is damaged. Okay, and you don't tell the you simply sample these
 situations uniformly, and you don't tell the algorithm which situation it is in. Now, if
 you look at if you compare two methods, one where you directly learn the weights, you
 learn a fixed policy to solve, you know, this is one task, right, this is one task. And
 all of these three things appear with equal probability. So you have to learn one policy
 to make all of this work. If you learn the weights directly, and you don't have a power,
 like there's no doubt that like a powerful RL approach could deal with this task. But
 if in this case, if you just put a standard weight learner with the same number of the
 same size of policy as the Hebbian they compare to, if you put a weight learner on it, it
 will not be able to solve this task satisfactorily, what it will do is it will say, well, I need
 one set of rules that make me walk as far as possible as often as possible. So if you
 can see at the table, I'm already showing you the results right here. The table right
 here, if you have these static weights, you can see that it's performing pretty well in
 two out of three situations, right. So it what it basically does, it says, okay, here
 is what where there's damage, what it does is it says, I'm going to learn to walk with
 my left leg using my left front leg. That means when I have no damage or damage to the
 right front leg, I'm just fine. And I'm just going to take the hit basically, where I have
 damage to the left front leg, because I'm it's just going to suck. So they solved they
 solve this like walk more than 100 steps. So that doesn't it, since it can only learn
 a fixed policy, it basically discards the case where there's damage to the left front
 leg, it takes that hit in order to be better in the other two methods, you can see it's
 outperforming the Hebbian rule in the other two methods. But this shows you kind of the
 difference and the power that these Hebbian rules or these generally neuroplasticity might
 have because the Hebbian one is perfectly capable of at least in part adapting to the
 different situations. Now you can see that is not symmetric. Also the Hebbian rules they
 learn to know there's 860 and there's 440 of a thing that should actually be symmetric,
 we do expect a drop when there's damage, but it's not symmetric, which means that also
 the Hebbian rules they kind of randomly focus on one over the other, but at least they're
 able in some degree to adapt to both. And that's because it depending on the input,
 you know, it has a rule in there that basically says, well, if the if the back left leg and
 the front light right leg, you know, if they fire together, then I want to, if they if
 they fire together, the sensors that show me that they're moving, if they fire together,
 I'm going to wire them together, because that's how I walk, you know, front, right, back,
 left, and then the other way around. And if that's not the case, I'm not going to wire
 them together. So that would be the situation where we have damage. Instead, if they are
 not wired together, I'm going to, you can do this in the next layer of the neural network,
 wire these other two things together, you know, if if the first thing is not the case,
 I'm going to wire these other two things together to make up for that loss. And there you can
 see there is kind of this logic built into the network. Now, again, I know you can do
 this with learning a fixed policy, you can achieve the same effects. The point here is
 just to show that given kind of a same size networks and so on, that you that there might
 be there might be like a qualitative difference in certain situations. Again, by no means
 this is meant to outcompete RL or anything like this. Okay, so we'll we went there. Now,
 how are these rules actually learned? And there we have to again make a distinction
 that is completely separate from the Hebbian non-Hebbian way. Okay, so the Hebbian non-Hebbian
 distinction was, do we learn the weights of the policy network directly? Or do we learn
 the rules to update the weights? Now the question is, whatever we learn, how do we learn it?
 And again, we have to draw the distinction this time between, I'm going to say classic
 R, even though the terminology is not really correct, classic RL and evolutionary methods.
 Okay, so in classic RL, what I would do is I would use my weights in order to obtain
 a reward. And then I would update my weights. So my delta W would be proportional to the
 gradient of W of the reward. Okay, so in the classic RL, especially in this is a policy
 gradient method right now, so I use my policy, my weights to get the reward, and then I would
 calculate a gradient. And you know, usually the reward isn't differentiable. So you have
 this reinforced trick in order to pull the reward out. And you can read all of this up
 if you look at policy gradient, the basic policy gradient methods. But this here tells
 me I need a gradient, usually this is going to be the reward times the gradient of my
 FW of my input. So what this means is, what this means is that if my reward is high, then
 I just want to know, what do I need to do to make more of what I just did? Okay, and
 the great hint ensures that for every single weight in your neural network, you know what
 to do. So the gradient means that I have an exact handle on how do I need to change this
 weight? How do I need to change that weight? How do I need to change this weight in order
 if the reward is high, and because of this multiplication here, I want to make more of
 what I just did. And the gradient tells me how. If the reward is low, on the other hand,
 I want to make less of what I just did. But also the gradient tells me how that can be
 achieved. I simply go into the other direction than I would if the reward is high. In evolutionary
 methods, we don't have, we don't do this gradient calculation. Now there can be advantages to
 not doing gradient calculation. Sometimes backpropagation simply isn't possible. Even
 if it is possible, and this is maybe the case where we're now, what we need to learn in
 our case is these rules to update the rules. And imagine you have an episode and that's
 kind of episode, you have step, step, step, step, and in each step, these rules are applied,
 right? In each of these steps, the rules are applied. And at the end, you get a reward.
 So what you need to do is to back propagate that reward through all the steps and then
 through all the rules. Okay. And that might be just computationally not feasible or the
 rules, the rules right here are pretty, pretty easy, but the rules might not be differentiable.
 You actually have the same problem in general in classic RL as well. But you know, you can
 cut off time steps and so on. There are various hacks. In any case, there can be advantages
 to not having that gradient and evolutionary methods are a way to do that. In evolutionary
 method, usually you are don't train one agent, you train a population of agents. So you have
 a bunch of these neural network agents in here. And the way you update the neural network
 agent is you simply let them run, you know, you let them run your app, the episode. So
 this is your W, one of them, you let them run the episode, they get a reward. And then
 you can do multiple things. So this depends on the evolutionary method. So you can either
 pick out the best performing agent, or you can update each agent according to some rule.
 The goal here is simply to basically, you always want to take your weights, you want
 to add some noise to them. And you want to see, does it get better or worse? If it gets
 better, good. If it gets worse, not good. Okay, the difference is without the gradient,
 you don't have a handle on how do you need to change each individual weight. All you
 can do is basically random walk and observe what happens. And if the random walk is, you
 know, turns out to be good, you go more into that direction of that random walk. So it's
 sort of a sort of a poor man's gradient method in these evolutionary methods. Again, completely
 independent of what we learn, you can use the evolutionary method to learn the fixed
 weights. And that's what actually what happens in the table I've shown you below. Or you
 can use the evolutionary method to learn the Hebbian update rules. As well, you can use
 RL to learn the fixed weight or the update rules. In this paper, they use evolutionary
 methods to learn the Hebbian update rules. And they compare mostly with using evolutionary
 methods to learn the fixed weights. Okay, the exact evolutionary step they use right
 here is the following. So HT here is going to be the thing that you learn. Now as compared
 to W being the network weights, H is going to be the Hebbian weights, since we learn
 the Hebbian weights. So how they'll update each agent is going to be they'll take the
 Hebbian weights. And this this here is how you update, right? This is your delta H. How
 do you update the Hebbian weights? Well, what you do is you you perform in random perturbations.
 So I take my weights and I add noise. I just add noise. Okay, so I I'm here. And I just
 make a bunch of versions of it. And then I observe how well are these versions doing?
 So how well are my random perturbations doing? This is going to be the fitness fi right here
 is going to be the fitness. And then I'm just going to perform a weighted average. So this
 is my weighted average of these new solutions. Okay, so if this solution here did pretty
 well, and this solution did pretty poorly, I want to walk, you know, in this direction.
 And then again, I do the same thing here from here, I do a bunch of perturbations. And maybe
 this one did pretty well. And this one did pretty poorly, I want to walk in this direction,
 and so on. Okay, so that's how you you'll change the you'll change weights or rules
 or whatever you want in an evolutionary method. As you know, it's pretty easy. It's easier
 than reinforcement learning, no back prop, no nothing. Basically a black box optimizer.
 There are more complicated evolutionary methods, but no, we don't go into those here right
 now. Okay, so again, I've already shown you these results. Now I said these static weights
 are also with evolutionary method, they also report what you would get with like a RL approach,
 like PPO, you would get kind of the same thing as they get as they get here. So, oh, sorry,
 this is not the same as the table. Yeah, I was confused for a second. This here is for
 the car environment. Okay, this is this vision based environment. So with their method, they
 get like an 870 rewards with the heavy and based approach. With the static weight, but
 still evolutionary method, they get a much lower reward. In fact, the heavy and based
 approach is about the same as you get here with an RL algorithm. And as we said, the
 RL algorithm more complicated. And if you use like a state of the art RL algorithm,
 not just PPO, you get a bit of a better performance, but not that much if you look at the actual
 numbers. So, you know, pretty cool to see that, again, this is not outperforming anything.
 This is simply showing that you can do that. They do a number of experiments where they
 go in the episode and they kind of change stuff in the episode. And one cool thing here
 is that they go and you know, this is an episode. So at the episode, you start with a random
 network each time in this heavy and setting. And then pretty quickly, the rules adapt for
 a high performing right. So it starts to walk, it reconfigures itself and starts to walk.
 The reward here again, it doesn't have access to that, but we can measure it, of course.
 And then at this step A right here, they simply go to the weights and zero them out. So they
 just delete these weights right here. And only 10 time steps later, it has reconfigured
 itself as you can see right here in order to walk again. So 10 time steps later, reconfigures
 itself, reconfigures itself. And after a short while right here, it's back to its kind of
 original performance, as you can see. So that's, I'd say that's fairly impressive in this very
 short amount of time able to recover from such an intervention. If you do this, of course,
 if you do this to your policy network, that's statically learned, it's going to be garbage.
 But I guess the fair comparison would be to delete the heavy and rules themselves. And
 you know, so it's not like it's not like this can adapt to new situations, or something
 like this, this is still learned for particular environments, right. But the point here is
 that you learn the rules. And this is kind of a study on neuroplasticity. Now, my question
 actually would be why this diagonal pattern appears. And I have not seen like a clear
 explanation. Especially is this anti diagonal pattern, it's not so much here in the output
 layer, right, this is the output layer, there are 21 actions or so. And this one is this,
 this dimension. So not that much there. But there seems to be this rule. And this is not
 the case at the beginning, right, you saw the beginning you saw at the beginning, it
 was pretty random matrix. So why? Why? Yeah, here, pretty random. And then there's this
 diagonal pattern, I don't know why. If you know, let me know. I mean, it's anti diagonal,
 maybe it is actually diagonal and the forward, the fully connected layer is just defined
 as something like WT times x. And but maybe this also depends on the random initialization.
 But there is no inherent way why particular neuron would, you know, care about sending
 information to like the same height of neuron on the other side. Or is there? I don't know
 I'm so is this a property of the evolutionary or the learning rules? It seems not because
 the learning rules don't depend on the position. I'm genuinely confused about this. And maybe,
 you know, maybe they've written it somewhere, and I've just overlooked it, though. I, they
 do reference it, they say, oh, there's this diagonal pattern appearing, but I don't think
 they ever say why it is diagonal. Okay, I might just be I might just be real dumb. Yeah.
 So they also, you know, they do some more experiments, they show, for example, that
 if you just have random Hebbian coefficients, then your algorithm just jumps around kind
 of in in weight space around the zero point. However, if you actually learn these Hebbian
 coefficients, as they do, you have like this clear attractor here. And you have these kind
 of oscillating curves. When, you know, when when you do that, and you can see here in
 the different situations where things are damaged, and so on. So all in all, I think
 it's a pretty interesting study. And I think this neuroplasticity is, it's a different
 way, you know, it's unclear to say if it will ever deliver the performance that RL delivers,
 but certainly there are situations where such plasticity is desired. And if we can also
 combine this with greater generalization performance, then, you know, we have agents that can quickly
 kind of reconfigure. And a lot of work by these these kind of open ended learning community
 also plays into this roles, all in all pretty, pretty cool, non standard way of doing things.
 Last thing, the broader impact statement. Every now and then we'll look at a broader
 impact statement, since these are new, just to get kind of an overview of what they look
 like. So they say the ethical and mutual societal consequences of this work are hard to predict,
 but likely similar to other work dealing with more adaptive agent and robots. In particular,
 by giving the robots the ability to still function when injured could make it easier
 from them being deployed in areas that have both a positive and negative impact on society.
 Okay, well, again, this it's not really giving robots the ability to still function when
 they're injured. I first I thought first I thought, okay, they train it when it's fully
 functioning, but then they damage it during test time. But as I understand it, as I understand
 the paper, they already train it with the damaged versions, they just don't tell the
 algorithm in which version it is right now. So it's not the same as being able to work
 when injured unless you've specifically trained for it. In this case, again, I could be wrong
 about this. Yeah. In the very long term robots that can adapt could help in industrial automation
 or help to care for the elderly. On the other hand, more adaptive robots could also be more
 easily used for military applications. The approach presented in this paper is far from
 being deployed in these areas, but it's important to discuss its potential long term consequences
 early on. Now, okay, so let's evaluate the broader impact statement. Let's, well, the
 first check to do is always to simply replace whatever their method is with the word technology.
 Okay, so let's do that. In the very long term, technology could help in industrial automation
 or help to care for the elderly. Check. On the other hand, technology could also be more
 easily used for military application. Check. Technology is far from being deployed in these
 areas. Okay, I guess some technology isn't, but advanced technology. Yeah. So again, the
 rule for broader impact statements seem to be you take whatever your method is and you
 go up until you find, you know, you're basically at technology or something equivalent, because
 no one actually I've never seen a broader impact statement that writes about the actual
 thing in the paper, they always go up like one layer or two. And then it basically regresses
 to technology, even even though very few papers actually would be able to discuss their particular
 thing, but you know, and that and then in terms of guidelines on broader impact statement,
 this one is missing, there's there's always this, the holy trifecta. So the holy trifecta
 is you go like a, you know, like you're a, you're a Catholic, you go with your finger
 to your head, chest, left and right. And you say technology good, technology bad, technology
 biased. Okay, so you want to write a broader impact statement, go up the layers, technology,
 good bad bias, and we're missing the bias here. So that's, you know, I'm just following
 what these guidelines to broader impact statements are. I don't make the rules. I'm sorry that
 the heavy ins make the rules apparently. I'm not heavy in. Okay, I've I hope you've enjoyed
 this paper and this video. Let me know what you think. Check out the videos that they
 have. I'll link them. And with that, I wish you a pleasant day. Bye bye.