Fast and Slow Learning of Recurrent Independent Mechanisms (Machine Learning Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper recurrent independent mechanisms metarim deep learning tutorial introduction to deep learning what is deep learning machine learning paper deep reinforcement learning reinforcement learning meta learning yoshua bengio bentio mila grid world fast and slow learning reinforcement learning attention catastrophic forgetting lifelong learning multitask learning
#metarim #deeprl #catastrophicforgetting

Reinforcement Learning is very tricky in environments where the objective shifts over time. This paper explores agents in multi-task environments that are usually subject to catastrophic forgetting. Building on the concept of Recurrent Independent Mechanisms (RIM), the authors propose to separate the learning procedures for the mechanism parameters (fast) and the attention parameters (slow) and achieve superior results and more stability, and even better zero-shot transfer performance.

OUTLINE:
0:00 - Intro & Overview
3:30 - Recombining pieces of knowledge
11:30 - Controllers as recurrent neural networks
14:20 - Recurrent Independent Mechanisms
21:20 - Learning at different time scales
28:40 - Experimental Results & My Criticism
44:20 - Conclusion & Comments

Paper: https://arxiv.org/abs/2105.08710
RIM Paper: https://arxiv.org/abs/1909.10893

Abstract:
Decomposing knowledge into interchangeable pieces promises a generalization advantage when there are changes in distribution. A learning agent interacting with its environment is likely to be faced with situations requiring novel combinations of existing pieces of knowledge. We hypothesize that such a decomposition of knowledge is particularly relevant for being able to generalize in a systematic manner to out-of-distribution changes. To study these ideas, we propose a particular training framework in which we assume that the pieces of knowledge an agent needs and its reward function are stationary and can be re-used across tasks. An attention mechanism dynamically selects which modules can be adapted to the current task, and the parameters of the selected modules are allowed to change quickly as the learner is confronted with variations in what it experiences, while the parameters of the attention mechanisms act as stable, slowly changing, meta-parameters. We focus on pieces of knowledge captured by an ensemble of modules sparsely communicating with each other via a bottleneck of attention. We find that meta-learning the modular aspects of the proposed system greatly helps in achieving faster adaptation in a reinforcement learning setup involving navigation in a partially observed grid world with image-level input. We also find that reversing the role of parameters and meta-parameters does not work nearly as well, suggesting a particular role for fast adaptation of the dynamically selected modules.

Authors: Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, Yoshua Bengio

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there! Today we're looking at fast and slow learning of recurrent independent mechanisms by Kanika Madan, Rosemary Nankö, Aniruddh Goyal, Bernard Schilkopf and Joshua Benjo. So this paper on a high level proposes an update to a previous paper which was about recurrent independent mechanisms. The update it proposes is to learn the individual parameters of the different subsystems that comprise recurrent independent mechanisms at different time scales. The idea behind recurrent independent mechanisms is that you have sub-modules in a reinforcement learning agent that specialize on different sub-tasks that the agent has to do. Then you have sort of higher level modules which are attention based modules that select those sub-modules and decide how they communicate with each other. As I said, this paper here builds on that and proposes to learn these higher level parameters at different time scales than the lower level parameters such that the higher level units can generalize to multiple tasks. This helps you in environments where you have to do multiple tasks. So we're going to go over this paper and we're mostly also going to go over what recurrent independent mechanisms are. As I already said, this paper doesn't introduce recurrent independent mechanisms. That's a previous paper. It has some overlap in authors. So keep this in mind as we go through it. If you're specifically interested in recurrent independent mechanisms, I invite you to go read the previous paper. We'll go over both our IAMs and the update to it. In the end, this paper demonstrates that by decoupling the learning, you get benefits in environments where the structure of multi-task, multi-objective is given. It can generalize to unseen tasks pretty well. And on the other hand, I think for what this paper does right here, for the fact that it simply proposes this update, I don't think it does enough to demonstrate really that this is something worthwhile or it doesn't analyze it enough, I feel. And they also call this what they're doing meta learning, which I don't really agree to call this meta learning. But you'll see for yourself, we'll go over the paper. And yeah, bear with me. So as always, if you like content like this, don't hesitate to share it out and tell all your friends about it. And tell me what you think in the comments. They say in the abstract right here, decomposing knowledge into interchangeable pieces promises a generalization advantage when there are changes in distribution, a learning agent interacting with its environment is likely to be faced with situations requiring novel combinations of existing pieces of knowledge. So the hypothesis here is that if you are in an environment that has sort of different tasks inside of it that that where the environment itself changes, so your objective changes as well, then it might be helpful to recombine old knowledge. And the situation you have to have in mind with this paper is one of their core environments here is sort of a grid world environment. And the grid world environment is simply have this grid. And the agent occupies one cell right here, maybe the agent is here. And the agent can sort of move around here and do different actions. And there, there's going to be different things in this environment. So maybe there's like a key right here, this is a key. And maybe there's like a door over here. And the agent will get an instruction. Now the instruction in this environment might be get the key and go to then go to the door, then go to the door. Okay, so this might be the instruction. It might actually always be the same instruction in this particular environment. But if you change the key, and you change the door, where they are, that's already like different tasks, or it's not it's not the same environment all the time, you can also vary the size of these environments pretty easily. So all these tasks, these different tasks, they share some underlying structure, which is there's always kind of this world, and there's a key, and there is a door, and there might be a wall right here. So they all share this structure. However, what exactly you have to do differs from episode to episode. You can also imagine that there is maybe I don't know, maybe there's like an orange here. So there's an orange right here. And then the text instruction will say, get or go, go eat the orange. So now the agent has to ignore the key and the door and go to the orange, right. And additionally, so you can modulate this a lot. Additionally, you can say, okay, the agent maybe only sees its surrounding, maybe like this, right. So the agent only sees whatever is in front of it and a little bit to the side. So it needs to sort of turn around and explore. There's lots of variations. The important thing is that there's an environment that has some kind of over over structure overarching structure. And there's different tasks, and each episode is sort of a new task that the agent needs to solve. Now, what happens if the agent here is implemented in as in classic reinforcement or deep reinforcement learning as one big box like one neural network, and then you perform your episodes and you update the neural network, the parameters of the neural network according to your reward. If you solve one task, you're you will update according to that task, right. So if you solve the key, the key door task, let's call that, then your neural network, all the parameters will be updated with respect to that task, right. The way you train a neural network is that you change the parameters such that your loss decreases. So you train your neural network to solve that task as well as possible. But now the task changes, right, then all of a sudden, it's get the orange. Now all of a sudden, this doesn't give you reward anymore, right. And now the orange gives you a reward. So all the parameters you're going to change in order to serve this new task, you know, finding the orange, by the way, this is supposed to be like a little light spec. I'm terrible at this. I'm absolutely terrible at this. It's, it's like an orange donut. But you get what I mean, this, in general, in the fields of like lifelong learning and multitask learning, and so on, this is known as catastrophic forgetting. Catastrophic forgetting. I don't even know why I bother to write, like no one can read anyway. So there is lots of work in preventing catastrophic forgetting in these types of situations. And the way that this or the previous paper, the recurrent independent mechanisms proposed to do that is, let's not implement our agent as one big box, rather, let's implement it as a collection of like little sub modules. And these little sub modules, they focus on individual sub tasks. Okay, so a sub tasks might be fine, go to somewhere, okay, with the somewhere being a parameter that's then taken from the instructions, or maybe one one parameter specifically for recognizing the orange. So now, and the other one is for recognizing the key. Now, if the instructions say go to the key, the module that is recognizing the key might become active, and the module that is that is for going somewhere might become active, and the combination of the two might then get you to the key. So in each time step, the idea is let's only activate a sub part of these modules, not all of them at the same time. And now only these modules will be active, because they are relevant for the current tasks. And then only these modules will receive a learning signal, and not the other modules, okay, the other modules will stay fixed for that particular, for that particular step on in time. And this makes sense if you if you think about it, right, if your module isn't relevant for the task, then it shouldn't receive a learning update. And that's how you try to prevent catastrophic forgetting. So if this here, this module down here remembers to or can recognize the orange, and right now you're trying to find the key and get to the door, then if you don't, if you do update that module, it will be in service of the goal of finding the key and getting to the door. So it will forget the orange. However, if you decide no, this module isn't relevant for the current task, and then you prevent an update to it, then it won't forget the orange, it will only come into life once the task is actually about the orange. And then of course, you want the learning signal. So that's the idea right here, to prevent catastrophic forgetting, I do have my doubts that that is is so like that that scales to because the combinatorics of catastrophic forgetting are rather large, and therefore, but, you know, depending on how you factor the independent things you need to do, it, it is a good idea. Okay, so that's the core idea. It is that instead of having this one box, you have a lot of small boxes. And now you do this, right? These reinforcement learning problems, they're often implemented as like recurrent networks. And it's not a, it's not by chance that this thing is called a recurrent independent mechanisms. Because each of these little boxes like the big box would be is a recurrent neural network. So the way that these things work is that you have your different your inputs, which is frame by frame by frame, right. And the input goes through some sort of an encoder into a hidden state. And you do have your hidden state, that's from so the hidden state that the agent itself carries, this is kind of its internal memory. And you use the input frame of the game. So this is frame one, this is frame two, this is frame three, use the input frame, and your own hidden state to produce the next hidden state. And you can easily use this to create a new state. And you can easily implement this with some sort of an LSTM, right. And then you use that and that to produce the next hidden state. So that's the normal way of how things are done. Now in the so that's if you just have like an LSTM controller. Now if you have a recurrent independent mechanism controller, then your hidden state will be sort of a it will consist of many hidden states. So the hidden state itself will be a collection of hidden states, right. And so these are supposed to be little vectors. And then the input comes in here, and then only a subset is selected. So maybe this one and this one are selected. Now, the way that this works is, I shouldn't even draw one circle here, I should actually draw four circles. So you have four LSTM controllers, and only two of them are selected, I'm going to tell you how they're selected in a second. Actually, I'm going to tell you right now, probably that's better. So what what you do is you now let's let's do that after so you select two, you deactivate the other two. And the way you produce your next hidden state is sorry is simply you copy over the hidden states of the deactivated modules. So you just copy those over. So they remain. And you would update the hidden states of the modules that you selected. So only those modules are active. All right. So now, yeah, so that's, that's that. And there's also a communication step at the end. We'll go into that here, because here's the diagram. So down here, you see what I've just told you, this is the system. Okay, you have to imagine there is the last frame right here, there is the next frame down here, the frame and also the so that's the observation and the instruction, they go through some sort of an encoder, which would also be the same encoder up here and down there. Then there is the hidden state which is here in blue. So these are the independent mechanisms. Wait, that's the wrong blue. So we have in this case, four, four independent mechanisms, those would actually carry over over time, the state, the internal state of the agent. And then at each time step, you have an output of a value head and a policy head, the method they use right here is proximal policy optimization, as far as I understand it. This is a variant on actor critic method. If you don't know about deep reinforcement learning or proximal policy optimization, or actor critic methods, or why we need value and policy heads, I invite you to go look that up that it's fairly simple. It's very basic algorithm, where you can do reinforcement learning, you can calculate a loss and then you can back propagate to these either to the encoder and also to the to the parameters in the recurrent cells here. Okay, so how do we decide which modules are activated and which ones aren't, and that goes through an attention mechanism. And that's what they call here input attention. So input attention is the following, you have your input, okay. And you do have the encoder for the input, which is like maybe some concoction, some alchemic concoction of neural network, right, that gives you a vector like an embedding of the input. Now, you go to your little modules, each of them will have a hidden state already. And they get to do attention to that input. So the input will emit keys and queries. Now you can do this in multiple heads. But ultimately, let's do one vector. Okay, so here is a key. Sorry, it will emit keys and values. Okay, there is a key, and it will also emit the value we can we can just get we can just do like say the value is the input itself, if we do not have a if we don't have multiple heads, but ultimately, they emit keys and values. So they emit keys and values, and every single one of the mechanisms emits some sort of a query. So in essence, the input outputs a descriptor for what it contains, right, that's how you have to think about attention. And the the each of the mechanisms outputs a query for what they would like to see. So they get to look at their hidden state. And they get to decide what kind of information would I like to read from the input or what it's it's more like a filter, what kind of input is relevant to me. So the mechanism that cares about the orange, it would output probably a query for saying, is there something orangey in the input, either in the instructions or in the picture? Is there like something about an orange there? And the the one that cares about the key would obviously say, well, is there something about the key in there, but you can also imagine more abstract things. And then the attention is computed via inner product. And you can see here, it's those two mechanisms that are closest in inner product to the key. And then only those two get, get selected for this particular time step. And those get eliminated, not eliminated, but only the two on the right get to update the hidden state, as you can see right here. The ones that are not selected, they the hidden state is simply carried over. Whereas the ones that are selected, they actually get to do computation and update their hidden state. Now at the end of the update of the hidden state, there is a communication step. So these are not fully independent, they do get to communicate with each other. And so they here they have a new hidden state, and here they have an old hidden state. And now we get to communicate with each other. And again, the way this works is that every single one of them processes the input, actually, so the input goes through all of them. And all of these emit again, a query and sorry, a key of them emit a vector saying, you know, what did I get out of this input, even the ones that were not selected, they emit some sort of information. And the ones that were activated, they get to emit a query for what they would like to see of the other modules. And that's how you get the intercommunication, right? That's how you get to like higher order, independent mechanisms. So you could actually get a mechanism for going somewhere. And then that mechanism would query sort of another mechanism that says, well, where do I need to go? And the other mechanism that was like, well, I, I know where to go, because the instruction said, find an orange, and I'm the orange module. So I located the orange. So they get to communicate to to each other. So that there's going to be attention based communication, where the active modules read from both the other active modules and the inactive modules. And then you go to the next step, and you repeat and then the next step, it could be that different modules are activated, right? So these two attention mechanisms, the first one called the input attention, that selects the active modules, and then the second one called the communication attention that says how the different, how the different modules communicate with each other, those are sort of the higher level of communication higher level modules that control the flow of information of the lower level modules. And now, in the recurrent independent mechanisms paper, this, as I understand it, just learned end to end. Okay. Now this paper comes into action and says, wait a minute, shouldn't like, if, if we have the same environment, but different tasks, okay, so here you see individual episodes, and these individual episodes are comprised of a couple of time steps, okay. Now, they say, if we want to learn these little modules, such that they share knowledge, like they learn the independent things, and they can be recombined in different ways across the tasks, shouldn't we sort of, when we learn the individual modules, yes, we do the what they call fast update, we do the classic RL, where we learn maybe frame by frame or from short sequences within an episode. Okay. So if you know the goal, then let's learn the little pieces that make the goal happen. But in order to learn to select the pieces, you should look across different spans across different episodes. So that's what they call the slow update right here. So they propose to learn these meta parameters or what they call them, the communication parameters in a slower fashion, feeding in longer episodes. And here you can see it even spans across the different tasks. And the idea here is that the, these slower parameters, they consider longer time spans, they see multiple tasks at the same time, and they learn how to select the different modules, depending on the current input, the current task. And yeah, so by seeing different variants of that, in a single episodes, they get to they get to know the differences and the commonalities between tasks. Now that is a high goal. So here, my first problem is they call these like meta sequences. And yes, okay, they are meta sequences, but I disagree that that is meta learning. So what they ultimately do is here is algorithm one. So they randomly initialize the parameters of the attention units. And here the, the little mechanism units, they randomly initialize them. By the way, the also the the policy parameters are part of the meta unit parameters, and the value head parameters are then part of the attention parameters, they're not actually part of these modules, but they're learned also on different time scales. Okay, so the policy is learned fast, and the value is learned slow. That's just because feelings. So, well not done, we sample a batch, a batch of tasks, and then for each task, we sample a trajectory. And then we learn the modules, the mechanisms in the fashion, right, we, we keep the attention units, the attention right, we keep the attention parameters constant. That doesn't mean we always select the same module. The attention parameters being constant means that the way the queries and the keys are generated from the input that remains fixed. But it's still going to be differently selected modules from from from time to time. It's just that the way in which we select which ones are active aren't updated from time step to time step. And keeping that fixed, we learn the individual little things. We learn the mechanisms in a very classic fashion. So you can see right here, these are individual episodes, okay. The loss function is the proximal policy optimization loss, very classic with like an entropy term, and so on, they have it somewhere here. So this is a very classic PPO loss. This thing right here, you have this clip loss for the policy, you can see here is the so here is you have the probability ratio, which is sort of like the policy parameter, this is the current policy, this is the old policy. And then you have the value function loss, and then you have an entropy parameter loss. So quite a standard loss for reinforcement learning. And you learn that from individual episodes, and you update the parameters of the mechanisms, as we said, right, so you only activate the modules that are currently that are selected by the attention, and the back propagation would reflect that. In then in the second step, you sample again trajectories from tasks, but then instead of keeping the tasks and the episodes separate, you now concatenate all of them into what they call meta sequences. And then you update your attention parameters using those meta sequences while keeping the mechanisms constant. So in the first step, you learn, given sort of the activation policy of the mechanisms, how should the mechanisms behave in order to achieve good reward? So how they're selected remains constant, so they, they just get selected, and then they're, they're meant to maximize the reward. So any any mechanism here, you know, when they're selected, they're just being like, okay, what do I need to do to solve the current problem? And if they are selected in a consistent mechanism, that will cause them to specialize, right? If one is always selected, when the the orange thing is in the input, it will sort of start to specialize in these kinds of tasks. And in the other step, the mechanisms are kept constant. So you have the little sub modules that can achieve or can can can do certain sub tasks. And now you're trying to select the best ones of them. So you're trying to train the attention mechanism, how do you facilitate the selection and communication between these given fixed mechanisms, such that the reward is the highest. So in this two step fashion, the little mechanisms get better at the tasks they're tasked with, which causes them to to specialize if they're selected correctly. And then the selection itself is updated, which in turn makes the learning signal for the mechanisms better, and then better mechanisms make the learning signal for the selection better, and so on. You can imagine that this two step process is sort of, you know, kind of swinging itself up, bootstrapping itself up to very, very good interlocking pieces of things. Okay, in the experiments that looks fairly promising, you can see often see so they, they're not very often see so they not probably you can't see that the blue one is vanilla, which is sort of an LSTM controller, the green ones is the recurrent independent mechanism one, while the red one, I don't have red here I have orange, red one is this new two step approach. It's not always the case. And reinforcement learning is quite tricky. But this being largely the same authors, I guess, they do at least have a good comparison to recurrent independent mechanisms. Though I have to say this is measured in frames. So how many frames did you consume? And that is an important thing, because sample efficiency is important. But also given how complicated this scheme is, I wonder if this is slower or faster than just training both things at the same time, like the recurrent independent mechanisms did. Okay, so again, the difference between this and the last paper is simply that they, they propose this two step process where you have one step here, and another step here, instead of learning these two things jointly. And they do so deliberately in environments where you have multiple tasks given. So, you know, like, it's another lesson in, hey, you know, you need to evaluate on the things where you are really, really meant to be good at, and you need to evaluate in the quantity that you're meant to be good at. I'm not sure if time here would show the same plots if you had like in the x axis as time or computation or anything like this, it might very well be. So they demonstrate that they do, you know, a lot of have a lot of success with this, they demonstrate that if they train on, let's say small environments that they call difficult environments, that the meta rims, that's their system, the modular is the old paper and vanilla is the base implementation, they demonstrate that, even though they all get to fairly good success rate and reward on the difficult problems, if you make it zero shot, more difficult, so you increase the size of the problem with without ever having trained on the bigger problems, you make that room a lot bigger for finding the key, the these meta, what they call meta rims, they generalize a lot better than the other ones, right, you can see right here, the other ones largely fail, and they claim their system claim their system generalizes a lot better. So reinforcement learning, experimental results are very, very tricky, right, you can you you've already seen sort of the, just the, the bars here, the error bars up here, and that's after a long probably experimentation, maybe, and also selecting the right metrics and so on. Here, we don't even get bars. And here, it's, it's quite tricky, because not only do, for example, the vanilla ones generalize worse, they also start at a worse point, right, so they start at much less reward. And maybe that's responsible for them not generalizing so well, if you were to actually push like point nine, five to point nine, seven doesn't see much. But if you look, it's like, almost half the error, right? So like, if the maximum reward is one, then this gets, you know, five less than the maximum reward, and this only gets three less, this is quite a reduction, maybe that's the reason why at zero shot transfers to the more difficult environment. Also, here, the modular ones, which you have to remember is the exact same architecture as the meta learned ones, they don't even have a good success in these tasks. So the hypothesis of this paper here is that if you learn all these things at the same time, you will still be subject to catastrophic forgetting in these environments where you have multiple tasks, right, by learning the high level parameters in a slower way, in a first of all, in an independent way. Second of all, in a in a way where they see a longer sequences of things. And I do believe also, and this is also a bit unclear, I also do believe they do less update steps, maybe not. No, I think that it's just that their their steps that they consider the time steps they consider are four times more than the time steps that the individual that the learning here considers. So line six has some number of steps, n number of steps, and line nine here considers four times n, the number of steps, okay. So they consider longer time scales. If you want some other numbers, they always have five of these. So they always have five, which is what they call little n. And of the five, there are always k equals three active. So there are always three or five things active at any given point in time. And that is a bit of a different problem I have here. You know, to their contribution is, let's learn these higher level parameter independently, and in a more slow fashion. That's the contribution, right? Not the recurrent independent mechanisms, the the separation. Now, I would expect there to be a lot more investigation into what exactly this separation and slower learning is doing. They do have some ablations right here. But not many most ablations are about the recurrent independent mechanisms itself. So for example, here, they compare k equals three and two, and they show look across the episode, different modules become active as time progresses, which gives you an indication that yes, in fact, the different modules do specialize in different things, which is cool, right? That is not a property of this separation. That's a property of recurrent independent mechanisms. And here again, they the ablation they do here is different case of different number of sub modules being active. And you can see that if all the modules are active all the time, you have the pink curve, which is quite bad. And if only some modules are active here, like k equals three, you get a much better performance. Now, I would expect that that you actually try to go to k equals one or something like this to show maybe there's an optimal subset and so on. But again, this is a property of recurrent independent mechanisms. Only here where they say shorter meta episode. So here they say, what if we do the same thing that works well, but we make this meta episode shorter. And then you can see that the curve here, it also it sort of follows the trajectory of the of the worst baseline. Now, that is one thing right where they make they don't say how much shorter they make it, they just say we make it shorter. And that hurts. I mean, okay. Here, they analyze the value function, which is cool, you can sort of see that the value function reacts to different things in the environment. Again, that is not a that is not a property of what they're doing. And here, choice of attention, of attention, this is ablation choice of attention parameters as slow parameters. Okay, so they say now, let's do a different thing, let's actually flip, let's learn the attention parameters in a fast way. And the meta parameters in a sorry, the mechanism parameters in a slow way. And that's what they call meta flip. And here they show they show that that performs worse. Okay, so the the top one here is the meta what they propose. And the bottom one here is the flipped one where they learn the other parameters slow and the attention parameters fast. And again, okay, that's a that's a thing, right? But it's, it's not so much worse, honestly, like, and at some point, they say, well, it's somewhat worse. And in the texts, and they say that is did not perform very well, right here, this did not perform very well. And I disagree a bit, like it performed okay, like it's certainly better than the than the vanilla one, it looks like it may be at the same as the vanilla one. It doesn't seem super duper bad. And I just don't think this is since this paper is about adding this thing, the addition of this thing, and the sort of, you know, how much that contributes, and what exactly of the thing makes the algorithm stronger. I don't think that's explored enough in this paper, I think too much space is wasted on exploring like the value function and which modules are active, which we already know from the recurrent independent mechanisms, right? There are, in fact, two things going on, right? There is the slowness, there is the fact of, hey, let's learn one set of parameters more slowly than another set of parameters. That's one thing. And the other thing is, hey, let's decouple learning the two parameters. Now, the decoupling actually is what I think makes it not meta. This is simply decoupling. This is not meta learning, as far as I'm concerned. This is not learning to learn or anything like this. It's simply that we have two different things, and we learn them at two different times. This is very much like, you know, in the beginning of GANs, you have whatever your generator, and your discriminator, and here and here you have your, your data set. And here you have your binary classification, and here you have your latent vector. Okay, these, this is basic drawing of a GAN. And what people used to do, at least at the beginning, before we realized how we can stabilize GAN training, is they did these independently. They said, I'm going to do one step, learning the discriminator, and then I'm going to do another step, learning the generator, instead of updating them both at the same time. And at the beginning, we even did things like, hey, let's learn the generator for five steps, and let's learn the discriminator only for one step. And then we learned that we can do the same thing for the discriminator, but only for one step, once we get to the discriminator. So it is exactly the same thing. It was that was not meta learning. This is simply the fact that if you have a system where the parameters are sort of entangled with each other, like the discriminator depends on the output of another system, which itself has got to get you into trouble that can get you into instability. And therefore, it might be a good idea to separate these and if one system is sort of stronger than the other system, it might also be effective to learn these at different time scales, there's nothing sort of to do with meta learning. And it's two different things, right? This time scale and the separation are two different things. And yeah, these are not entangled here. And they also compare with what they call slow LR, they say, well, in order to compare what we can also do is we can simply learn the parameters of the attention and the mechanisms at the same time, but we can give the we can give the attention simply a lower learning rate. Like we divide the instead of dividing the number of steps by four, we divide the learning rate by four, and they stay show that doesn't work. And I mean, it's not a surprise that doesn't work. That is absolutely not the same thing, right? It's and I'm not even sure what it's supposed to show, I guess it's supposed to show that that you need the separation instead, the slowness itself isn't a thing. But I don't think you, even if the slowness was a thing, it's it is not that you can simply replace the number of steps by a smaller learning rate. Yeah, in any case, but it is it is at least like some kind of experiment that that shows something about the system, right? What I would expect from an experiment like this is, yeah, here again, like what the modules are learning, which is cool, like it's cool that you show, look, this module is learning this, this one is active when that happens, and so on. And we can ablate the winner modules. So what they do is they take the modules that are selected, and then randomly drop out some of them, and they discover, well, what is this? And then they can actually say, well, the more we drop out, the less well it works. Wow. But there's no investigation into, okay, what is the effect of learning one thing more slowly? How much is the effect? Can we modulate that? Can we set the number of slow steps equal to five to six to 10 to 20? You know, can we can we discuss how long these meta episodes need to be like here is just like shorter, okay, but there's no indication like how long do they need to be? What's a good length? Then give us give us like the time penalty that we incur here, not only the frames, right? What's what's the time penalty? Might there be already something good about simply separating the updates? You know, like all of this kind of stuff is not really explored in this paper. So again, there is really cool parts about this paper, it makes sense to separate these two because you have an interdependent system reinforcement learning is brittle enough already. And it really seems to help against this catastrophic forgetting. However, for the fact that this paper simply adds this two step approach, I don't think it does enough to show what they're doing and to show the reasons of why what they're doing works works. And also I object to this being called meta learning. So that is my opinion. Please tell me your opinion. This was a bit more ranty than I usually do. But I hope you're still here. And I'll see you next time. Bye bye.
 Hi there! Today we're looking at fast and slow learning of recurrent independent mechanisms
 by Kanika Madan, Rosemary Nankö, Aniruddh Goyal, Bernard Schilkopf and Joshua Benjo.
 So this paper on a high level proposes an update to a previous paper which was about recurrent
 independent mechanisms. The update it proposes is to learn the individual parameters of the
 different subsystems that comprise recurrent independent mechanisms at different time scales.
 The idea behind recurrent independent mechanisms is that you have sub-modules in a reinforcement
 learning agent that specialize on different sub-tasks that the agent has to do. Then you
 have sort of higher level modules which are attention based modules that select those sub-modules
 and decide how they communicate with each other. As I said, this paper here builds on that and
 proposes to learn these higher level parameters at different time scales than the lower level
 parameters such that the higher level units can generalize to multiple tasks. This helps you in
 environments where you have to do multiple tasks. So we're going to go over this paper and we're
 mostly also going to go over what recurrent independent mechanisms are. As I already said,
 this paper doesn't introduce recurrent independent mechanisms. That's a previous paper. It has some
 overlap in authors. So keep this in mind as we go through it. If you're specifically interested
 in recurrent independent mechanisms, I invite you to go read the previous paper. We'll go over both
 our IAMs and the update to it. In the end, this paper demonstrates that by decoupling the learning,
 you get benefits in environments where the structure of multi-task, multi-objective
 is given. It can generalize to unseen tasks pretty well. And on the other hand, I think
 for what this paper does right here, for the fact that it simply proposes this update, I don't think
 it does enough to demonstrate really that this is something worthwhile or it doesn't analyze it
 enough, I feel. And they also call this what they're doing meta learning, which I don't really agree
 to call this meta learning. But you'll see for yourself, we'll go over the paper. And yeah,
 bear with me. So as always, if you like content like this, don't hesitate to share it out and
 tell all your friends about it. And tell me what you think in the comments. They say in the abstract
 right here, decomposing knowledge into interchangeable pieces promises a generalization advantage
 when there are changes in distribution, a learning agent interacting with its environment is likely
 to be faced with situations requiring novel combinations of existing pieces of knowledge.
 So the hypothesis here is that if you are in an environment that has sort of different tasks
 inside of it that that where the environment itself changes, so your objective changes as well,
 then it might be helpful to recombine old knowledge. And the situation you have to have
 in mind with this paper is one of their core environments here is sort of a grid world
 environment. And the grid world environment is simply have this grid. And the agent occupies
 one cell right here, maybe the agent is here. And the agent can sort of move around here and
 do different actions. And there, there's going to be different things in this environment. So maybe
 there's like a key right here, this is a key. And maybe there's like a door over here. And
 the agent will get an instruction. Now the instruction in this environment might be
 get the key and go to then go to the door, then go to the door. Okay, so this might be the
 instruction. It might actually always be the same instruction in this particular environment. But
 if you change the key, and you change the door, where they are, that's already like different
 tasks, or it's not it's not the same environment all the time, you can also vary the size of these
 environments pretty easily. So all these tasks, these different tasks, they share some underlying
 structure, which is there's always kind of this world, and there's a key, and there is a door,
 and there might be a wall right here. So they all share this structure. However, what exactly you
 have to do differs from episode to episode. You can also imagine that there is maybe I don't know,
 maybe there's like an orange here. So there's an orange right here. And then the text instruction
 will say, get or go, go eat the orange. So now the agent has to ignore the key and the door and go
 to the orange, right. And additionally, so you can modulate this a lot. Additionally, you can say,
 okay, the agent maybe only sees its surrounding, maybe like this, right. So the agent only sees
 whatever is in front of it and a little bit to the side. So it needs to sort of turn around and
 explore. There's lots of variations. The important thing is that there's an environment that has some
 kind of over over structure overarching structure. And there's different tasks, and each episode is
 sort of a new task that the agent needs to solve. Now, what happens if the agent here is implemented
 in as in classic reinforcement or deep reinforcement learning as one big box like one
 neural network, and then you perform your episodes and you update the neural network,
 the parameters of the neural network according to your reward. If you solve one task, you're you
 will update according to that task, right. So if you solve the key, the key door task, let's call
 that, then your neural network, all the parameters will be updated with respect to that task, right.
 The way you train a neural network is that you change the parameters such that your loss decreases.
 So you train your neural network to solve that task as well as possible. But now the task changes,
 right, then all of a sudden, it's get the orange. Now all of a sudden, this doesn't give you reward
 anymore, right. And now the orange gives you a reward. So all the parameters you're going to
 change in order to serve this new task, you know, finding the orange, by the way, this is supposed
 to be like a little light spec. I'm terrible at this. I'm absolutely terrible at this. It's,
 it's like an orange donut. But you get what I mean, this, in general, in the fields of like
 lifelong learning and multitask learning, and so on, this is known as catastrophic forgetting.
 Catastrophic forgetting. I don't even know why I bother to write, like no one can read anyway.
 So there is lots of work in preventing catastrophic forgetting in these types of situations.
 And the way that this or the previous paper, the recurrent independent mechanisms proposed to do
 that is, let's not implement our agent as one big box, rather, let's implement it as a collection
 of like little sub modules. And these little sub modules, they focus on individual sub tasks. Okay,
 so a sub tasks might be fine, go to somewhere, okay, with the somewhere being a parameter that's
 then taken from the instructions, or maybe one one parameter specifically for recognizing the orange.
 So now, and the other one is for recognizing the key. Now, if the instructions say go to the key,
 the module that is recognizing the key might become active, and the module that is that is
 for going somewhere might become active, and the combination of the two might then get you to the
 key. So in each time step, the idea is let's only activate a sub part of these modules,
 not all of them at the same time. And now only these modules will be active, because they are
 relevant for the current tasks. And then only these modules will receive a learning signal,
 and not the other modules, okay, the other modules will stay fixed for that particular,
 for that particular step on in time. And this makes sense if you if you think about it, right,
 if your module isn't relevant for the task, then it shouldn't receive a learning update.
 And that's how you try to prevent catastrophic forgetting. So if this here, this module down here
 remembers to or can recognize the orange, and right now you're trying to find the key and get
 to the door, then if you don't, if you do update that module, it will be in service of the goal of
 finding the key and getting to the door. So it will forget the orange. However, if you decide no,
 this module isn't relevant for the current task, and then you prevent an update to it, then it
 won't forget the orange, it will only come into life once the task is actually about the orange.
 And then of course, you want the learning signal. So that's the idea right here, to prevent
 catastrophic forgetting, I do have my doubts that that is is so like that that scales to because the
 combinatorics of catastrophic forgetting are rather large, and therefore, but, you know, depending on
 how you factor the independent things you need to do, it, it is a good idea. Okay, so that's the
 core idea. It is that instead of having this one box, you have a lot of small boxes. And now you
 do this, right? These reinforcement learning problems, they're often implemented as like
 recurrent networks. And it's not a, it's not by chance that this thing is called a
 recurrent independent mechanisms. Because each of these little boxes like the big box would be is
 a recurrent neural network. So the way that these things work is that you have your different your
 inputs, which is frame by frame by frame, right. And the input goes through some sort of an encoder
 into a hidden state. And you do have your hidden state, that's from so the hidden state that the
 agent itself carries, this is kind of its internal memory. And you use the input frame of the game.
 So this is frame one, this is frame two, this is frame three, use the input frame, and your own
 hidden state to produce the next hidden state. And you can easily use this to create a new
 state. And you can easily implement this with some sort of an LSTM, right. And then you use that and
 that to produce the next hidden state. So that's the normal way of how things are done. Now in the
 so that's if you just have like an LSTM controller. Now if you have a recurrent independent mechanism
 controller, then your hidden state will be sort of a it will consist of many hidden states. So the
 hidden state itself will be a collection of hidden states, right. And so these are supposed to be
 little vectors. And then the input comes in here, and then only a subset is selected. So maybe
 this one and this one are selected. Now, the way that this works is, I shouldn't even draw one
 circle here, I should actually draw four circles. So you have four LSTM controllers, and only two of
 them are selected, I'm going to tell you how they're selected in a second. Actually, I'm going to
 tell you right now, probably that's better. So what what you do is you now let's let's do that after
 so you select two, you deactivate the other two. And the way you produce your next hidden state is
 sorry is simply you copy over the hidden states of the deactivated modules. So you just copy those
 over. So they remain. And you would update the hidden states of the modules that you selected.
 So only those modules are active. All right. So now, yeah, so that's, that's that. And there's
 also a communication step at the end. We'll go into that here, because here's the diagram. So down
 here, you see what I've just told you, this is the system. Okay, you have to imagine there is the
 last frame right here, there is the next frame down here, the frame and also the so that's the
 observation and the instruction, they go through some sort of an encoder, which would also be the
 same encoder up here and down there. Then there is the hidden state which is here in blue. So these
 are the independent mechanisms. Wait, that's the wrong blue. So we have in this case, four,
 four independent mechanisms, those would actually carry over over time, the state, the internal
 state of the agent. And then at each time step, you have an output of a value head and a policy
 head, the method they use right here is proximal policy optimization, as far as I understand it.
 This is a variant on actor critic method. If you don't know about deep reinforcement learning or
 proximal policy optimization, or actor critic methods, or why we need value and policy heads,
 I invite you to go look that up that it's fairly simple. It's very basic algorithm,
 where you can do reinforcement learning, you can calculate a loss and then you can back propagate
 to these either to the encoder and also to the to the parameters in the recurrent cells here.
 Okay, so how do we decide which modules are activated and which ones aren't, and that goes
 through an attention mechanism. And that's what they call here input attention. So input attention
 is the following, you have your input, okay. And you do have the encoder for the input, which is
 like maybe some concoction, some alchemic concoction of neural network, right, that gives you
 a vector like an embedding of the input. Now, you go to your little modules, each of them will have
 a hidden state already. And they get to do attention to that input. So the input will emit
 keys and queries. Now you can do this in multiple heads. But ultimately, let's do one vector. Okay,
 so here is a key. Sorry, it will emit keys and values. Okay, there is a key, and it will also
 emit the value we can we can just get we can just do like say the value is the input itself, if we
 do not have a if we don't have multiple heads, but ultimately, they emit keys and values. So
 they emit keys and values, and every single one of the mechanisms emits some sort of a query.
 So in essence, the input outputs a descriptor for what it contains, right, that's how you have to
 think about attention. And the the each of the mechanisms outputs a query for what they would
 like to see. So they get to look at their hidden state. And they get to decide what kind of
 information would I like to read from the input or what it's it's more like a filter, what kind
 of input is relevant to me. So the mechanism that cares about the orange, it would output probably
 a query for saying, is there something orangey in the input, either in the instructions or in
 the picture? Is there like something about an orange there? And the the one that cares about
 the key would obviously say, well, is there something about the key in there, but you can
 also imagine more abstract things. And then the attention is computed via inner product.
 And you can see here, it's those two mechanisms that are closest in inner product to the key.
 And then only those two get, get selected for this particular time step. And those get eliminated,
 not eliminated, but only the two on the right get to update the hidden state, as you can see
 right here. The ones that are not selected, they the hidden state is simply carried over.
 Whereas the ones that are selected, they actually get to do computation and update their hidden
 state. Now at the end of the update of the hidden state, there is a communication step. So these are
 not fully independent, they do get to communicate with each other. And so they here they have a new
 hidden state, and here they have an old hidden state. And now we get to communicate with each
 other. And again, the way this works is that every single one of them processes the input,
 actually, so the input goes through all of them. And all of these emit again, a query and sorry,
 a key of them emit a vector saying, you know, what did I get out of this input, even the ones
 that were not selected, they emit some sort of information. And the ones that were activated,
 they get to emit a query for what they would like to see of the other modules. And that's how you
 get the intercommunication, right? That's how you get to like higher order, independent mechanisms.
 So you could actually get a mechanism for going somewhere. And then that mechanism would query
 sort of another mechanism that says, well, where do I need to go? And the other mechanism that was
 like, well, I, I know where to go, because the instruction said, find an orange, and I'm the
 orange module. So I located the orange. So they get to communicate to to each other. So that there's
 going to be attention based communication, where the active modules read from both the other active
 modules and the inactive modules. And then you go to the next step, and you repeat and then the next
 step, it could be that different modules are activated, right? So these two attention
 mechanisms, the first one called the input attention, that selects the active modules,
 and then the second one called the communication attention that says how the different, how the
 different modules communicate with each other, those are sort of the higher level of communication
 higher level modules that control the flow of information of the lower level modules. And now,
 in the recurrent independent mechanisms paper, this, as I understand it, just learned end to end.
 Okay. Now this paper comes into action and says, wait a minute, shouldn't like, if, if we have
 the same environment, but different tasks, okay, so here you see individual episodes,
 and these individual episodes are comprised of a couple of time steps, okay. Now, they say, if we
 want to learn these little modules, such that they share knowledge, like they learn the independent
 things, and they can be recombined in different ways across the tasks, shouldn't we sort of,
 when we learn the individual modules, yes, we do the what they call fast update, we do the classic
 RL, where we learn maybe frame by frame or from short sequences within an episode. Okay. So if
 you know the goal, then let's learn the little pieces that make the goal happen. But in order
 to learn to select the pieces, you should look across different spans across different episodes.
 So that's what they call the slow update right here. So they propose to learn these meta parameters
 or what they call them, the communication parameters in a slower fashion, feeding in
 longer episodes. And here you can see it even spans across the different tasks. And the idea
 here is that the, these slower parameters, they consider longer time spans, they see multiple
 tasks at the same time, and they learn how to select the different modules, depending on
 the current input, the current task. And yeah, so by seeing different variants of that,
 in a single episodes, they get to they get to know the differences and the commonalities between
 tasks. Now that is a high goal. So here, my first problem is they call these like meta sequences.
 And yes, okay, they are meta sequences, but I disagree that that is meta learning. So what
 they ultimately do is here is algorithm one. So they randomly initialize the parameters of the
 attention units. And here the, the little mechanism units, they randomly initialize them.
 By the way, the also the the policy parameters are part of the meta unit parameters, and the value
 head parameters are then part of the attention parameters, they're not actually part of these
 modules, but they're learned also on different time scales. Okay, so the policy is learned
 fast, and the value is learned slow. That's just because feelings. So, well not done, we sample a
 batch, a batch of tasks, and then for each task, we sample a trajectory. And then we learn the
 modules, the mechanisms in the fashion, right, we, we keep the attention units, the attention
 right, we keep the attention parameters constant. That doesn't mean we always select the same module.
 The attention parameters being constant means that the way the queries and the keys are generated
 from the input that remains fixed. But it's still going to be differently selected modules from
 from from time to time. It's just that the way in which we select which ones are active aren't
 updated from time step to time step. And keeping that fixed, we learn the individual little things.
 We learn the mechanisms in a very classic fashion. So you can see right here, these are individual
 episodes, okay. The loss function is the proximal policy optimization loss, very classic with like
 an entropy term, and so on, they have it somewhere here. So this is a very classic PPO loss.
 This thing right here, you have this clip loss for the policy, you can see here is the
 so here is you have the probability ratio, which is sort of like the policy parameter,
 this is the current policy, this is the old policy.
 And then you have the value function loss, and then you have an entropy parameter loss.
 So quite a standard loss for reinforcement learning. And you learn that from individual
 episodes, and you update the parameters of the mechanisms, as we said, right, so you only
 activate the modules that are currently that are selected by the attention, and the back propagation
 would reflect that. In then in the second step, you sample again trajectories from tasks, but then
 instead of keeping the tasks and the episodes separate, you now concatenate all of them into
 what they call meta sequences. And then you update your attention parameters using those
 meta sequences while keeping the mechanisms constant. So in the first step, you learn,
 given sort of the activation policy of the mechanisms, how should the mechanisms
 behave in order to achieve good reward? So how they're selected remains constant,
 so they, they just get selected, and then they're, they're meant to maximize the reward.
 So any any mechanism here, you know, when they're selected, they're just being like, okay,
 what do I need to do to solve the current problem? And if they are selected in a consistent
 mechanism, that will cause them to specialize, right? If one is always selected, when the the
 orange thing is in the input, it will sort of start to specialize in these kinds of tasks.
 And in the other step, the mechanisms are kept constant. So you have the little sub modules
 that can achieve or can can can do certain sub tasks. And now you're trying to select the best
 ones of them. So you're trying to train the attention mechanism, how do you facilitate
 the selection and communication between these given fixed mechanisms, such that the reward is
 the highest. So in this two step fashion, the little mechanisms get better at the tasks they're
 tasked with, which causes them to to specialize if they're selected correctly. And then the
 selection itself is updated, which in turn makes the learning signal for the mechanisms better,
 and then better mechanisms make the learning signal for the selection better, and so on.
 You can imagine that this two step process is sort of, you know, kind of swinging itself up,
 bootstrapping itself up to very, very good interlocking pieces of things. Okay, in the
 experiments that looks fairly promising, you can see often see so they, they're not very
 often see so they not probably you can't see that the blue one is vanilla, which is sort of an LSTM
 controller, the green ones is the recurrent independent mechanism one, while the red one,
 I don't have red here I have orange, red one is this new two step approach. It's not always the
 case. And reinforcement learning is quite tricky. But this being largely the same authors, I guess,
 they do at least have a good comparison to recurrent independent mechanisms. Though I have
 to say this is measured in frames. So how many frames did you consume? And that is an important
 thing, because sample efficiency is important. But also given how complicated this scheme is,
 I wonder if this is slower or faster than just training both things at the same time,
 like the recurrent independent mechanisms did. Okay, so again, the difference between this and
 the last paper is simply that they, they propose this two step process where you have one step
 here, and another step here, instead of learning these two things jointly. And they do so deliberately
 in environments where you have multiple tasks given. So, you know, like, it's another lesson in,
 hey, you know, you need to evaluate on the things where you are really, really meant to be good at,
 and you need to evaluate in the quantity that you're meant to be good at. I'm not sure if time
 here would show the same plots if you had like in the x axis as time or computation or anything
 like this, it might very well be. So they demonstrate that they do, you know, a lot of
 have a lot of success with this, they demonstrate that if they train on, let's say small
 environments that they call difficult environments, that the meta rims, that's their system,
 the modular is the old paper and vanilla is the base implementation, they demonstrate that,
 even though they all get to fairly good success rate and reward on the difficult problems,
 if you make it zero shot, more difficult, so you increase the size of the problem with without
 ever having trained on the bigger problems, you make that room a lot bigger for finding the key,
 the these meta, what they call meta rims, they generalize a lot better than the other ones,
 right, you can see right here, the other ones largely fail, and they claim their system
 claim their system generalizes a lot better. So reinforcement learning, experimental results
 are very, very tricky, right, you can you you've already seen sort of the, just the, the bars here,
 the error bars up here, and that's after a long probably experimentation, maybe, and also selecting
 the right metrics and so on. Here, we don't even get bars. And here, it's, it's quite tricky,
 because not only do, for example, the vanilla ones generalize worse, they also start at a worse
 point, right, so they start at much less reward. And maybe that's responsible for them not
 generalizing so well, if you were to actually push like point nine, five to point nine, seven doesn't see much.
 But if you look, it's like, almost half the error, right? So like, if the maximum reward is one,
 then this gets, you know, five less than the maximum reward, and this only gets three less,
 this is quite a reduction, maybe that's the reason why at zero shot transfers to the more difficult
 environment. Also, here, the modular ones, which you have to remember is the exact same architecture
 as the meta learned ones, they don't even have a good success in these tasks. So the hypothesis of
 this paper here is that if you learn all these things at the same time, you will still be subject
 to catastrophic forgetting in these environments where you have multiple tasks, right, by learning
 the high level parameters in a slower way, in a first of all, in an independent way. Second of all,
 in a in a way where they see a longer sequences of things. And I do believe also, and this is also a
 bit unclear, I also do believe they do less update steps, maybe not. No, I think that it's just that
 their their steps that they consider the time steps they consider are four times more than the
 time steps that the individual that the learning here considers. So line six has some number of
 steps, n number of steps, and line nine here considers four times n, the number of steps,
 okay. So they consider longer time scales. If you want some other numbers, they always have
 five of these. So they always have five, which is what they call little n. And of the five,
 there are always k equals three active. So there are always three or five things active at any
 given point in time. And that is a bit of a different problem I have here. You know, to
 their contribution is, let's learn these higher level parameter independently, and in a more slow
 fashion. That's the contribution, right? Not the recurrent independent mechanisms, the the separation.
 Now, I would expect there to be a lot more investigation into what exactly this separation
 and slower learning is doing. They do have some ablations right here. But not many most ablations
 are about the recurrent independent mechanisms itself. So for example, here, they compare k
 equals three and two, and they show look across the episode, different modules become active
 as time progresses, which gives you an indication that yes, in fact, the different modules do
 specialize in different things, which is cool, right? That is not a property of this separation.
 That's a property of recurrent independent mechanisms. And here again, they the ablation
 they do here is different case of different number of sub modules being active. And you can see that
 if all the modules are active all the time, you have the pink curve, which is quite bad. And if
 only some modules are active here, like k equals three, you get a much better performance. Now,
 I would expect that that you actually try to go to k equals one or something like this to show
 maybe there's an optimal subset and so on. But again, this is a property of recurrent independent
 mechanisms. Only here where they say shorter meta episode. So here they say, what if we do the same
 thing that works well, but we make this meta episode shorter. And then you can see that the
 curve here, it also it sort of follows the trajectory of the of the worst baseline. Now,
 that is one thing right where they make they don't say how much shorter they make it, they just say
 we make it shorter. And that hurts. I mean, okay. Here, they analyze the value function, which is
 cool, you can sort of see that the value function reacts to different things in the environment.
 Again, that is not a that is not a property of what they're doing. And here, choice of attention,
 of attention, this is ablation choice of attention parameters as slow parameters. Okay, so they say
 now, let's do a different thing, let's actually flip, let's learn the attention parameters in a
 fast way. And the meta parameters in a sorry, the mechanism parameters in a slow way. And that's
 what they call meta flip. And here they show they show that that performs worse. Okay, so the the
 top one here is the meta what they propose. And the bottom one here is the flipped one where they
 learn the other parameters slow and the attention parameters fast. And again, okay, that's a that's
 a thing, right? But it's, it's not so much worse, honestly, like, and at some point, they say, well,
 it's somewhat worse. And in the texts, and they say that is did not perform very well, right here,
 this did not perform very well. And I disagree a bit, like it performed okay, like it's certainly
 better than the than the vanilla one, it looks like it may be at the same as the vanilla one. It
 doesn't seem super duper bad. And I just don't think this is since this paper is about adding
 this thing, the addition of this thing, and the sort of, you know, how much that contributes,
 and what exactly of the thing makes the algorithm stronger. I don't think that's explored enough in
 this paper, I think too much space is wasted on exploring like the value function and which modules
 are active, which we already know from the recurrent independent mechanisms, right? There are,
 in fact, two things going on, right? There is the slowness, there is the fact of, hey, let's learn
 one set of parameters more slowly than another set of parameters. That's one thing. And the other
 thing is, hey, let's decouple learning the two parameters. Now, the decoupling actually is what
 I think makes it not meta. This is simply decoupling. This is not meta learning, as far as
 I'm concerned. This is not learning to learn or anything like this. It's simply that we have two
 different things, and we learn them at two different times. This is very much like,
 you know, in the beginning of GANs, you have whatever your generator, and your discriminator,
 and here and here you have your, your data set. And here you have your binary classification,
 and here you have your latent vector. Okay, these, this is basic drawing of a GAN. And
 what people used to do, at least at the beginning, before we realized how we can stabilize GAN
 training, is they did these independently. They said, I'm going to do one step, learning the
 discriminator, and then I'm going to do another step, learning the generator, instead of updating
 them both at the same time. And at the beginning, we even did things like, hey, let's learn the
 generator for five steps, and let's learn the discriminator only for one step. And then we
 learned that we can do the same thing for the discriminator, but only for one step, once we
 get to the discriminator. So it is exactly the same thing. It was that was not meta learning.
 This is simply the fact that if you have a system where the parameters are sort of entangled with
 each other, like the discriminator depends on the output of another system, which itself has
 got to get you into trouble that can get you into instability. And therefore, it might be a good idea
 to separate these and if one system is sort of stronger than the other system, it might also
 be effective to learn these at different time scales, there's nothing sort of to do with meta
 learning. And it's two different things, right? This time scale and the separation are two different
 things. And yeah, these are not entangled here. And they also compare with what they call slow
 LR, they say, well, in order to compare what we can also do is we can simply learn the parameters
 of the attention and the mechanisms at the same time, but we can give the we can give the
 attention simply a lower learning rate. Like we divide the instead of dividing the number of
 steps by four, we divide the learning rate by four, and they stay show that doesn't work. And
 I mean, it's not a surprise that doesn't work. That is absolutely not the same thing, right?
 It's and I'm not even sure what it's supposed to show, I guess it's supposed to show that
 that you need the separation instead, the slowness itself isn't a thing. But I don't think you,
 even if the slowness was a thing, it's it is not that you can simply replace the number of steps
 by a smaller learning rate. Yeah, in any case, but it is it is at least like some kind of experiment
 that that shows something about the system, right? What I would expect from an experiment like this
 is, yeah, here again, like what the modules are learning, which is cool, like it's cool that you
 show, look, this module is learning this, this one is active when that happens, and so on. And we can
 ablate the winner modules. So what they do is they take the modules that are selected, and then
 randomly drop out some of them, and they discover, well, what is this? And then they can actually
 say, well, the more we drop out, the less well it works. Wow. But there's no investigation into,
 okay, what is the effect of learning one thing more slowly? How much is the effect? Can we
 modulate that? Can we set the number of slow steps equal to five to six to 10 to 20? You know, can we
 can we discuss how long these meta episodes need to be like here is just like shorter, okay, but
 there's no indication like how long do they need to be? What's a good length? Then give us give us
 like the time penalty that we incur here, not only the frames, right? What's what's the time
 penalty? Might there be already something good about simply separating the updates?
 You know, like all of this kind of stuff is not really explored in this paper. So again, there is
 really cool parts about this paper, it makes sense to separate these two because you have
 an interdependent system reinforcement learning is brittle enough already. And it really seems
 to help against this catastrophic forgetting. However, for the fact that this paper simply adds
 this two step approach, I don't think it does enough to show what they're doing and to show
 the reasons of why what they're doing works works. And also I object to this being called meta
 learning. So that is my opinion. Please tell me your opinion. This was a bit more ranty than I
 usually do. But I hope you're still here. And I'll see you next time. Bye bye.