Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments (Review)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper active dendrites neurons dendrites biological deep learning deep learning biology numenta numenta research numenta deep learning dendrites deep learning deep learning tutorial hierarchical temporal memory computational neuroscience reinforcement learning robotics multi task learning continuous learning continual learning permuted mnist
#multitasklearning #biology #neuralnetworks

Catastrophic forgetting is a big problem in mutli-task and continual learning. Gradients of different objectives tend to conflict, and new tasks tend to override past knowledge. In biological neural networks, each neuron carries a complex network of dendrites that mitigate such forgetting by recognizing the context of an input signal. This paper introduces Active Dendrites, which carries over the principle of context-sensitive gating by dendrites into the deep learning world. Various experiments show the benefit in combatting catastrophic forgetting, while preserving sparsity and limited parameter counts.

OUTLINE:
0:00 - Introduction
1:20 - Paper Overview
3:15 - Catastrophic forgetting in continuous and multi-task learning
9:30 - Dendrites in biological neurons
16:55 - Sparse representations in biology
18:35 - Active dendrites in deep learning
34:15 - Experiments on multi-task learning
39:00 - Experiments in continual learning and adaptive prototyping
49:20 - Analyzing the inner workings of the algorithm
53:30 - Is this the same as just training a larger network?
59:15 - How does this relate to attention mechanisms?
1:02:55 - Final thoughts and comments

Paper: https://arxiv.org/abs/2201.00042
Blog: https://numenta.com/blog/2021/11/08/can-active-dendrites-mitigate-catastrophic-forgetting

ERRATA:
- I was made aware of this by https://twitter.com/ChainlessCoder: "That axon you showed of the pyramidal neuron, is actually the apical dendrite of the neuron". Sorry, my bad :)

Abstract:
A key challenge for AI is to build embodied systems that operate in dynamically changing environments. Such systems must adapt to changing task contexts and learn continuously. Although standard deep learning systems achieve state of the art results on static benchmarks, they often struggle in dynamic scenarios. In these settings, error signals from multiple contexts can interfere with one another, ultimately leading to a phenomenon known as catastrophic forgetting. In this article we investigate biologically inspired architectures as solutions to these problems. Specifically, we show that the biophysical properties of dendrites and local inhibitory systems enable networks to dynamically restrict and route information in a context-specific manner. Our key contributions are as follows. First, we propose a novel artificial neural network architecture that incorporates active dendrites and sparse representations into the standard deep learning framework. Next, we study the performance of this architecture on two separate benchmarks requiring task-based adaptation: Meta-World, a multi-task reinforcement learning environment where a robotic agent must learn to solve a variety of manipulation tasks simultaneously; and a continual learning benchmark in which the model's prediction task changes throughout training. Analysis on both benchmarks demonstrates the emergence of overlapping but distinct and sparse subnetworks, allowing the system to fluidly learn multiple tasks with minimal forgetting. Our neural implementation marks the first time a single architecture has achieved competitive results on both multi-task and continual learning settings. Our research sheds light on how biological properties of neurons can inform deep learning systems to address dynamic scenarios that are typically impossible for traditional ANNs to solve.

Authors: Abhiram Iyer, Karan Grewal, Akash Velu, Lucas Oliveira Souza, Jeremy Forest, Subutai Ahmad

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, this is a comprehensive paper review on a paper called Avoiding Catastrophe, Active Dendrites Enable Multitask Learning in Dynamic Environments. This is a very cool paper because it combines ideas that come from biology, which are active dendrites and ideas that come from deep learning, namely the problems that we face in multitask learning and in continuous learning. Catastrophic forgetting is one of the main problems of these areas and the method of active dendrites directly inspired by biology can really help with that. So this video is a comprehensive review on the method of active dendrites in deep learning as the paper describes it. By the end of the video, you'll have a good understanding of what is in the paper. In the next video that I'll publish tomorrow, there will be an interview with the authors, which was also super interesting. And I definitely invite you to check out both. As always, if you have any comments, please leave them in the comments on YouTube. Leave a like if you do like the video and I'll see you around. Bye bye. Hello there. Today we're going to look at Avoiding Catastrophe, Active Dendrites Enable Multitask Learning in Dynamic Environments. This is by researchers of Nementa, Cornell and Stanford. So this paper proposes to bring some of what has been lost in translation from real biological neurons to deep learning neurons to bring some of that back into the deep learning neurons, specifically the concept of what they call active dendrites and also a bit of sparsity that is to be found in biological neurons. So they bring these back into deep learning neural networks. And it turns out that that is pretty useful to combat something known as catastrophic forgetting, thus the title of the paper, Avoiding Catastrophe. So catastrophic forgetting is a phenomenon where in multitask learning or continual learning, a network has to learn many things at once. And then these things interfere with one another. And it turns out that our methods of training neural networks using backpropagation aren't really good at that. So either they don't learn any of the tasks because they conflict with each other, or in continual learning, they do this catastrophic forgetting where as soon as a new task comes in, they've completely forget about the old task. So many solutions obviously have been proposed. And this right here isn't like is not entirely ultra novel, but it is interesting. It ties together biology and sort of practical applied deep learning. And it does have some connections to, for example, modern transformer architectures and so on. So I'd also be interested to hear what you think how this stuff is all connected. So they start out saying that the artificial neural networks, they call these ANNs. So whenever you do in this paper, ANNs means sort of the deep learning neural networks, we have to be a bit careful when we talk about things that involve biology, because neural networks is an ambiguous term there, like the neural networks is an ambiguous term because it appears in both domains. So they they claim they fail dramatically when learning multiple tasks, a phenomenon known as catastrophic forgetting. And I already said catastrophic forgetting, it essentially means that you can't learn many things at once. So it says learning multiple sequential tasks can lead to significant interference between tasks. They look at two different they look at two different tasks right here. One is multi task reinforcement learning. And the other one is continual learning. So in multi task reinforcement learning, it's essentially reinforcement learning with multiple tasks. So you're some sort of an agent, and you're in some sort of environment, and you have this basic loop of sending an action and getting back some kind of observation and reward. However, however, there are multi there are many tasks in this environment. So maybe you see it and maybe you don't. But as part of the definition of the problem, I think in this particular environment, you also get back kind of an indicator of which let's call that T the task indicator. So which task you currently supposed to fulfill. So the same environment has many tasks. And then obviously, your reward is going to be dependent on which task is currently active. So you're going to give the agent a mixture. So every new episode, the agent tackles the task is different, and therefore, if the agent just does the same thing as in the last episode, it might get a completely different reward because the task is different, right. So that is multi task reinforcement learning. And it turns out that and this papers have established this before and I think we have even made a video on some of them that if you look at the gradients, they often conflict with one another. So learning one task would pull a weight in some direction and learning another task would pull it sort of in a different direction. And there are papers that try to make these gradients as like orthogonal as possible or project them somehow into a task specific subspace. But as it stands, conflicting gradients can arise in these multi task settings. And therefore, the classic way of training neural networks with back propagation to update all the weights at the same time, just isn't very conducive. Even worse in continual learning. So here, we're not necessarily in reinforcement learning anymore, although we could be. So this is this is simply continual learning, where you present a neural network. So you have a neural network, the neural network is able to, you know, take whatever picture, let's say it's a picture classification and give you some sort of a class label for that picture. And now you have different tasks. So you have task one, task one might be classify, you know, classify cats from dogs, then task two might be classify, I don't know, cows from beavers, task, and so on. So there is also a bit of a specification gap. Some of these continual learning benchmarks, they will always have the same classes, but different data sets, some will have different classes, some will have new classes, and so on. In this particular case, we're looking at permuted MNIST, which is sort of the MNIST data set. So you know, there is whatever picture, and there is some sort of handwritten digit in here. And the the permuted MNIST data set is simply that every task that you consider, so task one would have a permutation applied to all the pixels in in this picture, but always the same permutation. And then task two would apply sort of a different permutation, permutation one, permutation two. So it's kind of a different task. It's the same classes, you're still classifying digits into zero to nine, but the permutation is different. Therefore, it's like you have to learn a new task if you don't have some sort of built in symmetry prior in your neural network. Obviously this, we're not going to use conv nets right here, because conv nets would make no sense if your pixels are permuted. We're simply going to use feed forward networks. The goal isn't to get state of the art. The goal is to show the difference between what if we use regular neural networks, and you can imagine right here, if I train on task one right here, and task one has some kind of a permutation in the pixels, I'm able, you know, these neural networks, they're able to learn that because if they're feed forward networks, they don't care about neighborhood anyway. So they they are able to, you know, we train we train these weights right here to to completion. And then I activate task two, right? Right after task one, I stop giving the network data from task one, and I start giving in data from task two. So also different permutation, I also label my images, give it to tasks two. Now I'm going to train these weights, I continue training these weights. And there is some effect when we talk about large language model pre training in that whatever you pre train on that kind of stays around. So any fine tuning in large language models isn't going to completely erase the pre training. So it actually matters what you pre train. Although this is not the same right here. First of all, we're dealing with way smaller networks. And these way smaller networks, they're able to be kind of overwritten mostly. And also we're dealing with classification tasks right here, and not some sort of language modeling task. So yeah, these these weights, they will just be overwritten to the point where task one is forgotten. It's nowhere. So we've again, if we draw up some sort of a weight, task one would pull it in this direction, that would be the gradient. So the weight would slowly update by update going this direction. And then all of a sudden, we activate tasks to which will pull it in this direction. So the weight would then travel into this direction, and essentially forget about task one. So it is nowhere near where it should be for task one. As I said, there are some methods of solving this with orthogonal projections and so on. But as a basic rule, our deep networks aren't very good at that. So what do we do about it? This paper's idea is that since our deep networks use a model of the neuron that looks very much like the thing on the left, so you have your your input weights, which are commonly known as the weight matrix or the weights of the layer. This is just one row or column, I guess. Well, it depends on how you specify the layer. But these are just all the input weights going into one neuron, they're summed up. So this is the matrix multiplication. And then there is some sort of a nonlinearity right here, which could be a sigmoid, which could be a tan h, which could be a ReLU. And that's essentially still the model that we have. This is like an over like it's decades old, this this model. And it served us pretty well, but it has forgotten some very important aspect of biology. Here on the right, you see a pyramidal neuron, a pyramidal, a pyramidal, I'm just going to call it pyramidal because pyramid. So this is obviously way different. So well, first of all, it's not a schematic, it's kind of like an actual drawing, you see the axon right here. And the axon splits up into different parts, which is, you know, is like our regular neurons, they connect to all the neurons in the next layer. Although one difference is you can already see that there are way less connections from here than you would have in a fully connected layer. So there is a degree of sparsity in biological neural networks that is not represented in the deep neural networks that we build. And then the inputs right here, we just consider all the inputs to be the same. However, there is a difference between what they call proximal inputs and distal inputs. So proximal inputs would be inputs that are very close to the cell's body. And those behave very much like the linear influence that we see in our model. However, there are also these distal, by the way, these things are called dendrites. There's a difference between the axon, which is this thing here, and the dendrites, which is this thing here. Every neuron has one axon, but can have many, many dendrites. And dendrites are sort of like, they're just kind of elongations of the cell body. So any other axon could dock either directly on the cell body or close to it, or could dock on any of the dendrites. So you can make connections from axon to body or from axon to dendrites. And dendrites are kind of like harbors, like ports or docks for incoming traffic. Yeah, that's how I can explain it. However, these distal dendrites, they're not acting like as much as linear things. What they are doing is, and this paper describes that, is they act like their own little subunit that computes its own function. So it's almost like a mini neuron inside a neuron. And that mini neuron can then influence or modulate the cell body. So whenever that mini neuron is, for example, very high, is very activated, it will raise or lower the activation threshold for the main cell body. So it can sort of influence the main cell body in a multiplicative way. And that's exactly what we're going to see in this architecture. So yeah, I've sort of skipped a lot of the text right here. Um, yeah, if you're a Patreon, you get these notes, I hope they help. I've never considered my scribbles to be super duper helpful, but I've started pre annotating and I hope it helps someone. But yeah, these are mostly for me to see what I have to look at. So what does that have to do with continual learning? Well, they describe right here, they hypothesize that biological properties of pyramidal neurons in the neocortex can enable targeted context specific representations that avoid interference. So pyramidal neurons, which comprise most cells in the neocortex are significantly more sophisticated, demonstrate a wide range of complex nonlinear dendrite specific integrative properties. And they are hypothesizing that this modulation property that we've just discussed, this modulation property could battle this catastrophic forgetting. Specifically, what they say is that, well, we have many of these dendritic distal sub modules, and these could learn and there are some biological evidence for that to recognize different contexts in which you are in. And depending on which of these is active, that means which context is recognized, it can modulate the body of the cell. So the cell could react differently depending on the context. And that is one of the ingredients exactly that we need to avoid this catastrophic forgetting or do multiple tasks at the same time is to say, hey, I'm only going to activate my cell body if I'm in the correct context, meaning for example, a particular task is active. So the cell body can learn its weights to specialize on a given task and rely on the sub units to recognize when it needs to fire. And obviously, if there's some structure to the tasks, we can also think of these being sub tasks. So sub tasks are sort of being activated that can then generalize and be integrated into multiple tasks and so on. So there's a bit of related work. The active dendrites that is pretty much pretty much what I just described. You can see each distal dendritic segment acts as a separate active sub unit performing its own local computation. When input to an active dendritic segment reaches a threshold, the segment initiates a dendritic spike. So this is not a neural like axon spike. It's a dendritic spike that travels to the cell body. Okay, I've apparently memorized this passage. It can depolarize the neuron for an extended period of time, sometimes as long as half a second. They don't model time dependency right here, by the way. That's something they don't integrate right here. During this time, yeah, the cell is significantly closer to its firing threshold and any new input is more likely to make the cell fire. This suggests that active dendrites have a modulatory, long lasting impact on the cell's response with very different role than proximal or feed forward inputs. So they say they typically receive contextual input that is a different input than received in proximal segments. Proximal are the near ones. These context signals can arrive from other neurons in the same layer, neurons in other layers or from the top down feedback. Another thing they don't model right here is any sort of top down feedback or same layer or anything like this. I'm just taking this away. What they do model is these dendritic subunits. The second thing they're very interested in is sparsity. So sparse representations are ubiquitous in biological neural networks, not so much in deep neural networks. They claim that studies show that relatively few neurons spike in response to a sensory stimulus across multiple sensory modalities. Sparsity is also present in the connectivity. And they claim that one advantage of sparsity in representations is that vectors for two separate entities have low overlap. So they're now talking about deep networks because biological networks don't have vectors. So they're talking about how if you impose sparsity in a deep neural network, and you are in high dimensions, then your representations likely will not collide because a lot of the entries are zero. Low representation overlap among unrelated inputs may be particularly useful when an artificial neural network is learning multiple unrelated tasks. And that's why they are interested in the sparse representations. Because if different things aren't likely to overlap, they're not likely to interfere with each other. And therefore they might be useful to combat catastrophic forgetting. So two things. We're going to implement these active dendrites into our models, and also we're going to implement a degree of sparsity. And we're going to observe how these two things work together to combat the catastrophic forgetting phenomenon. That is essentially what this paper suggests. So let's look at exactly how they do this. I think it's best to jump to the model right here. So this is one of the models or one of the architectures they use. This is the actual arch, they use two layer neural networks. So yeah, this is these are these are not these are not huge networks that they use right here. It is for reinforcement learning. So it is kind of a soft actor critic, they use this benchmark right here, where a robotic arm needs to perform multiple tasks in the same world. And in this particular task, the agent always gets the information which task is active. So which task is active goes into this context vector on the left, this is a one hot vector that is fed as a context signal. What's special about this network is that first of all, you can see that there is a linear layer and that is not some classic linear layer that is a special linear layer, namely the active dendrite linear layer. So the active dendrite linear layer has a feed forward signal. And that feed forward signal is treated just as a classic deep neural network feed forward signal. So that would be the feed forward signal would essentially be whatever the input here is, in this case, probably the robots state or something, and its position and it's maybe the position of the whatever object it needs to grab, if that's not always at the same place and so on. So that's the state input. And if it if we're only one task, the network could just learn from this input. However, this is multiple tasks, so it gets the context vector, the alternative, the baseline what the baseline will do is it would append the context vector right here, and just sort of extend this feed forward layer. And it would say, well, the network essentially has access to this information right here in its input. So it should technically be able to handle that. However, they're going to show that, you know, they're going to implement this in a baseline going to show that that's not as helpful as what they're doing. So we have a feed forward signal. And that computes some output, you can see that's independent of this context vector. So the feed forward layer, the weights of the feed forward layer, which sit approximately here, they're going to be, you know, multiplied by the weight matrix summed up. And then there's some output signal right here, just in a classic feed forward layer, the context vector comes in here. And what it's what it's going to do, remember, this is a one hot vector. For now, they make it more complicated later, it is going to be matched with each of what these things are, these things are called dendritic segments. So it is going to be matched with each of them, and the matching is simply done via an inner product. That's what this little sum symbol does right here. So there's an inner product between the context vector and the dendritic segment. And then they're going to select whatever dendritic segment matched the highest and that is going into here. And then here is a modulation function. So the signal that is the highest, the highest inner product with whatever dendritic segment is going out here and modulates that signal, and that's going to be the output. Now let's look at how these dendritic segments work, because that's really sort of the meat right here. Here you can see the forward signal, the forward signal is your classic signal right here. There's a weight matrix or vector in this case, there's the input, there's a bias. The dendritic segments are, they're just vectors. These are trained, okay, every single one of these dendritic segments is a set of weights that is trained and it's different as far as I can understand each neuron has its own dendritic segments and for each dendritic segments, it has its own weights. So there's no weight sharing going on among the dendritic segments, which would, I think, break the whole thing, although I guess one could come up with some sort of smart like meta weight sharing right here. But the idea is that, as you can see from the formula, we're simply going to take the context vector, calculate the inner product with all of these dendritic segments, take the max dendritic segment, that's going to be some kind of a number, right? This is an inner product. So this is the strength of whichever dendritic segment matched the most. And then we're going to take a non-linearity, in this case, a sigmoid function, and we're going to multiply the at the feet forward signal that we have with this sigmoid function of this inner product. So this can, you know, the sigmoid is between zero and one, I think. Yeah, I think they retain the sign, so they take the max absolute value in the end. But let's leave that out for now. So whichever segment matches the most, that's some number that goes through a sigmoid. So let's think about this. When is this thing one? It's one whenever one of these dendritic segments activated, right? So we take since we take the max, one of them needs to activate, and then this thing is one. So these dendritic segments, they are sort of like, like receptors for contexts that where this neuron could be relevant. So they are sort of like, you know, feature detectors. And if they they expose some kind of some kind of vector, they are obviously vectors. So in the space, there's like here, like, you know, I have maybe I have three of these dendritic segments, and I say, well, I'm interested if if my representation, if my context representation is any of those three in that direction, then I'm interested. So if the context comes in like this, they're just like, no, no one is interested. Therefore, the sigmoided maximum is going to be zero. And it's going to block the signal right here. However, if the context comes in is very close to what one of these segments is, then it's like, oh, wow, this actually might be relevant for this neuron. Therefore, the sigmoid. So the inner product is high, the sigmoid of the inner product is high, and the signal is going to be propagated through. Interestingly, in the experiments, they always expose like as many dendritic segments per neuron as they have tasks, which I thought to criticize that because I was like, well, that's kind of cheating. But now I don't even know if that is necessarily like, wouldn't one dendritic segment suffice? Like if it could perfectly recognize if every neuron was only relevant for one task, and if that could be perfectly recognized by the context vector, I guess that would work. But this is more powerful, right? You can present a number of situations where you would be interested in. Ah, I guess, okay, if you have as many dendritic segments as you have tasks, then every neuron could be relevant for every task. So a neuron could be relevant for all tasks or for just two of the tasks and so on. So yeah, I still maintain it's a bit of cheating to make as many dendritic segments as you have tasks, because that's implicitly telling the network how many tasks you have. But you do get the task as the context. So you already know anyway, right? In any case, that's what this network does. It exposes these things, it's able to take this context signal and modulate that signal. The second thing it does is this k winner takes all. And this is very much like maybe the sort of sparse mixture of experts that you might know from transformers or the concept. So what it does is it simply calculates a maximum maximum activation over the entire layer and it only lets through the highest the highest k many things. So it's k winner takes all k could be three or five or something like this. But in any case, it is not as many as you have neurons. And all the other neurons, they're just set to zero. Therefore, they also don't receive any gradient. So here you can see how these two things play together. First of all, we're going to modulate so we're going to block a lot of the signals right here. Blocking means we're just going to multiply them by a very small number if they're not relevant. And then it's not just that they're very small. Actually, we're just going to pick like the top five. So all the numbers that are small, we're just going to eliminate completely. I don't know if this you know, this method of achieving sparsity is necessarily the best one to pick the K best, or if it'd be better to just threshold somewhere. Because K, then is some sort of other hyper parameter that you might, you know, set via cheating, or that you might have to try out and some some sort of a threshold might be more robust, especially since the sigmoid is fairly, fairly steep function. Yeah, that's, that's the architecture, essentially. So I hope you can see how this sort of connects to to other things. Especially, I'm interested in this modulation property. And I'm also interested in in the sparsity approach. Obviously, if you have sparse representations, there's not going to be any gradient flowing back through the neurons that weren't activated. And therefore, there's not going to be any gradient into these neurons. That means these weights here aren't trained for that particular neuron. It means these dendritic segments, which are, again, these are parameters trainable parameters. So these blue arrows are back propagate trainable, they will only update if the neuron has actually been selected in its forward pass. So they're random at the beginning, and then with time, they will fine tune for specific contexts. So they will sort of move. And yeah, there is a bit of a danger that some of these are just become ghost parameters. But I guess as stuff moves around, and as initializations are diverse and random enough, almost everything will will become sort of selected at some point, if your inputs are diverse enough. Yeah, so that's that. I've skipped a lot of these a lot of the text right here. You can see the K, the K WTA, the K winner takes all representation, we're simply going to let the signal through. If it's in the top K activations, and it's zero, otherwise. Yeah. Exactly. So here they say only the neurons that were selected by the WTA function will have non zero activations and thus non zero gradients, only the weights corresponding to those neurons will be updated. And that's how the two things work together to battle catastrophic forgetting in that, if the context, if the dendritic segments successfully learn to recognize different tasks, that means that only the neurons that are involved in a particular tasks will will be updated by that task. And therefore, the network will not will not forget the other tasks or not forget them as easily. Because the sparsity also the sparsity kind of forces not all parameters to be updated. And the dendritic segments forces these sparse updates to be in a very structured, very consistent fashion. And yeah, they also say that only the dendritic segment J that was chosen by the max operator is updated, all other segments remain untouched. So even if a neuron is part of this K top K activations, only one dendritic segment is updated, namely the one that matched the most with the context. And this again ensures that maybe if a neuron is relevant to different tasks, the other dendritic segments they can they can keep their place. Even if we train in a new task where this neuron is also relevant, if it was relevant to an old task that might be stored in a different dendritic segment than the one that is activated right now. And that dendritic segment due to the max operator will not receive a gradient and will just remain as it is. Of course, this doesn't scale, you know, forever. And to all degrees of noise, and there is a there is a way in which tasks can be too related. So I would guess that in a model like this, if tasks are very related, they will activate the same dendritic segments and therefore override each other. But then also if tasks are very related, you would expect that there is some form of generalization or crossover among them. But the difficulty has never been that much with generalization. It has always been with the fact that if you think of, for example, large language models, I also think of large language models as continual training, they often they don't even run in a single epoch over some of the data, and they still learn from it. So they see a data point once right and, and then, you know, that's that's that and they still are able to incorporate that somehow. So how are they not subject to catastrophic forgetting, they also in a way implement different tasks because I can query GPT-3 with so much stuff, like it can do so much different diverse things. It is all it is like a bit of, you know, sure, it's always the same loss and the gradients don't necessarily conflict of that loss. It's kind of a multitask learning. And one key difference is that GPT-3 is presented with sort of an IID shuffled sample of the training data. However, here, the all the data of task one comes first, and then all the data of tasks two comes later. So even if there's some generalization aspect, I would expect if tasks are close together, task two will override task one, because the same dendritic segments might activate. And just from the model here, they don't have a way to, I feel they don't have a way to battle that maybe they are there of a different opinion, but maybe some sort of how should I say this, some sort of a contrastive method, like a contrastive addition to these dendritic segments, like pushing them apart from each other for different tasks, you know, if they have the task information or just plain pushing them apart from each other, maybe hallucinating pseudo tasks for that, maybe a way to automatically adjust to how close together or far apart the different tasks are. Yeah, that's just my, what I would guess might help. But maybe I'm completely wrong. Tell me what you think. They say we hypothesize that a functional specialization will emerge where different dendritic segments will each learn to identify specific context vectors. So that's the model. Now they go into the experiments. As we already said, they do two things, multitask reinforcement learning. This is this robot thing. So it's all at the same time. In this particular case, it's not one after another. It's all at the same time. I think each batch is always from the same task, but like the next batch will be of a different task, I think. Yeah, but it's different tasks, right? So the same actions don't lead to the same reward. And that is means conflicting gradients. They use a very basic RL algorithm right here, which is not necessarily important for our discussion, just to say that the networks are quite small, right? They have two hidden layers, each with 2800 neurons, which, okay, that's that's sizable. So they're, they're quite, they're quite fat hidden layers, but it's just two of them. And then each one is followed by a K winner takes all activation function. And then there's a final output layer. They say the first layer has standard neurons, whereas the second layer hidden, the second hidden layer contains active dendrite neurons, which are modulated by the context vector. In this case, the context vector just encodes the task ID as a one hot vector. And yeah, each active dendrite neuron in our network has exactly 10 dendritic segments, the same as the number of tasks to learn, they do ablations where they increase that number of dendritic segments. But yeah, I do think they're giving their model the absolute best chance to learn right here, by setting some some of these parameters with essentially, okay, it's not hidden information in this particular case, but it is in the next case where we're not getting the task ID, as you will see. So this is how the model looks. There's the state vector, there's feed forward, we have some sparsity enforced by these, notice that it's really interesting that sparsity is even enforced here without any without any modulation. And they do also some ablations on that. But I'd be interested why they didn't choose to also have dendritic segments in the first layer. It seems quite odd, honestly, to set up an experiment like this. Yeah. And the other thing is, they say, although we control the hidden sizes to yield approximately the same number of total nonzero parameters, we note that MLP baseline contains nearly 500k more nonzero parameters than our active dendrite networks. They speak a lot of these nonzero parameters, and they count the network sizes in nonzero parameters. So I would be interested what's the difference between parameters and nonzero parameters and what it was is a nonzero. I've not seen this exactly explained in the paper. Is that like at the end of training, if a parameter is zero, you don't count it? Or is it somehow different? I don't know. But safe to say they do try to make the networks as you know, with the same number of parameters, which means that if they have these dendritic segments, which are quite a number of parameters, they have to, I mean, not that many compared, but they have to turn down the the other parameters. So here, you can see the results at the beginning, the active dendrites network in blue is sort of underperforming, but then it overtakes the baseline, the MLP baseline. And yeah, the errors here, the variances are quite large, as you can see. They do run another analysis where they just select the top five for each. And you can see that it separates a bit more cleanly, although I'm not sure if that is like, is that is that a thing? Like, can you say I'm just going to select like the top five of each to reduce the variance? I'm not sure if the the the max distribution is the same as the mean distribution. Like could I do that in practice? Maybe not if I just have one run, which is essentially what I'd want to do in practice. I couldn't necessarily do that. I don't know. In any case, they beat the MLP baseline in both cases, you can see that sometimes there are pretty significant differences, especially in what they claim are the harder tasks like the pick place tasks. And these are also the tasks that have very little overlap with the other tasks. So you would expect greater interference. And that's where they have a lot of gains in gains against the the baselines. In continual learning, they use this permuted MNIST as we've discussed. And so yeah, here's here's sort of the comparison. Yeah, you can see also you can see here the variants are huge for some of these tasks. Yeah, in the permuted MNIST data set, they okay, they don't have a graph, I believe. But in the permuted MNIST data set, they also are beating or are advancing against the baseline significantly. So we have somewhere, there are the results. So you can see right here, there isn't a baseline in this particular diagram. But you can see that the drop off is not very steep. And usually if you do this with regular MLPs, they just fail, like they they fail, which means that so this test accuracy is on all the tasks you've seen so far. So you get presented with whatever 20 tasks in sequence, and you evaluate on all of them. With regular MLPs, they just suck at this, like they forget the previous tasks. And yeah, that's that's that. So the fact that these networks are able to sort of hold up across and here you can see up to like 100 tasks is already pretty remarkable. They have two different variants. One where the prototype is given while training, which essentially means they have information about which tasks they're in. And one is where the prototype is inferred. And they describe these up here. So what they do, they now switch over from not providing the task ID as a context signal because that's kind of cheating. And they provide now these this prototype. So what is a prototype? A prototype is essentially a data point or it can be a latent vector. But here I think it's just a data point that is kind of the mean data point. So this would be the prototype of task A, the mean data point of all the data points in a particular task. So they provide that as the context as the context signal. Now what they can do now is here you can see how that works. It's just a mean. Well, I told you what they can do is if they don't have a task annotation, if they don't know what task goes with a particular data point, they can simply collect data points during training. They can say, well, here's a data point. Here is one. Here is one. Right. And it helps that they have the guarantee that each batch has the same task. And then they say, well, okay, we're going to make a prototype right here. And that's going to be our context vector. And then the next batch comes in and it's kind of like over here and they say, well, this is not very close. So we're going to make a new prototype right here. And then the next batch comes in and it's like here and they say, ah, that's probably of the same thing again. So we're going to use that prototype to provide to the system. So it's kind of this heuristic thing, averaging the data points, which I find to be quite weak, like averaging the pure data points is like, it might work in permuted MNIST, but there's definitely room for improvement right there, because that is not going to be informative at all in in many or most tasks. And obviously, there's also like a hyperparameter to set, like, you know, what's the appropriate distance measure right here? And also, this just going into this as the context signal. And the context signal is essentially just worked out by inner product as we saw up, sorry, up here. So the signal is just it's just an inner product with some of these U vectors. If this gets any more complicated, there's going to need to be a lot of machinery in front of the context vector, like, I would expect we need to pass it at least through some hidden layers to compute something of value. But for permuted MNIST, it's going to be enough, right? So they recognize which tasks they're in. Now, I am interested why exactly they switched from providing the task ID, like, at least in first in a first instance, why they switched over to providing these prototypes right here as the context signal, right, just experimentally, they have this one experiment in this one setting, where they they just provide the task ID, and then they have the other setting where they do something different. I would I would get it if they did both things in the same setting. But having two different settings and just doing two different things is a bit suspicious, I guess. And also here, you can see they provided actually to both layers, and not just to one layer. I would like to know the story behind this. They also compare to a baseline, which is called SI. So SI, as they describe here, it is a thing that operates solely at the level of synapses, it maintains an additional parameter per weight that controls the speed of weights adapting to specific tasks. The two approaches are complementary. That's why they can be combined. You can see on the right, so on the left hand side, you can see what happens if you infer these prototypes during training. And you can see it's just a little bit worse, which I think is like 100%. So I don't know how much better or worse they would be if they actually gave the task ID. But I think this distance right here, that is only going to be possible on permuted MNIST. Maybe I'm wrong. Maybe I'm wrong. So here you can see, interestingly, right, here's the active DEND, right? It it this is kind of the curve from the left. And then these SI method just by itself actually beats the active DEND, right? However, you can combine both as you can see, and both together are stronger and give you an even better, better boost. So that is, I mean, it's, it's, it's, it's good if you can combine all the tricks that you had so far. I would have liked to have here like a like, okay, the MLPs, they just suck. Because right now, it's not exactly clear how much they suck. Although I'm sure that there's some appendix table, and I haven't looked, I haven't found it. The paper is quite long. So here they compare to a different method, which is called xDG, which is context dependent gating, sorry, they say this is the implementation closest to theirs. This is another idea. However, that one uses hard coded distinct sub network for each task. So this is pre allocated, it pre allocate says you sub network, you're for task one, you're for task two, you're for task three, they engineer this in a way where they expect some overlap between the tasks and some separate neurons. And then they only train the sub network. So they need the task ID to be provided. The implementation of tasks specific subset of the hidden layer, other neurons are forced to have an activation value of zero. This requires a task ID that determines exactly which neurons to turn on or off. It turns out so the way they emphasize all of this is that it turns out that they do beat the baseline as you can see right here. When you just do them by themselves, but as soon as you combine them with this SI technique, the xDG outperforms the active tendrites. So obviously they need to highlight the differences right here, which is a good tactic, right? And it's valid, they do do more. So here they say task information is inferred, it's not provided via this prototyping, where this provides a system with a task ID during training and testing. And it's important to see that even if they do the prototyping with the information of the task ID, they claim that during inference time, there is no task ID provided. And they simply, you know, they see whatever if a data point is whatever prototype the data point is closest to, that's the prototype they take. The second thing, sub networks automatically emerge via the use of dendritic segments in their model, whereas the baseline, it pre allocates different sub networks for each task. And that's that's legitimate. However, I don't I can't shake the feeling that they've like evaluated it. And then this thing was better. And they were like, ah, rats. Now what can we what can we do? Okay, we can't beat it. How can we make it? How can we make it different enough? And maybe that's when they decided, okay, let's try to like not provide the task ID. But let's try to come up with like, a dynamic way of figuring out the task or something like this. And that's the story behind why this prototyping exists, or maybe that that has like, that just turned out like it is, I don't know. But you know, it's it's interesting. It's interesting to see sort of there might there might be a research process behind this. And which is cool, because the research process sort of leads to more innovation, which is neat. And important question one that which I also had during reading of this paper. And no, that's not it. This is we're going to get to that. First, they check their hypotheses. So they say the hypotheses of our work are twofold. First, active dendrite networks modulate an individual neurons activations for each task. Second, the winner takes all activations use this modulation to activate sub networks that correspond to each task. They provide some evidence for this. So here, on the left and the right, you see the two tasks they tackle. And they give you an impression of which hidden units are active for which particular task. And they you can see that it's fairly sparse. So if you look at any given column or at any given row, then not many light up in dark green, which means that not many things are activated per tasks and a given unit is kind of specialized to particular tasks or a particular set of tasks. Now, without a comparison to a sort of regular neural network, or without a comparison to one of the two features of the network ablated, it's kind of hard to to see whether this is a lot or not a lot, especially on the on the right, you can also see like is this sparse, or is this not sparse? I don't know. I'm going to guess it is. Yeah, so I don't know, I'm going to believe them that this is especially sparse. And I think they also measured it at some point, actually the sparsity, but just the graphic alone isn't this isn't necessarily enough for me. They look at single neurons. So in the single neuron, they wonder which dendritic segment is responding to which task, right, there's a neuron A and neuron B. And you can see at initialization, a lot of the segments are responding to a lot of the tasks. However, after learning, it becomes much more quiet, and only very few segments are responding to to any or each of the tasks. However, also here, first of all, it's not, it's not super clear what we are to compare this with, because this could just be this could just be a phenomenon of kind of like the scale of stuff being wrong. Like at initialization, just the scaling of things being kind of out of out of whack, because you can see right here, there are entire regions that are just kind of dimming down, right? So yeah, obviously, a given a given neuron isn't going to respond to all the tasks, right with all the segments, it's not going to be involved in all of the tasks that would actually, you know, this this is a valid prediction of their hypotheses. And you can also see that especially neuron B here, if you look at segment eight, multiple dendritic segments are reacting to signal eight, which might be an indication that there is some, you know, they have learned to recognize different features that all indicate that for no segment eight response to multiple tasks. Ah, okay, that's, that's different. Okay, negate my argument. Forget what I said. I thought I thought it was a smart recognition. But you know, it's it is it is definitely evidence for the fact that there's specialization going on, but without a comparison to anything, it's hard to tell if that is that or just some sort of a scaling, scaling issue that just after training things are scaled differently. But just, you know, from from all the other evidence, they make a convincing case that there is this sparsity and specialization going on. So here is the last thing I want to discuss. And this is a question that I had when reading this paper, which is, aren't like, isn't this isn't there an equivalence to larger networks? Like aren't you just sort of sort of, you know, designing this this network in this special way? And can't I achieve the same thing with sort of a regular neural network if I just make it a bit larger? They say multiple studies have suggested that that dendritic computations performed by pyramidal neurons can be approximated by artificial neural networks that have one or more hidden layers from a computational and deep learning perspective. This is equivalent to claiming that ANNs with dendrites can be substituted by larger ANNs without dendrites, supposedly. And I have tried so they are going to make the case right here that that is not the case that they are outperforming, for example, three layer MLPs, which are about the same size and MLPs that are much larger, so much deeper. So they're going to outperform them at you can see right here number of tasks 100. Oh, this is this is probably the graph I was looking for before, no? Yeah. So here you can see how much how much the the MLPs suck. So yeah, they show that even if you scale them up, in fact, the 10 layer MLP is even worse, which is interesting, which might be might be interesting in itself. Like, why is it? Why is it worse? And is there like a crossover point here? But in any case, these MLPs, they get the context vector as an input, right? So technically, technically, they have all the information to do the same thing. However, the paper argues that it's the training procedure, back propagation, updating all the weights for the given data that is presented to us. This is particular to an ID setting of data, which we don't have right here. So no matter how big you make your neural network, supposedly, if they are correct, it would always result in the same problems due to the way that you train them. On the left, you see an ablation of the two ingredients. So the active dendrites only, the sparse representations only, and the combination. One second. So they do certainly give empirical evidence. And by the way, here is also an ablation on having more dendritic segments. On the top, they're trying to learn 10 tasks. On the bottom, they're trying to learn 150 tasks. And it's interesting to see that the gains here are kind of negligible, although maybe that's just a property that they're very close to 100% already. And here you can kind of see gains until 50. And then, well, okay, I might be imagining things that there's stronger gains here than here after you pass sort of the number of tasks barrier. But safe to say that, you know, more dendritic segments might also be useful. And maybe my skepticism of them setting parameters exactly, exactly as many as sort of exactly to the number of tasks they have is not super warranted. Also interesting is the fixed number of dendritic segments and varying activation density level. So here is this k, so how many things they let through each layer, you can see increases to the right. So you activate 100%, which would regress to a classic MLP. See if you activate 100%, it's really bad. And there are two things right here. Again, they're trying to learn 10 tasks or 50 tasks. Interestingly, interestingly, if at the beginning, obviously, you let nothing through, it kind of sucks, then you let some things through, it's already really good. And then it gets better. So there's some kind of an optimum around 10% ish or so. Interestingly, that's the case for both the things, even though one is trying to learn significantly more tasks, which is interesting, right? Then there is a drop off for both things, which you would expect. But then there is kind of like a flat flattening, followed by another drop off. And it's also interesting to to think about why that's the case. So here it might be that this is the situation where very few things are overlapping. And therefore the network is able to use specialized sub networks for all the things that it needs to do. And in this entire region up until here, it might be the case, you see it kind of drops off at the end after like 80%. It might be the case that most of the things are shared. However, the network can kind of encode stuff in the non shared part. And that can itself within the network kind of modulate whatever the shared stuff is doing. It's kind of like a shared feature extractor, followed by some modulation of the non shared parts. I would Yeah, it's interesting to think and then that crashes together once there is no more non shared parts. And there's no way of doing anything different in the different task settings. I was thinking myself, you know, getting back, sorry, getting back to can I just achieve the same thing with a larger network, I was thinking myself of how to do that. So they claim, No, you cannot. And I guess it's true. Let's think of okay, let's leave the sparsity away. Let's just think of this dendritic activation, right? I have my x that's multiplied by by W. And let's also leave the biases away. So I have my x vector down here, I have some W, which is a weight matrix. So everything's connected to everything. To till here. Now can I also and I have my context vector, can I somehow build a feed forward network that would also you know, have the appropriate weight connections that I could build myself the function W x times sigmoid, you see, let's also leave away the max right right here, I guess we can't. That's an integral part. And yeah, it's not clear to me how that would work necessarily with with a single layer. And it's also not entirely clear to me how that would work with multiple layers, like, you would have to build some very, like various contraptions of additions. Maybe you know, once you get a relu out on all of that, it might be more possible. But it's not easy to get this multiplicative interactions between signals working in a feed forward network. However, however, in transformers, that might be different, right? So you know, this here, this, you know, we can do this in transformers, I guess in feed forward networks, too. And then the max, we have we have softmaxes in transformers, right? So what we could do is we could have these things here as, let's call them queries, right? And these things here are the keys. And we apply the softmax in a transformer. And the values might just be a constant vector of ones. So the values might just be constant vector of ones, which would mean that if we multiply the softmax by this thing, we would simply select sort of the maximum out of that, and that's going to be one and everything else might be zero. Maybe I might. Maybe I'm I have this wrong, but maybe not. Yeah, I guess that that would work, right? So and then in the next layer, so that could be our output signal for layer one. And that could be our output signal for layer one in a different attention head. And then the multiplicative interaction again, we can get by via attention because attention constructs the attention constructs the weights dynamically by multiplication. So we could take this as as keys and maybe also queries. And then simply this could be the values right here. And then we multiply them together. And that's going to be a multiplicative interaction between that signal over here and the signal over here. So I guess transformers could model something like this. It's not easy. It's not going to be in one layer. It's not going to be non shared potentially right as it is here. So here nothing is shared of the parameters. But I would I would argue that the more powerful method of the transformer doing these dynamic weights, you know, there might actually be some connection here. And as we said, for the sparsity, we have sort of the sparse mixture of experts, which is kind of sort of a little bit similar. So looking through the rest of the paper, I don't I don't think I have anything annotated right here. There are hyper parameters. There are tables and more results and methods. But that's essentially it what I had to say about this paper. I like this paper because it sort of connects, connects biological concepts, it tries to reintroduce them, it augments the fundamental architecture that we have. So this is not very task specific, right. And I think this can be augmented by quite a bit with these sort of side puts and context signals. And maybe we need to we can think about modulating inputs. There's also an interesting connection, by the way, to like LSTMs, which essentially do exactly this right. An LSTM has like a C signal and an H signal. I don't exactly remember what they stand for. But let's just call C context and H the hidden state. And then there is the X the input of that particular sequence. And then there's like, there's like various ways of multiplying them and adding them and concatenating them and multiplying those here, right, and then modulating them via some sort of gating and forget gates and so on. So it is very reminiscent of an just an LSTM, just not recurrent, but sort of this this gating mechanism, except the LSTM obviously constructs the context signal and the hidden signal from the same from the same state. So somewhere here, there are then outputs again, like the context and the hidden state for the next vector. But it's interesting connections to all the things we have so far. And you know, maybe maybe we could bring them together in sort of more simple, more unified form. And I like that they applied it specifically to a particular task. And they can show look, this helps for this particular thing. Alright, that was it for me. I know this was a bit longer, but is a long paper, is a bit out of the box. And I hope you learned something I did certainly. Let me know what you think and bye bye.
 Hello, this is a comprehensive paper review on a paper called Avoiding Catastrophe, Active
 Dendrites Enable Multitask Learning in Dynamic Environments.
 This is a very cool paper because it combines ideas that come from biology, which are active
 dendrites and ideas that come from deep learning, namely the problems that we face in multitask
 learning and in continuous learning.
 Catastrophic forgetting is one of the main problems of these areas and the method of
 active dendrites directly inspired by biology can really help with that.
 So this video is a comprehensive review on the method of active dendrites in deep learning
 as the paper describes it.
 By the end of the video, you'll have a good understanding of what is in the paper.
 In the next video that I'll publish tomorrow, there will be an interview with the authors,
 which was also super interesting.
 And I definitely invite you to check out both.
 As always, if you have any comments, please leave them in the comments on YouTube.
 Leave a like if you do like the video and I'll see you around.
 Bye bye.
 Hello there.
 Today we're going to look at Avoiding Catastrophe, Active Dendrites Enable Multitask Learning
 in Dynamic Environments.
 This is by researchers of Nementa, Cornell and Stanford.
 So this paper proposes to bring some of what has been lost in translation from real biological
 neurons to deep learning neurons to bring some of that back into the deep learning neurons,
 specifically the concept of what they call active dendrites and also a bit of sparsity
 that is to be found in biological neurons.
 So they bring these back into deep learning neural networks.
 And it turns out that that is pretty useful to combat something known as catastrophic
 forgetting, thus the title of the paper, Avoiding Catastrophe.
 So catastrophic forgetting is a phenomenon where in multitask learning or continual learning,
 a network has to learn many things at once.
 And then these things interfere with one another.
 And it turns out that our methods of training neural networks using backpropagation aren't
 really good at that.
 So either they don't learn any of the tasks because they conflict with each other, or
 in continual learning, they do this catastrophic forgetting where as soon as a new task comes
 in, they've completely forget about the old task.
 So many solutions obviously have been proposed.
 And this right here isn't like is not entirely ultra novel, but it is interesting.
 It ties together biology and sort of practical applied deep learning.
 And it does have some connections to, for example, modern transformer architectures
 and so on.
 So I'd also be interested to hear what you think how this stuff is all connected.
 So they start out saying that the artificial neural networks, they call these ANNs.
 So whenever you do in this paper, ANNs means sort of the deep learning neural networks,
 we have to be a bit careful when we talk about things that involve biology, because neural
 networks is an ambiguous term there, like the neural networks is an ambiguous term because
 it appears in both domains.
 So they they claim they fail dramatically when learning multiple tasks, a phenomenon
 known as catastrophic forgetting.
 And I already said catastrophic forgetting, it essentially means that you can't learn
 many things at once.
 So it says learning multiple sequential tasks can lead to significant interference between
 tasks.
 They look at two different they look at two different tasks right here.
 One is multi task reinforcement learning.
 And the other one is continual learning.
 So in multi task reinforcement learning, it's essentially reinforcement learning with multiple
 tasks.
 So you're some sort of an agent, and you're in some sort of environment, and you have
 this basic loop of sending an action and getting back some kind of observation and reward.
 However, however, there are multi there are many tasks in this environment.
 So maybe you see it and maybe you don't.
 But as part of the definition of the problem, I think in this particular environment, you
 also get back kind of an indicator of which let's call that T the task indicator.
 So which task you currently supposed to fulfill.
 So the same environment has many tasks.
 And then obviously, your reward is going to be dependent on which task is currently active.
 So you're going to give the agent a mixture.
 So every new episode, the agent tackles the task is different, and therefore, if the agent
 just does the same thing as in the last episode, it might get a completely different reward
 because the task is different, right.
 So that is multi task reinforcement learning.
 And it turns out that and this papers have established this before and I think we have
 even made a video on some of them that if you look at the gradients, they often conflict
 with one another.
 So learning one task would pull a weight in some direction and learning another task would
 pull it sort of in a different direction.
 And there are papers that try to make these gradients as like orthogonal as possible or
 project them somehow into a task specific subspace.
 But as it stands, conflicting gradients can arise in these multi task settings.
 And therefore, the classic way of training neural networks with back propagation to update
 all the weights at the same time, just isn't very conducive.
 Even worse in continual learning.
 So here, we're not necessarily in reinforcement learning anymore, although we could be.
 So this is this is simply continual learning, where you present a neural network.
 So you have a neural network, the neural network is able to, you know, take whatever picture,
 let's say it's a picture classification and give you some sort of a class label for that
 picture.
 And now you have different tasks.
 So you have task one, task one might be classify, you know, classify cats from dogs, then task
 two might be classify, I don't know, cows from beavers, task, and so on.
 So there is also a bit of a specification gap.
 Some of these continual learning benchmarks, they will always have the same classes, but
 different data sets, some will have different classes, some will have new classes, and so
 on.
 In this particular case, we're looking at permuted MNIST, which is sort of the MNIST
 data set.
 So you know, there is whatever picture, and there is some sort of handwritten digit in
 here.
 And the the permuted MNIST data set is simply that every task that you consider, so task
 one would have a permutation applied to all the pixels in in this picture, but always
 the same permutation.
 And then task two would apply sort of a different permutation, permutation one, permutation
 two.
 So it's kind of a different task.
 It's the same classes, you're still classifying digits into zero to nine, but the permutation
 is different.
 Therefore, it's like you have to learn a new task if you don't have some sort of built
 in symmetry prior in your neural network.
 Obviously this, we're not going to use conv nets right here, because conv nets would make
 no sense if your pixels are permuted.
 We're simply going to use feed forward networks.
 The goal isn't to get state of the art.
 The goal is to show the difference between what if we use regular neural networks, and
 you can imagine right here, if I train on task one right here, and task one has some
 kind of a permutation in the pixels, I'm able, you know, these neural networks, they're able
 to learn that because if they're feed forward networks, they don't care about neighborhood
 anyway.
 So they they are able to, you know, we train we train these weights right here to to completion.
 And then I activate task two, right?
 Right after task one, I stop giving the network data from task one, and I start giving in
 data from task two.
 So also different permutation, I also label my images, give it to tasks two.
 Now I'm going to train these weights, I continue training these weights.
 And there is some effect when we talk about large language model pre training in that
 whatever you pre train on that kind of stays around.
 So any fine tuning in large language models isn't going to completely erase the pre training.
 So it actually matters what you pre train.
 Although this is not the same right here.
 First of all, we're dealing with way smaller networks.
 And these way smaller networks, they're able to be kind of overwritten mostly.
 And also we're dealing with classification tasks right here, and not some sort of language
 modeling task.
 So yeah, these these weights, they will just be overwritten to the point where task one
 is forgotten.
 It's nowhere.
 So we've again, if we draw up some sort of a weight, task one would pull it in this direction,
 that would be the gradient.
 So the weight would slowly update by update going this direction.
 And then all of a sudden, we activate tasks to which will pull it in this direction.
 So the weight would then travel into this direction, and essentially forget about task
 one.
 So it is nowhere near where it should be for task one.
 As I said, there are some methods of solving this with orthogonal projections and so on.
 But as a basic rule, our deep networks aren't very good at that.
 So what do we do about it?
 This paper's idea is that since our deep networks use a model of the neuron that looks very
 much like the thing on the left, so you have your your input weights, which are commonly
 known as the weight matrix or the weights of the layer.
 This is just one row or column, I guess.
 Well, it depends on how you specify the layer.
 But these are just all the input weights going into one neuron, they're summed up.
 So this is the matrix multiplication.
 And then there is some sort of a nonlinearity right here, which could be a sigmoid, which
 could be a tan h, which could be a ReLU.
 And that's essentially still the model that we have.
 This is like an over like it's decades old, this this model.
 And it served us pretty well, but it has forgotten some very important aspect of biology.
 Here on the right, you see a pyramidal neuron, a pyramidal, a pyramidal, I'm just going to
 call it pyramidal because pyramid.
 So this is obviously way different.
 So well, first of all, it's not a schematic, it's kind of like an actual drawing, you see
 the axon right here.
 And the axon splits up into different parts, which is, you know, is like our regular neurons,
 they connect to all the neurons in the next layer.
 Although one difference is you can already see that there are way less connections from
 here than you would have in a fully connected layer.
 So there is a degree of sparsity in biological neural networks that is not represented in
 the deep neural networks that we build.
 And then the inputs right here, we just consider all the inputs to be the same.
 However, there is a difference between what they call proximal inputs and distal inputs.
 So proximal inputs would be inputs that are very close to the cell's body.
 And those behave very much like the linear influence that we see in our model.
 However, there are also these distal, by the way, these things are called dendrites.
 There's a difference between the axon, which is this thing here, and the dendrites, which
 is this thing here.
 Every neuron has one axon, but can have many, many dendrites.
 And dendrites are sort of like, they're just kind of elongations of the cell body.
 So any other axon could dock either directly on the cell body or close to it, or could
 dock on any of the dendrites.
 So you can make connections from axon to body or from axon to dendrites.
 And dendrites are kind of like harbors, like ports or docks for incoming traffic.
 Yeah, that's how I can explain it.
 However, these distal dendrites, they're not acting like as much as linear things.
 What they are doing is, and this paper describes that, is they act like their own little subunit
 that computes its own function.
 So it's almost like a mini neuron inside a neuron.
 And that mini neuron can then influence or modulate the cell body.
 So whenever that mini neuron is, for example, very high, is very activated, it will raise
 or lower the activation threshold for the main cell body.
 So it can sort of influence the main cell body in a multiplicative way.
 And that's exactly what we're going to see in this architecture.
 So yeah, I've sort of skipped a lot of the text right here.
 Um, yeah, if you're a Patreon, you get these notes, I hope they help.
 I've never considered my scribbles to be super duper helpful, but I've started pre annotating
 and I hope it helps someone.
 But yeah, these are mostly for me to see what I have to look at.
 So what does that have to do with continual learning?
 Well, they describe right here, they hypothesize that biological properties of pyramidal neurons
 in the neocortex can enable targeted context specific representations that avoid interference.
 So pyramidal neurons, which comprise most cells in the neocortex are significantly more
 sophisticated, demonstrate a wide range of complex nonlinear dendrite specific integrative
 properties.
 And they are hypothesizing that this modulation property that we've just discussed, this modulation
 property could battle this catastrophic forgetting.
 Specifically, what they say is that, well, we have many of these dendritic distal sub
 modules, and these could learn and there are some biological evidence for that to recognize
 different contexts in which you are in.
 And depending on which of these is active, that means which context is recognized, it
 can modulate the body of the cell.
 So the cell could react differently depending on the context.
 And that is one of the ingredients exactly that we need to avoid this catastrophic forgetting
 or do multiple tasks at the same time is to say, hey, I'm only going to activate my cell
 body if I'm in the correct context, meaning for example, a particular task is active.
 So the cell body can learn its weights to specialize on a given task and rely on the
 sub units to recognize when it needs to fire.
 And obviously, if there's some structure to the tasks, we can also think of these being
 sub tasks.
 So sub tasks are sort of being activated that can then generalize and be integrated into
 multiple tasks and so on.
 So there's a bit of related work.
 The active dendrites that is pretty much pretty much what I just described.
 You can see each distal dendritic segment acts as a separate active sub unit performing
 its own local computation.
 When input to an active dendritic segment reaches a threshold, the segment initiates
 a dendritic spike.
 So this is not a neural like axon spike.
 It's a dendritic spike that travels to the cell body.
 Okay, I've apparently memorized this passage.
 It can depolarize the neuron for an extended period of time, sometimes as long as half
 a second.
 They don't model time dependency right here, by the way.
 That's something they don't integrate right here.
 During this time, yeah, the cell is significantly closer to its firing threshold and any new
 input is more likely to make the cell fire.
 This suggests that active dendrites have a modulatory, long lasting impact on the cell's
 response with very different role than proximal or feed forward inputs.
 So they say they typically receive contextual input that is a different input than received
 in proximal segments.
 Proximal are the near ones.
 These context signals can arrive from other neurons in the same layer, neurons in other
 layers or from the top down feedback.
 Another thing they don't model right here is any sort of top down feedback or same layer
 or anything like this.
 I'm just taking this away.
 What they do model is these dendritic subunits.
 The second thing they're very interested in is sparsity.
 So sparse representations are ubiquitous in biological neural networks, not so much in
 deep neural networks.
 They claim that studies show that relatively few neurons spike in response to a sensory
 stimulus across multiple sensory modalities.
 Sparsity is also present in the connectivity.
 And they claim that one advantage of sparsity in representations is that vectors for two
 separate entities have low overlap.
 So they're now talking about deep networks because biological networks don't have vectors.
 So they're talking about how if you impose sparsity in a deep neural network, and you
 are in high dimensions, then your representations likely will not collide because a lot of the
 entries are zero.
 Low representation overlap among unrelated inputs may be particularly useful when an
 artificial neural network is learning multiple unrelated tasks.
 And that's why they are interested in the sparse representations.
 Because if different things aren't likely to overlap, they're not likely to interfere
 with each other.
 And therefore they might be useful to combat catastrophic forgetting.
 So two things.
 We're going to implement these active dendrites into our models, and also we're going to implement
 a degree of sparsity.
 And we're going to observe how these two things work together to combat the catastrophic forgetting
 phenomenon.
 That is essentially what this paper suggests.
 So let's look at exactly how they do this.
 I think it's best to jump to the model right here.
 So this is one of the models or one of the architectures they use.
 This is the actual arch, they use two layer neural networks.
 So yeah, this is these are these are not these are not huge networks that they use right
 here.
 It is for reinforcement learning.
 So it is kind of a soft actor critic, they use this benchmark right here, where a robotic
 arm needs to perform multiple tasks in the same world.
 And in this particular task, the agent always gets the information which task is active.
 So which task is active goes into this context vector on the left, this is a one hot vector
 that is fed as a context signal.
 What's special about this network is that first of all, you can see that there is a
 linear layer and that is not some classic linear layer that is a special linear layer,
 namely the active dendrite linear layer.
 So the active dendrite linear layer has a feed forward signal.
 And that feed forward signal is treated just as a classic deep neural network feed forward
 signal.
 So that would be the feed forward signal would essentially be whatever the input here is,
 in this case, probably the robots state or something, and its position and it's maybe
 the position of the whatever object it needs to grab, if that's not always at the same
 place and so on.
 So that's the state input.
 And if it if we're only one task, the network could just learn from this input.
 However, this is multiple tasks, so it gets the context vector, the alternative, the baseline
 what the baseline will do is it would append the context vector right here, and just sort
 of extend this feed forward layer.
 And it would say, well, the network essentially has access to this information right here
 in its input.
 So it should technically be able to handle that.
 However, they're going to show that, you know, they're going to implement this in a baseline
 going to show that that's not as helpful as what they're doing.
 So we have a feed forward signal.
 And that computes some output, you can see that's independent of this context vector.
 So the feed forward layer, the weights of the feed forward layer, which sit approximately
 here, they're going to be, you know, multiplied by the weight matrix summed up.
 And then there's some output signal right here, just in a classic feed forward layer,
 the context vector comes in here.
 And what it's what it's going to do, remember, this is a one hot vector.
 For now, they make it more complicated later, it is going to be matched with each of what
 these things are, these things are called dendritic segments.
 So it is going to be matched with each of them, and the matching is simply done via
 an inner product.
 That's what this little sum symbol does right here.
 So there's an inner product between the context vector and the dendritic segment.
 And then they're going to select whatever dendritic segment matched the highest and
 that is going into here.
 And then here is a modulation function.
 So the signal that is the highest, the highest inner product with whatever dendritic segment
 is going out here and modulates that signal, and that's going to be the output.
 Now let's look at how these dendritic segments work, because that's really sort of the meat
 right here.
 Here you can see the forward signal, the forward signal is your classic signal right here.
 There's a weight matrix or vector in this case, there's the input, there's a bias.
 The dendritic segments are, they're just vectors.
 These are trained, okay, every single one of these dendritic segments is a set of weights
 that is trained and it's different as far as I can understand each neuron has its own
 dendritic segments and for each dendritic segments, it has its own weights.
 So there's no weight sharing going on among the dendritic segments, which would, I think,
 break the whole thing, although I guess one could come up with some sort of smart like
 meta weight sharing right here.
 But the idea is that, as you can see from the formula, we're simply going to take the
 context vector, calculate the inner product with all of these dendritic segments, take
 the max dendritic segment, that's going to be some kind of a number, right?
 This is an inner product.
 So this is the strength of whichever dendritic segment matched the most.
 And then we're going to take a non-linearity, in this case, a sigmoid function, and we're
 going to multiply the at the feet forward signal that we have with this sigmoid function
 of this inner product.
 So this can, you know, the sigmoid is between zero and one, I think.
 Yeah, I think they retain the sign, so they take the max absolute value in the end.
 But let's leave that out for now.
 So whichever segment matches the most, that's some number that goes through a sigmoid.
 So let's think about this.
 When is this thing one?
 It's one whenever one of these dendritic segments activated, right?
 So we take since we take the max, one of them needs to activate, and then this thing is
 one.
 So these dendritic segments, they are sort of like, like receptors for contexts that
 where this neuron could be relevant.
 So they are sort of like, you know, feature detectors.
 And if they they expose some kind of some kind of vector, they are obviously vectors.
 So in the space, there's like here, like, you know, I have maybe I have three of these
 dendritic segments, and I say, well, I'm interested if if my representation, if my context representation
 is any of those three in that direction, then I'm interested.
 So if the context comes in like this, they're just like, no, no one is interested.
 Therefore, the sigmoided maximum is going to be zero.
 And it's going to block the signal right here.
 However, if the context comes in is very close to what one of these segments is, then it's
 like, oh, wow, this actually might be relevant for this neuron.
 Therefore, the sigmoid.
 So the inner product is high, the sigmoid of the inner product is high, and the signal
 is going to be propagated through.
 Interestingly, in the experiments, they always expose like as many dendritic segments per
 neuron as they have tasks, which I thought to criticize that because I was like, well,
 that's kind of cheating.
 But now I don't even know if that is necessarily like, wouldn't one dendritic segment suffice?
 Like if it could perfectly recognize if every neuron was only relevant for one task, and
 if that could be perfectly recognized by the context vector, I guess that would work.
 But this is more powerful, right?
 You can present a number of situations where you would be interested in.
 Ah, I guess, okay, if you have as many dendritic segments as you have tasks, then every neuron
 could be relevant for every task.
 So a neuron could be relevant for all tasks or for just two of the tasks and so on.
 So yeah, I still maintain it's a bit of cheating to make as many dendritic segments as you
 have tasks, because that's implicitly telling the network how many tasks you have.
 But you do get the task as the context.
 So you already know anyway, right?
 In any case, that's what this network does.
 It exposes these things, it's able to take this context signal and modulate that signal.
 The second thing it does is this k winner takes all.
 And this is very much like maybe the sort of sparse mixture of experts that you might
 know from transformers or the concept.
 So what it does is it simply calculates a maximum maximum activation over the entire
 layer and it only lets through the highest the highest k many things.
 So it's k winner takes all k could be three or five or something like this.
 But in any case, it is not as many as you have neurons.
 And all the other neurons, they're just set to zero.
 Therefore, they also don't receive any gradient.
 So here you can see how these two things play together.
 First of all, we're going to modulate so we're going to block a lot of the signals right
 here.
 Blocking means we're just going to multiply them by a very small number if they're not
 relevant.
 And then it's not just that they're very small.
 Actually, we're just going to pick like the top five.
 So all the numbers that are small, we're just going to eliminate completely.
 I don't know if this you know, this method of achieving sparsity is necessarily the best
 one to pick the K best, or if it'd be better to just threshold somewhere.
 Because K, then is some sort of other hyper parameter that you might, you know, set via
 cheating, or that you might have to try out and some some sort of a threshold might be
 more robust, especially since the sigmoid is fairly, fairly steep function.
 Yeah, that's, that's the architecture, essentially.
 So I hope you can see how this sort of connects to to other things.
 Especially, I'm interested in this modulation property.
 And I'm also interested in in the sparsity approach.
 Obviously, if you have sparse representations, there's not going to be any gradient flowing
 back through the neurons that weren't activated.
 And therefore, there's not going to be any gradient into these neurons.
 That means these weights here aren't trained for that particular neuron.
 It means these dendritic segments, which are, again, these are parameters trainable parameters.
 So these blue arrows are back propagate trainable, they will only update if the neuron has actually
 been selected in its forward pass.
 So they're random at the beginning, and then with time, they will fine tune for specific
 contexts.
 So they will sort of move.
 And yeah, there is a bit of a danger that some of these are just become ghost parameters.
 But I guess as stuff moves around, and as initializations are diverse and random enough,
 almost everything will will become sort of selected at some point, if your inputs are
 diverse enough.
 Yeah, so that's that.
 I've skipped a lot of these a lot of the text right here.
 You can see the K, the K WTA, the K winner takes all representation, we're simply going
 to let the signal through.
 If it's in the top K activations, and it's zero, otherwise.
 Yeah.
 Exactly.
 So here they say only the neurons that were selected by the WTA function will have non
 zero activations and thus non zero gradients, only the weights corresponding to those neurons
 will be updated.
 And that's how the two things work together to battle catastrophic forgetting in that,
 if the context, if the dendritic segments successfully learn to recognize different
 tasks, that means that only the neurons that are involved in a particular tasks will will
 be updated by that task.
 And therefore, the network will not will not forget the other tasks or not forget them
 as easily.
 Because the sparsity also the sparsity kind of forces not all parameters to be updated.
 And the dendritic segments forces these sparse updates to be in a very structured, very consistent
 fashion.
 And yeah, they also say that only the dendritic segment J that was chosen by the max operator
 is updated, all other segments remain untouched.
 So even if a neuron is part of this K top K activations, only one dendritic segment
 is updated, namely the one that matched the most with the context.
 And this again ensures that maybe if a neuron is relevant to different tasks, the other
 dendritic segments they can they can keep their place.
 Even if we train in a new task where this neuron is also relevant, if it was relevant
 to an old task that might be stored in a different dendritic segment than the one that is activated
 right now.
 And that dendritic segment due to the max operator will not receive a gradient and will
 just remain as it is.
 Of course, this doesn't scale, you know, forever.
 And to all degrees of noise, and there is a there is a way in which tasks can be too
 related.
 So I would guess that in a model like this, if tasks are very related, they will activate
 the same dendritic segments and therefore override each other.
 But then also if tasks are very related, you would expect that there is some form of generalization
 or crossover among them.
 But the difficulty has never been that much with generalization.
 It has always been with the fact that if you think of, for example, large language models,
 I also think of large language models as continual training, they often they don't even run in
 a single epoch over some of the data, and they still learn from it.
 So they see a data point once right and, and then, you know, that's that's that and they
 still are able to incorporate that somehow.
 So how are they not subject to catastrophic forgetting, they also in a way implement
 different tasks because I can query GPT-3 with so much stuff, like it can do so much
 different diverse things.
 It is all it is like a bit of, you know, sure, it's always the same loss and the gradients
 don't necessarily conflict of that loss.
 It's kind of a multitask learning.
 And one key difference is that GPT-3 is presented with sort of an IID shuffled sample of the
 training data.
 However, here, the all the data of task one comes first, and then all the data of tasks
 two comes later.
 So even if there's some generalization aspect, I would expect if tasks are close together,
 task two will override task one, because the same dendritic segments might activate.
 And just from the model here, they don't have a way to, I feel they don't have a way to
 battle that maybe they are there of a different opinion, but maybe some sort of how should
 I say this, some sort of a contrastive method, like a contrastive addition to these dendritic
 segments, like pushing them apart from each other for different tasks, you know, if they
 have the task information or just plain pushing them apart from each other, maybe hallucinating
 pseudo tasks for that, maybe a way to automatically adjust to how close together or far apart
 the different tasks are.
 Yeah, that's just my, what I would guess might help.
 But maybe I'm completely wrong.
 Tell me what you think.
 They say we hypothesize that a functional specialization will emerge where different
 dendritic segments will each learn to identify specific context vectors.
 So that's the model.
 Now they go into the experiments.
 As we already said, they do two things, multitask reinforcement learning.
 This is this robot thing.
 So it's all at the same time.
 In this particular case, it's not one after another.
 It's all at the same time.
 I think each batch is always from the same task, but like the next batch will be of a
 different task, I think.
 Yeah, but it's different tasks, right?
 So the same actions don't lead to the same reward.
 And that is means conflicting gradients.
 They use a very basic RL algorithm right here, which is not necessarily important for our
 discussion, just to say that the networks are quite small, right?
 They have two hidden layers, each with 2800 neurons, which, okay, that's that's sizable.
 So they're, they're quite, they're quite fat hidden layers, but it's just two of them.
 And then each one is followed by a K winner takes all activation function.
 And then there's a final output layer.
 They say the first layer has standard neurons, whereas the second layer hidden, the second
 hidden layer contains active dendrite neurons, which are modulated by the context vector.
 In this case, the context vector just encodes the task ID as a one hot vector.
 And yeah, each active dendrite neuron in our network has exactly 10 dendritic segments,
 the same as the number of tasks to learn, they do ablations where they increase that
 number of dendritic segments.
 But yeah, I do think they're giving their model the absolute best chance to learn right
 here, by setting some some of these parameters with essentially, okay, it's not hidden information
 in this particular case, but it is in the next case where we're not getting the task
 ID, as you will see.
 So this is how the model looks.
 There's the state vector, there's feed forward, we have some sparsity enforced by these, notice
 that it's really interesting that sparsity is even enforced here without any without
 any modulation.
 And they do also some ablations on that.
 But I'd be interested why they didn't choose to also have dendritic segments in the first
 layer.
 It seems quite odd, honestly, to set up an experiment like this.
 Yeah.
 And the other thing is, they say, although we control the hidden sizes to yield approximately
 the same number of total nonzero parameters, we note that MLP baseline contains nearly
 500k more nonzero parameters than our active dendrite networks.
 They speak a lot of these nonzero parameters, and they count the network sizes in nonzero
 parameters.
 So I would be interested what's the difference between parameters and nonzero parameters
 and what it was is a nonzero.
 I've not seen this exactly explained in the paper.
 Is that like at the end of training, if a parameter is zero, you don't count it?
 Or is it somehow different?
 I don't know.
 But safe to say they do try to make the networks as you know, with the same number of parameters,
 which means that if they have these dendritic segments, which are quite a number of parameters,
 they have to, I mean, not that many compared, but they have to turn down the the other parameters.
 So here, you can see the results at the beginning, the active dendrites network in blue is sort
 of underperforming, but then it overtakes the baseline, the MLP baseline.
 And yeah, the errors here, the variances are quite large, as you can see.
 They do run another analysis where they just select the top five for each.
 And you can see that it separates a bit more cleanly, although I'm not sure if that is
 like, is that is that a thing?
 Like, can you say I'm just going to select like the top five of each to reduce the variance?
 I'm not sure if the the the max distribution is the same as the mean distribution.
 Like could I do that in practice?
 Maybe not if I just have one run, which is essentially what I'd want to do in practice.
 I couldn't necessarily do that.
 I don't know.
 In any case, they beat the MLP baseline in both cases, you can see that sometimes there
 are pretty significant differences, especially in what they claim are the harder tasks like
 the pick place tasks.
 And these are also the tasks that have very little overlap with the other tasks.
 So you would expect greater interference.
 And that's where they have a lot of gains in gains against the the baselines.
 In continual learning, they use this permuted MNIST as we've discussed.
 And so yeah, here's here's sort of the comparison.
 Yeah, you can see also you can see here the variants are huge for some of these tasks.
 Yeah, in the permuted MNIST data set, they okay, they don't have a graph, I believe.
 But in the permuted MNIST data set, they also are beating or are advancing against the baseline
 significantly.
 So we have somewhere, there are the results.
 So you can see right here, there isn't a baseline in this particular diagram.
 But you can see that the drop off is not very steep.
 And usually if you do this with regular MLPs, they just fail, like they they fail, which
 means that so this test accuracy is on all the tasks you've seen so far.
 So you get presented with whatever 20 tasks in sequence, and you evaluate on all of them.
 With regular MLPs, they just suck at this, like they forget the previous tasks.
 And yeah, that's that's that.
 So the fact that these networks are able to sort of hold up across and here you can see
 up to like 100 tasks is already pretty remarkable.
 They have two different variants.
 One where the prototype is given while training, which essentially means they have information
 about which tasks they're in.
 And one is where the prototype is inferred.
 And they describe these up here.
 So what they do, they now switch over from not providing the task ID as a context signal
 because that's kind of cheating.
 And they provide now these this prototype.
 So what is a prototype?
 A prototype is essentially a data point or it can be a latent vector.
 But here I think it's just a data point that is kind of the mean data point.
 So this would be the prototype of task A, the mean data point of all the data points
 in a particular task.
 So they provide that as the context as the context signal.
 Now what they can do now is here you can see how that works.
 It's just a mean.
 Well, I told you what they can do is if they don't have a task annotation, if they don't
 know what task goes with a particular data point, they can simply collect data points
 during training.
 They can say, well, here's a data point.
 Here is one.
 Here is one.
 Right.
 And it helps that they have the guarantee that each batch has the same task.
 And then they say, well, okay, we're going to make a prototype right here.
 And that's going to be our context vector.
 And then the next batch comes in and it's kind of like over here and they say, well,
 this is not very close.
 So we're going to make a new prototype right here.
 And then the next batch comes in and it's like here and they say, ah, that's probably
 of the same thing again.
 So we're going to use that prototype to provide to the system.
 So it's kind of this heuristic thing, averaging the data points, which I find to be quite
 weak, like averaging the pure data points is like, it might work in permuted MNIST,
 but there's definitely room for improvement right there, because that is not going to
 be informative at all in in many or most tasks.
 And obviously, there's also like a hyperparameter to set, like, you know, what's the appropriate
 distance measure right here?
 And also, this just going into this as the context signal.
 And the context signal is essentially just worked out by inner product as we saw up,
 sorry, up here.
 So the signal is just it's just an inner product with some of these U vectors.
 If this gets any more complicated, there's going to need to be a lot of machinery in
 front of the context vector, like, I would expect we need to pass it at least through
 some hidden layers to compute something of value.
 But for permuted MNIST, it's going to be enough, right?
 So they recognize which tasks they're in.
 Now, I am interested why exactly they switched from providing the task ID, like, at least
 in first in a first instance, why they switched over to providing these prototypes right here
 as the context signal, right, just experimentally, they have this one experiment in this one
 setting, where they they just provide the task ID, and then they have the other setting
 where they do something different.
 I would I would get it if they did both things in the same setting.
 But having two different settings and just doing two different things is a bit suspicious,
 I guess.
 And also here, you can see they provided actually to both layers, and not just to one layer.
 I would like to know the story behind this.
 They also compare to a baseline, which is called SI.
 So SI, as they describe here, it is a thing that operates solely at the level of synapses,
 it maintains an additional parameter per weight that controls the speed of weights adapting
 to specific tasks.
 The two approaches are complementary.
 That's why they can be combined.
 You can see on the right, so on the left hand side, you can see what happens if you infer
 these prototypes during training.
 And you can see it's just a little bit worse, which I think is like 100%.
 So I don't know how much better or worse they would be if they actually gave the task ID.
 But I think this distance right here, that is only going to be possible on permuted MNIST.
 Maybe I'm wrong.
 Maybe I'm wrong.
 So here you can see, interestingly, right, here's the active DEND, right?
 It it this is kind of the curve from the left.
 And then these SI method just by itself actually beats the active DEND, right?
 However, you can combine both as you can see, and both together are stronger and give you
 an even better, better boost.
 So that is, I mean, it's, it's, it's, it's good if you can combine all the tricks that
 you had so far.
 I would have liked to have here like a like, okay, the MLPs, they just suck.
 Because right now, it's not exactly clear how much they suck.
 Although I'm sure that there's some appendix table, and I haven't looked, I haven't found
 it.
 The paper is quite long.
 So here they compare to a different method, which is called xDG, which is context dependent
 gating, sorry, they say this is the implementation closest to theirs.
 This is another idea.
 However, that one uses hard coded distinct sub network for each task.
 So this is pre allocated, it pre allocate says you sub network, you're for task one,
 you're for task two, you're for task three, they engineer this in a way where they expect
 some overlap between the tasks and some separate neurons.
 And then they only train the sub network.
 So they need the task ID to be provided.
 The implementation of tasks specific subset of the hidden layer, other neurons are forced
 to have an activation value of zero.
 This requires a task ID that determines exactly which neurons to turn on or off.
 It turns out so the way they emphasize all of this is that it turns out that they do
 beat the baseline as you can see right here.
 When you just do them by themselves, but as soon as you combine them with this SI technique,
 the xDG outperforms the active tendrites.
 So obviously they need to highlight the differences right here, which is a good tactic, right?
 And it's valid, they do do more.
 So here they say task information is inferred, it's not provided via this prototyping, where
 this provides a system with a task ID during training and testing.
 And it's important to see that even if they do the prototyping with the information of
 the task ID, they claim that during inference time, there is no task ID provided.
 And they simply, you know, they see whatever if a data point is whatever prototype the
 data point is closest to, that's the prototype they take.
 The second thing, sub networks automatically emerge via the use of dendritic segments in
 their model, whereas the baseline, it pre allocates different sub networks for each
 task.
 And that's that's legitimate.
 However, I don't I can't shake the feeling that they've like evaluated it.
 And then this thing was better.
 And they were like, ah, rats.
 Now what can we what can we do?
 Okay, we can't beat it.
 How can we make it?
 How can we make it different enough?
 And maybe that's when they decided, okay, let's try to like not provide the task ID.
 But let's try to come up with like, a dynamic way of figuring out the task or something
 like this.
 And that's the story behind why this prototyping exists, or maybe that that has like, that
 just turned out like it is, I don't know.
 But you know, it's it's interesting.
 It's interesting to see sort of there might there might be a research process behind this.
 And which is cool, because the research process sort of leads to more innovation, which is
 neat.
 And important question one that which I also had during reading of this paper.
 And no, that's not it.
 This is we're going to get to that.
 First, they check their hypotheses.
 So they say the hypotheses of our work are twofold.
 First, active dendrite networks modulate an individual neurons activations for each task.
 Second, the winner takes all activations use this modulation to activate sub networks that
 correspond to each task.
 They provide some evidence for this.
 So here, on the left and the right, you see the two tasks they tackle.
 And they give you an impression of which hidden units are active for which particular task.
 And they you can see that it's fairly sparse.
 So if you look at any given column or at any given row, then not many light up in dark
 green, which means that not many things are activated per tasks and a given unit is kind
 of specialized to particular tasks or a particular set of tasks.
 Now, without a comparison to a sort of regular neural network, or without a comparison to
 one of the two features of the network ablated, it's kind of hard to to see whether this is
 a lot or not a lot, especially on the on the right, you can also see like is this sparse,
 or is this not sparse?
 I don't know.
 I'm going to guess it is.
 Yeah, so I don't know, I'm going to believe them that this is especially sparse.
 And I think they also measured it at some point, actually the sparsity, but just the
 graphic alone isn't this isn't necessarily enough for me.
 They look at single neurons. So in the single neuron, they wonder which dendritic segment
 is responding to which task, right, there's a neuron A and neuron B. And you can see at
 initialization, a lot of the segments are responding to a lot of the tasks.
 However, after learning, it becomes much more quiet, and only very few segments are responding
 to to any or each of the tasks.
 However, also here, first of all, it's not, it's not super clear what we are to compare
 this with, because this could just be this could just be a phenomenon of kind of like
 the scale of stuff being wrong.
 Like at initialization, just the scaling of things being kind of out of out of whack,
 because you can see right here, there are entire regions that are just kind of dimming
 down, right?
 So yeah, obviously, a given a given neuron isn't going to respond to all the tasks, right
 with all the segments, it's not going to be involved in all of the tasks that would actually,
 you know, this this is a valid prediction of their hypotheses.
 And you can also see that especially neuron B here, if you look at segment eight, multiple
 dendritic segments are reacting to signal eight, which might be an indication that there
 is some, you know, they have learned to recognize different features that all indicate that
 for no segment eight response to multiple tasks.
 Ah, okay, that's, that's different.
 Okay, negate my argument.
 Forget what I said.
 I thought I thought it was a smart recognition.
 But you know, it's it is it is definitely evidence for the fact that there's specialization
 going on, but without a comparison to anything, it's hard to tell if that is that or just
 some sort of a scaling, scaling issue that just after training things are scaled differently.
 But just, you know, from from all the other evidence, they make a convincing case that
 there is this sparsity and specialization going on.
 So here is the last thing I want to discuss.
 And this is a question that I had when reading this paper, which is, aren't like, isn't this
 isn't there an equivalence to larger networks?
 Like aren't you just sort of sort of, you know, designing this this network in this
 special way?
 And can't I achieve the same thing with sort of a regular neural network if I just make
 it a bit larger?
 They say multiple studies have suggested that that dendritic computations performed by pyramidal
 neurons can be approximated by artificial neural networks that have one or more hidden
 layers from a computational and deep learning perspective.
 This is equivalent to claiming that ANNs with dendrites can be substituted by larger ANNs
 without dendrites, supposedly.
 And I have tried so they are going to make the case right here that that is not the case
 that they are outperforming, for example, three layer MLPs, which are about the same
 size and MLPs that are much larger, so much deeper.
 So they're going to outperform them at you can see right here number of tasks 100.
 Oh, this is this is probably the graph I was looking for before, no?
 Yeah.
 So here you can see how much how much the the MLPs suck.
 So yeah, they show that even if you scale them up, in fact, the 10 layer MLP is even
 worse, which is interesting, which might be might be interesting in itself.
 Like, why is it?
 Why is it worse?
 And is there like a crossover point here?
 But in any case, these MLPs, they get the context vector as an input, right?
 So technically, technically, they have all the information to do the same thing.
 However, the paper argues that it's the training procedure, back propagation, updating all
 the weights for the given data that is presented to us.
 This is particular to an ID setting of data, which we don't have right here.
 So no matter how big you make your neural network, supposedly, if they are correct,
 it would always result in the same problems due to the way that you train them.
 On the left, you see an ablation of the two ingredients.
 So the active dendrites only, the sparse representations only, and the combination.
 One second.
 So they do certainly give empirical evidence.
 And by the way, here is also an ablation on having more dendritic segments.
 On the top, they're trying to learn 10 tasks.
 On the bottom, they're trying to learn 150 tasks.
 And it's interesting to see that the gains here are kind of negligible, although maybe
 that's just a property that they're very close to 100% already.
 And here you can kind of see gains until 50.
 And then, well, okay, I might be imagining things that there's stronger gains here than
 here after you pass sort of the number of tasks barrier.
 But safe to say that, you know, more dendritic segments might also be useful.
 And maybe my skepticism of them setting parameters exactly, exactly as many as sort of exactly
 to the number of tasks they have is not super warranted.
 Also interesting is the fixed number of dendritic segments and varying activation density level.
 So here is this k, so how many things they let through each layer, you can see increases
 to the right.
 So you activate 100%, which would regress to a classic MLP.
 See if you activate 100%, it's really bad.
 And there are two things right here.
 Again, they're trying to learn 10 tasks or 50 tasks.
 Interestingly, interestingly, if at the beginning, obviously, you let nothing through, it kind
 of sucks, then you let some things through, it's already really good.
 And then it gets better.
 So there's some kind of an optimum around 10% ish or so.
 Interestingly, that's the case for both the things, even though one is trying to learn
 significantly more tasks, which is interesting, right?
 Then there is a drop off for both things, which you would expect.
 But then there is kind of like a flat flattening, followed by another drop off.
 And it's also interesting to to think about why that's the case.
 So here it might be that this is the situation where very few things are overlapping.
 And therefore the network is able to use specialized sub networks for all the things that it needs
 to do.
 And in this entire region up until here, it might be the case, you see it kind of drops
 off at the end after like 80%.
 It might be the case that most of the things are shared.
 However, the network can kind of encode stuff in the non shared part.
 And that can itself within the network kind of modulate whatever the shared stuff is doing.
 It's kind of like a shared feature extractor, followed by some modulation of the non shared
 parts.
 I would Yeah, it's interesting to think and then that crashes together once there is no
 more non shared parts.
 And there's no way of doing anything different in the different task settings.
 I was thinking myself, you know, getting back, sorry, getting back to can I just achieve
 the same thing with a larger network, I was thinking myself of how to do that.
 So they claim, No, you cannot.
 And I guess it's true.
 Let's think of okay, let's leave the sparsity away.
 Let's just think of this dendritic activation, right?
 I have my x that's multiplied by by W. And let's also leave the biases away.
 So I have my x vector down here, I have some W, which is a weight matrix.
 So everything's connected to everything.
 To till here.
 Now can I also and I have my context vector, can I somehow build a feed forward network
 that would also you know, have the appropriate weight connections that I could build myself
 the function W x times sigmoid, you see, let's also leave away the max right right here,
 I guess we can't.
 That's an integral part.
 And yeah, it's not clear to me how that would work necessarily with with a single layer.
 And it's also not entirely clear to me how that would work with multiple layers, like,
 you would have to build some very, like various contraptions of additions.
 Maybe you know, once you get a relu out on all of that, it might be more possible.
 But it's not easy to get this multiplicative interactions between signals working in a
 feed forward network.
 However, however, in transformers, that might be different, right?
 So you know, this here, this, you know, we can do this in transformers, I guess in feed
 forward networks, too.
 And then the max, we have we have softmaxes in transformers, right?
 So what we could do is we could have these things here as, let's call them queries, right?
 And these things here are the keys.
 And we apply the softmax in a transformer.
 And the values might just be a constant vector of ones.
 So the values might just be constant vector of ones, which would mean that if we multiply
 the softmax by this thing, we would simply select sort of the maximum out of that, and
 that's going to be one and everything else might be zero.
 Maybe I might.
 Maybe I'm I have this wrong, but maybe not.
 Yeah, I guess that that would work, right?
 So and then in the next layer, so that could be our output signal for layer one.
 And that could be our output signal for layer one in a different attention head.
 And then the multiplicative interaction again, we can get by via attention because attention
 constructs the attention constructs the weights dynamically by multiplication.
 So we could take this as as keys and maybe also queries.
 And then simply this could be the values right here.
 And then we multiply them together.
 And that's going to be a multiplicative interaction between that signal over here and the signal
 over here.
 So I guess transformers could model something like this.
 It's not easy.
 It's not going to be in one layer.
 It's not going to be non shared potentially right as it is here.
 So here nothing is shared of the parameters.
 But I would I would argue that the more powerful method of the transformer doing these dynamic
 weights, you know, there might actually be some connection here.
 And as we said, for the sparsity, we have sort of the sparse mixture of experts, which
 is kind of sort of a little bit similar.
 So looking through the rest of the paper, I don't I don't think I have anything annotated
 right here.
 There are hyper parameters.
 There are tables and more results and methods.
 But that's essentially it what I had to say about this paper.
 I like this paper because it sort of connects, connects biological concepts, it tries to
 reintroduce them, it augments the fundamental architecture that we have.
 So this is not very task specific, right.
 And I think this can be augmented by quite a bit with these sort of side puts and context
 signals.
 And maybe we need to we can think about modulating inputs.
 There's also an interesting connection, by the way, to like LSTMs, which essentially
 do exactly this right.
 An LSTM has like a C signal and an H signal.
 I don't exactly remember what they stand for.
 But let's just call C context and H the hidden state.
 And then there is the X the input of that particular sequence.
 And then there's like, there's like various ways of multiplying them and adding them and
 concatenating them and multiplying those here, right, and then modulating them via some sort
 of gating and forget gates and so on.
 So it is very reminiscent of an just an LSTM, just not recurrent, but sort of this this
 gating mechanism, except the LSTM obviously constructs the context signal and the hidden
 signal from the same from the same state.
 So somewhere here, there are then outputs again, like the context and the hidden state
 for the next vector.
 But it's interesting connections to all the things we have so far.
 And you know, maybe maybe we could bring them together in sort of more simple, more unified
 form.
 And I like that they applied it specifically to a particular task.
 And they can show look, this helps for this particular thing.
 Alright, that was it for me.
 I know this was a bit longer, but is a long paper, is a bit out of the box.
 And I hope you learned something I did certainly.
 Let me know what you think and bye bye.