Active Dendrites avoid catastrophic forgetting - Interview with the Authors
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper active dendrites neurons dendrites biological deep learning deep learning biology numenta numenta research numenta deep learning dendrites deep learning deep learning tutorial hierarchical temporal memory computational neuroscience reinforcement learning robotics multi task learning continuous learning continual learning permuted mnist
#multitasklearning #biology #neuralnetworks

This is an interview with the paper's authors: Abhiram Iyer, Karan Grewal, and Akash Velu!
Paper Review Video: https://youtu.be/O_dJ31T01i8

Check out Zak's course on Graph Neural Networks (discount with this link): https://www.graphneuralnets.com/p/introduction-to-gnns?coupon_code=SUNGLASSES&affcode=999036_lzknae-d

Catastrophic forgetting is a big problem in mutli-task and continual learning. Gradients of different objectives tend to conflict, and new tasks tend to override past knowledge. In biological neural networks, each neuron carries a complex network of dendrites that mitigate such forgetting by recognizing the context of an input signal. This paper introduces Active Dendrites, which carries over the principle of context-sensitive gating by dendrites into the deep learning world. Various experiments show the benefit in combatting catastrophic forgetting, while preserving sparsity and limited parameter counts.

OUTLINE:
0:00 - Intro
0:55 - Sponsor: GNN Course
2:30 - How did the idea come to be?
7:05 - What roles do the different parts of the method play?
8:50 - What was missing in the paper review?
10:35 - Are biological concepts viable if we still have backprop?
11:50 - How many dendrites are necessary?
14:10 - Why is there a plateau in the sparsity plot?
20:50 - How does task difficulty play into the algorithm?
24:10 - Why are there different setups in the experiments?
30:00 - Is there a place for unsupervised pre-training?
32:50 - How can we apply the online prototyping to more difficult tasks?
37:00 - What did not work out during the project?
41:30 - How do you debug a project like this?
47:10 - How is this related to other architectures?
51:10 - What other things from neuroscience are to be included?
55:50 - Don't miss the awesome ending :)

Paper: https://arxiv.org/abs/2201.00042
Blog: https://numenta.com/blog/2021/11/08/can-active-dendrites-mitigate-catastrophic-forgetting

Link to the GNN course (with discount): https://www.graphneuralnets.com/p/introduction-to-gnns?coupon_code=SUNGLASSES&affcode=999036_lzknae-d

Abstract:
A key challenge for AI is to build embodied systems that operate in dynamically changing environments. Such systems must adapt to changing task contexts and learn continuously. Although standard deep learning systems achieve state of the art results on static benchmarks, they often struggle in dynamic scenarios. In these settings, error signals from multiple contexts can interfere with one another, ultimately leading to a phenomenon known as catastrophic forgetting. In this article we investigate biologically inspired architectures as solutions to these problems. Specifically, we show that the biophysical properties of dendrites and local inhibitory systems enable networks to dynamically restrict and route information in a context-specific manner. Our key contributions are as follows. First, we propose a novel artificial neural network architecture that incorporates active dendrites and sparse representations into the standard deep learning framework. Next, we study the performance of this architecture on two separate benchmarks requiring task-based adaptation: Meta-World, a multi-task reinforcement learning environment where a robotic agent must learn to solve a variety of manipulation tasks simultaneously; and a continual learning benchmark in which the model's prediction task changes throughout training. Analysis on both benchmarks demonstrates the emergence of overlapping but distinct and sparse subnetworks, allowing the system to fluidly learn multiple tasks with minimal forgetting. Our neural implementation marks the first time a single architecture has achieved competitive results on both multi-task and continual learning settings. Our research sheds light on how biological properties of neurons can inform deep learning systems to address dynamic scenarios that are typically impossible for traditional ANNs to solve.

Authors: Abhiram Iyer, Karan Grewal, Akash Velu, Lucas Oliveira Souza, Jeremy Forest, Subutai Ahmad

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, this is an interview with the authors of the paper on active dendrites. Now, if you haven't seen it, I've made a comprehensive paper review video on this paper and I released that yesterday. If you watch this video as it comes out, which obviously you do today, I'm going to interview the authors and we've all seen my review. So we'll be able to directly dive in. So if you haven't seen the review yet, and you want to know what's in the paper, maybe that is a good place to start. The authors here were really helpful and really informative answering all of my questions and concerns that I had and even bringing up some new interesting insights. So I hope you learn something from this interview or at least that it entertains you. And if you have any comments, please let me know in the comments below the video. I'll see you around. Bye bye. Hey there, today's sponsor is the course on introduction to graph neural networks. This is a course by my friend Zach Jost, who is an expert in graph neural networks, and also runs the welcome AI overlords YouTube channel has a very interesting blog and does many other cool things. He's packed all his knowledge of graph neural networks into one course that will educate you on both the theoretical and hands on practical aspect on graph neural networks. Graph neural networks are really important. They're definitely one of the most interesting areas in deep learning right now they're on the upswing, they model data that has an underlying structure that is connected that is not really well fit for any of the classic formats like tables or images. They've also powered a lot of recent advances in scientific breakthroughs, such as alpha fold protein structure predictions, or better traffic predictions. So if you're interested in graph neural network, I'll definitely recommend you check out that course. If you use my link, you'll get a 15% discount on the course enrollment is open right now and lasts until April 1 or until spaces run out. The course is a six weeks course. It's cohort based, you'll get access to a community to discord community of other students, and you'll get all the materials and hands on experience. All right, let's get into the video now. See ya. Hi everyone, today I'm here with the three joint first authors of the paper on active dendrites, Abhi, Karan and Akash. And I'm very, very happy to have you all here. This paper covers many areas, it covers biology, it covers neural networks, it covers kind of different architectures of stuff. It's very cool that you all sort of are here and are able to sort of answer my questions. Welcome, all of you. Yeah, thanks, Janek. Thanks for having us. Thanks for having us. It's very interesting paper. So I saw this paper and I was intrigued because it's not often that a lot of people say they do biologically inspired things. But it's not often that someone really goes and says, look, you know, here's what's missing, let's build it in. And then it actually leads to something that works. And that is, you know, the hypothesis in your paper, the hypothesis you pose on what should happen are actually confirmed at the end. And this is, I think, a very good story arc for a paper and a really nice thing to write up. So is this, how did this come to be? How did you get the idea of bringing these very two distant, not too distant, but these two distant fields together of sort of neurobiology and deep learning? Well, at Numenta, we're interested, one of the things we're interested in is in continual learning and learning multiple tasks, more generally speaking. And so, you know, we're looking at, but a lot of neural networks and deep learning today focuses on trying to solve a single task. So we said, well, you know, how is biology enabling the ability to solve multiple things in sequence or, you know, at the same time, learning different things? And so, you know, there's been a lot of work out there on active dendrites. And so, and it's not exactly clear what their role was. But a little while back, we speculated that, hey, they might actually be helping at the neural level to allow for continual learning. And so if we can build this idea into deep learning, then there might be some prospect there for addressing problems like continual learning and multitask learning. So is it fair to say that it grew out of sort of a need to solve a task? I think it grew out of the need to solve multiple tasks in sequence, either learning them together or in sequence continuously. To add on to what Karan was saying is that we believe that active dendrites can really aid in achieving these specialized neural circuits. And we can apply these ideas directly to any neural network and show some competitive performance on various benchmarks that involve continual learning setups. So I guess the purpose of this project, if you were to just summarize it very briefly, is we just want to show a proof of concept for a new idea that can allow deep learning to work in more dynamic environments and scenarios. To kind of add on to what Karan and Abhi said. So at a higher level, I think we were kind of examining where a lot of modern deep networks fail, and that's in these streaming task settings and multitask settings. And the kind of inspiration for our solution was directed towards biology and biological neurons, which is a lot of what Numentos focuses on. And I think quite nicely we found these existing benchmarks and existing tasks that show that typical deep learning networks fail in these scenarios. And we were able to build in these biologically inspired neurons to improve the performance in such dynamic settings by using the fact that we believe active dendrites in biology kind of do this kind of context dependent adaptation in multiple tasks. What I found interesting is that even though you targeted a little bit towards multilayer perceptrons, in principle, this active dendrites architecture is sort of pluggable almost anywhere. So you could always imagine some sort of a context dependent signal that gets routed in and modulates the signal that exists. So I think what I'm trying to find out is there are a number of things happening in this model. There is first of all the modulation itself, which is a relatively it's not really a known concept, at least in classical deep learning, we always have weighted sums, we rarely have the situation where two parts of the signal are multiplied together, or one modulates the other, it happens a little bit in LSTM and so on. The other one is the sort of recognition of a context and, you know, being context dependent. And then a third thing is this, this sparsity. Now, you have sort of combined all of them. Is there one thing that you think is specifically important? Or is it sort of the combination of things that is really what makes the difference? You have some ablations in the paper. What can you say about this? I think it's the combination of all these things acting together. So it's the it's the it's the dendrites, which are, you know, up modulating and down modulating certain neurons to determine which ones should become which which to determine which sub network should be invoked. And then it's as far as you on top of that, which is ensuring that, you know, a large portion of the network is essentially not performing or learning a certain task. And it's those two things together, which, which, which really gets at this idea of using specialized sub networks for different things. So I wouldn't say any any one one thing that stands out more than the others. So when we get let's get into the paper itself, you've seen my review of it, with respect to just framing the problem and maybe framing the architecture as such, is there do you think I have captured what you've tried to say? Do you think I've left something important out or have put emphasis on or have not put emphasis on something that you would like to put emphasis on when it comes to like, what the architecture is, what it does and how it works? I think your explanations for the architecture, at least we're very good. I think it does definitely does capture what we were trying to trying to say. And the whole point to kind of reiterate is that the same model with the same principles should work on completely separate areas. One is the multitask reinforcement learning. The other one is continual learning with permuted MNIST. And I think you touched upon that idea too. So yeah, I think that the kind of motivation that I think you in towards the beginning of your review, you showed you kind of compared the typical weighted linear sum neuron with the active dendrites neuron. And I think our motivation in coming up with this architecture was how can we incorporate a lot of these properties into active dendrites with having dendritic segments being able to either up modulate or down modulate certain neurons in a way that didn't completely change from normal back propagation trainable networks. So this architecture kind of brings in that flavor of having dendrites influence certain neurons, but does so in a way that mathematically allows for back propagation to train the networks and I think you touched on that pretty well as well. Do you think it's valid to sort of bring in biological concepts even though we train with back propagation? Because it's very evident that at least pure like correct back propagation isn't happening in the brain. Do you think it's still valid to bring in the concepts and maybe the brain is doing something like backprop? Or do you think we're sort of just kind of taking inspiration from biology in order to solve some of our problems? I think it's more so the latter. Of course, the most accurate biological neural network would likely not use back propagation, right? But this is one area where I think the goal was can we make deep learning just a little bit more plausible? And in doing so, can we make it a little bit more dynamic? So we're not necessarily here to remove backprop entirely and say that that's the best way that the dendrites in this architecture can work. Although certainly that is how it works in biology. The point was, can we just augment traditional deep neural nets to work in more dynamic scenarios? Now I had some criticisms with respect to just like that details of your architecture. For example, you always or you often choose the number of dendritic segments to match the number of tasks that you have, which obviously, if I was a researcher, I would do the same. But can you say maybe something about how this is in the brain? Like what numbers are we talking about? How many of these sub networks that are composed of distal dendrites? How many are there approximately? Do you know? Do you have an idea? And what can you say about how many we should build into a problem where we maybe don't know how many tasks we expect? From what I recall, probably in the order of hundreds or thousands of individual dendrite segments for each individual neuron, actually, it might even be more than that. The actual numbers escape me. But regarding what you said earlier about having the number of tasks be equal to the number of segments here, we found that actually, even though in a lot of the experiments we report here, we do set the number of dendrites to the number of tasks. We found that we actually don't need to have that many. And we actually have further studies which show that we can actually keep the architecture fixed and increase the number of tasks we're doing. I'm talking about continual learning here because for multitask, we're focused on 10 specifically. We can increase the number of tasks and the performance actually doesn't change by much. So that shows that as we're increasing the number of dendrite segments, we actually end up overparameterizing the network quite a bit, which we don't need to do. Yeah. So this is the plot on the left right here. You just increase the number of dendritic segments and the top line is learning 10 tasks. And it doesn't get noticeably worse, which I find to be a very cool property. I don't want to have to set the parameter very specifically. I can just set it too high and it doesn't hurt, which is cool. Which leads me to the plot on the right where you discuss the sparsity. I'm going to guess that's the sparsity parameter. So that's the thing that ultimately controls k. And I find it peculiar, not that there is an optimal setting, which I would expect because that I can't set high that I have to set between 0 and 1. So there's going to be some optimum in between. But there's this two bump thing going on. So what's going on there? Why is it like really good at lows, like high sparsity, and then there's like this plateau, and then it just flat like crashes down. I think there in the beginning, you know, if you have if you have too much. So yeah, I always think in terms of sparsity, so I'm converting from density to sparsity. So if you have if it's too sparse, right, there's not enough signal going through. And that's why, you know, as you as you increase the amount of signal that you're allowing through as you're increasing the capacity of your representation, then you're going to get you're going to get an increase in performance. But then if you have if you're using up too many units to to create that, to create that representation, then you're going to get more interference, right. And as you have more interference, you're going to you're going to you're going to forget more and more network parameters are overwritten as you move on to subsequent tasks. And so you get a drop in accuracy. And towards the end, so you know, you notice that it does fall drastically. Honestly, I haven't thought too much about why that happens. Although it is it is a pretty, pretty monotonic fall, even though I guess in that in that upper curve, there's a slight bump with that could just be due to seeding or something like that. But yeah, Yeah, I was more referring to like the plateau itself, right? There's there's this plateau kind of, and I I know, I know that there could be almost like two two modes of using the sparsity in one mode, I have entire sub networks that do the job. And in the other mode, I have like a shared network. Yet I have like separate things that just kind of like track, track which task I'm on, which would sort of correspond to what the baseline is doing, right? When people say, well, the baseline has access to the task to it can just allocate some units. No, it's maybe not a perfect analogy. But I was just wondering, it was just interesting to see that there's this kind of this type of plateau. Yeah, that's that's something I guess, we haven't gone too deep into. But this might, this might just be a property of sparse representations and how and how much overlap there is as you as you as you increase the sparsity level, it could just be something to do with that. So in your paper, you make really, which I appreciate you make really sure that you sort of always have the same amount of let's say trainable parameters in your architectures. And you show that by arranging them correctly, you can you can achieve a better result. You always use this name of non zero parameters, right? Is there like, is there a difference? Are there large swaths of zero parameters in one or the one of these architectures? Yeah, so this is something that we control for. In the beginning, this is why we mentioned the idea of weight sparsity. So in the beginning, when when we're actually creating the architecture from scratch, we decide that some layers have an X percent sparsity level applied to it. And what that really means is that X percent of the parameters are zero throughout the entire part of training, and even towards the end. So that's why we express everything in non zero parameters. So the MLPs, for instance, at least in reinforcement learning, are trained with no weight sparsity. So it's completely dense. There are no zeros anywhere in the in the layers. And then the your your architecture, you sort of modulate the amount of sparsity. And that is on top of modulating the K parameter of the K winner takes all layers. Yeah, there's two aspects to the sparsity. So one is activation sparsity, which is like, at a hidden, like when you have a hidden state vector, how many neurons remain non zero after the activation is applied, which is a K winner activation. And then the second aspect of sparsity is weight sparsity, which is how connected are subsequent layers in the network. So if a lot of the units in the weight matrix are zero, then this models the fact that subsequent layers in the network are not very connected, they're sparsely connected. To I guess answer your question again on that is, it's not something with weight sparsity, at least it's something that it's not something we modulate, it's fixed. It's a fixed percentage that we find. And this can either be done through fine tuning, or just Yeah, just just experimentation. Okay, because I think yeah, I might I might have just over read that. But but I recall that in the introduction, you say, you know, both the weights and the both the weights and the the activations are sparse, but then sort of the I think the winner takes all really focuses on the on the activations itself. Have you experimented with setting, you know, something else than K to a number or a percentage, setting maybe a threshold for sparsity or something like this, where whenever a signal is strong enough, it is let through? We haven't, we haven't done anything like that. But we could do that. And you know, there is a chance that it could work out pretty well if we if we have a fixed threshold. But one potential downside there is that, you know, if you have if you have too many signals that cross the threshold, too many units whose activation crosses the threshold, you're going to get more interference when you train. Or if you have not not enough neurons whose activation crosses the threshold, you're going to get you're going to get that phenomenon which you're showing on the screen right now on the left side where you have a drop in accuracy because your representations don't have enough capacity. So that's why we we opted to go for a fixed value of K. But even if you know, we didn't have even if we did have a threshold, I think one of your critiques were here, you know, now we have another hyper parameter K that we're choosing. In the other case, I mean, we'd have to with our hyper parameter would just be the threshold value there, right? Obviously, yeah. Yeah. So to me, this this continual learning setup is very cool. And you can generate data very easily using this permuted MNIST. But there is a bit of an issue that I have. And that is that if I use permuted MNIST, there is another thing there's like all the tasks are like the same difficulty, right? They're essentially the same task. It's just permuted. So I need to learn. Yes, I need to learn like a different function. So this would be the permutation identity. And then the pixels are permuted somehow, right? So all the tasks are kind of the same, right? Which warrants a static network architecture and every context vector is kind of the same length, right? And all the dendrites, they can they can sort of specialize in each of their little task recognition. What would change here? Or is it is this a drastic requirement to your architecture? Or do you think if many of the tasks were wildly different from each other, and you have this a little bit in the robot example, so what can you tell about when tasks are very different in their difficulty, maybe in their amount of training data, like how do these things influence an architecture that's targeted towards continual learning? In our case, I think there might actually be similarities between different tasks. And so like, you know, for example, in this case, in permuted MNIST, right, there's a certain certain pixels are more likely to be white. And certain pixels are more likely to be black, depending on the permutation. So maybe, you know, two different permutations could have more overlap in terms of which pixels are white, which pixels are black, or they could be totally separate. And if they're more, if they're more similar, if the permutations are more similar, then we could expect that the the sub networks that are selected by the dendrites will probably have more are likely to overlap more in which neurons become active, since there's a lot of there's probably a lot of similar computation going on. But of course, you know, in that case, difficulty doesn't really change at all. I think to kind of add on to that, I think a lot of it depends on the quality of the context signal. Because ultimately, that's the part of the network that indicates to the active dendrites, what kind of task you're solving, how similar is it to previous tasks you might have seen and things like that. So I think that in this in this permuted MNIST case, the way we're computing the context does allow for this property that Karen just mentioned, where if there's some overlap in the input space, then the context signal for that will demonstrate this input and perhaps allow for overlapping subnetworks to emerge. Whereas if you have like wildly different tasks, which is something we see more in the robotics environment, then these context signals can like differ more and indicate that the sub networks must be like must not overlap. I think it would be really interesting. And we've talked about this before to try a similar setup in a continual like robotics learning case where you have a streaming set of like robotics tasks. And I think that would probably be a super interesting study to do. And something that hopefully we will try at some point in the future. So I had I had some observations with respect to your experimental setup. It's very cool that you do two different things. But there are also noticeable differences on how you implement the two different tasks, right in the first task, you give the task ID directly. In the second tasks, you do this, this this prototyping approach, which is a more advanced approach. Can you tell a little bit about how is there like a reason why in because I could also imagine you just give me the task ID in the second task, or I do the prototyping in the first task. Is there like a research process reason? Like did you find that some things did work or didn't work? Or how did this come about? That all of a sudden, we're introduced in the new task, we're introduced to this new way of detecting the context. I think in the context of the multi agent, like, sorry, the multitask reinforcement setup, like the environment setup itself gives the task ID. And I think that the concept of multitask learning itself is more focused on if you have different tasks, which may conflict with one another, in terms of the types of behavior you have to do, or the types of predictions, can how can you mathematically still optimize your like joint objective function without and still be able to perform well on all the tasks. And the problem shifts not so much from trying to infer what tasks you're doing, to more, you know what tasks you're doing, and you want to try to do all of them. How can we like optimize this joint objective. This is kind of the way we use this one hot task encoding is in line with passwords that deal with multitask learning and multitask reinforcement learning, where you have this like one hot task encoding that is provided. I do agree that like the one hot encoding is quite convenient and a little bit arbitrary, you can probably use like a denser representation for each task or try to infer it. But I think for the purposes of our experiments, this one hot encoding seemed simple as it was environment provided and kind of like the point of the multitask setup was to again like try to show that this network architecture prevents from like conflicting updates across tasks and avoids this like interfering updates from occurring. I think for continual learning, the kind of the kind of setup of the problem itself is a little bit bigger and that you have to you're not always provided with the task IDs and you have to infer this on the fly, which again, I think Karn can talk a little bit more about. Yeah, in continual learning, there are a couple other recent papers that have come out in the last couple of years and they're not providing task ID and the model actually needs to infer the task ID as it does some sort of modulation or whatever their technique is. So we thought that makes the problem a bit more challenging, a bit more interesting. So since we are working on continual learning and comparing to some of these other methods, let's also try to infer what the task should be. So if I hear this correctly, it's very much inspired by the environment itself, like what the problem is supposed to be. Because if I see something like this, I always have the vague suspicion that people try something and it didn't work and it's like, well, let's try something else. But there's also, I mean, I don't want to infer that. So it's always good to hear like, okay, this really came about through the environment. And I mean, it would be equally cool if it was the other thing. But I'm just always interested to hear so I can adjust my priors. What I think is just to add really quick, sorry, just to add really quickly, I think in the reinforcement learning setup as well, because the state space is shared across all the tasks, because essentially, it's hard to infer from the states, what task you might be doing if you weren't given such an ID. And the only information you would have is the reward signal. And that might not be enough to infer what the task is. So giving a task ID is part of the solution. Given that it's at the end, right? Yeah. It's like, you do something and then you get a reward and then you find out what task you just did. Okay, I agree with you. That's really not helpful at all. Also I think one thing to add here is that we did try a couple, so I think this is something you pointed out in your intro where the task IDs that we're using are one-on-encoded, right? At least for multitask RL. And that means that all these tasks are entirely orthogonal to each other. And it really doesn't reflect how similar one task is to another. And it really doesn't also reflect how different one task might be from another. So one thing that we were experimenting with, I think we mentioned briefly in the paper is that we tried having an embedding layer that effectively embeds this one-hot encode into some other higher dimensional representation and using this instead of that one-hot encode as a context. And I think what we eventually found was that using the embedding or not using the embedding produced fairly similar results. So we just decided to remove it for simplicity's sake. But one thing to note is that using the embedding allows you to represent contexts, I think, that are a little bit more nuanced in the sense that the embedding, since it's trained via end-to-end backprop, any task that is similar to another task would have a shared representation in that higher dimensional embedding. And ones that are really separate from each other would likewise correspond to huge distances apart in that higher dimensional space. But the one-hot encode is entirely orthogonal from each task, but it still worked out pretty well compared to the embedding. And if it gets more complicated, I think you could put entire sub-neural networks, instead of even that embedding layer, you could have non-linearities inferring more complicated task embedding or task relations. It is interesting though with respect to the context itself, to learn these things, all of this through backprop. And my question, I think I brought this up, is would this be like a candidate for maybe unsupervised pre-training that you sort of maybe collect episodes or something in your multitask RL and then just sort of decide based on this, you know, how do we structure our dendritic segments in order to recognize the context, maybe some sort of contrastive objective or anything like, is this something that came, I just blurt these things out when I do the reviews, right? I never know if they're entirely stupid or if people have thought about it or discarded it. Is that something that is a candidate? I don't think it's something that we considered. But an interesting thing to note is that if we did use this for some kind of unsupervised pre-training tactic, is that when you're actually fine-tuning the network, your context vectors are different. So that's something I think that would be the most important nuance to investigate. I personally don't know how well that would work if we trained on a set of contexts that are different during the unsupervised portion and then use a totally different set of contexts during the fine-tuning procedure. I would imagine that doesn't work well. So yeah. To add on to that, I think, yeah, kind of like when I heard you say that in your review, it was quite interesting. I think from the perspective of reinforcement learning at a high level, I don't know if this will work out, but it would be quite cool to see if you can train these dendritic segments to either produce... If you can train them to recognize different contexts and maybe guide exploration in different ways based on the context in an unsupervised manner and maybe do different things in different contexts as an exploration strategy, I think that'd be super cool. Again, I think the challenge there would be to come up with a clever way of generating contexts in an unsupervised way. So I think that would be an interesting area of investigation. It's still like, how do you come up with context signals in an unsupervised manner? A contrastive approach might be cool there. And given these contexts, how do you train these active dendrites to modulate neurons to do what you want it to do? And I think thinking about that in the lens of exploration in RL could be quite interesting. Yeah. You could sort of even prepare for contexts that you hadn't considered before, maybe new instructions in a familiar environment or something like this. You have this notion of prototyping to recognize the context, which I found very interesting because it's kind of like an unsupervised online way even, as the data streams in, you create these new prototypes and so on. And sure, there are some hyperparameters, but I think my main concern is that just taking the average of the samples as they come in right here, it's going to work for something very simple, like permuted MNIST or so. But this gets to its limits very quickly, right? If I think about ImageNet classification or so, it is quite limited. How can this idea be extended to, let's say, arbitrary complexity? Like, what would I have to do with this online prototyping approach to make it usable for more complex problems? Hey, look, I think you're absolutely right that this technique only works for something like permuted MNIST, where you get really good task separation through just averaging the examples from a single task. And that's why it works so well here, right? We actually evaluated how well this clustering procedure works, and it works pretty well. It's not misclassifying things when it's clustering the prototypes. But if we want something that's a bit more general and can apply to other domains, like ImageNet, as you mentioned, I think something along the lines of self-supervised learning might help there. That way, you're trying to build a context vector that is going to provide you sufficiently good task separation, and it's not as simple as just averaging. Does that get at your question? Yeah, no, absolutely. And I think also in meta-learning literature, there are prototyping methods that maybe process the raw input into an embedding space and then do clustering similar to what we're doing there. So I think that would be a quite simple approach that is similar in flavor to this one, but kind of embeds the raw input, like an ImageNet input, into some better clusterable space. Another thing I noticed, and this is a minor thing, but here you feed the context signal into both of your layers. And in the experiment before here, you draw this very accurately. You feed the context signal into only one of the layers, so it doesn't go in here. Is there a particular reason behind the choice of this? Yeah, so there's a bit of background regarding this. I want to say first that the continual learning and reinforcement learning projects started out as separate areas within Numenta. And the goal for this was really to see if the same principles of the same model could work equally in both of these areas. So while we did modulate both the layers in continual learning, the intuition for not doing so in reinforcement learning was a bit different. It was that the first layer should contain all the shared information the model needs, and you could really do this without activating any specific sub-networks, and that the second layer would then activate the context-dependent sub-networks for each task. But you're absolutely right that we could have tried doing in-depth experiments where we modulated both layers for the RL setup. I think we started doing that at the beginning of this project, but we found it worked reasonably well. But because of the time and computing constraints of running each of these RL experiments, we decided to stick with the original plan and really pick a few key experiments and key architectures to run, and really leave the ablations for the continual learning experiments, which are really significantly faster to run. But you are absolutely right, though. We just went off of our intuition on this one. It's just my reviewer too popping up and be like, hey! But it's good. It's even interesting to see that this is kind of a convergence of projects. Could you tell us a little bit more about just the research process? You already talked about how this came to be, but the process of researching this, it's kind of a new thing, right? You propose a new architecture. The tasks are, let's say, not that mainstream. People work on them, but they're not super mainstream. Was it smooth sailing from beginning to end, like stepwise improvement? Or was there points that just didn't work at all for a long time? Or are there entire avenues that you discarded and didn't end up working out? Could you let other people... I don't know what you can or want to disclose, but it's always interesting to hear what also didn't work out during a project. I can start off. When we first tried implementing some of these ideas behind dendrites, you noticed that we talk about this, that we're picking the maximum dendritic activation and we're using that to modulate. But actually, it was through the process of trial and error that we realized that we were just working on an initial toy task. We weren't working on continual learning back then. We found that, hey, we actually can't turn things off. We can only turn them on because you are picking the maximum value, right? So how do you get something that's super sparse? So we actually want to turn things off. So we're like, oh, okay, let's go back and let's actually not just pick the maximum, but pick the maximum and keep the sign. So if something's really negative, we're picking that. And so there's a whole appendix section and that's actually the detail of how... That's in the details of how we're actually implementing this. So through a bit of trial and error. And then also with going back to the prototype, for a while we were thinking, well, how can we get something that really provides sufficient task differentiation? So we tried a bunch of different things. Just like Avi mentioned, he had a linear embedding, which was created from his context. We also had one for continual learning, but that didn't really work too well either. And we ended up settling, converging on something that's really dumb and simple for permutativeness that ended up working out. Yeah. There's actually, just based off of what Karan was saying, if you go to figure 11, I think you had some points there as well. It's a visualization, if I remember correctly. Yeah, this one. 11. Yeah. So if you notice, we use the exact same gating technique for both continual learning and multitask reinforcement learning. And that's the absolute max gating. So you're picking not only the absolute max, but you're retaining the sign. And what you'll notice is that the initial intuition for doing this was that, as Karan just said, is you want to give each neuron the ability to either turn on or turn off. And it's very interesting because if you look at the results in multitask RL, you can see that for neuron B at least, you see some negative activations, those red squares that you see. So that's effectively the neuron being told to turn off. It's the exact opposite of a strongly positive activation. I think something that's very interesting to see is, at least for the two neurons that we've showed for continual learning on the right-hand side, you don't really see that happening. It's either the neuron doesn't receive high magnitudes of activation or it receives really high magnitudes, but it's all positive. So something interesting to note that we were, even in the multitask RL part, we were working trying to understand would max gating work better than absolute max gating in the sense that do we want to discard the sign or keep the sign? In the beginning, there was a lot of trial and error process. In multitask RL too, we had a good amount of time spent on understanding what the right sparsity levels were to apply for the weight sparsity and the feed forward layers. What we saw, I think, is also pretty intuitive. If you really increase your sparsity level to a really high sparsity, there's just not enough information in the network to keep training, and your accuracy plummets. But something that's interesting to note is that there's always a sweet spot for sparsity. Once you reach there, that's when the accuracy is the best. How do you debug these things? What is your main method? Is your main method mainly setting a parameter and then running things? Are there good ways to peek inside and what's happening? What are things that you look at to debug something like this? Like, oh, we are not sparse enough or we're too sparse or we don't turn off neurons or something like this. I think diagrams like this, which you have on your screen, are a perfect example, visualizations of how the dendrites are behaving. I think there was at one point early on, here you have in both cases after learning that different segments are responding to different tasks contexts. But there are cases early on where these diagrams looked exactly like just really just horizontal bars. So you have the same segment that's just winning all the time. So we realized, okay, well, this is not right. We don't want the same segment to always win. So that helps in identifying, okay, this is why the network is failing. So you would look at these things even during your research process. It's not just something that you made after the fact just to demonstrate to the readers. Yeah. Oh, yeah. This was a very helpful tool for debugging. Cool. I mean, that's really interesting to hear. A lot of the architecture decisions that were made in continual learning were used in multitask RL simply because I think each multitask experiment took 25 hours to run plus easily. So it was really hard to change a parameter, observe how the results and visualizations looked, and then sort of edit from there on. So a lot of the intuitions that we got in RL came from current continual learning experiments. So that was nice. Did you ever compare these things to, well, it's not too easy to compare, but sort of a baseline because there is the danger with these things that you kind of interpret. I think I said, well, couldn't this be just like the difference between the top and the bottom just be one is at initialization and one is trained and maybe has not much to do with sparsity? Did you ever compare this to something that isn't explicitly sparse or anything like this? Is there something you can say as a reference point? Yeah. So there's two things to note there. The first is that at least for this visualization, the activations are normalized with respect to when they were trained. So I think you mentioned this in your intro as well. You said that could it potentially be that you have really high activations in the beginning and the area that you've circled there in purple, it just sort of gets dimmed down. And I think the important thing to note is they're all normalized. So the range of values between the highest activated neurons are much higher than the lowest activated neurons after training than before training. But to address the second point, I think that's regarding figure 10, if you scroll up. And that was why don't we have like a baseline for this? Is it really that the active dendrites networks that are creating these hyper sparse sub networks? And to that, you're absolutely right. We should have had a nice diagram here that also showed how this would look in a baseline MLP. You're absolutely right. That's something that we could definitely include. I mean, I totally believe you that it's like very sparse. It's just that it's not it's not obvious from a diagram like this. Like what, you know, what what should I expect? I Yeah, but cool. Yeah, there is one one other thing in that big, by the way, like, I have mad respect for you for including the graph on the right. Like, like mad respect, like, 90% plus of researchers where they try something like this specifically because no one would notice if you leave this away, right? No one no one comes to you and says, Well, okay, maybe someone comes to you, but no, no one would seriously miss adding the SI to both of these things. And you, you know, at the left, you beat them very clearly. So, you know, huge respect for for including that that is, it's, it's, I think, to be commended and to be highlighted. I think, you know, when we present a new architecture like this, you know, we really want to show the community that, hey, we can we can do things like continual learning with our more biologically inspired ideas. And it's competitive with what's already out there, right? So even if we're not beating the state of the art, I think that that's perfectly fine. Even though you know, nowadays, a lot of machine learning has turned into this competition of, you know, getting getting the best numbers. And if you don't have the best numbers, apparently that that means you you won't be able to publish anymore. So yeah, to add on to that, I think the purpose of this paper is really something I said that we all said in the beginning, and now it's we really want to show a proof of concept for this completely novel architecture, where the goal is really not to get state of the art, I can see on either of these benchmarks. It's really about the promise of something new, something I think that deep learning is has been missing for the past, what 10 years or so. So yeah, it's exciting. And the last thing maybe we can get into is this comparison to other to other networks, because you you you very clearly address this in like a paragraph. And I think, wait, I have like even a transformer diagram somewhere, you clearly address this in a paragraph saying, like, isn't this just equivalent to to like a bigger network? And I try to myself also to come up with, you know, is there some way I could do the multiplication in like an MLP? And I'm fairly convinced there isn't. But there is a connection clearly to like LSTM which do modulate things with like forget gates and so on. They even have sigmoids, right? So they can they can module model this, this on or off, and also sparsity to an extent. And I also think that a transformer could conceivably like a two layer transformer could conceivably model the interaction right here. Did you explore at all, like the the inter like the connections of sort of this active dendrites framework to other models? Is there something you can say about that? I definitely think that these are great observations, by the way, that the kind of relationship between attention and transformers and like the gating and LSTMs and GRUs, there's definitely a relationship between those mechanisms and what we're doing here. I think in our research process, we definitely thought a lot about how this gating mechanism could be related to like things like multi headed attention, where basically you're doing a similar thing where you're matching keys and queries as vectors with an inner product and then using that as a way to see what parts of a sequence, for example, to weight when you're considering a certain position. I think the key difference in terms of I think the similarity is that for in the specific instance of attention, you are using learned weights to match a given input. So for example, in our active dendrites, you're matching the context with the set of dendritic segments and in attention, you're matching like the query vector with a set of keys. I think that the key difference is that the purpose for which it's done here in active dendrites, you're looking at a specific neuron and you're saying, okay, given the context, is this neuron relevant? In transformers, you're saying, okay, here's a position. What context around me in terms of the sentence, for example, is relevant for me? And how can I weight certain aspects of it? So I think it's a little bit like flipped in how an interpretation of the focus. Kind of shifting to the LSTM aspect, I think as a mechanism, it's quite similar in that the LSTM is actually like turn off or turn on certain units themselves to carry forward in time. I think, yeah, exactly. That's what's done here. I think the difference is now like focus more on the sparsity aspect of it. In LSTMs, you're doing like a weighted sum between what's in the past and what's current and saying, okay, let's pass this forward. And there's no aspect of like using this to enforce a level of sparsity. Here, we're saying, okay, let's turn off certain things and do that in order to remain sparse and pass forward this information. So there's definitely a relationship there. I think the interpretation is similar, but a little bit different. And I think in all of these things, again, to highlight, LSTMs and transformers, they're all trained, let's say, with back prop, and all the parameters are trained. So still, you'd run into the same problems where if you do discontinue learning, tasks would interfere with each other, no matter how much they can implement the multiplication. So that's definitely a difference. So in your outlook section, I haven't mentioned this in the video, but you discuss sort of what to do next. And you mentioned a lot of like, oh, yeah, we want to investigate maybe the combination of RL and continual learning and so on. Is there something that's here? Is there? Yeah, you said, you mentioned neuroscience a little bit, what would be sort of the next big things from neuroscience to include in deep learning architectures that aren't yet really done by other people? Like, is there something where, you know, you could say, well, if we had that, that's not really in our deep networks yet. But if we had that, that would be like, amazing. I think this is a very small point. But the dendrites that we're sort of modeling right now are, they can be considered the basal dendrites. I think you went over this briefly in your intro. And the basal dendrites are responsible for receiving this context and depolarizing the main cell to either fire or not, if that context was recognized. Something that we haven't looked into, which could be potentially interesting is modeling apical dendrites. And the apical dendrites receive feedback from other cells that also biases the soma to fire or not. I think that could be a potentially interesting way to also gate each individual neuron. I think standard deep learning doesn't do any of this anyway. They only consider the proximal dendrites, which is mimicked by the simple linear weighted sum to determine if the neuron is fired. But if we can gather all this other neuroscience background from all the other kinds of dendrites too, like apical dendrites, it could be a very potentially interesting architecture, like a very powerful one for dynamic scenarios. The issue of top down feedback or lateral inhibition or anything like this, a lot of people talk about it, but I haven't yet seen anyone successfully bring it into a deep network and actually do something useful with it. Definitely think beyond dendrites, just mechanisms like this would be super helpful. I think another aspect, which is a little bit quite different from what Avi just said, that would be quite interesting is the local learning rule aspects that are present in biological neurons and how they might relate to unsupervised learning in conditional machine learning. I think a lot of the unsupervised learning objectives are addendums to the loss function that we think might be useful and it just flows through the network. I might be wrong, but I don't think there's a lot of research until figuring out which parts of the network could focus on certain things in an unsupervised way, which might be better done in biological networks. I think thinking about that and getting inspiration to see what local learning rules in an unsupervised way could improve performance in modern deep learning would be super cool. Cool. Do you have anything to add, anything people should know or that we haven't talked about yet about the paper? People can get started with your code, which is online. I've seen that, which is very cool. Anything you want to get out there to the viewers? The take home message from this is what we want to be is that the brain is able to do a lot of different things. It's using different neural circuits to do it, but neural networks, as they've been designed decades ago, they're really just optimizing for one thing. They're great function approximators, but you don't just want to approximate one function. You want to be able to approximate multiple functions. We're trying to show that, hey, there are ways where we can get neural networks to actually have different sub-networks, different neural circuits that are able to be different function approximators. If we can do that, then neural networks will be able to operate in more dynamic, changing scenarios. I think that's really exciting because the world is constantly changing, but a lot of the applications for deep learning right now are the environments that they operate in, are static. If we can get to that, then that's great. Cool. Well, Akash, Karen, Avi, thank you very much for being here today. This was great fun and I learned a lot. Yeah, thanks, Yannick. Now you're influencing my fashion. Nice. I'll join the show. Thanks so much for being here. Yeah, I hope you continue this because it's really cool and I think we're missing it in deep learning. Thanks, Yannick. That was a lot of fun. It was a pleasure. Thanks for having us. Thanks for having me.
 Hello, this is an interview with the authors of the paper on active dendrites. Now, if
 you haven't seen it, I've made a comprehensive paper review video on this paper and I released
 that yesterday. If you watch this video as it comes out, which obviously you do today,
 I'm going to interview the authors and we've all seen my review. So we'll be able to directly
 dive in. So if you haven't seen the review yet, and you want to know what's in the paper,
 maybe that is a good place to start. The authors here were really helpful and really informative
 answering all of my questions and concerns that I had and even bringing up some new interesting
 insights. So I hope you learn something from this interview or at least that it entertains
 you. And if you have any comments, please let me know in the comments below the video.
 I'll see you around. Bye bye. Hey there, today's sponsor is the course on introduction to graph
 neural networks. This is a course by my friend Zach Jost, who is an expert in graph neural
 networks, and also runs the welcome AI overlords YouTube channel has a very interesting blog
 and does many other cool things. He's packed all his knowledge of graph neural networks
 into one course that will educate you on both the theoretical and hands on practical aspect
 on graph neural networks. Graph neural networks are really important. They're definitely one
 of the most interesting areas in deep learning right now they're on the upswing, they model
 data that has an underlying structure that is connected that is not really well fit for
 any of the classic formats like tables or images. They've also powered a lot of recent
 advances in scientific breakthroughs, such as alpha fold protein structure predictions,
 or better traffic predictions. So if you're interested in graph neural network, I'll definitely
 recommend you check out that course. If you use my link, you'll get a 15% discount on
 the course enrollment is open right now and lasts until April 1 or until spaces run out.
 The course is a six weeks course. It's cohort based, you'll get access to a community to
 discord community of other students, and you'll get all the materials and hands on experience.
 All right, let's get into the video now. See ya.
 Hi everyone, today I'm here with the three joint first authors of the paper on active
 dendrites, Abhi, Karan and Akash. And I'm very, very happy to have you all here. This paper
 covers many areas, it covers biology, it covers neural networks, it covers kind of different
 architectures of stuff. It's very cool that you all sort of are here and are able to sort
 of answer my questions. Welcome, all of you.
 Yeah, thanks, Janek. Thanks for having us.
 Thanks for having us.
 It's very interesting paper. So I saw this paper and I was intrigued because it's not
 often that a lot of people say they do biologically inspired things. But it's not often that someone
 really goes and says, look, you know, here's what's missing, let's build it in. And then
 it actually leads to something that works. And that is, you know, the hypothesis in your
 paper, the hypothesis you pose on what should happen are actually confirmed at the end.
 And this is, I think, a very good story arc for a paper and a really nice thing to write
 up. So is this, how did this come to be? How did you get the idea of bringing these very
 two distant, not too distant, but these two distant fields together of sort of neurobiology
 and deep learning?
 Well, at Numenta, we're interested, one of the things we're interested in is in continual
 learning and learning multiple tasks, more generally speaking. And so, you know, we're
 looking at, but a lot of neural networks and deep learning today focuses on trying to solve
 a single task. So we said, well, you know, how is biology enabling the ability to solve
 multiple things in sequence or, you know, at the same time, learning different things?
 And so, you know, there's been a lot of work out there on active dendrites. And so, and
 it's not exactly clear what their role was. But a little while back, we speculated that,
 hey, they might actually be helping at the neural level to allow for continual learning.
 And so if we can build this idea into deep learning, then there might be some prospect
 there for addressing problems like continual learning and multitask learning.
 So is it fair to say that it grew out of sort of a need to solve a task?
 I think it grew out of the need to solve multiple tasks in sequence, either learning them together
 or in sequence continuously. To add on to what Karan was saying is that we believe that
 active dendrites can really aid in achieving these specialized neural circuits. And we
 can apply these ideas directly to any neural network and show some competitive performance
 on various benchmarks that involve continual learning setups. So I guess the purpose of
 this project, if you were to just summarize it very briefly, is we just want to show a
 proof of concept for a new idea that can allow deep learning to work in more dynamic environments
 and scenarios. To kind of add on to what Karan and Abhi
 said. So at a higher level, I think we were kind of examining where a lot of modern deep
 networks fail, and that's in these streaming task settings and multitask settings. And
 the kind of inspiration for our solution was directed towards biology and biological neurons,
 which is a lot of what Numentos focuses on. And I think quite nicely we found these existing
 benchmarks and existing tasks that show that typical deep learning networks fail in these
 scenarios. And we were able to build in these biologically inspired neurons to improve the
 performance in such dynamic settings by using the fact that we believe active dendrites
 in biology kind of do this kind of context dependent adaptation in multiple tasks.
 What I found interesting is that even though you targeted a little bit towards multilayer
 perceptrons, in principle, this active dendrites architecture is sort of pluggable almost anywhere.
 So you could always imagine some sort of a context dependent signal that gets routed
 in and modulates the signal that exists. So I think what I'm trying to find out is there
 are a number of things happening in this model. There is first of all the modulation itself,
 which is a relatively it's not really a known concept, at least in classical deep learning,
 we always have weighted sums, we rarely have the situation where two parts of the signal
 are multiplied together, or one modulates the other, it happens a little bit in LSTM
 and so on. The other one is the sort of recognition of a context and, you know, being context
 dependent. And then a third thing is this, this sparsity. Now, you have sort of combined
 all of them. Is there one thing that you think is specifically important? Or is it sort of
 the combination of things that is really what makes the difference? You have some ablations
 in the paper. What can you say about this?
 I think it's the combination of all these things acting together. So it's the it's
 the it's the dendrites, which are, you know, up modulating and down modulating certain
 neurons to determine which ones should become which which to determine which sub network
 should be invoked. And then it's as far as you on top of that, which is ensuring that,
 you know, a large portion of the network is essentially not performing or learning a certain
 task. And it's those two things together, which, which, which really gets at this idea
 of using specialized sub networks for different things. So I wouldn't say any any one one
 thing that stands out more than the others.
 So when we get let's get into the paper itself, you've seen my review of it, with respect
 to just framing the problem and maybe framing the architecture as such, is there do you
 think I have captured what you've tried to say? Do you think I've left something important
 out or have put emphasis on or have not put emphasis on something that you would like
 to put emphasis on when it comes to like, what the architecture is, what it does and
 how it works?
 I think your explanations for the architecture, at least we're very good. I think it does
 definitely does capture what we were trying to trying to say. And the whole point to kind
 of reiterate is that the same model with the same principles should work on completely
 separate areas. One is the multitask reinforcement learning. The other one is continual learning
 with permuted MNIST. And I think you touched upon that idea too. So yeah,
 I think that the kind of motivation that I think you in towards the beginning of your
 review, you showed you kind of compared the typical weighted linear sum neuron with the
 active dendrites neuron. And I think our motivation in coming up with this architecture was how
 can we incorporate a lot of these properties into active dendrites with having dendritic
 segments being able to either up modulate or down modulate certain neurons in a way
 that didn't completely change from normal back propagation trainable networks. So this
 architecture kind of brings in that flavor of having dendrites influence certain neurons,
 but does so in a way that mathematically allows for back propagation to train the networks
 and I think you touched on that pretty well as well.
 Do you think it's valid to sort of bring in biological concepts even though we train with
 back propagation? Because it's very evident that at least pure like correct back propagation
 isn't happening in the brain. Do you think it's still valid to bring in the concepts
 and maybe the brain is doing something like backprop? Or do you think we're sort of just
 kind of taking inspiration from biology in order to solve some of our problems?
 I think it's more so the latter. Of course, the most accurate biological neural network
 would likely not use back propagation, right? But this is one area where I think the goal
 was can we make deep learning just a little bit more plausible? And in doing so, can we
 make it a little bit more dynamic? So we're not necessarily here to remove backprop entirely
 and say that that's the best way that the dendrites in this architecture can work. Although
 certainly that is how it works in biology. The point was, can we just augment traditional
 deep neural nets to work in more dynamic scenarios?
 Now I had some criticisms with respect to just like that details of your architecture.
 For example, you always or you often choose the number of dendritic segments to match
 the number of tasks that you have, which obviously, if I was a researcher, I would do the same.
 But can you say maybe something about how this is in the brain? Like what numbers are
 we talking about? How many of these sub networks that are composed of distal dendrites? How
 many are there approximately? Do you know? Do you have an idea? And what can you say
 about how many we should build into a problem where we maybe don't know how many tasks
 we expect?
 From what I recall, probably in the order of hundreds or thousands of individual dendrite
 segments for each individual neuron, actually, it might even be more than that. The actual
 numbers escape me. But regarding what you said earlier about having the number of tasks
 be equal to the number of segments here, we found that actually, even though in a lot
 of the experiments we report here, we do set the number of dendrites to the number of tasks.
 We found that we actually don't need to have that many. And we actually have further studies
 which show that we can actually keep the architecture fixed and increase the number of tasks we're
 doing. I'm talking about continual learning here because for multitask, we're focused
 on 10 specifically. We can increase the number of tasks and the performance actually doesn't
 change by much. So that shows that as we're increasing the number of dendrite segments,
 we actually end up overparameterizing the network quite a bit, which we don't need to
 do.
 Yeah. So this is the plot on the left right here. You just increase the number of dendritic
 segments and the top line is learning 10 tasks. And it doesn't get noticeably worse, which
 I find to be a very cool property. I don't want to have to set the parameter very specifically.
 I can just set it too high and it doesn't hurt, which is cool. Which leads me to the
 plot on the right where you discuss the sparsity. I'm going to guess that's the sparsity parameter.
 So that's the thing that ultimately controls k. And I find it peculiar, not that there
 is an optimal setting, which I would expect because that I can't set high that I have
 to set between 0 and 1. So there's going to be some optimum in between. But there's this
 two bump thing going on. So what's going on there? Why is it like really good at lows,
 like high sparsity, and then there's like this plateau, and then it just flat like crashes
 down.
 I think there in the beginning, you know, if you have if you have too much. So yeah,
 I always think in terms of sparsity, so I'm converting from density to sparsity. So if
 you have if it's too sparse, right, there's not enough signal going through. And that's
 why, you know, as you as you increase the amount of signal that you're allowing through
 as you're increasing the capacity of your representation, then you're going to get you're
 going to get an increase in performance. But then if you have if you're using up too many
 units to to create that, to create that representation, then you're going to get more interference,
 right. And as you have more interference, you're going to you're going to you're going
 to forget more and more network parameters are overwritten as you move on to subsequent
 tasks. And so you get a drop in accuracy. And towards the end, so you know, you notice
 that it does fall drastically. Honestly, I haven't thought too much about why that happens.
 Although it is it is a pretty, pretty monotonic fall, even though I guess in that in that
 upper curve, there's a slight bump with that could just be due to seeding or something
 like that. But yeah,
 Yeah, I was more referring to like the plateau itself, right? There's there's this plateau
 kind of, and I I know, I know that there could be almost like two two modes of using the
 sparsity in one mode, I have entire sub networks that do the job. And in the other mode, I
 have like a shared network. Yet I have like separate things that just kind of like track,
 track which task I'm on, which would sort of correspond to what the baseline is doing,
 right? When people say, well, the baseline has access to the task to it can just allocate
 some units. No, it's maybe not a perfect analogy. But I was just wondering, it was just interesting
 to see that there's this kind of this type of plateau.
 Yeah, that's that's something I guess, we haven't gone too deep into. But this might,
 this might just be a property of sparse representations and how and how much overlap there is as you
 as you as you increase the sparsity level, it could just be something to do with that.
 So in your paper, you make really, which I appreciate you make really sure that you sort
 of always have the same amount of let's say trainable parameters in your architectures.
 And you show that by arranging them correctly, you can you can achieve a better result. You
 always use this name of non zero parameters, right? Is there like, is there a difference?
 Are there large swaths of zero parameters in one or the one of these architectures?
 Yeah, so this is something that we control for. In the beginning, this is why we mentioned
 the idea of weight sparsity. So in the beginning, when when we're actually creating the architecture
 from scratch, we decide that some layers have an X percent sparsity level applied to it.
 And what that really means is that X percent of the parameters are zero throughout the
 entire part of training, and even towards the end. So that's why we express everything
 in non zero parameters. So the MLPs, for instance, at least in reinforcement learning, are trained
 with no weight sparsity. So it's completely dense. There are no zeros anywhere in the
 in the layers.
 And then the your your architecture, you sort of modulate the amount of sparsity. And that
 is on top of modulating the K parameter of the K winner takes all layers.
 Yeah, there's two aspects to the sparsity. So one is activation sparsity, which is like,
 at a hidden, like when you have a hidden state vector, how many neurons remain non zero after
 the activation is applied, which is a K winner activation. And then the second aspect of
 sparsity is weight sparsity, which is how connected are subsequent layers in the network.
 So if a lot of the units in the weight matrix are zero, then this models the fact that subsequent
 layers in the network are not very connected, they're sparsely connected.
 To I guess answer your question again on that is, it's not something with weight sparsity,
 at least it's something that it's not something we modulate, it's fixed. It's a fixed percentage
 that we find. And this can either be done through fine tuning, or just Yeah, just just
 experimentation.
 Okay, because I think yeah, I might I might have just over read that. But but I recall
 that in the introduction, you say, you know, both the weights and the both the weights
 and the the activations are sparse, but then sort of the I think the winner takes all really
 focuses on the on the activations itself. Have you experimented with setting, you know,
 something else than K to a number or a percentage, setting maybe a threshold for sparsity or
 something like this, where whenever a signal is strong enough, it is let through?
 We haven't, we haven't done anything like that. But we could do that. And you know,
 there is a chance that it could work out pretty well if we if we have a fixed threshold. But
 one potential downside there is that, you know, if you have if you have too many signals
 that cross the threshold, too many units whose activation crosses the threshold, you're going
 to get more interference when you train. Or if you have not not enough neurons whose activation
 crosses the threshold, you're going to get you're going to get that phenomenon which
 you're showing on the screen right now on the left side where you have a drop in accuracy
 because your representations don't have enough capacity. So that's why we we opted to go
 for a fixed value of K. But even if you know, we didn't have even if we did have a threshold,
 I think one of your critiques were here, you know, now we have another hyper parameter
 K that we're choosing. In the other case, I mean, we'd have to with our hyper parameter
 would just be the threshold value there, right? Obviously, yeah. Yeah. So to me, this this
 continual learning setup is very cool. And you can generate data very easily using this
 permuted MNIST. But there is a bit of an issue that I have. And that is that if I use permuted
 MNIST, there is another thing there's like all the tasks are like the same difficulty,
 right? They're essentially the same task. It's just permuted. So I need to learn. Yes,
 I need to learn like a different function. So this would be the permutation identity.
 And then the pixels are permuted somehow, right? So all the tasks are kind of the same,
 right? Which warrants a static network architecture and every context vector is kind of the same
 length, right? And all the dendrites, they can they can sort of specialize in each of
 their little task recognition. What would change here? Or is it is this a drastic requirement
 to your architecture? Or do you think if many of the tasks were wildly different from each
 other, and you have this a little bit in the robot example, so what can you tell about
 when tasks are very different in their difficulty, maybe in their amount of training data, like
 how do these things influence an architecture that's targeted towards continual learning?
 In our case, I think there might actually be similarities between different tasks. And
 so like, you know, for example, in this case, in permuted MNIST, right, there's a certain
 certain pixels are more likely to be white. And certain pixels are more likely to be black,
 depending on the permutation. So maybe, you know, two different permutations could have
 more overlap in terms of which pixels are white, which pixels are black, or they could
 be totally separate. And if they're more, if they're more similar, if the permutations
 are more similar, then we could expect that the the sub networks that are selected by
 the dendrites will probably have more are likely to overlap more in which neurons become
 active, since there's a lot of there's probably a lot of similar computation going on. But
 of course, you know, in that case, difficulty doesn't really change at all.
 I think to kind of add on to that, I think a lot of it depends on the quality of the
 context signal. Because ultimately, that's the part of the network that indicates to
 the active dendrites, what kind of task you're solving, how similar is it to previous tasks
 you might have seen and things like that. So I think that in this in this permuted MNIST
 case, the way we're computing the context does allow for this property that Karen just
 mentioned, where if there's some overlap in the input space, then the context signal for
 that will demonstrate this input and perhaps allow for overlapping subnetworks to emerge.
 Whereas if you have like wildly different tasks, which is something we see more in the
 robotics environment, then these context signals can like differ more and indicate that the
 sub networks must be like must not overlap. I think it would be really interesting. And
 we've talked about this before to try a similar setup in a continual like robotics learning
 case where you have a streaming set of like robotics tasks. And I think that would probably
 be a super interesting study to do. And something that hopefully we will try at some point in
 the future.
 So I had I had some observations with respect to your experimental setup. It's very cool
 that you do two different things. But there are also noticeable differences on how you
 implement the two different tasks, right in the first task, you give the task ID directly.
 In the second tasks, you do this, this this prototyping approach, which is a more advanced
 approach. Can you tell a little bit about how is there like a reason why in because
 I could also imagine you just give me the task ID in the second task, or I do the prototyping
 in the first task. Is there like a research process reason? Like did you find that some
 things did work or didn't work? Or how did this come about? That all of a sudden, we're
 introduced in the new task, we're introduced to this new way of detecting the context.
 I think in the context of the multi agent, like, sorry, the multitask reinforcement setup,
 like the environment setup itself gives the task ID. And I think that the concept of multitask
 learning itself is more focused on if you have different tasks, which may conflict with
 one another, in terms of the types of behavior you have to do, or the types of predictions,
 can how can you mathematically still optimize your like joint objective function without
 and still be able to perform well on all the tasks. And the problem shifts not so much
 from trying to infer what tasks you're doing, to more, you know what tasks you're doing,
 and you want to try to do all of them. How can we like optimize this joint objective.
 This is kind of the way we use this one hot task encoding is in line with passwords that
 deal with multitask learning and multitask reinforcement learning, where you have this
 like one hot task encoding that is provided. I do agree that like the one hot encoding
 is quite convenient and a little bit arbitrary, you can probably use like a denser representation
 for each task or try to infer it. But I think for the purposes of our experiments, this
 one hot encoding seemed simple as it was environment provided and kind of like the point of the
 multitask setup was to again like try to show that this network architecture prevents from
 like conflicting updates across tasks and avoids this like interfering updates from
 occurring. I think for continual learning, the kind of the kind of setup of the problem
 itself is a little bit bigger and that you have to you're not always provided with the
 task IDs and you have to infer this on the fly, which again, I think Karn can talk a
 little bit more about.
 Yeah, in continual learning, there are a couple other recent papers that have come out in
 the last couple of years and they're not providing task ID and the model actually needs to infer
 the task ID as it does some sort of modulation or whatever their technique is. So we thought
 that makes the problem a bit more challenging, a bit more interesting. So since we are working
 on continual learning and comparing to some of these other methods, let's also try to
 infer what the task should be.
 So if I hear this correctly, it's very much inspired by the environment itself, like what
 the problem is supposed to be. Because if I see something like this, I always have the
 vague suspicion that people try something and it didn't work and it's like, well, let's
 try something else. But there's also, I mean, I don't want to infer that. So it's always
 good to hear like, okay, this really came about through the environment. And I mean,
 it would be equally cool if it was the other thing. But I'm just always interested to hear
 so I can adjust my priors.
 What I think is just to add really quick, sorry, just to add really quickly, I think
 in the reinforcement learning setup as well, because the state space is shared across all
 the tasks, because essentially, it's hard to infer from the states, what task you might
 be doing if you weren't given such an ID. And the only information you would have is
 the reward signal. And that might not be enough to infer what the task is. So giving a task
 ID is part of the solution.
 Given that it's at the end, right?
 Yeah.
 It's like, you do something and then you get a reward and then you find out what task you
 just did. Okay, I agree with you. That's really not helpful at all.
 Also I think one thing to add here is that we did try a couple, so I think this is something
 you pointed out in your intro where the task IDs that we're using are one-on-encoded, right?
 At least for multitask RL. And that means that all these tasks are entirely orthogonal
 to each other. And it really doesn't reflect how similar one task is to another. And it
 really doesn't also reflect how different one task might be from another. So one thing
 that we were experimenting with, I think we mentioned briefly in the paper is that we
 tried having an embedding layer that effectively embeds this one-hot encode into some other
 higher dimensional representation and using this instead of that one-hot encode as a context.
 And I think what we eventually found was that using the embedding or not using the embedding
 produced fairly similar results. So we just decided to remove it for simplicity's sake.
 But one thing to note is that using the embedding allows you to represent contexts, I think,
 that are a little bit more nuanced in the sense that the embedding, since it's trained
 via end-to-end backprop, any task that is similar to another task would have a shared
 representation in that higher dimensional embedding. And ones that are really separate
 from each other would likewise correspond to huge distances apart in that higher dimensional
 space. But the one-hot encode is entirely orthogonal from each task, but it still worked
 out pretty well compared to the embedding.
 And if it gets more complicated, I think you could put entire sub-neural networks, instead
 of even that embedding layer, you could have non-linearities inferring more complicated
 task embedding or task relations. It is interesting though with respect to the context itself,
 to learn these things, all of this through backprop. And my question, I think I brought
 this up, is would this be like a candidate for maybe unsupervised pre-training that you
 sort of maybe collect episodes or something in your multitask RL and then just sort of
 decide based on this, you know, how do we structure our dendritic segments in order
 to recognize the context, maybe some sort of contrastive objective or anything like,
 is this something that came, I just blurt these things out when I do the reviews, right?
 I never know if they're entirely stupid or if people have thought about it or discarded
 it. Is that something that is a candidate?
 I don't think it's something that we considered. But an interesting thing to note is that if
 we did use this for some kind of unsupervised pre-training tactic, is that when you're
 actually fine-tuning the network, your context vectors are different. So that's something
 I think that would be the most important nuance to investigate. I personally don't
 know how well that would work if we trained on a set of contexts that are different during
 the unsupervised portion and then use a totally different set of contexts during the fine-tuning
 procedure. I would imagine that doesn't work well. So yeah.
 To add on to that, I think, yeah, kind of like when I heard you say that in your review,
 it was quite interesting. I think from the perspective of reinforcement learning at a
 high level, I don't know if this will work out, but it would be quite cool to see if
 you can train these dendritic segments to either produce... If you can train them to
 recognize different contexts and maybe guide exploration in different ways based on the
 context in an unsupervised manner and maybe do different things in different contexts
 as an exploration strategy, I think that'd be super cool. Again, I think the challenge
 there would be to come up with a clever way of generating contexts in an unsupervised
 way. So I think that would be an interesting area of investigation. It's still like, how
 do you come up with context signals in an unsupervised manner? A contrastive approach
 might be cool there. And given these contexts, how do you train these active dendrites to
 modulate neurons to do what you want it to do? And I think thinking about that in the
 lens of exploration in RL could be quite interesting.
 Yeah. You could sort of even prepare for contexts that you hadn't considered before, maybe new
 instructions in a familiar environment or something like this. You have this notion
 of prototyping to recognize the context, which I found very interesting because it's kind
 of like an unsupervised online way even, as the data streams in, you create these new
 prototypes and so on. And sure, there are some hyperparameters, but I think my main
 concern is that just taking the average of the samples as they come in right here, it's
 going to work for something very simple, like permuted MNIST or so. But this gets to its
 limits very quickly, right? If I think about ImageNet classification or so, it is quite
 limited. How can this idea be extended to, let's say, arbitrary complexity? Like, what
 would I have to do with this online prototyping approach to make it usable for more complex
 problems?
 Hey, look, I think you're absolutely right that this technique only works for something
 like permuted MNIST, where you get really good task separation through just averaging
 the examples from a single task. And that's why it works so well here, right? We actually
 evaluated how well this clustering procedure works, and it works pretty well. It's not
 misclassifying things when it's clustering the prototypes. But if we want something that's
 a bit more general and can apply to other domains, like ImageNet, as you mentioned,
 I think something along the lines of self-supervised learning might help there. That way, you're
 trying to build a context vector that is going to provide you sufficiently good task separation,
 and it's not as simple as just averaging. Does that get at your question?
 Yeah, no, absolutely.
 And I think also in meta-learning literature, there are prototyping methods that maybe process
 the raw input into an embedding space and then do clustering similar to what we're doing
 there. So I think that would be a quite simple approach that is similar in flavor to this
 one, but kind of embeds the raw input, like an ImageNet input, into some better clusterable
 space.
 Another thing I noticed, and this is a minor thing, but here you feed the context signal
 into both of your layers. And in the experiment before here, you draw this very accurately.
 You feed the context signal into only one of the layers, so it doesn't go in here. Is
 there a particular reason behind the choice of this?
 Yeah, so there's a bit of background regarding this. I want to say first that the continual
 learning and reinforcement learning projects started out as separate areas within Numenta.
 And the goal for this was really to see if the same principles of the same model could
 work equally in both of these areas. So while we did modulate both the layers in continual
 learning, the intuition for not doing so in reinforcement learning was a bit different.
 It was that the first layer should contain all the shared information the model needs,
 and you could really do this without activating any specific sub-networks, and that the second
 layer would then activate the context-dependent sub-networks for each task. But you're absolutely
 right that we could have tried doing in-depth experiments where we modulated both layers
 for the RL setup. I think we started doing that at the beginning of this project, but
 we found it worked reasonably well. But because of the time and computing constraints of running
 each of these RL experiments, we decided to stick with the original plan and really pick
 a few key experiments and key architectures to run, and really leave the ablations for
 the continual learning experiments, which are really significantly faster to run. But
 you are absolutely right, though. We just went off of our intuition on this one.
 It's just my reviewer too popping up and be like, hey! But it's good. It's even interesting
 to see that this is kind of a convergence of projects. Could you tell us a little bit
 more about just the research process? You already talked about how this came to be,
 but the process of researching this, it's kind of a new thing, right? You propose a
 new architecture. The tasks are, let's say, not that mainstream. People work on them,
 but they're not super mainstream. Was it smooth sailing from beginning to end, like stepwise
 improvement? Or was there points that just didn't work at all for a long time? Or are
 there entire avenues that you discarded and didn't end up working out? Could you let
 other people... I don't know what you can or want to disclose, but it's always interesting
 to hear what also didn't work out during a project.
 I can start off. When we first tried implementing some of these ideas behind dendrites, you
 noticed that we talk about this, that we're picking the maximum dendritic activation and
 we're using that to modulate. But actually, it was through the process of trial and error
 that we realized that we were just working on an initial toy task. We weren't working
 on continual learning back then. We found that, hey, we actually can't turn things
 off. We can only turn them on because you are picking the maximum value, right? So how
 do you get something that's super sparse? So we actually want to turn things off. So
 we're like, oh, okay, let's go back and let's actually not just pick the maximum, but pick
 the maximum and keep the sign. So if something's really negative, we're picking that. And so
 there's a whole appendix section and that's actually the detail of how... That's in the
 details of how we're actually implementing this. So through a bit of trial and error.
 And then also with going back to the prototype, for a while we were thinking, well, how can
 we get something that really provides sufficient task differentiation? So we tried a bunch
 of different things. Just like Avi mentioned, he had a linear embedding, which was created
 from his context. We also had one for continual learning, but that didn't really work too
 well either. And we ended up settling, converging on something that's really dumb and simple
 for permutativeness that ended up working out. Yeah.
 There's actually, just based off of what Karan was saying, if you go to figure 11, I think
 you had some points there as well. It's a visualization, if I remember correctly. Yeah,
 this one. 11. Yeah. So if you notice, we use the exact same gating technique for both continual
 learning and multitask reinforcement learning. And that's the absolute max gating. So you're
 picking not only the absolute max, but you're retaining the sign. And what you'll notice
 is that the initial intuition for doing this was that, as Karan just said, is you want
 to give each neuron the ability to either turn on or turn off. And it's very interesting
 because if you look at the results in multitask RL, you can see that for neuron B at least,
 you see some negative activations, those red squares that you see. So that's effectively
 the neuron being told to turn off. It's the exact opposite of a strongly positive activation.
 I think something that's very interesting to see is, at least for the two neurons that
 we've showed for continual learning on the right-hand side, you don't really see that
 happening. It's either the neuron doesn't receive high magnitudes of activation or it
 receives really high magnitudes, but it's all positive. So something interesting to
 note that we were, even in the multitask RL part, we were working trying to understand
 would max gating work better than absolute max gating in the sense that do we want to
 discard the sign or keep the sign? In the beginning, there was a lot of trial and error
 process. In multitask RL too, we had a good amount of time spent on understanding what
 the right sparsity levels were to apply for the weight sparsity and the feed forward layers.
 What we saw, I think, is also pretty intuitive. If you really increase your sparsity level
 to a really high sparsity, there's just not enough information in the network to keep
 training, and your accuracy plummets. But something that's interesting to note is that
 there's always a sweet spot for sparsity. Once you reach there, that's when the accuracy
 is the best.
 How do you debug these things? What is your main method? Is your main method mainly setting
 a parameter and then running things? Are there good ways to peek inside and what's
 happening? What are things that you look at to debug something like this? Like, oh, we
 are not sparse enough or we're too sparse or we don't turn off neurons or something
 like this.
 I think diagrams like this, which you have on your screen, are a perfect example, visualizations
 of how the dendrites are behaving. I think there was at one point early on, here you
 have in both cases after learning that different segments are responding to different tasks
 contexts. But there are cases early on where these diagrams looked exactly like just really
 just horizontal bars. So you have the same segment that's just winning all the time.
 So we realized, okay, well, this is not right. We don't want the same segment to always win.
 So that helps in identifying, okay, this is why the network is failing.
 So you would look at these things even during your research process. It's not just something
 that you made after the fact just to demonstrate to the readers.
 Yeah. Oh, yeah. This was a very helpful tool for debugging.
 Cool. I mean, that's really interesting to hear.
 A lot of the architecture decisions that were made in continual learning were used in multitask
 RL simply because I think each multitask experiment took 25 hours to run plus easily. So it was
 really hard to change a parameter, observe how the results and visualizations looked,
 and then sort of edit from there on. So a lot of the intuitions that we got in RL came
 from current continual learning experiments. So that was nice.
 Did you ever compare these things to, well, it's not too easy to compare, but sort of
 a baseline because there is the danger with these things that you kind of interpret. I
 think I said, well, couldn't this be just like the difference between the top and the
 bottom just be one is at initialization and one is trained and maybe has not much to do
 with sparsity? Did you ever compare this to something that isn't explicitly sparse or
 anything like this? Is there something you can say as a reference point?
 Yeah. So there's two things to note there. The first is that at least for this visualization,
 the activations are normalized with respect to when they were trained. So I think you
 mentioned this in your intro as well. You said that could it potentially be that you
 have really high activations in the beginning and the area that you've circled there in
 purple, it just sort of gets dimmed down. And I think the important thing to note is
 they're all normalized. So the range of values between the highest activated neurons are
 much higher than the lowest activated neurons after training than before training. But to
 address the second point, I think that's regarding figure 10, if you scroll up. And that was
 why don't we have like a baseline for this? Is it really that the active dendrites networks
 that are creating these hyper sparse sub networks? And to that, you're absolutely right. We should
 have had a nice diagram here that also showed how this would look in a baseline MLP. You're
 absolutely right. That's something that we could definitely include.
 I mean, I totally believe you that it's like very sparse. It's just that it's not it's
 not obvious from a diagram like this. Like what, you know, what what should I expect?
 I Yeah, but cool. Yeah, there is one one other thing in that big, by the way, like, I have
 mad respect for you for including the graph on the right. Like, like mad respect, like,
 90% plus of researchers where they try something like this specifically because no one would
 notice if you leave this away, right? No one no one comes to you and says, Well, okay,
 maybe someone comes to you, but no, no one would seriously miss adding the SI to both
 of these things. And you, you know, at the left, you beat them very clearly. So, you
 know, huge respect for for including that that is, it's, it's, I think, to be commended
 and to be highlighted. I think, you know, when we present a new architecture like this,
 you know, we really want to show the community that, hey, we can we can do things like continual
 learning with our more biologically inspired ideas. And it's competitive with what's already
 out there, right? So even if we're not beating the state of the art, I think that that's
 perfectly fine. Even though you know, nowadays, a lot of machine learning has turned into
 this competition of, you know, getting getting the best numbers. And if you don't have the
 best numbers, apparently that that means you you won't be able to publish anymore. So
 yeah, to add on to that, I think the purpose of this paper is really something I said that
 we all said in the beginning, and now it's we really want to show a proof of concept
 for this completely novel architecture, where the goal is really not to get state of the
 art, I can see on either of these benchmarks. It's really about the promise of something
 new, something I think that deep learning is has been missing for the past, what 10
 years or so. So yeah, it's exciting.
 And the last thing maybe we can get into is this comparison to other to other networks,
 because you you you very clearly address this in like a paragraph. And I think, wait, I
 have like even a transformer diagram somewhere, you clearly address this in a paragraph saying,
 like, isn't this just equivalent to to like a bigger network? And I try to myself also
 to come up with, you know, is there some way I could do the multiplication in like an MLP?
 And I'm fairly convinced there isn't. But there is a connection clearly to like LSTM
 which do modulate things with like forget gates and so on. They even have sigmoids,
 right? So they can they can module model this, this on or off, and also sparsity to an extent.
 And I also think that a transformer could conceivably like a two layer transformer could
 conceivably model the interaction right here. Did you explore at all, like the the inter
 like the connections of sort of this active dendrites framework to other models? Is there
 something you can say about that?
 I definitely think that these are great observations, by the way, that the kind of relationship
 between attention and transformers and like the gating and LSTMs and GRUs, there's definitely
 a relationship between those mechanisms and what we're doing here. I think in our research
 process, we definitely thought a lot about how this gating mechanism could be related
 to like things like multi headed attention, where basically you're doing a similar thing
 where you're matching keys and queries as vectors with an inner product and then using
 that as a way to see what parts of a sequence, for example, to weight when you're considering
 a certain position. I think the key difference in terms of I think the similarity is that
 for in the specific instance of attention, you are using learned weights to match a given
 input. So for example, in our active dendrites, you're matching the context with the set of
 dendritic segments and in attention, you're matching like the query vector with a set
 of keys. I think that the key difference is that the purpose for which it's done here
 in active dendrites, you're looking at a specific neuron and you're saying, okay, given the
 context, is this neuron relevant? In transformers, you're saying, okay, here's a position. What
 context around me in terms of the sentence, for example, is relevant for me? And how can
 I weight certain aspects of it? So I think it's a little bit like flipped in how an interpretation
 of the focus. Kind of shifting to the LSTM aspect, I think as a mechanism, it's quite
 similar in that the LSTM is actually like turn off or turn on certain units themselves
 to carry forward in time. I think, yeah, exactly. That's what's done here. I think the difference
 is now like focus more on the sparsity aspect of it. In LSTMs, you're doing like a weighted
 sum between what's in the past and what's current and saying, okay, let's pass this
 forward. And there's no aspect of like using this to enforce a level of sparsity. Here,
 we're saying, okay, let's turn off certain things and do that in order to remain sparse
 and pass forward this information. So there's definitely a relationship there. I think the
 interpretation is similar, but a little bit different. And I think in all of these things,
 again, to highlight, LSTMs and transformers, they're all trained, let's say, with back
 prop, and all the parameters are trained. So still, you'd run into the same problems
 where if you do discontinue learning, tasks would interfere with each other, no matter
 how much they can implement the multiplication. So that's definitely a difference. So in your
 outlook section, I haven't mentioned this in the video, but you discuss sort of what
 to do next. And you mentioned a lot of like, oh, yeah, we want to investigate maybe the
 combination of RL and continual learning and so on. Is there something that's here? Is
 there? Yeah, you said, you mentioned neuroscience a little bit, what would be sort of the next
 big things from neuroscience to include in deep learning architectures that aren't yet
 really done by other people? Like, is there something where, you know, you could say,
 well, if we had that, that's not really in our deep networks yet. But if we had that,
 that would be like, amazing. I think this is a very small point. But the
 dendrites that we're sort of modeling right now are, they can be considered the basal
 dendrites. I think you went over this briefly in your intro. And the basal dendrites are
 responsible for receiving this context and depolarizing the main cell to either fire
 or not, if that context was recognized. Something that we haven't looked into, which could be
 potentially interesting is modeling apical dendrites. And the apical dendrites receive
 feedback from other cells that also biases the soma to fire or not. I think that could
 be a potentially interesting way to also gate each individual neuron. I think standard deep
 learning doesn't do any of this anyway. They only consider the proximal dendrites, which
 is mimicked by the simple linear weighted sum to determine if the neuron is fired. But
 if we can gather all this other neuroscience background from all the other kinds of dendrites
 too, like apical dendrites, it could be a very potentially interesting architecture,
 like a very powerful one for dynamic scenarios.
 The issue of top down feedback or lateral inhibition or anything like this, a lot of
 people talk about it, but I haven't yet seen anyone successfully bring it into a deep network
 and actually do something useful with it. Definitely think beyond dendrites, just mechanisms
 like this would be super helpful.
 I think another aspect, which is a little bit quite different from what Avi just said,
 that would be quite interesting is the local learning rule aspects that are present in
 biological neurons and how they might relate to unsupervised learning in conditional machine
 learning. I think a lot of the unsupervised learning objectives are addendums to the loss
 function that we think might be useful and it just flows through the network. I might
 be wrong, but I don't think there's a lot of research until figuring out which parts
 of the network could focus on certain things in an unsupervised way, which might be better
 done in biological networks. I think thinking about that and getting inspiration to see
 what local learning rules in an unsupervised way could improve performance in modern deep
 learning would be super cool.
 Cool. Do you have anything to add, anything people should know or that we haven't talked
 about yet about the paper? People can get started with your code, which is online. I've
 seen that, which is very cool. Anything you want to get out there to the viewers?
 The take home message from this is what we want to be is that the brain is able to do
 a lot of different things. It's using different neural circuits to do it, but neural networks,
 as they've been designed decades ago, they're really just optimizing for one thing. They're
 great function approximators, but you don't just want to approximate one function. You
 want to be able to approximate multiple functions. We're trying to show that, hey, there are
 ways where we can get neural networks to actually have different sub-networks, different neural
 circuits that are able to be different function approximators. If we can do that, then neural
 networks will be able to operate in more dynamic, changing scenarios. I think that's really
 exciting because the world is constantly changing, but a lot of the applications for deep learning
 right now are the environments that they operate in, are static. If we can get to that, then
 that's great.
 Cool. Well, Akash, Karen, Avi, thank you very much for being here today. This was great
 fun and I learned a lot.
 Yeah, thanks, Yannick. Now you're influencing my fashion.
 Nice.
 I'll join the show.
 Thanks so much for being here. Yeah, I hope you continue this because it's really cool
 and I think we're missing it in deep learning.
 Thanks, Yannick. That was a lot of fun.
 It was a pleasure.
 Thanks for having us.
 Thanks for having me.