Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper gnn transformer graph biology neurons axon dendrites plausible biologically plausible backprop backpropagation dfa feedback alignment random projections
Backpropagation is one of the central components of modern deep learning. However, it's not biologically plausible, which limits the applicability of deep learning to understand how the human brain works. Direct Feedback Alignment is a biologically plausible alternative and this paper shows that, contrary to previous research, it can be successfully applied to modern deep architectures and solve challenging tasks.

OUTLINE:
0:00 - Intro & Overview
1:40 - The Problem with Backpropagation
10:25 - Direct Feedback Alignment
21:00 - My Intuition why DFA works
31:20 - Experiments

Paper: https://arxiv.org/abs/2006.12878
Code: https://github.com/lightonai/dfa-scales-to-modern-deep-learning
Referenced Paper by Arild Nøkland: https://arxiv.org/abs/1609.01596

Abstract:
Despite being the workhorse of deep learning, the backpropagation algorithm is no panacea. It enforces sequential layer updates, thus preventing efficient parallelization of the training process. Furthermore, its biological plausibility is being challenged. Alternative schemes have been devised; yet, under the constraint of synaptic asymmetry, none have scaled to modern deep learning tasks and architectures. Here, we challenge this perspective, and study the applicability of Direct Feedback Alignment to neural view synthesis, recommender systems, geometric learning, and natural language processing. In contrast with previous studies limited to computer vision tasks, our findings show that it successfully trains a large range of state-of-the-art deep learning architectures, with performance close to fine-tuned backpropagation. At variance with common beliefs, our work supports that challenging tasks can be tackled in the absence of weight transport.

Authors: Julien Launay, Iacopo Poli, François Boniface, Florent Krzakala

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we'll look at direct feedback alignment scales to modern deep learning tasks and architectures by Julia Alonet, Jacopo Poli, François Boniface and Florian Krzakala. So this paper on a high level it replaces the back propagation algorithm in deep learning architectures with this algorithm called direct feedback alignment which is more biologically plausible. The algorithm has been around for a while but it hasn't yet been shown to be applicable to really modern big deep learning architectures and then perform on par with backprop on modern deep learning tasks. This paper as I understand it is the first one to demonstrate that it can do that. So this is very much an engineering paper, an applied paper and we're going to mostly go into direct feedback alignment as such and I don't think we're going to go too much into what the actual empirical findings are because they even though they're impressive and it's a good piece of engineering I think they can be summarized pretty much by it works not yet on par with back propagation but into a promising direction. Alright as always if you like content like this consider sharing it out and leaving a like and tell me in the comments what you like. Of course subscribe if you aren't yet that that is you know essential otherwise how are we gonna hear from me in the future? Okay let's dive in. They say despite being the workhorse of deep learning the back propagation algorithm is no panacea. It enforces sequential layer updates thus preventing efficient parallelization of the training process. Furthermore its biological plausibility is being challenged. Alternative schemes have been devised yet under the constraints of synaptic asymmetry. None have scaled to modern deep learning tasks and architectures. Here we challenge this perspective and study the applicability of direct feedback alignment to neural view synthesis, recommender systems, geometric learning and natural language processing. In contrast with previous studies limited to computer vision tasks our findings show that it successfully trains a large range of state-of-the-art deep learning architectures with performance close to fine-tuned back propagation. At variance with common beliefs our work supports that challenging tasks can be tackled in the absence of weight transport. So there's a lot to unpack in this particular abstract right here. So first of all what's the problem with back propagation? Back propagation they have they have two corals right here. First of all it's preventing efficient parallelization of the training process. So what does that mean? So in back propagation I'm pretty sure you all know it's basic back propagation but you have an input to a neural network and the neural network has a bunch of layers so the input will travel layer by layer and at the end you'll get some output and your output y hat, let's call it here what the neural network thinks the let's say it's a classifier, thinks that the class of this particular X should be. Now in the data set you have your true label and then you compare that to your output label and you can compute a loss function. Now the whole question of the back propagation algorithm is how do I need to change my layers of the neural network in order to make the loss as small as possible? And for that you can use back propagation that means you can take that loss and you can back propagate it down the layers in order to update each layer individually. So the first problem they have here with the back propagation algorithm and it's not I mean it's kind of a secondary problem but it is that is sequential. So in order to update this layer right here you need to have already back propagated to this layer and then you need to back propagate further to this and to this layer so it's a sequential task you need to back propagate down the layers again whereas what is more plausible but what would be more efficient if we could somehow update all the layers in parallel but this is a minor quarrel. The bigger one is that back propagation isn't biologically plausible. We know that in real neurons you have your dendrites, your inputs and your axon and the signal only travels in one direction. We don't know of a feedback mechanism in true neurons in the brain that would allow for information sort of to flow in the opposite direction. There is information flowing in the opposite direction but I think it's too slow and it's so it's not really it can't be there's no analogous way of back propagation. There's no nothing in the brain that would take the role of the back propagation algorithm. Specifically if each layer is characterized by a weight matrix right here what back propagation does is it uses the transpose of that weight matrix to back propagate. So these arrows to the front right here they use the weight matrices and these arrows to the back they use the transposes of the weight matrices. So the transposes of the weight matrices sort of relay the information of what needs to change that would be the loss. What needs to change to make the losses small as possible. They relay this information down to the other layers and we don't know of any biological analogy to this mechanism right here. This transpose it acts as sort of a layer inverse and that is called weight transport. So weight transport means that you can you can do something like the transpose of the weights basically to carry to bring information from the next layer back to this layer. And in biology we don't have this and in direct feedback alignment we don't have this either. So direct feedback alignment the next thing here in this abstract is the algorithm that they are going to apply here. Direct feedback alignment and we'll go into what it is but it is more biologically plausible in that what it does is it takes the loss somehow and it distributes it globally to all of these layers like this. And it does so without requiring these transposes and also without requiring these sequential steps. So both of their proposed problems here would be solved by this. They say that in contrast with previous studies limited to computer vision tasks. So what people have tried to do is they have tried to apply this DFA algorithm to computer vision tasks. But in computer vision most architectures are CNNs and as I understand it as far as I understand it DFA can only right now be applied to linear layers. So something that is WX plus B and then a non-linearity. It cannot even though you can write the CNN as like a linear layer with constraints as I read this paper I think to interpret that you can only apply DFA to fully connected layers or things that look kind of like fully connected layers. So what they're going to do in their experiments is they're going to take these big architectures like transformers and replace parts of them with the parts that act as fully connected layers with DFA updates. So well they're not going to replace the layers but they're going to replace the back propagation part of it with DFA updates. It remains to say that they still use back propagation at some places where they can't replace the updates with DFA and that means where the layer isn't you know a fully connected layer or I guess it's too big. They somehow have to make it work so often they will not update for example the embedding layers and things like this. Okay so what they're saying is they go away from computer vision tasks because if you go to computer vision and CNNs rule that world right you can only do for feet-forward layers fully connected layers you're gonna lose already. So yeah it's kind of an unfair fight in that sense but even in absence of that they say we apply this to neural view synthesis, recommender systems, geometric learning and natural language processing. So these are quite diverse tasks and they're going to be quite diverse architectures that they are applying it to. For example in geometric learning I believe they do graph neural networks and there they replace the usually in graph neural networks there are fully connected layers that connect the two the vertices and the edges together and compute properties of them. So that's a pretty good point for using DFA right because what you're looking for is state-of-the-art tasks and architectures that still employ fully connected layers because there your algorithm can shine. Okay so that's it and they're basically going to show that this is performance is close to fine-tuned back propagation. Alright so what is DFA? What is this direct feedback alignment? And for that I actually want to jump papers right here and go to this other paper that describes DFA in a bit in a bit not more detail but in a graphic fashion. So this paper right here direct feedback alignment provides learning in deep neural networks by Arl Noecklund sorry Noecklund shows some theoretical properties about DFA. Now I don't want to go into the theory right here or in the math but I mainly like this paper for this particular graphic right here. So in the back propagation algorithm as you can see you forward propagate using these weight matrices and then you back propagate using the transposes of the weight matrices. Now one step after that is this thing right here it's called feedback alignment. It's not the same thing as a direct feedback alignment. In feedback alignment you simply say well I won't back prop using these transposes because I can't because that's not biologically possible. What I'll do is I'll use other matrices and these other matrices are going to be random matrices and by random matrices we really mean a matrix that is of you know the correct shape the same shape as this W transpose but each entry is going to be sampled from a like a random Gaussian right now I don't mean like the distribution of Gaussians but you fix this matrix once at the beginning of training by sampling from Gaussian and then you leave it there and that's going to be the matrix that you use for relaying the signal back through the layers. Now you might protest and say wait that's not gonna work because specifically this thing right here it you know that you need to know the weights here to know what you need to change in the lower layers you need to somehow have that information in there how are you gonna know what to change and that's a valid question and I will give my opinion of why this works okay in a second in two seconds first this is feedback alignment so simply use random matrices to back propagate so to say and then you have direct feedback alignment and direct feedback alignment goes a step further because in feedback alignment you still do this in a sequential manner direct feedback alignment simply takes whatever the top change should be the change to the top layer so how do I need to change the top layer and it back propagates that in a this global fashion to all the layers directly using random matrices okay and then this IFA we're not gonna look at today because that's not relevant for this other paper but I hope you can sort of see the overview here so let's go back scroll scroll scroll scroll scroll scroll scroll okay so here is the mathematical formulation of all of this and it pays to look at it to understand what's going on so they characterize a neural network right here as having n layers each neural network is the following each neural each layer takes whatever is the output of the last layer multiplies it by a weight matrix and that's going to be your a quantity you put a through a non-linearity to obtain the next layers input okay so the H is the output of this layer and the input of the next layer at the very end your last output is going to be your estimation of the labels so your last non-linearity is probably going to be something like a a softmax or something like this okay so how can we how can we have this as a concept in our heads if you have the neural network right here what you want to do is you want to forward prop always using your weight matrix W and then your non-linearity of that particular layer and then the last in the last layer you get your y hat as we saw before now the question is how can we adjust how can we adjust this W right here to make y hat more into the direction of y and here it's in here it's useful to think of the last layer as a vector output like usually we think of the loss function but in all of these algorithms they always start with the derivative of the loss function with respect to the last layer output so ay and ay is here right before the non-linearity if you remember this was f of ay and this here I guess is the softmax so if this is a classifier the ay here those are the logits and that's the output of your last layer so it instead of having y and y hat right sorry y hat right here it pays to maybe think of the output as a vector and the desired output as another vector and the desired output is of course going to be one hot vector in the case of in the case of a classification but it you know if you think of it like this then you'll recognize okay I need to change if this is my estimated output and I want to achieve this output I need to change it into this direction right to get more into the same direction as the output I want the entire question now becomes how do I tell the lower layers about this change right here this is the change that I want to make in the lower layers how do I get the lower layers such that they provide me with that signal with with the green signal instead of the red signal so I need to propagate this blue difference in the back propagation algorithm you can simply ask the system right so we've built entire frameworks on being able to back propagate tensorflow pytorch jacks whatever because with back propagation we can simply ask the system this question so here is how should I change the weights of my layer to make the loss smaller you can just ask that you can say what's the gradient of the loss with respect to the to my weights and the night negative sign here is because you want to make the loss smaller okay and that is going to be a straightforward calculation how does that calculation go it's going to involve this right here is the last layers output this right here as you can see over here is going to be this is going to be whatever comes back from the back propagation so in back propagation you always have to think of if you want to update these weights you need two quantities you need whatever comes from the bottom or came from the bottom during the forward pass and whatever comes from the top during the backward pass and this quantity here is going to be the one that came from the top and it's basically how you need to change the next layer in order to make the loss happier and by using this right here you pull it back to this layer so how do I need to change this layer and here you see that dreaded transpose of that weight matrix this is what we can't do in biology but this is what back propagation does so it pulls back how you need to change the next layer it pulls it back to this layer so this quantity right here is basically how do I need to change the output of this particular layer in order to make the loss happier and then you multiply it by the signal that comes from the bottom and that will give you how you need to change your weights okay so the green part is how does the output of the layer need to change and the multiplied by the blue part it's how do the weights need to change and of course the non-linearity is in there as well but let's let's just leave the non-linearity away because it's really not important for this particular thing so this is what backprop does what does DFA do DFA here again asks how should I change the weights of layer I and DFA says well first you need to compute this thing right here this is you see the derivative of the loss with respect to a y now a y is the output of the last layer these are in in our case for example your log it's okay note that this is still a gradient so it's not like we can't differentiate anymore we simply can't do back propagation from layer to layer okay so this is the quantity how do we need to change the last layers output and we're going to take that and simply feed it through this random matrix and then multiply again let's leave this away multiply it by the by this thing right here so if I get my colors correct like this again you have your neural network you want to update these weights the green is what comes from the top now it doesn't come from the next layer but the green actually comes from all the way at the end sorry you can't see that I still have to get used to that new frame of view so the green comes all the way from the end and the blue comes from down here okay so this is weird right because especially because this is just modulated by a random matrix so how can this possibly work that's the question and I you know I had some thoughts but I haven't read too much about it so I might be completely wrong or this might be completely known in the community I have no idea I'll just give my opinion right here so first of all you have to see you have to compare this to backprop so what's actually changing is this green part right here right we agree that this is the thing that's changing and what do we say does the green part mean the green part basically tells you how do you how should the output of this layer change okay by adjusting the weights in the direction of the thing on the right side of the equality sign you're going to change the output of the layer into the direction of that green part now in backpropagation the green part basically tells you how should the output of this layer change in order to make the loss as happy as possible now we don't have that anymore here we simply change the output of the layer into the into the direction of a random transformation of the of the change we would like to have in the output now okay that's the the first thing is we understand what's different and we understand what the green quantity means green quantity means how should the output of our layer change okay second thing if you look at the last layer of a neural network that that logits layer right what does it actually do let's say we have that's a three-dimensional last layer which means you have three classes right if your last layer is three-dimensional you have three classes each axis represents one class because you encode the classes as one hot vectors so this might be C the class label equals zero this might be C equals one this might be C equals two if you have something that you forward propagate through your neural network and let's say it comes out to be like this what would you classify that as now you classify that as the whatever class has the the biggest inner product with that vector which would be the C equals zero class right here and what is this quantity going to be how should you update this output in order to make the loss happier now that depends on your true label but let's say your true label is actually the zero label now what you want to do is you want to update that thing into the direction here right such that it is more aligned with the axis so what happens if you pull that back through a random matrix now the thing you have to know about random matrices like this is that they do approximately preserve distances and angles so technically if you pull this back what you're going to induce is another coordinate system in that other space now this can be a higher or a lower dimensional space I frankly I don't care but what you're going to induce is a coordinate system and what do you pull through that B matrix so this is the BI matrix you fix it right this is really important you fix it at the beginning of training it's always the same it preserves distances and angles approximately you pull back that quantity which is the okay my colors are all screwed which is the green arrow over here you pull back this green arrow here so what does it mean what so the output right here the output vector that came from the lower layers right that's you forward propagated that through your network so maybe in this layer it actually pointed here we don't know but let's say it pointed here if we pull back the green thing it might point here okay now this is since it's a random matrix we don't know we know that the angle is approximately preserved okay but you know that and these lengths are approximately preserved with relative to each other but it doesn't really tell you too much so why is this useful and to see why it's useful you need to consider other inputs we don't just in input this one vector we input a whole bunch of data now let's consider two other vectors so first I want to consider this this blue vector right here now the blue vector is also going to have a label of zero so what does the blue vectors update look like the blue vector is going to be pulled into this direction and I also want to consider this red vector right here the red vector is of class one so what does the red vectors update going to look like like this and if I consider now the red and the blue vector in this space right let's I just draw them at random like so okay what I do know actually that's that's for consistent let's draw the blue somewhere here and the red somewhere here what I do know is that the angles and distances are preserved so what is the green thing going to look like the update for the blue vector is going to be something like this and the update for the red vector is going to maybe be something like this you know away from from those so what is happening in that lower space you'll notice that the two vectors that are supposed to be in the same class this and this they are going to be pulled together now the direction they're pulled in that's determined by this random matrix but we know they're going to be pulled together because they are pulled together in this space in the final space okay and they're going to be pulled apart from the red vector okay because that red vector is going to to be pulled towards a different class in the in the last space and since the distances and angles are approximately preserved it's going to be pulled away from these in in this space so what this induces in my opinion is some sort of it induces this coordinate system where if you make the last layer axis aligned because you want to classify it it kind of clusters things that belong in the same class in these previous weight spaces right and because and if you do this layer by layer so if you do this in layer K and then you make the job easier for any layer K plus one that's in between here right because they are now the things in the same class are already together pretty okay now you map it through a weight and the non-linearity they might you know intertwine a bit again but there's they're more together than they would be otherwise so you make the job for the next layer easier which means that the next layer can also can even better cluster things and what you'll end up with in this last layer is the is a basically a class or next to last layer is basically a clustering where everything that's supposed to be in the same class is together and far apart from each other and since the last layer is the classification layer it's going to have a really easy job separating those classes and performing good classification so that's what I think is happening in this algorithm so even though the layers don't know how to change to help the last layer by the fact that these random matrices and induce a clustering together you know by back propagating these updates here it helps the last layer make it makes its job really easy and you know that's all the classifier needs and I want to I want to show again this is my opinion this is not anything of value it's just my hypothesis of why something like this could work I want to show you in this paper that I've shown you before right here they do actually do these experiments with DFA and they show that you can see top row shows feature obtained with back propagation bottom row shows features obtained with DFA I think these are input and features I'm not sure where exactly they are in the network but you can see that this clustering clearly emerges so oh yeah here from left to right input images first hidden layer second hidden layer third hidden layer so you can see that the clustering from layer to layer in backprop and also in DFA is better and better so the reason why backprop is good maybe it's just that because it also really induces clusterings like this I don't know maybe backprop does even does something on top of that because I mean backprop has all the properties of this and more right but still this this is congruent with my hypothesis of what's happening so what do they do with it they take this algorithm and they apply it to these architectures now let's for example look at one of them this neural view synthesis with neural radiance fields so neural radiance fields is a type of model to do this task of where you get a bunch of views of an object in 3d or you know a bunch of views around an object and you're supposed to render a new view and you can see that the DFA parameter or the DFA updated nerve neural radiance field model is pretty close to the back propagation updated one you can see it's a bit more blurry but it it works right and I think the this paper is really trying to show that look this works it doesn't work you know extremely well but it works and it works on a level that hasn't been seen before so here if you consider these results higher is better on the synthetic data set here even you see that if you have the same model with backprop it performs better than with DFA but the DFA for that model performs better than these other baseline models that have themselves been trained with back propagation so it's definitely in the direction of being competitive and that's the same thing they show with all of these experiments so they apply this to graph networks apply this to transformers and as I said it's it's not there yet you see that so in the transformers they have these settings where macro they just use it DFA for the individual blocks and micro they use it for each layer and already told you that you still in the attention mechanism you still have to use backprop within the attention mechanism but it is much more of a plausible algorithm than the back propagation through the entire network and they show that if they appropriately tweak the hyper parameters they do get into the direction of something that's performant at least with this macro strategy now this is nowhere close to this is nowhere close to what the to what the back propagation algorithm achieves but it's sort of it's sort of an indication that if the community could work as much on this as it has worked on back propagation then probably will make a lot of like we could we could push this to a place where it does perform on par with backprop or very close to it so I do invite you to go and look at the experiments they have a lot of lot of details on how they did it and exactly how you have to change the architectures to make DFA work and the hyper parameters and so on so that's really cool and they have some more outputs right here of the view synthesis and so on yeah if you are interested in that thing I again I don't want to disrespect it it's just I don't think there is much point in me going over it it's the results are always sort of the same that DFA it it's not there yet but it's a good direction yeah I hope this was informative let me know if you disagree about my assessment of DFA I could be completely wrong or you know I yeah or or this could be like well known to people already so yeah see you next time
 Hi there, today we'll look at direct feedback alignment scales to modern deep
 learning tasks and architectures by Julia Alonet, Jacopo Poli, François Boniface
 and Florian Krzakala. So this paper on a high level it replaces the back
 propagation algorithm in deep learning architectures with this algorithm called
 direct feedback alignment which is more biologically plausible. The algorithm has
 been around for a while but it hasn't yet been shown to be applicable to
 really modern big deep learning architectures and then perform on par
 with backprop on modern deep learning tasks. This paper as I understand it is
 the first one to demonstrate that it can do that. So this is very much an
 engineering paper, an applied paper and we're going to mostly go into direct
 feedback alignment as such and I don't think we're going to go too much into
 what the actual empirical findings are because they even though they're
 impressive and it's a good piece of engineering I think they can be
 summarized pretty much by it works not yet on par with back propagation but
 into a promising direction. Alright as always if you like content like this
 consider sharing it out and leaving a like and tell me in the comments what
 you like. Of course subscribe if you aren't yet that that is you know
 essential otherwise how are we gonna hear from me in the future? Okay let's
 dive in. They say despite being the workhorse of deep learning the back
 propagation algorithm is no panacea. It enforces sequential layer updates thus
 preventing efficient parallelization of the training process. Furthermore its
 biological plausibility is being challenged. Alternative schemes have been
 devised yet under the constraints of synaptic asymmetry. None have scaled to
 modern deep learning tasks and architectures. Here we challenge this
 perspective and study the applicability of direct feedback alignment to neural
 view synthesis, recommender systems, geometric learning and natural language
 processing. In contrast with previous studies limited to computer vision tasks
 our findings show that it successfully trains a large range of state-of-the-art
 deep learning architectures with performance close to fine-tuned back
 propagation. At variance with common beliefs our work supports that
 challenging tasks can be tackled in the absence of weight transport. So there's a
 lot to unpack in this particular abstract right here. So first of all what's the
 problem with back propagation? Back propagation they have they have two
 corals right here. First of all it's preventing efficient parallelization of
 the training process. So what does that mean? So in back propagation I'm pretty
 sure you all know it's basic back propagation but you have an input to a
 neural network and the neural network has a bunch of layers so the input will
 travel layer by layer and at the end you'll get some output and your output
 y hat, let's call it here what the neural network thinks the let's say it's a
 classifier, thinks that the class of this particular X should be. Now in the
 data set you have your true label and then you compare that to your output
 label and you can compute a loss function. Now the whole question of the
 back propagation algorithm is how do I need to change my layers of the neural
 network in order to make the loss as small as possible? And for that you can
 use back propagation that means you can take that loss and you can back
 propagate it down the layers in order to update each layer individually. So the
 first problem they have here with the back propagation algorithm and it's not
 I mean it's kind of a secondary problem but it is that is sequential. So in order
 to update this layer right here you need to have already back propagated to this
 layer and then you need to back propagate further to this and to this
 layer so it's a sequential task you need to back propagate down the layers again
 whereas what is more plausible but what would be more efficient if we could
 somehow update all the layers in parallel but this is a minor quarrel. The
 bigger one is that back propagation isn't biologically plausible. We know
 that in real neurons you have your dendrites, your inputs and your
 axon and the signal only travels in one direction. We don't know of a feedback
 mechanism in true neurons in the brain that would allow for information sort of
 to flow in the opposite direction. There is information flowing in the
 opposite direction but I think it's too slow and it's so it's not
 really it can't be there's no analogous way of back propagation. There's no
 nothing in the brain that would take the role of the back propagation algorithm.
 Specifically if each layer is characterized by a weight matrix right here
 what back propagation does is it uses the transpose of that weight matrix to
 back propagate. So these arrows to the front right here they use the
 weight matrices and these arrows to the back they use the transposes of the
 weight matrices. So the transposes of the weight matrices sort of relay the
 information of what needs to change that would be the loss. What needs to change
 to make the losses small as possible. They relay this information down to the
 other layers and we don't know of any biological analogy to this
 mechanism right here. This transpose it acts as sort of a layer inverse and that
 is called weight transport. So weight transport means that you can you can do
 something like the transpose of the weights basically to carry to bring
 information from the next layer back to this layer. And in biology we don't have
 this and in direct feedback alignment we don't have this either. So direct
 feedback alignment the next thing here in this abstract is the algorithm that
 they are going to apply here. Direct feedback alignment and we'll go into
 what it is but it is more biologically plausible in that what it does is it
 takes the loss somehow and it distributes it globally to all of these
 layers like this. And it does so without requiring these transposes and also
 without requiring these sequential steps. So both of their proposed problems here
 would be solved by this. They say that in contrast with previous studies
 limited to computer vision tasks. So what people have tried to do is they have
 tried to apply this DFA algorithm to computer vision tasks. But in computer
 vision most architectures are CNNs and as I understand it as far as I
 understand it DFA can only right now be applied to linear layers. So something
 that is WX plus B and then a non-linearity. It cannot even though you
 can write the CNN as like a linear layer with constraints as I read this paper I
 think to interpret that you can only apply DFA to fully connected layers or
 things that look kind of like fully connected layers. So what they're going to
 do in their experiments is they're going to take these big architectures like
 transformers and replace parts of them with the parts that act as fully
 connected layers with DFA updates. So well they're not going to replace
 the layers but they're going to replace the back propagation part of it with DFA
 updates. It remains to say that they still use back propagation at some
 places where they can't replace the updates with DFA and that means where
 the layer isn't you know a fully connected layer or I guess it's too big.
 They somehow have to make it work so often they will not update for example
 the embedding layers and things like this. Okay so what they're saying is they
 go away from computer vision tasks because if you go to computer vision and
 CNNs rule that world right you can only do for feet-forward layers fully
 connected layers you're gonna lose already. So yeah it's kind of an
 unfair fight in that sense but even in absence of that they say we
 apply this to neural view synthesis, recommender systems, geometric learning
 and natural language processing. So these are quite diverse tasks and they're
 going to be quite diverse architectures that they are applying it to. For example
 in geometric learning I believe they do graph neural networks and there
 they replace the usually in graph neural networks there are fully connected
 layers that connect the two the vertices and the edges together and compute
 properties of them. So that's a pretty good point for using DFA right because
 what you're looking for is state-of-the-art tasks and architectures
 that still employ fully connected layers because there your algorithm can shine.
 Okay so that's it and they're basically going to show that this is performance
 is close to fine-tuned back propagation. Alright so what is DFA? What is this
 direct feedback alignment? And for that I actually want to jump papers right here
 and go to this other paper that describes DFA in a bit in a bit not
 more detail but in a graphic fashion. So this paper right here direct feedback
 alignment provides learning in deep neural networks by Arl Noecklund
 sorry Noecklund shows some theoretical properties about DFA. Now I don't want to
 go into the theory right here or in the math but I mainly like this paper for
 this particular graphic right here. So in the back propagation algorithm as you
 can see you forward propagate using these weight matrices and then you back
 propagate using the transposes of the weight matrices. Now one step after that
 is this thing right here it's called feedback alignment. It's not the same
 thing as a direct feedback alignment. In feedback alignment you simply say well I
 won't back prop using these transposes because I can't because that's not
 biologically possible. What I'll do is I'll use other matrices and these other
 matrices are going to be random matrices and by random matrices we really mean a
 matrix that is of you know the correct shape the same shape as this W transpose
 but each entry is going to be sampled from a like a random Gaussian right now
 I don't mean like the distribution of Gaussians but you fix this matrix once
 at the beginning of training by sampling from Gaussian and then you leave it
 there and that's going to be the matrix that you use for relaying the signal
 back through the layers. Now you might protest and say wait that's not gonna
 work because specifically this thing right here it you know that you need to
 know the weights here to know what you need to change in the lower layers you
 need to somehow have that information in there how are you gonna know what to
 change and that's a valid question and I will give my opinion of why this works
 okay in a second in two seconds first this is feedback alignment so simply use
 random matrices to back propagate so to say and then you have direct feedback
 alignment and direct feedback alignment goes a step further because in feedback
 alignment you still do this in a sequential manner direct feedback
 alignment simply takes whatever the top change should be the change to the top
 layer so how do I need to change the top layer and it back propagates that in a
 this global fashion to all the layers directly using random matrices okay and
 then this IFA we're not gonna look at today because that's not relevant for
 this other paper but I hope you can sort of see the overview here so let's go back
 scroll scroll scroll scroll scroll scroll scroll okay so here is the
 mathematical formulation of all of this and it pays to look at it to understand
 what's going on so they characterize a neural network right here as having n
 layers each neural network is the following each neural each layer takes
 whatever is the output of the last layer multiplies it by a weight matrix and
 that's going to be your a quantity you put a through a non-linearity to obtain
 the next layers input okay so the H is the output of this layer and the input
 of the next layer at the very end your last output is going to be your
 estimation of the labels so your last non-linearity is probably going to be
 something like a a softmax or something like this okay so how can we how can we
 have this as a concept in our heads if you have the neural network right here
 what you want to do is you want to forward prop always using your weight
 matrix W and then your non-linearity of that particular layer and then the last
 in the last layer you get your y hat as we saw before now the question is how can
 we adjust how can we adjust this W right here to make y hat more into the
 direction of y and here it's in here it's useful to think of the last layer
 as a vector output like usually we think of the loss function but in all of these
 algorithms they always start with the derivative of the loss function with
 respect to the last layer output so ay and ay is here right before the
 non-linearity if you remember this was f of ay and this here I guess is the softmax
 so if this is a classifier the ay here those are the logits and that's the
 output of your last layer so it instead of having y and y hat right sorry y hat
 right here it pays to maybe think of the output as a vector and the desired
 output as another vector and the desired output is of course going to be one hot
 vector in the case of in the case of a classification but it you know if you
 think of it like this then you'll recognize okay I need to change if this
 is my estimated output and I want to achieve this output I need to change it
 into this direction right to get more into the same direction as the output I
 want the entire question now becomes how do I tell the lower layers about this
 change right here this is the change that I want to make in the lower layers
 how do I get the lower layers such that they provide me with that signal
 with with the green signal instead of the red signal so I need to propagate
 this blue difference in the back propagation algorithm you can simply ask
 the system right so we've built entire frameworks on being able to back
 propagate tensorflow pytorch jacks whatever because with back propagation
 we can simply ask the system this question so here is how should I change
 the weights of my layer to make the loss smaller you can just ask that you can
 say what's the gradient of the loss with respect to the to my weights and the
 night negative sign here is because you want to make the loss smaller okay and
 that is going to be a straightforward calculation how does that calculation go
 it's going to involve this right here is the last layers output this right here
 as you can see over here is going to be this is going to be whatever comes back
 from the back propagation so in back propagation you always have to think of
 if you want to update these weights you need two quantities you need whatever
 comes from the bottom or came from the bottom during the forward pass and
 whatever comes from the top during the backward pass and this quantity here is
 going to be the one that came from the top and it's basically how you need to
 change the next layer in order to make the loss happier and by using this right
 here you pull it back to this layer so how do I need to change this layer and
 here you see that dreaded transpose of that weight matrix this is what we can't
 do in biology but this is what back propagation does so it pulls back how you
 need to change the next layer it pulls it back to this layer so this quantity
 right here is basically how do I need to change the output of this particular
 layer in order to make the loss happier and then you multiply it by the signal
 that comes from the bottom and that will give you how you need to change your
 weights okay so the green part is how does the output of the layer need to
 change and the multiplied by the blue part it's how do the weights need to
 change and of course the non-linearity is in there as well but let's let's just
 leave the non-linearity away because it's really not important for this
 particular thing so this is what backprop does what does DFA do DFA here
 again asks how should I change the weights of layer I and DFA says well
 first you need to compute this thing right here this is you see the derivative
 of the loss with respect to a y now a y is the output of the last layer these
 are in in our case for example your log it's okay note that this is still a
 gradient so it's not like we can't differentiate anymore we simply can't do
 back propagation from layer to layer okay so this is the quantity how do we
 need to change the last layers output and we're going to take that and simply
 feed it through this random matrix and then multiply again let's leave this
 away multiply it by the by this thing right here so if I get my colors
 correct like this again you have your neural network you want to update these
 weights the green is what comes from the top now it doesn't come from the next
 layer but the green actually comes from all the way at the end sorry you can't
 see that I still have to get used to that new frame of view so the green
 comes all the way from the end and the blue comes from down here okay so this
 is weird right because especially because this is just modulated by a
 random matrix so how can this possibly work that's the question and I you know
 I had some thoughts but I haven't read too much about it so I might be
 completely wrong or this might be completely known in the community I have
 no idea I'll just give my opinion right here so first of all you have to see you
 have to compare this to backprop so what's actually changing is this green
 part right here right we agree that this is the thing that's changing and what
 do we say does the green part mean the green part basically tells you how do
 you how should the output of this layer change okay by adjusting the weights
 in the direction of the thing on the right side of the equality sign you're
 going to change the output of the layer into the direction of that green part
 now in backpropagation the green part basically tells you how should the
 output of this layer change in order to make the loss as happy as possible now
 we don't have that anymore here we simply change the output of the layer
 into the into the direction of a random transformation of the of the change we
 would like to have in the output now okay that's the the first thing is we
 understand what's different and we understand what the green quantity means
 green quantity means how should the output of our layer change okay second
 thing if you look at the last layer of a neural network that that logits layer
 right what does it actually do let's say we have that's a three-dimensional last
 layer which means you have three classes right if your last layer is
 three-dimensional you have three classes each axis represents one class because
 you encode the classes as one hot vectors so this might be C the class
 label equals zero this might be C equals one this might be C equals two if you
 have something that you forward propagate through your neural network
 and let's say it comes out to be like this what would you classify that as now
 you classify that as the whatever class has the the biggest inner product with
 that vector which would be the C equals zero class right here and what is this
 quantity going to be how should you update this output in order to make the
 loss happier now that depends on your true label but let's say your true label
 is actually the zero label now what you want to do is you want to update that
 thing into the direction here right such that it is more aligned with the axis so
 what happens if you pull that back through a random matrix now the thing
 you have to know about random matrices like this is that they do approximately
 preserve distances and angles so technically if you pull this back what
 you're going to induce is another coordinate system in that other space now
 this can be a higher or a lower dimensional space I frankly I don't care
 but what you're going to induce is a coordinate system and what do you pull
 through that B matrix so this is the BI matrix you fix it right this is really
 important you fix it at the beginning of training it's always the same it
 preserves distances and angles approximately you pull back that quantity
 which is the okay my colors are all screwed which is the green arrow over
 here you pull back this green arrow here so what does it mean what so the output
 right here the output vector that came from the lower layers right that's you
 forward propagated that through your network so maybe in this layer it
 actually pointed here we don't know but let's say it pointed here if we pull
 back the green thing it might point here okay now this is since it's a random
 matrix we don't know we know that the angle is approximately preserved okay
 but you know that and these lengths are approximately preserved with relative to
 each other but it doesn't really tell you too much so why is this useful and
 to see why it's useful you need to consider other inputs we don't just in
 input this one vector we input a whole bunch of data now let's consider two
 other vectors so first I want to consider this this blue vector right here now the
 blue vector is also going to have a label of zero so what does the blue
 vectors update look like the blue vector is going to be pulled into this
 direction and I also want to consider this red vector right here the red
 vector is of class one so what does the red vectors update going to look like
 like this and if I consider now the red and the blue vector in this space right
 let's I just draw them at random like so okay what I do know actually that's
 that's for consistent let's draw the blue somewhere here and the red
 somewhere here what I do know is that the angles and distances are preserved
 so what is the green thing going to look like the update for the blue vector is
 going to be something like this and the update for the red vector is going to
 maybe be something like this you know away from from those so what is
 happening in that lower space you'll notice that the two vectors that are
 supposed to be in the same class this and this they are going to be pulled
 together now the direction they're pulled in that's determined by this
 random matrix but we know they're going to be pulled together because they are
 pulled together in this space in the final space okay and they're going to
 be pulled apart from the red vector okay because that red vector is going to to
 be pulled towards a different class in the in the last space and since the
 distances and angles are approximately preserved it's going to be pulled away
 from these in in this space so what this induces in my opinion is some sort of it
 induces this coordinate system where if you make the last layer axis aligned
 because you want to classify it it kind of clusters things that belong in the
 same class in these previous weight spaces right and because and if you do
 this layer by layer so if you do this in layer K and then you make the job easier
 for any layer K plus one that's in between here right because they are now
 the things in the same class are already together pretty okay now you map it
 through a weight and the non-linearity they might you know intertwine a bit
 again but there's they're more together than they would be otherwise so you make
 the job for the next layer easier which means that the next layer can also can
 even better cluster things and what you'll end up with in this last layer is
 the is a basically a class or next to last layer is basically a clustering
 where everything that's supposed to be in the same class is together and far
 apart from each other and since the last layer is the classification layer it's
 going to have a really easy job separating those classes and performing
 good classification so that's what I think is happening in this algorithm so
 even though the layers don't know how to change to help the last layer by the
 fact that these random matrices and induce a clustering together you know by
 back propagating these updates here it helps the last layer make it makes its
 job really easy and you know that's all the classifier needs and I want to I
 want to show again this is my opinion this is not anything of value it's just
 my hypothesis of why something like this could work I want to show you in this
 paper that I've shown you before right here they do actually do these
 experiments with DFA and they show that you can see top row shows feature
 obtained with back propagation bottom row shows features obtained with DFA I
 think these are input and features I'm not sure where exactly they are in the
 network but you can see that this clustering clearly emerges so oh yeah
 here from left to right input images first hidden layer second hidden layer
 third hidden layer so you can see that the clustering from layer to layer in
 backprop and also in DFA is better and better so the reason why backprop is
 good maybe it's just that because it also really induces clusterings like
 this I don't know maybe backprop does even does something on top of that
 because I mean backprop has all the properties of this and more right but
 still this this is congruent with my hypothesis of what's happening so what
 do they do with it they take this algorithm and they apply it to these
 architectures now let's for example look at one of them this neural view
 synthesis with neural radiance fields so neural radiance fields is a type of
 model to do this task of where you get a bunch of views of an object in 3d or you
 know a bunch of views around an object and you're supposed to render a new view
 and you can see that the DFA parameter or the DFA updated nerve neural radiance
 field model is pretty close to the back propagation updated one you can see it's
 a bit more blurry but it it works right and I think the this paper is really
 trying to show that look this works it doesn't work you know extremely well but
 it works and it works on a level that hasn't been seen before so here if you
 consider these results higher is better on the synthetic data set here even you
 see that if you have the same model with backprop it performs better than with
 DFA but the DFA for that model performs better than these other baseline models
 that have themselves been trained with back propagation so it's definitely in
 the direction of being competitive and that's the same thing they show with all
 of these experiments so they apply this to graph networks apply this to
 transformers and as I said it's it's not there yet you see that so in the
 transformers they have these settings where macro they just use it DFA for the
 individual blocks and micro they use it for each layer and already told you that
 you still in the attention mechanism you still have to use backprop within the
 attention mechanism but it is much more of a plausible algorithm than the back
 propagation through the entire network and they show that if they appropriately
 tweak the hyper parameters they do get into the direction of something that's
 performant at least with this macro strategy now this is nowhere close to
 this is nowhere close to what the to what the back propagation algorithm
 achieves but it's sort of it's sort of an indication that if the community could
 work as much on this as it has worked on back propagation then probably will make
 a lot of like we could we could push this to a place where it does perform on
 par with backprop or very close to it so I do invite you to go and look at the
 experiments they have a lot of lot of details on how they did it and exactly
 how you have to change the architectures to make DFA work and the hyper parameters
 and so on so that's really cool and they have some more outputs right here of the
 view synthesis and so on yeah if you are interested in that thing I again I don't
 want to disrespect it it's just I don't think there is much point in me going
 over it it's the results are always sort of the same that DFA it it's not there
 yet but it's a good direction yeah I hope this was informative let me know if
 you disagree about my assessment of DFA I could be completely wrong or you know
 I yeah or or this could be like well known to people already so yeah see you
 next time