Hopfield Networks is All You Need (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper schmidhuber hochreiter lstm gru rnn hopfield attention attention is all you need transformer bert query key value routing pattern retrieval store error exponental binary continuous hopfield network lse energy function update rule metastable separation
#ai #transformer #attention

Hopfield Networks are one of the classic models of biological memory networks. This paper generalizes modern Hopfield Networks to continuous states and shows that the corresponding update rule is equal to the attention mechanism used in modern Transformers. It further analyzes a pre-trained BERT model through the lens of Hopfield Networks and uses a Hopfield Attention Layer to perform Immune Repertoire Classification.

OUTLINE:
0:00 - Intro & Overview
1:35 - Binary Hopfield Networks
5:55 - Continuous Hopfield Networks
8:15 - Update Rules & Energy Functions
13:30 - Connection to Transformers
14:35 - Hopfield Attention Layers
26:45 - Theoretical Analysis
48:10 - Investigating BERT
1:02:30 - Immune Repertoire Classification

Paper: https://arxiv.org/abs/2008.02217
Code: https://github.com/ml-jku/hopfield-layers
Immune Repertoire Classification Paper: https://arxiv.org/abs/2007.13505

My Video on Attention: https://youtu.be/iDulhoQ2pro
My Video on BERT: https://youtu.be/-9evrZnBorM

Abstract:
We show that the transformer attention mechanism is the update rule of a modern Hopfield network with continuous states. This new Hopfield network can store exponentially (with the dimension) many patterns, converges with one update, and has exponentially small retrieval errors. The number of stored patterns is traded off against convergence speed and retrieval error. The new Hopfield network has three types of energy minima (fixed points of the update): (1) global fixed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) fixed points which store a single pattern. Transformer and BERT models operate in their first layers preferably in the global averaging regime, while they operate in higher layers in metastable states. The gradient in transformers is maximal for metastable states, is uniformly distributed for global averaging, and vanishes for a fixed point near a stored pattern. Using the Hopfield network interpretation, we analyzed learning of transformer and BERT models. Learning starts with attention heads that average and then most of them switch to metastable states. However, the majority of heads in the first layers still averages and can be replaced by averaging, e.g. our proposed Gaussian weighting. In contrast, heads in the last layers steadily learn and seem to use metastable states to collect information created in lower layers. These heads seem to be a promising target for improving transformers. Neural networks with Hopfield networks outperform other methods on immune repertoire classification, where the Hopfield net stores several hundreds of thousands of patterns. We provide a new PyTorch layer called "Hopfield", which allows to equip deep learning architectures with modern Hopfield networks as a new powerful concept comprising pooling, memory, and attention. GitHub: this https URL

Authors: Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there. Today we'll look at hopfield networks is all you need by researchers from the Johannes Kepler University in Linz and the University of Oslo. So on high level this paper proposes a new type of hopfield networks that generalizes modern hopfield networks from binary patterns to continuous patterns and then shows that the retrieval update rule of these new hopfield networks is equivalent to the attention mechanism that's used in modern transformers. And it's actually a more general formulation of the attention mechanism and therefore it can be used to do kind of a variety of things to improve modern deep learning. And it also has a companion paper where it applies this to some kind of immunology research and achieves state-of-the-art in a task that is specifically suited to this type of attention. Alright let's dive in together we'll go over what this paper does what it proposes and so on. If you like pay if you like videos like this consider subscribing, you know sharing it out and I hope you're enjoying this. Alright also thanks to my discord community for you know very helpful bringing me up to speed on this paper. Super interesting discussions there. If you're not on our discord yet I invite you to join it's fun. Okay so what is a hopfield network? A hopfield network is a pretty kind of old style, an old conceptualization of a neural network. So in a hopfield network what your goal would be is you can conceptualize it as a bit of a neural network. So let's say we have five neurons or something like this. What your goal would be is to have a neural network where you can store so-called patterns and a pattern in this case would be a binary string of size 5. So for example 10100 or 11010 and you'd have a list of these patterns and what your goal would be is to store these patterns in the neural network such that and here you know we'll just consider everything to be sort of connected to everything else and what your goal would be in this is that you can kind of store patterns inside this neural network and you adjust the weights somehow. So this was as I said this was this was this is kind of an old model. You store you adapt the weights such that you store these patterns. And what does it mean for a pattern to be stored? If you have stored a pattern you will then be able to retrieve it and you retrieve a pattern in these kind of old style Hopfield networks by providing a partial pattern. So what you'll say is for example I want a pattern that starts with 110 and you give that to the network and there would be a so-called update rule and the update rule is kind of an internal rule. So let's just go through this. So here this 110 maybe this is 110 and then they would kind of send messages around so this update rule would somehow adjust the value of this and this neuron here to what's most compatible with the network weights and if the network weights have been adjusted correctly this will turn out then at the end of applying this update rule that this is a 1 and this is a 0 and therefore this pattern here is retrieved. Now had I input 101 at the beginning then the outcome would be different. Hopefully this pattern here would have been retrieved. So you can see the applications of this like you can have the first three digits as sort of a database key and then the last ones as sort of the value that you store along with it and then you can simply provide the first few. You can also provide you don't always have to provide three so this all depends. This is sort of an as I said an old conceptualization of neural networks so people were imagining that this is kind of how the brain works you know fire together wire together and also with research into this it turns out that you know you might think you know there's there's kind of five neurons so maybe I can store five different patterns you know accurately because if I store too many patterns right if I have many many many many patterns then I can't expect to be able to retrieve all the patterns again because some of them will just be so equal that you know many will start maybe with this and and I won't have a chance to to retrieve the one I want or the update rule will make a mistake so you might think this might be like five because I have five neurons or maybe ten because I have ten connections but it turns out that in modern hopfield networks with the appropriate update rule you can store exponentially many patterns in these networks exponentially many in the in the dimension of the in the dimension of the patterns and here I guess that would be the length of the pattern so this is a little bit surprising the kind of storage capacity of these networks and will this this paper here generalizes that to continuous to continuous states so what do we mean with continuous states I guess I mean continuous patterns so no longer is a pattern a binary string but a pattern now is a string of floating point numbers like 0.5 1.3 and so on and you know a string of floating or a sequence of floating point numbers is naturally depicted as a vector okay so our patterns are going to be different vectors that we store and you know in high dimensions that the the vectors will be kind of separated well from each other as long as we don't have too many but this paper shows that all these properties for the modern hopfield that works that hold for binary strings still hold if you go to these kind of continuous to these vector patterns that means you can store exponentially many patterns in the dimensions of the vector which is pretty surprising right because you'd think like you know after you have one vector per dimension that you know after that it might get a bit shaky but no you can actually store exponentially many that's pretty surprising and this paper is a lot about how to do that and the fact that that happens and so on so we've talked about update rules for these kind of hopfield networks and I haven't really specified what that is I've just said that you know I enter a pattern and then the network does something and outcomes outcomes the whatever the pattern that matches my query so this here is called a query you might already this is on purpose like the kind of overlap between the attention mechanism lingo and the hopfield network lingo we're going to conflate the two to kind of make clear where the two overlap if you don't know what an attention mechanism is or aren't familiar with it watch my video on attention is all you need once you watch that this video will make a lot more sense all right so in what the update rule does is specifically in the update rule that there isn't only one right there are many different proposals of hopfield networks and they all lead to different properties but what an update rule does ultimately is it minimizes what's called an energy so every type of hopfield network is associated with an energy function and this the energy function of the modern hopfield network for binary strings is this energy function right here so with X X is the pattern the pattern this is the kind of state of the hopfield network and these are the whatever is stored in the network and then the X here is the query that you enter into the network and then the energy here tells you this quantity you have to minimize this quantity in order to retrieve the pattern that you want okay now we are never directly working with the energy as such so what you could do is for example use backprop or something to use gradient descent to decrease the energy but usually along with an energy function comes an update function and the update function is what I've talked about here like you do something and then the network does something and then you get the pattern out what the network does is it minimizes its energy function and the update rule is made such that the corresponding energy function is minimized so the energy function is more like a theoretical consideration that you say okay here is my energy function of my hopfield network and the there will be a corresponding update rule that minimizes that energy function and if you use that update rule maybe multiple times then the energy function will be minimized and you will have retrieved your pattern or not if if you have too many patterns stored it might also fail right so they they say what the update rules are in the in the text here for the old hopfield networks but we're not really interested in the old ones we're interested in the ones that this paper cares about namely where the patterns that you store in the hopfield network are these vectors over our vector patterns and the query is also a vector pattern so you want to store all of these patterns into the hopfield network I'm gonna draw it like this here I'm gonna store it into the hopfield network and then after that you want to come up with a query and the query is like this and in the case of the binary strings we had something like well I sort of know half of my binary string now in the vector hopfield network it's more like well I sort of kind of know the direction that my vector should point in okay and you will read what you want to retrieve is the vector that has kind of a large inner product okay so if I enter this query into my hopfield network what I hope is that this vector here is retrieved now you see it's not exactly the same vector like they do point if I translate that here by I it's maybe something like this but so they are different but you want to say well I kind of know what I want I kind of want something like this and then the hopfield network would answer with oh I have something like this it's this right here okay so you that the connection to attention mechanism should become pretty pretty obvious right now but you know the to actually establish this formally is the kind of the point of this paper and you know it's pretty cool to see so they formulate this new energy right here this is the energy of this new continuous hopfield network specifically they have to have this term right here because they are now have continuous states and continuous queries this if you minimize the energy it basically means that your query can never you know go to infinity because you have the the query right here in the energy function the update rule is this right here and we'll look at that in a moment but remember the update rule is what you actually implement in code so if I if I have a query right here I plug it in here this is the state of my hopfield network and I apply this rule multiple times and out comes the kind of answer of the hopfield network to my question so the I input this and the out comes this after I update after I apply the update rule maybe multiple times right and interestingly you can already see that this here if you rewrite a bunch of these quantities so if you rewrite the beta here which is the softmax temperature in a way to be 1 over square root of D and if you take the query the psi here to be the query matrix and if you take the X here to be the key matrix then this is equivalent to the update or sorry the attention mechanism of a modern transformer so that's the point of the paper is that we can look at the transformer attention mechanism as a hopfield network and they have this interesting this interesting diagram at the end right here so the appendix you know this is typical I guess sep hoher I remember the cellul paper had like 60 pages of machine proof appendix this also this is like 70 page appendix crazy but at the end of the appendix you'll find this diagram right here now usually in an attention mechanism you have whatever the the input is so you have an input right here so this is attention mechanisms or at least transformers they work on sequences or sets of objects and from these you'll generate three things you'll generate the you'll generate the queries the keys and the values now you can either generate the queries from the same objects which would be self attention or you can generate the queries from like a different object over here it doesn't it doesn't matter too much for our discussions but either you you know have a reference input or you have you know this kind of same input all the way and then what you do is use three different heads or three different matrices to transform that input into queries keys and values so I often conceptualize this as you have kind of your input set and each of the input sets outputs a key and also each one which would be a vector and also each one outputs a query so I often draw this here the same sequence and each one outputs a query and the query sort of the query is kind of a request for information so the key exposes sort of what exposes something about the input here so this could be a sentence down here this could be my cat is very pretty and the the the vector the key vector right here could encode something like this is a noun or this is an animal or anything like this right and the query here it could ask for for other things so for example since this is cat this vector right here the query vector is generated from that you know token cat now it could recognize that cat is a noun and it could ask the other nodes to basically say are there any adjectives around here because you know adjectives because it itself is a noun it's the object of the sentence right it could ask are there any kind of adjectives that describe the object because that would be naturally a thing to ask if you were the noun you would want to know are there any kind of modifiers for for me so it could output the query and the query here could mean you know this direction could mean adjectives and you see here the word pretty is an adjective so it itself would output a key that says by the way I'm an adjective right so if the cat asks then if this node asks for an adjective and this outputs the adjective vector then because the inner product between the two things is high this will be routed here so attention mechanism is basically information routing that's how I always describe it but in this paper we look at it more like these here are the patterns that are stored in a hopfield network and I by inputting a query and the dot product being the update rule of the hopfield network I retrieve from the hopfield network I retrieve the appropriate pattern that I ask for okay and then you know the values the values are simply a modification of the keys in this form but a lot of people also do keys and values to be the same thing yeah but the this routing of information happens here where you multiply the queries and the keys and then you put a softmax over them okay so if you just look from the perspective of a single node like this node here this cat node what it would do is it would inner product its own query vector with all of the key vectors right so it would build an inner product with all of these and then it would normalize it would put it through a softmax which will kind of give it a distribution right so here would give it like so this is actually matches because my well my is also very important for cat this this this is just an accident I did not plan this this here this also well many things match but but in our example we would just say that this last one it's not only higher it's also wider it matches very well right and so the information routing would route mostly information from this pretty token to the cat token which makes sense in our case right this is the attention mechanism now since if we are interpreting this as a hopfield network and the update rule here is the dot product you can actually think of applying this rule multiple times so what happens now if we and this is where this update rule comes in what happens if we take this distribution and we don't aggregate the values like usually we would aggregate the values by this distribution what if we aggregate the keys by this distribution okay what comes out well if we look at this and you know let's just assume that this key right here matches really well but the others also match a little bit what would come out would be a weighted average where a lot of weight is put on this particular key so what will turn out would be something like something that's very close to that key you can see I'm going to draw the old key here in green and I'm going to draw the old query in blue so you see that it's then whatever comes out is not the query but it's also not that only key that matches right it's kind of a weighted average but with that key dominating okay now since you know in a hopfield network what we would do is we would go again we would put this new thing the red thing instead of the query vector okay so we would use this aggregated keys this weighted average as a new query vector for that node right here so we duplicate that node over here I'll use that query vector again and do the same thing again okay inner product with all of the query vectors and now since this is already an aggregate of the query vectors what's going to happen of course the distribution that's going to come out is going to be weighted even more heavily into the direction so let's make it even wider into the direction of that key that matches okay and you can pretty clearly see if I do that iteratively then that will lead to a situation where everything is like very low except that one key will sort of dominate the distribution and ultra high and ultra wide okay and that's how that's exactly how a hopfield network works right I would input the query which would be sort of what I want right I kind of know what I want okay and then I apply this rule multiple times and with each time I refine refine refine until I decide on a pattern the hopfield network is made for pattern retrieval and these here are the patterns that I want to retrieve so here the patterns aren't kind of stored in the network beforehand but the patterns are also generated like in an attention layer so the keys are generated by the previous layer or by these matrices but that doesn't matter for the hopfield network update rule so you see here that the attention mechanism can be interpreted as simply one step making one step of this update rule but you can think of making actually multiple steps and retrieving the particular key so you know deciding on a sort of a hard routing of particular information now that only works if if there are no other vectors that are close to that particular key right so if the query is this and you know the way I drew it here you can see that there are many there is this one and this one and this one that matches so technically the way I drew it what would happen most likely is no many no matter how many times you apply your update rule it would sort of result in kind of the average of the three keys right so because they're all matching and they would all contribute to that weighted average of the query in the next step and then that means basically the conversions would be to something in the middle and that's going to be a central point of this paper in which situation we are so they call the first part is retrieving a single pattern and they call the second situation where you have multiple patterns that all match that are not well separated from each other they call this a meta stable state and it's going to be pretty interesting to look at transform like BERT language models and look at where they actually are are they actually operating in this single pattern retrieval mode or are they operating in the meta stable state mode all right so here you can see it in the diagram the only thing different this from a hop field network sorry from an attention mechanism is this branch right here so here you ask do you want to do multiple updates after you've you've multiplied the queries and the keys do you want to do multiple updates if yes so if you're in a this hop field network situation you want to do multiple updates then you go back as you can see and you do you use the keys together with the output of the softmax to generate a new query so this query queue here is now generated from the output here and the key so the keys are the same these are this is the same thing it's just put here twice okay this is exactly what we discussed okay I hope that's some somehow clear that these the the attention mechanism is simply a one step hop field network pattern retrieval algorithm with a particular update rule that is that is matches this energy function that they propose right here of course they do this you know particularly because the update rule that turns out is the transformer update rule but but I actually don't know if they backwards engineered the energy function to match the transformer or if they first came up with a continuous hop field networks and then just kind of discovered that it's like the transformer will maybe never find out okay so let's go there are a couple of theorems I believe there are four five theorems right here that show that kind of make some points about this about this stuff and we'll go through them won't go through the proofs or any you know super in-depth meaning but it's pretty cool to go through them and they are proved very rigorously as I said there's a 70 page appendix so have a look at that if you're up for it okay so they say here we have an update rule this is our update rule for our new hop field networks so the first theorem they serve is the update rule that we propose converges globally if we apply the update rule repeatedly the energy for t goes equals infinity and the energy will converge sorry the energy will converge to a fixed point this being a fixed point for t equals sorry for t goes to infinity yeah if this is a fixed point basically saying that if I apply this update rule here over and over and over again it will it will make this energy function converge to a fixed it will make this energy function converge don't want to say anything mistakenly here or claim too much but that basically connects the update rule to the energy okay so just showing like this really is the update rule for that particular energy function okay now as as itself it's not super duper interesting yet but now we get to theorem 2 so theorem 2 for the iteration that's the update rule that we just looked at we have we have that this convergence holds as t goes to infinity for some stationary point furthermore this quantity here goes to zero so that means this is the the update at t plus one and this is the update at t and the difference between them goes to zero so that means not only does the energy converge but the iterates themselves converge so the algorithm actually converges the individual updates of the algorithm so this e new at some point that will no longer change because the the norm between it and the previous one will go to zero you can see that either the sequence here converges or in the other case the set of limit points yada yada is a connecting subset this is a bit over the top here they say okay it can either converge to a point or it can converge to a connected subset but if the loss is finite then any sequence generated by the iteration equation 3 converges to some fixed point so you know basically saying that here we oh this is not the loss I'm sorry no this is the domain never mind I am an idiot this is basically saying that this algorithm will converge okay and they define here what it means for a pattern to be stored and retrieved and that's for establishing what the kind of storage capacity for Hopfield network is so we've established that the update rule minimizes the appropriate energy and the update rule will converge at some point which means that we can you know if it converges we can retrieve the pattern that it converges to so now we define how many patterns can we actually store for that we need to know what does it mean for a pattern to be stored so we assume that we have patterns and these patterns are called X okay X I we have n different patterns each one is called X with a subscript we assume that around every pattern a sphere is given so how do we imagine this we have these patterns and this is this is just a space that they consider patterns of the on a sphere but we'll just conceptualize it as this will have a space there are patterns we want to store okay and we'll say around every pattern there is a sphere okay sphere like this and naturally the patterns are going to be there's going to be a notion of well separated patterns and you can imagine this a little bit like these spheres won't be touching each other if these spheres aren't touching each other that means that the patterns are kind of well separated and that means that if we initialize the query remember the query here is a vector that kind of sort of looks like a pattern and that means the query is kind of close to the pattern in some notion of distance so if we initialize the query somewhere in that sphere then it might if it converges to that sphere to that pattern then we retrieve the pattern okay now it gets a bit more complicated than this but not much more we say a pattern is stored if there is a single fixed point inside the sphere to which all points that start inside the sphere converge and none of the spheres intersect so the sphere of point I doesn't intersect with the sphere of point J so that's where we say all these spheres are non intersecting we say X is retrieved if the iteration equation 3 converged to the single fixed point in that sphere the retrieval error is the distance so you'll notice you have two things you have X I this is the actual pattern and you have X I star this is the retrieved pattern so these hopeful they don't always have to give you the same thing that you stored that's part of the the nature of continuous neural networks whatnot so for every sphere we say there is a pattern there is a sphere now we as pattern is stored if every I can start wherever I want in this sphere okay wherever I want it will always converge to a point that's inside the sphere and maybe that point isn't the pattern that I stored but actually this point right here but wherever I start I will always converge to that particular point if that's the case then I have stored this particular pattern now the fact is I don't retrieve this particular pattern I retrieve the blue thing but I can then define the error of retrieval the error of retrieval is simply the distance between the two things ideally this distance is very small right but you know we can't can't guarantee it now there are going to be theorems that deal exactly with this retrieval error but first you can see that here if if these spheres become larger you you can't accurately store a pattern anymore so this is the kind of ideal situation but there are also situations where these spheres you know if I have these patterns right here these spheres are so large kind of the the attractions of the patterns are so large that if I start let's say here then I don't converge to either of these two patterns I converge to like something in the middle I converge to maybe this point right here and that's going to be one of these meta stable states okay we're going to encounter situations like this but we're also going to encounter situations like this and the bottom thing isn't necessarily bad and that's what you have to keep in mind and yeah as I said we'll get to it but just keep this kind of sphere image in mind okay so first we'll just deal with the you know the up the top situation where we store patterns and then retrieve patterns so we'll assume a failure probability which is P and P is going to be no pretty pretty low for their example so they have P equals 0.001 you know like a 0.1 percent error probability of retrieving your pattern things like this and randomly chosen patterns on the sphere with radius M we define some constants yada yada yada then with probability 1 minus P the number of random patterns that can be stored and stored in the sense of having these spheres around them so that you can retrieve them accurately or at least you can retrieve something that's close to them is is bounded lower bounded by this quantity right here so there's the square root of P there is this constant C but then you see that D is in the exponent right here so that means it's exponential in the number of dimensions so that's that's pretty cool so if you add a dimension you exponentially increase the number of the number of patterns you can store and you know that's that is a kind of I mean it's it's been known for modern Hopfield networks with binary strings so it's not uber surprising but if you have you know it's not what you would imagine like that okay so they may give a few examples of these you have to accept these constants you know in a particular fashion such that this is given and so on but they say examples here are where C is something like 3 and D is 20 so if you were to add a 21st dimension then your I guess storage capacity would increase by a factor of 3 which pretty cool alright so this is how many that we can store infinitely not sorry exponentially many patterns in these networks now they deal they say the next theorem states that the update rule typically converges after one update if the patterns are well separated okay so if we're in a situation where these patterns are well separated which is kind of like this but you can also imagine this in terms of dot products because we operate in the space of dot products so if the patterns are well separated that sort of means that they all kind of sort of point away from each other and this notion of separation is going to be captured by this quantity right here this is the separation of example of pattern I which is just the inner product with itself minus the maximum inner product with any other pattern and this quantity is going to be large when no other pattern is close to it so when the separation is large then the update rule the retrieval rule of calculating you know I have a query calculate the inner product with all of those then I reweigh all of the patterns by that inner product by the softmax then I use that new thing as a query again and so on as we discussed it will converge to the closest pattern but this theorem says it actually converges pretty fast and here I have my problems with saying that it converges after one step typically converges after one update because that you know genuinely depends on a lot of constants as we'll see but it does converge exponentially fast in this separation constant as a theorem for says with query psi after one update the distance of the new point to the fixed point is exponentially small in the separation Delta I the precise bound using Jacobian and its value in the mean value theorem are the following so here you can see this is the distance between the updated psi after one step and the and the fixed point right here this is what it converges to is going to be the distance as it was before times this thing right here so you can see since this is a this is a multiplicative update and in this Jacobian so this is expanded down here this is this you can see here you have the you have this sorry yeah this is this so this is bounded by that you have the exponent the exponential function negative this separation right here so the higher the separation the faster this algorithm converges okay to say that it converges after one step is you know it might be a bit of bragging I don't know if this is a common thing if you have like an exponential convergence that you are allowed to say it's after one step I'm not sure especially what I'm not sure about is that you have n here as linear constants in that factor okay so if you if you and that's what they do in their code so if you look at their code and the codes available which is pretty cool it's implemented in pytorch as a general module that can you can just drop in so this is not only for transformers this is for you can replace like LSTM's you can replace pooling mechanisms you can you know do a whole bunch of stuff in their paper in the company paper they do this multi instance learning with giant sets on using these hopfield layers so pretty pretty cool this code is definitely worth kind of checking out and maybe you want to replace some stuff with you but the question is how many of these update steps should you do right because we looked at the diagram at least in the attention mechanism it seems like you have attention layers right you have a transformer and the transformer consists of you know you have this input right here and you go through layer layer layer layer layer and in each layer there's contained in it and one of these attention mechanism right this entire thing is in this layer okay and now if you interpret this as a hopfield network and you want to do multiple steps that means you go this branch right here so in each layer potentially you do multiple steps of these things so for whatever computational constraints transformers had already this will certainly make it worse but also you need to decide how many steps you want to do now you can hard code that of course but they say you should do these steps until this norm here until the norm between the old and the new is small enough so where is that so you can't measure how close you are to the convergence points right because you don't know in practice but you can measure how far you're away you can measure where did we have it you can measure this quantity right here that's something you can measure how far two iterates are apart so what you'll simply do is you'll measure that and if that is small enough then you'll you'll stop but that I guess is very related to this so how if you we've already proven it converges to this X star so I guess we can approximate this quantity right here with the quantity above and that tells you how many updates you need to do and that quantity is linear not only linear but actually here quadratic in n I don't care you know yes it's exponential in the separation but it's quadratic in n and if I've learned anything from kind of my fast code courses is that constants actually matter when you're not dealing with infinity with an infinite number of steps so the number of the number of steps you need to do I guess will depend on the sequence length in a quadratic fashion so I'm not sure you can always claim this is convergence in one step now I might be super mistaken here and none of this will can none of this actually makes a difference in the in the light of the exponential decay here but I would just I'm just a bit worried saying this usually converges in one step it's clear I guess why they do it right because the attention mechanism in transformers is a one-step application of this rule and this here is kind of a theoretical justification for interpreting this precisely as a hopfield network because you'd say well in a hopfield network you would do multiple steps but wait wait we can actually prove that even if you interpret it as a hopfield network you it can it usually converges after one step so what you're actually doing in a transformer is applying a hopfield network update rule to convergence so yeah I'm not yeah I might be bickering on a high level here luxury problems theorem five then says so theorem four is how fast does this converge theorem five the last theorem right here says that the retrieval error of a pattern then so this is the this is what you converge to and this is what you've stored is bounded by again something that's exponential in the separation right here as you can see okay so that was the theorem so if we go quickly through them again theorems one and two deal with the convergence of this algorithm and the fact that it actually minimizes the proposed energy then theorem three says you can store exponentially many patterns in terms of the dimension of your space and theorems four and five say that this update rule will converge exponentially fast after after one step if you believe that and the retrieval error will also go down exponentially fast with the number of update steps that you do okay that sounds pretty pretty pretty good but we've heard it it's very dependent on how well separated these patterns are and it turns out that is you know at least in transformers they aren't always well separated and that might be on purpose remember the the states here the patterns aren't pre stored like in a classic hopfield network but the patterns if you interpret an attention mechanism as this are also generated by the network itself so the pattern matrix that you retrieve from and the query are generated by the attention mechanism in this case as I said this is applicable to many many more domains than just this but yeah so there's another slight modification that you have to do to make this actually equivalent to an attention mechanism and that is you'll have to recast the value because usually what you'll do is you have some sort of input and then you make queries keys and values from that using different heads the only thing to make it formally equivalent is you have to make the values generated from the keys so the keys give rise to the values as you can see right here that you first multiply with the key matrix and then with the value matrix I think that's you know that I don't I doubt that this will will change anything if you if you the only way that could really change anything is if this matrix here would be super low rank like collapse the space of into like very few dimensions which the value matrix wouldn't do so you know but just letting you know that the technical equality requires this slight modification okay now we said that it might not you know be that this is always super well separate and you retrieve a single pattern and that's what they research here in a pre trained BERT model so they take a pre trained BERT model from I guess from hugging face and they run they just run a data set through it and what they do is so for each for each query and sorry for each attention head because you have multiple ones of these attention heads right in each layer so in each layer you have multiple of these heads for each head they look at over the course of the whole data set how do these softmax distributions look like so when you believe that this is a hopfield network and you believe that this converges in one step then if the patterns are well separated what we would expect is a distribution as we said like this okay there would be one dominant pattern that you retrieve you know that's what you want to retrieve that's what comes out but a bang you retrieve that accurate pattern anything else would mean that the hopfield network sort of failed right it wouldn't give you back one particular pattern so they have I think that's a pretty it's a pretty smart experiment they look how many bars do we need to add how many of these bars in the softmax distribution do we need to add to reach 90% so it depends a bit on the temperature of the softmax which is hard coded in attention mechanism bdb is 1 this squared over d so they say how many do we need to add to get to 0.9 to 90% of the mass of this distribution and if this is the hopfield network where you retrieve one pattern then one will be enough right one of these bars will probably be I don't know like 99% okay but there are other cases imagine the case where the patterns and the query you retrieve the spheres that it gives rise to are all like overlapping okay so what that will do is it won't converge to any particular pattern but the attractor space in this kind so you can imagine if you have two spheres that are apart from each other the update rule converges either so if it's closer to here it converge here if it's closer to here it'll converge here but if they are overlapping like this the energy landscape will actually make it such that it will neither if it starts somewhere it will neither converge to here nor to here it will actually converge to somewhere in the middle okay into the mean of the stored patterns and if we take that to the extreme what could be is it could be that the softmax distribution looks completely uniform okay which would basically mean that you know I don't care where my information comes from just average and this has its applications so if you for example want to make a sentiment classifier very cheap way to do that is to simply take pre-trained word embeddings like glove or word2vec you know assign each word word embedding and then just average the word embeddings okay and you count on the fact if there are a lot of kind of negative words in there like bad sad angry the word embedding kind of will you know reflect that and the average word embedding will point more into the bad direction and if there's a lot of happy words the average will point into the happy direction okay so there are applications of averaging information not caring particularly where it comes from and in that case what we'd expect is that this number and we'll call that so we'll call that the number K in this case it equals one but in this case K equals I guess N the number of inputs okay because we need well not maybe N but you know approximately we need almost all of them to to reach the 90% okay and there is an in-between and these are called these meta stable states where and the in-between is something like you'd have a couple of patterns here a couple here and a couple maybe here it's almost like a clustering like and these overlap and these overlap and these overlap but they don't overlap with each other which means that if you start somewhere here you would converge to the mean but not to the mean of all the patterns but just to the mean of these patterns and here here and here here so this this is like a clustering in latent space right so you can interpret these Hopfield update rules as somehow you know getting not going to a particular pattern but going to sort of a cluster and this is if you ask something like hey is there any adjective around right and all of these patterns they kind of overlap in that space in that query space of adjective they overlap and therefore the update rule would converge to sort of the mean which would basically say yes there is an adjective here right and the information would not be routed so that the distribution if we start here writing we converge to this the distribution would look something like small small small and then you'd have a couple of large ones all right you'd have like maybe two or three or four of large ones and these would exactly correspond to the patterns here so the information will be routed from all of those in that cluster to this particular note that asks the query okay these are these are what's called these meta stable states and what they do is they calculate over the entire data set this number K and here they show you the distribution so in these plots what you'll see is over the entire data set K goes into that direction so I guess let's go to Tiz here this this seems pretty easy so K is in this direction and this is simply the amount of like how so in each you let a data point run through it you measure K for that particular layer one you see this is layer one head four okay this is one layer one attention head and then you can see that the number K is distributed like this okay so contrast this to this head right here where it's a lot of weight on the number one or like very few numbers okay so these blue ones would be these are your typical like when you retrieve one particular pattern so this attention head we can sort of conclude in this particular attention head this is very specific it looks at its input it looks at its token and it decides what information do I want and it retrieves one particular thing from the other nodes okay whereas here it's more like kind of an averaging it's more like I want this kind of information and on average I don't even know what the sequence length is here I guess it's maybe 512 so of the 512 the median this number is always the median and median it collects information from 231 of them okay so you can see that this corresponds these green and orange ones correspond to these meta stable states where there's kind of an implicit clustering done in the in this space of attention whereas the blue ones they correspond to attention heads that ask for particular information retrieve one particular maybe few patterns and happy with that and the red ones here you can see that they often just average they just you know because K is so high means that I need all of the I need all of these bars to get to the 90% or I need almost all of them which basically means it's a uniform distribution right so it's like I don't care where information comes from just average whatever average I just want the average in some particular space and as we said that also has its uses interesting how this translate through so this here is as we go down the BERT model on the bottom of layer one you see there are a lot of these averaging operations going on so a lot of the heads are simply doing averaging and as you go up the layers the heads get more and more specific in the types of information they seek but then again in the last layers interestingly you get into a lot of these meta stable states again which I guess I get interpret this as you as you want I'm gonna leave this up to you but it sort of says like here you want kind of general patterns at the bottom and then the middle layers are kind of the logical workhorses so you look for very specific things in the input this is this is where I guess this is where the thinking happens so this is sort of pre-processing I'm just making stuff up here by the way this is this must be a no way true this is maybe thinking and this this here this might already be output again because you know after that you have language modeling or classification so this might already be like aggregating types of information this is how I sort of interpreted okay yeah so so this these these experiments are pretty pretty pretty interesting and here they have they do these are the last experiments for this paper they do an interesting experiment where they actually replace the attention heads by simply an average mechanism and later they actually replace them by Gaussians but in this case they simply average and they show that look if I replace layer one with this averaging the perplexity doesn't rise that much so it's pretty good even if I replace an entire layer here with averaging it perplexity goes more up and you can see the correspondence if you remember the previous plot the correspondence is pretty one-to-one with how much blue and green heads there are as contrast to how much red and orange ones there are so here you have lots of blue ones and you can see that the error kind of goes up and interestingly here you have more meta stable states at the end but still the perplexity goes up more so I guess you can only really replace the red ones with the averaging so this is always averaging in one particular layer and they go into more detail here where they say look this is this is layer 6 and this is layer 12 so this is one particular attention head from layer 6 and layer 12 and the updates don't be confused it goes in this direction okay I was confused at first and you can see right here this number K at first you know it's kind of spread out but then it pretty quickly converges to a very small number and there is this kind of point right here I don't know if the learning rates decrease I don't think so I think that's just kind of a a phase transition right here this is the blue line by the way the blue training line a phase transition where all of a sudden these just these attention heads they somehow decide okay this is the thing I want to specialize in this is the type of task I want like a sub task of linguistic sub task I want to specialize in and then they concentrate on one particular pattern per input so they are really specializing whereas in the last layer you see here that even during training they are sort of continuously learning so first they also do this averaging then they go into this meta stable region right this is this meta stable region K isn't one but also K isn't a very high number so they continuously learn and it's even indicative of this training might not be done here first of all and second of all it would be really interesting to see how this works out with you know sizes of transformers and like especially these these huge transformers just the fact that they can keep learning the more we train them might be you know be interpreted in the light of what kind of states they converge to and the fact that their attention heads I don't know how does this go on do they stay in the meta stable states because it makes sense to have meta stable states as I said it makes sense to kind of cluster things or are they simply into is this simply an intermediate step and if you go really far down they would actually also converge to the K equals one where they really specialize or if you do we need more attention heads for this I don't know it's just I think this is just the the beginning of kind of research in this direction I think just this kind of number K how it's how it's made it's pretty simple and apparently it's pretty pretty revealing so you know that's pretty cool so that was the paper and its experiments it's it's a pretty sizable paper as I said even the paper itself is ten pages and then there is this immune repertoire classification which I will like spend one minute looking at it so you have you have these set classifications so for each human you obtain a set of immune receptors and you simply obtain one label whether that human is immune to a particular disease or not and your task is kind and then a different human has a different set you have no idea which one of these things is responsible for it being for the human being for the human being immune or not in fact there is a you can't even decide based on these you can only decide based on like sub sequences of these and they might be in combination with each other so there might not be a single one responsible but like a combination but you don't have labels for the individual ones and you have different ones per human and they are different length all of this is just a giant giant task and you have many of them you have tens of thousands per human right so they build a system here where first they do these 1d convolutions to process the inside sequences and then they do this hop field attention mechanism or with with learned queries over these things and then they train on the output label and surprisingly that actually works even with tens of thousands of inside sequences and only one label for all of them and so they they achieve I guess favorable results compared to other baselines on this task using these hop field network which is pretty interesting but I let you look at that paper yourself so I hope this somehow made it a bit clear what happens here and it would actually be pretty interesting if we you know to see what happens if we just do maybe two rounds of these updates is this even desirable right is it desirable to run this to convergence is there something good about not running into convergence or does it actually not matter because it actually does converge in one step I don't know but have a look at the code it's pretty cool and I hope you enjoyed this video I'm sure you have many open questions as do I don't hesitate to ask me in the comments or join our discord as I said there are lots of helpful people on our discord and I'll see you next time bye bye
 Hi there. Today we'll look at hopfield networks is all you need by researchers
 from the Johannes Kepler University in Linz and the University of Oslo. So on
 high level this paper proposes a new type of hopfield networks that generalizes
 modern hopfield networks from binary patterns to continuous patterns and then
 shows that the retrieval update rule of these new hopfield networks is equivalent
 to the attention mechanism that's used in modern transformers. And it's actually a
 more general formulation of the attention mechanism and therefore it can
 be used to do kind of a variety of things to improve modern deep learning.
 And it also has a companion paper where it applies this to some kind of
 immunology research and achieves state-of-the-art in a task that is
 specifically suited to this type of attention. Alright let's dive in together
 we'll go over what this paper does what it proposes and so on. If you like pay if
 you like videos like this consider subscribing, you know sharing it out and
 I hope you're enjoying this. Alright also thanks to my discord community for you
 know very helpful bringing me up to speed on this paper. Super interesting
 discussions there. If you're not on our discord yet I invite you to join it's
 fun. Okay so what is a hopfield network? A hopfield network is a pretty kind of
 old style, an old conceptualization of a neural network. So in a hopfield network
 what your goal would be is you can conceptualize it as a bit of a neural
 network. So let's say we have five neurons or something like this. What your
 goal would be is to have a neural network where you can store so-called
 patterns and a pattern in this case would be a binary string of size 5. So
 for example 10100 or 11010 and you'd have a list of these patterns and what
 your goal would be is to store these patterns in the neural network such that
 and here you know we'll just consider everything to be sort of connected to
 everything else and what your goal would be in this is that you can kind of store
 patterns inside this neural network and you adjust the weights somehow. So this
 was as I said this was this was this is kind of an old model. You store you adapt
 the weights such that you store these patterns. And what does it mean for a
 pattern to be stored? If you have stored a pattern you will then be
 able to retrieve it and you retrieve a pattern in these kind of old style
 Hopfield networks by providing a partial pattern. So what you'll say is for
 example I want a pattern that starts with 110 and you give that to the
 network and there would be a so-called update rule and the update rule is kind
 of an internal rule. So let's just go through this. So here this 110 maybe
 this is 110 and then they would kind of send messages around so this update
 rule would somehow adjust the value of this and this neuron here to what's most
 compatible with the network weights and if the network weights have been
 adjusted correctly this will turn out then at the end of applying this update
 rule that this is a 1 and this is a 0 and therefore this pattern here is
 retrieved. Now had I input 101 at the beginning then the outcome would be
 different. Hopefully this pattern here would have been retrieved. So you
 can see the applications of this like you can have the first three digits as
 sort of a database key and then the last ones as sort of the value that you store
 along with it and then you can simply provide the first few. You can also
 provide you don't always have to provide three so this all depends. This is
 sort of an as I said an old conceptualization of neural networks so
 people were imagining that this is kind of how the brain works you know fire
 together wire together and also with research into this it turns out that you
 know you might think you know there's there's kind of five neurons so maybe I
 can store five different patterns you know accurately because if I store too
 many patterns right if I have many many many many patterns then I can't expect
 to be able to retrieve all the patterns again because some of them will just be
 so equal that you know many will start maybe with this and and I won't have a
 chance to to retrieve the one I want or the update rule will make a mistake so
 you might think this might be like five because I have five neurons or maybe ten
 because I have ten connections but it turns out that in modern hopfield
 networks with the appropriate update rule you can store exponentially many
 patterns in these networks exponentially many in the in the dimension of the in
 the dimension of the patterns and here I guess that would be the length of the
 pattern so this is a little bit surprising the kind of storage capacity
 of these networks and will this this paper here generalizes that to continuous
 to continuous states so what do we mean with continuous states I guess I mean
 continuous patterns so no longer is a pattern a binary string but a pattern
 now is a string of floating point numbers like 0.5 1.3 and so on and you
 know a string of floating or a sequence of floating point numbers is naturally
 depicted as a vector okay so our patterns are going to be different
 vectors that we store and you know in high dimensions that the the vectors
 will be kind of separated well from each other as long as we don't have too many
 but this paper shows that all these properties for the modern hopfield that
 works that hold for binary strings still hold if you go to these kind of
 continuous to these vector patterns that means you can store exponentially many
 patterns in the dimensions of the vector which is pretty surprising right
 because you'd think like you know after you have one vector per dimension that
 you know after that it might get a bit shaky but no you can actually store
 exponentially many that's pretty surprising and this paper is a lot about
 how to do that and the fact that that happens and so on so we've talked about
 update rules for these kind of hopfield networks and I haven't really specified
 what that is I've just said that you know I enter a pattern and then the
 network does something and outcomes outcomes the whatever the pattern that
 matches my query so this here is called a query you might already this is on
 purpose like the kind of overlap between the attention mechanism lingo and the
 hopfield network lingo we're going to conflate the two to kind of make clear
 where the two overlap if you don't know what an attention mechanism is or aren't
 familiar with it watch my video on attention is all you need once you watch
 that this video will make a lot more sense all right so in what the update
 rule does is specifically in the update rule that there isn't only one right
 there are many different proposals of hopfield networks and they all lead to
 different properties but what an update rule does ultimately is it minimizes
 what's called an energy so every type of hopfield network is associated with an
 energy function and this the energy function of the modern hopfield network
 for binary strings is this energy function right here so with X X is the
 pattern the pattern this is the kind of state of the hopfield network and these
 are the whatever is stored in the network and then the X here is the
 query that you enter into the network and then the energy here tells you this
 quantity you have to minimize this quantity in order to retrieve the
 pattern that you want okay now we are never directly working with the energy
 as such so what you could do is for example use backprop or something to
 use gradient descent to decrease the energy but usually along with an energy
 function comes an update function and the update function is what I've talked
 about here like you do something and then the network does something and then
 you get the pattern out what the network does is it minimizes its energy function
 and the update rule is made such that the corresponding energy function is
 minimized so the energy function is more like a theoretical consideration that
 you say okay here is my energy function of my hopfield network and the there
 will be a corresponding update rule that minimizes that energy function and if
 you use that update rule maybe multiple times then the energy function will be
 minimized and you will have retrieved your pattern or not if if you have too
 many patterns stored it might also fail right so they they say what the update
 rules are in the in the text here for the old hopfield networks but we're not
 really interested in the old ones we're interested in the ones that this paper
 cares about namely where the patterns that you store in the hopfield network
 are these vectors over our vector patterns and the query is also a vector
 pattern so you want to store all of these patterns into the hopfield network
 I'm gonna draw it like this here I'm gonna store it into the hopfield
 network and then after that you want to come up with a query and the query is
 like this and in the case of the binary strings we had something like well I
 sort of know half of my binary string now in the vector hopfield network it's
 more like well I sort of kind of know the direction that my vector should point
 in okay and you will read what you want to retrieve is the vector that has kind
 of a large inner product okay so if I enter this query into my hopfield
 network what I hope is that this vector here is retrieved now you see it's not
 exactly the same vector like they do point if I translate that here by I it's
 maybe something like this but so they are different but you want to say well I
 kind of know what I want I kind of want something like this and then the hopfield
 network would answer with oh I have something like this it's this right here
 okay so you that the connection to attention mechanism should become pretty
 pretty obvious right now but you know the to actually establish this formally
 is the kind of the point of this paper and you know it's pretty cool to see so
 they formulate this new energy right here this is the energy of this new
 continuous hopfield network specifically they have to have this term right here
 because they are now have continuous states and continuous queries this if
 you minimize the energy it basically means that your query can never you know
 go to infinity because you have the the query right here in the energy function
 the update rule is this right here and we'll look at that in a moment but
 remember the update rule is what you actually implement in code so if I if I
 have a query right here I plug it in here this is the state of my hopfield
 network and I apply this rule multiple times and out comes the kind of answer
 of the hopfield network to my question so the I input this and the out comes
 this after I update after I apply the update rule maybe multiple times right
 and interestingly you can already see that this here if you rewrite a bunch of
 these quantities so if you rewrite the beta here which is the softmax
 temperature in a way to be 1 over square root of D and if you take the query the
 psi here to be the query matrix and if you take the X here to be the key matrix
 then this is equivalent to the update or sorry the attention mechanism of a
 modern transformer so that's the point of the paper is that we can look at the
 transformer attention mechanism as a hopfield network and they have this
 interesting this interesting diagram at the end right here so the appendix you
 know this is typical I guess sep hoher I remember the cellul paper had like 60
 pages of machine proof appendix this also this is like 70 page appendix crazy
 but at the end of the appendix you'll find this diagram right here now usually
 in an attention mechanism you have whatever the the input is so you have an
 input right here so this is attention mechanisms or at least transformers they
 work on sequences or sets of objects and from these you'll generate three things
 you'll generate the you'll generate the queries the keys and the values now you
 can either generate the queries from the same objects which would be self
 attention or you can generate the queries from like a different object over
 here it doesn't it doesn't matter too much for our discussions but either you
 you know have a reference input or you have you know this kind of same input
 all the way and then what you do is use three different heads or three different
 matrices to transform that input into queries keys and values so I often
 conceptualize this as you have kind of your input set and each of the input
 sets outputs a key and also each one which would be a vector and also each
 one outputs a query so I often draw this here the same sequence and each one
 outputs a query and the query sort of the query is kind of a request for
 information so the key exposes sort of what exposes something about the input
 here so this could be a sentence down here this could be my cat is very pretty
 and the the the vector the key vector right here could encode something like
 this is a noun or this is an animal or anything like this right and the query
 here it could ask for for other things so for example since this is cat this
 vector right here the query vector is generated from that you know token cat
 now it could recognize that cat is a noun and it could ask the other nodes to
 basically say are there any adjectives around here because you know adjectives
 because it itself is a noun it's the object of the sentence right it could
 ask are there any kind of adjectives that describe the object because that
 would be naturally a thing to ask if you were the noun you would want to know are
 there any kind of modifiers for for me so it could output the query and the
 query here could mean you know this direction could mean adjectives and you
 see here the word pretty is an adjective so it itself would output a key that
 says by the way I'm an adjective right so if the cat asks then if this node
 asks for an adjective and this outputs the adjective vector then because the
 inner product between the two things is high this will be routed here so
 attention mechanism is basically information routing that's how I always
 describe it but in this paper we look at it more like these here are the patterns
 that are stored in a hopfield network and I by inputting a query and the dot
 product being the update rule of the hopfield network I retrieve from the
 hopfield network I retrieve the appropriate pattern that I ask for okay
 and then you know the values the values are simply a modification of the keys in
 this form but a lot of people also do keys and values to be the same thing
 yeah but the this routing of information happens here where you multiply the
 queries and the keys and then you put a softmax over them okay so if you just
 look from the perspective of a single node like this node here this cat node
 what it would do is it would inner product its own query vector with all
 of the key vectors right so it would build an inner product with all of these
 and then it would normalize it would put it through a softmax which will kind of
 give it a distribution right so here would give it like so this is actually
 matches because my well my is also very important for cat this this this is just
 an accident I did not plan this this here this also well many things match but
 but in our example we would just say that this last one it's not only higher
 it's also wider it matches very well right and so the information routing
 would route mostly information from this pretty token to the cat token which
 makes sense in our case right this is the attention mechanism now since if we
 are interpreting this as a hopfield network and the update rule here is the
 dot product you can actually think of applying this rule multiple times so
 what happens now if we and this is where this update rule comes in what happens
 if we take this distribution and we don't aggregate the values like usually
 we would aggregate the values by this distribution what if we aggregate the
 keys by this distribution okay what comes out well if we look at this and
 you know let's just assume that this key right here matches really well but the
 others also match a little bit what would come out would be a weighted
 average where a lot of weight is put on this particular key so what will turn
 out would be something like something that's very close to that key you can
 see I'm going to draw the old key here in green and I'm going to draw the old
 query in blue so you see that it's then whatever comes out is not the query but
 it's also not that only key that matches right it's kind of a weighted average
 but with that key dominating okay now since you know in a hopfield network
 what we would do is we would go again we would put this new thing the red thing
 instead of the query vector okay so we would use this aggregated keys this
 weighted average as a new query vector for that node right here so we duplicate
 that node over here I'll use that query vector again and do the same thing
 again okay inner product with all of the query vectors and now since this is
 already an aggregate of the query vectors what's going to happen of course
 the distribution that's going to come out is going to be weighted even more
 heavily into the direction so let's make it even wider into the direction of that
 key that matches okay and you can pretty clearly see if I do that iteratively
 then that will lead to a situation where everything is like very low except that
 one key will sort of dominate the distribution and ultra high and ultra
 wide okay and that's how that's exactly how a hopfield network works right I
 would input the query which would be sort of what I want right I kind of know
 what I want okay and then I apply this rule multiple times and with each time I
 refine refine refine until I decide on a pattern the hopfield network is made
 for pattern retrieval and these here are the patterns that I want to retrieve so
 here the patterns aren't kind of stored in the network beforehand but the
 patterns are also generated like in an attention layer so the keys are
 generated by the previous layer or by these matrices but that doesn't matter
 for the hopfield network update rule so you see here that the attention mechanism
 can be interpreted as simply one step making one step of this update rule but
 you can think of making actually multiple steps and retrieving the
 particular key so you know deciding on a sort of a hard routing of particular
 information now that only works if if there are no other vectors that are
 close to that particular key right so if the query is this and you know the way I
 drew it here you can see that there are many there is this one and this one and
 this one that matches so technically the way I drew it what would happen most
 likely is no many no matter how many times you apply your update rule it
 would sort of result in kind of the average of the three keys right so
 because they're all matching and they would all contribute to that weighted
 average of the query in the next step and then that means basically the
 conversions would be to something in the middle and that's going to be a central
 point of this paper in which situation we are so they call the first part is
 retrieving a single pattern and they call the second situation where you have
 multiple patterns that all match that are not well separated from each other
 they call this a meta stable state and it's going to be pretty interesting to
 look at transform like BERT language models and look at where they actually
 are are they actually operating in this single pattern retrieval mode or are they
 operating in the meta stable state mode all right so here you can see it in the
 diagram the only thing different this from a hop field network sorry from an
 attention mechanism is this branch right here so here you ask do you want to do
 multiple updates after you've you've multiplied the queries and the keys do
 you want to do multiple updates if yes so if you're in a this hop field network
 situation you want to do multiple updates then you go back as you can see
 and you do you use the keys together with the output of the softmax to
 generate a new query so this query queue here is now generated from the output
 here and the key so the keys are the same these are this is the same thing
 it's just put here twice okay this is exactly what we discussed okay I hope
 that's some somehow clear that these the the attention mechanism is simply a one
 step hop field network pattern retrieval algorithm with a particular update rule
 that is that is matches this energy function that they propose right here of
 course they do this you know particularly because the update rule that
 turns out is the transformer update rule but but I actually don't know if they
 backwards engineered the energy function to match the transformer or if they
 first came up with a continuous hop field networks and then just kind of
 discovered that it's like the transformer will maybe never find out
 okay so let's go there are a couple of theorems I believe there are four five
 theorems right here that show that kind of make some points about this about
 this stuff and we'll go through them won't go through the proofs or any you
 know super in-depth meaning but it's pretty cool to go through them and they
 are proved very rigorously as I said there's a 70 page appendix so have a
 look at that if you're up for it okay so they say here we have an update rule
 this is our update rule for our new hop field networks so the first theorem they
 serve is the update rule that we propose converges globally if we apply
 the update rule repeatedly the energy for t goes equals infinity and the
 energy will converge sorry the energy will converge to a fixed point this
 being a fixed point for t equals sorry for t goes to infinity yeah if this is
 a fixed point basically saying that if I apply this update rule here over and
 over and over again it will it will make this energy function converge to a fixed
 it will make this energy function converge don't want to say anything
 mistakenly here or claim too much but that basically connects the update rule
 to the energy okay so just showing like this really is the update rule for that
 particular energy function okay now as as itself it's not super duper
 interesting yet but now we get to theorem 2 so theorem 2 for the iteration
 that's the update rule that we just looked at we have we have that this
 convergence holds as t goes to infinity for some stationary point furthermore
 this quantity here goes to zero so that means this is the the update at t plus
 one and this is the update at t and the difference between them goes to zero so
 that means not only does the energy converge but the iterates themselves
 converge so the algorithm actually converges the individual updates of the
 algorithm so this e new at some point that will no longer change because the
 the norm between it and the previous one will go to zero you can see that either
 the sequence here converges or in the other case the set of limit points yada
 yada is a connecting subset this is a bit over the top here they say okay it
 can either converge to a point or it can converge to a connected subset but if
 the loss is finite then any sequence generated by the iteration equation 3
 converges to some fixed point so you know basically saying that here we oh
 this is not the loss I'm sorry no this is the domain never mind I am an idiot
 this is basically saying that this algorithm will converge okay and they
 define here what it means for a pattern to be stored and retrieved and that's
 for establishing what the kind of storage capacity for Hopfield network
 is so we've established that the update rule minimizes the appropriate energy
 and the update rule will converge at some point which means that we can you
 know if it converges we can retrieve the pattern that it converges to so now we
 define how many patterns can we actually store for that we need to know what does
 it mean for a pattern to be stored so we assume that we have patterns and these
 patterns are called X okay X I we have n different patterns each one is called X
 with a subscript we assume that around every pattern a sphere is given so how
 do we imagine this we have these patterns and this is this is just a space
 that they consider patterns of the on a sphere but we'll just conceptualize it as
 this will have a space there are patterns we want to store okay and we'll
 say around every pattern there is a sphere okay sphere like this and
 naturally the patterns are going to be there's going to be a notion of well
 separated patterns and you can imagine this a little bit like these spheres
 won't be touching each other if these spheres aren't touching each other that
 means that the patterns are kind of well separated and that means that if we
 initialize the query remember the query here is a vector that kind of sort of
 looks like a pattern and that means the query is kind of close to the pattern in
 some notion of distance so if we initialize the query somewhere in that
 sphere then it might if it converges to that sphere to that pattern then we
 retrieve the pattern okay now it gets a bit more complicated than this but not
 much more we say a pattern is stored if there is a single fixed point inside the
 sphere to which all points that start inside the sphere converge and none of
 the spheres intersect so the sphere of point I doesn't intersect with the
 sphere of point J so that's where we say all these spheres are non intersecting
 we say X is retrieved if the iteration equation 3 converged to the single fixed
 point in that sphere the retrieval error is the distance so you'll notice you
 have two things you have X I this is the actual pattern and you have X I star
 this is the retrieved pattern so these hopeful they don't always have to give
 you the same thing that you stored that's part of the the nature of
 continuous neural networks whatnot so for every sphere we say there is a
 pattern there is a sphere now we as pattern is stored if every I can start
 wherever I want in this sphere okay wherever I want it will always converge
 to a point that's inside the sphere and maybe that point isn't the pattern that
 I stored but actually this point right here but wherever I start I will always
 converge to that particular point if that's the case then I have stored this
 particular pattern now the fact is I don't retrieve this particular pattern I
 retrieve the blue thing but I can then define the error of retrieval the error
 of retrieval is simply the distance between the two things ideally this
 distance is very small right but you know we can't can't guarantee it now
 there are going to be theorems that deal exactly with this retrieval error but
 first you can see that here if if these spheres become larger you you can't
 accurately store a pattern anymore so this is the kind of ideal situation but
 there are also situations where these spheres you know if I have these
 patterns right here these spheres are so large kind of the the attractions of the
 patterns are so large that if I start let's say here then I don't converge to
 either of these two patterns I converge to like something in the middle I
 converge to maybe this point right here and that's going to be one of these
 meta stable states okay we're going to encounter situations like this but we're
 also going to encounter situations like this and the bottom thing isn't
 necessarily bad and that's what you have to keep in mind and yeah as I said we'll
 get to it but just keep this kind of sphere image in mind okay so first we'll
 just deal with the you know the up the top situation where we store patterns
 and then retrieve patterns so we'll assume a failure probability which is P
 and P is going to be no pretty pretty low for their example so they have P
 equals 0.001 you know like a 0.1 percent error probability of retrieving your
 pattern things like this and randomly chosen patterns on the sphere with
 radius M we define some constants yada yada yada then with probability 1 minus
 P the number of random patterns that can be stored and stored in the sense of
 having these spheres around them so that you can retrieve them accurately or at
 least you can retrieve something that's close to them is is bounded lower
 bounded by this quantity right here so there's the square root of P there is
 this constant C but then you see that D is in the exponent right here so that
 means it's exponential in the number of dimensions so that's that's pretty cool
 so if you add a dimension you exponentially increase the number of
 the number of patterns you can store and you know that's that is a kind of I
 mean it's it's been known for modern Hopfield networks with binary strings so
 it's not uber surprising but if you have you know it's not what you would imagine
 like that okay so they may give a few examples of these you have to
 accept these constants you know in a particular fashion such that this is
 given and so on but they say examples here are where C is something like 3
 and D is 20 so if you were to add a 21st dimension then your I guess storage
 capacity would increase by a factor of 3 which pretty cool alright so this is how
 many that we can store infinitely not sorry exponentially many patterns in
 these networks now they deal they say the next theorem states that the update
 rule typically converges after one update if the patterns are well
 separated okay so if we're in a situation where these patterns are well
 separated which is kind of like this but you can also imagine this in terms of
 dot products because we operate in the space of dot products so if the patterns
 are well separated that sort of means that they all kind of sort of point away
 from each other and this notion of separation is going to be captured by
 this quantity right here this is the separation of example of pattern I which
 is just the inner product with itself minus the maximum inner product with any
 other pattern and this quantity is going to be large when no other pattern is
 close to it so when the separation is large then the update rule the retrieval
 rule of calculating you know I have a query calculate the inner product with
 all of those then I reweigh all of the patterns by that inner product by the
 softmax then I use that new thing as a query again and so on as we discussed it
 will converge to the closest pattern but this theorem says it actually converges
 pretty fast and here I have my problems with saying that it converges after one
 step typically converges after one update because that you know genuinely
 depends on a lot of constants as we'll see but it does converge exponentially
 fast in this separation constant as a theorem for says with query psi after
 one update the distance of the new point to the fixed point is exponentially
 small in the separation Delta I the precise bound using Jacobian and its
 value in the mean value theorem are the following so here you can see this is
 the distance between the updated psi after one step and the and the fixed
 point right here this is what it converges to is going to be the
 distance as it was before times this thing right here so you can see since
 this is a this is a multiplicative update and in this Jacobian so this is
 expanded down here this is this you can see here you have the you have this
 sorry yeah this is this so this is bounded by that you have the exponent
 the exponential function negative this separation right here so the higher the
 separation the faster this algorithm converges okay to say that it converges
 after one step is you know it might be a bit of bragging I don't know if this is
 a common thing if you have like an exponential convergence that you are
 allowed to say it's after one step I'm not sure especially what I'm not sure
 about is that you have n here as linear constants in that factor okay so if you
 if you and that's what they do in their code so if you look at their code and
 the codes available which is pretty cool it's implemented in pytorch as a
 general module that can you can just drop in so this is not only for
 transformers this is for you can replace like LSTM's you can replace pooling
 mechanisms you can you know do a whole bunch of stuff in their paper in the
 company paper they do this multi instance learning with giant sets on
 using these hopfield layers so pretty pretty cool this code is definitely
 worth kind of checking out and maybe you want to replace some stuff with you but
 the question is how many of these update steps should you do right because we
 looked at the diagram at least in the attention mechanism it seems like you
 have attention layers right you have a transformer and the transformer consists
 of you know you have this input right here and you go through layer layer
 layer layer layer and in each layer there's contained in it and one of these
 attention mechanism right this entire thing is in this layer okay and now if
 you interpret this as a hopfield network and you want to do multiple steps that
 means you go this branch right here so in each layer potentially you do
 multiple steps of these things so for whatever computational constraints
 transformers had already this will certainly make it worse but also you
 need to decide how many steps you want to do now you can hard code that of
 course but they say you should do these steps until this norm here until the
 norm between the old and the new is small enough so where is that so you
 can't measure how close you are to the convergence points right because you
 don't know in practice but you can measure how far you're away you can
 measure where did we have it you can measure this quantity right here that's
 something you can measure how far two iterates are apart so what you'll simply
 do is you'll measure that and if that is small enough then you'll you'll stop but
 that I guess is very related to this so how if you we've already proven it
 converges to this X star so I guess we can approximate this quantity right here
 with the quantity above and that tells you how many updates you need to do and
 that quantity is linear not only linear but actually here quadratic in n I don't
 care you know yes it's exponential in the separation but it's quadratic in n
 and if I've learned anything from kind of my fast code courses is that constants
 actually matter when you're not dealing with infinity with an infinite number of
 steps so the number of the number of steps you need to do I guess will
 depend on the sequence length in a quadratic fashion so I'm not sure you
 can always claim this is convergence in one step now I might be super mistaken
 here and none of this will can none of this actually makes a difference in the
 in the light of the exponential decay here but I would just I'm just a bit
 worried saying this usually converges in one step it's clear I guess why they do
 it right because the attention mechanism in transformers is a one-step application
 of this rule and this here is kind of a theoretical justification for
 interpreting this precisely as a hopfield network because you'd say well
 in a hopfield network you would do multiple steps but wait wait we can
 actually prove that even if you interpret it as a hopfield network you
 it can it usually converges after one step so what you're actually doing in a
 transformer is applying a hopfield network update rule to convergence so
 yeah I'm not yeah I might be bickering on a high level here luxury problems
 theorem five then says so theorem four is how fast does this converge theorem
 five the last theorem right here says that the retrieval error of a pattern
 then so this is the this is what you converge to and this is what you've
 stored is bounded by again something that's exponential in the separation
 right here as you can see okay so that was the theorem so if we go quickly
 through them again theorems one and two deal with the convergence of this
 algorithm and the fact that it actually minimizes the proposed energy then
 theorem three says you can store exponentially many patterns in terms of
 the dimension of your space and theorems four and five say that this update rule
 will converge exponentially fast after after one step if you believe that and
 the retrieval error will also go down exponentially fast with the number of
 update steps that you do okay that sounds pretty pretty pretty good but
 we've heard it it's very dependent on how well separated these patterns are
 and it turns out that is you know at least in transformers they aren't
 always well separated and that might be on purpose remember the the states here
 the patterns aren't pre stored like in a classic hopfield network but the
 patterns if you interpret an attention mechanism as this are also generated by
 the network itself so the pattern matrix that you retrieve from and the query are
 generated by the attention mechanism in this case as I said this is applicable
 to many many more domains than just this but yeah so there's another slight
 modification that you have to do to make this actually equivalent to an attention
 mechanism and that is you'll have to recast the value because usually what
 you'll do is you have some sort of input and then you make queries keys and
 values from that using different heads the only thing to make it formally
 equivalent is you have to make the values generated from the keys so the
 keys give rise to the values as you can see right here that you first multiply
 with the key matrix and then with the value matrix I think that's you know
 that I don't I doubt that this will will change anything if you if you the only
 way that could really change anything is if this matrix here would be super low
 rank like collapse the space of into like very few dimensions which the value
 matrix wouldn't do so you know but just letting you know that the technical
 equality requires this slight modification okay now we said that it
 might not you know be that this is always super well separate and you
 retrieve a single pattern and that's what they research here in a pre trained
 BERT model so they take a pre trained BERT model from I guess from hugging
 face and they run they just run a data set through it and what they do is so
 for each for each query and sorry for each attention head because you have
 multiple ones of these attention heads right in each layer so in each layer you
 have multiple of these heads for each head they look at over the course of the
 whole data set how do these softmax distributions look like so when you
 believe that this is a hopfield network and you believe that this converges in
 one step then if the patterns are well separated what we would expect is a
 distribution as we said like this okay there would be one dominant pattern
 that you retrieve you know that's what you want to retrieve that's what comes
 out but a bang you retrieve that accurate pattern anything else would
 mean that the hopfield network sort of failed right it wouldn't give you back
 one particular pattern so they have I think that's a pretty it's a pretty
 smart experiment they look how many bars do we need to add how many of these bars
 in the softmax distribution do we need to add to reach 90% so it depends a bit
 on the temperature of the softmax which is hard coded in attention mechanism
 bdb is 1 this squared over d so they say how many do we need to add to get to 0.9
 to 90% of the mass of this distribution and if this is the hopfield network
 where you retrieve one pattern then one will be enough right one of these bars
 will probably be I don't know like 99% okay but there are other cases imagine
 the case where the patterns and the query you retrieve the spheres that it
 gives rise to are all like overlapping okay so what that will do is it won't
 converge to any particular pattern but the attractor space in this kind so you
 can imagine if you have two spheres that are apart from each other the update
 rule converges either so if it's closer to here it converge here if it's closer
 to here it'll converge here but if they are overlapping like this the energy
 landscape will actually make it such that it will neither if it starts
 somewhere it will neither converge to here nor to here it will actually
 converge to somewhere in the middle okay into the mean of the stored patterns and
 if we take that to the extreme what could be is it could be that the softmax
 distribution looks completely uniform okay which would basically mean that you
 know I don't care where my information comes from just average and this has its
 applications so if you for example want to make a sentiment classifier very cheap
 way to do that is to simply take pre-trained word embeddings like glove
 or word2vec you know assign each word word embedding and then just average the
 word embeddings okay and you count on the fact if there are a lot of kind of
 negative words in there like bad sad angry the word embedding kind of will
 you know reflect that and the average word embedding will point more into the
 bad direction and if there's a lot of happy words the average will point into
 the happy direction okay so there are applications of averaging information
 not caring particularly where it comes from and in that case what we'd expect
 is that this number and we'll call that so we'll call that the number K in this
 case it equals one but in this case K equals I guess N the number of inputs
 okay because we need well not maybe N but you know approximately we need almost
 all of them to to reach the 90% okay and there is an in-between and these are
 called these meta stable states where and the in-between is something like you'd
 have a couple of patterns here a couple here and a couple maybe here it's almost
 like a clustering like and these overlap and these overlap and these overlap but
 they don't overlap with each other which means that if you start somewhere here
 you would converge to the mean but not to the mean of all the patterns but just
 to the mean of these patterns and here here and here here so this this is like
 a clustering in latent space right so you can interpret these Hopfield update
 rules as somehow you know getting not going to a particular pattern but going
 to sort of a cluster and this is if you ask something like hey is there any
 adjective around right and all of these patterns they kind of overlap in that
 space in that query space of adjective they overlap and therefore the update
 rule would converge to sort of the mean which would basically say yes there is
 an adjective here right and the information would not be routed so that
 the distribution if we start here writing we converge to this the
 distribution would look something like small small small and then you'd have a
 couple of large ones all right you'd have like maybe two or three or four of
 large ones and these would exactly correspond to the patterns here so the
 information will be routed from all of those in that cluster to this particular
 note that asks the query okay these are these are what's called these meta stable
 states and what they do is they calculate over the entire data set this
 number K and here they show you the distribution so in these plots what
 you'll see is over the entire data set K goes into that direction so I guess
 let's go to Tiz here this this seems pretty easy so K is in this direction
 and this is simply the amount of like how so in each you let a data point run
 through it you measure K for that particular layer one you see this is
 layer one head four okay this is one layer one attention head and then you
 can see that the number K is distributed like this okay so contrast this to this
 head right here where it's a lot of weight on the number one or like very
 few numbers okay so these blue ones would be these are your typical like
 when you retrieve one particular pattern so this attention head we can sort of
 conclude in this particular attention head this is very specific it looks at
 its input it looks at its token and it decides what information do I want and
 it retrieves one particular thing from the other nodes okay whereas here it's
 more like kind of an averaging it's more like I want this kind of information
 and on average I don't even know what the sequence length is here I guess it's
 maybe 512 so of the 512 the median this number is always the median and median
 it collects information from 231 of them okay so you can see that this
 corresponds these green and orange ones correspond to these meta stable states
 where there's kind of an implicit clustering done in the in this space of
 attention whereas the blue ones they correspond to attention heads that ask
 for particular information retrieve one particular maybe few patterns and happy
 with that and the red ones here you can see that they often just average they
 just you know because K is so high means that I need all of the I need all of
 these bars to get to the 90% or I need almost all of them which basically means
 it's a uniform distribution right so it's like I don't care where information
 comes from just average whatever average I just want the average in some
 particular space and as we said that also has its uses interesting how this
 translate through so this here is as we go down the BERT model on the bottom of
 layer one you see there are a lot of these averaging operations going on so a
 lot of the heads are simply doing averaging and as you go up the layers
 the heads get more and more specific in the types of information they seek but
 then again in the last layers interestingly you get into a lot of
 these meta stable states again which I guess I get interpret this as you as you
 want I'm gonna leave this up to you but it sort of says like here you want kind
 of general patterns at the bottom and then the middle layers are kind of the
 logical workhorses so you look for very specific things in the input this is
 this is where I guess this is where the thinking happens so this is sort of
 pre-processing I'm just making stuff up here by the way this is this must be a
 no way true this is maybe thinking and this this here this might already be
 output again because you know after that you have language modeling or
 classification so this might already be like aggregating types of information
 this is how I sort of interpreted okay yeah so so this these these experiments
 are pretty pretty pretty interesting and here they have they do these are the
 last experiments for this paper they do an interesting experiment where they
 actually replace the attention heads by simply an average mechanism and later
 they actually replace them by Gaussians but in this case they simply average and
 they show that look if I replace layer one with this averaging the perplexity
 doesn't rise that much so it's pretty good even if I replace an entire layer
 here with averaging it perplexity goes more up and you can see the
 correspondence if you remember the previous plot the correspondence is
 pretty one-to-one with how much blue and green heads there are as contrast to how
 much red and orange ones there are so here you have lots of blue ones and you
 can see that the error kind of goes up and interestingly here you have more
 meta stable states at the end but still the perplexity goes up more so I guess
 you can only really replace the red ones with the averaging so this is always
 averaging in one particular layer and they go into more detail here where they
 say look this is this is layer 6 and this is layer 12 so this is one
 particular attention head from layer 6 and layer 12 and the updates don't be
 confused it goes in this direction okay I was confused at first and you can see
 right here this number K at first you know it's kind of spread out but then
 it pretty quickly converges to a very small number and there is this kind of
 point right here I don't know if the learning rates decrease I don't think so
 I think that's just kind of a a phase transition right here this is the blue
 line by the way the blue training line a phase transition where all of a sudden
 these just these attention heads they somehow decide okay this is the thing I
 want to specialize in this is the type of task I want like a sub task of
 linguistic sub task I want to specialize in and then they concentrate on one
 particular pattern per input so they are really specializing whereas in the last
 layer you see here that even during training they are sort of continuously
 learning so first they also do this averaging then they go into this meta
 stable region right this is this meta stable region K isn't one but also K
 isn't a very high number so they continuously learn and it's even
 indicative of this training might not be done here first of all and second of all
 it would be really interesting to see how this works out with you know sizes of
 transformers and like especially these these huge transformers just the fact
 that they can keep learning the more we train them might be you know be
 interpreted in the light of what kind of states they converge to and the fact
 that their attention heads I don't know how does this go on do they stay in the
 meta stable states because it makes sense to have meta stable states as I
 said it makes sense to kind of cluster things or are they simply into is this
 simply an intermediate step and if you go really far down they would actually
 also converge to the K equals one where they really specialize or if you do we
 need more attention heads for this I don't know it's just I think this is
 just the the beginning of kind of research in this direction I think just
 this kind of number K how it's how it's made it's pretty simple and apparently
 it's pretty pretty revealing so you know that's pretty cool so that was the paper
 and its experiments it's it's a pretty sizable paper as I said even the paper
 itself is ten pages and then there is this immune repertoire classification
 which I will like spend one minute looking at it so you have you have these
 set classifications so for each human you obtain a set of immune receptors and
 you simply obtain one label whether that human is immune to a particular disease
 or not and your task is kind and then a different human has a different set you
 have no idea which one of these things is responsible for it being for the
 human being for the human being immune or not in fact there is a you can't even
 decide based on these you can only decide based on like sub sequences of
 these and they might be in combination with each other so there might not be a
 single one responsible but like a combination but you don't have labels for
 the individual ones and you have different ones per human and they are
 different length all of this is just a giant giant task and you have many of
 them you have tens of thousands per human right so they build a system here
 where first they do these 1d convolutions to process the inside
 sequences and then they do this hop field attention mechanism or with with
 learned queries over these things and then they train on the output label and
 surprisingly that actually works even with tens of thousands of inside
 sequences and only one label for all of them and so they they achieve I guess
 favorable results compared to other baselines on this task using these hop
 field network which is pretty interesting but I let you look at that
 paper yourself so I hope this somehow made it a bit clear what happens here
 and it would actually be pretty interesting if we you know to see what
 happens if we just do maybe two rounds of these updates is this even desirable
 right is it desirable to run this to convergence is there something good
 about not running into convergence or does it actually not matter because it
 actually does converge in one step I don't know but have a look at the code
 it's pretty cool and I hope you enjoyed this video I'm sure you have many open
 questions as do I don't hesitate to ask me in the comments or join our discord
 as I said there are lots of helpful people on our discord and I'll see you
 next time bye bye