A bio-inspired bistable recurrent cell allows for long-lasting memory (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper gru lstm schmidhuber bistable bistability neurons biological spiking tanh stable attractor fixed points memory memorize sparse long sequence history storage remember rnn recurrent neural network gated recurrent unit forget backpropagation biologically inspired
Even though LSTMs and GRUs solve the vanishing and exploding gradient problems, they have trouble learning to remember things over very long time spans. Inspired from bistability, a property of biological neurons, this paper constructs a recurrent cell with an inherent memory property, with only minimal modification to existing architectures.

OUTLINE:
0:00 - Intro & Overview
1:10 - Recurrent Neural Networks
6:00 - Gated Recurrent Unit
14:40 - Neuronal Bistability
22:50 - Bistable Recurrent Cell
31:00 - Neuromodulation
32:50 - Copy First Benchmark
37:35 - Denoising Benchmark
48:00 - Conclusion & Comments

Paper: https://arxiv.org/abs/2006.05252
Code: https://github.com/nvecoven/BRC

Abstract:
Recurrent neural networks (RNNs) provide state-of-the-art performances in a wide variety of tasks that require memory. These performances can often be achieved thanks to gated recurrent cells such as gated recurrent units (GRU) and long short-term memory (LSTM). Standard gated cells share a layer internal state to store information at the network level, and long term memory is shaped by network-wide recurrent connection weights. Biological neurons on the other hand are capable of holding information at the cellular level for an arbitrary long amount of time through a process called bistability. Through bistability, cells can stabilize to different stable states depending on their own past state and inputs, which permits the durable storing of past information in neuron state. In this work, we take inspiration from biological neuron bistability to embed RNNs with long-lasting memory at the cellular level. This leads to the introduction of a new bistable biologically-inspired recurrent cell that is shown to strongly improves RNN performance on time-series which require very long memory, despite using only cellular connections (all recurrent connections are from neurons to themselves, i.e. a neuron state is not influenced by the state of other neurons). Furthermore, equipping this cell with recurrent neuromodulation permits to link them to standard GRU cells, taking a step towards the biological plausibility of GRU.

Authors: Nicolas Vecoven, Damien Ernst, Guillaume Drion

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we're looking at a bio-inspired bistable recurrent cell, allows for long-lasting memory by Nicolas Vécovin, Damien Ernst and Gion Drion of the University of Liège. This paper here is not a paper that wants to push state-of-the-art on anything, it is a paper that takes a concept from the biological research on actual neurons, which is the bistability property and tries to introduce it to recurrent neural networks. And on toy data or small data they show that this has the interesting property that these recurrent neural networks can then remember important things for much much longer than our current recurrent architectures can do. I believe this is a very interesting paper and it's a nice refresher from the whole state-of-the-art number pushing papers. So dive in with me to explore this. If you like content like this also consider subscribing if you aren't and sharing it out and leaving a like and a comment if you have any sort of comments. They basically say recurrent neural networks provide state-of-the-art performance in a wide variety of tasks that will require memory, which is true. So we have these recurrent neural networks and what the recurrent neural networks do is they're basically... So a classic recurrent neural network goes something like this. There is a hidden state at time step T and there is a sequence of inputs that you have to work with. So we'll call them x1, x2, x3, x4 and so on. And then at some point you have to provide an output. This could be at every single time step or sometimes it's just at the end you have to provide an output y. So for example this here could be a piece of text and you need to decide whether or not that piece of text, maybe it's an email, whether or not that's spam. This could be a time series of a patient in an ICU and you need to decide whether or not to give some medication to the patient. So the applications of this are very wide and any sort of series data will do. So there's this hidden state and at each time step this hidden state is updated to a new hidden state. So this call this h0. It's updated to a new hidden state by incorporating the input. So somehow the input x and the previous hidden state are made into a new hidden state. And then the next input is taken in and by using this hidden state a new hidden state is made and so on. So one property here is that the newest hidden state always only depends on the previous hidden state and it doesn't really directly depend on like the hidden state to before itself. It only depends on the hidden state right before itself and the input that corresponds to it. So this is the information flow. The other important property here is that these connections that make a hidden state into the next hidden state and also that incorporate the input, they're always the same. So these functions here that incorporate the input, they're always the same in each time step. So the parameters are shared between them and the same goes for the functions here that transform one hidden state into the next hidden state. Of course there is a joint function between the two that actually produces the next hidden state. So these weights are all shared and for each time step and that's what makes the network recurrent. So we call the single time step here, we call that a recurrent cell. And the question now is how do you construct a recurrent cell? Usually recurrent neural networks they run into this problem of either gradient explosion or vanishing gradients because usually this here, if you are into neural networks you know this, this is a weight matrix multiplied by the previous hidden state and if you just multiply the same weight matrix over and over and over again it pretty much depends on the singular value of that weight matrix if the top singular value is higher than one then the signal is going to explode and if it's lower than one the signal is going to fade over time and there's pretty much nothing you can do. So classic RNNs have looked like this right here. So the next hidden state is a nonlinear function G and G can be some non-linearity like a sigmoid or a hyperbolic tangent but it's a function of the current input and the last hidden state by simply matrix multiplying these two things by some weight matrices and then adding them up. So that's what we've just looked at. Now this is problematic as I said because of the vanishing or exploding gradients and therefore people have come up with methods to solve this and you might know things like LSTMs and GRUs that are used to solve this. Now these cells here are much more complicated than the standard cell that we saw here but they also are much more effective because they don't have this vanishing or exploding gradient problems. Their promise is that they can remember things for longer because they allow the gradient to flow without these problems during backpropagation. Now how does one of these look? In this paper they mainly look at the GRU, the gated recurrent unit which is a simpler version of the LSTM. The LSTM is slightly more complex but the principles are the same. So they look at the GRU right here. What does the GRU do? These are the formulas for the GRU and we're going to try to deconstruct these formulas. So as you can see the inputs are the same. The inputs are going to be this points input, this time steps input and the last hidden state. Those are all the quantities that we need and we need to output somehow the next hidden state. The last hidden state is then used to predict the y by the way in all of these cases. So first we need to calculate two things called Z and R and both of them are the same. They're multiplying these two things by weight matrices and then running them through a sigmoid non-linearity. Let's do that. Let's say we have the last hidden state here and we have Xt here. So first we're going to calculate the Zt and the Rt from that. Now every one of these arrows right here is a multiplication by a weight matrix. So every one of these arrows is transforming the input and let's also let's join this into a sigmoid node and that gives you Zt and let's join these into a sigmoid that gives you Rt. Okay so far so good. Now we need to combine all of this in this last line right here. So you can see that the Z thing here sort of acts as a switch. So Z is the result of a sigmoid and therefore it's between 0 and 1 and here this is the Hadamard product. This is the element-wise product between vectors which sort of means this is like a gating. This is like a switch. If it's 1 it selects this quantity and if it's 0 it selects the quantity over here and of course it can be between 0 and 1 but those are the ends of the spectrum. So Z is a switch that selects between the last hidden state. So let's draw that right here. So the last hidden state goes here and is one of the options of the output. And the option is given by Z. So Zt let's draw a switch like this maybe. So Zt is responsible for modulating this switch right here. Okay this gives you the next hidden state. You see Zt modulates that switch so Ht is the one possibility that the switch can select. What's the other possibility? The other possibility is this quantity right here which is a hyperbolic tangent of whatever that is. So that is combined of X. So let's go from the back right here. Tanh. What's the input to the tanh? It's two things. First of all the X is an input to the tanh so we can draw directly a line from here. The X modulated. Every arrow as you might remember can be a function. Not all arrows are functions like this arrow right here is not a function it's just an arrow. Maybe that's confusing. You get what I mean. And the next thing is R times the hidden the last hidden state or the last hidden state modulated by this matrix. So R is acting as another gate. R can be again between 0 and 1 because it's the result of a sigmoid. So this hidden state will also go here. It will be modulated running out of colors here. It will be modulated by R here as a sort of gate. So R can either close or open this gate right here and then that's fed into the tanh. So it's a rather complicated setup as you can see right here. So let's analyze this. First of all the hidden state is either the last hidden state or it is something new and that's modulated by this Z right here. And Z is calculated from the hidden state and the current input. So this allows the cell to basically look at the hidden state is sort of the information of what happened so far and the current input is the new information that it gets from the sequence. And it sort of gets to look at these two things and decides do I even want to update my hidden state? If not I can just select this path right here and then nothing happens to the hidden state. The next hidden state will be exactly the same as the last hidden state. If it decides, if it thinks wow this new thing in the sequence that's actually important I should remember that. Because remember the task of the network sometimes is to remember things from the sequence. I think we drew this over here. So if this is an email and we want to detect whether it's spam then this word in the sequence right here might be really important because it would say something like gold, like buy gold. These two things might be buy gold and you need to remember that in the hidden state because the only way that information from X is going to flow to Y is through the hidden states. So you would want at this point you would want to remember this input in the hidden state so you would actually want to update the hidden state. And then this here might be not as important so you might want to say I don't want to I don't want to I still want my hidden state to be the old hidden state. So Z is that gate that allows us to do this. If we decide to update the hidden state then what do we do? Again if we decide to update the hidden state we can we can we will incorporate the new input but we will we can also decide to mix that how to mix that new input with the old hidden state. So if we decide to update the hidden state we don't simply discard the old hidden state because the old hidden state will still have a path to be sort of still there to be remembered but it's a longer path and it needs to go through this thing here and through this thing here. So this thing here decides which of the old hidden state pass through. So at each you can see right here this is an element-wise product this R is going to be between 0 and 1 at each point in the vector. So at each point in the vector the R decides is this worth remembering or not? And if it's not worth remembering then this is going to be 0 and that that position of the old hidden state is going to be 0 and then that's going to be forgotten and that's the opportunity for the hidden state to incorporate new information because then it can delete this old information and then it can incorporate the new input and that will result then on this path on the new hidden state. So there's two major things. First we can decide whether or not to even incorporate new information that's achieved by the Z gate and then we can decide which parts of the old hidden state if we want to update it which parts to forget that's the R gate and how to update it is then basically a result of the weight matrix that's associated with this function right here. Alright so that's the gated recurrent unit and it works a lot better than the classic RNNs. So having said that they now turn to this property of neuronal by stability that happens in actual neurons. So this here is sort of a model of a neuron with this property. Now forget everything we said about GRUs we're just going to look at this right now. What happens in a neuron usually is you have this is a single neuron you have input synapses from other neurons so these are connections coming from other neurons into you. They are accumulated right here. Usually they are just in a classic model of a neuron they're just summed up you would sum up all these all these input signals and then you decide you'd run it through like a step function. So if the sum of all the things is smaller than a particular threshold the output would be just nothing and if it's higher than a particular threshold then the output of the neuron would be sort of a firing of the neuron. This can be weighted and whatnot but in this case it's just a function of the inputs and that gives you your input signal. So this is like this is your input signal to the neuron. Now there is this property right here that makes it interesting. The signal goes out here and is integrated. This is an integrator and that's going to be in the output signal but there's this connection this back connection right here and that means that the signal that comes out at time step t is going to be fed back into the signal and actually added to the signal before itself and sort of self modulating right the signal comes out is sent back is added to this input and then sent through again and this here is just an integrator that's integrating everything that's happening. So if you if you look a bit closer you'll see that there is a minus here so it's actually not added it's subtracted and there is an F here which means that this is a nonlinear function. Now if this weren't a nonlinear function we can just sort of try or let's say this is a monotonic function we can sort of try to estimate what happens. If all of this right here is very high it's a high number big number this will be a big number then this sum will be a big number this output will be a big number what happens is this here will be a big number this is monotonic so it will also be a big number and that means it will subtract that big number so that means when whenever the neuron is going to to be very excited this feedback would actually push it back now when it is not very excited so when it's a very low number very negatively excited then the feedback would work in the exact opposite direction this will be very negative this will be very negative and this here would push it towards the positive so this neuron somehow self stabilizes over time to this to the zero point right here and that's simply if this F is the identity function right now so you can sort of see how this property works now we'll make it a bit more complicated in that we'll assume that this F here is not the identity function but let's say they have it somewhere but this right here so the F F of V post is this here it's V post minus alpha tan H of V post or is this the entire F yes that's this this thing right here if this is the case if this is that this if this is the signal minus the tan H then something very very very interesting happens and that that's depending on this alpha right here in that if this alpha is between if the alpha is between 0 and 1 then we simply have our monotonic function so here you can see how big V post is so how big the output signal is here that's the experiment we made before and here you can see what the feedback signal is okay or the integrator the integrated signal maybe this is in the wrong place and maybe F is just minus the tan H I'm not sure but in any case the way they build it after in the GRU it's pretty explicit so this is the thing we said before namely if if the signal is very high then this signal here will be high as well and because it's subtracted right here it's it's going to push the signal back towards zero again if this is lower than zero then this thing here will also be lower than zero and because it's subtracted it's going to push the signal towards zero so this thing here is the stable point it will always push it back towards zero however if we change the function and we change just the parameter alpha to be 1.5 a very different thing happens that you can see right here then it turns out if your output signal is very is very high the same thing happens is going to put me push back but if your output signal is between zero and this point right here there is a regime where actually even though the output signal is positive you will be pushed towards this point right here and therefore there is there are these two stable points right now and the stable point basically means if you deviate if the signal deviates from it it's going to be pushed back towards that point and you can see these two stable points they're not at zero they're actually at at these two points here and that's pretty interesting because that means you can potentially remember things with the cell right an output signal of zero it's basically not informative but here you can be in either the state here or in the state here and the little little perturbations will still keep you in that state so you could potentially be in this state right here as an output and the cell will just keep updating itself and be stable and always output that signal right here and then you could go ahead and if you then provide some huge input signal right here you could potentially throw this over to the other side over this hill and then it would stabilize at this point so this is sort of a way to remember things within these biological cells pretty cool now this here is a non filled out circle that means it's an unstable point it's technically stable in the sense that if you're exactly at zero you will remain at zero but if you perturb even a little bit you will go if you perturb a bit you will go away from it okay I hope this sort of property is right is clear and why this is so fascinating because we can use this these this fact that the stable points are not at zero and are more than just one stable point for remembering things and they're now trying to fill this into the gated recurrent unit so they call this the bi-stable recurrent cell BRC and the formulas are these right here maybe a little smaller come on can't zoom anymore okay it looks almost the same as the GRU so the formulas are these this and this so let's analyze the differences to the GRU the first most striking difference is that a lot of weight matrices here have become single numbers so or single vectors this here used to be a weight matrix and this used to be a matrix multiplication and you'll see this sort of throughout whenever the last hidden state is incorporated into these things then you'll see that it is no longer a weight matrix but is in fact a in a product with a vector a element wise product and that has a reason namely what they want to model is individual neurons so on a biological level and neuron can only feed back onto itself if there is a layer of neurons right here they can only each feed back onto themselves whereas in a recurrent neural network my hidden vector my hidden state is a vector and if I transform this into the next hidden state or any quantity let's say I transform this H into this R right here and this R is a vector too then any interaction is possible so any cell any entry in the vector here can influence any other vector because there's a big weight matrix in the middle they want to leave this away they want to model this as close as possible to actual layers of neurons and therefore they say okay the input X can you know be distributed to all the neurons because technically the input comes from some other neurons down here and they can all have connections to these neurons but these feedbacks we only really observe them in individual neuron this feedback cycle so that's why they model these recurrent weight products by just element wise products with vectors and then the second difference you again see that there is this switch right here this C switch and the C switch is like before it's a sigmoid with where combine the the output and the previous hidden state there's nothing new here so this switch is the same the cell has the possibility of letting in new information or just ignoring the new current information the XT the second thing is here and this is the same as well right the tanh this is a combination of the new information it's in case we want to let in the new information of the new information and you need to decide what things of the old information to forget or remember now the difference here is in this a so this a used to be again this sigmoid of the combination and now it's just slightly different it used to be sigmoid now it's one plus tanh this is a very very slight modification it's tanh because tanh is between minus one and one instead of zero and one like the sigmoid and the one plus makes it such that this is between zero and two and we've seen before that this critical behavior there is two regimes to these functions when it's between zero and one this behaves like a classic gated recurrent unit like a classic GRU but when it's between one and two then you have that exact behavior that we saw before of the bi-stability okay so depending on what the a is if the a is zero to one it's a classic cell and if the a is one to two it's a bi-stable cell and the network can decide by itself what it wants to do because here it has it can actually learn how to do that all right so this is the only change the only change really apart from it only being individual neurons feeding back on themselves is that now this is no longer between zero and one with the sigmoid this is now between zero and two because it's one plus the tan h very simple change but the effect of this is pretty pretty cool so they do some here is like a schematic drawing of this if this a is between zero and one again you have this stable state that's at zero but it's if it's between one and two you have two stable states at two non zero points and again this we already saw this but now this is for I believe this this recurrent cell this by modal recurrent cell not for the neuron itself and here they give an example of what happens when you run this particular signal this particular time series through a cell like this while fixing the C and the a parameters so now the C and their a parameter aren't learned they're just fixed and you see what happens now as you can see the the blue should be a classic the classic behavior so in this blue case what happens you see right here this C is moderately low so we saw the C is the switch of whether to leave in old information or take up new information if it's low it means we want to take up new information this is reasonably low and that's why when the signal goes up here the blue line goes up as well and when the signal goes down the blue line goes down again and so on so the blue line pretty straightforwardly follows the signal right here okay now in contrast to this the red line is over this threshold so a is fixed at 1.5 C is still at 0.2 so again when this line goes up then this line goes up but because this is near a this stable point if it goes down again it doesn't appear to go down enough it sort of remembers that state it was in it doesn't go down with the signal only now that it goes down even further it's over this threshold so we were in this situation now and the first bump down was only like to here and that pushed it up again but now it jumps over here because the signal is even lower and then the cell sort of switches to another state as you can see here it goes down but then this bump here is not enough to bring it up again so it kind of remains in this state so you can see the it sort of remembers the input and small deviations or small changes in signal don't manage to throw it away from that only larger things only it needs to go very the signal needs to go very much down in order for it to change state so that's pretty cool that there's this remembering behavior and now remember in the actual implementation these C and A parameters this C and this A right here aren't fixed they are also determined by the cell itself and therefore the cell can decide by itself when it wants to remember things how hard it wants to remember things and so on so we're going to check this out in an actual implementation so there's this one last modification they make where they say okay they tried this and it doesn't really work because it works sometimes but there is this issue of these neurons connecting only back on themselves which really makes the model much less powerful than a classic recurrent cell it's closer to biology but it's much less powerful and there is this property they say of a neuromodulation where technically in real neurons the one neuron here could influence another neuron by modulating these A and C parameters okay these A and C parameters this is called neuromodulation so there are interconnections between the neurons that influence how much other neurons remember and forget things so they decide let's model that and lo and behold we're now back to having weight matrices right here so this this is sort of they say this is a not really a super biologically plausible way of implementing neuromodulation but it's sort of it's an easier way and it brings us closer to the G back to the GRU and yeah so now the only difference to the GRU is that the fact that here there was a sigmoid now it's a 1 plus tan H okay I find this this pretty cool so now also the only difference here is this property of bi stability this is the only difference and now we can actually compare so let's compare they first give they do these sort of benchmarks which are they're pretty pretty neat so they have this first benchmark where it's the copy first input benchmark I'm having some trouble here moving this paper around with my fingers so the copy first input benchmark is simply a time series in this benchmark the network is presented with a one-dimensional time series of T time steps and the each entry is a is a random number after receiving the last time step the network output value should approximate the very very first input step okay so all the network needs to do is remember the first thing it sees and that's that should be learnable right that should be learnable because you can so you can it's not specified whether the zero with hidden state the initial hidden state is given into the network but technically it doesn't matter because it can just learn whatever that is I can learn to have a designated bit in this hidden state so this hidden state is of size 100 I believe one designated bit in the hidden state of whether it has already encountered the first thing or not if it has not encountered it means that it's at the first time step therefore it should incorporate the new information into the hidden state and if and also set this bit and then for each subsequent step it can see I've already set this bit and it can simply close that gate that makes it incorporate new information so it should be able to carry this information all the way to the end by simply always closing that gate after the first step and what happens in this so as you can see when the result is all the results up here so this is after three so they train it for 300,000 gradient descent iterations and you can see that when these time steps when the series are pretty small the LSTM or the GRUs tend to perform well but you can see that these BRCs they they don't tend to perform poorly they're just performing worse right it's zero it's still the 0.01 regime or something like this of error however when you go up to like 300 steps then you can see the GRUs and the LSTM they start to fail because they are not made explicitly to remember for that long they don't have this by stability property whereas now these things excel you can see they're still pretty low and at 600 steps these things completely fail they completely forget the input so and the NBRC at least is still able to remember the first thing pretty pretty well and yeah so the second one is no this is the first experiment still the copy input benchmark you can see right here that even at this three at this 100 thing where the GRU still learns it it learns it much much later than the BRC which learns it pretty fast only here when the when it's only five when that series are only five steps long does the GRU slightly outperform the BRC so the general notion here is that these classic cells are more powerful in like classic tasks whereas these things are shining whenever these things fail because they can't remember things for very long so they're not these new cells are not state-of-the-art yet possibly there are still some modifications to be made we've had a pretty long history of optimizing GRUs and LSTMs they haven't always worked so well as they do now because we kind of know how to handle them and I expect if these cells here take off especially these NBRC then with time will be as proficient at handling them and they will probably become on par or even outperform the LSTMs or GRUs on every day like on all the tasks and then be especially good on tasks where you have to remember things but for now they're outperformed by LSTMs and GRUs okay so the second thing is a more interesting experiment the denoising benchmark where they say the the copy input benchmark is interesting as a means to highlight the memorization capacity of the recurrent neural network but it does not tackle its ability to successfully exploit complex relationships between different elements of the input signal to predict the output so they have a new benchmark in the denoising benchmark the network is presented with a two-dimensional time series of t time steps five different time steps are sampled uniformly with okay and are communicated in the network okay I'll just tell you what's going on so this new time series is two dimensional in the lower dimension you simply have a bunch of random numbers like five eight two nine actually these are numbers sampled from a uniform Gaussian or so so they're not actually five eight two and nine but you can imagine it like this five eight two nine three four zero two and so on and in the second dimension you have a negative one I believe almost anywhere and then at some points you have a one negative one again and then you have a one and a negative one again and at the last point of the sequence you'll have a zero and so the zero is simply a marker that it's the end of the sequence what the network needs to do is it needs to output all the elements so the output of the network should be in this case should be nine four so all the elements where there was a one in order okay so it remember what it needs to learn it needs to learn to every time it sees a one in the second dimension it needs to take the first dimension put it somehow into the hidden state and then carry that hidden state forward and sees a one again it needs to take the second thing also put it into the hidden state but not override the first thing it put into the hidden state like if it were to just realize I need to put this into the hidden state then it would almost surely override the previous information so it needs to be able to say I've already kind of in my H is going to be a vector of a hundred dimensions it needs to be able to say well I've already stored a bunch of stuff in that part of the vector maybe I should store that thing here over here so this is fairly complex things to remember and technically GRU's and LSTMs are able to do it but as we'll see they're not as much the results are in this table where you can clearly see that whenever the n so the n the n is a parameter that is how far how far in this direction are these ones so when n is 0 the ones can be anywhere but when n here is like 5 that means that the last five ones surely don't contain a one that means only the first whatever a L minus L minus 5 contain the one so the higher this number n is the harder the task because your learning signal is way way farther away from the from what's when you get the output so you can see when the n is low then the GRU's and the LSTMs they perform pretty well but also these cells perform pretty well they're just not performing as well however when the task gets harder and you actually need to learn a sparse signal over a long period of time where in between you don't get any signal the GRU's and the LSTMs fail while the BRC's would still be able to learn these kinds of things so that's that's fairly cool now it's if from a researcher's perspective I wonder if they just first tried this task you know as I described it and then they discovered like ah crap they can still do it and like okay how can we make it such that there's a difference okay let's actually make the task harder like this and then they did that I wonder if they always had the idea with the end here or just introduced this after after it they they failed to produce a difference in the first place I'm not sure but they have they have another benchmark but they basically show that these cells are actually good can incorporate this information can reason about what they need to remember and whatnot and in the end they also have this sequential MNIST where they just feed an MNIST digit digit by digit and at the end I think that the output of the neural network needs to be the class of the of the MNIST digit and again here they have a parameter called N black which means that so they have an MNIST digit it's like a three they unroll it to a single vector right they feed this one by one into the recurrent network and then after that they attach a certain number of just empty pixels black pixels and after that the network needs to predict the Y you can see if they ask the network the class of the digit immediately after it's done then the G are using the LSTM perform fairly well as do the BRCs but if you attach a whole bunch of these black pixels remember an MNIST digit has some seven sorry seven hundred and eighty four maybe entries so attaching 300 black pixels is quite significant in in terms of the length of these sequences and then the GRUs and the LSTMs they can't learn they can't learn to ignore these things because the learning signal is just too far away right here but these things they can because they can exploit this by stability property and remember things again I wonder how this came to be it seems pretty funny but the last thing they do is they investigate what happens in their cells and this I feel is the most interesting part and they do this on this denoising benchmark so the task we've looked at before where you need to remember five randomly selected numbers that are indicated by the second dimension here they show a sequence where the five numbers occur at 3100 246 at 300 and at 376 so these are the five positions where the sequence indicates that the network should remember the thing in the first dimension and then output they analyze two things they analyze the proportion of bi-stable neurons so basically they analyze these out these a quantities and they analyze how many of the neurons in the layer have an a that's higher than one which means that they are in this bi-stable mode and also they analyze what's the average value of C so see if you remember if this is high it means it doesn't let in new information and if this is low it means it lets in new information if you first look at the C you can see that every single time when the second dimension indicates that this is one of the inputs to remember this the network drops immediately drops the C values the different colors here are different layers they build they have a recurrent network has multiple layers of these cells as is usual in the recurrent neural networks so this C as you can see it goes up pretty quickly but then as soon as one of these inputs appear the C drops which basically means that the network realizes it now must let in the new information and then it immediately shoots back up makes it seem like so the network says okay as long as so all of these inputs here they have the negative one in the second dimension right so it recognizes it says there's no reason for me to incorporate that information it's not important and as soon as the second input comes it immediately shoots down again now you can see this here is the last layer of the network the highest layer so sort of the highest abstractive information and you can see that from input to input this value of C gets higher and higher and these spikes as they go down but they go down to a higher and higher point which you know is is the fact that it recognizes it needs to let in new information but it lets in less and less new information the more things it needs to remember so not only does it recognize wait I need to remember this it also recognizes but I probably shouldn't shouldn't you know completely forget what I had previously because I it is important for me to remember these previous things so that's a pretty cool demonstration the fact that these go down at the input and the fact that generally they go up every time after a new input is incorporated into the hidden state this basically this shows that the or this is a pretty good indication that what they're saying is really happening right okay the second thing shows almost the same it shows how many of these neurons are actually in their bi-stable mode and you can also see right here that especially in the last layer you can see that the number of neurons in the bi-stable mode goes up and up and up and up after each of these steps and these spikes here correspond to always the points where they have to let in new information okay cool so I find that I find this to be pretty cool and I find this last experiment to be the coolest where they can actually show look here there's a pretty good indication that the thing we we build does what we say it does they also actually have a proof here of the bi-stability when this a is higher than one I won't go through this right here but if you want you can look at that I'm excited to see what happens with these kinds of architectures in the future because it seems to be a pretty minor modification and maybe with a little bit of more modification or if we sort of just tune this a little bit and kind of figure out what we have to do to make these things actually compete with the classic GRUs and LSTMs in regimes where a long memory isn't necessary I feel this could be a you know kind of a standard building block in the recurrent neural network toolkit even though it's been sort of outperformed by transformers in previous years alright that was it for me and I hope you had fun with this paper I invite you to check it out and bye bye
 Hi there, today we're looking at a bio-inspired bistable recurrent cell,
 allows for long-lasting memory by Nicolas Vécovin, Damien Ernst and
 Gion Drion of the University of Liège. This paper here is not a paper that wants
 to push state-of-the-art on anything, it is a paper that takes a concept from the
 biological research on actual neurons, which is the bistability property and
 tries to introduce it to recurrent neural networks. And on toy data or small
 data they show that this has the interesting property that these
 recurrent neural networks can then remember important things for much much
 longer than our current recurrent architectures can do. I believe this
 is a very interesting paper and it's a nice refresher from the whole
 state-of-the-art number pushing papers. So dive in with me to explore this. If you
 like content like this also consider subscribing if you aren't and sharing it
 out and leaving a like and a comment if you have any sort of comments.
 They basically say recurrent neural networks provide state-of-the-art
 performance in a wide variety of tasks that will require memory, which is true.
 So we have these recurrent neural networks and what the recurrent neural
 networks do is they're basically... So a classic recurrent neural network goes
 something like this. There is a hidden state at time step T and there is a
 sequence of inputs that you have to work with. So we'll call them x1, x2, x3, x4
 and so on. And then at some point you have to provide an output. This could be
 at every single time step or sometimes it's just at the end you have to provide
 an output y. So for example this here could be a piece of text and you need to
 decide whether or not that piece of text, maybe it's an email, whether or not
 that's spam. This could be a time series of a patient in an ICU and you need to
 decide whether or not to give some medication to the patient. So the
 applications of this are very wide and any sort of series data will do. So
 there's this hidden state and at each time step this hidden state is updated
 to a new hidden state. So this call this h0. It's updated to a new
 hidden state by incorporating the input. So somehow the input x and the previous
 hidden state are made into a new hidden state. And then the next input is taken
 in and by using this hidden state a new hidden state is made and so on. So one
 property here is that the newest hidden state always only depends on the
 previous hidden state and it doesn't really directly depend on like the
 hidden state to before itself. It only depends on the hidden state right before
 itself and the input that corresponds to it. So this is the information flow. The
 other important property here is that these connections that make a hidden
 state into the next hidden state and also that incorporate the input, they're
 always the same. So these functions here that incorporate the input,
 they're always the same in each time step. So the parameters are shared
 between them and the same goes for the functions here that transform
 one hidden state into the next hidden state. Of course there is a joint
 function between the two that actually produces the next hidden state. So these
 weights are all shared and for each time step and that's what makes the network
 recurrent. So we call the single time step here, we call that a recurrent cell.
 And the question now is how do you construct a recurrent cell? Usually
 recurrent neural networks they run into this problem of either gradient
 explosion or vanishing gradients because usually this here, if you are into
 neural networks you know this, this is a weight matrix multiplied by the previous
 hidden state and if you just multiply the same weight matrix over and over and
 over again it pretty much depends on the singular value of that weight matrix
 if the top singular value is higher than one then the signal is going to explode
 and if it's lower than one the signal is going to fade over time and there's
 pretty much nothing you can do. So classic RNNs have looked like
 this right here. So the next hidden state is a nonlinear function G and G can be
 some non-linearity like a sigmoid or a hyperbolic tangent but it's a
 function of the current input and the last hidden state by simply matrix
 multiplying these two things by some weight matrices and then adding them up.
 So that's what we've just looked at. Now this is problematic as I said because of
 the vanishing or exploding gradients and therefore people have come up with
 methods to solve this and you might know things like LSTMs and GRUs that are
 used to solve this. Now these cells here are much more complicated than the
 standard cell that we saw here but they also are much more effective because
 they don't have this vanishing or exploding gradient problems. Their
 promise is that they can remember things for longer because they allow the
 gradient to flow without these problems during backpropagation. Now how does one
 of these look? In this paper they mainly look at the GRU, the gated recurrent unit
 which is a simpler version of the LSTM. The LSTM is slightly more complex but
 the principles are the same. So they look at the GRU right here. What does the GRU
 do? These are the formulas for the GRU and we're going to try to
 deconstruct these formulas. So as you can see the inputs are the same. The inputs
 are going to be this points input, this time steps input and the last hidden
 state. Those are all the quantities that we need and we need to output
 somehow the next hidden state. The last hidden state is then used to predict
 the y by the way in all of these cases. So first we need to calculate two things
 called Z and R and both of them are the same. They're multiplying these two
 things by weight matrices and then running them through a sigmoid non-linearity.
 Let's do that. Let's say we have the last hidden state here and we
 have Xt here. So first we're going to calculate the Zt and the Rt from that.
 Now every one of these arrows right here is a multiplication by a weight matrix.
 So every one of these arrows is transforming the input and let's also
 let's join this into a sigmoid node and that gives you Zt and let's join these
 into a sigmoid that gives you Rt. Okay so far so good. Now we need to combine all
 of this in this last line right here. So you can see that the Z thing here sort
 of acts as a switch. So Z is the result of a sigmoid and therefore it's between
 0 and 1 and here this is the Hadamard product. This is the element-wise
 product between vectors which sort of means this is like a gating. This is like
 a switch. If it's 1 it selects this quantity and if it's 0 it selects the
 quantity over here and of course it can be between 0 and 1 but those are the
 ends of the spectrum. So Z is a switch that selects between the last hidden
 state. So let's draw that right here. So the last hidden state goes here and is
 one of the options of the output. And the option is given by Z. So Zt
 let's draw a switch like this maybe. So Zt is responsible for
 modulating this switch right here. Okay this gives you the next hidden state. You
 see Zt modulates that switch so Ht is the one possibility that the switch can
 select. What's the other possibility? The other possibility is this quantity right
 here which is a hyperbolic tangent of whatever that is. So that is combined of
 X. So let's go from the back right here. Tanh. What's the input to the tanh?
 It's two things. First of all the X is an input to the tanh so we can draw
 directly a line from here. The X modulated. Every arrow as you might
 remember can be a function. Not all arrows are functions like this
 arrow right here is not a function it's just an arrow. Maybe that's confusing.
 You get what I mean. And the next thing is R times the hidden the last hidden
 state or the last hidden state modulated by this matrix. So R is
 acting as another gate. R can be again between 0 and 1 because it's the result
 of a sigmoid. So this hidden state will also go here. It will be modulated running
 out of colors here. It will be modulated by R here as a sort of gate. So R can either
 close or open this gate right here and then that's fed into the tanh. So it's
 a rather complicated setup as you can see right here. So let's analyze this.
 First of all the hidden state is either the last hidden state or it is something
 new and that's modulated by this Z right here. And Z is calculated from the hidden
 state and the current input. So this allows the cell to basically look at
 the hidden state is sort of the information of what happened so far and
 the current input is the new information that it gets from the sequence. And it
 sort of gets to look at these two things and decides do I even want to update my
 hidden state? If not I can just select this path right here and then nothing
 happens to the hidden state. The next hidden state will be exactly the same as
 the last hidden state. If it decides, if it thinks wow this new thing in the
 sequence that's actually important I should remember that. Because
 remember the task of the network sometimes is to remember things from
 the sequence. I think we drew this over here. So if this is an email and we want
 to detect whether it's spam then this word in the sequence right here might be
 really important because it would say something like gold, like buy gold. These
 two things might be buy gold and you need to remember that in the hidden
 state because the only way that information from X is going to flow to
 Y is through the hidden states. So you would want at this point you would want
 to remember this input in the hidden state so you would actually want to
 update the hidden state. And then this here might be not as important so you
 might want to say I don't want to I don't want to I still want my hidden
 state to be the old hidden state. So Z is that gate that allows us to do
 this. If we decide to update the hidden state then what do we do? Again if we
 decide to update the hidden state we can we can we will incorporate the new
 input but we will we can also decide to mix that how to mix that new input with
 the old hidden state. So if we decide to update the hidden state we don't
 simply discard the old hidden state because the old hidden state will still
 have a path to be sort of still there to be remembered but it's a
 longer path and it needs to go through this thing here and through this thing
 here. So this thing here decides which of the old hidden state pass through. So at
 each you can see right here this is an element-wise product this R is going to
 be between 0 and 1 at each point in the vector. So at each point in the vector
 the R decides is this worth remembering or not? And if it's not worth
 remembering then this is going to be 0 and that that position of the old hidden
 state is going to be 0 and then that's going to be forgotten and that's the
 opportunity for the hidden state to incorporate new information because
 then it can delete this old information and then it can incorporate
 the new input and that will result then on this path on the new hidden state.
 So there's two major things. First we can decide whether or not to even
 incorporate new information that's achieved by the Z gate and then we can
 decide which parts of the old hidden state if we want to update it which
 parts to forget that's the R gate and how to update it is then basically a
 result of the weight matrix that's associated with this function right here.
 Alright so that's the gated recurrent unit and it works a lot better than the
 classic RNNs. So having said that they now turn to this property of neuronal
 by stability that happens in actual neurons. So this here is sort of a model
 of a neuron with this property. Now forget everything we said about GRUs
 we're just going to look at this right now. What happens in a neuron usually is
 you have this is a single neuron you have input synapses from other neurons
 so these are connections coming from other neurons into you. They are
 accumulated right here. Usually they are just in a classic model of a neuron
 they're just summed up you would sum up all these all these input signals and
 then you decide you'd run it through like a step function. So if the sum of
 all the things is smaller than a particular threshold the output would be
 just nothing and if it's higher than a particular threshold then the output of
 the neuron would be sort of a firing of the neuron. This can be weighted and
 whatnot but in this case it's just a function of the inputs and that gives
 you your input signal. So this is like this is your input signal to
 the neuron. Now there is this property right here that makes it interesting.
 The signal goes out here and is integrated. This is an integrator and that's
 going to be in the output signal but there's this connection this back
 connection right here and that means that the signal that comes out at time
 step t is going to be fed back into the signal and actually added to the signal
 before itself and sort of self modulating right the signal comes out
 is sent back is added to this input and then sent through again and this here is
 just an integrator that's integrating everything that's happening. So if you
 if you look a bit closer you'll see that there is a minus here so it's actually
 not added it's subtracted and there is an F here which means that this is a
 nonlinear function. Now if this weren't a nonlinear function we can just sort of
 try or let's say this is a monotonic function we can sort of try to estimate
 what happens. If all of this right here is very high it's a high number big
 number this will be a big number then this sum will be a big number this
 output will be a big number what happens is this here will be a big number this
 is monotonic so it will also be a big number and that means it will subtract
 that big number so that means when whenever the neuron is going to to be
 very excited this feedback would actually push it back now when it is not
 very excited so when it's a very low number very negatively excited then the
 feedback would work in the exact opposite direction this will be very
 negative this will be very negative and this here would push it towards the
 positive so this neuron somehow self stabilizes over time to this to the zero
 point right here and that's simply if this F is the identity function right
 now so you can sort of see how this property works now we'll make it a bit
 more complicated in that we'll assume that this F here is not the identity
 function but let's say they have it somewhere but this right here so the F
 F of V post is this here it's V post minus alpha tan H of V post or is this
 the entire F yes that's this this thing right here if this is the case if this
 is that this if this is the signal minus the tan H then something very very very
 interesting happens and that that's depending on this alpha right here in
 that if this alpha is between if the alpha is between 0 and 1 then we simply
 have our monotonic function so here you can see how big V post is so how big the
 output signal is here that's the experiment we made before and here you
 can see what the feedback signal is okay or the integrator the integrated signal
 maybe this is in the wrong place and maybe F is just minus the tan H I'm not
 sure but in any case the way they build it after in the GRU it's pretty explicit
 so this is the thing we said before namely if if the signal is very high
 then this signal here will be high as well and because it's subtracted right
 here it's it's going to push the signal back towards zero again if this is
 lower than zero then this thing here will also be lower than zero and because
 it's subtracted it's going to push the signal towards zero so this thing here
 is the stable point it will always push it back towards zero however if we
 change the function and we change just the parameter alpha to be 1.5 a very
 different thing happens that you can see right here then it turns out if your
 output signal is very is very high the same thing happens is going to put me
 push back but if your output signal is between zero and this point right here
 there is a regime where actually even though the output signal is positive you
 will be pushed towards this point right here and therefore there is there are
 these two stable points right now and the stable point basically means if you
 deviate if the signal deviates from it it's going to be pushed back towards
 that point and you can see these two stable points they're not at zero they're
 actually at at these two points here and that's pretty interesting because that
 means you can potentially remember things with the cell right an output
 signal of zero it's basically not informative but here you can be in either
 the state here or in the state here and the little little perturbations will
 still keep you in that state so you could potentially be in this state right
 here as an output and the cell will just keep updating itself and be stable and
 always output that signal right here and then you could go ahead and if you then
 provide some huge input signal right here you could potentially throw this
 over to the other side over this hill and then it would stabilize at this point
 so this is sort of a way to remember things within these biological cells
 pretty cool now this here is a non filled out circle that means it's an
 unstable point it's technically stable in the sense that if you're exactly at
 zero you will remain at zero but if you perturb even a little bit you will go if
 you perturb a bit you will go away from it okay I hope this sort of property is
 right is clear and why this is so fascinating because we can use this
 these this fact that the stable points are not at zero and are more than just
 one stable point for remembering things and they're now trying to fill this into
 the gated recurrent unit so they call this the bi-stable recurrent cell BRC
 and the formulas are these right here maybe a little smaller come on can't
 zoom anymore okay it looks almost the same as the GRU so the formulas are
 these this and this so let's analyze the differences to the GRU the first most
 striking difference is that a lot of weight matrices here have become single
 numbers so or single vectors this here used to be a weight matrix and this used
 to be a matrix multiplication and you'll see this sort of throughout whenever the
 last hidden state is incorporated into these things then you'll see that it is
 no longer a weight matrix but is in fact a in a product with a vector a element
 wise product and that has a reason namely what they want to model is
 individual neurons so on a biological level and neuron can only feed back onto
 itself if there is a layer of neurons right here they can only each feed back
 onto themselves whereas in a recurrent neural network my hidden vector my
 hidden state is a vector and if I transform this into the next hidden
 state or any quantity let's say I transform this H into this R right here
 and this R is a vector too then any interaction is possible so any cell any
 entry in the vector here can influence any other vector because there's a big
 weight matrix in the middle they want to leave this away they want to model this
 as close as possible to actual layers of neurons and therefore they say okay the
 input X can you know be distributed to all the neurons because technically the
 input comes from some other neurons down here and they can all have connections
 to these neurons but these feedbacks we only really observe them in individual
 neuron this feedback cycle so that's why they model these recurrent weight
 products by just element wise products with vectors and then the second
 difference you again see that there is this switch right here this C switch and
 the C switch is like before it's a sigmoid with where combine the the output
 and the previous hidden state there's nothing new here so this switch is the
 same the cell has the possibility of letting in new information or just
 ignoring the new current information the XT the second thing is here and this is
 the same as well right the tanh this is a combination of the new information
 it's in case we want to let in the new information of the new information and
 you need to decide what things of the old information to forget or remember now
 the difference here is in this a so this a used to be again this sigmoid of the
 combination and now it's just slightly different it used to be sigmoid now it's
 one plus tanh this is a very very slight modification it's tanh because tanh is
 between minus one and one instead of zero and one like the sigmoid and the
 one plus makes it such that this is between zero and two and we've seen
 before that this critical behavior there is two regimes to these functions when
 it's between zero and one this behaves like a classic gated recurrent unit like
 a classic GRU but when it's between one and two then you have that exact
 behavior that we saw before of the bi-stability okay so depending on what
 the a is if the a is zero to one it's a classic cell and if the a is one to two
 it's a bi-stable cell and the network can decide by itself what it wants to do
 because here it has it can actually learn how to do that all right so this
 is the only change the only change really apart from it only being
 individual neurons feeding back on themselves is that now this is no longer
 between zero and one with the sigmoid this is now between zero and two because
 it's one plus the tan h very simple change but the effect of this is pretty
 pretty cool so they do some here is like a schematic drawing of this if this a is
 between zero and one again you have this stable state that's at zero but it's
 if it's between one and two you have two stable states at two non zero points
 and again this we already saw this but now this is for I believe this this
 recurrent cell this by modal recurrent cell not for the neuron itself and here
 they give an example of what happens when you run this particular signal this
 particular time series through a cell like this while fixing the C and the a
 parameters so now the C and their a parameter aren't learned they're just
 fixed and you see what happens now as you can see the the blue should be a
 classic the classic behavior so in this blue case what happens you see right
 here this C is moderately low so we saw the C is the switch of whether to leave
 in old information or take up new information if it's low it means we want
 to take up new information this is reasonably low and that's why when the
 signal goes up here the blue line goes up as well and when the signal goes down
 the blue line goes down again and so on so the blue line pretty
 straightforwardly follows the signal right here okay now in contrast to this
 the red line is over this threshold so a is fixed at 1.5 C is still at 0.2 so
 again when this line goes up then this line goes up but because this is near a
 this stable point if it goes down again it doesn't appear to go down enough it
 sort of remembers that state it was in it doesn't go down with the signal only
 now that it goes down even further it's over this threshold so we were in this
 situation now and the first bump down was only like to here and that pushed it
 up again but now it jumps over here because the signal is even lower and
 then the cell sort of switches to another state as you can see here it
 goes down but then this bump here is not enough to bring it up again so it kind
 of remains in this state so you can see the it sort of remembers the input and
 small deviations or small changes in signal don't manage to throw it away
 from that only larger things only it needs to go very the signal needs to go
 very much down in order for it to change state so that's pretty cool that there's
 this remembering behavior and now remember in the actual implementation
 these C and A parameters this C and this A right here aren't fixed they are also
 determined by the cell itself and therefore the cell can decide by itself
 when it wants to remember things how hard it wants to remember things and so
 on so we're going to check this out in an actual implementation so there's this
 one last modification they make where they say okay they tried this and it
 doesn't really work because it works sometimes but there is this issue of
 these neurons connecting only back on themselves which really makes the model
 much less powerful than a classic recurrent cell it's closer to biology
 but it's much less powerful and there is this property they say of a neuromodulation
 where technically in real neurons the one neuron here could influence another
 neuron by modulating these A and C parameters okay these A and C parameters
 this is called neuromodulation so there are interconnections between the neurons
 that influence how much other neurons remember and forget things so they
 decide let's model that and lo and behold we're now back to having weight
 matrices right here so this this is sort of they say this is a not really a
 super biologically plausible way of implementing neuromodulation but it's
 sort of it's an easier way and it brings us closer to the G back to the GRU and
 yeah so now the only difference to the GRU is that the fact that here there
 was a sigmoid now it's a 1 plus tan H okay I find this this pretty cool so now
 also the only difference here is this property of bi stability this is the
 only difference and now we can actually compare so let's compare they first
 give they do these sort of benchmarks which are they're pretty pretty neat so
 they have this first benchmark where it's the copy first input benchmark I'm
 having some trouble here moving this paper around with my fingers so the copy
 first input benchmark is simply a time series in this benchmark the network is
 presented with a one-dimensional time series of T time steps and the each
 entry is a is a random number after receiving the last time step the network
 output value should approximate the very very first input step okay so all the
 network needs to do is remember the first thing it sees and that's that
 should be learnable right that should be learnable because you can so you can
 it's not specified whether the zero with hidden state the initial hidden state is
 given into the network but technically it doesn't matter because it can just
 learn whatever that is I can learn to have a designated bit in this hidden
 state so this hidden state is of size 100 I believe one designated bit in the
 hidden state of whether it has already encountered the first thing or not if it
 has not encountered it means that it's at the first time step therefore it
 should incorporate the new information into the hidden state and if and also
 set this bit and then for each subsequent step it can see I've already
 set this bit and it can simply close that gate that makes it incorporate new
 information so it should be able to carry this information all the way to
 the end by simply always closing that gate after the first step and what
 happens in this so as you can see when the result is all the results up here so
 this is after three so they train it for 300,000 gradient descent iterations and
 you can see that when these time steps when the series are pretty small the
 LSTM or the GRUs tend to perform well but you can see that these BRCs they
 they don't tend to perform poorly they're just performing worse right it's
 zero it's still the 0.01 regime or something like this of error however
 when you go up to like 300 steps then you can see the GRUs and the LSTM they
 start to fail because they are not made explicitly to remember for that long
 they don't have this by stability property whereas now these things excel
 you can see they're still pretty low and at 600 steps these things completely
 fail they completely forget the input so and the NBRC at least is still able to
 remember the first thing pretty pretty well and yeah so the second one is no
 this is the first experiment still the copy input benchmark you can see right
 here that even at this three at this 100 thing where the GRU still learns it it
 learns it much much later than the BRC which learns it pretty fast only here
 when the when it's only five when that series are only five steps long does the
 GRU slightly outperform the BRC so the general notion here is that these
 classic cells are more powerful in like classic tasks whereas these things are
 shining whenever these things fail because they can't remember things for
 very long so they're not these new cells are not state-of-the-art yet possibly
 there are still some modifications to be made we've had a pretty long history of
 optimizing GRUs and LSTMs they haven't always worked so well as they do now
 because we kind of know how to handle them and I expect if these cells here
 take off especially these NBRC then with time will be as proficient at handling
 them and they will probably become on par or even outperform the LSTMs or GRUs on
 every day like on all the tasks and then be especially good on tasks where you
 have to remember things but for now they're outperformed by LSTMs and GRUs
 okay so the second thing is a more interesting experiment the denoising
 benchmark where they say the the copy input benchmark is interesting as a
 means to highlight the memorization capacity of the recurrent neural network
 but it does not tackle its ability to successfully exploit complex
 relationships between different elements of the input signal to predict the output
 so they have a new benchmark in the denoising benchmark the network is
 presented with a two-dimensional time series of t time steps five different
 time steps are sampled uniformly with okay and are communicated in the network
 okay I'll just tell you what's going on so this new time series is two
 dimensional in the lower dimension you simply have a bunch of random numbers
 like five eight two nine actually these are numbers sampled from a uniform
 Gaussian or so so they're not actually five eight two and nine but you can
 imagine it like this five eight two nine three four zero two and so on and in the
 second dimension you have a negative one I believe almost anywhere and then at
 some points you have a one negative one again and then you have a one and a
 negative one again and at the last point of the sequence you'll have a zero and
 so the zero is simply a marker that it's the end of the sequence what the
 network needs to do is it needs to output all the elements so the output of
 the network should be in this case should be nine four so all the elements
 where there was a one in order okay so it remember what it needs to learn it
 needs to learn to every time it sees a one in the second dimension it needs to
 take the first dimension put it somehow into the hidden state and then carry
 that hidden state forward and sees a one again it needs to take the second thing
 also put it into the hidden state but not override the first thing it put into
 the hidden state like if it were to just realize I need to put this into the
 hidden state then it would almost surely override the previous information so it
 needs to be able to say I've already kind of in my H is going to be a vector
 of a hundred dimensions it needs to be able to say well I've already stored a
 bunch of stuff in that part of the vector maybe I should store that thing
 here over here so this is fairly complex things to remember and technically GRU's
 and LSTMs are able to do it but as we'll see they're not as much the results are
 in this table where you can clearly see that whenever the n so the n the n is a
 parameter that is how far how far in this direction are these ones so when n
 is 0 the ones can be anywhere but when n here is like 5 that means that the last
 five ones surely don't contain a one that means only the first whatever a L
 minus L minus 5 contain the one so the higher this number n is the harder the
 task because your learning signal is way way farther away from the from what's
 when you get the output so you can see when the n is low then the GRU's and the
 LSTMs they perform pretty well but also these cells perform pretty well they're
 just not performing as well however when the task gets harder and you actually
 need to learn a sparse signal over a long period of time where in between you
 don't get any signal the GRU's and the LSTMs fail while the BRC's would still
 be able to learn these kinds of things so that's that's fairly cool now it's if
 from a researcher's perspective I wonder if they just first tried this task you
 know as I described it and then they discovered like ah crap they can still
 do it and like okay how can we make it such that there's a difference okay
 let's actually make the task harder like this and then they did that I wonder if
 they always had the idea with the end here or just introduced this after after
 it they they failed to produce a difference in the first place I'm not
 sure but they have they have another benchmark but they basically show that
 these cells are actually good can incorporate this information can reason
 about what they need to remember and whatnot and in the end they also have
 this sequential MNIST where they just feed an MNIST digit digit by digit and
 at the end I think that the output of the neural network needs to be the class
 of the of the MNIST digit and again here they have a parameter called N black
 which means that so they have an MNIST digit it's like a three they unroll it to
 a single vector right they feed this one by one into the recurrent network and
 then after that they attach a certain number of just empty pixels black pixels
 and after that the network needs to predict the Y you can see if they ask
 the network the class of the digit immediately after it's done then the G
 are using the LSTM perform fairly well as do the BRCs but if you attach a whole
 bunch of these black pixels remember an MNIST digit has some seven sorry seven
 hundred and eighty four maybe entries so attaching 300 black pixels is quite
 significant in in terms of the length of these sequences and then the GRUs and
 the LSTMs they can't learn they can't learn to ignore these things because the
 learning signal is just too far away right here but these things they can
 because they can exploit this by stability property and remember things
 again I wonder how this came to be it seems pretty funny but the last thing
 they do is they investigate what happens in their cells and this I feel is the
 most interesting part and they do this on this denoising benchmark so the task
 we've looked at before where you need to remember five randomly selected numbers
 that are indicated by the second dimension here they show a sequence
 where the five numbers occur at 3100 246 at 300 and at 376 so these are the five
 positions where the sequence indicates that the network should remember the
 thing in the first dimension and then output they analyze two things they
 analyze the proportion of bi-stable neurons so basically they analyze these
 out these a quantities and they analyze how many of the neurons in the layer
 have an a that's higher than one which means that they are in this bi-stable
 mode and also they analyze what's the average value of C so see if you
 remember if this is high it means it doesn't let in new information and if
 this is low it means it lets in new information if you first look at the C
 you can see that every single time when the second dimension indicates that this
 is one of the inputs to remember this the network drops immediately drops the
 C values the different colors here are different layers they build they have a
 recurrent network has multiple layers of these cells as is usual in the
 recurrent neural networks so this C as you can see it goes up pretty quickly
 but then as soon as one of these inputs appear the C drops which basically means
 that the network realizes it now must let in the new information and then it
 immediately shoots back up makes it seem like so the network says okay as long as
 so all of these inputs here they have the negative one in the second dimension
 right so it recognizes it says there's no reason for me to incorporate that
 information it's not important and as soon as the second input comes it
 immediately shoots down again now you can see this here is the last layer of
 the network the highest layer so sort of the highest abstractive information and
 you can see that from input to input this value of C gets higher and higher
 and these spikes as they go down but they go down to a higher and higher
 point which you know is is the fact that it recognizes it needs to let in new
 information but it lets in less and less new information the more things it needs
 to remember so not only does it recognize wait I need to remember this
 it also recognizes but I probably shouldn't shouldn't you know completely
 forget what I had previously because I it is important for me to remember these
 previous things so that's a pretty cool demonstration the fact that these go
 down at the input and the fact that generally they go up every time after a
 new input is incorporated into the hidden state this basically this shows
 that the or this is a pretty good indication that what they're saying is
 really happening right okay the second thing shows almost the same it shows how
 many of these neurons are actually in their bi-stable mode and you can also
 see right here that especially in the last layer you can see that the number
 of neurons in the bi-stable mode goes up and up and up and up after each of these
 steps and these spikes here correspond to always the points where they have to
 let in new information okay cool so I find that I find this to be pretty cool
 and I find this last experiment to be the coolest where they can actually show
 look here there's a pretty good indication that the thing we we build
 does what we say it does they also actually have a proof here of the
 bi-stability when this a is higher than one I won't go through this right here
 but if you want you can look at that I'm excited to see what happens with these
 kinds of architectures in the future because it seems to be a pretty minor
 modification and maybe with a little bit of more modification or if we sort of
 just tune this a little bit and kind of figure out what we have to do to make
 these things actually compete with the classic GRUs and LSTMs in regimes where
 a long memory isn't necessary I feel this could be a you know kind of a
 standard building block in the recurrent neural network toolkit even though it's
 been sort of outperformed by transformers in previous years alright
 that was it for me and I hope you had fun with this paper I invite you to
 check it out and bye bye