Self-classifying MNIST Digits (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper biology biological alive living message passing global state local state information cellular automata neural cellular automata neural ca convolution recurrent rnn pixels cell state latent state distill distill pub mnist neural network digit classification
#ai #biology #machinelearning

Neural Cellular Automata are models for how living creatures can use local message passing to reach global consensus without a central authority. This paper teaches pixels of an image to communicate with each other and figure out as a group which digit they represent. On the way, the authors have to deal with pesky side-effects that come from applying the Cross-Entropy Loss in combination with a Softmax layer, but ultimately achieve a self-sustaining, stable and continuous algorithm that models living systems.

OUTLINE:
0:00 - Intro & Overview
3:10 - Neural Cellular Automata
7:30 - Global Agreement via Message-Passing
11:05 - Neural CAs as Recurrent Convolutions
14:30 - Training Continuously Alive Systems
17:30 - Problems with Cross-Entropy
26:10 - Out-of-Distribution Robustness
27:10 - Chimeric Digits
27:45 - Visualizing Latent State Dimensions
29:05 - Conclusion & Comments

Paper: https://distill.pub/2020/selforg/mnist/

My Video on Neural CAs: https://youtu.be/9Kec_7WFyp0

Abstract:
Growing Neural Cellular Automata [1] demonstrated how simple cellular automata (CAs) can learn to self-organise into complex shapes while being resistant to perturbations. Such a computational model approximates a solution to an open question in biology, namely, how do cells cooperate to create a complex multicellular anatomy and work to regenerate it upon damage? The model parameterizing the cells’ rules is parameter-efficient, end-to-end differentiable, and illustrates a new approach to modeling the regulation of anatomical homeostasis. In this work, we use a version of this model to show how CAs can be applied to a common task in machine learning: classification. We pose the question: can CAs use local message passing to achieve global agreement on what digit they compose?

Authors: Ettore Randazzo, Alexander Mordvintsev, Eyvind Niklasson, Michael Levin, Sam Greydanus

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Check this out. So what you're seeing here is neurocellular automata that are learned to communicate with each other what digit they compose. So every pixel you see is like a little cell and it communicates with its neighbors and only its immediate neighbors about kind of its surroundings. And by doing that all these cells that are connected components have to agree as to what digits they compose. And here you can see the seven symbolized by gray and the three symbolized by green reach an agreement. There are some interesting properties about these cellular automata. Namely here you can see that half of this thinks it's a two and the rest thinks it's a zero. However, let's see when I complete this. No, it's too smart for this. Well, look at that. Now it thinks it's an eight. So you can clearly see there's like some message passing, some evolution going on across the states right here. It doesn't work perfectly. I found it thinks a lot of times that it is in fact a zero. As you can see right here. But so the goal is that this this direction of research isn't about state of the art in digit classification as you might be able to determine right here. It's about neurocellular automata. And I highly recommend if you don't know yet, go watch my video or read the previous article in this distil pub journal about growing neurocellular automata. This paper here is a follow up. It's called self classifying MNIST digits. And it's by Ettore Randazzo, Alexander Mortwintsev, Edwin Nicholson and sorry, I'd been Nicholson, Michael Levin and Sam Graydenes. So this paper is an evolution of the previous paper. And I'm going to switch back and forth here between the website and the thing where I can scribble on. So bear with me for that. They're saying that growing neurocellular automata demonstrated how simple cellular automata can learn to self organize into complex shapes while being resistant to perturbation. So that was the last paper. Such a computational model approximates a solution to an open question biology, namely how do cells cooperate to create a complex multicellular anatomy and work to regenerate it upon damage. Also from the last paper. The model parametrizing the cell rule is parameter efficient and to indifferentiable and illustrates a new approach to modeling the regulation of anatomical homeostasis. Homeostasis. OK. In this work, we use a version of this model to show how cellular automata can be applied to common task and machine learning classification. We pose the question, can cellular automata use local message passing to achieve global agreement on what digit they compose? So that's the question right here. Now, again, I've done a video on cellular automata, but really, really briefly. What you saw above is that there's an image and it's rasterized, of course rasterized in two pixels, and each pixel represents one cell. So you can think of this as basically nodes in a graph and each cell is connected to its immediate neighbors. So each cell, let's take this one, is connected to all its immediate neighbors like so. And of course, each cell, each other cell again is connected to its immediate neighbors. Now, all they know is basically the so if I draw something on this canvas, let's say I draw, let's take this, I draw a two. OK, then you look at this cell right here. And of course, the cell, this is going to be, it's the line would be thicker. So it's either going to be on or off. It's either going to be I painted on it or I didn't paint on it. And it can be in different variations, like there is an alpha level. But ultimately, the each cell can only register whatever was set painted on it. OK, so each cell can be dead or alive and dead cells, they will not send around any messages and dead cells is everywhere where there is no color at all. So this would be a dead cell. This would be a dead cell. This one wouldn't be a dead cell because there is a little bit of color. This would be a dead cell right here. So with this, so you can see that most cells here are actually dead. Now, the cells that aren't dead, they register whatever is painted on them like this cell or this cell or this cell. And then they need to communicate that to each other. And the goal is that all these cells that are alive, like these cells right here, all the cells that are alive, they pass messages to each other such that they all come to an agreement what digit they compose. And if you imagine you're this cell right here, all you see is that there is a bit of purple on you. Right. There is a bit of purple and it could be alpha level 200 out of 255. And only by registering this and communicating this to your neighbors and receiving messages and then passing on those messages to other neighbors, all of these cells need to come to an agreement. So how do these cells agree? Each cell, in fact, has a cell state. So each of these cells has a cell state. And that cell state, first and foremost, composes is composed of 10 different slots, one for each class. So what does it mean to agree on something at the end of this procedure or over time? Each cell in each round of communication can update its own cell state. And whatever number is highest right here, so this could be a high number, this could be a low number, wrong, sideways histograms, whatever one is the highest right here, that's what the cell believes the class is. So you immediately see a bit how this is going to be trained. So this is going to be trained by these authors taking an MNIST digit, placing that on the cells, letting this whatever procedure run. If the procedure is differentiable, you let it run for a number of time steps. And in each time step, you basically impose a cross entropy classification loss on these 10 entries in the cell state. That way you train the cells to output the correct digit. Now, each cell has to do that by itself. So the goal is to devise a communication algorithm such that each cell communicates with each other cell such that at the end, all the cells will be updated as to what the global state is, as to what the digit comprises. So what is this message passing right here? And for that, I think we need to first of all imagine what is actually passed around here. So if you see this sample above right here and you imagine, let's say we are actually in this configuration on the left and there is a slight bend. Let's say here we're in this part of the number two, there's a slight bend right here. So what you can see, maybe let me draw this a bit more clear, is that, for example, this the blue cell will register will by message passing. It can register that there is an alive cell right here. But this alive cell will also register that there is no, there is a dead cell next to it. So it can pass on that message to the blue cell and the blue cell will sort of know that there is kind of a border over there. Then also diagonally to the blue cell, it will register itself. Wow, there is a dead cell right here. And that's right below this alive cell above. So there must be some kind of a bend right here. You can already see how through this sort of message passing and this cell right here, of course, will its neighbor is also dead. Through this message passing, these cells can kind of figure out together the kind of more global shapes and they will recognize, ah, there is a bend. It's something like this. Right. And then other cells, maybe down here, will figure out, well, there is actually a corner right here. OK. And then other cells on top here, they will figure out, well, there is actually a bend like this. And then they can communicate this to each other. So these cells right here that have the corner, they will at some point receive this integrated message that there is a bend on top. And then they can make sense of that, right, and say, well, we are a corner and there is a bend on top. And there is so there must be a digit that's something like this. Right. And you can already see that at that point, they can be fairly sure that this is a two. So you can see that the combination of message passing and kind of think of each cell thinking by itself can give rise to this kind of each cell coming into global agreement, not only agreement, but correct agreement. So the message passing itself, again, described in the last paper, but really briefly, there is these 10 entries right here that decide on what the cell believes the state is. And then you can have extra entries that are just kind of latent state. There is no loss imposed on these latent variables, but ultimately the cell state consists of this long vector. And then this vector is passed on to all the neighbors. OK, this vector is passed to all the neighbors and all the neighbors send their own state vector to this cell. Now, the state vectors of all the neighbor cells are then integrated. So each one has this vector, vector, vector, vector, vector. These are all integrated together with the own state of the of the cell in a linear fashion. So there is like a small neural network in between. And that will update the cell state. In fact, I think they calculate a diff to the cell state. They don't calculate the new cell state by definition. They actually calculate a diff. And this should remind you of... So if we just look at this one dimensionally, right? So here's the cell and there is its neighbor, its neighbor, its neighbor, neighbor, and then the diagonal neighbors. And we want to update this cell right here as a linear combination of all the cells surrounding it and itself. And we want to do that for each. So each cell has the same update rule. So it doesn't matter where the cell is. You're trying to come up with one rule how to integrate the surrounding states into the cell itself. This is so the biological kind of reasoning behind it is that all the cells follow the same rules. But by virtue of where they are and how they communicate, these global patterns can arise. And this cell will update and then if we consider the next cell next to it, it has its neighbors. It will update according to its neighbors. This should remind you of a convolution, right? Because this is exactly convolution. So there will be a convolutional operator, a 3x3 convolutional operator right here. This can be multi-channel, of course, because we have multiple channels right here in the cell state. So the convolution will be learned once globally, which is exactly what a convolutional operator is, a convolutional kernel. It will be learned to update these cell states. In fact, it's a residual convolutional connection, right? This goes through the convolutional kernel and is then added together with the signal itself to give rise to the new cell states. So one convolution across the entire image will take care of updating all the cells. It's one round of message passing. And then now contrary to a convolutional neural network, where then the signal would go into the next layer into the next convolutional kernel. Sorry. This is then repeated with the same convolutional kernel, right? The message passing algorithm is the same in each round. So this is a recurrent neural network with a residual convolution as an operator. That is the model for kind of the biological cell communication algorithm. So these are these neural cellular automata. The difference to the last paper is twofold. First of all, in the last paper, we had RGB values up here. Now it's the class labels. So these are also passed around so that the cell passes to its neighbors what it believes the current labels are, but also these hidden features right here. And we'll come to this in a second. And the second difference is that the dead and alive cells are static. So where these dead cells, where the dead cells and where the alive cells are, that never changes. That used to change in the last paper. Here it never changes. It's only about passing the messages around between the cells. All right. So this is basically it. So this is a model for agreement between cells. I think it's pretty cool. I would still like to go more into what kind of what exactly happens, what kind of messages are passed around. But they do this a little bit. So they have a bunch of experiments. How do they train this stuff? Basically, how do they train this stuff that I can, you know, I can change it in between and it will actually it will update it live. So the cells, you can't only do this once. The cells must have a notion of continuously being alive, continuously updating themselves, continuously being prepared that there is some sort of a modification to the cell. And that's they do this by. So here you can see, can I zoom? Well, I can't. Now I can. Here you can see that this is how they train it. So they just initialize the cell states randomly. That's why you see there are just random colors right here. These are MNIST digits. And then they train these cells, all of them, to predict the label of the MNIST digits, which they have in the training set. And then so you can see, once you've trained it, that happens fairly, fairly quickly. And then after 200 steps, they simply switch out the digit. OK, they leave all the cells as they are. Of course, some cells will be dead now and some cells will be alive. The ones that come alive will just be initialized randomly. But there are always going to be cells that are going to be present in both digits. And those will just keep the label. But, you know, usually the the digit here changes with a 90 percent probability. And since this is one long run of a recurrent network, the network sort of changes. That network sort of has to always be prepared for a change because it's trained with this mutation. So it's trained for 200 steps in the first digit. And then it's switched and trained for 200 steps with the second label. That causes these cells to kind of always be ready for change. And that's, yeah. So you can see there are still some artifacts where the cells that they're not quite sure and so on. And in fact, they get worse over time. So if you pay real close attention towards the end of these cycles, it actually gets worse. So after a while, some of them will start flickering up again. And that's a problem they've observed. And they go into this right here. So they have these graphs of accuracy over time. So accuracy means average cell accuracy. So they just take all the cells and they see how many of them are correct. And you can see at the beginning of training pretty quickly. So in the beginning, this is inference. So inference, of course, you also do over time. Right. So this is in inference, you provide a digit, you initialize randomly, and then you let these cells communicate. So you run the recurrent convolutional algorithm and you count how many cells output the correct label at each step. And pretty quickly reaches high up. And then you can see at the mutation, it drops down to random again, but also pretty quickly recover. So it sounds pretty good. And you can see a teeny tiny bit right here. It's kind of going down after, you know, over time. And so they determine they need to do something about this. In fact, they first of all, they want to make a point that you have to figure out what exactly is happening. So here they have average cell accuracy. But what they also decide to measure is average total agreement across the batch. Average total agreement basically means how many of the cells within a digit agree with each other on the label, which is sort of a measure. If this is really an MNIST digit, you know, it should be perfectly in one class and not the other. I know there's some ambiguity. But so what you should have at least, even if the cells are wrong, you should have a total agreement in the cells. If this is in fact a digit, the cells should somehow agree with each other because that's what you train them to. You train them to agree with each other. And you can see again here as well, pretty quickly you have an agreement after a number of steps. And then that agreement drops again, strangely, right? Because they've already reached an agreement. You might think this will sort of maybe it will hamper down, but it might slightly go up. But no, it actually slightly goes down over time. So why is that? They also analyze this here, and I'm sorry about this chopped up graph. But you can see that the here are the state values, here are the sizes, the real numerical sizes of these entries in the states. And you can see that they grow over time. So not only do they grow until the agreement is reached, but also they keep growing after that. And here are the diffs from state to state. And you can also see that these never go to zero. So why is that? And they have a hypothesis right here. In fact, they have the hypothesis this is due to the cross entropy loss. Now the cross entropy loss is kind of the most famous loss for classification. So usually what you'll have is your neural network will output some distribution like this. Let's say it's three classes. So it believes that class number two here is the correct class. And then you have a label, which you transform into a one-hot distribution, where this is one, these are zero. And then you perform this cross entropy loss between the two, saying that the left thing should be more equal to the right thing. And you do that in the sense of... So this is the kind of the entropy formulation. But what you actually do is this y log p. So p here is going to be the distribution that you output and y is going to be the distribution that the network outputs. You can pretty clearly see y is going to be zero for all the classes that are wrong. So the entire loss reduces to simply the probability here of the... Sorry, there is a negative. The probability of the class that is correct. So what you want to do is you want to push that up. Now, of course, just looking at the loss, only the correct class is pushed up. Nothing else is done. Now, you also know that most of the time we combine this with a so-called softmax operator. So what our network outputs isn't actually a distribution, it's what we call logit, so an unnormalized distribution. So what it actually outputs could be something like this. A high number, a negative number and a negative number. And only by matter of normalization, we reach this distribution. So the softmax operator will take care of normalizing. And also the softmax operator, because of the normalization, when we back propagate this loss, it causes this logit here to rise and it causes these ones to lower because of this normalization step, not actually because of the loss. So I think they correctly say here is the cross entropy loss, but it is the cross entropy loss combined with the softmax operator that we usually use in neural networks that makes this phenomenon happen. So what is actually happening here? If you look at the softmax operator, what it does is it's like e to the x divided by the sum of e to the x prime overall, or overall other classes, so you can fairly easily see that this exponential function here is never, ever, ever going to be zero. So you can never have a zero entry right here. So the loss forces you to push this thing up, but because you can never have zero entries there, of course, this can never be one. So you can never actually reach perfect loss. And what does it do to the logits? You cannot reach perfect loss, but the gradient will always push you into the direction of upping this logit and downing this. So raising the one that is correct and lowering actually into the negative direction, the ones that aren't correct. So you can see that if we do this once, no problem. If we do this in a single neural network, forward propagate, calculate loss, not a problem. But if we do this over and over and over and over again in a convolutional neural network and we let it run for infinite time, of course, what is going to happen is that these things are going to explode more and more and more. So these losses are going to get bigger and bigger, which makes the entire rest of the network behave in a bigger and bigger fashion. And that is exactly what you see here, because these simply the numerical values in the states, they will be bigger and bigger and bigger because they push the network into the direction of more and more and more reducing the loss, thereby raising the logits. So there's it's very disproportionate. At the end, you have to raise the logits by a lot to reduce the loss a little bit. But the network doesn't care because that's what it was trained to do. So they hypothesize if we use an L2 loss, this shouldn't happen. Now in an L2 loss, you do not compare, you don't output logits, you output actual probabilities, and you simply compare the L2 distance to them. So if you compare the L2 distance right here, yes, you will push this one up. But if you push it too high, then it's too high and then it will be pushed down again until it is exactly the same level as the other one. Now, the disadvantages here is that, of course, this isn't actually forced to be a valid probability distribution. You can normalize it, yes, but you can go too high. So you can output probabilities higher than one and so on. So there's a whole slew of problems that come with this, but you can counter this. So beside using an L2 loss, they also have another on top idea in that they always add noise to these residual updates that they do after the convolution, just kind of to keep the network on its toes, saying that everything can always change with noise. So in each step, it basically has to do some of some correction with respect to that noise. And here you can see the clear difference, especially in the lower plot, where the total agreement before this blue line was when it went down over time. And now with the L2 loss and a little bit more with this residual noise, it manages to keep the total agreement up and solve that problem. And you can also see that the average magnitude of the updates no longer is rising over time, but actually it's keeping the same for the cell states and the updates converge towards zero. Of course, not as much with the noise because the noise makes them. The noise will make them non-zero, the updates, but still they are at the same magnitude. And they manage to correct that noise and not incorporate more and more and more like the cross entropy loss. So this, I don't want to go into the last few bits, except this one. These cells have some interesting properties, notably they're also resistant to kind of out of distribution errors. You can see that in this video where you can see it's classifying it fairly solidly as 1s. But as soon as you draw a shape that is not kind of in the training set or in the classes of the training set, the cells keep disagreeing with each other. So this you can see as sort of kind of a robustness to out of distribution samples. And it's also pretty interesting to see that the messages here, where they go from. So you can fairly clearly see that if you draw some kind of shape, that the message passing starts at kind of the most symbolic parts of the digits. And here they have some chimeric digits or something they call it like this. And just pay attention to where the messages start. And you can clearly see that this sort of local determination of what a digit is will spread out over time to the other cells. And I thought there was this last thing. This thing. Yes. So here, not only do they visualize the cell state, so the color of the cell, and that's the thing on the left, is always the first 10 entries in this hidden state. But on the right, they also visualize the other hidden entries. And so each entry is represented by a two color thing where blue is very low number, red is a very high number. And here you can see what these latent states pass around. And also you can fairly clearly see that they do pass around these kind of typical sub shapes of the digit. So in the case of a zero, that's going to be a bend. In the case of a four, that's going to be these ends and corners of the numbers. And you can see that over time, as these messages pass, also the cell states on the left, the visible states, the class labels change over time. This lends a lot of credence, especially the six I like. Or the two. You can see in the different, if you kind of look at the different latent states, that the kind of typical, the bends, the corners, every latent state is sort of assigned to one of them. And then they pass this information around in order to reach an agreement. So I like this research, pretty cool research. I don't want to say it's very useful, but certainly it's very interesting. And I also like the format in this distilled format. I think that's sort of the future of research rather than eight page PDFs. You can look at it, it's interactive, you can have a little demo in it. You can write for as long as you want. And yeah, it's just overall better. This is still going. Doesn't know what it is. So lastly, you can, as I said, you can clearly see that, look, if I do this, it's a zero. But if I do this, then the stem part will immediately go for a six because that's indicative of a six. But then it will disagree with the zero part of the digit. In fact, I seem to be unable to write a six. Is that an American six? Maybe. Yeah, so with that, I'll leave this here. I think this is, again, very interesting, this kind of biological models. And certainly, if you're looking for an exciting research directions, this might be it. And you do not need a lot of resources to do this. This is very parameter efficient, as we saw in the last paper. And certainly kind of a niche right now. So that was it for me. I hope you enjoyed this. If you liked it, share it out and bye bye. See you next time.
 Check this out.
 So what you're seeing here is neurocellular automata that are learned to communicate with each other what digit they compose.
 So every pixel you see is like a little cell and it communicates with its neighbors and only its immediate neighbors about kind of its surroundings.
 And by doing that all these cells that are connected components have to agree as to what digits they compose.
 And here you can see the seven symbolized by gray and the three symbolized by green reach an agreement.
 There are some interesting properties about these cellular automata.
 Namely here you can see that half of this thinks it's a two and the rest thinks it's a zero.
 However, let's see when I complete this. No, it's too smart for this.
 Well, look at that. Now it thinks it's an eight.
 So you can clearly see there's like some message passing, some evolution going on across the states right here.
 It doesn't work perfectly. I found it thinks a lot of times that it is in fact a zero.
 As you can see right here. But so the goal is that this this direction of research isn't about state of the art in digit classification as you might be able to determine right here.
 It's about neurocellular automata.
 And I highly recommend if you don't know yet, go watch my video or read the previous article in this distil pub journal about growing neurocellular automata.
 This paper here is a follow up. It's called self classifying MNIST digits.
 And it's by Ettore Randazzo, Alexander Mortwintsev, Edwin Nicholson and sorry, I'd been Nicholson, Michael Levin and Sam Graydenes.
 So this paper is an evolution of the previous paper.
 And I'm going to switch back and forth here between the website and the thing where I can scribble on.
 So bear with me for that.
 They're saying that growing neurocellular automata demonstrated how simple cellular automata can learn to self organize into complex shapes while being resistant to perturbation.
 So that was the last paper.
 Such a computational model approximates a solution to an open question biology, namely how do cells cooperate to create a complex multicellular anatomy and work to regenerate it upon damage.
 Also from the last paper.
 The model parametrizing the cell rule is parameter efficient and to indifferentiable and illustrates a new approach to modeling the regulation of anatomical homeostasis.
 Homeostasis. OK.
 In this work, we use a version of this model to show how cellular automata can be applied to common task and machine learning classification.
 We pose the question, can cellular automata use local message passing to achieve global agreement on what digit they compose?
 So that's the question right here.
 Now, again, I've done a video on cellular automata, but really, really briefly.
 What you saw above is that there's an image and it's rasterized, of course rasterized in two pixels, and each pixel represents one cell.
 So you can think of this as basically nodes in a graph and each cell is connected to its immediate neighbors.
 So each cell, let's take this one, is connected to all its immediate neighbors like so.
 And of course, each cell, each other cell again is connected to its immediate neighbors.
 Now, all they know is basically the so if I draw something on this canvas, let's say I draw, let's take this, I draw a two.
 OK, then you look at this cell right here.
 And of course, the cell, this is going to be, it's the line would be thicker. So it's either going to be on or off.
 It's either going to be I painted on it or I didn't paint on it.
 And it can be in different variations, like there is an alpha level.
 But ultimately, the each cell can only register whatever was set painted on it.
 OK, so each cell can be dead or alive and dead cells, they will not send around any messages and dead cells is everywhere where there is no color at all.
 So this would be a dead cell. This would be a dead cell.
 This one wouldn't be a dead cell because there is a little bit of color.
 This would be a dead cell right here.
 So with this, so you can see that most cells here are actually dead.
 Now, the cells that aren't dead, they register whatever is painted on them like this cell or this cell or this cell.
 And then they need to communicate that to each other.
 And the goal is that all these cells that are alive, like these cells right here, all the cells that are alive,
 they pass messages to each other such that they all come to an agreement what digit they compose.
 And if you imagine you're this cell right here, all you see is that there is a bit of purple on you.
 Right. There is a bit of purple and it could be alpha level 200 out of 255.
 And only by registering this and communicating this to your neighbors and receiving messages and then passing on those messages to other neighbors, all of these cells need to come to an agreement.
 So how do these cells agree? Each cell, in fact, has a cell state.
 So each of these cells has a cell state.
 And that cell state, first and foremost, composes is composed of 10 different slots, one for each class.
 So what does it mean to agree on something at the end of this procedure or over time?
 Each cell in each round of communication can update its own cell state.
 And whatever number is highest right here, so this could be a high number, this could be a low number, wrong, sideways histograms, whatever one is the highest right here, that's what the cell believes the class is.
 So you immediately see a bit how this is going to be trained.
 So this is going to be trained by these authors taking an MNIST digit, placing that on the cells, letting this whatever procedure run.
 If the procedure is differentiable, you let it run for a number of time steps.
 And in each time step, you basically impose a cross entropy classification loss on these 10 entries in the cell state.
 That way you train the cells to output the correct digit.
 Now, each cell has to do that by itself.
 So the goal is to devise a communication algorithm such that each cell communicates with each other cell such that at the end, all the cells will be updated as to what the global state is, as to what the digit comprises.
 So what is this message passing right here?
 And for that, I think we need to first of all imagine what is actually passed around here.
 So if you see this sample above right here and you imagine, let's say we are actually in this configuration on the left and there is a slight bend.
 Let's say here we're in this part of the number two, there's a slight bend right here.
 So what you can see, maybe let me draw this a bit more clear, is that, for example, this the blue cell will register will by message passing.
 It can register that there is an alive cell right here.
 But this alive cell will also register that there is no, there is a dead cell next to it.
 So it can pass on that message to the blue cell and the blue cell will sort of know that there is kind of a border over there.
 Then also diagonally to the blue cell, it will register itself.
 Wow, there is a dead cell right here.
 And that's right below this alive cell above.
 So there must be some kind of a bend right here.
 You can already see how through this sort of message passing and this cell right here, of course, will its neighbor is also dead.
 Through this message passing, these cells can kind of figure out together the kind of more global shapes and they will recognize, ah, there is a bend.
 It's something like this.
 Right. And then other cells, maybe down here, will figure out, well, there is actually a corner right here.
 OK. And then other cells on top here, they will figure out, well, there is actually a bend like this.
 And then they can communicate this to each other.
 So these cells right here that have the corner, they will at some point receive this integrated message that there is a bend on top.
 And then they can make sense of that, right, and say, well, we are a corner and there is a bend on top.
 And there is so there must be a digit that's something like this.
 Right. And you can already see that at that point, they can be fairly sure that this is a two.
 So you can see that the combination of message passing and kind of think of each cell thinking by itself can give rise to this kind of
 each cell coming into global agreement, not only agreement, but correct agreement.
 So the message passing itself, again, described in the last paper, but really briefly, there is these 10 entries right here that decide on what the cell believes the state is.
 And then you can have extra entries that are just kind of latent state.
 There is no loss imposed on these latent variables, but ultimately the cell state consists of this long vector.
 And then this vector is passed on to all the neighbors.
 OK, this vector is passed to all the neighbors and all the neighbors send their own state vector to this cell.
 Now, the state vectors of all the neighbor cells are then integrated.
 So each one has this vector, vector, vector, vector, vector.
 These are all integrated together with the own state of the of the cell in a linear fashion.
 So there is like a small neural network in between.
 And that will update the cell state.
 In fact, I think they calculate a diff to the cell state.
 They don't calculate the new cell state by definition.
 They actually calculate a diff. And this should remind you of...
 So if we just look at this one dimensionally, right?
 So here's the cell and there is its neighbor, its neighbor, its neighbor, neighbor, and then the diagonal neighbors.
 And we want to update this cell right here as a linear combination of all the cells surrounding it and itself.
 And we want to do that for each. So each cell has the same update rule.
 So it doesn't matter where the cell is.
 You're trying to come up with one rule how to integrate the surrounding states into the cell itself.
 This is so the biological kind of reasoning behind it is that all the cells follow the same rules.
 But by virtue of where they are and how they communicate, these global patterns can arise.
 And this cell will update and then if we consider the next cell next to it, it has its neighbors.
 It will update according to its neighbors.
 This should remind you of a convolution, right? Because this is exactly convolution.
 So there will be a convolutional operator, a 3x3 convolutional operator right here.
 This can be multi-channel, of course, because we have multiple channels right here in the cell state.
 So the convolution will be learned once globally, which is exactly what a convolutional operator is, a convolutional kernel.
 It will be learned to update these cell states.
 In fact, it's a residual convolutional connection, right?
 This goes through the convolutional kernel and is then added together with the signal itself to give rise to the new cell states.
 So one convolution across the entire image will take care of updating all the cells.
 It's one round of message passing.
 And then now contrary to a convolutional neural network, where then the signal would go into the next layer into the next convolutional kernel.
 Sorry.
 This is then repeated with the same convolutional kernel, right?
 The message passing algorithm is the same in each round.
 So this is a recurrent neural network with a residual convolution as an operator.
 That is the model for kind of the biological cell communication algorithm.
 So these are these neural cellular automata.
 The difference to the last paper is twofold.
 First of all, in the last paper, we had RGB values up here.
 Now it's the class labels.
 So these are also passed around so that the cell passes to its neighbors what it believes the current labels are, but also these hidden features right here.
 And we'll come to this in a second.
 And the second difference is that the dead and alive cells are static.
 So where these dead cells, where the dead cells and where the alive cells are, that never changes.
 That used to change in the last paper.
 Here it never changes.
 It's only about passing the messages around between the cells.
 All right.
 So this is basically it.
 So this is a model for agreement between cells.
 I think it's pretty cool.
 I would still like to go more into what kind of what exactly happens, what kind of messages are passed around.
 But they do this a little bit.
 So they have a bunch of experiments.
 How do they train this stuff?
 Basically, how do they train this stuff that I can, you know, I can change it in between and it will actually it will update it live.
 So the cells, you can't only do this once.
 The cells must have a notion of continuously being alive, continuously updating themselves,
 continuously being prepared that there is some sort of a modification to the cell.
 And that's they do this by.
 So here you can see, can I zoom?
 Well, I can't.
 Now I can.
 Here you can see that this is how they train it.
 So they just initialize the cell states randomly.
 That's why you see there are just random colors right here.
 These are MNIST digits.
 And then they train these cells, all of them, to predict the label of the MNIST digits, which they have in the training set.
 And then so you can see, once you've trained it, that happens fairly, fairly quickly.
 And then after 200 steps, they simply switch out the digit.
 OK, they leave all the cells as they are.
 Of course, some cells will be dead now and some cells will be alive.
 The ones that come alive will just be initialized randomly.
 But there are always going to be cells that are going to be present in both digits.
 And those will just keep the label.
 But, you know, usually the the digit here changes with a 90 percent probability.
 And since this is one long run of a recurrent network, the network sort of changes.
 That network sort of has to always be prepared for a change because it's trained with this mutation.
 So it's trained for 200 steps in the first digit.
 And then it's switched and trained for 200 steps with the second label.
 That causes these cells to kind of always be ready for change.
 And that's, yeah.
 So you can see there are still some artifacts where the cells that they're not quite sure and so on.
 And in fact, they get worse over time.
 So if you pay real close attention towards the end of these cycles, it actually gets worse.
 So after a while, some of them will start flickering up again.
 And that's a problem they've observed.
 And they go into this right here.
 So they have these graphs of accuracy over time.
 So accuracy means average cell accuracy.
 So they just take all the cells and they see how many of them are correct.
 And you can see at the beginning of training pretty quickly.
 So in the beginning, this is inference.
 So inference, of course, you also do over time.
 Right.
 So this is in inference, you provide a digit, you initialize randomly, and then you let these cells communicate.
 So you run the recurrent convolutional algorithm and you count how many cells output the correct label at each step.
 And pretty quickly reaches high up.
 And then you can see at the mutation, it drops down to random again, but also pretty quickly recover.
 So it sounds pretty good.
 And you can see a teeny tiny bit right here.
 It's kind of going down after, you know, over time.
 And so they determine they need to do something about this.
 In fact, they first of all, they want to make a point that you have to figure out what exactly is happening.
 So here they have average cell accuracy.
 But what they also decide to measure is average total agreement across the batch.
 Average total agreement basically means how many of the cells within a digit agree with each other on the label,
 which is sort of a measure.
 If this is really an MNIST digit, you know, it should be perfectly in one class and not the other.
 I know there's some ambiguity.
 But so what you should have at least, even if the cells are wrong, you should have a total agreement in the cells.
 If this is in fact a digit, the cells should somehow agree with each other because that's what you train them to.
 You train them to agree with each other.
 And you can see again here as well, pretty quickly you have an agreement after a number of steps.
 And then that agreement drops again, strangely, right?
 Because they've already reached an agreement.
 You might think this will sort of maybe it will hamper down, but it might slightly go up.
 But no, it actually slightly goes down over time.
 So why is that?
 They also analyze this here, and I'm sorry about this chopped up graph.
 But you can see that the here are the state values, here are the sizes, the real numerical sizes of these entries in the states.
 And you can see that they grow over time.
 So not only do they grow until the agreement is reached, but also they keep growing after that.
 And here are the diffs from state to state. And you can also see that these never go to zero.
 So why is that? And they have a hypothesis right here.
 In fact, they have the hypothesis this is due to the cross entropy loss.
 Now the cross entropy loss is kind of the most famous loss for classification.
 So usually what you'll have is your neural network will output some distribution like this.
 Let's say it's three classes.
 So it believes that class number two here is the correct class.
 And then you have a label, which you transform into a one-hot distribution, where this is one, these are zero.
 And then you perform this cross entropy loss between the two, saying that the left thing should be more equal to the right thing.
 And you do that in the sense of... So this is the kind of the entropy formulation.
 But what you actually do is this y log p.
 So p here is going to be the distribution that you output and y is going to be the distribution that the network outputs.
 You can pretty clearly see y is going to be zero for all the classes that are wrong.
 So the entire loss reduces to simply the probability here of the... Sorry, there is a negative.
 The probability of the class that is correct.
 So what you want to do is you want to push that up.
 Now, of course, just looking at the loss, only the correct class is pushed up.
 Nothing else is done.
 Now, you also know that most of the time we combine this with a so-called softmax operator.
 So what our network outputs isn't actually a distribution, it's what we call logit, so an unnormalized distribution.
 So what it actually outputs could be something like this.
 A high number, a negative number and a negative number.
 And only by matter of normalization, we reach this distribution.
 So the softmax operator will take care of normalizing.
 And also the softmax operator, because of the normalization, when we back propagate this loss,
 it causes this logit here to rise and it causes these ones to lower because of this normalization step, not actually because of the loss.
 So I think they correctly say here is the cross entropy loss,
 but it is the cross entropy loss combined with the softmax operator that we usually use in neural networks that makes this phenomenon happen.
 So what is actually happening here?
 If you look at the softmax operator, what it does is it's like e to the x divided by the sum of e to the x prime overall,
 or overall other classes, so you can fairly easily see that this exponential function here is never, ever, ever going to be zero.
 So you can never have a zero entry right here.
 So the loss forces you to push this thing up, but because you can never have zero entries there, of course, this can never be one.
 So you can never actually reach perfect loss. And what does it do to the logits?
 You cannot reach perfect loss, but the gradient will always push you into the direction of upping this logit and downing this.
 So raising the one that is correct and lowering actually into the negative direction, the ones that aren't correct.
 So you can see that if we do this once, no problem.
 If we do this in a single neural network, forward propagate, calculate loss, not a problem.
 But if we do this over and over and over and over again in a convolutional neural network and we let it run for infinite time,
 of course, what is going to happen is that these things are going to explode more and more and more.
 So these losses are going to get bigger and bigger, which makes the entire rest of the network behave in a bigger and bigger fashion.
 And that is exactly what you see here, because these simply the numerical values in the states,
 they will be bigger and bigger and bigger because they push the network into the direction of more and more and more reducing the loss, thereby raising the logits.
 So there's it's very disproportionate. At the end, you have to raise the logits by a lot to reduce the loss a little bit.
 But the network doesn't care because that's what it was trained to do.
 So they hypothesize if we use an L2 loss, this shouldn't happen.
 Now in an L2 loss, you do not compare, you don't output logits, you output actual probabilities, and you simply compare the L2 distance to them.
 So if you compare the L2 distance right here, yes, you will push this one up.
 But if you push it too high, then it's too high and then it will be pushed down again until it is exactly the same level as the other one.
 Now, the disadvantages here is that, of course, this isn't actually forced to be a valid probability distribution.
 You can normalize it, yes, but you can go too high.
 So you can output probabilities higher than one and so on.
 So there's a whole slew of problems that come with this, but you can counter this.
 So beside using an L2 loss, they also have another on top idea in that they always add noise to these residual updates that they do after the convolution,
 just kind of to keep the network on its toes, saying that everything can always change with noise.
 So in each step, it basically has to do some of some correction with respect to that noise.
 And here you can see the clear difference, especially in the lower plot, where the total agreement before this blue line was when it went down over time.
 And now with the L2 loss and a little bit more with this residual noise, it manages to keep the total agreement up and solve that problem.
 And you can also see that the average magnitude of the updates no longer is rising over time, but actually it's keeping the same for the cell states and the updates converge towards zero.
 Of course, not as much with the noise because the noise makes them.
 The noise will make them non-zero, the updates, but still they are at the same magnitude.
 And they manage to correct that noise and not incorporate more and more and more like the cross entropy loss.
 So this, I don't want to go into the last few bits, except this one.
 These cells have some interesting properties, notably they're also resistant to kind of out of distribution errors.
 You can see that in this video where you can see it's classifying it fairly solidly as 1s.
 But as soon as you draw a shape that is not kind of in the training set or in the classes of the training set, the cells keep disagreeing with each other.
 So this you can see as sort of kind of a robustness to out of distribution samples.
 And it's also pretty interesting to see that the messages here, where they go from.
 So you can fairly clearly see that if you draw some kind of shape, that the message passing starts at kind of the most symbolic parts of the digits.
 And here they have some chimeric digits or something they call it like this.
 And just pay attention to where the messages start.
 And you can clearly see that this sort of local determination of what a digit is will spread out over time to the other cells.
 And I thought there was this last thing.
 This thing. Yes. So here, not only do they visualize the cell state, so the color of the cell, and that's the thing on the left, is always the first 10 entries in this hidden state.
 But on the right, they also visualize the other hidden entries.
 And so each entry is represented by a two color thing where blue is very low number, red is a very high number.
 And here you can see what these latent states pass around.
 And also you can fairly clearly see that they do pass around these kind of typical sub shapes of the digit.
 So in the case of a zero, that's going to be a bend.
 In the case of a four, that's going to be these ends and corners of the numbers.
 And you can see that over time, as these messages pass, also the cell states on the left, the visible states, the class labels change over time.
 This lends a lot of credence, especially the six I like.
 Or the two. You can see in the different, if you kind of look at the different latent states, that the kind of typical, the bends, the corners,
 every latent state is sort of assigned to one of them.
 And then they pass this information around in order to reach an agreement.
 So I like this research, pretty cool research.
 I don't want to say it's very useful, but certainly it's very interesting.
 And I also like the format in this distilled format.
 I think that's sort of the future of research rather than eight page PDFs.
 You can look at it, it's interactive, you can have a little demo in it.
 You can write for as long as you want.
 And yeah, it's just overall better. This is still going.
 Doesn't know what it is.
 So lastly, you can, as I said, you can clearly see that, look, if I do this, it's a zero.
 But if I do this, then the stem part will immediately go for a six because that's indicative of a six.
 But then it will disagree with the zero part of the digit.
 In fact, I seem to be unable to write a six. Is that an American six? Maybe.
 Yeah, so with that, I'll leave this here.
 I think this is, again, very interesting, this kind of biological models.
 And certainly, if you're looking for an exciting research directions, this might be it.
 And you do not need a lot of resources to do this.
 This is very parameter efficient, as we saw in the last paper.
 And certainly kind of a niche right now.
 So that was it for me. I hope you enjoyed this.
 If you liked it, share it out and bye bye. See you next time.