GLOM: How to represent part-whole hierarchies in a neural network (Geoff Hinton's Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence geoff hinton geoff hinton capsule networks geoff hinton neural networks geoffrey hinton geoffrey hinton deep learning geoffrey hinton glom hinton glom glom model deep learning tutorial introduction to deep learning capsule networks computer vision capsule networks explained google brain google ai schmidhuber transformer attention mechanism consensus algorithm column
#glom #hinton #capsules

Geoffrey Hinton describes GLOM, a Computer Vision model that combines transformers, neural fields, contrastive learning, capsule networks, denoising autoencoders and RNNs. GLOM decomposes an image into a parse tree of objects and their parts. However, unlike previous systems, the parse tree is constructed dynamically and differently for each input, without changing the underlying neural network. This is done by a multi-step consensus algorithm that runs over different levels of abstraction at each location of an image simultaneously. GLOM is just an idea for now but suggests a radically new approach to AI visual scene understanding.

OUTLINE:
0:00 - Intro & Overview
3:10 - Object Recognition as Parse Trees
5:40 - Capsule Networks
8:00 - GLOM Architecture Overview
13:10 - Top-Down and Bottom-Up communication
18:30 - Emergence of Islands
22:00 - Cross-Column Attention Mechanism
27:10 - My Improvements for the Attention Mechanism
35:25 - Some Design Decisions
43:25 - Training GLOM as a Denoising Autoencoder & Contrastive Learning
52:20 - Coordinate Transformations & Representing Uncertainty
57:05 - How GLOM handles Video
1:01:10 - Conclusion & Comments

Paper: https://arxiv.org/abs/2102.12627

Abstract:
This paper does not describe a working system. Instead, it presents a single idea about representation which allows advances made by several different groups to be combined into an imaginary system called GLOM. The advances include transformers, neural fields, contrastive representation learning, distillation and capsules. GLOM answers the question: How can a neural network with a fixed architecture parse an image into a part-whole hierarchy which has a different structure for each image? The idea is simply to use islands of identical vectors to represent the nodes in the parse tree. If GLOM can be made to work, it should significantly improve the interpretability of the representations produced by transformer-like systems when applied to vision or language

Authors: Geoffrey Hinton

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there. Today we'll look at how to represent part-whole hierarchies in a neural network by the legend himself Jeffrey Hinton. He describes a system also known as GLOM that is a new approach to processing visual information using neural networks. And interestingly, the paper starts off by saying this paper does not describe a working system. So this is an idea paper, Jeffrey Hinton's suggestion of how we should go about solving vision or furthering vision in the AI community. He says openly, these are just ideas. Please prove me right, prove me wrong, try them out, and so on. And I absolutely welcome this. Idea papers is a thing that I think we have lost as a community because everything needs to be state of the art and so on. This is super cool, and I encourage more people to do it. I'm not saying you're going to have the same kind of success with an idea paper as Jeff Hinton. He is banking on his name in large part with this, but nevertheless it's just an archive paper. I see people complaining, this would never be possible if it wasn't. Yeah, it wouldn't. People wouldn't pay attention, but you're welcome to write your ideas and post them on archive, or write a blog post, make a YouTube video. Anyone has opinions? So go ahead. So to the paper itself, GLOM, as you can see here, GLOM stems from agglomeration, is a system that instead presents a single idea about representation, which allows advances made by several different groups to be combined into an imaginary system called GLOM. The advances include transformers, neural field, contrastive representation learning, distillation, and capsules. GLOM answers the question, how can a neural network with fixed architecture parse an image into a part-whole hierarchy, which has different structure for each image? The idea is simply to use islands of identical vectors to represent the nodes in the parse tree. If GLOM can be made to work, it should significantly improve the interpretability of the representations produced by transformer-like systems when applied to vision or language. That's the abstract. We'll dive into the system. We'll see what it's about. I think I can actually make a suggestion to improve it. But maybe I'm way behind other folks. So what is the GLOM system? And what are these parse tree about? And why does it combine all of these things? And for that, we look at so it has two core diagrams here. This is the first diagram. This is the second diagram. And at first sight, they have little to do with each other. So let me try to go about it like this. If you have an image, and Hinton looks at vision very much in terms of you have an image or a video, and you want to parse the image into kind of a tree. And the tree should be sort of like a tree of objects and their parts. So let's say it's an image of a car. So the whole notion is very, very object centric. So this is like my best attempt at a car. And a parse tree for this image would look something like this. All right. So this whole thing here is a car. So that's going to be your top node in the parse tree. The car has different parts, namely, it has this cabin, it has a motor and has wheels. So that is going to be those are going to be kind of downstream of that parse tree. Then the cabin itself is going to have two segments here, windows, and maybe here is the door area. So that is going to be window window door, and so on. So you get that we what we want to do is we want to look at an image, sort of create this parse tree over here, this is very much into the into the area of go fi good old fashioned AI people that want to understand a the world in terms of their symbolic representations and relation of the symbols to each other. However, what Hinton is saying is that if you simply do this, it's, it's, you know, you can't really do this with neural networks, neural networks are continuous, and so on. So what would you have to do in In addition, we know that the brain doesn't reconfigure itself every single time you get a new input. So the brain, even though it has some neuroplasticity, while you look at the world and do inference in the world, the connections stay the same. So what we need to do is we need to come up with a system that when we input one image, it can give us one parse tree. But when we input another image, it can give us some kind of other parse tree, maybe now there are two objects in the image. And this one has one descendant only, which in turn has two descendants, and so on, you see the point, the tree structure needs to be different each time. This in part was addressed by Hinton's capsule networks. So in the capsule networks, Hinton's idea was sort of, okay, I'm going to have these capsules here in different layers. And I'm going to have kind of lots of capsules in these layers, lots of capsules in these layers. And I'm going over capsules, because it's kind of important here. So Hinton's idea with capsules was that the first layer of capsules would sort of recognize the smallest parts. So this would be kind of the wheel capsule. And this would be sort of the window capsule, and so on. So there would be a single capsule for every part that could possibly be in an image, right? You already see the limitations. Because if you want to recognize the whole world, you need many capsules. But nevertheless, this was the idea. So a capsule would be active if there was the given object in the image. And then the next thing here, this would be kind of the the motor capsule. So the motor motor capsule, and this would be the cabin capsule, and so on. So the window would activate the cabin capsule, but the door capsule would also activate the cabin capsule, and so on. And the wheel would maybe activate it would maybe activate, I don't know, the wheel should probably be here as well, wheel at this level would activate that and then all of these things here would activate the car capsule. So you can see that this parse tree here is generated dynamically, right? These connections, this routing in capsules is generated every time different. So in the next image, there could be a different object, different capsules are activated, different things are routed together, the parse tree is different. However, you need these many, many capsules for that every one capsule per possible part in the image. And that was just infeasible. And also the routing was very cumbersome in these capsules. So here we go with a new approach. And this new approach is what Hinton describes as the glom architecture is composed of a large number of columns, which all use exactly the same weight. Each column is a stack of spatially local auto encoders that learn multiple levels of representation for what is happening in a small image patch. Okay, so we're going to build up some kind of imagination here. At the at the bottom level, we have our image. So our image is going to be lying flat on the ground, maybe you can see it like this. And it is going to be divided into pixels or small patches, whatever you want. But these are would be called locations. So it would be divided like this into different locations. I am not good at perspective drawing. In any case, above each location, there would be one of these columns. And these columns, I can draw one here, these columns would sort of stack up like this. And these columns would be divided into multiple levels. So there would be a bottom level, which would be this there would be a middle level, higher level, and so on. Hinton suggests about five levels should probably do. And every single level of this column tries to represent the location at the image, right this location down here in a different resolution. So the very bottom level might be aware that there is a part of a wheel like let's say this is actually let's say this is a cat. So here, there's probably Yep, yep. Okay, so you can see there is there is an ear or a part of an ear that stays as a part of an ear in this location. So the very bottom thing would probably represent something like the very structure of the fur. So the bottom thing would represent what's going on at you know, the micro level really the location level, the next layer would represent what's going on at this location in a kind of a broader sense. So that might recognize that that that's an that's actually part of an ear, right. So it goes beyond the location. If you think convolutional neural networks, you're right. So you're going to have a very similar network. So if you think convolutional neural networks, you're in the right ballpark, but we're going to implement this differently. The next layer will recognize well, this location is part of a of a cat of a cat's head. And then the next location will recognize well, this thing is a cat. Now there there is a cat at other places. But at this location, there is a cat, and so on. So maybe we don't have more and this locate at this particular image. But if you consider a different column, like this, this column right here, and you look at what's going on in that column, you'll see similar. So in the top layer, let's just consider the cat the top layer, in the top layer, it might say, well, there's a cat too. But it's also part of it's part of a cat's neck, neck. And then here it's maybe there's a bunch of, well, I don't know, a chin. And there is also a fine first structure of the chin. So you get the idea, every column will build up these rep these representations. And these are vectors. So these are embedding vectors. So at the bottom location, you'd have the fur vector, and then this vector is the ear, whereas here over here, the chin would be very different, or be a different vector at the same layer. So the only thing that agrees here is the cat vector, the cat vector in this top layer would agree between both of these columns. I hope you get the idea, you have a column above each of the locations, every single layer in the column represents that particular location, but at a different level of abstraction and a different level of I don't want to say resolution, but it it would consider more and more of its neighbors. The question is, how does it consider its neighbors? And how do you learn these things, right? So how do you learn these different abstractions? And that's where these columns, they communicate with each other. So Hinton imagines that this is a process over time, where the columns iteratively communicate to each other. And within the column, the layers communicate to each other. And this is one of these first diagrams right here. So this is one single column over time. Okay, this is this would be the, this would be the fur at the ear, this would be the cat's ear, and this would be cat. Okay, so the information that so the embeddings are updated by sending information around every single information around every single embedding, which means that every single vector at every single layer of every single column is updated by simply averaging four things. So we have the embedding at layer l, at time step t plus one is going to be sorry at layer l location x is going to be a sum between the four parts, the four following parts, it's going to be the embedding at the last time step, right? So this is sort of a recurrent neural network. We the new embedding is the old embedding plus it's going to be a function at a top down, that's what Hinton calls top down function of the embedding at the same location in the previous time step at one layer above. So l plus one it is also going to be receiving information from the upwards, I think bottom up, because the bottom up embedding of layer l minus one at the same location at time step t. All right, so this would that's what you can see right here. The green arrows are each level each layer simply passes information to the next time step. This is if any if nothing else happens, you just keep your embedding. Then each embedding also sends itself through a neural network one layer above itself. That's the blue arrows. So the blue arrows here are these and you every everything is a neural network here, every arrow except the green ones, but the green ones could be too. So every arrow is a neural network. So this is a neural network sending information above. And this is intuitive, right? So the ear embedding would sort of send information about itself like saying like, hey, I'm a cat ear sends it above and it goes through a neural network because it needs to be transformed. The neural network has to learn. Well, if it's a cat ear at that level, it might be a cat at the top level. And lastly, every single layer sends information down and that is the red arrows right here. They're also neural networks. So the cat ear says, well, I'm a cat ear. So downstream of myself, there might be, you know, some first structure. So all of these embeddings, they try to predict each other, they try to predict the neighbors of themselves. And Hinton's idea is that by aggregating over time, they will sort of reach a consensus of what is in these columns. Okay, there are a few things missing right here. The one thing that's missing and Hinton pointed this out that all of these different columns that we've drawn, they use the same weights. Okay, so, and he discusses this at the end of the paper, it's not really biologically plausible, but there's an ensemble effect. We won't go into that. But all these, these, so the blue arrows are always the same for each time step, but not necessarily the same between different layers. So that might be this F might be different from this F down here. However, the function passing information from from layer L to layer L plus one is the same in every single column across the image. It's a bit like a convolutional network in terms of weight sharing. So you can imagine it as one by one convolutional network in that sense. But except the information does not only go up the layers, it also goes down the layers over time. As I said, this is an iterative procedure, goes up, down, and laterally. The second thing is, now that you ask, oh, well, if every single column has the same weights, wouldn't that simply sort of how how can you localize any information? And the answer is that you have a side input, like in a neural field, you have a side input annotating each location, basically a positional encoding, honestly. So in in addition to what the image patch looks like, you also get your kind of either your x y coordinates, or you could also get your relative coordinates to some other coordinate frame in there. And so the network knows where it is. And that's going to be important, because what Hinton wants to build are these islands. So the imagination of Hinton is that this is going to be somewhere in between like after time step 10, and you want to run it for 100. And he imagines that there will what will emerge are these sort of islands. So imagine the image is now a 1d vector down here. Or you can imagine these columns in 2d, whatever fits, you know, whatever fits your brain better. But imagine the images, the image is simply the image is simply a 1d line right here. He imagines that the bottom vectors, they will just, you know, happily kind of be describing whatever that is at the very bottom level. But then at the next level, once it goes to sort of higher resolution or lower resolution, higher abstraction, there will be there must necessarily be vectors that are the same if the system works and look at these two vectors and look at these two vectors, they are the same because they now describe objects that are larger than one location, right, the cat's head is larger than simply one location. Therefore, at the layer that represents the cat's head, you expect because these are all all neural all the up and down functions in the same layer have the same weight, you expect that the embedding of a cat's head is the same in in the different columns. Right, that this is if the system works, this must be the case. And then as you go up, you expect more and more of these what what Hinton calls islands to emerge, right. So they they agree. And the idea. The idea between all of this message passing is that over time, all of these things kind of reinforce each other. So we looked at a column before, and we maybe said, okay, so this vector down here, it gets information from the top saying, hey, you know, there's a cat here. So you might be like a cat ear or a cat eye or something like this. And then it gets information from the bottom saying, well, there's a bit of there's, you know, fur here, and there's some cartilage showing and so on. And it has already sort of figured out that it might be an ear. And these informations they own they reinforce itself now like they'd be like, okay, you know, you're saying I'm part of a head and you're saying there's a bit of fur and cartilage. And I already kind of noticed that I'm a bit like an ear. So I'm probably more an ear. So the idea is that over time, you have this consensus algorithm, there's one thing missing. And that is, how do the different columns communicate with each other. So I said there are different parts, there is one missing. And that one missing is going to be, I'm just going to call it whatever a and a is going to be an attention mechanism across all the other columns at the same layer. So if we look here, this cell receives information from above from below from itself, and also, in an attention mechanism way, it's going to receive information from all of the different, all of the different embeddings at the same layer, you can see that, you know, hidden puts in everything we got in here. Now the attention, he says, is easier. And So these are the four parts right here. At each discrete time, and in each column separately, the embedding at a level is updated to be the weighted average of four contributions. The prediction produced by the bottom up neural net acting on the embedding at the level below acting on the embedding at the level below at the previous time, the prediction produced by the top down neural net acting on the embedding at the level above at the previous time, the embedding vector at the previous time step, these three we got, and then the attention weighted average of the embeddings at the same level, right at the same level in nearby columns at the previous time. So nearby, he, sorry, he later backpedals a bit, I think, on nearby and what nearby exactly means. And he at some parts, so this this is idea, I think this is still up for debate. And this is, I think, where I can help. But what he wants to do is he wants to aggregate, he wants to attention aggregate, and he wants to simplify attention. So instead, what we usually have is we're going to produce queries, and keys and values, queries, keys, and values, and they're all going to be different functions of our input. And then we're going to do query times key transposed softmax of that times value, and that is going to be our attention mechanism that allows you know, arbitrary information to be routed around and so on. Hinton says, Nope, what I want is simply that all the queries, the keys and the values, they're all just equal to the embeddings themselves. So the attention mechanism would work out to be the softmax of x times x transposed times x. And what that does is if you yourself are the query, and every vector also itself is the key, what do you attend to, you attend to vectors that are very similar to yourself. And you can see that in Hinton's diagram, the one we circled dark blue, what would it attend to? Well, it would probably attend to its left hand neighbor, the one you can see circled, I'm going to circle it. This one, it will probably attend a lot to this one, it might not attend so much. And the ones over here, it might not attend at all. So what we're going to do is we're going to try to attend to this one to be sure that we have the right thing. You see, this is a consensus algorithm, it is not meant as a way to pass information around, this is not meant like in a transformer as a way to do computation because we have no trainable weights in this process. It is simply meant as a consensus algorithm. So in imagines that by doing this, by sort of attending to things that are similar to you and then integrating their values, there will be these islands forming. And that's what you see right here. You can imagine if two vectors are already close at the same layer, this mechanism will make them even closer. So this is a sort of a clustering algorithm. And so that my question is that these drawings, you look at them, they are very specifically constructed, they're constructed such that a parse tree is emerging. So when you look at this, you have a clear sense I can probably I can probably move all of that crap out of the way. You can see the parse tree, right? Because the black thing is going to be the top node right here, let's leave away the scene level embedding for now, the black thing is going to be the top node. And then it has two child nodes, this one, and this one. And then it has four, every one of those has two child nodes. But it's not it doesn't have to be in this case. So this dynamically and every one of them, you know, the black ones are individual. This is dynamically constructing a parse tree, right? The parse tree here is something like this. And then the the the So this is pretty cool. But it is also drawn deliberately such that a core problem does not arise. And the core problem would be something like, well, what if this vector here was actually also pointing like this, okay, so it is not in it is not in the same. It is not in the same area of the parse tree, right? If you go down the parse tree, it is actually here. Now, if we do what Hinton says, and if for this vector here, we do this aggregation via attention on the same layer, what we will attend to is this vector over here. Now, this is probably not meant to be because this vector over here, it can represent the same thing. But you can see it's not in the in the same path of the parse tree. And he mentions this a little bit throughout, but not necessarily clear. And the drawing makes it seem like there's no problem. But I hope you can see how this is a problem. The attention would pull in information from over here. However, the whole parse tree here and the island on the top layer suggests that these two things should be parsed independently from each other and therefore also processed independently from each other. So here is my suggestion to to extend this and maybe Hinton's already thought of this. But I would suggest that the this attention mechanism here is modulated by how close two things are in the parse tree. So what would that be? So for a given a given vector, it would be how much do you attend to this vector right here? Well, a lot because it agrees with you, right? It you know, this the softmax of the inner product would be high, it agrees with you. And also it is in the same, it is the same branch of the parse tree. So that's perfect, right? This one right here doesn't agree with you, but is in the same branch. So it could potentially later agree with you through a consensus algorithm. However, this one over here, I, you probably shouldn't attend to that too much, even though it points in the same direction, because it's in a different branch of the parse tree, you shouldn't attend zero to it like because these branches on top, they could change. And you know, by you sending information there, this one could change the the top structure here that could agree more with your branch of the parse tree and so on. So my suggestion would be that let's not only get the softmax of the, let's not only get the softmax of the current layer things, but let's do x times and here we're going to have a sum. So this is going to be k. And let's say we're at we're at layer L. And this is layer one, this is layer two, this is layer three, going to number them from the top, actually from the bottom layer m, layer m minus one, and this is layer L. I suck at this. So from the current layer, I want to go up the hierarchy until layer one. And I'm going to take the softmax of the representation at layer L at layer k, where I'm at x k transposed like this. What we aggregate is still the the values on the current layer, but how much we should attend to that should be dependent on the parse tree. And we do that like this. And maybe we have like a kind of a lambda k, L minus k, L minus k. I hope you get what I mean. So how much how much you aggregate this sum here, the sum here is weird. This should go probably. Hi, it's future Yannick. And I just wanted to write that down again. So because I've made some mistakes, obviously, the sum here should be within the softmax because you want to aggregate the distributions in log space, and the softmax should still be valid, you know, distribution. And then the lambda is exponentiated by k and k now properly runs from the zero to all the way up the stacks. So big L would be the total number of layers and little l would be the layer where you're currently at. And you can clearly see that the contribution of these attention matrices it is so lambda would be something smaller than one. And therefore, the contribution is in the current layer is the strongest, but also in the next one up is a bit weaker than one more up is even a bit weaker and so on. So you'd still have essentially the same mechanism as Hinton is suggesting controlling for the fact that things are in different branches of the parse tree. All right, back to classic Yannick who is thoroughly confused by these things. Yeah, I'm not good at I'm not good at coming up with math on the spot. But I hope you can see what it's doing. So it is if, if you simply take the first k, you would simply stay at that layer and it would be what Hinton said. But what I'm saying is you should also consider how much your top your higher layer, one layer up from you agrees with one layer up from the thing you want to attend to. So you also compute that inner product between between the embeddings, and you add that to the softmax distribution. So initially, the softmax distribution would be like you should tend to this thing and this thing, and this thing a lot. But then the next up hierarchy would maybe say, well, we agree, because you know, these are in the same thing, but this one, maybe not so much. And you would add those together, maybe with a lambda factor in here, and then you go one layer up and it would say, well, okay, everything over here basically agrees, right and here, no, but everything over here basically doesn't agree. So you would add that maybe with a lambda squared, as you go up the layers, it would be less and less important, but still you'd consider it. All right. Now, if this is gonna work out, site the channel. Now back to what Hinton says that this is actually the system. This is the system as in a nutshell, you're gonna input the image at the bottom. And Hinton says you could use like a convent at the very bottom to get it into the columns. But then you're going to every time step pass information up the columns down the columns, and between the same layer of the different columns. And that's going to, in some point, this is going to stabilize, I don't know if it has cycles, it probably doesn't have cycles. This is good. Yeah, probably does not have cycles. So at some point, this comes to an end. And if that comes to an end, it should be that the object level embeddings agree on an object, the part level embeddings agree on what parts there are, the sub parts agree, and so on. And they form these islands, these islands give rise to a parse tree. And the parse tree can tell you what object is there, what is it made of, and where are these parts in the image, and so on. So exactly, that is it. And now we're going to look at what Hinton calls some design decisions. How many levels are there? About five. Okay, we can skip that. How fine grained are the locations? Hinton says you could be as fine grained as pixels, or they could correspond to larger image patches. You and he says you could do convolutional neural network to get it in there. Does the bottom up net look at nearby locations? He says, yes, the bottom up net, so this this is not the attention network, that's the bottom up network, it could look at nearby locations. But Hinton imagines that if you have bottom up, top down, and if you have attention drawing information, and if you maybe limit that attention to a neighborhood, then then the the attention will do the job because you can have instead of looking at neighboring locations in the bottom up network, you can simply in two time steps, aggregate that information. So you can do bottom up here, bottom up here, and then using the attention, the lateral mechanism, you can pass that information around this way. And also, it is not as biasing the network to the immediate neighborhood. So the attention mechanism can sort of look farther, which conflicts with what he's saying on top that the attention mechanism might only be looking at the neighbors. I think there are different possibilities here. And only looking at neighbors is actually one of the solution to the problem of having, you know, kind of similar vectors at very distant locations at down the levels. But I think it's not as as good a solutions to simply look at how close things are in pixel space, because even though things are close in pixel space, they might be far away in the parse tree space. How does the attention work? We've already looked at this. So the way that one location attends to another location is going to be the softmax of the inner product between the embeddings here. And the values are also going to be just the embeddings that layer at that layer. The visual input, he says convolutional net could be used. Color and texture. He says, he makes he gives this example, like if you know, if an object is entirely pale or entirely green, or entirely, I don't even know how to pronounce this, the color of a part is straightforward. But what color is the whole object. So this entire notion of capsules, by the way, imagines this as these embeddings represent kind of properties of the object so that the the cat ear embedding represents not only the fact that it is a cat ear, but also different properties about the cat ear and even its location in the image is in the embedding. And, you know, we know that transformers, they must be doing something like this, because we feed in positional embeddings, for example, at the very bottom, and it can still, you know, compute things in terms of positions. So that's the there's an intrinsic connection between kind of capsules and the kind of transformer architecture. He says, one of the motivations of Glom was idea that the whole object has a compound color, which might be called pale green or move. And at the object level, every location belonging to the object has exactly the same compound color. So the object is whatever this all over, when deciding which other locations the object level attend to preference would be given two locations with a similar compound color. So what he's saying right here is that, you know, you could give preference to two similar color locations, when you decide what you want to attend to. But the color isn't as easy as simply saying what color is there in the location that you are at. But you could be so if this is green, and this here is blue, then the bottom layer would say yes, I'm green. And yes, I'm blue. But they could also be saying, well, I am part of a green blue object, right. And then the the higher layer here, you know, attending or caring about multiple or bigger region, its color would then be, you know, green blue, and the consensus could reach on, well, we are a green blue object, even though the object isn't a pure green or pure blue all throughout. So he I think, yeah, it's it's I think it's a side suggestion, maybe he has this as a core motivation between the system. But it's just interesting to see how he thinks of things and he extends the color here to textures and even shapes. Shapes, the individual texture elements have their own shapes and poses in spatial relationships, but an object with a textured surface has exactly the same texture everywhere at the object level. Glom extends this idea to shapes, an object may have parts that are very different from one another, but at the object level, it has exactly the same compound shape in all of the location that it occupies. Basically saying that, okay, every pixel that's part of a cat head is a cat head has the shape of a cat head, even though the individual locations might not recognize that, and that information could be passed around through this consensus mechanism over time. So the cluster discovery versus cluster formation, we've seen that and he makes a lot of he makes a lot of analogies to face recognition. But yeah, the clusters are not the islands of similar embedding vectors at a level can be viewed as clusters, but these clusters are not discovered in immutable data. They are formed by the interaction between the intra level process that favors islands of similarity and dynamically changing suggestions coming from the locations embedding at adjacent levels. So the core here is really this consensus algorithm that creates these clusters. And yeah, the clustering algorithm doesn't work by simply looking at embeddings and deciding which ones go together, but the embeddings themselves update themselves in order to form clusters. And yeah, this is replicating embedding vectors. This is a response to a criticism that I guess he got where someone said, well, why don't why do you represent if you have these, you know, these columns at the bottom, it makes sense, you have all the different vectors. But then as you go up, you know, you have that kind of the same vector for all locations, because it's the same object. Why does it make sense to replicate that everywhere, and not just have one, because, you know, in a database, we just have one. And it basically says that in order to reach the consensus first, of all, it's important to have different vectors, they might be slightly different. So they might have some nuance in them, because, you know, they might get pulled into different directions from the sign of bottom up signal, then from the consensus algorithm on the same layer. So I, you know, I believe that it is that is important. Here, I think it's just this is a criticism he got. And then he decided to put this in here, learning islands. So what we haven't discussed about this yet is how this is trained and Hinton says this is trained as a denoising auto encoder. Let us assume that Glom is trained to reconstruct at its output, the uncorrupted version of an image from which some region has been have been removed. So he goes into self supervised learning with the system. This objective should ensure that information about the input is preserved during the forward pass. And if the regions are sufficiently large, it should also ensure that identifying familiar objects will be helpful for filling in the missing regions. To encourage islands of near identity, we need to add a regularizer. And experience shows that a regularizer that simply encourages similarity between the embeddings of nearby locations can cause representations to collapse. All the embedding vectors may become very small, so that they are all very similar. And the reconstruction will then use very large weights to deal with the very small scale to prevent collapse. And then he says contrastive learning is the answer to this. So how do you regularize the model such that this consensus is formed? He says contrastive learning might be useful, but you can't simply apply it straight out. So it learns to make representations of two different crops of the same image agree, and the representations of two crops from different images disagree. But this is not a sensible thing to do if our aim is to recognize objects. If crop one contains objects A and B and crop two from the same image contains objects B and C, it does not make sense to demand that the representation of the two crops is the same at the object level. Okay, so he says that contrastive learning is good, but you have to pay very careful attention at which layer you employ it. Because if you go down far enough, then contrastive learning, especially this type where you crop the image into different parts, and you say, well, since it's the same image, the representations should agree. Hinton would say, well, at the top layer, yes, but at the bottom layer, certainly not, because they display different things. So you have to be careful where you apply this contrastive learning. And he gives a bunch of suggestions on how to solve that. He says things like, well, negative examples, for example, might not might not even be needed. Well, that's it. Sorry, that's a different thing. So the obvious solution is to regularize the bottom up and top down neural networks by encouraging each of them to predict the consensus option. Yeah, this is the weighted geometric mean of the predictions coming from the top down and bottom up networks, the attention weighted average of the embeddings at nearby locations at the previous time step, the previous state of and I guess, and there should be an end and the previous state of the embedding training, the inter level prediction to agree with the consensus will clearly make the islands found during feed forward inference be more coherent. So he says you could regularize the model to to regress to the consensus option. So it's sort of like a self a self regression. And he asks whether or not that will lead to a collapse, because if you don't have negative examples and contrastive learning, this could lead to simply a collapse. An important question is whether this type of training will necessarily cause collapse if it is not accompanied by training the inter level predictions to be different for negative examples that use the consensus options for unrelated spatial contexts. So here is that problem. Right. If you use the consensus opinion for unrelated spatial context, that might be a problem. He says using layer batch norm should reduce the tendency to collapse, but a more important consideration may be the achievability of the goal. It goes into why regularization could help. And he says if however, an embedding at one location is free to choose which embeddings at other locations it should resemble, the goal can be achieved almost perfectly by learning to form islands of identical vectors and attending almost entirely to other locations that are in the same island. And I don't know, I don't know if this is what I suggested. So I guess this is kind of a convoluted paragraph, and I had to also read it multiple times and I still don't exactly know what he's trying to say right here. But I think what he's saying is that what we want to do is we want to sort of regularize the network to produce this consensus, right. So we have a bottom up signal, a top down signal, we have a current value, and we have the signal from the attention mechanism. Now, what we want to do is we want to reach a consensus such that these islands form. However, if you attend to any sort of things here that have nothing to do with you, you might not be able to reach this consensus, right. That's, I think that's the problem I think he's touching on the problem that I said before. So what he says is, you know, what you should do is you should simply attend to things that are in the same islands already. So if an embedding at one location is free to choose which embedding at other locations it should resemble, the goal can be achieved by learning to form islands of identical vectors and attending almost entirely to other locations that are in the same islands. Now, I think here, what he's doing, he makes the case for the attention mechanism itself, right. So he says, if, if we simply draw in information from the same layer here, you know, anything, any old information might come in, and we might collapse and or we might never reach consensus because any old information might come in. However, if we simply draw in information from the selected neighbors that already are in the same group in the same island as me, then this consensus algorithm works. So if the network, the network is now forced kind of to learn to build these islands of similar things in order to make this consensus work, if we regularize this consensus, then we can actually create a consensus that is similar to the one that we have in the same group. So I think that's the way to make this consensus work if we regularize this consensus. So I believe he makes the case for the attention mechanism. I don't think he, in this case, considers kind of the up the next up layer islands, what I would say is you need to go to the columns in order to decide which things, which locations, right, it's free to choose which embeddings at other locations it should resemble. I think, yeah, this is the case for the attention mechanism. Okay, I hope you're still half with me. If not, I'm, I'm a bit confused because I think what he's doing is he says, contrastive learning would be good, you can use it, but you have to be careful at which layer you do it. Another regularizer to form these islands would be this regularize the network to conform to the consensus option, opinion. However, if you simply aggregate information from the same layer, then that wouldn't work because, you know, the different things in the same layer might correspond to completely different parts of the image. Drawing in information from there would not help you. How do you solve this? By introducing the very attention mechanism that he introduced in order to only draw in information from parts of the same layer that actually are related to you. Okay, the next thing, the next consideration he does is representing coordinate transformations. So how does this represent coordinate transformations? There was a capsule net paper where he explicitly represents coordinate transformations in kind of four dimension quaternion space. And he says that is probably not needed because you don't want to, he says you could represent this by four by four matrices. However, if you simply allocate 16 numbers in each embedding vector, in order to represent the part whole coordinate transformation, like the transformation that relates the part to the whole, that does not make it easy to represent uncertainty about the aspects of pose and certainty about others. So the problem here is that we know that humans, when they watch something right here, when they watch a scene, like this is a chair, and there is a person, a very tiny person on the chair, we don't see necessarily the coordinate frame of the world. What we see is we see the coordinate frame of the chair, like maybe this is the center, and we see the person in relation to the chair, our brain seems to do this intuitively, and hinting things that a system like this should also do it intuitively. So somehow, the coordinate transformations involved going from the eye to the reference through the frame of the chair, and then from the chair to the person, they should be somehow in encoded in this network. However, he also says that it's probably not necessary to encode them explicitly as you know, explicit coordinate transformations, because not only does that make it harder, probably to learn, but also, you can't represent uncertainty. In fact, you can represent uncertainty, that's the next thing right here, much better by having a higher dimensional thing that you're trying to guess, right? If you are trying to guess a distribution with three components, and you simply have a three dimensional vector, you have no way of representing uncertainty. However, if you have a nine dimensional vector, you can have three opinions about the distribution. So this is an opinion, this is an opinion, and then this is an opinion. And then you can sort of aggregate and you can say, well, I'm pretty sure about these two things, because all my opinions are pretty close. But this one here, I'm not so sure because my individual things say different things, things say things. All right, I've this video is too long. So that's his argument right here, we don't need explicit representing of uncertainty, because by simply over parameterizing, we can already represent uncertainty well. And we also don't need disentangled position information and, and so on. Sorry, we don't need different position information. Because, again, the network can take care of that. And he gives a good example, like why would you have disentangled coordinate frame if you have an image. And in the image, the picture in it is this. How do you know if that is a rhomboid shape? Or if it is a rec, if it is a rectangular piece of paper viewed from the side, I should probably draw it way closer, something like something like this. I suck at this. You get probably get what I mean. Like, if it is a different object, it has a like the object and the coordinate transformation are dependent upon each other. And so it makes sense for the neural network to actually entangle the two, because the two things depend on each other. In essence, he's just saying, don't worry about explicitly representing all of the different things. We got it, like the neural network can do all of these things, like uncertainty or position, and pose transformations. So here he compares it to different other architectures. Comparison to CNN, comparison to transformers, comparison to capsule models. And at the end, it goes into video. At the very beginning, he says the paper is about actually a video system. And you can kind of see that because we go through this algorithm in multiple time steps, right? You have, it's like you analyze an image with these columns, which gives you sort of a 3D, 3D tensor with the image at the bottom. And you go in the next time step, you have a new 3D tensor, right? You pass this whole information around with the image at the bottom. Hinton says, well, why does that need to be the same image? That could also be different images. So you could use the system to analyze video. So what he does is he says, at the same time, you do this time step to find agreement, you could actually swap out the video frame, the X, you can swap out the video frame, and produce a slightly different video frame. And you could actually have a kind of an ensemble regularizing effect. So as the whole columns here, the whole system comes to a consensus over time, you feed in different information at the bottom. And what he says is that, you know, if this is a slow enough video, then the top layers here would probably could still reach an agreement, while the bottom layers would change rapidly. But that could be sort of an ensemble or a regularizer regularizing effect that it even has. So he intrinsically connects these two time dimensions, because they would be separate, right, you could input a video, and then in, you know, in each frame, you could do this consensus finding algorithm. But he says, No, it's actually cool to consider them together to do the consensus finding while you sort of watch the video. It's just not clear that you always need the same amount of consensus finding steps as you need as you have video frames. So maybe you want to, maybe you want to take like five consensus steps per video frame or the other way around. Not sure. In any case, I think that's a pretty cool idea. And he says things like, if the changes are rapid, there is no time available to iteratively settle on a good set of embedding vectors for interpreting a specific frame. This means that the GLOM architecture cannot correctly interpret complicated shapes. If the images are changing rapidly, try taking an irregularly shaped potato and throwing it up in the air such a way that it rotates at one or two cycles per second. Even if you smoothly track the potato, you cannot see what shape it is. Now I don't have a potato, but I can give you an avocado. So if you give me a second, how's that? Could you track the shape? I don't know. Probably Hinton's correct. All right. He talks about is this biologically plausible? And I don't want to go too much into this. He discusses some restrictions like, yeah, we still use backprop and is backprop plausible and so on. I love this sentence. In the long run, however, we are all dead. And then the footnote saying there are alternative facts. But yeah, he discusses whether it's biologically plausible. How could you modify it to make it more plausible? For example, when you want to do contrastive learning, there is evidence that dreams during so during sleep, you do contrastive learning, like you produce the negative examples during sleep, and then during the day, you collect the positive examples, and so on. So I think this is a more speculative part of the paper, but it's pretty cool to it's pretty cool to read it. And lastly, he goes into discussion he also says that this paper is too long already. I'm going to just briefly talk about this. And he trashes the neuro symbolic people a bit like he trashes the people that say no, no, you know, neural networks can never do whatever. And he says pretty clearly look, neural networks can represent trees, I've given you a system also BERT can output parse trees. So shut up, I guess. And he comes up with this glom BERT name, which, you know, is is already coined if you wanted to do glom BERT, that's already taken. Sorry. I also by the way also coined then I coined the name may go mania. Right now. Okay, if you want to if you want to use it, it better be a pretty cool machine learning system and be based on glom. Right, that was the paper. I think it's a cool system. It has a bunch of parts that are maybe not super friendly to hardware at the time like this iterative procedure. But honestly, it is not much more than a neural network, sorry, a recurrent neural network with very complicated recurrence functions. The video extension might be a bit tricky. And, but the rest and the regularization might be a bit tricky, the exact objective. So the denoising auto encoder objective isn't super detailed in the paper, he simply says, reconstruct the corrupted version of the input. How exactly the input happens, maybe there's a CNN, maybe the CNN feeds information into actually multiple layers. None of that is exactly specified. So there's lots to figure out. I do think the ideas are very cool. And I love idea papers. And therefore, I recommend that if you're interested more, give this thing a read, give this video a like, share it out, and I'll see you next time. Bye bye.
 Hi there. Today we'll look at how to represent part-whole hierarchies in a neural network
 by the legend himself Jeffrey Hinton. He describes a system also known as GLOM that is a new approach
 to processing visual information using neural networks. And interestingly, the paper starts
 off by saying this paper does not describe a working system. So this is an idea paper,
 Jeffrey Hinton's suggestion of how we should go about solving vision or furthering vision
 in the AI community. He says openly, these are just ideas. Please prove me right, prove me wrong,
 try them out, and so on. And I absolutely welcome this. Idea papers is a thing that I think we have
 lost as a community because everything needs to be state of the art and so on.
 This is super cool, and I encourage more people to do it. I'm not saying you're going to have the
 same kind of success with an idea paper as Jeff Hinton. He is banking on his name in large part
 with this, but nevertheless it's just an archive paper. I see people complaining, this would never
 be possible if it wasn't. Yeah, it wouldn't. People wouldn't pay attention, but you're welcome to
 write your ideas and post them on archive, or write a blog post, make a YouTube video.
 Anyone has opinions? So go ahead. So to the paper itself, GLOM, as you can see here,
 GLOM stems from agglomeration, is a system that instead presents a single idea about
 representation, which allows advances made by several different groups to be combined into
 an imaginary system called GLOM. The advances include transformers, neural field, contrastive
 representation learning, distillation, and capsules. GLOM answers the question, how can a
 neural network with fixed architecture parse an image into a part-whole hierarchy, which has
 different structure for each image? The idea is simply to use islands of identical vectors to
 represent the nodes in the parse tree. If GLOM can be made to work, it should significantly improve
 the interpretability of the representations produced by transformer-like systems when applied
 to vision or language. That's the abstract. We'll dive into the system. We'll see what it's about.
 I think I can actually make a suggestion to improve it. But maybe I'm way behind other folks. So
 what is the GLOM system? And what are these parse tree about? And why does it combine all of these
 things? And for that, we look at so it has two core diagrams here. This is the first diagram.
 This is the second diagram. And at first sight, they have little to do with each other. So let
 me try to go about it like this. If you have an image, and Hinton looks at vision very much in
 terms of you have an image or a video, and you want to parse the image into kind of a tree.
 And the tree should be sort of like a tree of objects and their parts. So let's say it's
 an image of a car. So the whole notion is very, very object centric. So this is like my best attempt
 at a car. And a parse tree for this image would look something like this. All right. So this whole
 thing here is a car. So that's going to be your top node in the parse tree. The car has different
 parts, namely, it has this cabin, it has a motor and has wheels. So that is going to be those are
 going to be kind of downstream of that parse tree. Then the cabin itself is going to have two
 segments here, windows, and maybe here is the door area. So that is going to be window window door,
 and so on. So you get that we what we want to do is we want to look at an image, sort of create
 this parse tree over here, this is very much into the into the area of go fi good old fashioned AI
 people that want to understand a the world in terms of their symbolic representations and relation
 of the symbols to each other. However, what Hinton is saying is that if you simply do this, it's,
 it's, you know, you can't really do this with neural networks, neural networks are continuous,
 and so on. So what would you have to do in In addition, we know that the brain doesn't
 reconfigure itself every single time you get a new input. So the brain, even though it has some
 neuroplasticity, while you look at the world and do inference in the world, the connections stay
 the same. So what we need to do is we need to come up with a system that when we input one image,
 it can give us one parse tree. But when we input another image, it can give us some kind of other
 parse tree, maybe now there are two objects in the image. And this one has one descendant only,
 which in turn has two descendants, and so on, you see the point, the tree structure needs to be
 different each time. This in part was addressed by Hinton's capsule networks. So in the capsule
 networks, Hinton's idea was sort of, okay, I'm going to have these capsules here in different layers.
 And I'm going to have kind of lots of capsules in these layers, lots of capsules in these layers.
 And I'm going over capsules, because it's kind of important here. So Hinton's idea with capsules
 was that the first layer of capsules would sort of recognize the smallest parts. So this would be
 kind of the wheel capsule. And this would be sort of the window capsule, and so on. So there would
 be a single capsule for every part that could possibly be in an image, right? You already see
 the limitations. Because if you want to recognize the whole world, you need many capsules. But
 nevertheless, this was the idea. So a capsule would be active if there was the given object in the
 image. And then the next thing here, this would be kind of the the motor capsule. So the motor
 motor capsule, and this would be the cabin capsule, and so on. So the window would activate the cabin
 capsule, but the door capsule would also activate the cabin capsule, and so on. And the wheel would
 maybe activate it would maybe activate, I don't know, the wheel should probably be here as well,
 wheel at this level would activate that and then all of these things here would activate the car
 capsule. So you can see that this parse tree here is generated dynamically, right? These connections,
 this routing in capsules is generated every time different. So in the next image, there could be a
 different object, different capsules are activated, different things are routed together, the parse
 tree is different. However, you need these many, many capsules for that every one capsule per
 possible part in the image. And that was just infeasible. And also the routing was very
 cumbersome in these capsules. So here we go with a new approach. And this new approach is what
 Hinton describes as the glom architecture is composed of a large number of columns,
 which all use exactly the same weight. Each column is a stack of spatially local auto encoders that
 learn multiple levels of representation for what is happening in a small image patch. Okay, so
 we're going to build up some kind of imagination here. At the at the bottom level, we have our
 image. So our image is going to be lying flat on the ground, maybe you can see it like this.
 And it is going to be divided into pixels or small patches, whatever you want. But these are
 would be called locations. So it would be divided like this into different locations. I am not good
 at perspective drawing. In any case, above each location, there would be one of these columns.
 And these columns, I can draw one here, these columns would sort of stack up like this.
 And these columns would be divided into multiple levels. So there would be a bottom level,
 which would be this there would be a middle level, higher level, and so on. Hinton suggests about
 five levels should probably do. And every single level of this column tries to represent the
 location at the image, right this location down here in a different resolution. So the very bottom
 level might be aware that there is a part of a wheel like let's say this is actually let's say
 this is a cat. So here, there's probably Yep, yep. Okay, so you can see there is there is an ear or a
 part of an ear that stays as a part of an ear in this location. So the very bottom thing would
 probably represent something like the very structure of the fur. So the bottom thing would
 represent what's going on at you know, the micro level really the location level, the next layer
 would represent what's going on at this location in a kind of a broader sense. So that might
 recognize that that that's an that's actually part of an ear, right. So it goes beyond the location.
 If you think convolutional neural networks, you're right. So you're going to have a very
 similar network. So if you think convolutional neural networks, you're in the right ballpark,
 but we're going to implement this differently. The next layer will recognize well, this location
 is part of a of a cat of a cat's head. And then the next location will recognize well, this thing
 is a cat. Now there there is a cat at other places. But at this location, there is a cat, and so on.
 So maybe we don't have more and this locate at this particular image. But if you consider a
 different column, like this, this column right here, and you look at what's going on in that column,
 you'll see similar. So in the top layer, let's just consider the cat the top layer, in the top
 layer, it might say, well, there's a cat too. But it's also part of it's part of a cat's neck,
 neck. And then here it's maybe there's a bunch of, well, I don't know, a chin.
 And there is also a fine first structure of the chin. So you get the idea, every column will build
 up these rep these representations. And these are vectors. So these are embedding vectors. So
 at the bottom location, you'd have the fur vector, and then this vector is the ear, whereas here over
 here, the chin would be very different, or be a different vector at the same layer. So the only
 thing that agrees here is the cat vector, the cat vector in this top layer would agree between both
 of these columns. I hope you get the idea, you have a column above each of the locations,
 every single layer in the column represents that particular location, but at a different
 level of abstraction and a different level of I don't want to say resolution, but it it would
 consider more and more of its neighbors. The question is, how does it consider its neighbors?
 And how do you learn these things, right? So how do you learn these different abstractions?
 And that's where these columns, they communicate with each other. So Hinton imagines that this is
 a process over time, where the columns iteratively communicate to each other. And within the column,
 the layers communicate to each other. And this is one of these first diagrams right here.
 So this is one single column over time. Okay, this is this would be the, this would be the fur
 at the ear, this would be the cat's ear, and this would be cat. Okay, so
 the information that so the embeddings are updated by sending information around every single
 information around every single embedding, which means that every single vector at every single
 layer of every single column is updated by simply averaging four things. So we have the embedding
 at layer l, at time step t plus one is going to be sorry at layer l location x is going to be
 a sum between the four parts, the four following parts, it's going to be the embedding at the last
 time step, right? So this is sort of a recurrent neural network. We the new embedding is the old
 embedding plus it's going to be a function at a top down, that's what Hinton calls top down function
 of the embedding at the same location in the previous time step at one layer above. So l plus one
 it is also going to be receiving information from the upwards, I think bottom up, because the bottom
 up embedding of layer l minus one at the same location at time step t. All right, so this would
 that's what you can see right here. The green arrows are each level each layer simply passes
 information to the next time step. This is if any if nothing else happens, you just keep your embedding.
 Then each embedding also sends itself through a neural network one layer above itself. That's the
 blue arrows. So the blue arrows here are these and you every everything is a neural network here,
 every arrow except the green ones, but the green ones could be too. So every arrow is a neural
 network. So this is a neural network sending information above. And this is intuitive, right?
 So the ear embedding would sort of send information about itself like saying like, hey, I'm a cat ear
 sends it above and it goes through a neural network because it needs to be transformed.
 The neural network has to learn. Well, if it's a cat ear at that level, it might be a cat at the
 top level. And lastly, every single layer sends information down and that is the red arrows right
 here. They're also neural networks. So the cat ear says, well, I'm a cat ear. So downstream of myself,
 there might be, you know, some first structure. So all of these embeddings, they try to predict
 each other, they try to predict the neighbors of themselves. And Hinton's idea is that by
 aggregating over time, they will sort of reach a consensus of what is in these columns.
 Okay, there are a few things missing right here. The one thing that's missing and Hinton pointed
 this out that all of these different columns that we've drawn, they use the same weights. Okay, so,
 and he discusses this at the end of the paper, it's not really biologically plausible,
 but there's an ensemble effect. We won't go into that. But all these, these, so the blue
 arrows are always the same for each time step, but not necessarily the same between different
 layers. So that might be this F might be different from this F down here. However, the function
 passing information from from layer L to layer L plus one is the same in every single column across
 the image. It's a bit like a convolutional network in terms of weight sharing. So you can imagine it
 as one by one convolutional network in that sense. But except the information does not only go up
 the layers, it also goes down the layers over time. As I said, this is an iterative procedure,
 goes up, down, and laterally. The second thing is, now that you ask, oh, well, if every single column
 has the same weights, wouldn't that simply sort of how how can you localize any information?
 And the answer is that you have a side input, like in a neural field, you have a side input
 annotating each location, basically a positional encoding, honestly. So in in addition to what the
 image patch looks like, you also get your kind of either your x y coordinates, or you could also get
 your relative coordinates to some other coordinate frame in there. And so the network knows where it
 is. And that's going to be important, because what Hinton wants to build are these islands. So the
 imagination of Hinton is that this is going to be somewhere in between like after time step 10, and
 you want to run it for 100. And he imagines that there will what will emerge are these sort of
 islands. So imagine the image is now a 1d vector down here. Or you can imagine these columns in 2d,
 whatever fits, you know, whatever fits your brain better. But imagine the images, the image is simply
 the image is simply a 1d line right here. He imagines that the bottom vectors, they will just,
 you know, happily kind of be describing whatever that is at the very bottom level. But then at the
 next level, once it goes to sort of higher resolution or lower resolution, higher abstraction,
 there will be there must necessarily be vectors that are the same if the system works and look
 at these two vectors and look at these two vectors, they are the same because they now describe
 objects that are larger than one location, right, the cat's head is larger than simply one location.
 Therefore, at the layer that represents the cat's head, you expect because these are all all neural
 all the up and down functions in the same layer have the same weight, you expect that the embedding
 of a cat's head is the same in in the different columns. Right, that this is if the system works,
 this must be the case. And then as you go up, you expect more and more of these what what Hinton calls
 islands to emerge, right. So they they agree. And the idea. The idea between all of this message
 passing is that over time, all of these things kind of reinforce each other. So we looked at
 a column before, and we maybe said, okay, so this vector down here, it gets information from the top
 saying, hey, you know, there's a cat here. So you might be like a cat ear or a cat eye or something
 like this. And then it gets information from the bottom saying, well, there's a bit of there's,
 you know, fur here, and there's some cartilage showing and so on. And it has already sort of
 figured out that it might be an ear. And these informations they own they reinforce itself now
 like they'd be like, okay, you know, you're saying I'm part of a head and you're saying there's a bit
 of fur and cartilage. And I already kind of noticed that I'm a bit like an ear. So I'm probably more
 an ear. So the idea is that over time, you have this consensus algorithm, there's one thing missing.
 And that is, how do the different columns communicate with each other. So I said there
 are different parts, there is one missing. And that one missing is going to be, I'm just going
 to call it whatever a and a is going to be an attention mechanism across all the other columns
 at the same layer. So if we look here, this cell receives information from above from below from
 itself, and also, in an attention mechanism way, it's going to receive information from all of the
 different, all of the different embeddings at the same layer, you can see
 that, you know, hidden puts in everything we got in here. Now the attention, he says, is easier. And
 So these are the four parts right here. At each discrete time, and in each column separately,
 the embedding at a level is updated to be the weighted average of four contributions.
 The prediction produced by the bottom up neural net acting on the embedding at the level below
 acting on the embedding at the level below at the previous time, the prediction produced
 by the top down neural net acting on the embedding at the level above at the previous time,
 the embedding vector at the previous time step, these three we got, and then the attention
 weighted average of the embeddings at the same level, right at the same level
 in nearby columns at the previous time. So nearby, he, sorry, he later backpedals a bit, I think, on
 nearby and what nearby exactly means. And he at some parts, so this this is idea, I think this is
 still up for debate. And this is, I think, where I can help. But what he wants to do is he wants to
 aggregate, he wants to attention aggregate, and he wants to simplify attention. So instead,
 what we usually have is we're going to produce queries, and keys and values, queries, keys,
 and values, and they're all going to be different functions of our input. And then we're going to do
 query times key transposed softmax of that times value, and that is going to be our attention
 mechanism that allows you know, arbitrary information to be routed around and so on. Hinton
 says, Nope, what I want is simply that all the queries, the keys and the values, they're all just
 equal to the embeddings themselves. So
 the attention mechanism would work out to be the softmax
 of x times x transposed times x. And what that does is if you yourself are the query, and every
 vector also itself is the key, what do you attend to, you attend to vectors that are very similar
 to yourself. And you can see that in Hinton's diagram, the one we circled dark blue, what would
 it attend to? Well, it would probably attend to its left hand neighbor, the one you can see circled,
 I'm going to circle it. This one, it will probably attend a lot to this one, it might not attend so
 much. And the ones over here, it might not attend at all. So what we're going to do is we're going
 to try to attend to this one to be sure that we have the right thing. You see, this is a
 consensus algorithm, it is not meant as a way to pass information around, this is not meant like
 in a transformer as a way to do computation because we have no trainable weights in this process.
 It is simply meant as a consensus algorithm. So in imagines that by doing this, by sort of attending
 to things that are similar to you and then integrating their values, there will be these
 islands forming. And that's what you see right here. You can imagine if two vectors are already
 close at the same layer, this mechanism will make them even closer. So this is a sort of a clustering
 algorithm. And so that my question is that these drawings, you look at them, they are very
 specifically constructed, they're constructed such that a parse tree is emerging. So when you look at
 this, you have a clear sense I can probably I can probably move all of that crap out of the way.
 You can see the parse tree, right? Because the black thing is going to be the top node right here,
 let's leave away the scene level embedding for now, the black thing is going to be the top node.
 And then it has two child nodes, this one, and this one. And then it has four, every one of those
 has two child nodes. But it's not it doesn't have to be in this case. So this dynamically and every
 one of them, you know, the black ones are individual. This is dynamically constructing
 a parse tree, right? The parse tree here is something like this. And then the the the
 So this is pretty cool. But it is also drawn deliberately such that a core problem does not
 arise. And the core problem would be something like, well, what if this vector here was actually also
 pointing like this, okay, so it is not in it is not in the same. It is not in the same area of the
 parse tree, right? If you go down the parse tree, it is actually here. Now, if we do what Hinton says,
 and if for this vector here, we do this aggregation via attention on the same layer,
 what we will attend to is this vector over here. Now, this is probably not meant to be because this
 vector over here, it can represent the same thing. But you can see it's not in the in the same path
 of the parse tree. And he mentions this a little bit throughout, but not necessarily clear.
 And the drawing makes it seem like there's no problem. But I hope you can see how this is a
 problem. The attention would pull in information from over here. However, the whole parse tree
 here and the island on the top layer suggests that these two things should be parsed independently
 from each other and therefore also processed independently from each other. So here is my
 suggestion to to extend this and maybe Hinton's already thought of this. But I would suggest that
 the this attention mechanism here is modulated by how close two things are in the parse tree.
 So what would that be? So for a given a given vector, it would be how much do you attend
 to this vector right here? Well, a lot because it agrees with you, right? It you know, this the
 softmax of the inner product would be high, it agrees with you. And also it is in the same,
 it is the same branch of the parse tree. So that's perfect, right? This one right here doesn't agree
 with you, but is in the same branch. So it could potentially later agree with you through a consensus
 algorithm. However, this one over here, I, you probably shouldn't attend to that too much,
 even though it points in the same direction, because it's in a different branch of the parse
 tree, you shouldn't attend zero to it like because these branches on top, they could change. And you
 know, by you sending information there, this one could change the the top structure here that could
 agree more with your branch of the parse tree and so on. So my suggestion would be that let's not only
 get the softmax of the, let's not only get the softmax of the current layer things, but let's do
 x times and here we're going to have a sum. So this is going to be k. And let's say we're at
 we're at layer L. And this is layer one, this is layer two, this is layer three, going to number
 them from the top, actually from the bottom layer m, layer m minus one, and this is layer L.
 I suck at this. So from the current layer, I want to go up the hierarchy until layer one.
 And I'm going to take the softmax of the representation at layer L at layer k, where I'm at
 x k transposed like this. What we aggregate is still the the values on the current layer,
 but how much we should attend to that should be dependent on the parse tree. And we do that
 like this. And maybe we have like a kind of a lambda k, L minus k, L minus k. I hope you get
 what I mean. So how much how much you aggregate this sum here, the sum here is weird. This should
 go probably. Hi, it's future Yannick. And I just wanted to write that down again. So because I've
 made some mistakes, obviously, the sum here should be within the softmax because you want to
 aggregate the distributions in log space, and the softmax should still be valid, you know,
 distribution. And then the lambda is exponentiated by k and k now properly runs from the zero to all
 the way up the stacks. So big L would be the total number of layers and little l would be the layer
 where you're currently at. And you can clearly see that the contribution of these attention matrices
 it is so lambda would be something smaller than one. And therefore, the contribution is in the
 current layer is the strongest, but also in the next one up is a bit weaker than one more up is
 even a bit weaker and so on. So you'd still have essentially the same mechanism as Hinton is suggesting
 controlling for the fact that things are in different branches of the parse tree. All right,
 back to classic Yannick who is thoroughly confused by these things. Yeah, I'm not good at I'm not good
 at coming up with math on the spot. But I hope you can see what it's doing. So it is if, if you
 simply take the first k, you would simply stay at that layer and it would be what Hinton said.
 But what I'm saying is you should also consider how much your top your higher layer, one layer up
 from you agrees with one layer up from the thing you want to attend to. So you also compute that
 inner product between between the embeddings, and you add that to the softmax distribution. So
 initially, the softmax distribution would be like you should tend to this thing and this thing,
 and this thing a lot. But then the next up hierarchy would maybe say, well, we agree,
 because you know, these are in the same thing, but this one, maybe not so much. And you would add
 those together, maybe with a lambda factor in here, and then you go one layer up and it would say,
 well, okay, everything over here basically agrees, right and here, no, but everything over here
 basically doesn't agree. So you would add that maybe with a lambda squared, as you go up the
 layers, it would be less and less important, but still you'd consider it. All right. Now,
 if this is gonna work out, site the channel. Now back to what Hinton says that this is actually
 the system. This is the system as in a nutshell, you're gonna input the image at the bottom.
 And Hinton says you could use like a convent at the very bottom to get it into the columns. But
 then you're going to every time step pass information up the columns down the columns,
 and between the same layer of the different columns. And that's going to, in some point,
 this is going to stabilize, I don't know if it has cycles, it probably doesn't have cycles.
 This is good. Yeah, probably does not have cycles. So at some point, this comes to an end. And if
 that comes to an end, it should be that the object level embeddings agree on an object,
 the part level embeddings agree on what parts there are, the sub parts agree, and so on. And
 they form these islands, these islands give rise to a parse tree. And the parse tree can tell you
 what object is there, what is it made of, and where are these parts in the image, and so on. So
 exactly, that is it. And now we're going to look at what Hinton calls some design decisions. How
 many levels are there? About five. Okay, we can skip that. How fine grained are the locations?
 Hinton says you could be as fine grained as pixels, or they could correspond to larger image patches.
 You and he says you could do convolutional neural network to get it in there.
 Does the bottom up net look at nearby locations? He says, yes, the bottom up net, so this this is
 not the attention network, that's the bottom up network, it could look at nearby locations.
 But Hinton imagines that if you have bottom up, top down, and if you have attention drawing
 information, and if you maybe limit that attention to a neighborhood, then then the the attention
 will do the job because you can have instead of looking at neighboring locations in the bottom
 up network, you can simply in two time steps, aggregate that information. So you can do bottom
 up here, bottom up here, and then using the attention, the lateral mechanism, you can pass
 that information around this way. And also, it is not as biasing the network to the immediate
 neighborhood. So the attention mechanism can sort of look farther, which conflicts with what he's
 saying on top that the attention mechanism might only be looking at the neighbors. I think there
 are different possibilities here. And only looking at neighbors is actually one of the solution
 to the problem of having, you know, kind of similar vectors at very distant locations at
 down the levels. But I think it's not as as good a solutions to simply look at how close things
 are in pixel space, because even though things are close in pixel space, they might be far away
 in the parse tree space. How does the attention work? We've already looked at this. So the way
 that one location attends to another location is going to be the softmax of the inner product
 between the embeddings here. And the values are also going to be just the embeddings that layer
 at that layer. The visual input, he says convolutional net could be used. Color and texture.
 He says, he makes he gives this example, like if you know, if an object is entirely pale or
 entirely green, or entirely, I don't even know how to pronounce this, the color of a part is
 straightforward. But what color is the whole object. So this entire notion of capsules, by the way,
 imagines this as these embeddings represent kind of properties of the object so that the
 the cat ear embedding represents not only the fact that it is a cat ear, but also different
 properties about the cat ear and even its location in the image is in the embedding. And, you know,
 we know that transformers, they must be doing something like this, because we feed in positional
 embeddings, for example, at the very bottom, and it can still, you know, compute things in terms
 of positions. So that's the there's an intrinsic connection between kind of capsules and the kind
 of transformer architecture. He says, one of the motivations of Glom was idea that the whole object
 has a compound color, which might be called pale green or move. And at the object level,
 every location belonging to the object has exactly the same compound color.
 So the object is whatever this all over, when deciding which other locations the object level
 attend to preference would be given two locations with a similar compound color. So what he's saying
 right here is that, you know, you could give preference to two similar color locations,
 when you decide what you want to attend to. But the color isn't as easy as simply saying what color
 is there in the location that you are at. But you could be so if this is green, and this here is blue,
 then the bottom layer would say yes, I'm green. And yes, I'm blue. But they could also be saying,
 well, I am part of a green blue object, right. And then the the higher layer here, you know,
 attending or caring about multiple or bigger region, its color would then be, you know, green
 blue, and the consensus could reach on, well, we are a green blue object, even though the object
 isn't a pure green or pure blue all throughout. So he I think, yeah, it's it's I think it's a side
 suggestion, maybe he has this as a core motivation between the system. But it's just interesting to
 see how he thinks of things and he extends the color here to textures and even shapes.
 Shapes, the individual texture elements have their own shapes and poses in spatial relationships,
 but an object with a textured surface has exactly the same texture everywhere at the object level.
 Glom extends this idea to shapes, an object may have parts that are very different from one another,
 but at the object level, it has exactly the same compound shape in all of the location that it
 occupies. Basically saying that, okay, every pixel that's part of a cat head is a cat head has the
 shape of a cat head, even though the individual locations might not recognize that, and that
 information could be passed around through this consensus mechanism over time. So the cluster
 discovery versus cluster formation, we've seen that and he makes a lot of he makes a lot of
 analogies to face recognition. But yeah, the clusters are not the islands of similar embedding
 vectors at a level can be viewed as clusters, but these clusters are not discovered in immutable data.
 They are formed by the interaction between the intra level process that favors islands of
 similarity and dynamically changing suggestions coming from the locations embedding at adjacent
 levels. So the core here is really this consensus algorithm that creates these clusters. And yeah,
 the clustering algorithm doesn't work by simply looking at embeddings and deciding which ones go
 together, but the embeddings themselves update themselves in order to form clusters.
 And yeah, this is replicating embedding vectors. This is a response to a criticism that I guess he
 got where someone said, well, why don't why do you represent if you have these, you know, these
 columns at the bottom, it makes sense, you have all the different vectors. But then as you go up,
 you know, you have that kind of the same vector for all locations, because it's the same object.
 Why does it make sense to replicate that everywhere, and not just have one, because, you know,
 in a database, we just have one. And it basically says that in order to reach the consensus first,
 of all, it's important to have different vectors, they might be slightly different. So they might
 have some nuance in them, because, you know, they might get pulled into different directions
 from the sign of bottom up signal, then from the consensus algorithm on the same layer. So I, you
 know, I believe that it is that is important. Here, I think it's just this is a criticism he got.
 And then he decided to put this in here, learning islands. So what we haven't discussed about this
 yet is how this is trained and Hinton says this is trained as a denoising auto encoder. Let us
 assume that Glom is trained to reconstruct at its output, the uncorrupted version of an image from
 which some region has been have been removed. So he goes into self supervised learning with the system.
 This objective should ensure that information about the input is preserved during the forward
 pass. And if the regions are sufficiently large, it should also ensure that identifying familiar
 objects will be helpful for filling in the missing regions. To encourage islands of near identity,
 we need to add a regularizer. And experience shows that a regularizer that simply encourages
 similarity between the embeddings of nearby locations can cause representations to collapse.
 All the embedding vectors may become very small, so that they are all very similar. And the
 reconstruction will then use very large weights to deal with the very small scale to prevent collapse.
 And then he says contrastive learning is the answer to this. So how do you regularize the model
 such that this consensus is formed? He says contrastive learning might be useful, but you
 can't simply apply it straight out. So it learns to make representations of two different crops of
 the same image agree, and the representations of two crops from different images disagree.
 But this is not a sensible thing to do if our aim is to recognize objects. If crop one contains
 objects A and B and crop two from the same image contains objects B and C, it does not make sense
 to demand that the representation of the two crops is the same at the object level. Okay, so he says
 that contrastive learning is good, but you have to pay very careful attention at which layer you
 employ it. Because if you go down far enough, then contrastive learning, especially this type
 where you crop the image into different parts, and you say, well, since it's the same image,
 the representations should agree. Hinton would say, well, at the top layer, yes, but at the bottom
 layer, certainly not, because they display different things. So you have to be careful
 where you apply this contrastive learning. And he gives a bunch of suggestions on how to solve that.
 He says things like, well, negative examples, for example, might not might not even be needed.
 Well, that's it. Sorry, that's a different thing. So the obvious solution is to regularize
 the bottom up and top down neural networks by encouraging each of them to predict the
 consensus option. Yeah, this is the weighted geometric mean of the predictions coming from
 the top down and bottom up networks, the attention weighted average of the embeddings at nearby
 locations at the previous time step, the previous state of and I guess, and there should be an end
 and the previous state of the embedding training, the inter level prediction to agree with the
 consensus will clearly make the islands found during feed forward inference be more coherent.
 So he says you could regularize the model to to regress to the consensus option. So
 it's sort of like a self a self regression. And he asks whether or not that will lead to a collapse,
 because if you don't have negative examples and contrastive learning, this could lead to simply a
 collapse. An important question is whether this type of training will necessarily cause collapse
 if it is not accompanied by training the inter level predictions to be different for negative
 examples that use the consensus options for unrelated spatial contexts. So here is that
 problem. Right. If you use the consensus opinion for unrelated spatial context,
 that might be a problem. He says using layer batch norm should reduce the tendency to collapse,
 but a more important consideration may be the achievability of the goal. It goes into why
 regularization could help. And he says if however, an embedding at one location is free to choose
 which embeddings at other locations it should resemble, the goal can be achieved almost
 perfectly by learning to form islands of identical vectors and attending almost entirely to other
 locations that are in the same island. And I don't know, I don't know if this is what I suggested.
 So I guess this is kind of a convoluted paragraph, and I had to also read it multiple times and I
 still don't exactly know what he's trying to say right here. But I think what he's saying is that
 what we want to do is we want to sort of regularize the network to produce this consensus,
 right. So we have a bottom up signal, a top down signal, we have a current value,
 and we have the signal from the attention mechanism. Now, what we want to do is we want to
 reach a consensus such that these islands form. However, if you attend to any sort of things here
 that have nothing to do with you, you might not be able to reach this consensus, right. That's,
 I think that's the problem I think he's touching on the problem that I said before.
 So what he says is, you know, what you should do is you should simply attend to things that are
 in the same islands already. So if an embedding at one location is free to choose which embedding
 at other locations it should resemble, the goal can be achieved by learning to form islands of
 identical vectors and attending almost entirely to other locations that are in the same islands.
 Now, I think here, what he's doing, he makes the case for the attention mechanism itself,
 right. So he says, if, if we simply draw in information from the same layer here,
 you know, anything, any old information might come in, and we might collapse and or we might
 never reach consensus because any old information might come in. However, if we simply draw in
 information from the selected neighbors that already are in the same group in the same island
 as me, then this consensus algorithm works. So if the network, the network is now forced kind of
 to learn to build these islands of similar things in order to make this consensus work, if we
 regularize this consensus, then we can actually create a consensus that is similar to the one
 that we have in the same group. So I think that's the way to make this consensus work if we
 regularize this consensus. So I believe he makes the case for the attention mechanism. I don't think
 he, in this case, considers kind of the up the next up layer islands, what I would say is you need
 to go to the columns in order to decide which things, which locations, right, it's free to
 choose which embeddings at other locations it should resemble. I think, yeah, this is the case
 for the attention mechanism. Okay, I hope you're still half with me. If not, I'm, I'm a bit confused
 because I think what he's doing is he says, contrastive learning would be good, you can use it,
 but you have to be careful at which layer you do it. Another regularizer to form these islands
 would be this regularize the network to conform to the consensus option, opinion. However, if you
 simply aggregate information from the same layer, then that wouldn't work because, you know, the
 different things in the same layer might correspond to completely different parts of the image.
 Drawing in information from there would not help you. How do you solve this? By introducing the
 very attention mechanism that he introduced in order to only draw in information from parts of
 the same layer that actually are related to you. Okay, the next thing, the next consideration he
 does is representing coordinate transformations. So how does this represent coordinate transformations?
 There was a capsule net paper where he explicitly represents coordinate transformations in kind of
 four dimension quaternion space. And he says that is probably not needed because you don't want to,
 he says you could represent this by four by four matrices. However, if you simply allocate 16
 numbers in each embedding vector, in order to represent the part whole coordinate transformation,
 like the transformation that relates the part to the whole, that does not make it easy to represent
 uncertainty about the aspects of pose and certainty about others. So the problem here is that we know
 that humans, when they watch something right here, when they watch a scene, like this is a chair,
 and there is a person, a very tiny person on the chair, we don't see necessarily the coordinate
 frame of the world. What we see is we see the coordinate frame of the chair, like maybe this is
 the center, and we see the person in relation to the chair, our brain seems to do this intuitively,
 and hinting things that a system like this should also do it intuitively. So somehow,
 the coordinate transformations involved going from the eye to the reference through the frame
 of the chair, and then from the chair to the person, they should be somehow in encoded in this
 network. However, he also says that it's probably not necessary to encode them explicitly as you
 know, explicit coordinate transformations, because not only does that make it harder,
 probably to learn, but also, you can't represent uncertainty. In fact, you can represent uncertainty,
 that's the next thing right here, much better by having a higher dimensional thing that you're
 trying to guess, right? If you are trying to guess a distribution with three components,
 and you simply have a three dimensional vector, you have no way of representing uncertainty.
 However, if you have a nine dimensional vector, you can have three opinions about the distribution.
 So this is an opinion, this is an opinion, and then this is an opinion. And then you can sort
 of aggregate and you can say, well, I'm pretty sure about these two things, because all my opinions
 are pretty close. But this one here, I'm not so sure because my individual things say different
 things, things say things. All right, I've this video is too long. So that's his argument right
 here, we don't need explicit representing of uncertainty, because by simply over parameterizing,
 we can already represent uncertainty well. And we also don't need disentangled position information
 and, and so on. Sorry, we don't need different position information. Because, again, the network
 can take care of that. And he gives a good example, like why would you have disentangled
 coordinate frame if you have an image. And in the image, the picture in it is this.
 How do you know if that is a rhomboid shape? Or if it is a
 rec, if it is a rectangular piece of paper viewed from the side, I should probably draw it way closer,
 something like something like this. I suck at this. You get probably get what I mean. Like,
 if it is a different object, it has a like the object and the coordinate transformation are
 dependent upon each other. And so it makes sense for the neural network to actually entangle the two,
 because the two things depend on each other. In essence, he's just saying, don't worry about
 explicitly representing all of the different things. We got it, like the neural network can do
 all of these things, like uncertainty or position, and pose transformations. So here he compares it
 to different other architectures. Comparison to CNN, comparison to transformers, comparison to
 capsule models. And at the end, it goes into video. At the very beginning, he says the paper is about
 actually a video system. And you can kind of see that because we go through this algorithm in
 multiple time steps, right? You have, it's like you analyze an image with these columns, which gives
 you sort of a 3D, 3D tensor with the image at the bottom. And you go in the next time step, you have
 a new 3D tensor, right? You pass this whole information around with the image at the bottom.
 Hinton says, well, why does that need to be the same image? That could also be different images.
 So you could use the system to analyze video. So what he does is he says, at the same time,
 you do this time step to find agreement, you could actually swap out the video frame, the X,
 you can swap out the video frame, and produce a slightly different video frame. And you could
 actually have a kind of an ensemble regularizing effect. So as the whole columns here, the whole
 system comes to a consensus over time, you feed in different information at the bottom. And what
 he says is that, you know, if this is a slow enough video, then the top layers here would probably
 could still reach an agreement, while the bottom layers would change rapidly. But that could be
 sort of an ensemble or a regularizer regularizing effect that it even has. So he intrinsically
 connects these two time dimensions, because they would be separate, right, you could input a video,
 and then in, you know, in each frame, you could do this consensus finding algorithm. But he says,
 No, it's actually cool to consider them together to do the consensus finding while you sort of
 watch the video. It's just not clear that you always need the same amount of consensus finding
 steps as you need as you have video frames. So maybe you want to, maybe you want to take like
 five consensus steps per video frame or the other way around. Not sure. In any case, I think that's
 a pretty cool idea. And he says things like, if the changes are rapid, there is no time available
 to iteratively settle on a good set of embedding vectors for interpreting a specific frame.
 This means that the GLOM architecture cannot correctly interpret complicated shapes. If the
 images are changing rapidly, try taking an irregularly shaped potato and throwing it up
 in the air such a way that it rotates at one or two cycles per second. Even if you smoothly track
 the potato, you cannot see what shape it is. Now I don't have a potato, but I can give you an avocado.
 So if you give me a second, how's that? Could you track the shape? I don't know.
 Probably Hinton's correct. All right. He talks about is this biologically plausible? And I don't
 want to go too much into this. He discusses some restrictions like, yeah, we still use backprop
 and is backprop plausible and so on. I love this sentence. In the long run, however, we are all
 dead. And then the footnote saying there are alternative facts. But yeah, he discusses whether
 it's biologically plausible. How could you modify it to make it more plausible? For example,
 when you want to do contrastive learning, there is evidence that dreams during so during sleep,
 you do contrastive learning, like you produce the negative examples during sleep, and then during
 the day, you collect the positive examples, and so on. So I think this is a more speculative part
 of the paper, but it's pretty cool to it's pretty cool to read it. And lastly, he goes into discussion
 he also says that this paper is too long already. I'm going to just briefly talk about this. And he
 trashes the neuro symbolic people a bit like he trashes the people that say no, no, you know,
 neural networks can never do whatever. And he says pretty clearly look, neural networks can represent
 trees, I've given you a system also BERT can output parse trees. So shut up, I guess. And he comes
 up with this glom BERT name, which, you know, is is already coined if you wanted to do glom BERT,
 that's already taken. Sorry. I also by the way also coined then I coined the name may go mania.
 Right now. Okay, if you want to if you want to use it, it better be a pretty cool machine learning
 system and be based on glom. Right, that was the paper. I think it's a cool system. It has a bunch
 of parts that are maybe not super friendly to hardware at the time like this iterative procedure.
 But honestly, it is not much more than a neural network, sorry, a recurrent neural network with
 very complicated recurrence functions. The video extension might be a bit tricky. And, but the rest
 and the regularization might be a bit tricky, the exact objective. So the denoising auto encoder
 objective isn't super detailed in the paper, he simply says, reconstruct the corrupted version of
 the input. How exactly the input happens, maybe there's a CNN, maybe the CNN feeds information
 into actually multiple layers. None of that is exactly specified. So there's lots to figure out.
 I do think the ideas are very cool. And I love idea papers. And therefore, I recommend that if
 you're interested more, give this thing a read, give this video a like, share it out,
 and I'll see you next time. Bye bye.