Involution: Inverting the Inherence of Convolution for Visual Recognition (Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper what is deep learning deep learning tutorial introduction to deep learning computer vision convolutional neural network convolutions alternative cnn attention self attention attention mechanism for vision weight sharing neural networks convolutions vision cnn vision involution vision image segmentation rednet resnet residual neural networks bytedance ai
#involution #computervision #attention

Convolutional Neural Networks (CNNs) have dominated computer vision for almost a decade by applying two fundamental principles: Spatial agnosticism and channel-specific computations. Involution aims to invert these principles and presents a spatial-specific computation, which is also channel-agnostic. The resulting Involution Operator and RedNet architecture are a compromise between classic Convolutions and the newer Local Self-Attention architectures and perform favorably in terms of computation accuracy tradeoff when compared to either.

OUTLINE:
0:00 - Intro & Overview
3:00 - Principles of Convolution
10:50 - Towards spatial-specific computations
17:00 - The Involution Operator
20:00 - Comparison to Self-Attention
25:15 - Experimental Results
30:30 - Comments & Conclusion

Paper: https://arxiv.org/abs/2103.06255
Code: https://github.com/d-li14/involution

Abstract:
Convolution has been the core ingredient of modern neural networks, triggering the surge of deep learning in vision. In this work, we rethink the inherent principles of standard convolution for vision tasks, specifically spatial-agnostic and channel-specific. Instead, we present a novel atomic operation for deep neural networks by inverting the aforementioned design principles of convolution, coined as involution. We additionally demystify the recent popular self-attention operator and subsume it into our involution family as an over-complicated instantiation. The proposed involution operator could be leveraged as fundamental bricks to build the new generation of neural networks for visual recognition, powering different deep learning models on several prevalent benchmarks, including ImageNet classification, COCO detection and segmentation, together with Cityscapes segmentation. Our involution-based models improve the performance of convolutional baselines using ResNet-50 by up to 1.6% top-1 accuracy, 2.5% and 2.4% bounding box AP, and 4.7% mean IoU absolutely while compressing the computational cost to 66%, 65%, 72%, and 57% on the above benchmarks, respectively. Code and pre-trained models for all the tasks are available at this https URL.

Authors: Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, Qifeng Chen

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there! Today we're looking at involution, inverting the inheritance of convolution for visual recognition by a number of researchers of the Hong Kong University of Science and Technology, ByteDance AI lab and Peking University. In this paper on a high level the researchers try to replace the good old convolution operator in CNNs by this new thing called an involution. In its essence, involution is about halfway between a convolution and a self attention kind of operation. And it turns out that with some clever weight-sharing scheme you can achieve very good performance compared to CNNs and self-attention networks, while keeping the number of parameters and the computational cost relatively low. This I think is very much worth trying for anyone who does not operate on extremely large-scale problems. We'll get into that a bit more when we go into the experiments, but for now let's go through the paper, through what involution is, what it does, how it's different. If you like this, don't hesitate to share it out, it would help a lot. We're on the road to a hundred K subscribers and with every subscriber I get a subscriber. I stole that joke. They say here in the abstract, convolution has been the core ingredient of modern neural networks triggering the surge of deep learning in vision. AlexNet, ResNet, etc. Convolution, even though transformers are slowly taking over computer vision, convolutions are still very very much used and if you're not on a super large scale problem a convolutional neural network is still very probably the best way to go if you have a computer vision problem. They say we rethink the inherent principles of standard convolution for vision tasks, specifically spatial agnostic and channel specific. Instead we present a novel atomic operation for deep neural networks by inverting the aforementioned design principles of convolution, coined an involution. They say we additionally demystify the recent popular self-attention operator and subsume it into our involution family as an over complicated instantiation. A lot of statements in this paper are true, especially further down. A lot of the experiments are really cool, but it is a bit of an over statement what they say right here. Their claim is that if you have a convolution, what you do is something that's spatial agnostic and channel specific, which means that in a convolutional neural network when you have an image, let's say, with a bunch of pixels, these are now true pixels, not patches, and you run a convolutional layer over it, you run a convolutional kernel over it, you put the center of the kernel at some pixel, so the kernel will be something like a 3x3 kernel, you put that on the center here, so it overlaps here, you multiply element-wise, and then you aggregate. You can do that in multiple channels, but essentially you do that. Then after you've done that, you move the kernel one, let's say to the right, you shift it, so the center is here, you do the same thing again, and you shift it, you do the same thing again. It's spatial agnostic because it repeats the same computation over and over and over across the image, and it doesn't care where the computation is. It does the same computation, and that is the selling point of convolutional neural networks. They are translation invariant. It's a form of weight sharing, you share the weights across the locations, and therefore you don't really care where stuff is in the image. The CNN will be able to recognize it just as well, and you don't need to learn over and over and over the same principle just because it's in different parts of the image. This is spatial agnostic. What does channel specific mean? For that we have to go into the multiple channels realm. If your image has multiple channels, let's say I'm going to draw a new image right here with a bunch of pixels, and it has multiple channels, that means you can imagine it sort of as a 3D tensor here, where each pixel is a column, and every column is a vector of a certain dimensionality. The original image has of course three channels, which is red, green, and blue, but if you have intermediate representations these channels can grow to sizes of hundreds of channels. The point of the channels is that every entry here is a number, and every number can capture one aspect of what's described in that particular pixel. Maybe the first channel is a corner, the second one is an edge, the third one is a blue pixel, the fourth one is probably a cat here, and so on. These are the different features in the channels. A convolution operator is channel specific, that means if you have the kernel... Now convolutional kernels aren't as easy as I drew them, they're in fact four dimensional tensors. They are four dimensional tensors, which makes it a little bit complicated for me to draw, honestly. However, you can imagine that you have one kernel like so, that has the same amount of channels as your image. Now you can still do the same operation. You can overlay your kernel on a part of the image, overlay it like so, and then you can do element-wise multiplication, and then you do a sum, you sum it all up. After you do this operation, you do a big sum over all the elements of whatever your kernel multiplied with your image, and that gives you one number. You do an all-reduce, one number gives you one number. So you do this, so this is one kernel, but you have another one right here. You do the same thing, and that gives you also one number. You have another kernel, I think you get the idea, you have another kernel here. You have many of those kernels per layer. If you've never looked at how the weights look when you instantiate these layers in a deep learning framework, I encourage you to do so. A convolutional layer will have weights that are of the size kernel size by kernel size, by input channels, by output channels. It's a 4D tensor, and this orange part here is just one of those sub tensors. In fact you have as many as you have output channels. That gives you, of course when you then go over all of these, that gives you the next layer. So that becomes in the next layer. This is the next layer representation, at the point where you overlaid the kernel in the last layer, that will become this column right here. So you have the orange thing in the first, the blue thing in the second channel, green thing in the third channel, and so on. I hope this is relatively clear. So you have in fact one convolutional kernel per output channel. So if you call the orange thing here a convolutional kernel, then you have one kernel per output channel. That means it's channel specific. This is a conscious choice and it makes sense when you think about it, because each output channel means something different. If my output channel means is there a cat at this particular location, then I might want to aggregate the last layer's representation differently than if my output channel says, well is this part of the sky, or is there a corner here, or something like this. So I want to aggregate the weights differently. That's why I have to have a different set of weights here, here, and here, because they mean different things. So it's spatial agnostic, because it does the same computation at every location. It's channel specific, because it does a different computation at each channel, even though it does it for all the locations equally. Now we're prepared to invert that. So convolution promises we invert this. What we want to do is something spatial specific and channel agnostic. So the first thing here is the channel agnostic. If you've seen my last video about MLP mixer, this is very much the same idea. The idea is just of, hey why do we have different things here? Why do I have different computations? Can't we just apply the same principle we apply to the spatial thing, where we say we just slide the same computation over the image, and that is generally fine. That's weight sharing, it's actually good. Why don't we just do this here? Why don't we aggregate the information in the same way for all the different channels? So you can do that. You can just have one kernel. So instead of having a number of output channels, many kernel. So the involution will come up with simply one kernel that it shares across all of the channels. They have a little picture down here. Just look at the last step right here. Wow sorry, I crossed that out. Here this is the kernel that they have. Sorry, it's not even by number of channels. It's actually you just flatten this thing. So it's a k by k by 1 kernel and you simply push that, put that over a location in the image and then you share the computation across. So the image here, given that this is all in the same colors, it means that you just multiply, you broadcast. That's the word I was looking for. You broadcast the operation across the channels and then you aggregate after that. So you can see what involution does is broadcast and then not reduce. You don't reduce at the end to a single number, but you keep the channels as they are. That's why you only need a k by k by 1, because you don't have the different computation for each output channel and you don't reduce across the input channels. So you get away with a lot less parameters. That's even wrong here. Just a k by k kernel. Now that's one part. The other part is why don't we do something that's spatial specific. Now remember what spatial agnostic was. Spatial agnostic was we slide the same kernel across the image. What they're saying in first instance, they're saying things like, or they said something, don't know where it was in the picture, but they say what we could do is, if we have an image, and we do something spatial specific, what that means is we could have a kernel that's just as big as the image. Then no more sliding across it. It's simply you multiply those things together, you broadcast it across these channels of the image, and there you go. Also something that that MLP mixer does, they just say whatever, we don't do slidey slidey anymore. They do weight sharing, but essentially you're trying to get rid of this sliding over. You have different weight for each location. That means that the computation actually differs from where stuff is in the image. We know that that is somewhat important, because usually the sky is up and objects in these natural images that humans take might be more in the middle than anywhere else. Text goes from left to right. It's not all super translation and location invariant. It makes sense to have weights that are different for each position. But then they run into a problem. They say we couldn't do that very well, because now we can't just input pictures of different resolutions. That's one problem. I think the other problem is that this might not work too well. They come up with a different thing. They say can't we make a compromise? They don't call it a compromise. They call it something different. But they say look, can we come up with a scheme where we can retain a kernel that's approximately this size, like a small kernel, but it is different for each location. We still do the classic convolution way of doing things, in that we do these local aggregations across neighboring pixels. However the kernel that we use here is different from the kernel that we use here. That's different from the kernel that we use here. How could you make a computation where the kernel is always different? You do that by coming up with the kernel in a dynamic way. The authors here say, let's say we're at this pixel right here. We care about this neighborhood. How can we come up on the fly with a kernel for this particular pixel? Their answer is, let's just generate it from the pixel. This is the full involution diagram. We've now arrived at this. They are at this neighborhood, which is outlined here in this black scaffolding grid thing. The center pixel is the red pixel here. They say we look at that pixel and all its channels. We use that pixel and only that pixel. Not the neighborhood. We use that pixel to come up with the kernel. They have a computation here, which of course is going to be a small neural network. This is a two-layer neural network that comes up with the kernel. You see this is simply a reshape. You compute the kernel across the neighborhood from the pixel itself. That means that every single pixel here, unless it's the exact same pixel, so the exact same color in the first layer, or the exact same representation in the intermediate layers, every single location gets its own kernel for the convolution. The computation I've already told you is a small neural network. Specifically it's a bottleneck neural network. It takes the pixel representation as a vector, bottlenecks it. There is a non-linearity here and then it expands it again to the size of the actual kernel. Then you use that kernel and you broadcast it instead of having one kernel per input channel. Then you multiply and then you don't reduce across the input channels. That alleviates you from having to have multiple kernels, one for each output channel. This is the whole convolution pipeline. I would say there are multiple different concepts here. This coming up with the kernel on the fly is one concept. Then this broadcasting scheme is an entirely different concept. You could do both independently of each other. They do them together. They do ablations further down, but it's two new things in one. The first thing here is that you might think of a tension mechanism as you look at that. It's a form of fast weights. The weights of the computation are computed on the fly from the data itself. That is exactly what an attention mechanism does. However, here you do it in a slightly different way. They say that they have a discussion about attention right here. They say there are a bunch of differences. In attention what you'd have is you don't only compute your weights from the actual location where you are, even in local self-attention. You actually compute your weights from more than just the pixel where you are. You compute it from the entire region you care about. That's the first thing. The second thing is that in self-attention you have the queries and the keys. You have your data, your neighborhood, let's say. Each of those things produces a query and a key. Everyone produces a query and a key. Then you do this sort of quadratic thing in order to determine how you should aggregate your information. In involution you simply don't produce keys. You only produce queries, if you will, or only keys, however you want to look at it. Then you don't do the quadratic thing. Rather you immediately interpret this as the weights of aggregation. You can write this, and they say that, you can interpret this as the positional encodings already being present in these weights, because it's now specific to a position. Whereas in the attention literature you'd have to supply positional encodings. In order for the algorithm to know that this is a different thing, that this here is a different thing from this thing here, you need to supply it with positional encodings. Not here, because the individual channels of this thing immediately refer to different positions. This neural network is very aware of what position is where relative to the pixel you're considering. They say the success of involution explains in part why other people had lots of success with leaving away the keys and only using positional encodings together with the query. If I'm not mistaken, I think you could frame the lambda networks into this category, where at some point they never do this attention. However they rely heavily on positional encodings. However you can learn those ahead of time or statically. This is the connection to attention. The connection to attention is that the weights are constructed on the fly. However here there's no quadratic interaction, there is no softmax and so on. You construct the weights from the pixel in the center. To frame attention as a more complicated instantiation of our idea, that's a bit out there. The authors here say that attention is just a more complicated thing. The second thing I worry a bit about is that they say that this is position specific. They started out with saying that convolution is spatial agnostic. We want to do something spatial specific. This here is also spatial agnostic. If you get the same pixel at different locations in the image, this thing will produce the same weights and the computation will be the same. In fact you do this entire computation right here. That is a spatially agnostic computation. The difference here is the same difference that you have between slow weights and fast weights. You simply construct the weights of the actual computation on the fly. However the way you construct these weights remains position agnostic. The second thing is that the weight sharing is a bit of an independent thing. I get that the two work well together, but the broadcasting and weight sharing thing across the channels is almost a much simpler mention. It's a bit related to the fact that if you have a depth separated convolution and you simply share the weights across that, that's about what it boils down to. What does that give us? In fact it gives us a lot. In this paper they do experiments and they compare against for example ResNets and other networks with similar number of parameters. I like these experiments here in that you can see they always make sure that they have the lowest number of parameters among the things they compare with. Yet they show that they still beat these models. They compare ResNet with the same number of layers. This is standalone ResNet. Here is the axial ResNet. You can see that this outperforms on these tabs. This is ImageNet. They also have different things such as this segmentation task. I think they have a picture down here. This segmentation task where they perform better. This is the baseline and you can see the involution network. I think the effect that you see right here. The fact that they are better in this number is really cool. It's probably a bit due to the fact that they do this on the fly computation of weights. Which is a more powerful idea than the static weights of a convolution. The lower number of parameters I think is more a result of their weight sharing. They tout here how that they are on par with ResNet 101 regarding the top one recognition accuracy. While saving 65% of storage and computation. I think that the saving of computation is more due to the weight sharing mechanism. I think they've just selected tasks and they might be important tasks. It was just the case that in these tasks whether or not you share the weights probably doesn't matter. It doesn't hit you as hard or is even beneficial if you don't have enough data. Therefore that's why they have less parameters. What you can also observe here is that differences. They get continuously smaller as you move up the scale of network. This is all on the same data set but it would be interesting to see how this performs on a really large scale. My intuition is that as you go larger and larger in scale. This approach is going to top out and lose out to the more general architectures like attention. It's a clown world now. In these regimes and I would argue these are the regimes where a lot of practitioners care about. These and actually smaller regimes. This seems to perform reasonably well. You can see right here the curves here when you compare compute to accuracy is very favorable. Especially if you're in this region here. If you're in the low resource region it might be something that you want to try out. It remains to be seen how well this is pre-trainable and fine-tunable. It's something you might want to try. If you try to only use parts of it it would be interesting to see. If we still do convolution but we do this weight sharing scheme. They also have a notion of grouping in the channels. As the attention mechanism has it. Sharing a single kernel across all channels obviously underperforms in accuracy. Considering channel redundancy of evolution kernels. As long as the channels shared in a group to an acceptable range. The channel agnostic behavior will not only preserve the performance. But also reduce the parameter count and computational cost. This will also permit the larger kernel size under the same budget. It's the same reasoning as people introducing groups or different heads in multi-head attention. Try all of this stuff out. I think it's worth it. The code is available right here. I'll also put a link to that. That was it from me for this paper. I wish you a very pleasant day of the week. Bye bye.
 Hello there! Today we're looking at involution, inverting the inheritance of
 convolution for visual recognition by a number of researchers of the Hong Kong
 University of Science and Technology, ByteDance AI lab and Peking University.
 In this paper on a high level the researchers try to replace the good old
 convolution operator in CNNs by this new thing called an involution. In its
 essence, involution is about halfway between a convolution and a self
 attention kind of operation. And it turns out that with some clever
 weight-sharing scheme you can achieve very good performance compared to CNNs
 and self-attention networks, while keeping the number of parameters and the
 computational cost relatively low. This I think is very much worth trying for
 anyone who does not operate on extremely large-scale problems.
 We'll get into that a bit more when we go into the experiments, but for now
 let's go through the paper, through what involution is, what it does, how it's
 different. If you like this, don't hesitate to share it out, it
 would help a lot. We're on the road to a hundred K subscribers and with every
 subscriber I get a subscriber. I stole that joke. They say here in the
 abstract, convolution has been the core ingredient of modern neural networks
 triggering the surge of deep learning in vision.
 AlexNet, ResNet, etc. Convolution, even though transformers are slowly taking
 over computer vision, convolutions are still very very much used and if you're
 not on a super large scale problem a convolutional neural network is still
 very probably the best way to go if you have a computer vision problem. They say
 we rethink the inherent principles of standard convolution for vision tasks,
 specifically spatial agnostic and channel specific. Instead we present a
 novel atomic operation for deep neural networks by inverting the aforementioned
 design principles of convolution, coined an involution. They say we
 additionally demystify the recent popular self-attention operator and
 subsume it into our involution family as an over complicated instantiation.
 A lot of statements in this paper are true, especially further
 down. A lot of the experiments are really cool, but it is a bit of an over
 statement what they say right here. Their claim is that if you have a
 convolution, what you do is something that's spatial agnostic and
 channel specific, which means that in a convolutional neural network when
 you have an image, let's say, with a bunch of pixels, these are now true pixels, not
 patches, and you run a convolutional layer over it, you run a convolutional
 kernel over it, you put the center of the kernel at some pixel, so the kernel
 will be something like a 3x3 kernel, you put that on the center here, so it
 overlaps here, you multiply element-wise, and then you aggregate. You can do
 that in multiple channels, but essentially you do that. Then after
 you've done that, you move the kernel one, let's say to the right, you
 shift it, so the center is here, you do the same thing again, and you shift it,
 you do the same thing again. It's spatial agnostic because it repeats the
 same computation over and over and over across the image, and it doesn't care
 where the computation is. It does the same computation, and that is the
 selling point of convolutional neural networks. They are translation
 invariant. It's a form of weight sharing, you share the weights
 across the locations, and therefore you don't really care where stuff is in the
 image. The CNN will be able to recognize it just as well, and you don't
 need to learn over and over and over the same principle just because it's in
 different parts of the image. This is spatial agnostic. What does channel
 specific mean? For that we have to go into the multiple channels realm.
 If your image has multiple channels, let's say I'm going to draw a new image
 right here with a bunch of pixels, and it has multiple channels, that means you can
 imagine it sort of as a 3D tensor here, where each pixel is a column, and every
 column is a vector of a certain dimensionality. The original image
 has of course three channels, which is red, green, and blue, but if you have
 intermediate representations these channels can grow to sizes of hundreds
 of channels. The point of the channels is that every entry here is a number, and
 every number can capture one aspect of what's described in that
 particular pixel. Maybe the first channel is a corner, the
 second one is an edge, the third one is a blue
 pixel, the fourth one is probably a cat here, and so on. These are the
 different features in the channels. A convolution operator is channel
 specific, that means if you have the kernel... Now convolutional kernels aren't
 as easy as I drew them, they're in fact four dimensional tensors.
 They are four dimensional tensors, which makes it a little bit
 complicated for me to draw, honestly. However, you can imagine that you
 have one kernel like so, that has the same amount of channels as your image.
 Now you can still do the same operation. You can overlay your
 kernel on a part of the image, overlay it like so, and then
 you can do element-wise multiplication, and then you do a sum, you sum it all up.
 After you do this operation, you do a big sum over all the elements of
 whatever your kernel multiplied with your image, and that gives you one
 number. You do an all-reduce, one number gives you one number. So you do
 this, so this is one kernel, but you have another one right here.
 You do the same thing, and that gives you also one number.
 You have another kernel, I think you get the idea, you have another kernel
 here. You have many of those kernels per layer. If you've
 never looked at how the weights look when you instantiate these
 layers in a deep learning framework, I encourage you to do so. A
 convolutional layer will have weights that are of the size kernel size by
 kernel size, by input channels, by output channels. It's a 4D tensor, and this
 orange part here is just one of those sub tensors. In fact you have as many as
 you have output channels. That gives you, of course when you then go over
 all of these, that gives you the next layer. So that becomes in the next layer.
 This is the next layer representation, at the point where you
 overlaid the kernel in the last layer, that will become this column right here.
 So you have the orange thing in the first, the blue thing in the second
 channel, green thing in the third channel, and so on. I hope this is relatively clear.
 So you have in fact one convolutional kernel per output channel. So if you
 call the orange thing here a convolutional kernel, then you have one
 kernel per output channel. That means it's channel specific. This is a
 conscious choice and it makes sense when you think about it, because each
 output channel means something different. If my output channel
 means is there a cat at this particular location, then I might want to
 aggregate the last layer's representation differently than if my
 output channel says, well is this part of the sky, or is there a corner here,
 or something like this. So I want to aggregate the weights differently.
 That's why I have to have a different set of weights here, here, and here, because
 they mean different things. So it's spatial agnostic, because it does the same
 computation at every location. It's channel specific, because it does a
 different computation at each channel, even though it does it for all the
 locations equally. Now we're prepared to invert that. So convolution
 promises we invert this. What we want to do is something spatial specific and
 channel agnostic. So the first thing here is the channel agnostic.
 If you've seen my last video about MLP mixer, this is very much the same idea.
 The idea is just of, hey why do we have different things here? Why do I have
 different computations? Can't we just apply the same principle we apply
 to the spatial thing, where we say we just slide the same computation
 over the image, and that is generally fine. That's weight sharing, it's actually
 good. Why don't we just do this here? Why don't we aggregate the information in
 the same way for all the different channels? So you can do that.
 You can just have one kernel. So instead of having a number of output channels,
 many kernel. So the involution will come up with simply one kernel that
 it shares across all of the channels. They have a little
 picture down here. Just look at the last step right here.
 Wow sorry, I crossed that out. Here this is the kernel that they have.
 Sorry, it's not even by number of channels. It's actually you
 just flatten this thing. So it's a k by k by 1 kernel and you simply
 push that, put that over a location in the image and then you share the
 computation across. So the image here, given that this is all in the same
 colors, it means that you just multiply, you broadcast. That's the word I was
 looking for. You broadcast the operation across the channels and then you
 aggregate after that. So you can see what involution does is broadcast and then
 not reduce. You don't reduce at the end to a single number, but you keep
 the channels as they are. That's why you only need a k by k by 1,
 because you don't have the different computation for each output channel and
 you don't reduce across the input channels. So you get away with a lot
 less parameters. That's even wrong here. Just a k by k kernel. Now that's
 one part. The other part is why don't we do something that's spatial
 specific. Now remember what spatial agnostic was.
 Spatial agnostic was we slide the same kernel across the image. What they're
 saying in first instance, they're saying things like, or they said something,
 don't know where it was in the picture, but they say what we could do is,
 if we have an image, and we do something spatial specific,
 what that means is we could have a kernel that's just as big as the image.
 Then no more sliding across it. It's simply you multiply those
 things together, you broadcast it across these channels of the image,
 and there you go. Also something that that MLP mixer does,
 they just say whatever, we don't do slidey slidey anymore.
 They do weight sharing, but essentially you're trying to get rid of this sliding
 over. You have different weight for each location. That means that the
 computation actually differs from where stuff is in the image. We know that
 that is somewhat important, because usually the sky is up and objects
 in these natural images that humans take might be more in the middle than
 anywhere else. Text goes from left to right. It's not all super
 translation and location invariant. It makes sense to have weights that are
 different for each position. But then they run into a problem. They say we
 couldn't do that very well, because now we can't just input pictures of
 different resolutions. That's one problem. I think the other problem is
 that this might not work too well. They come up with a different thing.
 They say can't we make a compromise? They don't call it a compromise.
 They call it something different. But they say look, can we come up with a
 scheme where we can retain a kernel that's approximately this size, like a
 small kernel, but it is different for each location. We still do the
 classic convolution way of doing things, in that we do these local aggregations
 across neighboring pixels. However the kernel that we use here is different
 from the kernel that we use here. That's different from the kernel that we
 use here. How could you make a computation where the kernel is always
 different? You do that by coming up with the kernel in a dynamic way.
 The authors here say, let's say we're at this pixel right here. We care
 about this neighborhood. How can we come up on the fly with a kernel for this
 particular pixel? Their answer is, let's just generate it from the pixel.
 This is the full involution diagram. We've now arrived at this. They are at
 this neighborhood, which is outlined here in this black scaffolding grid
 thing. The center pixel is the red pixel here. They say we look at that
 pixel and all its channels. We use that pixel and only that pixel. Not the
 neighborhood. We use that pixel to come up with the kernel. They have a
 computation here, which of course is going to be a small neural network.
 This is a two-layer neural network that comes up with the kernel. You see this
 is simply a reshape. You compute the kernel
 across the neighborhood from the pixel itself. That means that every
 single pixel here, unless it's the exact same pixel, so the exact same color in
 the first layer, or the exact same representation in the intermediate
 layers, every single location gets its own kernel for the convolution. The
 computation I've already told you is a small neural network. Specifically it's
 a bottleneck neural network. It takes the pixel representation as a
 vector, bottlenecks it. There is a non-linearity here and then it expands
 it again to the size of the actual kernel. Then you use that kernel and
 you broadcast it instead of having one kernel per input channel. Then
 you multiply and then you don't reduce across the input channels.
 That alleviates you from having to have
 multiple kernels, one for each output channel. This is the whole
 convolution pipeline. I would say there are multiple different
 concepts here. This coming up with the kernel on the fly is one concept.
 Then this broadcasting scheme is an entirely different concept. You could do
 both independently of each other. They do them together. They do
 ablations further down, but it's two new things in one. The first
 thing here is that you might think of a tension mechanism as
 you look at that. It's a form of fast weights. The weights of the
 computation are computed on the fly from the data itself. That is exactly
 what an attention mechanism does. However, here you do it in a slightly
 different way. They say that they have a discussion about
 attention right here. They say there are a bunch of differences.
 In attention what you'd have is you don't only compute your
 weights from the actual location where you are, even in local self-attention.
 You actually compute your weights from more than just the pixel where you are.
 You compute it from the entire region you care about. That's the first thing.
 The second thing is that in self-attention you have the
 queries and the keys. You have your data, your neighborhood, let's say.
 Each of those things produces a query and a key.
 Everyone produces a query and a key. Then you do this sort of
 quadratic thing in order to determine how you should aggregate your
 information. In involution you simply don't produce keys. You
 only produce queries, if you will, or only keys, however you want to look at it.
 Then you don't do the quadratic thing. Rather you immediately interpret
 this as the weights of aggregation. You can write this, and they say that,
 you can interpret this as the positional encodings already
 being present in these weights, because it's now specific to a position.
 Whereas in the attention literature you'd have to supply positional encodings.
 In order for the algorithm to know that this is a different thing,
 that this here is a different thing from this thing here, you need to
 supply it with positional encodings. Not here, because the individual
 channels of this thing immediately refer to different positions.
 This neural network is very aware of what position is where relative
 to the pixel you're considering. They say the success of involution
 explains in part why other people had lots of success with leaving away the
 keys and only using positional encodings together with the query.
 If I'm not mistaken, I think you could frame the lambda networks
 into this category, where at some point they never do this attention.
 However they rely heavily on positional encodings.
 However you can learn those ahead of time or statically.
 This is the connection to attention.
 The connection to attention is that the weights are constructed on the fly.
 However here there's no quadratic interaction, there is no softmax and so on.
 You construct the weights from the pixel in the center.
 To frame attention as a more complicated instantiation of our idea,
 that's a bit out there. The authors here say that attention is just a more complicated thing.
 The second thing I worry a bit about is that they say that this is position specific.
 They started out with saying that convolution is spatial agnostic.
 We want to do something spatial specific.
 This here is also spatial agnostic. If you get the same pixel at different locations in the image,
 this thing will produce the same weights and the computation will be the same.
 In fact you do this entire computation right here.
 That is a spatially agnostic computation.
 The difference here is the same difference that you have between slow weights and fast weights.
 You simply construct the weights of the actual computation on the fly.
 However the way you construct these weights remains position agnostic.
 The second thing is that the weight sharing is a bit of an independent thing.
 I get that the two work well together, but the broadcasting and weight sharing thing across the channels
 is almost a much simpler mention.
 It's a bit related to the fact that if you have a depth separated convolution
 and you simply share the weights across that, that's about what it boils down to.
 What does that give us? In fact it gives us a lot.
 In this paper they do experiments and they compare against for example
 ResNets and other networks with similar number of parameters.
 I like these experiments here in that you can see they always make sure that they have the lowest number of parameters
 among the things they compare with.
 Yet they show that they still beat these models.
 They compare ResNet with the same number of layers.
 This is standalone ResNet.
 Here is the axial ResNet.
 You can see that this outperforms on these tabs.
 This is ImageNet.
 They also have different things such as this segmentation task.
 I think they have a picture down here.
 This segmentation task where they perform better.
 This is the baseline and you can see the involution network.
 I think the effect that you see right here.
 The fact that they are better in this number is really cool.
 It's probably a bit due to the fact that they do this on the fly computation of weights.
 Which is a more powerful idea than the static weights of a convolution.
 The lower number of parameters I think is more a result of their weight sharing.
 They tout here how that they are on par with ResNet 101
 regarding the top one recognition accuracy.
 While saving 65% of storage and computation.
 I think that the saving of computation is more due to the weight sharing mechanism.
 I think they've just selected tasks and they might be important tasks.
 It was just the case that in these tasks whether or not you share the weights probably doesn't matter.
 It doesn't hit you as hard or is even beneficial if you don't have enough data.
 Therefore that's why they have less parameters.
 What you can also observe here is that differences.
 They get continuously smaller as you move up the scale of network.
 This is all on the same data set but it would be interesting to see how this performs on a really large scale.
 My intuition is that as you go larger and larger in scale.
 This approach is going to top out and lose out to the more general architectures like attention.
 It's a clown world now.
 In these regimes and I would argue these are the regimes where a lot of practitioners care about.
 These and actually smaller regimes.
 This seems to perform reasonably well.
 You can see right here the curves here when you compare compute to accuracy is very favorable.
 Especially if you're in this region here.
 If you're in the low resource region it might be something that you want to try out.
 It remains to be seen how well this is pre-trainable and fine-tunable.
 It's something you might want to try.
 If you try to only use parts of it it would be interesting to see.
 If we still do convolution but we do this weight sharing scheme.
 They also have a notion of grouping in the channels.
 As the attention mechanism has it.
 Sharing a single kernel across all channels obviously underperforms in accuracy.
 Considering channel redundancy of evolution kernels.
 As long as the channels shared in a group to an acceptable range.
 The channel agnostic behavior will not only preserve the performance.
 But also reduce the parameter count and computational cost.
 This will also permit the larger kernel size under the same budget.
 It's the same reasoning as people introducing groups or different heads in multi-head attention.
 Try all of this stuff out. I think it's worth it.
 The code is available right here.
 I'll also put a link to that.
 That was it from me for this paper.
 I wish you a very pleasant day of the week.
 Bye bye.