Gauge Equivariant Convolutional Networks and the Icosahedral CNN
machine learning deep learning artificial intelligence ai data science convolution convolutional neural networks cnn manifolds curvature parallel transport gauge gauge transformation icosahedron weight sharing coordinate frame invariant coordinate system equivariance sphere spherical
Ever wanted to do a convolution on a Klein Bottle? This paper defines CNNs over manifolds such that they are independent of which coordinate frame you choose. Amazingly, this then results in an efficient practical method to achieve state-of-the-art in several tasks!

https://arxiv.org/abs/1902.04615

Abstract:
The principle of equivariance to symmetry transformations enables a theoretically grounded approach to neural network architecture design. Equivariant networks have shown excellent performance and data efficiency on vision and medical imaging problems that exhibit symmetries. Here we show how this principle can be extended beyond global symmetries to local gauge transformations. This enables the development of a very general class of convolutional neural networks on manifolds that depend only on the intrinsic geometry, and which includes many popular methods from equivariant and geometric deep learning. We implement gauge equivariant CNNs for signals defined on the surface of the icosahedron, which provides a reasonable approximation of the sphere. By choosing to work with this very regular manifold, we are able to implement the gauge equivariant convolution using a single conv2d call, making it a highly scalable and practical alternative to Spherical CNNs. Using this method, we demonstrate substantial improvements over previous methods on the task of segmenting omnidirectional images and global climate patterns.

Authors: Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, Max Welling
 What you're looking at here are manifolds. Specifically you're looking at 2D manifolds embedded in a 3D space. So naturally these are some kind of bodies that have a surface and one of the things you might want to do with a manifold like this is to define a convolutional neural network to work on this surface. So usually we have convolutional neural network working on flat surfaces such as images. But what if you could actually work on a manifold like this? An easy example is a sphere. You might want to work on a sphere. Why is that useful? Maybe you want to predict the climate and then you actually want to work on the Earth's surface which is approximated by a sphere. So today we'll look at the following paper. Gauge-equivariant convolutional networks and the icosahedral CNN by Tachokohen, Maurice Weiler, Burkai Kichang, and Max Welling. So as I already said this paper tries to define convolutional neural networks on any kind of manifold. So what's the problem inherently when you're doing this? Can't you just, you know, place a filter move it around like you do in a regular CNN? That's exactly the problem actually. So if you have a picture, and let me draw a picture of a cat, right? Cat here, here, here, here, eye, eye. Alright, cat smiling. This is a terrible cat. What you do is you have your filter, right, and that's a little patch in the image. You're just going to move this filter, move it around, move it around, and at each point you convolve the filter. If this is larger, you convolve each of the elements of the filter. Here maybe you have nine elements. So each of these elements here is convolved with the underlying image. At the end you aggregate all of them into a single point, usually by adding them up. And there you, from this image, you produce a new image that is a different thing. So if this kernel here, for example, is a specific kernel that detects lines, you might end up with, or that detects specifically up-down lines, you might end up with just the lines that go up and down in this. So the eyes here, here, right. So this might be the result of this convolution. Of course in CNN these convolutional kernels then are learned as parameters. So it seems pretty easy, right? You just simply take a kernel and kind of shift it around. At each point you convolve the underlying image and that's it. Well it's not so easy if you work on a manifold. And why is that? It's illustrated here on a sphere. So if you have a sphere and you place a kernel, it really matters which direction you place the kernel in. Of course I mean it does on an image, but bear with me. So here you place a kernel in the direction of this arrow, right? You place the kernel maybe like this here, you place your little kernel on it, and you say up. Basically up is here, right? And then you move that kernel around and ultimately you want to move it all the way to the other side of the sphere. So back here you want to move it over there, you want to move it all around the sphere, right? Now what happens if you move it this way, right? You convolve here, you move it this way, you convolve here. You see already by the red arrows where is up. Up is where the red arrows point, right? If you move it along here the red arrows will always point up up up up up. Okay so you arrive back here with your kernel. I'm gonna try to draw this dashed with the up in the kernel being this direction, because you've moved it around like so. But if you for some reason choose to move your kernel in another direction, namely in this direction up here, then as you can see if you place it here and then you place it here, you place it here, you place it back here and ultimately here. Where is up? If you just keep track of where up is in your kernel it's always going to be to the front of the sphere. So on one hand you have up being to the back here and on the other hand you have one up being to the front here. So this doesn't match. So it actually depends on which path you take from this original point to any other point. It depends which path you take, how your kernel is gonna end up there. And that's of course very unfortunate because we're not used to this on this on this 2d thing. Because if I you know move it down first and then up here, over here sorry, where is up in my... so if up is here, if it's down here, up is here and over here up is here. And if I kind of move it straight over here and then down and then here and then here, you see up is always the same direction. There is no problem in a flat surface. That's why we can simply define it as we do. But in a sphere or any kind of manifold it's called parallel transport is path dependent in technical terms. The way you transport a thing from one place to another really depends on the path you take. So this paper is trying to address this problem and define a convolution on any manifold. So how is this done? First of all to define a convolution on the curved surface what they do is they say okay we have a convolutional filter and the convolutional filter is actually some sort of a flat object and it works in what's called the tangent space of the manifold. The tangent space is a flat space that you can define at any point on the manifold. So here the manifold is the sphere. At some point P you define the tangent space as simply the tangent kind of a sheet, a straight sheet touching the surface at point P. So this is now a flat space where we can define a let's say a regular convolutional kernel as we did laying it up here. The question is how do you map points from the sphere to this tangent space and back and that's happening via this exponential map. The exponential map in this sense is not the same as the exponential map that you are used to by simply you know exponentiating things. The exponential map here basically means if I want to go from a point in the tangent space to a point on the manifold what I do is I take this vector here which is a straight vector in the tangent space and I go on the manifold in this direction for a predefined length. So this is usually a length of one on the manifold. For a predefined length I walk into this direction along the geodesic. It's along the shortest path into this direction and then I stop and where I end up that's where I basically end up. So that's the corresponding point to this point here on the tangent space. So to define a convolution fully it means that first you lay your kernel and then for each element in the kernel you will multiply that kernel entry, let me use a blue here, multiply that kernel entry by the corresponding point on the manifold itself. So by mapping this point in the tangent space to the manifold. You can also say you basically back project from the manifold to the tangent space and there you do your regular convolution. So that's how you define a convolution in the classic sense if you have for example a sphere and what the authors here of course noticed already is that this is dependent on how you get there and in technical terms it's called this is dependent on your gauge. So the gauge basically is defining this coordinate frame in the tangent space. So this tangent vector here is an abstract object, it's just a vector, but in order to do something with it, in order to do something with a kernel and convolution and so on, you have to express it in numbers and numbers are expressed with respect to a base usually. If you have a vector v here you can express it with respect to this two basis vectors. So maybe v is here is 2 and here is 3. So v can be represented as the vector 2, 3 with respect to the base e1, e2. And so this choice of base basically is what's called a gauge. Now I'm probably butchering this topic completely for any physicists or mathematicians listening but just kind of give you an impression. So this choice of bases is called a gauge and we can imagine a different choice of bases. So let me draw another basis here. So another basis might be 1, 2. So e1 is here, e2 is here. So the new coordinates here would be something like v can also be expressed in this new basis as say 1, here's maybe 1 and this is very far so this is maybe 5. So 5 in this direction. And to transform between the two there is formulas basically from from you know them from linear algebra from vector spaces. In general they're called gauge transformations and if we want our convolution to be invariant to the basically chosen coordinate frames we have to say in technical terms what we mean is the convolution should be gauge-equivariant. That means no matter which base we choose. If we choose this base or if we choose this the result should basically be the same. So within the computation of the convolution we must account for the fact of which gauge is chosen and then basically have the result be invariant. And with the result we don't mean the numbers of the result because these will change but we mean the the actual object that is resulting, the geometric object that is resulting should be equivalent under gauge transformations. So this is a it sounds very technical but the way I understand it is basically you want to define a convolution on these manifolds such that you it's such that the result is not dependent on exactly how you shift the kernel around as long as you account for the fact that you shifted it around this way should give you the same the same result. So for this they define a condition and the condition is that the kernel must behave as such. So the V is the input here and G minus 1 is a a transformation of the of the gauge as I understand it. And so basically if you transform the input by a different coordinate frame then at the kernel applied to that different input must behave exactly as the kernel applied to the original input and then perturbed by these two operations. So this is this you might notice this you might know things like this from discussions maybe of what it means for a function to be linear or something where the function applied to a transformed version must correspond to the function applied to the original version of the input transformed so the result transformed by some some operation. So if this holds so this is a condition on the kernel of the convolution and if you so if you define your convolution in this way this is a modification to the convolution on the tangent space that we had then your result will be gauge equivalent. What is this transformation and what is this new convolution they define they say if you do the convolution this way then these things will hold. So what is this this way basically again you convolve the kernel with the input but you the f here is the input k is the kernel but what you do if we come up here again what you do you have to do a slight modification your kernel here if you want to convolve it let's say this point here you would not combine this point with the point along the exponential map corresponding to it right this point here but what you would do is you would transport this point back along the geodesic to here and then you would and then you would compute your regular convolution. So this means sorry this is what this term here means technically. If you don't understand it don't worry I don't either I guess this is simply saying that if you perform convolutions in on manifolds in this way and you have the appropriate kernel then they will be gauge equivalent. So this is pretty cool because what they do next is they define the convolution on an icosahedron and an icosahedron is a shape a 3d geometric shape that's made of like triangles and I can try to maybe they have drawn it yes so all right this is an icosahedron and so they can now define a convolution on this with where a filter is basically the filter looks like this it's this kind of hexagon I yes and the and the filter is kind of shifted around and of course it's the problem is whenever it shifts over one of these boundaries here or whenever it shifts over the these corners here what do you do what do you do then because if you look at it you can't basically flatten the corner if you try to flatten the corner you're gonna have this wedge sticking out that's terrible you're gonna have a wedge here sticking out if you try to flatten the corner so you have to define basically the convolution on this they do it in their framework and specifically what they do is they flatten and pad the icosahedron to this representation so they put it into five pieces they have to pad a bit you see here each colored edge here this colored edge corresponds to this colored edge so that would be padded from here to nicely define this convolution and then they put this into a regular 2d image with the color things they are sometimes repeated in this image and then they define the filters in this following way so this these are the filters for basically for a six channel input image and what they have to do is they have to do a weight sharing between the filters in a very specific way and in order for the kernel to have these properties they need to see replicate these filters down here and if you look the different colors in these different let's call them channels they each have different intensities and if you look down here they're all slightly different which means they're all slightly different linear combinations of the of the filter up here or rotations basically they're all differently arranged but they're basically this blue field here is this blue field but is also let's see this one and this one and this one and this one so the the weights here are these original filters are basically arranged such that the weights are shared in this form down here but if you do this if you arrange them like this when you replicate each filter basically six times because you also want six output channels then the filter will have the desired properties and your convolution will be gauge equivalent so they apply this to to ICO M this so the complete algorithm is actually down here they can actually use if they pad the image in the correct way to the 2d image and expand the kernel to arrange it as we just saw they can use a regular 2d convolution to compute their result and that's pretty cool and this means this also is very very very efficient on this Ico Sahedron so what they do is they apply this to Ico M NIST where they project basically they project M NIST on an Ico Sahedron so they take the image M NIST and they project it onto this and then they try to classify it on that I can actually show that their method outperforms other method and learns these invariances so learns the the symmetries of the Ico Sahedron or basic sorry is invariant to them being invariant to the symmetries means you don't have to learn them anymore if you're not invariant to symmetries it means you have to learn each one of them separately right but if you're invariant to symmetries then you have only have to learn one thing once and then if the Ico Sahedron is rotated you're just like ma that's just the same thing as this other thing they also do this interestingly to climate pattern segmentation and also a kind of 2d or 3d omni-directional segmentation where you're in a room a 3d room and you have an omni-directional picture sorry from everywhere you have a picture a 3d sphere picture from everywhere you're asked to segment things in the room and actually outperform all other methods on these data sets so I find this extremely cool that kind of this ultra theoretical work starting out as ultra theoretical then gets implemented into something that beats state-of-the-art methods on relevant tasks alright so that was just a brief overview and a very dirty look at these things but I hope you got something out of it and thus far that was it for me bye bye
 What you're looking at here are manifolds. Specifically you're looking at
 2D manifolds embedded in a 3D space. So naturally these are some kind of bodies
 that have a surface and one of the things you might want to do with a
 manifold like this is to define a convolutional neural network to work on
 this surface. So usually we have convolutional neural network working on
 flat surfaces such as images. But what if you could actually work on a manifold
 like this? An easy example is a sphere. You might want to work on a sphere. Why is
 that useful? Maybe you want to predict the climate and then you actually want
 to work on the Earth's surface which is approximated by a sphere. So today we'll
 look at the following paper. Gauge-equivariant convolutional networks
 and the icosahedral CNN by Tachokohen, Maurice Weiler, Burkai Kichang,
 and Max Welling. So as I already said this paper tries to define
 convolutional neural networks on any kind of manifold. So what's the problem
 inherently when you're doing this? Can't you just, you know, place a filter
 move it around like you do in a regular CNN? That's exactly the problem actually.
 So if you have a picture, and let me draw a picture of a cat, right? Cat here, here,
 here, here, eye, eye. Alright, cat smiling. This is a terrible cat. What you do is you
 have your filter, right, and that's a little patch in the image. You're just
 going to move this filter, move it around, move it around, and at each point you
 convolve the filter. If this is larger, you convolve each of the elements of the
 filter. Here maybe you have nine elements. So each of these elements here is
 convolved with the underlying image. At the end you aggregate all of them into a
 single point, usually by adding them up. And there you, from this image, you
 produce a new image that is a different thing. So if this kernel here, for example,
 is a specific kernel that detects lines, you might end up with, or that detects
 specifically up-down lines, you might end up with just the lines that go up and
 down in this. So the eyes here, here, right. So this might be the result of this
 convolution. Of course in CNN these convolutional kernels then are learned
 as parameters. So it seems pretty easy, right? You just simply take a kernel and
 kind of shift it around. At each point you convolve the underlying image
 and that's it. Well it's not so easy if you work on a manifold. And why is that?
 It's illustrated here on a sphere. So if you have a sphere and you place a kernel,
 it really matters which direction you place the kernel in. Of course I mean it
 does on an image, but bear with me. So here you place a kernel in the direction
 of this arrow, right? You place the kernel maybe like this here, you place your little
 kernel on it, and you say up. Basically up is here, right? And then you move that
 kernel around and ultimately you want to move it all the way to the other side of
 the sphere. So back here you want to move it over there, you want to move it all
 around the sphere, right? Now what happens if you move it this way, right? You
 convolve here, you move it this way, you convolve here. You see already by the red
 arrows where is up. Up is where the red arrows point, right? If you move it along
 here the red arrows will always point up up up up up. Okay so you arrive back here
 with your kernel. I'm gonna try to draw this dashed with the up in the
 kernel being this direction, because you've moved it around like so. But if
 you for some reason choose to move your kernel in another direction, namely in
 this direction up here, then as you can see if you place it here and then you
 place it here, you place it here, you place it back here and ultimately here.
 Where is up? If you just keep track of where up is in your kernel it's always
 going to be to the front of the sphere. So on one hand you have up being to the
 back here and on the other hand you have one up being to the front here. So this
 doesn't match. So it actually depends on which path you take from this original
 point to any other point. It depends which path you take, how your kernel is
 gonna end up there. And that's of course very unfortunate because we're not used
 to this on this on this 2d thing. Because if I you know move it down first and then
 up here, over here sorry, where is up in my... so if up is here, if it's down here, up
 is here and over here up is here. And if I kind of move it straight over here and
 then down and then here and then here, you see up is always the same direction.
 There is no problem in a flat surface. That's why we can simply define it
 as we do. But in a sphere or any kind of manifold it's called parallel
 transport is path dependent in technical terms. The way you transport a thing from
 one place to another really depends on the path you take. So this paper is
 trying to address this problem and define a convolution on any manifold. So
 how is this done? First of all to define a convolution on the curved surface what
 they do is they say okay we have a convolutional filter and the
 convolutional filter is actually some sort of a flat object and it works in
 what's called the tangent space of the manifold. The tangent space is a flat
 space that you can define at any point on the manifold. So here the manifold is
 the sphere. At some point P you define the tangent space as simply the tangent
 kind of a sheet, a straight sheet touching the surface at point P. So this
 is now a flat space where we can define a let's say a regular convolutional
 kernel as we did laying it up here. The question is how do you map
 points from the sphere to this tangent space and back and that's happening via
 this exponential map. The exponential map in this sense is not the same as the
 exponential map that you are used to by simply you know exponentiating things.
 The exponential map here basically means if I want to go from a point in
 the tangent space to a point on the manifold what I do is I take this vector
 here which is a straight vector in the tangent space and I go on the manifold in
 this direction for a predefined length. So this is usually a length of one on
 the manifold. For a predefined length I walk into this direction along the
 geodesic. It's along the shortest path into this direction and then I stop and
 where I end up that's where I basically end up. So that's the corresponding point
 to this point here on the tangent space. So to define a convolution fully it means
 that first you lay your kernel and then for each element in the kernel you will
 multiply that kernel entry, let me use a blue here, multiply that kernel entry by
 the corresponding point on the manifold itself. So by mapping this
 point in the tangent space to the manifold. You can also say you basically
 back project from the manifold to the tangent space and there you do your
 regular convolution. So that's how you define a convolution in the classic sense
 if you have for example a sphere and what the authors here of course noticed
 already is that this is dependent on how you get there and in technical terms
 it's called this is dependent on your gauge. So the gauge basically is defining
 this coordinate frame in the tangent space. So this tangent vector here is an
 abstract object, it's just a vector, but in order to do something with it, in
 order to do something with a kernel and convolution and so on, you have to
 express it in numbers and numbers are expressed with respect to a base
 usually. If you have a vector v here you can express it with respect to this two
 basis vectors. So maybe v is here is 2 and here is 3. So v can be represented
 as the vector 2, 3 with respect to the base e1, e2. And so this choice of base
 basically is what's called a gauge. Now I'm probably butchering this topic
 completely for any physicists or mathematicians listening but just kind
 of give you an impression. So this choice of bases is called a gauge and we
 can imagine a different choice of bases. So let me draw another basis here. So
 another basis might be 1, 2. So e1 is here, e2 is here. So the new
 coordinates here would be something like v can also be expressed in this new
 basis as say 1, here's maybe 1 and this is very far so this is maybe 5. So 5 in
 this direction. And to transform between the two there is formulas basically from
 from you know them from linear algebra from vector spaces. In general they're
 called gauge transformations and if we want our convolution to be invariant to
 the basically chosen coordinate frames we have to say in technical terms what
 we mean is the convolution should be gauge-equivariant. That means no matter
 which base we choose. If we choose this base or if we choose this the result
 should basically be the same. So within the computation of the convolution we
 must account for the fact of which gauge is chosen and then basically have the
 result be invariant. And with the result we don't mean the numbers of the result
 because these will change but we mean the the actual object that is resulting,
 the geometric object that is resulting should be equivalent under gauge
 transformations. So this is a it sounds very technical but the way I understand
 it is basically you want to define a convolution on these manifolds such that
 you it's such that the result is not dependent on exactly how you shift the
 kernel around as long as you account for the fact that you shifted it around this
 way should give you the same the same result. So for this they define a
 condition and the condition is that the kernel must behave as such. So the V is
 the input here and G minus 1 is a a transformation of the of the gauge as I
 understand it. And so basically if you transform the input by a different
 coordinate frame then at the kernel applied to that different input must
 behave exactly as the kernel applied to the original input and then perturbed by
 these two operations. So this is this you might notice this you might know things
 like this from discussions maybe of what it means for a function to be linear or
 something where the function applied to a transformed version must correspond
 to the function applied to the original version of the input transformed so the
 result transformed by some some operation. So if this holds so this is a
 condition on the kernel of the convolution and if you so if you define
 your convolution in this way this is a modification to the convolution on the
 tangent space that we had then your result will be gauge
 equivalent. What is this transformation and what is this new
 convolution they define they say if you do the convolution this way then these
 things will hold. So what is this this way basically again you convolve the
 kernel with the input but you the f here is the input k is the kernel but what
 you do if we come up here again what you do you have to do a slight
 modification your kernel here if you want to convolve it let's say this point
 here you would not combine this point with the point along the exponential map
 corresponding to it right this point here but what you would do is you would
 transport this point back along the geodesic to here and then you would and
 then you would compute your regular convolution. So this means sorry this is
 what this term here means technically. If you don't understand it don't worry I
 don't either I guess this is simply saying that if you perform convolutions
 in on manifolds in this way and you have the appropriate kernel then they will be
 gauge equivalent. So this is pretty cool because what they do next is they define
 the convolution on an icosahedron and an icosahedron is a shape a 3d geometric
 shape that's made of like triangles and I can try to maybe they have drawn it
 yes so all right this is an icosahedron and so they can now define a
 convolution on this with where a filter is basically the filter looks like this
 it's this kind of hexagon I yes and the and the filter is kind of shifted around
 and of course it's the problem is whenever it shifts over one of these
 boundaries here or whenever it shifts over the these corners here what do you
 do what do you do then because if you look at it you can't basically flatten
 the corner if you try to flatten the corner you're gonna have this wedge
 sticking out that's terrible you're gonna have a wedge here sticking out if
 you try to flatten the corner so you have to define basically the convolution
 on this they do it in their framework and specifically what they do is they
 flatten and pad the icosahedron to this representation so they put it into five
 pieces they have to pad a bit you see here each colored edge here this colored
 edge corresponds to this colored edge so that would be padded from here to nicely
 define this convolution and then they put this into a regular 2d image with
 the color things they are sometimes repeated in this image and then they
 define the filters in this following way so this these are the filters for
 basically for a six channel input image and what they have to do is they have to
 do a weight sharing between the filters in a very specific way and in order for
 the kernel to have these properties they need to see replicate these filters down
 here and if you look the different colors in these different let's call
 them channels they each have different intensities and if you look down here
 they're all slightly different which means they're all slightly different
 linear combinations of the of the filter up here or rotations basically
 they're all differently arranged but they're basically this blue field here
 is this blue field but is also let's see this one and this one and this one and
 this one so the the weights here are these original filters are basically
 arranged such that the weights are shared in this form down here but if you
 do this if you arrange them like this when you replicate each filter basically
 six times because you also want six output channels then the filter will have
 the desired properties and your convolution will be gauge equivalent so
 they apply this to to ICO M this so the complete algorithm is actually down here
 they can actually use if they pad the image in the correct way to the 2d image
 and expand the kernel to arrange it as we just saw they can use a regular 2d
 convolution to compute their result and that's pretty cool and this means this
 also is very very very efficient on this Ico Sahedron so what they do is they
 apply this to Ico M NIST where they project basically they project M NIST on
 an Ico Sahedron so they take the image M NIST and they project it onto this and
 then they try to classify it on that I can actually show that their method
 outperforms other method and learns these invariances so learns the the
 symmetries of the Ico Sahedron or basic sorry is invariant to them being
 invariant to the symmetries means you don't have to learn them anymore if
 you're not invariant to symmetries it means you have to learn each one of them
 separately right but if you're invariant to symmetries then you have only have to
 learn one thing once and then if the Ico Sahedron is rotated you're just like
 ma that's just the same thing as this other thing they also do this
 interestingly to climate pattern segmentation and also a kind of 2d or 3d
 omni-directional segmentation where you're in a room a 3d room and you have
 an omni-directional picture sorry from everywhere you have a picture a 3d
 sphere picture from everywhere you're asked to segment things in the room and
 actually outperform all other methods on these data sets so I find this extremely
 cool that kind of this ultra theoretical work starting out as ultra theoretical
 then gets implemented into something that beats state-of-the-art methods on
 relevant tasks alright so that was just a brief overview and a very dirty look
 at these things but I hope you got something out of it and thus far that was
 it for me bye bye