Gradient Origin Networks (Paper Explained w/ Live Coding)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper gon gradient negative gradient implicit implicit representation siren sirens deep neural networks convolutional neural network dnns mnist cifar10 fashion mnist gradient descent sgd inner loop backpropagation live code code machine learning code research research paper
Neural networks for implicit representations, such as SIRENs, have been very successful at modeling natural signals. However, in the classical approach, each data point requires its own neural network to be fit. This paper extends implicit representations to an entire dataset by introducing latent vectors of data points to SIRENs. Interestingly, the paper shows that such latent vectors can be obtained without the need for an explicit encoder, by simply looking at the negative gradient of the zero-vector through the representation function.

OUTLINE:
0:00 - Intro & Overview
2:10 - Implicit Generative Models
5:30 - Implicitly Represent a Dataset
11:00 - Gradient Origin Networks
23:55 - Relation to Gradient Descent
28:05 - Messing with their Code
37:40 - Implicit Encoders
38:50 - Using GONs as classifiers
40:55 - Experiments & Conclusion

Paper: https://arxiv.org/abs/2007.02798
Code: https://github.com/cwkx/GON
Project Page: https://cwkx.github.io/data/GON/

My Video on SIREN: https://youtu.be/Q5g3p9Zwjrk

Abstract:
This paper proposes a new type of implicit generative model that is able to quickly learn a latent representation without an explicit encoder. This is achieved with an implicit neural network that takes as inputs points in the coordinate space alongside a latent vector initialised with zeros. The gradients of the data fitting loss with respect to this zero vector are jointly optimised to act as latent points that capture the data manifold. The results show similar characteristics to autoencoders, but with fewer parameters and the advantages of implicit representation networks.

Authors: Sam Bond-Taylor, Chris G. Willcocks

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
 Hi there, today we'll look at gradient origin networks by Sam Bond Taylor and Chris G Wilcox of Durham University. So on a high level, this paper trains implicit representation networks, but not on single data points, but on entire data set. It does so by using a latent encoding of each data point. And it doesn't obtain that encoding through an explicit encoder, but by simply looking at the gradient of the latent variable when initialized at the origin. So it's a bit of a weird formulation, and I've seen this paper upvoted on Reddit, and the top comments would always say like, I don't really get it, I don't really get it. And I thought, you know, maybe I'm completely wrong, but I can just give my opinion kind of what's going on in this paper. Now also, most people on Reddit or a lot did say, I don't really get it, but here is what I think is going on. And then listing something and that's there is where I stopped reading. So as to not be kind of as to form my own opinion, I like to kind of understand papers by myself. So again, maybe I'm completely wrong. But here is my opinion. If you like opinions, hit the like button and subscribe if you aren't yet. And yeah, share this video out maybe that helps someone else understand. So this paper is a very short paper, it is four pages. And it's a dense paper, it definitely can warrant it definitely can warrant making a longer paper out of it. That being said, it's an archive paper for now. So you know, there's nothing wrong with archiving kind of unfinished work. But we're just going to look at it and try to understand it. Okay. So the abstract says this paper proposes a new type of implicit generative model that is able to quickly learn a latent representation without an explicit encoder. So for that, you need to know what an implicit generative model is. And I've covered one type of implicit generative models, specifically the type that they're using here, what they're called siren. So sirens are implicit representation networks. And I've made a video about sirens. So if you don't know what that is, go look it up. But very quickly, a siren will is a neural network to represent a single data point. So each data point in a data set is represented by its own neural network. And the neural network, so this might be a bit foreign to you. But usually you have some kind of image, right? And it's simply represented as an as an array of RGB coordinates, right? It's it's simply an array of this is like one zero point five and so on. So all the pixels are in this array. This is the explicit representation of that data point. Now, this here is a long list, and it has some regularities to it. So that's why you can also think of an implicit representation of the data point. The implicit representation works as follows. You imagine again your image, your image is made up of pixels, and these pixels are on X and Y coordinates. So this pixel right here would be zero zero. This pixel right here would be zero one, and so on. A siren is or generally an implicit representation network is a network that takes in any X and Y coordinate as the input. So the input itself is the numerical X and Y coordinate of that picture, and it passes it through a neural network and outcomes the RGB value. OK, so an entire picture is represented by this neural network. The neural network maps each coordinate to its RGB value. And here you can see that a single picture can become an entire data set for this neural network. In fact, it has to, because for a different picture, of course, there is a different mapping from X and Y coordinates to RGB coordinates. But this allows you to do multiple things. So first of all, this neural network can be smaller than the explicit representation. Second of all, it can capture some regularity in the data. Usually sirens have sine waves as nonlinearities in the neural network here, which is also a bit special, but lends itself very well to capture natural signals, because natural signals are often repeated at different scales and derivatives of themselves and so on. So I've covered this all in my video. And also this allows you to have a continuous representation rather than a discrete representation. Like here, you just have each pixel. Now you have a continuous representation. All right, so these are implicit representation models or implicit generative models, or these neural networks right here that map from coordinates to two colors. Now what's the problem with this is, as we said, you need one neural network per data point. Now, the idea that these people here go with is that can't we do kind of the same thing, but except we have one neural network per data point, we want to have the same neural network for the entire data set. So again, they want to have a neural network that somehow outputs RGB coordinates. But now it's not for a single image. Now we have a data set. Okay, and the data set has many images like this is image i, this is image j, this is image k. So what we could do is we could simply tell the neural network the x and y coordinate that we where we would like the RGB values to know. And we could also tell it which image it is right, k, or i or j. And this will give us a neural network right here that can represent the entire data set because it always can see, ah, I want of image j, I want these and these x y coordinate doesn't help you very much though, because it still has to learn for each image individually, how to encode it, how to produce it. What's much more interesting is, if you kind of mix this with the kind of old style, the kind of old style generative models. So in old style generative models, let's consider for example, an auto encoder. So in an auto encoder, what you would do is you would take your image and you would put it through an encoder. And this encoder will give you a latent variable z. And then you would put it through a decoder again. And that would give you an image. So your generative model now is this part right here, and this z variable is your latent encoding of this data point. Now, if you train these models correctly, be this a be this a an auto encoder or a variational auto encoder, or the green part can actually just be a GAN, right? If you train this correctly, then this z right here will be sort of a a latent encoding of the what the what of the information in the image itself. Okay. And that can generalize. So now I can input a picture that the model has never seen during training. And the encoder will map it to a latent representation that sort of makes sense that is able to reconstruct the image that I've put in. Okay, so the your hope with these latent representation is is that there is some kind of data manifold somewhere in hidden in the in the entire space of parameters. And as long as you're on that data manifold, you will produce a sensible data point. And this is kind of a continuous and so on. So even though you've only seen a few during training, if you have a new one during testing, then you can sort of it will be mapped to a correct place on the data manifold and it will produce a data point again. And you've seen this right, you've seen these interpolations in GANs where you can interpolate in latent space, and, and so on. The problem here is that, you know, in so in GANs, we sample these things right here. So that's a different story. But in VAEs, we need this encoder or in auto encoders, we need this encoder to obtain a latent representation for a given data point. In GANs, there is no way if we have an image, there is no way to obtain the corresponding Z variable. If we don't have an encoder, right, and that's the problem we're tackling right here. So here, what we want to do is we want to give the X and Y, we want to give the Z, we say we have some way of obtaining a latent representation of one of the image right here. And from that, we want to generate the RGB variables. Now the question is, think of again, the question is, how do we obtain the Z variable without having without having access to the encoder? And that's that's the problem of this paper. And this paper proposes a solution. So they say this is achieved with an implicit neural network that takes as inputs points in the coordinate space, alongside a latent vector initialized with zero. So that's the model that we saw. That's this, this is sorry about that. This is this right here, it takes in the coordinates, this is the coordinates, and it takes in the latent vector Z. Now, this whole point with it being initialized at zeros will get will get to that in one second. Okay. For the fact right now is just that the represent the implicit neural network also takes the identity of the image. So each image, the image is always going to have the same Z. And then we sort of say which x and y coordinate of that image we want. So the Z is per image. And then each image has all the x and y coordinates of, you know, itself. So yeah. So if yeah, you you I think you can follow. They go on they say the gradients of the data fitting loss with respect to this zero vector are jointly optimized to act as latent points that capture the data manifold. So this is where this is where I already got lost reading the first time through the results show similar characteristics to auto encoders, but with fewer parameters and the advantages of implicit representation networks. Okay, so we'll actually we'll, we'll jump to this right here. So this is the this is the comparison between a variational auto encoder and the gradient origin network. So in a variational auto encoder, what you would do is you would have this explicit encoder right here, as we said, and in the variational auto encoder, you don't obtain the latent representation directly, you actually obtain the distribution in terms of the mean and standard deviation of the latent representation. And then you sample from that distribution to obtain that latent representation. I think the point here is simply to show that you first of all, you do need an encoder, which you do need to train. And second of all, it's kind of a complicated process to get that latent representation for the data point x. And then you need to decoder that generates an image. And then you have the loss right here that compares the two that is used to train the encoder and the decoder. Whereas in the gradient origin networks, what you do is you start you basically have a function f and the function f it's a bit weird right here, the function f uses two things. So this here is that z, which is termed zero here. But in fact, it's the latent representation of the image, which is derived from the image itself. And I don't really know, so I guess you can hear you can input this x is derived from the image itself by some way that doesn't require parameters that is not learned. And it also takes in these coordinates, and it produces that image. Now let's disentangle two things right here, what we're going to see is equally applicable to non implicit neural networks. So for the rest of this paper, I'm not saying it's going to work as well, maybe it's going to work specifically well with implicit neural networks. But we need to differentiate the these two things. So the first thing is explicit versus implicit. Okay, we're simply going to view these as functions that take a z and give you an x. Okay, if this is this is most notably the explicit version, the implicit version is simply that we're going to take a z along with all the x and y of the image. And we're going to obtain the R, g and b values of all the images, right, which is equal to the x. So this this entire set of RGB values is equal to the x, and we input the entire set right here. So essentially, it's simply a function that takes in a latent representation of an image and gives you back a image. The second thing, which is an entirely different thing, in my opinion, is how do we obtain a z from an x? So how do we get to have an image? How do we obtain the corresponding latent representation? And such that such that. So this must be such that this function right here, the function that gives you the x from the z will reproduce the x. Okay. So how do we obtain the correct latent representation for any for any input data point? Two different things. Don't. So I think they're not dependent on each other, except, as I said, they might work especially well together or something like this. All right. So this becomes a lot easier right now in this formula. So this is the thing ultimately that they optimize. They optimize the this thing and it's introduced like I don't know why they limited themselves to four pages here. And again, this is work in progress, as I understand it. But it is it is not it's like cold water. It's like, you know, an expressive neural network can be trained in this space to mimic this by minimizing the gradient origin network loss function. That's that's it. That's what you that's what you get. And then you get the loss thrown in your face. But let's deconstruct it. So this g thing right here, what's it? This is the loss that you minimize. Okay, you can see that this is simply an integral of this loss function over your entire coordinate space. So see here is the entire coordinate space. So this is for a given for a given image, right for a given image f x, you would minimize this actually across your across your entire data set. So you would minimize the parameters of f f here is going to be your generator neural network, your siren, whatever you minimize over the parameters of f across your entire data set. Okay, so this is your standard loss function. And this is some across your entire data set. Cool. So what are you going to minimize, you're going to minimize each data point consists of an integral over the coordinate space, which you can't see of this loss function right here. Now, this is simply due to the fact that this is an implicit representation. If this were an explicit representation, it would simply be the loss function of that data point, okay. So don't don't be scared by the integral. I'm usually scared by integrals, I never get them. And then I try to talk to them and people be like, do you think you know a remany an integral or a little big integral? And I'm like, okay, but in in this case, this is this simply means that you want the loss of each of the coordinates and you want to sum them up, right, which is the same as simply the the normal loss function with respect to a data point. This right here is the data point itself. As you can see, this is the this is your natural signal. So this is the function that you don't know. This is the true image function that maps the coordinates to the RGB space. In the case of explicit representation, this here is simply x. Okay, and forget about this integral for now. Cool. So we have a loss between x and whatever this is right here. This is a bit too long and whatever this is right here, you can see the loss function between two things. So what is this thing? The loss function, I can tell you the one they use in this particular paper is the L two loss. This is simply the reconstruction loss between a data point and its its reconstruction. Okay, so this part on the right is what's going to make the reconstruction. You can see, yes, our F here is going to be our siren, our neural network that will take in a Z. So F is one of these function explicit or implicit that takes in a Z and gives you x the the reconstruction. Now the question is, what does F take in? F takes in two things. First of all, the coordinates concatenated with the thing on the right. And you remember, we said that instead of giving x y to the implicit representation, we now give x y and z where z is the latent vector of the image we're trying to reconstruct. So if we were to see this as a non implicit method, we can simply leave away this right. So we as we leave away the x and y coordinates in a in a GAN or a VAE, we simply give it this thing right here. Again, we're trying to disentangle the implicit network, the implicit generator from how we are going to obtain the Z. So this is not important. So what remains is this quantity right here. So this must be our Z for the image. Okay, this thing. So what's this thing? I'm running slowly out of colors. This thing is going to be somehow the negative gradient of something. Again, you have the integral right here of the loss function. This again is x. This here again, we can leave this away. We can leave away the integral and you'll start to see kind of a repetitive thing. So this is going to be the gradient somehow of your loss function with that. Again, there is x and then there is f of z zero. So this is somehow an x to an x hat as well. But it's a special x hat. Let's call it x hat prime or x hat zero. Because the input is not z, but the input is now z zero. Okay, this is kind of a complicated thing. So I'm going to explain what's going on right here. Maybe in drawing. So what you want to do is you want to start out with z zero, which is an initial guess of what your latent representation is. You do it without looking even at the image, at the data point. You simply start with one. And there are multiple ways to do this. And this paper right here simply says we're going to see zero is just going to be a constant value zero, the constant value zero. That's why it's called gradient origin networks, because you always start with your z zero, your initial guess of your latent representation is the origin. Okay. Then you use F, your neural network to obtain a estimate, a first estimate of what your image could look like. Again, you have not looked at the image, you're simply taking the z zero and you produce an image. Then you somehow somehow obtain a better representation z. And that you use your F again to obtain x hat. And then from that x hat, you can now compare this to your x and that will give you your loss that you back propagate. So two things here, you can see you use F twice, which means that your loss, if you back propagate it, you must somehow back propagate to both of these things. Okay, so this is the first the first thing if you back propagate. The second thing is what's this thing right here? How are we going to obtain somehow a better z? And the better z is going to be obtained by basically looking at the gradient. So you've seen that we have a gradient of z zero of the loss of x and f of z zero. That's that thing here is going to be your z. z equals that. What does it mean? It basically means that so you've tried to produce an image, but this is the real image that you want to get and the loss measures how far apart you are from that real image. How would you need to change your initial guess in order to make that loss go down? So the negative here is to make the loss go down because otherwise it would make the loss go up. Okay, so it basically simply says how do you need to change your z zero in order to decrease the loss in order to get a better z for representing this particular image right here. And in the paper here is where I kind of disagree because in the paper they say that this in a single step this gives you the correct z or something like this. And I don't agree. They say with respect to the origin we obtain a latent vector that minimizes the reconstruction loss is obtained in a single step thereby playing the similar role to an explicit encoder. So this is true. This is kind of like an encoder, right? You simply ask what z would I need to put in in order to make this representation be a better sorry in order to make the latent representation be a better latent representation for the particular image x. However, if you compare so what is this? This is essentially gradient descent in the latent space, right? And the fact that we look at the explicit gradient is only because they started at the zero point right here. The fact that they started at the zero point means that here they can just leave away the following. So if we were to do gradient descent, what you would do is you would say this my z is going to be equal to z zero minus this thing, right? Now it looks much more like gradient descent in the latent space because you have some initial guess and then you update it using the gradient. Now there is no learning rate right here. So the learning rate is one in this case. So this is and again, the z zero because it's zero, you can just leave it away. So this is simply one single step of gradient descent in the latent space in order to get a better z right here. However, this is not a this is doesn't it doesn't guarantee you that in the single step you're actually going to find the correct zero even an appropriate z simply means that you're going to find a better z than z zero for that particular image. And this can work right. And again, because you back propagate to both of the F's, you say you basically say I want my neural network first of all to reconstruct the data point better from a given latent representation and I also want my neural network to give me a latent representation basically to help my latent to help this procedure. You back propagate through the gradient descent procedure. So you say I want my neural network to help me obtain a better latent representation if I do one step of gradient descent. So therefore it's not just pure gradient descent in that space. It actually the back propagation makes it such that your neural network also supports that supports obtaining a good representation in one step. Okay, now that we've disentangled this, basically, you can see two things. First of all, you could probably get an even better representation by doing multiple steps of gradient descent right here, maybe adjusting the learning rate a bit. It depends right because you have to back propagate through all the gradient descent steps. But pretty sure you could probably improve this by doing multiple steps. Second of all, it doesn't really matter that this is a constant zero. It gives you know, there's a cool name gradient origin networks, but you could probably start with any constant or even here's the thing even non constant initial points, you could sample them from a distribution and so on. Okay, so let's change like let's imagine changing z zero to be sampled from some normal distribution. And then it looks much more like a game, right? Alright, so here we go. I've cloned the repo and I ran the code once just to make sure that the data is downloaded and everything. And the code is, you know, pretty, pretty easy. So there is one file, and I didn't do it in the colab because the colab was, I think, a bit slow for me. I don't know if I've caught a wrong runtime. But essentially, there is a bunch of setup code, they know these siren layers and so on. And then you have the real deal thing right here. So you have the step. So we do 500 steps. And in each step, we as you can see right here, we start with zeros as z, then we put this into f concatenated with the coordinates. So the coordinates is like a kind of a mesh grid type thing. We obtain the inner loss right here, we do a gradient with respect so of the inner loss with respect to z. And then the negative gradient that's going to become our outer z. So this z up here is z zero, and this z down here is going to be our true z from the paper. We are going to concatenate that again, with the coordinates to obtain the g, which is the kind of reconstruction of x. And then our outer loss is going to be simply this reconstruction loss right here. And then we're going to backward to all of the parameters. Okay. So first hypothesis is that this here is simply kind of gradient descent. So what we should be able to do is first, let's run let's run this. So I've run this like that. So this is shipping it to a GPU server. And as you will be able to see, the loss will be output. And it's going to kind of decrease the loss over the course of 500 steps. And we can also look at the samples. So while that's happening, what we can do is we can actually already prepare what we want to do. So if this is really gradient descent, we should be basically just able to do this z minus this gradient right here, because it's zeros. We would simply expect this to yield the same loss. So we're going to do this, and then we're going to ship this off to the server again. Sorry. So we were here. And okay, the logs failed. All right. So this is called images. I have this thing set up such that it's called logs. But you can basically see that the loss right here was from 24 going to down to about 13 or so over the course of training. So by subtracting z minus the gradient, we there really shouldn't be any change, right? Because z is zero at the beginning. So again, we're going to run this. And while it's running, we're going to prepare the different things. So my hypothesis is that we can maybe we could make this z here pretty much anything. So let's do it. Let's put it into ones. Again, you see that the loss, I guess, you know, we get an idea of kind of the noisiness of this thing. And 2119, and so on. We can in fact, over here, we might be able to if we ship it to a different GPU, might be able to run two things in parallel. So this now is when we just start with ones instead of zeros. So let's see how that happens. While that's the case. So you can see right here that we ended up at also about 1413. This pretty much is the same if you can we can look at the images that it's produced. So the reconstructions look kind of like this of fashion MNIST, the samples kind of look like this. And the interval interpolations, you can look at those as well. But we're mainly interested also in the in the kind of loss right here. You can see that with the ones, pretty much the same thing is happening. So let's say we actually change this to a normal distribution. What does that do? And while that's happening, we're going to revert this to the original zeros. And we're going to investigate what happens if we just do more than one step of gradient descent. So in order to do that, it's actually pretty easy. So this here is the gradient descent step. What we can do is we can simply double that. So now if this is correct, I'm pretty sure this is correct. So the normal initialized isn't really the hit right here, as you can see. Wow. Okay. The normal isn't. Maybe it's because it's too large. I'm not sure. The other thing is deterministic, so that's going to be a lot easier. We can quickly go back and let's go ones. Let's go to normal. And let's multiply it with a tiny 0.01 or so. I just want to see whether this works. I have no big hopes. Okay. So we are here again, and we're going to make this into two different things. Two steps of gradient descent. All right. So now we have two steps of gradient descent. And let's see whether that helps. Ah, okay. So the normal distribution already helps or is not worse. We simply initialized it with too big of a variance. The 0.01 seems to be some kind of magic number for normal distributions and neural networks. So on the right side over here, and you can see we're a bit off, but I guess with a bit of tuning you could do that. And it gets down to about the same loss as you saw. If we look at the images that this produced, I'm going to guess they seem a bit worse, but it kind of works. On the right side, however, if you do more than one step of gradient descent, wah, wah, wee wah. You see, we already started lower losses. And since this is gradient descent, we can also, you know, there's no need why the learning rate should be one. So let's try to divide it by a generous three and then by maybe, you know, it's a six, like a decreasing learning rate seems like a rather good idea. And yeah, let's just take the two steps with the decreasing learning rate. Oops. So you can see that the loss now is way down just because we did two steps of gradient descent and the reconstructions, I'm going to guess they're almost perfect. So we're now, I guess we're overfitting a bit. So this is now trading off kind of power of the encoder decoder and so on. But ultimately, yeah, so let's just for the last part, just try to have this gradient descent with the decreasing step size and see where that gets us if that gets us to even a lower reconstruction loss. And that will be our investigation into the code right here. Okay. Do do do do do. Okay, we start with 19. Maybe we're as good as before. That's fine, you know. But I hope I hope that kind of gives a bit of evidence to my point that this is basically reversing a generator by using gradient descent, which has been around for a while. And I happen to know someone who who once attempted to write a paper about it. So yeah, but it's it's with implicit networks, which are pretty cool. So you know, maybe this might work especially well with them given that the gradient of a siren is a gradient and is a siren, and so on. Yep, as you can see, this works as well decreasing learning rate. And now you can go nuts. Oh, nine. Wow. This is the lowest loss we've gotten so far. Right? Yeah. So pretty cool. Interpolations look like things. These are the best samples. I think these are the best samples we've seen today. Maybe not. I'm not sure. Let's look at the interpolations quickly. Yeah, this looks like interpolations. I mean, if you squint. Okay, this was it for coding. See ya. Now, GANs have come with encoders before or it much more looks like a variational auto encoder as well. The difference here is we replace the encoder. So this here is our encoder, right? This is our implicit encoder is simply gradient descent. This has also been done before for GANs. So people train GANs and then they try to find the latent representation by back propagating. And some people even do this while training. They do gradient descent and then either do or do not back prop through the GAN, through the gradient descent procedure. So in a way or another, this is kind of sort of like those ideas, not saying it is equal. And again, there could be like some special interaction because you actually back prop through both these things and there could be some special interaction because these are implicit neural networks. However, I very much view these as two different things. The cool, there is a rather cool derivation of that where you can say, okay, you can also use it as a classifier by basically doing this. And now hope you can understand this much better. So what we'll have is we'll have the classification loss for sample X is going to be your cross entropy loss between two things. Okay. Well, can you please go down again? Thanks. So your cross your loss between two things is going to be the loss between your label Y. So that's one thing. And usually you have the feature, the logits on this side, right? Now you can see right here, you have an F that's probably that something that gives you the logits from your features. And here your features aren't going to be the data point itself, but your features are going to be the Z variable that comes with the data point. So basically you use this as a feature producer. And the feature producer is made by again, minimizing this reconstruction loss. Now I'm not sure this is going to work really well for classifiers because classifiers generally don't require you to reconstruct things. And we know this, you know, people try to, this is like you were to have a variational autoencoder and then simply use that encoder as a feature producer for a classifier, which generally doesn't work very well. But you know, you can, you can do it right here. And the cool thing is that you can actually use the implicit representation network F to give you features for the entire data sample Z. So you're kind of freed from the coordinate representation here and you get kind of a latent vector back. So this is how you would use an implicit neural network in order to do classification. That's I think, you know, pretty, pretty cool derivation of this. So here they make some empirical claims, which I don't, I don't want to go too much into, but there are certain advantages, certain practical advantages of doing things like this. Like you can have very, very few parameters to represent an entire set of data. The interpolations here work nicely as you can see. And I think generally they make the claim that this trains fast and you can see after three seconds, it already has a lot of information about the data set and it does some sensible things. Okay. So the code is available and in fact, I'll probably enter, inter parse into this video a let's actually test our hypotheses, right? Let's test these hypotheses that I said. So first hypothesis is probably we can start with something else than the constant zero and second hypothesis is we can probably improve by doing multiple steps of gradient descent in the inner loop. Yes, I, this might be somewhere in this video. And if not, it comes at the end like right now. Okay. So I'll see you next time. Bye bye.
 Hi there, today we'll look at gradient origin networks by Sam Bond Taylor and Chris G Wilcox
 of Durham University.
 So on a high level, this paper trains implicit representation networks, but not on single
 data points, but on entire data set.
 It does so by using a latent encoding of each data point.
 And it doesn't obtain that encoding through an explicit encoder, but by simply looking
 at the gradient of the latent variable when initialized at the origin.
 So it's a bit of a weird formulation, and I've seen this paper upvoted on Reddit, and
 the top comments would always say like, I don't really get it, I don't really get it.
 And I thought, you know, maybe I'm completely wrong, but I can just give my opinion kind
 of what's going on in this paper.
 Now also, most people on Reddit or a lot did say, I don't really get it, but here is what
 I think is going on.
 And then listing something and that's there is where I stopped reading.
 So as to not be kind of as to form my own opinion, I like to kind of understand papers
 by myself.
 So again, maybe I'm completely wrong.
 But here is my opinion.
 If you like opinions, hit the like button and subscribe if you aren't yet.
 And yeah, share this video out maybe that helps someone else understand.
 So this paper is a very short paper, it is four pages.
 And it's a dense paper, it definitely can warrant it definitely can warrant making a
 longer paper out of it.
 That being said, it's an archive paper for now.
 So you know, there's nothing wrong with archiving kind of unfinished work.
 But we're just going to look at it and try to understand it.
 Okay.
 So the abstract says this paper proposes a new type of implicit generative model that
 is able to quickly learn a latent representation without an explicit encoder.
 So for that, you need to know what an implicit generative model is.
 And I've covered one type of implicit generative models, specifically the type that they're
 using here, what they're called siren.
 So sirens are implicit representation networks.
 And I've made a video about sirens.
 So if you don't know what that is, go look it up.
 But very quickly, a siren will is a neural network to represent a single data point.
 So each data point in a data set is represented by its own neural network.
 And the neural network, so this might be a bit foreign to you.
 But usually you have some kind of image, right?
 And it's simply represented as an as an array of RGB coordinates, right?
 It's it's simply an array of this is like one zero point five and so on.
 So all the pixels are in this array.
 This is the explicit representation of that data point.
 Now, this here is a long list, and it has some regularities to it.
 So that's why you can also think of an implicit representation of the data point.
 The implicit representation works as follows.
 You imagine again your image, your image is made up of pixels, and these pixels are on
 X and Y coordinates.
 So this pixel right here would be zero zero.
 This pixel right here would be zero one, and so on.
 A siren is or generally an implicit representation network is a network that takes in any X and
 Y coordinate as the input.
 So the input itself is the numerical X and Y coordinate of that picture, and it passes
 it through a neural network and outcomes the RGB value.
 OK, so an entire picture is represented by this neural network.
 The neural network maps each coordinate to its RGB value.
 And here you can see that a single picture can become an entire data set for this neural
 network.
 In fact, it has to, because for a different picture, of course, there is a different mapping
 from X and Y coordinates to RGB coordinates.
 But this allows you to do multiple things.
 So first of all, this neural network can be smaller than the explicit representation.
 Second of all, it can capture some regularity in the data.
 Usually sirens have sine waves as nonlinearities in the neural network here, which is also
 a bit special, but lends itself very well to capture natural signals, because natural
 signals are often repeated at different scales and derivatives of themselves and so on.
 So I've covered this all in my video.
 And also this allows you to have a continuous representation rather than a discrete representation.
 Like here, you just have each pixel.
 Now you have a continuous representation.
 All right, so these are implicit representation models or implicit generative models, or these
 neural networks right here that map from coordinates to two colors.
 Now what's the problem with this is, as we said, you need one neural network per data
 point.
 Now, the idea that these people here go with is that can't we do kind of the same thing,
 but except we have one neural network per data point, we want to have the same neural
 network for the entire data set.
 So again, they want to have a neural network that somehow outputs RGB coordinates.
 But now it's not for a single image.
 Now we have a data set.
 Okay, and the data set has many images like this is image i, this is image j, this is
 image k.
 So what we could do is we could simply tell the neural network the x and y coordinate
 that we where we would like the RGB values to know.
 And we could also tell it which image it is right, k, or i or j.
 And this will give us a neural network right here that can represent the entire data set
 because it always can see, ah, I want of image j, I want these and these x y coordinate doesn't
 help you very much though, because it still has to learn for each image individually,
 how to encode it, how to produce it.
 What's much more interesting is, if you kind of mix this with the kind of old style, the
 kind of old style generative models.
 So in old style generative models, let's consider for example, an auto encoder.
 So in an auto encoder, what you would do is you would take your image and you would put
 it through an encoder.
 And this encoder will give you a latent variable z.
 And then you would put it through a decoder again.
 And that would give you an image.
 So your generative model now is this part right here, and this z variable is your latent
 encoding of this data point.
 Now, if you train these models correctly, be this a be this a an auto encoder or a variational
 auto encoder, or the green part can actually just be a GAN, right?
 If you train this correctly, then this z right here will be sort of a a latent encoding of
 the what the what of the information in the image itself.
 Okay.
 And that can generalize.
 So now I can input a picture that the model has never seen during training.
 And the encoder will map it to a latent representation that sort of makes sense that is able to reconstruct
 the image that I've put in.
 Okay, so the your hope with these latent representation is is that there is some kind of data manifold
 somewhere in hidden in the in the entire space of parameters.
 And as long as you're on that data manifold, you will produce a sensible data point.
 And this is kind of a continuous and so on.
 So even though you've only seen a few during training, if you have a new one during testing,
 then you can sort of it will be mapped to a correct place on the data manifold and it
 will produce a data point again.
 And you've seen this right, you've seen these interpolations in GANs where you can interpolate
 in latent space, and, and so on.
 The problem here is that, you know, in so in GANs, we sample these things right here.
 So that's a different story.
 But in VAEs, we need this encoder or in auto encoders, we need this encoder to obtain a
 latent representation for a given data point.
 In GANs, there is no way if we have an image, there is no way to obtain the corresponding
 Z variable.
 If we don't have an encoder, right, and that's the problem we're tackling right here.
 So here, what we want to do is we want to give the X and Y, we want to give the Z, we
 say we have some way of obtaining a latent representation of one of the image right here.
 And from that, we want to generate the RGB variables.
 Now the question is, think of again, the question is, how do we obtain the Z variable without
 having without having access to the encoder?
 And that's that's the problem of this paper.
 And this paper proposes a solution.
 So they say this is achieved with an implicit neural network that takes as inputs points
 in the coordinate space, alongside a latent vector initialized with zero.
 So that's the model that we saw.
 That's this, this is sorry about that.
 This is this right here, it takes in the coordinates, this is the coordinates, and it takes in the
 latent vector Z.
 Now, this whole point with it being initialized at zeros will get will get to that in one
 second.
 Okay.
 For the fact right now is just that the represent the implicit neural network also takes the
 identity of the image.
 So each image, the image is always going to have the same Z.
 And then we sort of say which x and y coordinate of that image we want.
 So the Z is per image.
 And then each image has all the x and y coordinates of, you know, itself.
 So yeah.
 So if yeah, you you I think you can follow.
 They go on they say the gradients of the data fitting loss with respect to this zero vector
 are jointly optimized to act as latent points that capture the data manifold.
 So this is where this is where I already got lost reading the first time through the results
 show similar characteristics to auto encoders, but with fewer parameters and the advantages
 of implicit representation networks.
 Okay, so we'll actually we'll, we'll jump to this right here.
 So this is the this is the comparison between a variational auto encoder and the gradient
 origin network.
 So in a variational auto encoder, what you would do is you would have this explicit encoder
 right here, as we said, and in the variational auto encoder, you don't obtain the latent
 representation directly, you actually obtain the distribution in terms of the mean and
 standard deviation of the latent representation.
 And then you sample from that distribution to obtain that latent representation.
 I think the point here is simply to show that you first of all, you do need an encoder,
 which you do need to train.
 And second of all, it's kind of a complicated process to get that latent representation
 for the data point x.
 And then you need to decoder that generates an image.
 And then you have the loss right here that compares the two that is used to train the
 encoder and the decoder.
 Whereas in the gradient origin networks, what you do is you start you basically have a function
 f and the function f it's a bit weird right here, the function f uses two things.
 So this here is that z, which is termed zero here.
 But in fact, it's the latent representation of the image, which is derived from the image
 itself.
 And I don't really know, so I guess you can hear you can input this x is derived from
 the image itself by some way that doesn't require parameters that is not learned.
 And it also takes in these coordinates, and it produces that image.
 Now let's disentangle two things right here, what we're going to see is equally applicable
 to non implicit neural networks.
 So for the rest of this paper, I'm not saying it's going to work as well, maybe it's going
 to work specifically well with implicit neural networks.
 But we need to differentiate the these two things.
 So the first thing is explicit versus implicit.
 Okay, we're simply going to view these as functions that take a z and give you an x.
 Okay, if this is this is most notably the explicit version, the implicit version is
 simply that we're going to take a z along with all the x and y of the image.
 And we're going to obtain the R, g and b values of all the images, right, which is equal to
 the x.
 So this this entire set of RGB values is equal to the x, and we input the entire set right
 here.
 So essentially, it's simply a function that takes in a latent representation of an image
 and gives you back a image.
 The second thing, which is an entirely different thing, in my opinion, is how do we obtain
 a z from an x?
 So how do we get to have an image?
 How do we obtain the corresponding latent representation?
 And such that such that.
 So this must be such that this function right here, the function that gives you the x from
 the z will reproduce the x.
 Okay.
 So how do we obtain the correct latent representation for any for any input data point?
 Two different things.
 Don't.
 So I think they're not dependent on each other, except, as I said, they might work especially
 well together or something like this.
 All right.
 So this becomes a lot easier right now in this formula.
 So this is the thing ultimately that they optimize.
 They optimize the this thing and it's introduced like I don't know why they limited themselves
 to four pages here.
 And again, this is work in progress, as I understand it.
 But it is it is not it's like cold water.
 It's like, you know, an expressive neural network can be trained in this space to mimic
 this by minimizing the gradient origin network loss function.
 That's that's it.
 That's what you that's what you get.
 And then you get the loss thrown in your face.
 But let's deconstruct it.
 So this g thing right here, what's it?
 This is the loss that you minimize.
 Okay, you can see that this is simply an integral of this loss function over your entire coordinate
 space.
 So see here is the entire coordinate space.
 So this is for a given for a given image, right for a given image f x, you would minimize
 this actually across your across your entire data set.
 So you would minimize the parameters of f f here is going to be your generator neural
 network, your siren, whatever you minimize over the parameters of f across your entire
 data set.
 Okay, so this is your standard loss function.
 And this is some across your entire data set.
 Cool.
 So what are you going to minimize, you're going to minimize each data point consists
 of an integral over the coordinate space, which you can't see of this loss function
 right here.
 Now, this is simply due to the fact that this is an implicit representation.
 If this were an explicit representation, it would simply be the loss function of that
 data point, okay.
 So don't don't be scared by the integral.
 I'm usually scared by integrals, I never get them.
 And then I try to talk to them and people be like, do you think you know a remany an
 integral or a little big integral?
 And I'm like, okay, but in in this case, this is this simply means that you want the loss
 of each of the coordinates and you want to sum them up, right, which is the same as simply
 the the normal loss function with respect to a data point.
 This right here is the data point itself.
 As you can see, this is the this is your natural signal.
 So this is the function that you don't know.
 This is the true image function that maps the coordinates to the RGB space.
 In the case of explicit representation, this here is simply x.
 Okay, and forget about this integral for now.
 Cool.
 So we have a loss between x and whatever this is right here.
 This is a bit too long and whatever this is right here, you can see the loss function
 between two things.
 So what is this thing?
 The loss function, I can tell you the one they use in this particular paper is the L
 two loss.
 This is simply the reconstruction loss between a data point and its its reconstruction.
 Okay, so this part on the right is what's going to make the reconstruction.
 You can see, yes, our F here is going to be our siren, our neural network that will take
 in a Z.
 So F is one of these function explicit or implicit that takes in a Z and gives you x
 the the reconstruction.
 Now the question is, what does F take in?
 F takes in two things.
 First of all, the coordinates concatenated with the thing on the right.
 And you remember, we said that instead of giving x y to the implicit representation,
 we now give x y and z where z is the latent vector of the image we're trying to reconstruct.
 So if we were to see this as a non implicit method, we can simply leave away this right.
 So we as we leave away the x and y coordinates in a in a GAN or a VAE, we simply give it
 this thing right here.
 Again, we're trying to disentangle the implicit network, the implicit generator from how we
 are going to obtain the Z.
 So this is not important.
 So what remains is this quantity right here.
 So this must be our Z for the image.
 Okay, this thing.
 So what's this thing?
 I'm running slowly out of colors.
 This thing is going to be somehow the negative gradient of something.
 Again, you have the integral right here of the loss function.
 This again is x.
 This here again, we can leave this away.
 We can leave away the integral and you'll start to see kind of a repetitive thing.
 So this is going to be the gradient somehow of your loss function with that.
 Again, there is x and then there is f of z zero.
 So this is somehow an x to an x hat as well.
 But it's a special x hat.
 Let's call it x hat prime or x hat zero.
 Because the input is not z, but the input is now z zero.
 Okay, this is kind of a complicated thing.
 So I'm going to explain what's going on right here.
 Maybe in drawing.
 So what you want to do is you want to start out with z zero, which is an initial guess
 of what your latent representation is.
 You do it without looking even at the image, at the data point.
 You simply start with one.
 And there are multiple ways to do this.
 And this paper right here simply says we're going to see zero is just going to be a constant
 value zero, the constant value zero.
 That's why it's called gradient origin networks, because you always start with your z zero,
 your initial guess of your latent representation is the origin.
 Okay.
 Then you use F, your neural network to obtain a estimate, a first estimate of what your
 image could look like.
 Again, you have not looked at the image, you're simply taking the z zero and you produce an
 image.
 Then you somehow somehow obtain a better representation z.
 And that you use your F again to obtain x hat.
 And then from that x hat, you can now compare this to your x and that will give you your
 loss that you back propagate.
 So two things here, you can see you use F twice, which means that your loss, if you
 back propagate it, you must somehow back propagate to both of these things.
 Okay, so this is the first the first thing if you back propagate.
 The second thing is what's this thing right here?
 How are we going to obtain somehow a better z?
 And the better z is going to be obtained by basically looking at the gradient.
 So you've seen that we have a gradient of z zero of the loss of x and f of z zero.
 That's that thing here is going to be your z.
 z equals that.
 What does it mean?
 It basically means that so you've tried to produce an image, but this is the real image
 that you want to get and the loss measures how far apart you are from that real image.
 How would you need to change your initial guess in order to make that loss go down?
 So the negative here is to make the loss go down because otherwise it would make the loss
 go up.
 Okay, so it basically simply says how do you need to change your z zero in order to decrease
 the loss in order to get a better z for representing this particular image right here.
 And in the paper here is where I kind of disagree because in the paper they say that this in
 a single step this gives you the correct z or something like this.
 And I don't agree.
 They say with respect to the origin we obtain a latent vector that minimizes the reconstruction
 loss is obtained in a single step thereby playing the similar role to an explicit encoder.
 So this is true.
 This is kind of like an encoder, right?
 You simply ask what z would I need to put in in order to make this representation be
 a better sorry in order to make the latent representation be a better latent representation
 for the particular image x.
 However, if you compare so what is this?
 This is essentially gradient descent in the latent space, right?
 And the fact that we look at the explicit gradient is only because they started at the
 zero point right here.
 The fact that they started at the zero point means that here they can just leave away the
 following.
 So if we were to do gradient descent, what you would do is you would say this my z is
 going to be equal to z zero minus this thing, right?
 Now it looks much more like gradient descent in the latent space because you have some
 initial guess and then you update it using the gradient.
 Now there is no learning rate right here.
 So the learning rate is one in this case.
 So this is and again, the z zero because it's zero, you can just leave it away.
 So this is simply one single step of gradient descent in the latent space in order to get
 a better z right here.
 However, this is not a this is doesn't it doesn't guarantee you that in the single step
 you're actually going to find the correct zero even an appropriate z simply means that
 you're going to find a better z than z zero for that particular image.
 And this can work right.
 And again, because you back propagate to both of the F's, you say you basically say I want
 my neural network first of all to reconstruct the data point better from a given latent
 representation and I also want my neural network to give me a latent representation basically
 to help my latent to help this procedure.
 You back propagate through the gradient descent procedure.
 So you say I want my neural network to help me obtain a better latent representation if
 I do one step of gradient descent.
 So therefore it's not just pure gradient descent in that space.
 It actually the back propagation makes it such that your neural network also supports
 that supports obtaining a good representation in one step.
 Okay, now that we've disentangled this, basically, you can see two things.
 First of all, you could probably get an even better representation by doing multiple steps
 of gradient descent right here, maybe adjusting the learning rate a bit.
 It depends right because you have to back propagate through all the gradient descent
 steps.
 But pretty sure you could probably improve this by doing multiple steps.
 Second of all, it doesn't really matter that this is a constant zero.
 It gives you know, there's a cool name gradient origin networks, but you could probably start
 with any constant or even here's the thing even non constant initial points, you could
 sample them from a distribution and so on.
 Okay, so let's change like let's imagine changing z zero to be sampled from some normal distribution.
 And then it looks much more like a game, right?
 Alright, so here we go.
 I've cloned the repo and I ran the code once just to make sure that the data is downloaded
 and everything.
 And the code is, you know, pretty, pretty easy.
 So there is one file, and I didn't do it in the colab because the colab was, I think,
 a bit slow for me.
 I don't know if I've caught a wrong runtime.
 But essentially, there is a bunch of setup code, they know these siren layers and so
 on.
 And then you have the real deal thing right here.
 So you have the step.
 So we do 500 steps.
 And in each step, we as you can see right here, we start with zeros as z, then we put
 this into f concatenated with the coordinates.
 So the coordinates is like a kind of a mesh grid type thing.
 We obtain the inner loss right here, we do a gradient with respect so of the inner loss
 with respect to z.
 And then the negative gradient that's going to become our outer z.
 So this z up here is z zero, and this z down here is going to be our true z from the paper.
 We are going to concatenate that again, with the coordinates to obtain the g, which is
 the kind of reconstruction of x.
 And then our outer loss is going to be simply this reconstruction loss right here.
 And then we're going to backward to all of the parameters.
 Okay.
 So first hypothesis is that this here is simply kind of gradient descent.
 So what we should be able to do is first, let's run let's run this.
 So I've run this like that.
 So this is shipping it to a GPU server.
 And as you will be able to see, the loss will be output.
 And it's going to kind of decrease the loss over the course of 500 steps.
 And we can also look at the samples.
 So while that's happening, what we can do is we can actually already prepare what we
 want to do.
 So if this is really gradient descent, we should be basically just able to do this z
 minus this gradient right here, because it's zeros.
 We would simply expect this to yield the same loss.
 So we're going to do this, and then we're going to ship this off to the server again.
 Sorry.
 So we were here.
 And okay, the logs failed.
 All right.
 So this is called images.
 I have this thing set up such that it's called logs.
 But you can basically see that the loss right here was from 24 going to down to about 13
 or so over the course of training.
 So by subtracting z minus the gradient, we there really shouldn't be any change, right?
 Because z is zero at the beginning.
 So again, we're going to run this.
 And while it's running, we're going to prepare the different things.
 So my hypothesis is that we can maybe we could make this z here pretty much anything.
 So let's do it.
 Let's put it into ones.
 Again, you see that the loss, I guess, you know, we get an idea of kind of the noisiness
 of this thing.
 And 2119, and so on.
 We can in fact, over here, we might be able to if we ship it to a different GPU, might
 be able to run two things in parallel.
 So this now is when we just start with ones instead of zeros.
 So let's see how that happens.
 While that's the case.
 So you can see right here that we ended up at also about 1413.
 This pretty much is the same if you can we can look at the images that it's produced.
 So the reconstructions look kind of like this of fashion MNIST, the samples kind of look
 like this.
 And the interval interpolations, you can look at those as well.
 But we're mainly interested also in the in the kind of loss right here.
 You can see that with the ones, pretty much the same thing is happening.
 So let's say we actually change this to a normal distribution.
 What does that do?
 And while that's happening, we're going to revert this to the original zeros.
 And we're going to investigate what happens if we just do more than one step of gradient
 descent.
 So in order to do that, it's actually pretty easy.
 So this here is the gradient descent step.
 What we can do is we can simply double that.
 So now if this is correct, I'm pretty sure this is correct.
 So the normal initialized isn't really the hit right here, as you can see.
 Wow.
 Okay.
 The normal isn't.
 Maybe it's because it's too large.
 I'm not sure.
 The other thing is deterministic, so that's going to be a lot easier.
 We can quickly go back and let's go ones.
 Let's go to normal.
 And let's multiply it with a tiny 0.01 or so.
 I just want to see whether this works.
 I have no big hopes.
 Okay.
 So we are here again, and we're going to make this into two different things.
 Two steps of gradient descent.
 All right.
 So now we have two steps of gradient descent.
 And let's see whether that helps.
 Ah, okay.
 So the normal distribution already helps or is not worse.
 We simply initialized it with too big of a variance.
 The 0.01 seems to be some kind of magic number for normal distributions and neural networks.
 So on the right side over here, and you can see we're a bit off, but I guess with a bit
 of tuning you could do that.
 And it gets down to about the same loss as you saw.
 If we look at the images that this produced, I'm going to guess they seem a bit worse,
 but it kind of works.
 On the right side, however, if you do more than one step of gradient descent, wah, wah,
 wee wah.
 You see, we already started lower losses.
 And since this is gradient descent, we can also, you know, there's no need why the learning
 rate should be one.
 So let's try to divide it by a generous three and then by maybe, you know, it's a six, like
 a decreasing learning rate seems like a rather good idea.
 And yeah, let's just take the two steps with the decreasing learning rate.
 Oops.
 So you can see that the loss now is way down just because we did two steps of gradient
 descent and the reconstructions, I'm going to guess they're almost perfect.
 So we're now, I guess we're overfitting a bit.
 So this is now trading off kind of power of the encoder decoder and so on.
 But ultimately, yeah, so let's just for the last part, just try to have this gradient
 descent with the decreasing step size and see where that gets us if that gets us to
 even a lower reconstruction loss.
 And that will be our investigation into the code right here.
 Okay.
 Do do do do do.
 Okay, we start with 19.
 Maybe we're as good as before.
 That's fine, you know.
 But I hope I hope that kind of gives a bit of evidence to my point that this is basically
 reversing a generator by using gradient descent, which has been around for a while.
 And I happen to know someone who who once attempted to write a paper about it.
 So yeah, but it's it's with implicit networks, which are pretty cool.
 So you know, maybe this might work especially well with them given that the gradient of
 a siren is a gradient and is a siren, and so on.
 Yep, as you can see, this works as well decreasing learning rate.
 And now you can go nuts.
 Oh, nine.
 Wow.
 This is the lowest loss we've gotten so far.
 Right?
 Yeah.
 So pretty cool.
 Interpolations look like things.
 These are the best samples.
 I think these are the best samples we've seen today.
 Maybe not.
 I'm not sure.
 Let's look at the interpolations quickly.
 Yeah, this looks like interpolations.
 I mean, if you squint.
 Okay, this was it for coding.
 See ya.
 Now, GANs have come with encoders before or it much more looks like a variational auto
 encoder as well.
 The difference here is we replace the encoder.
 So this here is our encoder, right?
 This is our implicit encoder is simply gradient descent.
 This has also been done before for GANs.
 So people train GANs and then they try to find the latent representation by back propagating.
 And some people even do this while training.
 They do gradient descent and then either do or do not back prop through the GAN, through
 the gradient descent procedure.
 So in a way or another, this is kind of sort of like those ideas, not saying it is equal.
 And again, there could be like some special interaction because you actually back prop
 through both these things and there could be some special interaction because these
 are implicit neural networks.
 However, I very much view these as two different things.
 The cool, there is a rather cool derivation of that where you can say, okay, you can also
 use it as a classifier by basically doing this.
 And now hope you can understand this much better.
 So what we'll have is we'll have the classification loss for sample X is going to be your cross
 entropy loss between two things.
 Okay.
 Well, can you please go down again?
 Thanks.
 So your cross your loss between two things is going to be the loss between your label
 Y.
 So that's one thing.
 And usually you have the feature, the logits on this side, right?
 Now you can see right here, you have an F that's probably that something that gives
 you the logits from your features.
 And here your features aren't going to be the data point itself, but your features are
 going to be the Z variable that comes with the data point.
 So basically you use this as a feature producer.
 And the feature producer is made by again, minimizing this reconstruction loss.
 Now I'm not sure this is going to work really well for classifiers because classifiers generally
 don't require you to reconstruct things.
 And we know this, you know, people try to, this is like you were to have a variational
 autoencoder and then simply use that encoder as a feature producer for a classifier, which
 generally doesn't work very well.
 But you know, you can, you can do it right here.
 And the cool thing is that you can actually use the implicit representation network F
 to give you features for the entire data sample Z.
 So you're kind of freed from the coordinate representation here and you get kind of a
 latent vector back.
 So this is how you would use an implicit neural network in order to do classification.
 That's I think, you know, pretty, pretty cool derivation of this.
 So here they make some empirical claims, which I don't, I don't want to go too much into,
 but there are certain advantages, certain practical advantages of doing things like
 this.
 Like you can have very, very few parameters to represent an entire set of data.
 The interpolations here work nicely as you can see.
 And I think generally they make the claim that this trains fast and you can see after
 three seconds, it already has a lot of information about the data set and it does some sensible
 things.
 Okay.
 So the code is available and in fact, I'll probably enter, inter parse into this video
 a let's actually test our hypotheses, right?
 Let's test these hypotheses that I said.
 So first hypothesis is probably we can start with something else than the constant zero
 and second hypothesis is we can probably improve by doing multiple steps of gradient descent
 in the inner loop.
 Yes, I, this might be somewhere in this video.
 And if not, it comes at the end like right now.
 Okay.
 So I'll see you next time.
 Bye bye.