Noether Networks: Meta-Learning Useful Conserved Quantities (w/ the authors)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper noether networks noether's theroem noether theorem symmetries neural network bias neural network symmetries inductive biases conserved quantities pendulum neural network physics deep learning physics deep learning symmetries group convolutions with the authors paper explained deep learning prediction test time optimization tailoring neural network tailoring
#deeplearning #noether #symmetries

This video includes an interview with first author Ferran Alet!
Encoding inductive biases has been a long established methods to provide deep networks with the ability to learn from less data. Especially useful are encodings of symmetry properties of the data, such as the convolution's translation invariance. But such symmetries are often hard to program explicitly, and can only be encoded exactly when done in a direct fashion. Noether Networks use Noether's theorem connecting symmetries to conserved quantities and are able to dynamically and approximately enforce symmetry properties upon deep neural networks.

OUTLINE:
0:00 - Intro & Overview
18:10 - Interview Start
21:20 - Symmetry priors vs conserved quantities
23:25 - Example: Pendulum
27:45 - Noether Network Model Overview
35:35 - Optimizing the Noether Loss
41:00 - Is the computation graph stable?
46:30 - Increasing the inference time computation
48:45 - Why dynamically modify the model?
55:30 - Experimental Results & Discussion

Paper: https://arxiv.org/abs/2112.03321
Website: https://dylandoblar.github.io/noether-networks/
Code: https://github.com/dylandoblar/noether-networks

Abstract:
Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases. Useful biases often exploit symmetries in the prediction problem, such as convolutional networks relying on translation equivariance. Automatically discovering these useful symmetries holds the potential to greatly improve the performance of ML systems, but still remains a challenge. In this work, we focus on sequential prediction problems and take inspiration from Noether's theorem to reduce the problem of finding inductive biases to meta-learning useful conserved quantities. We propose Noether Networks: a new type of architecture where a meta-learned conservation loss is optimized inside the prediction function. We show, theoretically and experimentally, that Noether Networks improve prediction quality, providing a general framework for discovering inductive biases in sequential problems.

Authors: Ferran Alet, Dylan Doblar, Allan Zhou, Joshua Tenenbaum, Kenji Kawaguchi, Chelsea Finn

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 But the intuition is that knowing these five conserved quantities is going to tell me a bit about what my prediction should be. And so it's kind of free information that I get to know. Hello there! Today we'll look at Nöter Networks Meta-Learning Useful Conserved Quantities by Ferran Oled and Dylan Doblar and others. This is another one of the with the authors installations videos, whatever, where I just discuss the paper briefly right now and then we'll jump into an interview with one of the first authors, with Ferran, and we'll go through the paper together. And I think Ferran can explain this so much better than I can. And I'm also able to ask some of my dumb questions. So this was a lot of fun and I definitely invite you to stick around. If you already know a little bit what the paper is about, feel free to skip ahead. If you don't know what the paper is about, the paper essentially deals with neural networks that predict dynamical systems. And in these dynamical systems, very often there are these conserved quantities that are part of it. For example, in a physical system, energy is conserved, momentum is conserved, and things like this. And under this constraint, you can build in this constraint into the predictive neural network so that the neural network does a better job. And they build these neuter networks in order to dynamically learn these conserved quantities, and then adjust at runtime during forward propagation, tailor the loss to conserve these quantities. And I think that's really cool. It's different. And yeah, that's what I like about it. So pretty brief introduction, this paper obviously is named after Neuter's theorem, which essentially they say here loosely states the following. For every continuous symmetry property of a dynamical system, there is a corresponding quantity whose value is conserved in time. For example, they say a system of planets interacting via gravity, the system is translation invariant in all three cardinal directions. Neuter's theorem asserts that there must be a conserved quantity for each of these symmetries. In this case, linear momentum is conserved. So the symmetry in space as translations is accompanied by a conserved quantity, which is linear momentum. Now, we don't always obviously know these quantities. And they're not always super explicit. And they're not always exact. So what we are going to be dealing with here is predictions of dynamical systems. And the example here is the prediction of a video of like a physical interaction. So this is a thing here on an inclined plane, it sort of slides down, and then collides with this other thing right here. And the goal is to predict the next frames of this video. Now, we could just build a neural network to just to predict these things frame by frame by frame. And that would go certainly well, if we had a lot of data. However, if we don't have a lot of data, what we need to do is we need to build in inductive biases. And inductive biases, what people usually do is they build in these symmetries directly, for example, they build in the physical laws, they know how the world works. And they say, you know, whether I translated to the left or to the right, it doesn't really matter, and so on. But building in these symmetries, and I think we know this from geometric deep learning, building in these symmetries is very powerful, but it can also be cumbersome, because you have to define them beforehand. This paper goes ahead and says, you know, what's real, what's a lot easier than building in symmetries directly is building in a constraint to conserve a given quantity. And that is a lot easier. And there's a potential that you can actually learn it from data. And with Noether's theorem, we know that the two things are equivalent. So if a system conserves a quantity, it essentially encodes a symmetry in the system. So what do we do? This is the very high level overview over these networks, we take to this entire thing here is one forward propagation, we take the original frame, we put it through a forward predicting neural network, which is this f theta right here. This is a network that simply forward predicts frames as we I said initially. So we forward predict forward predict forward predict, this gives us an initial set of of outputs right here, these x tilde, now these are going to be pretty, pretty bad, not pretty bad. But if we don't have a lot of data to learn from these, we don't expect them to be particularly good. And that's the regime we are here. What we do then is we're trying to adjust this f thing right here. In the moment, so during the forward propagation, we're going to update our predicting neural network by this neutral loss. So we're going to do an update, a temporary update to the weights of the f network. And we're going to do this into direction of this neutral loss. So you can see here, we have these networks G lying around, and G is always the same network. So what we're going to do is we're going to feed each frame that we predicted through G. And G always being the same network, it will output the same thing. And now obviously, if you know, given given that how I made this introduction, you might already have guessed that G is the part that predicts the quantity to be preserved. So what we want to do is we want to put all these things through G. And then we want to these these will give us a bunch of outputs, right? G here and here and here and here will output some things and the things can either be a number or an entire vector, right, an embedding vector. So essentially, G takes this thing right here, actually takes two consecutive frames, and embeds it into some space. And now, ideally, all these G's would output the same thing, which would mean which would mean that we have conserved some quantity and therefore encoded some symmetry. However, initially, these G's are not going to output the same thing. So we are going to attempt to change the F function such that the G's output more the same thing, there is a loss involved right here. This is the neutral loss, they call it, and it is defined down here. So you can see all this really is, is it's either defined in one of two ways. Either you take the difference between the G function of the initial frame and the frame at time point t, or and you calculate the difference, or you calculate the difference between consecutive frames. In either way, since you sum across all the frames, this means that all the outputs of the G network will should approximately be the same. Now, what do you do with this information? Again, we're still we're still during one forward propagation. So what do you do with this information, you calculate this neutral loss, which is one we just described, and then sorry for skipping around so much, you're going to do one update step. So these are the parameters of the F network, we're going to do one update step into the direction of the gradient. And it's the direction of the gradient with respect to the parameters of the F network. So this is the forward predicting network. So essentially, we are saying, how do I need to update my forward predicting network, such that, right, such that the frames that it outputs, the frames that it predicts in the future, make it such that the G functions of all of these frames are more similar to each other, or more similar to the G function of that first frame. So we're going to in time update the F function right here. And after that, we're going to forward propagate again, with this new F function, and thereby obtain our final prediction. This is one, this is like an inner optimization that we do during forward propagation. I find this to be pretty cool. Now they just do they just do one gradient step, obviously. Otherwise, you know, you could do a lot of things and you could like program in Adam and Ada grad, not only one like gradient step, which is one SGD step, essentially. But even with one step, that is good enough. So again, they here is the entire training procedure in an algorithm, you can see that. Let's start down here, they start with randomly initialized weights, these weights here are for the G network, these weights are for the F network, they sample batches for each batch, they predict the sequence. Now the sequence prediction is this entire thing we just looked at. So the sequence prediction is, I'm going to start at the initial frames, I'm going to use the F, the original F, the one I currently have, unconditional, let's say to forward predict all of the frames once, then I'm going to put all of these predictions here into this neutral loss, I'm going to calculate the gradient, how do I need to update this F for this particular data point to make the G functions output, the more similar things, I'm going to attain new parameters, again, these are just temporary parameters, I'm going to use these temporary parameters here to do another round of forward prediction, which gives me my final estimate, I could probably repeat this again. And or I could do multiple steps right here, I could probably do a lot of things, but this is sort of the simplest case. And then I will return these, what do I do with them? You can see right here, this is my output. Now I'm going to input these things into what's called the task loss. And the task loss in our case here is just the video prediction loss. So that's going to be some L2 distance between the frames, the output and the frames that actually, so that these are the output frames, these are the frames that are actually in the video. And then I'm going to just run back prop on that. So update the parameters of both G and F on the task loss. So what does it mean? G is going to be updated such that if I do this whole sequence again, if I do the whole sequence of predicting, then tailoring my loss to G, right, I tailor my loss to the G function, G is going to be updated such that next time, if I tailor my loss to it, it's going to lead to a better outcome overall. And F is going to be updated. Similarly, it's going to be updated such that, well, next time, if I do this whole procedure of first predicting these, which I'm going to use the parameters, then updating the parameters, and then updating the parameters using G, and then predicting again, I update my F such that this whole procedure will result in a better loss. Now, I think this is the magic of our back propagation frameworks that we can even think of these types of things, because, I mean, behold, actually writing this down and implementing the backwards pass here yourself, that'd be crazy. So this is the entire algorithm right here. Now, again, given that there are, as you can see, some hyperparameters here, such as the learning rates, they only do one gradient step, as we mentioned. So this isn't an exact enforcement of that constraint, right? This is only an approximate enforcement. Essentially, the only additional constraint that we introduce here is this requirement that the G function is the same G function on all the forward predicted things. And that is our knowledge that we are dealing with a dynamical system. And in this dynamical system, some quantities should be preserved. The way we build the losses means that G can simply output a constant value, otherwise, it would not be useful to the loss, right? But also the way we build the loss means that it is not an exact constraint, like we would build this into the architecture that a quantity must be conserved. So it's able to deal with real world data, such as this video where even sometimes a hand may come in, there's friction and so on. It's not an exactly conserving system, right? And the way we do this in the moment in the forward pass update using this neutral loss, that means that I can now tailor whatever like I can tailor the inductive bias for this particular sample. So I can learn it's kind of meta learning thing, right? What I learn is how to in the moment, adjust my loss function to this particular sample of data. Now, as I said, obviously, if you had more data and all, maybe you wouldn't need this, but it does help a lot in their experiments in the in these regimes where you do not have a lot of data, they have a theoretical section right here, where they have a reduced case and show that it can be useful to impose these constraints, then they have a bunch of experimental settings, among other things, they also they don't only do what I just said with the video prediction, but they also do a prediction where they don't not everything is a neural network. So where the things they predict are actual physical quantities, and they do it using symbolic regression. And this is the same method except it's not neural networks, it's symbolic regression. And what that does is, it comes up with these equations, for example, for the ideal pendulum, as you can see, these equations are insanely close, like they recover the correct equations. And these are symbolic regressions. So the it's not you don't you didn't only have to come up with the number right here, you actually, the network had to come up not the network, the system had to come up with the entire equation, given some basic building blocks of variables, and you can square stuff, and you can take the cosine of stuff. So these experiments show that the method can indeed recover physical quantities that are conserved if you present them with a scenario where this is the case, and they use either ideal scenarios, so ideal data generation, but they also use real world data from pendulums, where obviously you have energy dissipating, and then you can, you can compare. So here, I believe they do compare with what they say is a baseline. So as that predicts into the future, the longer prediction they do, the worse that gets. Or, I guess the losses over here, you can see that. But then also, the Hamiltonian neural networks, which enforce exact constraints, they enforce the quantities to be preserved exactly. If you face them with real world data, you can see right here, the quantities aren't changed at all, yet the loss still goes up because the quantity isn't actually conserved in the real data. And the neural networks do follow the ground truth data much more closely, because they can model also in exact constraints and not super strict enforcement of these constraints, which is what I think we need in real world data. They do have a bunch of other experiments, especially as I said, also video prediction where they do outperform various baselines, they investigate where the network pays attention to and whether or not you can actually move or do a lot more inner iteration steps than just one, because we just did one inner iteration steps there, there is no reason why this should remain at one. And here they show that even though they only trained with one at inference time, they can actually take a bunch more and the outer loss will still go down. So this all validates a little bit of the reasoning behind the method. Yeah, I don't want to take up too much of your time right here because I want to jump into the interview. Let me know what you think of these more interviewee style paper reviews. I quite enjoyed the interview. And I do think it's pretty useful to have the authors there because they can correct me pretty instantly. All right, see you over there. Okay, cool. Hi, everyone. Today I have with me Ferran Aled, who is one of the primary authors of the Nöter Networks paper and here to discuss with us probably a little bit about the intrinsics of the paper. And maybe also for me personally, because the paper is very technical, it's very technical. It's a new field for me as well, connecting physics to machine learning, building all of this into neural networks. There's also a bit of symbolic regression in there. So I feel a lot of things are coming together here. I found the paper pretty cool and it's new and that's what's interesting. So Ferran, thank you very much for being here. Yeah, thanks for the invitation. Wonderful to be here. Thanks. So your paper deals with, do you call it Nöter Networks, how do you pronounce? I pronounce it Nöter Networks, but I think I'm not German, so I'm not sure I'm pronouncing it properly. I'm not a German either, but I think that the author was called Nöter. Yeah, so you're pronouncing it more properly than I am. Maybe. But essentially, could you give us maybe just first an insight, where does the name, because the name is kind of distinct, right? Because there is the Nöter Theorem. What does the Nöter Theorem say in general? Yeah, so the Nöter Theorem was kind of the inspiration for our work. And the intuition is that for every symmetry of a dynamical system, there is a certain conservation law that's going to apply to that system. So for instance, imagine you have a planetary system of planets moving around. The physics laws don't change from today to tomorrow. That means that there's a time symmetry of the system. And here, Nöter's theorem tells you, oh, if there is a symmetry here, that means that there must be a quantity that's conserved over time. And in this case, for time symmetry, there is energy that's being conserved. So we use that as a motivation, not that the technical details, more like the higher level message of the theorem, to build a new machine learning model. And the intuition is that in machine learning, symmetries are one of the core ways in which we've improved data efficiency and model performance. And so it would be very cool if we could kind of automatically learn some of these symmetries. But symmetries are kind of hard to quantify and get a hold of computationally. And the intuition is that they talk about kind of counterfactuals and kind of global in the sense that when I was telling you about this time symmetry, I was saying, if I were to look at the planetary system tomorrow, the laws of physics would be the same. But I don't have access to the data for tomorrow. It's a kind of counterfactual. So the model cannot handle this. Instead, conserved quantities can be directly measured. I can check, oh, this quantity, which I will call energy, is being conserved on my actual data. And that makes it very easy to quantify. Yeah, we've heard in, I think in the recent past, even a lot of people attempting to get more out of symmetries out of neural network with I'm thinking of, I'm thinking of like, group convolutional neural networks, and so on that try to actively build in symmetries into neural networks. But it seems like they can only do that in situations where they know the symmetry that will appear, they already know a molecule doesn't matter which way I look at it, right, so I can directly build that in. But your reasoning is that because assessing conserved quantities is an easier task than assessing symmetries, it might be possible to learn the conserved quantities dynamically actually learn them from data. Is that approximately correct? Yes, exactly. Exactly. So and the theorem is the motivation because it tells us that conserved quantities are kind of on the same level of powerful as symmetries for dynamical systems, in particular, if you're doing image classification that does not apply because image classification is not a dynamical system. But that's the intuition. Yes. And you even have some slack in there you discuss, you know, we can, we, it doesn't even have to be absolutely conserved quantity, it doesn't have to be an absolute symmetry that we deal with. By learning it from data, we can even handle approximate symmetries. Is that right? That's another thing that may be a bit different from our work than other works, which is that some symmetries are only approximately conserved or conserved quantities are only approximately conserved. So for instance, you have if you have a dissipative system, like in the real world restriction, and so you actually lose energy, you don't consider if you don't consider the entire system, you're usually have small losses. So in this case, you would say you would like to say, oh, energy is conserved, but not quite. So it's fine if you if your prediction doesn't fully conserve energy. But knowing about energy conservation maybe helps you with the overall prediction. And maybe I want to want to get to sort of a little bit of an example of where so people can imagine this a little bit more. Now, I only have a mouse here because I forgot the iPad because I'm stupid. But maybe we can give the small example of a pendulum, right? So here's a pendulum, it hangs here, and it sort of gets down here. And here's the little ball. And the pendulum is accurately described by I think the angle right here that it's sort of off the off the main axis, and also its momentum, let's say it swings in this direction with a certain with a certain speed. And this describes the pendulum. Now your model focuses on predicting the future, let's say, or at least from from what I can tell. So what your model would be able to do is it would be able to predict the next time step right here, right? Then it's a bit here, here. Sorry, it's a little bit more up to the left, right? So it's a little bit more up and then it's it's even more up over here and then it swings back and so on it swings back over. Now, can you explain to us what are sort of the what is the symmetry here? And what are the conserved quantities? Yeah, so in this case, for the pendulum, we know that if we were to swing the pendulum now and 10 minutes from now, the physics wouldn't change. And so we know that there's a time symmetry. And so in this case, we would say, oh, there's a time symmetry and then another theorem would would tell us, oh, energy is conserved. So in this case, energy is a mixture of the kinetic energy, which is how much movement there is, and more movement, the more energy, and potential energy, which in this case is because of gravity. So a combination of these must be conserved. We don't know exactly how which formula and that's what we're going to automatically discover. I see. And the original approach, I think, would just be that here, this arrow, I parameterize this with some neural network, right? I just say, you know, here, I plug in neural network, I predict the next time step, and the next time step, and the next time step, and that it will maybe work, right? But it will, let's say, will only implicitly make use, it will not actually make use of the fact that something is conserved. So you go ahead and you say, since this is a dynamical system, we know more about the system, we can impose additional constraints. And the additional constraints right here, if I see this correctly, essentially, at every time step, you say, I want to build a neural network that's always going to be the same neural network that takes a state, let's say the pendulum in this state, and predicts a quantity, let's call that, no, G is the name of the network, let's call the quantity, I don't know, alpha. And I want to use that same neural network in all the different states that I find this thing in. And it always needs to predict the same thing, right? Since it needs to figure out a quantity that is conserved. And now it is, if I just train a neural network to always predict the same number right here, I would just end up with a neural network that is predicting some kind of a constant, right? Yeah. So your method figures out how do I need to build, first of all, this predictive neural network to predict this conserved quantity, such that it actually predicts something useful. But then also, how do I make this network right here actually use the fact that this other network predicts common quantities, right? Yeah, exactly. So that's why the word useful in our title, because there is many conserved quantities that are kind of not useful. And so we want to find those that are helpful for loss, final loss. So in machine learning, we usually care about some performance, whatever it is. And so that's exactly what we, that our objective just cares about that. And the useful quantities are just a proxy and intermediate thing for getting us to better performance. Yeah. And so here you have this main diagram, I think that that would be considered the main diagram describing your method. And this is on a task that is a video prediction task. And it's about sliding something down an incline. Could you maybe describe what the task here is? The frames are a bit low resolution. So this is the physics 101 data set from Josh Tenenbaum's group. I think Jesun was the first author. And they have a collection of videos. And in this case, they have a hand dropping an object passively, like it just lets it drop down and the object falls down. And there's a second object at the end of the ramp, they collide. And then the other one, sometimes depending on the masses and the friction and whatnot, the dynamics are kind of can change. That's the data set. And does, so that there are multiple videos and it's always different objects or? Like some objects could be common between videos, but there's lots of objects. So it's not always the same object. And that's kind of the point, the fact that it can vary. So one nice thing about the other networks is that they can deal with raw video. So some usually conserved quantities, you get them from kind of state data. Like when I was telling you, when we were talking about the pendulum, it's kind of, you have the exact position of the pendulum, you have the momentum of the pendulum, you don't have a pixel video of the pendulum. And here, because we deal with neural networks that predict the conserved quantities, you can hopefully get conserved quantities from video. Yeah. So here, the diagram shows a little bit of what you're, what you are trying to do, but also what you're trying to avoid. So the bottom path right here, if I see this correctly, that would be if I did nothing else, except the bottom path, I would build this neural network to just predict sort of the future time steps. And that often turns out poorly. I don't know, this is a quite a pixel-ish mess, but it's sort of, it's sort of, all of a sudden, there are like three objects instead of two, and the one is kind of gone or split up. And it's a bit of a mess. And you attribute this to the fact that it's just a video prediction or? Yeah, well, in this case, to analyze it and to make the problem challenging, we made the, like there was very few data. In general, you can, it's all like symmetries and inductive biases are going to be most useful when the problem is hard and then there is like less data. So in this case, there was a few ones of videos and also because video prediction is pretty long. So at the very few, like at the beginning of the frames, like the first few frames, there was not that much mistakes. But when you go very far into the future, then it's much harder. So those two problems, lack of data and the fact that you go a lot into the future. Your method is, and you also have an algorithm described somewhere. It's a bit of a, it's a algorithm that is, oh, right here. It's an algorithm that has multiple steps in it. And one special part is that you have this sort of inner optimization loop right here. Now, I want to maybe go back to the diagram and let's go, let's walk through it once before we, before we, you know, take a look at the formulas and all we can walk through it once. So the first thing that happens, if I understand correctly is you take your first input and you do exactly what we just said, you run it through a forward prediction neural network that just tries to predict the future, just plain by itself. Right. So this has, this has a bit of a, of a default thing, but now you try to improve that. And this is all, this is the entire thing we're describing right now. That is one forward pass through your system. So you would take every single prediction that you made and you would feed it through this G network right here. And this G network is, you call it an embedding network. That is the thing ultimately that's trying to predict a conserved quantity. But it's not, it's not necessarily just outputting one number. It's outputting an entire vector. So it's an outputting and embedding vector. And the, the goal obviously is that for all of these inputs, it should output the same embedding vector. But so, ah, so, but this is, this is going to be, let's say trained such that across the dataset, it works well. So maybe, you know, for this video sequence, it's going to predict approximately the vector A for all the frames if it works well. And for another sequence with two different objects that obviously have a different total energy or so, it might predict a different embedding vector. Exactly. But all the same across the, across the video sequence. Okay. So this is how we can imagine you train this G network to sort of predict whatever is special about this particular data point, but inside of the data point conserved among all the frames. Exactly. Because if it was the same A for everyone, then you would have the issue that you mentioned at the beginning, then it's a useless conserved quantity. Yeah. So it's, it's almost like a bit of a description of the scene as such, right? That makes the video predictors life easier if you have sort of this, this global description. Yeah. Yeah. So the intuition, I think is, let's think about when the, if, if the network G was very good at predicting the conserved quantities and perfectly told you, oh, these five quantities, I know for certain that they're going to be conserved. Then we could, we will see the next step. We haven't gone through it yet, but the intuition is that knowing these five conserved quantities is going to tell me a bit about what my prediction should be. And so it's kind of free information that I get to know about constraints. So it's kind of an unsupervised loss that I have access at test time. Yeah. It restricts, it restricts what you can output, right? Because ideally the F network should only output whatever the G network says is, is the same, right? If the F network can only output things that the G network will embed to the same place in the embedding space or a similar place. Yes. There's just to be a hundred percent precise. There is lots of images that could make the network G happy because it only constrains like a few dimensions, but it has to make the network G say, oh, this is approximately what you had at the beginning. Yeah. Okay. And so that comes in in the next step. So here, what you do, you use, you take the input again and you route it through this F network again, but now this F network doesn't, is not like a free form predictor, but it actually takes, has somehow the notion of, of this information that the G network output out of the initial sequence again. And you do this in a very special way in that you actually take the parameters of F and you update them on the fly. Yes. You update them on the, so this is within a forward pass. You actually update the parameters into the direction of the gradient of G. Exactly. Yes. So, yeah, sorry. This is, I think that that it takes it. Yeah. So here you have this neutral loss. Yes, exactly. Which do you maybe want to talk about this briefly? Yes. So about another loss. Yeah, sure. So the other loss essentially is telling you, you should have, you should conserve G. So the, you know, for a fact that, so there's two ways of conserving G. They're roughly equivalent. If you fully impose them, if you don't fully impose them, they're not equivalent. That's why we put the approximate sign. So let's look at the term A here. It's basically saying, oh, you should conserve G. And so it should be, all of them should be equal to what G was telling you for the input X naught. So if you make the embedding of your prediction, note that X of T has kind of a tilde on top of it. So your prediction for XT should have the same conserved quantities as your input. And that's what your first term is. And just an MSC over this neural embedding. The second one is very similar. Sometimes it's a bit more useful, more stable, because instead of, if instead of comparing to the very beginning, you compare to the previous time step, you have a more immediate signal. And you basically say you should conserve it. Every time you apply F, you should conserve G. So that's the other basically important observation. And now we update theta and theta are the theta are the parameters of F, right? Theta are the parameters of F. We update these on the fly. And I suppose that we just do this in the moment. And for the next data point, we go back to the original parameters and do this again. So this is sort of an on the fly update for a temporary update of these parameters into the direction of this quantity right here. So this is the gradient of exactly the loss that we just discussed with respect to the parameters of F. So essentially, it says, what parameters would make F more apt at fulfilling this loss, which essentially means that these which how do we need to change F such that these forward predictions make the G conservation happier? Exactly. Exactly. So this is some previous work of ours, which we call tailoring. And the idea of tailoring is just because of what you said, that the fact that the adaptation is customized for each individual data point. And the idea there was a general way of encoding inductive biases with unsupervised auxiliary losses. So auxiliary losses in general, you say, for instance, one thing we could say is, oh, why not we add energy conservation when we train? Sometimes auxiliary losses would say, okay, I train for good predictions and I train for energy conservation at training time. But if you do that, you're not going to enforce energy conservation at test time. Because at test time, you're going to have a generalization gap in energy conservation. But energy conservation or any type of conservation or any auxiliary loss can be checked before making the prediction at test time or at training time. Inside the prediction function, I can first make my prediction and see, okay, do I like it? Does my auxiliary loss, does my unsupervised loss like this prediction? And if not, I can take a gradient step or multiple gradient steps to improve my unsupervised loss, in this case, the conservation loss. And so this makes it much better for the particular point we care about, which is the one we are making a prediction for. It's a bit surprising because it's a single data point. And maybe you have trained with a million data points. So the question is, why does one data point matter if we've trained with one million data points? Well, the idea is that you're training on the exact point you care about. So enforcing inductive bias in the exact point you care about right now for which you're making the prediction is going to have a very big impact. And so in this case, this gradient step improves the prediction just for that one point. Yeah, maybe it's also important to highlight that the parameter here, this theta that we start with, and also the parameters of G, those are the ones that will be learned during the training procedure across the entire training data set. And then the parameters here, those are always constructed in the moment, data point by data point, to, as you say, tailor the inductive bias. And the inductive bias, in this case, would sort of be this entire term right here, essentially says, how do I need to change my predictor in order to conserve the particular thing that G decides is the common quantity for this data point? Yeah. And this gives rise to the algorithm. So here is what we just discussed. This is the forward prediction sequence with this inner optimization step. So we first predict this plane sequence, then we temporarily update the parameters. And that allows us to again do the forward pass, but now with the updated F function, and that gives us sort of our final predictions. And as you can see here, during the training, we sample always batches, we forward predict using this inner update, and then we take outer gradients. And the L task here, that would just be what you call the task loss. This would be the video prediction loss or something like this. Okay. So I have a lot of questions. First of all, this, it seems quite intricate, right? Because if I think, okay, these outer gradients right here, especially this gradient right here, this is, how do I need to change theta? Now, okay, how do I need to change theta? This depends on these predictions right here. These predictions right here have one forward pass using theta, then have a gradient with respect to theta right here inside of them. And all of those come from this quantity, which is already a forward pass using theta. Is this actually how it's implemented in practice? Do you do stop gradient somewhere? Do you have any hacks? Or is this actually, because it seems mighty unstable, right? Does this actually work as you specify? Okay. Yeah, that's a good question. So in general, it depends. So if it was a single prediction, so if it was like the default, sometimes we've applied this kind of prediction time optimization, the day learning procedure to regular tasks like image classification, I think like this, it's not that unstable because you're just kind of doubling the computation graph because you make one prediction and then gradient step and then double that prediction. So that's fine. Now here you have two issues, the fact that you're taking the gradient step and the fact that you have many predictions that kind of build upon one upon the other. So that could get tricky. In practice, we've seen that if the overall training regime is stable, then it works fine. But if the overall thing is already unstable, then it's extremely tricky to add things there. So for instance, one thing we realized was that because video prediction is very expensive, and basically we couldn't fit that many examples on a GPU, literally, I think two or four. So we were initially using vice normalization. And so that was making the training, the vanilla training of the vanilla neural network. So just F already unstable. And when we were adding our another network improvement on top of it, it couldn't learn anything. We'd swap the batch normalization for layer normalization. Then the vanilla training was very, very stable. And then suddenly the neural networks worked out of the box. And we think that that's because the original gradients, because of the batch normalization, if you compute the batch statistic with a very small batch, it's already very crazy unstable. And then we couldn't learn. When the other thing is already stable, then it seems for us it worked pretty out of the box when we swapped the layer normalization. Okay, that sounds good. Yeah, I would expect so. Yeah. So for instance, I would expect, for instance, if we were to do 100 steps or many more steps, for instance, we were discussing before how there were two losses that sometimes we tried one or the other. The reason we came up with a second loss that conserves the conserved quantity between this time step and the next time step was when we were using batch normalization, we were wondering, oh, is our another network unstable? And then we realized, okay, no, it's the vanilla network that was unstable. But that was part of our concern, because there are some papers that mention that when you're backpropagating through a very deep graph, then the gradients are sometimes not very informative. In our case, we found that when the thing is pretty stable, it seems to work fine. But I could expect that if you make very, very long predictions or your thing is already unstable, then it only adds to the instability taking the second error. Yeah. Yeah. And another thing that struck me is that there is only, right, there's only one gradient step here. Mm-hmm. You take one gradient step and I'm going to, yeah, that might also be something where stability or computational graph size, first of all, you just do a gradient step. Many things would be possible, right? You could do an AdaGrad step, you could do an Adam step, you could do a line search or a Newton step or anything like this, but you have chosen to do the most simple thing, which is a single gradient step, right? I think the key word here is what you said about simple. We could have done anything else, but I think simplicity is something to value a lot in research, I feel. And so we went for the simplest thing. Yeah. And so one gradient step. And you can train with three gradient steps and we've sometimes done that. It's a bit better because this allows you to take smaller gradient steps and then sometimes you optimize the inner loss further, better. But in terms of one, simplicity, if it works with one, it's better. And two, especially when you present the algorithm in a paper, you really want to show the simplest version. And then usually people now know that, okay, if you can take one gradient step, you can usually take more than one gradient step and it will just make the computation graph larger, but that's fine. So we were striving for simplicity both when we were implementing and then when we were showing the algorithm. And you do have experiments that show that even though you learn with one gradient step, and that is down here somewhere, even though you learn with one gradient step, you can in fact, at inference time, then perform more than one gradient step. And that up to a sizable amount of steps, like up to a hundred steps or so here, will actually improve the outer loss. Right. Yes. Yes. We think that essentially the inner loss is kind of a projection loss, right? Because you keep saying, okay, why don't you make G happier and happier? And especially in the theory section, we go a bit about this, but essentially there is many futures you could have predicted. And some of them make G higher. Imagine it's only one quantity for now. Some of them will make G higher. Some of them will make G lower. And when you're forced to conserve G, all these futures say, okay, no, you should conserve G and therefore it's kind of projecting one dimension. And so in particular for conserved quantities, applying the same laws over and over, it's kind of stable because you will just keep going closer to these manifold of predictions that conserve G. Yep. So there's no, let's say, danger of overdoing. I mean, there's a little bit, but as I said, it hits after like a hundred steps, which is quite a bit, right? Given that you train with one. Yes. So eventually, especially because also these are neural networks, so it's not like it's a, for instance, if when we've tried with this with hard-coded losses in the previous data in paper and it's the true conserved quantity and the energy is truly conserved, then you can freely do that and it will keep going down. But because it's a neural network, then suddenly I think you're going outside, it's kind of a distribution shift. You train G to be useful for one or two or three grand steps. Now you're using it for a hundred. It doesn't make you any promises. Yep. That makes sense. Now, so I wanted to also come back a little bit to a more conceptual idea. Maybe this is also a question about tailoring in general, what you do here, that you essentially adjust the parameters of your forward predictor on the fly. There are many ways you could have combined the two networks, right? The one network that essentially predicts the conserved quantity and the other one that forward predicts. For example, you could have optimized the predictions themselves at runtime to make both of them happy. You could have, I don't know, you could have just learned it as one thing and not even bothered with runtime optimization. Why did you choose this tailoring approach in particular? It seems a bit cumbersome, right? And it's not maybe the first choice one would come up with. What are the advantages here? So there's two things in your question. Let me answer one after the other. So there is one, why the prediction time procedure, the runtime procedure. And then the other one is why adapt theta instead of X. So let me start why the runtime procedure. It goes back to what we were talking a bit like 10 minutes ago or so. The fact that the alternative to tailoring is auxiliary losses, which are, you could say, okay, we are going to learn an auxiliary loss that is going to be helpful for the final prediction. So there's two points here that I think could be improved. The first one is we are trying to learn an inductive bias. So for instance, one very cool thing about Hamiltonian neural networks or CNNs or transformers is that the inductive bias that they encode into the network applies at training time, but also applies at test time. So you know that you have equivariance at test time. And you know that your prediction satisfy these inductive bias. And so auxiliary losses, if you train for energy conservation or whatever loss you want, do not enforce, do not satisfy inductive bias. And so for it to be a proper inductive bias, it has to be satisfied also at test time. And that's why we optimize it at runtime. You also have to optimize it at training time, because if you optimize it only at test time, then you have a distribution shift. So that's why it has to be optimized inside the prediction function. So that's the first reason why to be a proper inductive bias, it has to be optimized at runtime. The second question, oh, sorry, and there's a second reason why we also do that instead of auxiliary losses. And the reason is that there is a very immediate signal. So imagine you encode energy conservation at training time, then it's a very loose signal to the final test prediction, because you're saying, okay, this is going to affect my final training parameters. And then I'm going to use my training parameters on a validation set. And this is going to lead me to good predictions. But this is only happens, you only can look at the effect at the very end of training, and then you're going to use that on validation. And so you could do that. And I think there's people that do that using implicit gradients. But the signal is much, much more cumbersome. And so you can use the implicit gradients, and then you can use the implicit gradients to optimize the signal. So the signal is much, much more cumbersome. Instead, if you use if you say, okay, no, the way I'm optimizing this is inside the prediction function, then you can literally compute the grain, the computation graph and optimize it. So that's the reason why we do that at runtime. Okay, second point in your question was why theta and not x. And that's a great very stark difference between both options in the previous in the tailoring paper. And we have a, we think we understand why the intuition is optimizing x actually helps. Experimentally, it makes sense that it helps. And it also empirically found that it helps. But it helps very little. The reason being that you can, it may find like an adversarial example on that optimizes G perfectly and makes G very happy with very small changes. If you optimize theta in that theta has kind of the geometry of the task, it knows the ways that it the ways to change the output condition on the input that kind of still do not deviate too much from what it has learned. So theta captures the dynamics and says, okay, I probably got it a bit wrong because I'm not conserving G. So but I don't want to deviate too much from what I've learned. So optimizing theta still make sure that you're satisfied what you've learned so far. And then it leads to much, much larger improvements. I mean, it does bring up like just right now, it does seem like might be possible to set up some adversarial setting right here where you could maybe use G as sort of a discriminator, not optimizing x directly, but sort of optimizing the parameters of F in maybe more of an adversarial setting. So not directly taking a gradient step with respect to the loss, but maybe saying, you know, is the is according to what G outputs, is this a real sample or is it a sample that I have predicted? Is this anything on your radar? Yeah, I think it's, I think there's something like what you said that that they're going to be there. In particular, I think G has a feeling like this adversarial discriminator because it's telling you, oh, if you're not satisfying G conservation, then most likely you are wrong, especially if you don't satisfy it by a large amount because again, they're approximately conserved. So that's one. So one thing I'm interested in going forward, and I think that that could be a venue for many future works, is that we focused a lot on when we were trying to make predictions on kind of generative networks. The fact that you're sorry, generative, not in the sense of self-supervised learning, but more in like you predict the next input, given the output, given the input, you have to generate the thing. G is like a checking network and checking sometimes is easier, right? You just have to say, stand back and say, okay, I like it, I don't like it. And that may be much easier to do. And also the type of network that you have that you build in may be very different architecturally, maybe the type of networks that we want to encode and construct may be architecturally different from the F networks. And maybe combining these proposal networks with these checking networks may make different architecture classes that could be useful. Yeah, I wanted to get a little bit more into... So you have experimental results where you compare to various baselines, like, you know, without... And obviously, obviously you're better than them, which is what we've come to expect from machine learning papers. I want to focus a little bit on also here you have an investigation into what the conservation, what the embedding network, this G network actually looks at. Do you maybe want to comment on this a little bit and why this makes you a little... Why this makes you comfortable, say, like comparing this to conserving quantities and why your assumptions might be correct? Yeah. So we were able to check the fact that we were learning conserved quantities in two ways. One, the symbolic experiment on the physics based, we were able to recover energies, but in the video, it's very hard to know, are you learning anything meaningful? And so we were able, okay, let's inspect what the G network is looking at. One thing here, just to be precise, is that we have to... It's a dynamical system, so we have to have some notion of velocity. So G was actually taking two consecutive frames to be able to have any chance of visualizing the velocity. But here, okay, we only look at one of the frames and we say, okay, where is it looking at? And if it's not looking at this reasonable stuff, then maybe it's not doing anything. And so if you look at the Nodder loss, it's an MSC of multiple dimensions. In our case, we tried... That hyperparameter didn't really matter experimentally. I'll come back to this a bit later. But let's say we fixed it to 64, so it was predicting 64 numbers. But if you think about it, you can rotate and exchange the dimensions and whatnot. So really what matters only is the PCA of this. So you can take the PCA and look at what's the most important dimensions and then the least important. And we found that even though we were trying to conserve 64 different numbers, in practice, there were only four to six that mattered. And in particular, the first one mattered a lot. 84% of the variance was captured by the first dimension. So it's the one on the left. And it was comforting to see that this dimension was looking at the right stuff. So in particular, it looks primarily at the object that's falling down. You can see it in red. And then we also saw that it was often looking at the edge. We think that this is because there were two types of... Here, they're both right to left, but there were sometimes sequences that the object was falling left to right. So we think that the edge of the ramp was a good signal on measuring this. And it also looks very faintly, but it also looks a bit at the object waiting to be hit. So that was very comforting to see. So you can see, for instance, other dimensions that were much less important than the first one, they are not very meaningful at all. And then the fourth one and the sixth one do have some meaning. We think that the fourth one was carrying more about four-inch type stuff. And we think that maybe it's because there was sometimes a hand that was going on there. We don't know. And the sixth one, we found that it was following blue objects very closely. So here, of course, we only show one example over time. So this is a time sequence as we track the object. On the appendix, we show that it basically didn't matter. The example didn't matter. It reproduced very nicely. And that also gave us confidence that the G network was learning something meaningful. Cool. So I have this question. You have a lot of these physics examples, right? Which also comes close to your notion of in physical systems, in dynamical systems, there are these conserved quantities and so on. Is it fair to say that probably in most video prediction tasks, unless it's like, I don't know, a SpongeBob video where every four seconds there is a cut, in most video prediction tasks, I can reasonably say if a model just observes the pixel information, then probably it's going to find some of these conserved things. It's almost like a prior on stuff over time moves slowly and in according to physical reality or something like this. Yeah, exactly. I think there's probably some type of prior like this that enforcing the fact that some things are approximately conserved is going to be useful beyond physics. It's true that we've because of the motivation, especially we thought that that's the most likely thing to work. And also the message was clear, but we think that possibly in other types of videos, well, even many videos are essentially everything is physics. If you're in the real world, cars or people moving around, but they also have some intrinsic movement that doesn't follow passive physics laws. But there's always something in mind, except cuts between scenes. Yeah, that cut you'll get goodbye. Do you have anything other? Is there a prominent example where this type of model would fail? Fail. So I think, I mean, I was thinking maybe, yes, I know. One easy example of something that would fail is you have a video and you often have things that enter the video that were not in the video. Then here you get into trouble because there's something that was not observed. It's the same thing that we were talking energy dissipation before. If you consider the entire system, then maybe there's something that's going to get conserved. You consider heat and whatnot. But anything that you cannot observe then enforces some things that are not getting conserved. So yeah, extra objects that appear and disappear, then you're going to get into trouble. Yeah, I was like going to mention the exact same thing. And I mean, it's still going to be the case that the G network, it can just output something like, well, the energy of the entire universe is still the same, right? But that then ceases to be useful. Yes, exactly. So yeah, things and one other thing I think conversely, it could be that there's a lot of work that will need to be done if the camera is moving a lot, because then all of these objects will for sure appear that were not there because you're looking at stuff that was not there. So if you look at the videos, this video is a static, the camera is static, sorry, the scene is not static. But so most likely some work will need to be done in this case. One good thing about this is that we're not fully imposing the conservation. So some approximately, actually the fact that it's approximate allows us to handle things that were not previously possible before, but still you will get into trouble if you keep entering stuff. But it's, I mean, just out of intuition, it seems more likely that the network detects something like, there's a blue bunch of pixels and an orange bunch of pixels, and these pixels sort of move together as objects rather than the network from video somehow determining, aha, there's laws of physics and there's gravity and there's friction and there's sliding. The first situation seems a bit more likely here, right? Yes, yes. Actually, so just to give a bit of context of how we came up with this idea. Initially, the original tailoring paper, we initially came up with applications on adversarial examples and contrastive learning. And I had the feeling that it could be applied to inductive devices, but I was not fully sure. I didn't know exactly how. And then Ross DeDrake gave a talk at MIT, it's online on the YouTube EI seminar. And he was telling us how it's very hard to encode inductive devices in neural networks. And in their case, basically they were predicting how a robot was pushing a bunch of carrot, and the carrot was moving around and they trained a carrot predictor. And it worked fine, very good prediction, but then they used it for planning a test time and suddenly it was not conserving carrot. It was making carrot disappear instead of bringing it to the proper place. And they were like, okay, neural networks don't work, so we're going to use a constrained linear model. And they were going to solve the problem this way. But I was like, okay, maybe we can actually, if we enforced it inside the prediction function, it would conserve carrot. And then that was the motivation that led us going to this direction. Cool. Is there anything else you want to say about the experimental results? We touched on sort of upping the inner steps and the grad chem, but is there anything special you want to say about sort of your tests on, for example, the pendulums or... Yeah, I think some of the experiments, it depends on how much time we have, but on the pendulum there was a symbolic component, so the G doesn't have to be fully neural. So it's the first experiment. The G is kind of a program with some parameter, like a formula. And there we search over formulas because it's a state information, the pendulum that you draw, like the angle and the momentum. And there we search over formulas, and then there's some parameters as well that get trained over with gradient descent. And there we saw that, okay, we are able to recover the true formulas of the energy, and then we can use the data to recover the true formulas of the energy, and it leads to better prediction than a vanilla MLP that does not learn about conservations. And there also you can see that actually you can even handle these approximate constraints where you have real data, which then the networks that have the hard-coded constraints can't handle as well. Yeah, exactly. So there is a cool paper, Hamiltonian Neural Networks, that encodes, I think the graph is a bit above, I think, that basically... Yeah, here, this one, perfect. So it's a very cool paper that they construct the network in such a way that it conserves the energy. And so we thought it was a very good comparison because it improves a lot above a vanilla MLP that does not conserve energy. So if you look on the right, this is changing HNN conserve quantity, which is what they believe is... They predict it's going to be some of the energy. You can see the baseline neural network, which is just the F basically, just F, quickly loses energy. And therefore, this is going to lead to much worse predictions. On the left, you can see the MSC goes up. If you fully impose energy, well, this is a much better inductive bias, the fact that energy is conserved. And you can see that the predictions are much better. But if you only softly encode it, then we show that we can do much better. And then we compare to actually knowing the loss, the formula for the energy. And we see that essentially the performance is pretty much the same. We are able to discover it and then use it to softly encode energy conservation. Nice. Seems like a good deal. I mean, it's really cool that if you know something about your problem, this is sort of another way that you can directly encode that even in sort of a soft way. I think the softness is something super useful, especially in the real world, compared to sort of the really hard constraints that often these asymmetry conserving neural networks have. Yeah, yeah, exactly. Cool. Yeah, I think this is about it for this paper. Is there anything you want to... You have a theoretical section. We didn't talk much about the symbolic regression, but I think we've gotten sort of to the essence. Is there anything else you want to add to this or anything people should know that your code is online? Yeah, the code is online. So it can be easily built upon. It's on with PyTorch, but I think actually JAX will make it this type of things of parameter, a kind of this tailoring process that essentially you have a parameter per example with JAX are very... It's very, very easy to encode and parallelize, so that will also make it easier. But with PyTorch, it's already pretty easy to the... With PyTorch higher, it's very easy to implement. So I think that should be easy to build up. I just wanted to point out that this was a group effort. So in particular, Dylan Doblar was also a co-first author in this work and did a lot of the experiments. And then we also had Alan Cho and Chelsea Finn from Stanford collaborating on this work because we found they had a really cool paper on learning discrete symmetries, meta-learning symmetries by reparameterization. And then we also had Professor Josh Tenenbaum from MIT cognitive science and Kenji Kawaguchi from the University of Singapore. Cool. Excellent. Well, Ferran, thank you so much for being here with us today. And all the best. I hope you have great, great ideas in the future. Thank you.
 But the intuition is that knowing these five conserved quantities is going to tell me a bit
 about what my prediction should be. And so it's kind of free information that I get to know.
 Hello there! Today we'll look at Nöter Networks Meta-Learning Useful Conserved Quantities by
 Ferran Oled and Dylan Doblar and others. This is another one of the with the authors installations
 videos, whatever, where I just discuss the paper briefly right now and then we'll jump into an
 interview with one of the first authors, with Ferran, and we'll go through the paper together.
 And I think Ferran can explain this so much better than I can. And I'm also able to ask some of my
 dumb questions. So this was a lot of fun and I definitely invite you to stick around. If you
 already know a little bit what the paper is about, feel free to skip ahead. If you don't know what
 the paper is about, the paper essentially deals with neural networks that predict dynamical systems.
 And in these dynamical systems, very often there are these conserved quantities that are
 part of it. For example, in a physical system, energy is conserved, momentum is conserved,
 and things like this. And under this constraint, you can build in this constraint into the
 predictive neural network so that the neural network does a better job. And they build these
 neuter networks in order to dynamically learn these conserved quantities, and then adjust at runtime
 during forward propagation, tailor the loss to conserve these quantities. And I think that's
 really cool. It's different. And yeah, that's what I like about it. So pretty brief introduction,
 this paper obviously is named after Neuter's theorem, which essentially they say here loosely
 states the following. For every continuous symmetry property of a dynamical system,
 there is a corresponding quantity whose value is conserved in time. For example, they say a system
 of planets interacting via gravity, the system is translation invariant in all three cardinal
 directions. Neuter's theorem asserts that there must be a conserved quantity for each of these
 symmetries. In this case, linear momentum is conserved. So the symmetry in space as translations
 is accompanied by a conserved quantity, which is linear momentum. Now, we don't always obviously
 know these quantities. And they're not always super explicit. And they're not always exact.
 So what we are going to be dealing with here is predictions of dynamical systems. And the example
 here is the prediction of a video of like a physical interaction. So this is a thing here
 on an inclined plane, it sort of slides down, and then collides with this other thing right here.
 And the goal is to predict the next frames of this video. Now, we could just build a neural
 network to just to predict these things frame by frame by frame. And that would go certainly well,
 if we had a lot of data. However, if we don't have a lot of data, what we need to do is we need to
 build in inductive biases. And inductive biases, what people usually do is they build in these
 symmetries directly, for example, they build in the physical laws, they know how the world works.
 And they say, you know, whether I translated to the left or to the right, it doesn't really matter,
 and so on. But building in these symmetries, and I think we know this from geometric deep learning,
 building in these symmetries is very powerful, but it can also be cumbersome, because you have to
 define them beforehand. This paper goes ahead and says, you know, what's real, what's a lot easier
 than building in symmetries directly is building in a constraint to conserve a given quantity.
 And that is a lot easier. And there's a potential that you can actually learn it from data. And with
 Noether's theorem, we know that the two things are equivalent. So if a system conserves a quantity,
 it essentially encodes a symmetry in the system. So what do we do? This is the very high level
 overview over these networks, we take to this entire thing here is one forward propagation,
 we take the original frame, we put it through a forward predicting neural network, which is this
 f theta right here. This is a network that simply forward predicts frames as we I said initially.
 So we forward predict forward predict forward predict, this gives us an initial set of of
 outputs right here, these x tilde, now these are going to be pretty, pretty bad, not pretty bad.
 But if we don't have a lot of data to learn from these, we don't expect them to be particularly
 good. And that's the regime we are here. What we do then is we're trying to adjust this f thing
 right here. In the moment, so during the forward propagation, we're going to update our predicting
 neural network by this neutral loss. So we're going to do an update, a temporary update to the weights
 of the f network. And we're going to do this into direction of this neutral loss. So you can see here,
 we have these networks G lying around, and G is always the same network. So what we're going to do
 is we're going to feed each frame that we predicted through G. And G always being the same network,
 it will output the same thing. And now obviously, if you know, given given that how I made this
 introduction, you might already have guessed that G is the part that predicts the quantity to be
 preserved. So what we want to do is we want to put all these things through G. And then we want to
 these these will give us a bunch of outputs, right? G here and here and here and here will output
 some things and the things can either be a number or an entire vector, right, an embedding vector.
 So essentially, G takes this thing right here, actually takes two consecutive frames, and embeds
 it into some space. And now, ideally, all these G's would output the same thing, which would mean
 which would mean that we have conserved some quantity and therefore encoded some symmetry.
 However, initially, these G's are not going to output the same thing. So we are going to
 attempt to change the F function such that the G's output more the same thing, there is a loss
 involved right here. This is the neutral loss, they call it, and it is defined down here. So you can
 see all this really is, is it's either defined in one of two ways. Either you take the difference
 between the G function of the initial frame and the frame at time point t, or and you calculate
 the difference, or you calculate the difference between consecutive frames. In either way, since
 you sum across all the frames, this means that all the outputs of the G network will should
 approximately be the same. Now, what do you do with this information? Again, we're still
 we're still during one forward propagation. So what do you do with this information, you calculate
 this neutral loss, which is one we just described, and then sorry for skipping around so much,
 you're going to do one update step. So these are the parameters of the F network, we're going to
 do one update step into the direction of the gradient. And it's the direction of the gradient
 with respect to the parameters of the F network. So this is the forward predicting network. So
 essentially, we are saying, how do I need to update my forward predicting network, such that,
 right, such that the frames that it outputs, the frames that it predicts in the future,
 make it such that the G functions of all of these frames are more similar to each other,
 or more similar to the G function of that first frame. So we're going to in time update the F
 function right here. And after that, we're going to forward propagate again, with this new F
 function, and thereby obtain our final prediction. This is one, this is like an inner optimization
 that we do during forward propagation. I find this to be pretty cool. Now they just do they just do
 one gradient step, obviously. Otherwise, you know, you could do a lot of things and you could like
 program in Adam and Ada grad, not only one like gradient step, which is one SGD step, essentially.
 But even with one step, that is good enough. So again, they here is the entire training procedure
 in an algorithm, you can see that. Let's start down here, they start with randomly initialized
 weights, these weights here are for the G network, these weights are for the F network, they sample
 batches for each batch, they predict the sequence. Now the sequence prediction is this entire thing
 we just looked at. So the sequence prediction is, I'm going to start at the initial frames,
 I'm going to use the F, the original F, the one I currently have, unconditional, let's say to forward
 predict all of the frames once, then I'm going to put all of these predictions here into this
 neutral loss, I'm going to calculate the gradient, how do I need to update this F for this particular
 data point to make the G functions output, the more similar things, I'm going to attain new
 parameters, again, these are just temporary parameters, I'm going to use these temporary
 parameters here to do another round of forward prediction, which gives me my final estimate,
 I could probably repeat this again. And or I could do multiple steps right here, I could probably do
 a lot of things, but this is sort of the simplest case. And then I will return these, what do I do
 with them? You can see right here, this is my output. Now I'm going to input these things into
 what's called the task loss. And the task loss in our case here is just the video prediction loss.
 So that's going to be some L2 distance between the frames, the output and the frames that actually,
 so that these are the output frames, these are the frames that are actually in the video. And then
 I'm going to just run back prop on that. So update the parameters of both G and F on the task loss.
 So what does it mean? G is going to be updated such that if I do this whole sequence again,
 if I do the whole sequence of predicting, then tailoring my loss to G, right, I tailor my loss
 to the G function, G is going to be updated such that next time, if I tailor my loss to it,
 it's going to lead to a better outcome overall. And F is going to be updated. Similarly,
 it's going to be updated such that, well, next time, if I do this whole procedure of first
 predicting these, which I'm going to use the parameters, then updating the parameters,
 and then updating the parameters using G, and then predicting again, I update my F such that
 this whole procedure will result in a better loss. Now, I think this is the magic of our back
 propagation frameworks that we can even think of these types of things, because, I mean, behold,
 actually writing this down and implementing the backwards pass here yourself, that'd be crazy.
 So this is the entire algorithm right here. Now, again, given that there are, as you can see,
 some hyperparameters here, such as the learning rates, they only do one gradient step, as we
 mentioned. So this isn't an exact enforcement of that constraint, right? This is only an
 approximate enforcement. Essentially, the only additional constraint that we introduce here
 is this requirement that the G function is the same G function on all the forward predicted things.
 And that is our knowledge that we are dealing with a dynamical system. And in this dynamical system,
 some quantities should be preserved. The way we build the losses means that G can simply output
 a constant value, otherwise, it would not be useful to the loss, right? But also the way we
 build the loss means that it is not an exact constraint, like we would build this into the
 architecture that a quantity must be conserved. So it's able to deal with real world data, such as
 this video where even sometimes a hand may come in, there's friction and so on. It's not an exactly
 conserving system, right? And the way we do this in the moment in the forward pass update using this
 neutral loss, that means that I can now tailor whatever like I can tailor the inductive bias
 for this particular sample. So I can learn it's kind of meta learning thing, right? What I learn
 is how to in the moment, adjust my loss function to this particular sample of data. Now, as I said,
 obviously, if you had more data and all, maybe you wouldn't need this, but it does help a lot
 in their experiments in the in these regimes where you do not have a lot of data, they have a
 theoretical section right here, where they have a reduced case and show that it can be useful
 to impose these constraints, then they have a bunch of experimental settings, among other things,
 they also they don't only do what I just said with the video prediction, but they also do a
 prediction where they don't not everything is a neural network. So where the things they predict
 are actual physical quantities, and they do it using symbolic regression. And this is the same
 method except it's not neural networks, it's symbolic regression. And what that does is,
 it comes up with these equations, for example, for the ideal pendulum, as you can see,
 these equations are insanely close, like they recover the correct equations. And these are
 symbolic regressions. So the it's not you don't you didn't only have to come up with the number
 right here, you actually, the network had to come up not the network, the system had to come up with
 the entire equation, given some basic building blocks of variables, and you can square stuff,
 and you can take the cosine of stuff. So these experiments show that the method can indeed
 recover physical quantities that are conserved if you present them with a scenario where this is
 the case, and they use either ideal scenarios, so ideal data generation, but they also use real
 world data from pendulums, where obviously you have energy dissipating, and then you can,
 you can compare. So here, I believe they do compare with what they say is a baseline. So
 as that predicts into the future, the longer prediction they do, the worse that gets. Or,
 I guess the losses over here, you can see that. But then also, the Hamiltonian neural networks,
 which enforce exact constraints, they enforce the quantities to be preserved exactly.
 If you face them with real world data, you can see right here, the quantities aren't changed at all,
 yet the loss still goes up because the quantity isn't actually conserved in the real data. And
 the neural networks do follow the ground truth data much more closely, because they can model
 also in exact constraints and not super strict enforcement of these constraints, which is what
 I think we need in real world data. They do have a bunch of other experiments, especially as I said,
 also video prediction where they do outperform various baselines, they investigate where the
 network pays attention to and whether or not you can actually move or do a lot more inner iteration
 steps than just one, because we just did one inner iteration steps there, there is no reason why this
 should remain at one. And here they show that even though they only trained with one at inference
 time, they can actually take a bunch more and the outer loss will still go down. So this all validates
 a little bit of the reasoning behind the method. Yeah, I don't want to take up too much of your time
 right here because I want to jump into the interview. Let me know what you think of these
 more interviewee style paper reviews. I quite enjoyed the interview. And I do think it's pretty
 useful to have the authors there because they can correct me pretty instantly. All right, see you over
 there. Okay, cool. Hi, everyone. Today I have with me Ferran Aled, who is one of the primary authors
 of the Nöter Networks paper and here to discuss with us probably a little bit about the intrinsics
 of the paper. And maybe also for me personally, because the paper is very technical, it's very
 technical. It's a new field for me as well, connecting physics to machine learning, building
 all of this into neural networks. There's also a bit of symbolic regression in there. So I feel a
 lot of things are coming together here. I found the paper pretty cool and it's new and that's
 what's interesting. So Ferran, thank you very much for being here. Yeah, thanks for the invitation.
 Wonderful to be here. Thanks. So your paper deals with, do you call it Nöter Networks,
 how do you pronounce? I pronounce it Nöter Networks, but I think I'm not German,
 so I'm not sure I'm pronouncing it properly. I'm not a German either, but I think that
 the author was called Nöter. Yeah, so you're pronouncing it more properly than I am.
 Maybe. But essentially, could you give us maybe just first an insight, where does the name,
 because the name is kind of distinct, right? Because there is the Nöter Theorem. What does
 the Nöter Theorem say in general? Yeah, so the Nöter Theorem was kind of the inspiration for
 our work. And the intuition is that for every symmetry of a dynamical system, there is a certain
 conservation law that's going to apply to that system. So for instance, imagine you have a
 planetary system of planets moving around. The physics laws don't change from today to tomorrow.
 That means that there's a time symmetry of the system. And here, Nöter's theorem tells you, oh,
 if there is a symmetry here, that means that there must be a quantity that's conserved
 over time. And in this case, for time symmetry, there is energy that's being conserved. So we
 use that as a motivation, not that the technical details, more like the higher level message of
 the theorem, to build a new machine learning model. And the intuition is that in machine learning,
 symmetries are one of the core ways in which we've improved data efficiency and model performance.
 And so it would be very cool if we could kind of automatically learn some of these symmetries.
 But symmetries are kind of hard to quantify and get a hold of computationally. And the intuition
 is that they talk about kind of counterfactuals and kind of global in the sense that when I was
 telling you about this time symmetry, I was saying, if I were to look at the planetary system tomorrow,
 the laws of physics would be the same. But I don't have access to the data for tomorrow. It's a kind
 of counterfactual. So the model cannot handle this. Instead, conserved quantities can be directly
 measured. I can check, oh, this quantity, which I will call energy, is being conserved on my actual
 data. And that makes it very easy to quantify. Yeah, we've heard in, I think in the recent past,
 even a lot of people attempting to get more out of symmetries out of neural network with I'm thinking
 of, I'm thinking of like, group convolutional neural networks, and so on that try to actively
 build in symmetries into neural networks. But it seems like they can only do that in situations
 where they know the symmetry that will appear, they already know a molecule doesn't matter which
 way I look at it, right, so I can directly build that in. But your reasoning is that because
 assessing conserved quantities is an easier task than assessing symmetries, it might be possible
 to learn the conserved quantities dynamically actually learn them from data. Is that approximately
 correct? Yes, exactly. Exactly. So and the theorem is the motivation because it tells us that
 conserved quantities are kind of on the same level of powerful as symmetries for dynamical systems,
 in particular, if you're doing image classification that does not apply because image classification
 is not a dynamical system. But that's the intuition. Yes. And you even have some slack in there you
 discuss, you know, we can, we, it doesn't even have to be absolutely conserved quantity, it doesn't
 have to be an absolute symmetry that we deal with. By learning it from data, we can even handle
 approximate symmetries. Is that right? That's another thing that may be a bit different from
 our work than other works, which is that some symmetries are only approximately conserved or
 conserved quantities are only approximately conserved. So for instance, you have if you have a
 dissipative system, like in the real world restriction, and so you actually lose energy,
 you don't consider if you don't consider the entire system, you're usually have small losses.
 So in this case, you would say you would like to say, oh, energy is conserved, but not quite. So
 it's fine if you if your prediction doesn't fully conserve energy. But knowing about energy
 conservation maybe helps you with the overall prediction. And maybe I want to want to get to
 sort of a little bit of an example of where so people can imagine this a little bit more. Now,
 I only have a mouse here because I forgot the iPad because I'm stupid. But maybe we can give
 the small example of a pendulum, right? So here's a pendulum, it hangs here, and it sort of gets down
 here. And here's the little ball. And the pendulum is accurately described by I think the angle
 right here that it's sort of off the off the main axis, and also its momentum, let's say it swings
 in this direction with a certain with a certain speed. And this describes the pendulum. Now your
 model focuses on predicting the future, let's say, or at least from from what I can tell. So
 what your model would be able to do is it would be able to predict the next time step right here,
 right? Then it's a bit here, here. Sorry, it's a little bit more up to the left, right? So it's a
 little bit more up and then it's it's even more up over here and then it swings back and so on it
 swings back over. Now, can you explain to us what are sort of the what is the symmetry here? And
 what are the conserved quantities? Yeah, so in this case, for the pendulum, we know that if we
 were to swing the pendulum now and 10 minutes from now, the physics wouldn't change. And so we know
 that there's a time symmetry. And so in this case, we would say, oh, there's a time symmetry and then
 another theorem would would tell us, oh, energy is conserved. So in this case, energy is a mixture
 of the kinetic energy, which is how much movement there is, and more movement, the more energy,
 and potential energy, which in this case is because of gravity. So a combination of these
 must be conserved. We don't know exactly how which formula and that's what we're going to
 automatically discover. I see. And the original approach, I think, would just be that here,
 this arrow, I parameterize this with some neural network, right? I just say, you know, here,
 I plug in neural network, I predict the next time step, and the next time step, and the next time
 step, and that it will maybe work, right? But it will, let's say, will only implicitly make use,
 it will not actually make use of the fact that something is conserved. So you go ahead and you
 say, since this is a dynamical system, we know more about the system, we can impose additional
 constraints. And the additional constraints right here, if I see this correctly, essentially, at
 every time step, you say, I want to build a neural network that's always going to be the same neural
 network that takes a state, let's say the pendulum in this state, and predicts a quantity, let's call
 that, no, G is the name of the network, let's call the quantity, I don't know, alpha. And I want to
 use that same neural network in all the different states that I find this thing in. And it always
 needs to predict the same thing, right? Since it needs to figure out a quantity that is conserved.
 And now it is, if I just train a neural network to always predict the same number right here,
 I would just end up with a neural network that is predicting some kind of a constant, right?
 Yeah. So your method figures out how do I need to build, first of all, this predictive neural
 network to predict this conserved quantity, such that it actually predicts something useful. But
 then also, how do I make this network right here actually use the fact that this other network
 predicts common quantities, right? Yeah, exactly. So that's why the word useful in our title,
 because there is many conserved quantities that are kind of not useful. And so we want to find
 those that are helpful for loss, final loss. So in machine learning, we usually care about
 some performance, whatever it is. And so that's exactly what we, that our objective just cares
 about that. And the useful quantities are just a proxy and intermediate thing for getting us to
 better performance. Yeah. And so here you have this main diagram, I think that that would be
 considered the main diagram describing your method. And this is on a task that is a video
 prediction task. And it's about sliding something down an incline. Could you maybe describe what
 the task here is? The frames are a bit low resolution. So this is the physics 101 data set
 from Josh Tenenbaum's group. I think Jesun was the first author. And they have a collection of
 videos. And in this case, they have a hand dropping an object passively, like it just lets it drop
 down and the object falls down. And there's a second object at the end of the ramp, they collide.
 And then the other one, sometimes depending on the masses and the friction and whatnot,
 the dynamics are kind of can change. That's the data set. And does, so that there are multiple
 videos and it's always different objects or? Like some objects could be common between videos,
 but there's lots of objects. So it's not always the same object. And that's kind of the point,
 the fact that it can vary. So one nice thing about the other networks is that they can deal with
 raw video. So some usually conserved quantities, you get them from kind of state data. Like when
 I was telling you, when we were talking about the pendulum, it's kind of, you have the exact
 position of the pendulum, you have the momentum of the pendulum, you don't have a pixel video of the
 pendulum. And here, because we deal with neural networks that predict the conserved quantities,
 you can hopefully get conserved quantities from video. Yeah. So here, the diagram shows a little
 bit of what you're, what you are trying to do, but also what you're trying to avoid. So the bottom
 path right here, if I see this correctly, that would be if I did nothing else, except the bottom
 path, I would build this neural network to just predict sort of the future time steps. And that
 often turns out poorly. I don't know, this is a quite a pixel-ish mess, but it's sort of, it's
 sort of, all of a sudden, there are like three objects instead of two, and the one is kind of
 gone or split up. And it's a bit of a mess. And you attribute this to the fact that it's just a video
 prediction or? Yeah, well, in this case, to analyze it and to make the problem challenging, we made
 the, like there was very few data. In general, you can, it's all like symmetries and inductive
 biases are going to be most useful when the problem is hard and then there is like less data. So in
 this case, there was a few ones of videos and also because video prediction is pretty long. So at the
 very few, like at the beginning of the frames, like the first few frames, there was not that
 much mistakes. But when you go very far into the future, then it's much harder. So those two
 problems, lack of data and the fact that you go a lot into the future. Your method is, and you also
 have an algorithm described somewhere. It's a bit of a, it's a algorithm that is, oh, right here.
 It's an algorithm that has multiple steps in it. And one special part is that you have this sort of
 inner optimization loop right here. Now, I want to maybe go back to the diagram and let's go, let's
 walk through it once before we, before we, you know, take a look at the formulas and all we can
 walk through it once. So the first thing that happens, if I understand correctly is you take
 your first input and you do exactly what we just said, you run it through a forward prediction
 neural network that just tries to predict the future, just plain by itself. Right. So this has,
 this has a bit of a, of a default thing, but now you try to improve that. And this is all,
 this is the entire thing we're describing right now. That is one forward pass through your system.
 So you would take every single prediction that you made and you would feed it through this
 G network right here. And this G network is, you call it an embedding network. That is the thing
 ultimately that's trying to predict a conserved quantity. But it's not, it's not necessarily just
 outputting one number. It's outputting an entire vector. So it's an outputting and embedding
 vector. And the, the goal obviously is that for all of these inputs, it should output the same
 embedding vector. But so, ah, so, but this is, this is going to be, let's say trained such that
 across the dataset, it works well. So maybe, you know, for this video sequence, it's going to
 predict approximately the vector A for all the frames if it works well. And for another sequence
 with two different objects that obviously have a different total energy or so, it might predict
 a different embedding vector. Exactly. But all the same across the, across the video sequence. Okay.
 So this is how we can imagine you train this G network to sort of predict whatever is special
 about this particular data point, but inside of the data point conserved among all the frames.
 Exactly. Because if it was the same A for everyone, then you would have the issue that you mentioned
 at the beginning, then it's a useless conserved quantity. Yeah. So it's, it's almost like a bit
 of a description of the scene as such, right? That makes the video predictors life easier
 if you have sort of this, this global description. Yeah. Yeah. So the intuition, I think is, let's
 think about when the, if, if the network G was very good at predicting the conserved quantities
 and perfectly told you, oh, these five quantities, I know for certain that they're going to be
 conserved. Then we could, we will see the next step. We haven't gone through it yet, but the
 intuition is that knowing these five conserved quantities is going to tell me a bit about what
 my prediction should be. And so it's kind of free information that I get to know about constraints.
 So it's kind of an unsupervised loss that I have access at test time. Yeah. It restricts, it restricts
 what you can output, right? Because ideally the F network should only output whatever the G network
 says is, is the same, right? If the F network can only output things that the G network will embed
 to the same place in the embedding space or a similar place. Yes. There's just to be a hundred
 percent precise. There is lots of images that could make the network G happy because it only
 constrains like a few dimensions, but it has to make the network G say, oh, this is approximately
 what you had at the beginning. Yeah. Okay. And so that comes in in the next step. So here, what you
 do, you use, you take the input again and you route it through this F network again, but now this F
 network doesn't, is not like a free form predictor, but it actually takes, has somehow the notion
 of, of this information that the G network output out of the initial sequence again. And you do this
 in a very special way in that you actually take the parameters of F and you update them on the fly.
 Yes. You update them on the, so this is within a forward pass. You actually update the parameters
 into the direction of the gradient of G. Exactly. Yes. So, yeah, sorry. This is,
 I think that that it takes it. Yeah. So here you have this neutral loss. Yes, exactly. Which do you
 maybe want to talk about this briefly? Yes. So about another loss. Yeah, sure. So the other
 loss essentially is telling you, you should have, you should conserve G. So the, you know, for a
 fact that, so there's two ways of conserving G. They're roughly equivalent. If you fully impose
 them, if you don't fully impose them, they're not equivalent. That's why we put the approximate
 sign. So let's look at the term A here. It's basically saying, oh, you should conserve G.
 And so it should be, all of them should be equal to what G was telling you for the input X naught.
 So if you make the embedding of your prediction, note that X of T has kind of a tilde on top of
 it. So your prediction for XT should have the same conserved quantities as your input. And that's
 what your first term is. And just an MSC over this neural embedding. The second one is very similar.
 Sometimes it's a bit more useful, more stable, because instead of, if instead of comparing to
 the very beginning, you compare to the previous time step, you have a more immediate signal.
 And you basically say you should conserve it. Every time you apply F, you should conserve G.
 So that's the other basically important observation. And now we update theta and theta are the
 theta are the parameters of F, right? Theta are the parameters of F. We update these on the fly.
 And I suppose that we just do this in the moment. And for the next data point, we go back to the
 original parameters and do this again. So this is sort of an on the fly update for a temporary
 update of these parameters into the direction of this quantity right here. So this is the gradient
 of exactly the loss that we just discussed with respect to the parameters of F. So essentially,
 it says, what parameters would make F more apt at fulfilling this loss, which essentially means that
 these which how do we need to change F such that these forward predictions make the G
 conservation happier? Exactly. Exactly. So this is some previous work of ours, which we call
 tailoring. And the idea of tailoring is just because of what you said, that the fact that
 the adaptation is customized for each individual data point. And the idea there was a general way
 of encoding inductive biases with unsupervised auxiliary losses. So auxiliary losses in general,
 you say, for instance, one thing we could say is, oh, why not we add energy conservation when we
 train? Sometimes auxiliary losses would say, okay, I train for good predictions and I train
 for energy conservation at training time. But if you do that, you're not going to
 enforce energy conservation at test time. Because at test time, you're going to have a
 generalization gap in energy conservation. But energy conservation or any type of conservation
 or any auxiliary loss can be checked before making the prediction at test time or at training time.
 Inside the prediction function, I can first make my prediction and see, okay, do I like it? Does my
 auxiliary loss, does my unsupervised loss like this prediction? And if not, I can take a gradient
 step or multiple gradient steps to improve my unsupervised loss, in this case, the conservation
 loss. And so this makes it much better for the particular point we care about, which is the one
 we are making a prediction for. It's a bit surprising because it's a single data point.
 And maybe you have trained with a million data points. So the question is, why does one data
 point matter if we've trained with one million data points? Well, the idea is that you're training
 on the exact point you care about. So enforcing inductive bias in the exact point you care about
 right now for which you're making the prediction is going to have a very big impact. And so in this
 case, this gradient step improves the prediction just for that one point. Yeah, maybe it's also
 important to highlight that the parameter here, this theta that we start with, and also the
 parameters of G, those are the ones that will be learned during the training procedure across the
 entire training data set. And then the parameters here, those are always constructed in the moment,
 data point by data point, to, as you say, tailor the inductive bias. And the inductive bias,
 in this case, would sort of be this entire term right here, essentially says, how do I need to
 change my predictor in order to conserve the particular thing that G decides is the common
 quantity for this data point? Yeah. And this gives rise to the algorithm. So here is what we just
 discussed. This is the forward prediction sequence with this inner optimization step. So we first
 predict this plane sequence, then we temporarily update the parameters. And that allows us to again
 do the forward pass, but now with the updated F function, and that gives us sort of our final
 predictions. And as you can see here, during the training, we sample always batches, we forward
 predict using this inner update, and then we take outer gradients. And the L task here, that would
 just be what you call the task loss. This would be the video prediction loss or something like this.
 Okay. So I have a lot of questions. First of all, this, it seems quite intricate, right? Because if
 I think, okay, these outer gradients right here, especially this gradient right here, this is,
 how do I need to change theta? Now, okay, how do I need to change theta? This depends on these
 predictions right here. These predictions right here have one forward pass using theta, then
 have a gradient with respect to theta right here inside of them. And all of those come from this
 quantity, which is already a forward pass using theta. Is this actually how it's implemented in
 practice? Do you do stop gradient somewhere? Do you have any hacks? Or is this actually,
 because it seems mighty unstable, right? Does this actually work as you specify?
 Okay. Yeah, that's a good question. So in general, it depends. So if it was a single prediction,
 so if it was like the default, sometimes we've applied this kind of prediction time optimization,
 the day learning procedure to regular tasks like image classification, I think like this,
 it's not that unstable because you're just kind of doubling the computation graph because you
 make one prediction and then gradient step and then double that prediction. So that's fine.
 Now here you have two issues, the fact that you're taking the gradient step and the fact that you
 have many predictions that kind of build upon one upon the other. So that could get tricky.
 In practice, we've seen that if the overall training regime is stable, then it works fine.
 But if the overall thing is already unstable, then it's extremely tricky to add things there.
 So for instance, one thing we realized was that because video prediction is very expensive,
 and basically we couldn't fit that many examples on a GPU, literally, I think two or four.
 So we were initially using vice normalization. And so that was making the training, the vanilla
 training of the vanilla neural network. So just F already unstable. And when we were adding our
 another network improvement on top of it, it couldn't learn anything. We'd swap the batch
 normalization for layer normalization. Then the vanilla training was very, very stable. And then
 suddenly the neural networks worked out of the box. And we think that that's because
 the original gradients, because of the batch normalization, if you compute the batch statistic
 with a very small batch, it's already very crazy unstable. And then we couldn't learn.
 When the other thing is already stable, then it seems for us it worked pretty out of the box
 when we swapped the layer normalization. Okay, that sounds good. Yeah, I would expect so.
 Yeah. So for instance, I would expect, for instance, if we were to do 100 steps or many more steps,
 for instance, we were discussing before how there were two losses that sometimes we tried one or
 the other. The reason we came up with a second loss that conserves the conserved quantity between
 this time step and the next time step was when we were using batch normalization, we were wondering,
 oh, is our another network unstable? And then we realized, okay, no, it's the vanilla network
 that was unstable. But that was part of our concern, because there are some papers that
 mention that when you're backpropagating through a very deep graph, then the gradients are sometimes
 not very informative. In our case, we found that when the thing is pretty stable, it seems to work
 fine. But I could expect that if you make very, very long predictions or your thing is already
 unstable, then it only adds to the instability taking the second error. Yeah. Yeah. And another
 thing that struck me is that there is only, right, there's only one gradient step here.
 Mm-hmm. You take one gradient step and I'm going to, yeah, that might also be something where
 stability or computational graph size, first of all, you just do a gradient step. Many things
 would be possible, right? You could do an AdaGrad step, you could do an Adam step, you could do
 a line search or a Newton step or anything like this, but you have chosen to do the most simple
 thing, which is a single gradient step, right? I think the key word here is what you said about
 simple. We could have done anything else, but I think simplicity is something to value a lot in
 research, I feel. And so we went for the simplest thing. Yeah. And so one gradient step. And you can
 train with three gradient steps and we've sometimes done that. It's a bit better because this allows
 you to take smaller gradient steps and then sometimes you optimize the inner loss further,
 better. But in terms of one, simplicity, if it works with one, it's better. And two, especially
 when you present the algorithm in a paper, you really want to show the simplest version. And then
 usually people now know that, okay, if you can take one gradient step, you can usually take more
 than one gradient step and it will just make the computation graph larger, but that's fine. So we
 were striving for simplicity both when we were implementing and then when we were showing the
 algorithm. And you do have experiments that show that even though you learn with one gradient step,
 and that is down here somewhere, even though you learn with one gradient step, you can in fact,
 at inference time, then perform more than one gradient step. And that up to a sizable amount
 of steps, like up to a hundred steps or so here, will actually improve the outer loss.
 Right. Yes. Yes. We think that essentially the inner loss is kind of a projection loss, right?
 Because you keep saying, okay, why don't you make G happier and happier? And especially in the theory
 section, we go a bit about this, but essentially there is many futures you could have predicted.
 And some of them make G higher. Imagine it's only one quantity for now. Some of them will make G
 higher. Some of them will make G lower. And when you're forced to conserve G, all these futures say,
 okay, no, you should conserve G and therefore it's kind of projecting one dimension. And so
 in particular for conserved quantities, applying the same laws over and over, it's kind of stable
 because you will just keep going closer to these manifold of predictions that conserve G.
 Yep. So there's no, let's say, danger of overdoing. I mean, there's a little bit,
 but as I said, it hits after like a hundred steps, which is quite a bit, right? Given that you train
 with one. Yes. So eventually, especially because also these are neural networks, so it's not like
 it's a, for instance, if when we've tried with this with hard-coded losses in the previous
 data in paper and it's the true conserved quantity and the energy is truly conserved,
 then you can freely do that and it will keep going down. But because it's a neural network,
 then suddenly I think you're going outside, it's kind of a distribution shift. You train G to be
 useful for one or two or three grand steps. Now you're using it for a hundred. It doesn't make you
 any promises. Yep. That makes sense. Now, so I wanted to also come back a little bit to a more
 conceptual idea. Maybe this is also a question about tailoring in general, what you do here,
 that you essentially adjust the parameters of your forward predictor on the fly. There are
 many ways you could have combined the two networks, right? The one network that essentially
 predicts the conserved quantity and the other one that forward predicts. For example, you could have
 optimized the predictions themselves at runtime to make both of them happy. You could have,
 I don't know, you could have just learned it as one thing and not even bothered with runtime
 optimization. Why did you choose this tailoring approach in particular? It seems a bit cumbersome,
 right? And it's not maybe the first choice one would come up with. What are the advantages here?
 So there's two things in your question. Let me answer one after the other. So there is one,
 why the prediction time procedure, the runtime procedure. And then the other one is why adapt
 theta instead of X. So let me start why the runtime procedure. It goes back to what we were
 talking a bit like 10 minutes ago or so. The fact that the alternative to tailoring is auxiliary
 losses, which are, you could say, okay, we are going to learn an auxiliary loss that is going
 to be helpful for the final prediction. So there's two points here that I think could be improved.
 The first one is we are trying to learn an inductive bias. So for instance, one very cool
 thing about Hamiltonian neural networks or CNNs or transformers is that the inductive bias that they
 encode into the network applies at training time, but also applies at test time. So you know that
 you have equivariance at test time. And you know that your prediction satisfy these inductive bias.
 And so auxiliary losses, if you train for energy conservation or whatever loss you want, do not
 enforce, do not satisfy inductive bias. And so for it to be a proper inductive bias, it has to be
 satisfied also at test time. And that's why we optimize it at runtime. You also have to optimize
 it at training time, because if you optimize it only at test time, then you have a distribution
 shift. So that's why it has to be optimized inside the prediction function. So that's the first
 reason why to be a proper inductive bias, it has to be optimized at runtime. The second question,
 oh, sorry, and there's a second reason why we also do that instead of auxiliary losses.
 And the reason is that there is a very immediate signal. So imagine you encode energy conservation
 at training time, then it's a very loose signal to the final test prediction, because
 you're saying, okay, this is going to affect my final training parameters. And then I'm going to
 use my training parameters on a validation set. And this is going to lead me to good predictions.
 But this is only happens, you only can look at the effect at the very end of training, and then
 you're going to use that on validation. And so you could do that. And I think there's people that do
 that using implicit gradients. But the signal is much, much more cumbersome. And so you can use
 the implicit gradients, and then you can use the implicit gradients to optimize the signal.
 So the signal is much, much more cumbersome. Instead, if you use if you say, okay, no,
 the way I'm optimizing this is inside the prediction function, then you can literally
 compute the grain, the computation graph and optimize it. So that's the reason why we do that
 at runtime. Okay, second point in your question was why theta and not x. And that's a great
 very stark difference between both options in the previous in the tailoring paper. And we have a,
 we think we understand why the intuition is optimizing x actually helps. Experimentally,
 it makes sense that it helps. And it also empirically found that it helps. But it helps
 very little. The reason being that you can, it may find like an adversarial example on that
 optimizes G perfectly and makes G very happy with very small changes. If you optimize theta in
 that theta has kind of the geometry of the task, it knows the ways that it the ways to change the
 output condition on the input that kind of still do not deviate too much from what it has learned.
 So theta captures the dynamics and says, okay, I probably got it a bit wrong because I'm not
 conserving G. So but I don't want to deviate too much from what I've learned. So optimizing theta
 still make sure that you're satisfied what you've learned so far. And then it leads to much, much
 larger improvements. I mean, it does bring up like just right now, it does seem like might be
 possible to set up some adversarial setting right here where you could maybe use G as sort of a
 discriminator, not optimizing x directly, but sort of optimizing the parameters of F in maybe more
 of an adversarial setting. So not directly taking a gradient step with respect to the loss, but maybe
 saying, you know, is the is according to what G outputs, is this a real sample or is it a sample
 that I have predicted? Is this anything on your radar? Yeah, I think it's, I think there's
 something like what you said that that they're going to be there. In particular, I think G has
 a feeling like this adversarial discriminator because it's telling you, oh, if you're not
 satisfying G conservation, then most likely you are wrong, especially if you don't satisfy it by a
 large amount because again, they're approximately conserved. So that's one. So one thing I'm
 interested in going forward, and I think that that could be a venue for many future works,
 is that we focused a lot on when we were trying to make predictions on kind of generative networks.
 The fact that you're sorry, generative, not in the sense of self-supervised learning,
 but more in like you predict the next input, given the output, given the input, you have to
 generate the thing. G is like a checking network and checking sometimes is easier, right? You just
 have to say, stand back and say, okay, I like it, I don't like it. And that may be much easier to do.
 And also the type of network that you have that you build in may be very different architecturally,
 maybe the type of networks that we want to encode and construct may be architecturally different
 from the F networks. And maybe combining these proposal networks with these checking networks
 may make different architecture classes that could be useful.
 Yeah, I wanted to get a little bit more into... So you have experimental results where you compare
 to various baselines, like, you know, without... And obviously, obviously you're better than them,
 which is what we've come to expect from machine learning papers. I want to focus a little bit
 on also here you have an investigation into what the conservation, what the embedding network,
 this G network actually looks at. Do you maybe want to comment on this a little bit and why this
 makes you a little... Why this makes you comfortable, say, like comparing this to conserving
 quantities and why your assumptions might be correct? Yeah. So we were able to check the fact
 that we were learning conserved quantities in two ways. One, the symbolic experiment
 on the physics based, we were able to recover energies, but in the video, it's very hard to know,
 are you learning anything meaningful? And so we were able, okay, let's inspect what the G network
 is looking at. One thing here, just to be precise, is that we have to... It's a dynamical system,
 so we have to have some notion of velocity. So G was actually taking two consecutive frames
 to be able to have any chance of visualizing the velocity. But here, okay, we only look at one of
 the frames and we say, okay, where is it looking at? And if it's not looking at this reasonable stuff,
 then maybe it's not doing anything. And so if you look at the Nodder loss, it's an MSC
 of multiple dimensions. In our case, we tried... That hyperparameter didn't really matter
 experimentally. I'll come back to this a bit later. But let's say we fixed it to 64,
 so it was predicting 64 numbers. But if you think about it, you can rotate and exchange the
 dimensions and whatnot. So really what matters only is the PCA of this. So you can take the PCA
 and look at what's the most important dimensions and then the least important. And we found that
 even though we were trying to conserve 64 different numbers, in practice, there were only four to six
 that mattered. And in particular, the first one mattered a lot. 84% of the variance was captured
 by the first dimension. So it's the one on the left. And it was comforting to see that
 this dimension was looking at the right stuff. So in particular, it looks primarily at the object
 that's falling down. You can see it in red. And then we also saw that it was often looking at the
 edge. We think that this is because there were two types of... Here, they're both right to left,
 but there were sometimes sequences that the object was falling left to right. So we think that the
 edge of the ramp was a good signal on measuring this. And it also looks very faintly, but it also
 looks a bit at the object waiting to be hit. So that was very comforting to see. So you can see,
 for instance, other dimensions that were much less important than the first one, they are not
 very meaningful at all. And then the fourth one and the sixth one do have some meaning.
 We think that the fourth one was carrying more about four-inch type stuff. And we think that
 maybe it's because there was sometimes a hand that was going on there. We don't know. And the sixth
 one, we found that it was following blue objects very closely. So here, of course, we only show
 one example over time. So this is a time sequence as we track the object. On the appendix, we show
 that it basically didn't matter. The example didn't matter. It reproduced very nicely. And that also
 gave us confidence that the G network was learning something meaningful. Cool. So I have this question.
 You have a lot of these physics examples, right? Which also comes close to your notion of
 in physical systems, in dynamical systems, there are these conserved quantities and so on.
 Is it fair to say that probably in most video prediction tasks, unless it's like,
 I don't know, a SpongeBob video where every four seconds there is a cut, in most video prediction
 tasks, I can reasonably say if a model just observes the pixel information, then probably
 it's going to find some of these conserved things. It's almost like a prior on stuff over time
 moves slowly and in according to physical reality or something like this.
 Yeah, exactly. I think there's probably some type of prior like this that enforcing the fact that
 some things are approximately conserved is going to be useful beyond physics. It's true that we've
 because of the motivation, especially we thought that that's the most likely thing to work. And
 also the message was clear, but we think that possibly in other types of videos, well, even
 many videos are essentially everything is physics. If you're in the real world,
 cars or people moving around, but they also have some intrinsic movement that doesn't follow
 passive physics laws. But there's always something in mind, except cuts between scenes.
 Yeah, that cut you'll get goodbye.
 Do you have anything other? Is there a prominent example where this type of model would fail?
 Fail. So I think, I mean, I was thinking maybe, yes, I know. One easy example of something that
 would fail is you have a video and you often have things that enter the video that were not in the
 video. Then here you get into trouble because there's something that was not observed. It's
 the same thing that we were talking energy dissipation before. If you consider the entire
 system, then maybe there's something that's going to get conserved. You consider heat and whatnot.
 But anything that you cannot observe then enforces some things that are not getting
 conserved. So yeah, extra objects that appear and disappear, then you're going to get into trouble.
 Yeah, I was like going to mention the exact same thing. And I mean, it's still going to be the
 case that the G network, it can just output something like, well, the energy of the entire
 universe is still the same, right? But that then ceases to be useful.
 Yes, exactly. So yeah, things and one other thing I think conversely, it could be that
 there's a lot of work that will need to be done if the camera is moving a lot, because then all of
 these objects will for sure appear that were not there because you're looking at stuff that was not
 there. So if you look at the videos, this video is a static, the camera is static, sorry, the scene is
 not static. But so most likely some work will need to be done in this case. One good thing about this
 is that we're not fully imposing the conservation. So some approximately, actually the fact that it's
 approximate allows us to handle things that were not previously possible before, but still you will
 get into trouble if you keep entering stuff. But it's, I mean, just out of intuition, it seems
 more likely that the network detects something like, there's a blue bunch of pixels and an
 orange bunch of pixels, and these pixels sort of move together as objects rather than the network
 from video somehow determining, aha, there's laws of physics and there's gravity and there's
 friction and there's sliding. The first situation seems a bit more likely here, right?
 Yes, yes. Actually, so just to give a bit of context of how we came up with this idea.
 Initially, the original tailoring paper, we initially came up with applications on
 adversarial examples and contrastive learning. And I had the feeling that it could be applied
 to inductive devices, but I was not fully sure. I didn't know exactly how. And then
 Ross DeDrake gave a talk at MIT, it's online on the YouTube EI seminar. And he was telling us how
 it's very hard to encode inductive devices in neural networks. And in their case, basically
 they were predicting how a robot was pushing a bunch of carrot, and the carrot was moving around
 and they trained a carrot predictor. And it worked fine, very good prediction, but then they used it
 for planning a test time and suddenly it was not conserving carrot. It was making carrot disappear
 instead of bringing it to the proper place. And they were like, okay, neural networks don't work,
 so we're going to use a constrained linear model. And they were going to solve the problem this way.
 But I was like, okay, maybe we can actually, if we enforced it inside the prediction function,
 it would conserve carrot. And then that was the motivation that led us going to this direction.
 Cool. Is there anything else you want to say about the experimental results? We touched on
 sort of upping the inner steps and the grad chem, but is there anything special you want to say about
 sort of your tests on, for example, the pendulums or...
 Yeah, I think some of the experiments, it depends on how much time we have, but on the
 pendulum there was a symbolic component, so the G doesn't have to be fully neural. So it's
 the first experiment. The G is kind of a program with some parameter, like a formula. And there we
 search over formulas because it's a state information, the pendulum that you draw,
 like the angle and the momentum. And there we search over formulas, and then there's some
 parameters as well that get trained over with gradient descent. And there we saw that, okay,
 we are able to recover the true formulas of the energy, and then we can use the
 data to recover the true formulas of the energy, and it leads to better prediction than a vanilla
 MLP that does not learn about conservations. And there also you can see that actually you
 can even handle these approximate constraints where you have real data, which then the networks
 that have the hard-coded constraints can't handle as well. Yeah, exactly. So there is a
 cool paper, Hamiltonian Neural Networks, that encodes, I think the graph is a bit above, I think,
 that basically... Yeah, here, this one, perfect. So it's a very cool paper that they construct
 the network in such a way that it conserves the energy. And so we thought it was a very good
 comparison because it improves a lot above a vanilla MLP that does not conserve energy. So
 if you look on the right, this is changing HNN conserve quantity, which is what they
 believe is... They predict it's going to be some of the energy. You can see the baseline neural
 network, which is just the F basically, just F, quickly loses energy. And therefore, this is
 going to lead to much worse predictions. On the left, you can see the MSC goes up. If you fully
 impose energy, well, this is a much better inductive bias, the fact that energy is conserved.
 And you can see that the predictions are much better. But if you only softly encode it, then
 we show that we can do much better. And then we compare to actually knowing the loss, the formula
 for the energy. And we see that essentially the performance is pretty much the same. We are able
 to discover it and then use it to softly encode energy conservation. Nice. Seems like a good deal.
 I mean, it's really cool that if you know something about your problem, this is sort of
 another way that you can directly encode that even in sort of a soft way. I think the softness
 is something super useful, especially in the real world, compared to sort of the really hard
 constraints that often these asymmetry conserving neural networks have. Yeah, yeah, exactly.
 Cool. Yeah, I think this is about it for this paper. Is there anything you want to... You have
 a theoretical section. We didn't talk much about the symbolic regression, but I think we've gotten
 sort of to the essence. Is there anything else you want to add to this or anything people should know
 that your code is online? Yeah, the code is online. So it can be easily built upon. It's on with PyTorch,
 but I think actually JAX will make it this type of things of parameter, a kind of this tailoring
 process that essentially you have a parameter per example with JAX are very... It's very, very easy
 to encode and parallelize, so that will also make it easier. But with PyTorch, it's already pretty
 easy to the... With PyTorch higher, it's very easy to implement. So I think that should be
 easy to build up. I just wanted to point out that this was a group effort. So in particular, Dylan
 Doblar was also a co-first author in this work and did a lot of the experiments. And then we also had
 Alan Cho and Chelsea Finn from Stanford collaborating on this work because we found
 they had a really cool paper on learning discrete symmetries, meta-learning symmetries
 by reparameterization. And then we also had Professor Josh Tenenbaum from MIT cognitive
 science and Kenji Kawaguchi from the University of Singapore. Cool. Excellent. Well, Ferran,
 thank you so much for being here with us today. And all the best. I hope you have great,
 great ideas in the future. Thank you.