Fourier Neural Operator for Parametric Partial Differential Equations (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper berkeley purdue mc hammer mchammer mit technology review pde partial differential equation navier stokes darcy flow burgers convolutions fft dfft fourier transform fourier neural operator neural operator fast fourier transform fourier modes flow turbulent flow fluid dynamics residual aerodynamics wind tunnel neural network layers numerical discretization
#ai #research #engineering

Numerical solvers for Partial Differential Equations are notoriously slow. They need to evolve their state by tiny steps in order to stay accurate, and they need to repeat this for each new problem. Neural Fourier Operators, the architecture proposed in this paper, can evolve a PDE in time by a single forward pass, and do so for an entire family of PDEs, as long as the training set covers them well. By performing crucial operations only in Fourier Space, this new architecture is also independent of the discretization or sampling of the underlying signal and has the potential to speed up many scientific applications.

OUTLINE:
0:00 - Intro & Overview
6:15 - Navier Stokes Problem Statement
11:00 - Formal Problem Definition
15:00 - Neural Operator
31:30 - Fourier Neural Operator
48:15 - Experimental Examples
50:35 - Code Walkthrough
1:01:00 - Summary & Conclusion

Paper: https://arxiv.org/abs/2010.08895
Blog: https://zongyi-li.github.io/blog/2020/fourier-pde/
Code: https://github.com/zongyi-li/fourier_neural_operator/blob/master/fourier_3d.py
MIT Technology Review: https://www.technologyreview.com/2020/10/30/1011435/ai-fourier-neural-network-cracks-navier-stokes-and-partial-differential-equations/

Abstract:
The classical development of neural networks has primarily focused on learning mappings between finite-dimensional Euclidean spaces. Recently, this has been generalized to neural operators that learn mappings between function spaces. For partial differential equations (PDEs), neural operators directly learn the mapping from any functional parametric dependence to the solution. Thus, they learn an entire family of PDEs, in contrast to classical methods which solve one instance of the equation. In this work, we formulate a new neural operator by parameterizing the integral kernel directly in Fourier space, allowing for an expressive and efficient architecture. We perform experiments on Burgers' equation, Darcy flow, and the Navier-Stokes equation (including the turbulent regime). Our Fourier neural operator shows state-of-the-art performance compared to existing neural network methodologies and it is up to three orders of magnitude faster compared to traditional PDE solvers.

Authors: Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 AI has cracked a key mathematical puzzle for understanding our world. Just in from MIT technology review and look at this puzzle right here. It's got the bumps, it's got the valleys, the surfaces, it's got the braille, it's got the bits, the ones and the zeros, not only going up and down like in the matrix, but going in circles. It's got it all. This puzzle is really hard as you can see and AI has just cracked it. I'm being a bit hyperbolic of course. This is actually about a new paper that can solve, numerically solve a particular type of partial differential equations way faster than anything before it. So this is about this new paper and we'll get into the paper in a second. It's pretty cool, but as you can see MC Hammer, the infamous MC Hammer has tweeted this out and he has actually a pretty cool Twitter feed where he regularly tweets about scientific papers and so on. So pretty cool cross-domain overlap. I recommend that. So we'll get into the paper, we'll get into the code a little bit as well because I think it helps to understand what's going on. I want to start out by, this is the blog post by one of the authors and it's pretty good to get a basic overview of the paper and here is the motivational example. So the motivational example is the Navier-Stokes equation, which is an equation in fluid dynamics. So you're trying to predict how a fluid evolves over time given a certain parameters like its viscosity and a forcing function. So basically how sticky it is and how hard you stir it and then you want to know how it evolves over time. You can see on the left is given an initial condition and I think on the right is sort of a rollout after the 10th time step until the 50th time step. And the ground truth is obtained with a sort of classic numerical solver where you do little time steps and you calculate the interactions and then this takes a lot of time and compute. And on the right is the prediction of this new Fourier neural operator that this paper develops. And you can see it's almost equal and the gist of it is that the thing on the right simply takes one forward propagation through a neural network. So it takes like 0.00 something of a second to compute the thing on the right, whereas the thing on the left is quite hard to compute and as I understand can take minutes. So here you see the motivational example. These things are described by partial differential equations, which are sort of linearized ways of describing how the system evolves over one time step. And it'd be cool if we could solve these faster because this is applications in aerodynamics and other types of engineering fields. All right, so let's jump into the paper. As always, if you like content like this, consider sharing it out, telling your friends about it and subscribing, of course. So the paper is called Fourier Neural Operator for Parametric Partial Differential Equations. And it's by Tsong Yi Li, Nikola Kovatsky, Kamjar Aziza Deneshelli, Borygede Liu, Kaushik Patacharya, Andrew Stewart and Anima Anankumar of Caltech and Purdue University. So I feel the paper is both very cool and a bit overhyped. So we're going to see what it does. It's for a particular type of PDEs. And it has a lot of, let's say, engineering choices that make it possible to solve with neural networks, but also that limit its applicability to where the classical methods would be applicable where this thing isn't. So there are tradeoffs definitely to reach the sort of speed up that they reach. But we'll get into this. First, I actually want to scroll down right here all the way because there is something that you don't often see in the sort of machine learning field. And that is here in the acknowledgments section. And I just find it interesting. Don't regard this as anyone. But here we are supported by the LWLL grants, which I understand is DARPA. Beyond Limits, which is like a makes soft or makes AI or systems for things like gas and oil and so on with British Petroleum as a main sponsor. Raytheon, which of course is a giant military manufacturer. We have the Army Research Laboratory and so on. So you can see that this is kind of, I don't know, I don't see this often. This is sort of a good bouquet of sponsorships. Of course, there's also Microsoft, Google, and so on. Yeah, but it's just interesting to see that the Army is pretty heavily into these things. And of course they would be. I mean, rockets need to fly and they need to be aerodynamic and so on. So yeah, I'm not saying this is bad or good. I just thought it was interesting that Raytheon would be a sponsor of this. All right, so let's dive in. As we said, we're interested in these types of problems right here, where you have this thing called... So there is this quantity called the vorticity, which as I understand is a derivation of the viscosity. So it sort of tells you how the fluid is moving right now. And so this state right here. And then you apply a sort of constant forcing function and you want to know how that evolves over time. So you can see at time step 15, you get sort of this picture. So these move past each other and see this moves here, this moves here. And then at time step 20, you can see they are fairly moved. This blue thing moves in here as well. And they just sort of mix. And there are certain parameters that make the fluid more sticky or not so sticky. And the interesting regimes is, I guess, when it's not very sticky, so not too sticky, but also not sticky enough. And then these really complicated patterns occur. And to predict them would be very, very valuable. So you want something that takes in this initial state right here and outputs all of these these future states. And usually this is done by these classical numerical solvers. So the Navier-Stokes equation is described by a set of partial differential equations. And you can see this down here. So Navier-Stokes equation is described by this set of equations right here. Is there? Yep. And you can see that the that this this is fairly complex. It includes partial derivatives, gradients, and so on. So this is the this is this vorticity, and it includes that on on both sides. And this is this the yeah, this is two derivatives, maybe. Or is it just the delta? I don't even know. I'm not an expert in partial differential equations by any means. So anything coming from that direction, don't take me for granted. I'm going to give you sort of the under the thing of what I understand from this paper. And so with respect to that entire area, I'm not an expert, I just can understand that this is fairly complex. And what you usually do is you take the initial state and you just evolve it in time. So you take this time parameter, and you do you go one little little time step, and then you calculate because these are all sort of linear linear equations, you calculate this one little time step into the future, you update your state, right? It's sort of like, you know, you have your points here and how they move, and how they move is given by their gradients. So these are all sort of linearized things. Now, you don't want to move them too much per time step, because ultimately, if this thing moves, and this thing moves, then the movement of this arrow will change because this thing over here moves, right? So you want to compute this one little time step into the future, like to here and this to here, and then you want to recompute all of these arrows. So maybe now that points a little bit more here, and that points a little bit more here. And then you want to update it again. So you have these sort of these these numerical solvers that go little tiny time step by little tiny time step, it's not even this if here if you see t equals 20 or something, it's not 20 time step for these solvers, but these usually go like 1000 or 100 steps per time step that is here, or something like this, they need to take very tiny steps to be accurate. And that takes a long time. So the idea is, can't we simply can't we simply simply input this, let's say this thing or or like something at time 15, and directly predict the thing at time 30. And that's exactly what this paper does. And a lot of papers have done this before, but without much success. So this paper proposes to do this in the Fourier domain, and we'll see the path that they take right there. So they go into the will shortly go into sort of the the basics right here. So what you want what you're looking for is a function G that takes an A and gives a U. So what are A and U, A and U are both function spaces. So A, A and U here are functions. So A is a function, as you can see, A is a function, and U is a function, but you can characterize them as data points. So in this in this way, there is a functions and data points are sort of interchangeable, you can see an image like this as a data point, where it's an image, but you can also see it as a function where every x and y coordinate is mapped to a value, right. So when when they talk about functions, very often they talk about this type of function, where you have x, y and t, so t is also t is zero here, x, so the function would x, y, t map that to some value, right here, the vorticity. And you want to transform this function. So this function would be A, A would be the function at time, let's say zero or something or the times zero to 15, you would want to map that to the function, the function U that also takes an x and the y, let's leave t out for the moment, also takes an x and the y and let's say t, but t is set to 30, and maps that to a vorticity, right. So you want to input a function and output a function, but it's the same as inputting an image and outputting an image in as for from an engineering perspective, of course, from a math perspective, it's a little bit different. But other than that, it's a fairly standard machine learning problem. So you have this, these sets A and U, and you're looking for this function, G that maps A to U. So we study maps, which maps G, which arises the solution operators of parametric PDEs. Suppose we have observations, where A is an IID sequence from probability measure mu, transported on I and U is the A transported by G, it is possibly corrupted with noise, we aim to build an approximation of G by constructing a parametric map. This G right here. So it's a bit of a mathy way of saying we have a bunch of data points where we were a this is the initial state goes to U, which is the state at some point in time. And we know that there is a function G, this is this G with this inverse cross, we know that there is a true function that maps any A to U. So a single function G that can if I input the initial state can give me the output state. And what I want to do is I want to approximate this by a parametric version. So these here are the parameters. And of course, as you can guess by now, G is going to be this G right here is going to be a neural network that is parameterized by theta. So these would be the layers of the neural network. And we're going to input A into the neural network, and we're going to get out U. So that's basically that there is quite a bit of math right here. And the math here is to derive what they call a neural operator. So here is one layer of this neural network. As we said, we're going to input A. Now A first thing that we do A is going to be, let's say up projected. So A is going to be made into a latent representation v zero. So this is let's call that here P. So there is a function P, which is going to be a little layer of neural network. And it is going to produce this v zero. So v zero is going to be a latent state of the neural network. And then there is going to be a number of these layers that transform this to v1, v2, v3. I think there are four layers of these in their particular implementation, but there don't need to be four layers. You can choose that, as you can choose any depth of neural network. And then at the end, you're going to project that down to whatever output you want. So U. So this function here is called Q. And these are just going to be neural networks. So P and Q are going to be your very, very classic up projections and down projections of data point. We'll get into sampling. Let's go actually right now. So one thing right here, and they stress this, is that they work in function space, right? They don't work on the, let's say they don't map the data point to the data point. What you could do is simply have like a convolutional neural network, an image to image network, and so on. But what is the problem with that? So if you have your A, which is your initial state, and it has these bunch of fluid things right here. And what you do when you have an image is you sample this, right? You sample this at different, sorry, maybe a regular grid. I am terrible at regular. So you sample this into a certain amount of pixels, and your neural network will operate on this, right? This will give you some kind of a tensor, which is, let's say we have a, so this is a seven by seven grid. Okay, so your neural network is going to expect this as an input dimension. And whatever U is, of course, so you map this to U, which is also going to be some sort of image, okay, where you need to output pixels. So again, you have some set resolution, and your neural network can only operate at that particular resolution. What they're doing right here is the cool thing about is it can operate at any resolution. So once you've learned the network, you can input higher resolution images, or you can output higher resolution images, any any sort of, you can deal with more resolution, less resolution sampled irregularly, you can deal with a lot of things once the neural network is their neural network is learned. And how do they do it? They do it by only ever acting point wise in the spatial domain. So what they're going to do is they're going to take this a, and now we get into the more critical things. So here, a and u aren't just the beginning state and the end state. In fact, in this Navier-Stokes example, a is a tensor like this. So a is going to be a tensor with slices, and each slice describes one time step up to a given time. So this here could be t equals zero. So there is kind of the initial distribution, and then t equals one and so on up until t equals like 10. Let's say I think they do 10. So they let this thing evolve for 10 time steps. And I'm going to guess they do it using one of these classical methods. And that's the input. So the input isn't just the initial state, the input is actually here is what happened in the first time 10 time steps. And then the output isn't just the output at some particular time, but the output is actually also a slice right here. Each slice here describes the output at a particular time. So this would be t equals 11 up until t equals 50. So this is u. So the top one is sort of the conceptual thing, but the bottom one is what really happens. So they input 10 time steps, and they get out the 40 subsequent time steps, they predict them all at once. So and now you can see that in this particular case, how I can understand this is at each pixel here, I want to know what what is that pixels value after what after like certain amount of time steps, okay, like 11 or 50 right here or 40. And of course, the result is going to not only depend on the time zero, but on the entire evolution of time zero to time 10. So this here is an entire column for that pixel. And this is akin to that particular pixel having this many channels. So here I can just say, well, these are technically 10 channels or 11 or something like this, I probably screwed up this should be t equals zero to nine, and then 10 to 49. But so this is this is an entire stack. This is we can interpret this as input channels right here. And we can interpret these as output channels. Okay, so ultimately, one pixel is going to have input channels, all the time steps that happened up until the point where we want to predict and the output channels are going to be at the same time all the time steps of what we want to predict. Okay, so these projections now coming back to this, they simply work in the channels. So these P and Q, they are one by one convolutions. And the one by one convolution simply up project and down project these features, you see, these are one by one convolutions. Actually they could be dense layers. Let's check that in the code later. But for sure, what they do is they only work point wise. So they don't they don't mix the individual pixels together. In here, you simply get at like a D by D grid with each has 10 channels. And then you simply up project that to so here you have D by D times 10. And then you up project that using P to D by D times and here is a parameter that you choose. So this is sort of your latent dimension. Okay. And you are going to transform this tensor keeping it in this D by D by W dimensionality until you back projected using Q to D by D by in this case, 40. Okay, so but this, this and this, they only work point wise. And that means there is no particular dependence on the D right here. So the next data point could actually have a different D as long as this pipeline right here can handle different dimensions, because the P and Q only act point wise, you're good. So what do what do these magic layers here do? So these are these Fourier neural operators, okay, they transform one hidden state into the next note that we have four of these layers. So they don't need to be the same as the number of time steps we're trying to predict, you see. And it's pretty clear from here. So we these four hidden layers, they're simply transforming this entire volume right here, this entire input volume, they are transforming this as a sequence of latent states, and then outputting this entire volume. So this down here has nothing to do with the time steps that we're trying to predict. It is simply a sequence of computations of latent computations. And you know, that in a neural network, the deeper you make it, the sort of more complicated functions arise. Even though of course, the universal approximation theorem says that with one hidden layer, you can do anything. But in general, if you have deeper neural networks, the more you can kind of make more complicated things. And so four seems to be a good number of complicated for these particular problems. So here's what one of these layers does. It is very much like a residual network. So here you have the the V is the hidden representation at t plus one and t plus one is not as I said, is not the time step in the in the Navier-Stokes sense of time evolution of the PDE. This is simply the layer t plus one. So I don't know why they maybe Yeah, maybe t here makes still makes sense. Is it not because it's large t? Yeah, so they have large t right here. Okay, maybe. But in the engineering sense, it is not. This is simply the layer. And you can see it's formulated as a function. But again, don't be like the x right here. This is simply the x and y and t coordinates. So this, this, all of this here can be represented as one big tensor x, y, t, or x, y channels or something like this. Okay, don't. So don't, don't be confused by the fact that these are formulated as functions. So what we want to do is we have two different things. So one neural, this is one neural network layer, as you can see, at the very end is a nonlinearity. This is a point wise nonlinearity. And this is in the original pixel space or in the original spatial space, the D by D space, each of the things gets a nonlinear function slapped on top, as is normal. Then this part is normal as well. This is simply a linear transformation of the input. Again, this is point wise. Okay, so this is a linear transformation. So so far, so good. We have a linear transformation of the input and a nonlinearity. The important part is this thing here. So what this thing is, this is a kernel function that depends on the initial condition. So not only on the last hidden state, but the initial condition and sort of is then applied by the last hidden representation, like like here, and then only x is applied. So notice the difference right here. This is at a point x, we're getting this function value, which means we're getting the entry of that tensor. And then we're applying the linear transformation. This makes it point wise. Here, first, we compute this function by this by applying this kernel to the input function, so to the entire input tensor, and only then we are looking for the particular entry. So that means this thing here is a point wise transformation of that tensor, while this thing here, it takes in the whole tensor and outputs a sort of new tensor. So this is going to be the magic. Here where k, it goes, you can see it goes from from u space to u space, maps to bounded linear operators on u, and is parameterized by theta, maybe what's this? I don't know. I never know. So the this this kernel, we choose this to be a kernel integral transformation parameterized by neural network. So they define the kernel integral operator as this. And you can see this is an integral over the D, D is the input space of u and a actually. So this is a function that's dependent not only on where you are in the tensor, but on the initial input this a, and then that's convolved. So this here is a, a integral over the entire space. So that's convolved with v, you can see that this is a convolution. And it's fairly complicated. So this alone tells you nothing. But luckily, they say that they restrict this. So it's a bit annoying when things always depend on this a, that means that each of these functions right here, each of these arrows right here, these are the neural operators, actually let's go here. Each of these Fourier neural operators right here. They would always also depend on this a here, like this, and like this, and like this. This is a bit annoying for deep learning, because we sort of want one layer's representation to go into the next one. So they simply make an engineering choice and say, nope, nope, nope. So they say, we impose, right, we impose. If we remove the dependence on the function a, we impose that the kernel is simply a function of x, not only x and w, but only x minus w. So now you have a sort of proper kernel function in there that we can handle. We obtain that four is a convolution operator. Okay, it wasn't a convolution before it was just an integral. But now if you restrict your kernel functions to this, you get a convolution, we exploit the fact in the following section by parameterizing k directly in Fourier space and using the fast Fourier transform to efficiently compute four. This leads to fast architecture, which abstains state of the art results for PDE problems. So there's quite a bit of math right here to finally arrive at this thing here. So what is all this math for? This math is for saying what we want, we want to build our neural network like this. And what we do is we simplify and specify this kernel thing until the kernel looks something like this. So we restrict the kernel to be a convolution. And since a convolution in Fourier space is just a multiplication, what we can do is instead of taking the function V and convolving it with this kernel, what we can do is we take the Fourier transform of the function V, then multiply it in Fourier space by this thing. And this thing is now simply a matrix that's learned in as a bunch of parameters. And then we do the inverse Fourier transform. Now you might ask why is this relevant? Why can't we just do a convolution like we do normally? And the reason is, so when you do a Fourier transform, what do you do? You have some kind of signal like... And so on. So you take this signal and you transform this into Fourier space. And here we just go like one vector. So here, as you know, in Fourier space, you have these basis functions, which are sort of these different parameterization of sine waves, or you can do it with cosine waves, and they get faster and faster, and so on. So you know that you can decompose any signal into its basis functions in this kind of periodic function space. So this function right here might have, you know, one times this function, plus 0.1 times this function, plus two times this function, minus five times this function, and so on. So you can describe any of that. Now for these type of PDEs that we're looking for, the special thing about them is they are fairly well described if you simply cut away the sort of top Fourier modes and only work with these because they are, you know, sort of the individual tiny ripples you might not want to take into account. So you can truncate the lower Fourier modes, and that's what they do exactly here. And they learn. So instead of transforming this signal directly into the next hidden representation, they go to Fourier space, cut the top Fourier modes. They have a way of making the next representation in Fourier space. And this is this r here. And that is simply a weight matrix that they multiply with. And that is, you can prove that that is the same as convolving in the original space. So multiplying in Fourier space is the same as convolving in the original space. And so they multiply the green numbers right here by r. Then you get something out. So I should maybe, this is way too much. So the green numbers you multiply by r to obtain new green numbers. So maybe r is the, is 2, 2, 4. So the new green numbers would be 2, 0.4. Then you do the inverse Fourier transform. So you get back to a signal. Now with 2 times this, so it might be bigger. And 0.4 times, so I can't even draw, but you sort of get the idea. You put it into Fourier space. You apply the function r, which is a multiplying by a matrix that you learn in Fourier space. You get new Fourier coefficients, you map them back. And there you have your next layers representation. Almost. Okay. So this is this Fourier neural operator and is described right here. What you do is you take your representation, your hidden representation, put it through a Fourier transform, which you can do in a differentiable fashion. You get these Fourier modes, which describes how to decompose the signal into these periodic functions. You take away the top modes, which is your sort of regularization. You apply r, which is in a dense layer of neural, not even that. It's a multiplication, okay, by a weight matrix. And then you obtain this, these new Fourier modes. You do the inverse, and then you have the next representation. Almost. What you do is we saw this before, a point wise transformation in the original pixel space. So this is very much like a residual network, right? Residual networks, they also have this. They have the implemented as one by one convolutions. So and then at the end, you apply the non linearity. What is good about this? Two things. First of all, throwing away the top Fourier modes is very advantageous to these types of problems that we have right here. You can see that the little jiggles right here, they will be sort of sorted out by the larger scale movements of the fluid. So throwing away the top modes is a sort of a regularization. It helps with generalization. And it's very easy in Fourier space. So these things other than natural images are described well by these Fourier spaces. And that, again, is an engineering choice. So you cannot not apply these things to everything. You can apply them to where this type of assumption holds. Second of all, this is now fully independent of the discretization of the input. Okay? Because when I take a picture and I sample it in a three by three, I can do a Fourier transform and I'll get all of these numbers right here. Okay? It's just, you know, the Fourier transform does a good job as possible. When I sample it in a seven by seven grid, like I sample it super densely, I do the same for transform, I get the same numbers right here. Okay? And it's not exactly the same. So they always claim it's the same. It's not exactly the same, of course, if you don't sample densely enough, your Fourier transform isn't going to be as accurate, let's say. So ideally, you want the Fourier transform of the real signal or the real underlying signal. But since you sample this, you can't have this. So there is a bit of a difference, but it is independent. So that's true. The function R that you learn simply operates on these Fourier modes. And these are fairly independent of how regularly you sample, of course, more regular, better, but still fairly independent. Yeah, so that's good. So if you have what they're going to do is they're going to have something like the three by three during training and then sample more densely during during inference, which is something you can do but understand that this is just it's just a form of interpolation, right? So the inverse Fourier transform simply gives you whatever you want interpolating using the Fourier modes it has. And of course, given a certain number of Fourier modes, which is quite small for them, I think it's something like eight or 12 higher resolution at some point doesn't help you anymore, because you've cut off the high resolution Fourier modes, I guess what can help you is this, this thing right here. But this thing right here only acts point wise. So you see, this is now fully independent of the discretization of the signal, which is a cool thing. So the two cool things about this entire stuff is that first of all, independent of discretization, second of all, these types of problems that we are having here, lend themselves very well to be described in Fourier space. Yeah, so that's why I'm saying this is for a particular type of problem. And also, there are a bunch of other things you can see right here. You have this entire input tensor right here, and this entire output tensor right here. And these can be fairly large, right, and all the intermediate representations have to be kind of at D by D by W. So this is, you can't go infinite time right here, like you could with a classic solver, like a numerical solver, all you need is the last time step, right, you go, what's the t equals one, then at t equals 1.1, 1.2, and so on, you just count up and you just go always from the last time step to the next time step here. Since it's in neural network, during training, you need to keep all of these tensors, the intermediate things, I guess you can do gradient checkpointing, but this is engineering wise, you predict all the future time steps at the same time. So you can't really go infinite in time. And how do you train this thing? You train it by simply giving it one of these A, right, you have a bunch of A's, so you have a bunch of these input tensors, a data set. And where you always say here is a one of these Navier-Stokes equation, sorry, type of problems, I've sampled it somehow, and I've let it run for 10 time steps. And then I've let it run for longer, u, so I let it run for longer. And here are time steps of this t equals zero to t equals nine or 10, let's go 10. And here is t equals 11 to t equals 50. So you have a data set, and this data set is fully computed by a classic forward solver. So you can't replace the forward solvers right yet, because you need them for generating training data, right? So this becomes your training data, this becomes generally your x and this becomes your y. And now you're learning this neural network, this entire thing to give you x to y. So you see, you still need the classic solvers to produce the training data. That's the first thing. The second thing is, you can pretty clearly see that the good thing is that now we can input any a so the classic solvers, you need to rerun them for each initial condition. Now we simply train with a bunch of initial conditions trained in neural network to predict what happens then, and then it can generalize to other initial conditions. But you know about generalization that the problem is, we can we can only trust our neural network, if the problem we're considering is very similar to what we had in the data set, it doesn't arbitrarily generalize. Okay, so that is, you know, it is something to remember. So I said, all of these things have trade offs trade off one there is you have to predict all time steps at the same time, which is hard on your memory, right? It limits the size of things you can do trade off to you can only really trust your network if the problem you're considering is within your data set vicinity. There are other problems that we've mentioned problem three, we've made very specific choices with respect to how our kernel looks that it's only ever dependent on x minus y. So therefore it is a convolution. There's all these these channels, you know, engineering choice, more you cut off the top Fourier modes, which limits the types of signals you can analyze. The next choice is the number of intermediate computation steps right here, which limits the complexity you can assume, and so on. So there are just I'm not saying you don't have choices in the other numerical solvers you probably do, but just to remember there that that this is the case. So someone might say, well, can't you can't you just if you want to predict for longer time steps, you could make this t equals 11. And then simply, you know, not not go in slices of one, but maybe going slices of 100. So this could be t equals 111, this could be t equals 211, and so on. And that is completely completely valid. What they actually do is they subdivide the space further. So instead of doing like 40 time steps, they are doing like 80 time steps, but still times 11 to 50, I believe. The problem with extrapolating like like this and leaving away time steps is that see here you have a supervision signal in your training for each of the times. And it it might be that the fact that so you know, time step 15 looks something like this. And I know I'm trimmed to M this time step 16 is just like a small evolution like this from right, it's it's like a small difference. And it could be that the neural networks, because they don't have internal dynamics, right, they don't internally like dynamically simulate this physical system, they simply learn to map things to things. And if if they are still related to each other a lot, then sort of they can make sense of it. So if one slice, so this could be the slice 15. This could be slice 16, if, if these are sort of related, you know, it can, it can make sense there is a relation between them. Also you can implement this as an RNN. And then also, from one step to the next, it sort of makes sense, you don't need an internal dynamic simulation. However, if you jump from time step 15 directly to time step 115, right, then it might look like it might look nothing like it, right, because it has evolved so much. And there can be quite chaotic dynamics. And that's the entire problem with PD is that the dynamics can be super complicated, and not easily predictable. So here, you don't really have a relation, right. And so since the neural network doesn't do internal dynamic simulation, it probably wouldn't I'm going to guess something like this wouldn't work too well, I could be wrong. But I'm going to guess classical solvers are still needed for this type of situation. So that's the other limiting factor is that you sort of are bound to data samples that can be statistically correlatively predicted from one another without having to do these physical, the real physical underlying simulations, though I have been proven wrong in the past. All right, so they talk a bit about how the fast Fourier transform plays into this. And there is actually an interesting thing, which we'll see at the code. And then they have three examples, like the Darcy flow burgers equation, and Navier Stokes equation. And they also do these Bayesian inverse problems, where I believe the what here what you have is sort of a thing at time step, you have the bottom thing given at some time step, and then you want to find out the original thing. And what you do is you have like an algorithm that is simply guessing. So you have a you given and you want to find out the a so the a is unknown. So you simply start with a zero and guess what you is going to be from that a zero. So you evolve your state a to you. And then if it's not entirely correct, you try again, you try a one. Okay, what does that give me now? You see you kind of play a game of guessing and you have an algorithm that does this guessing kind of smartly. So it says, Oh, now that's not the direction I want to go to, it's sort of a reinforcement learning algorithm a little bit. And the important part is it needs to do a lot of these forward evaluation, right, it needs to change a little bit, and then evaluate and see if the you that comes out is the same as the you that you want. So you want to find the initial state of any given evolved state. And if you need a lot of forward evaluations, it's going to be a problem if the if the forward evaluation is really slow, like these classical simulators. So these neural networks can really help right here, and I think they bring it down, they bring down the time it takes from 18 hours or so to two and a half minutes for this entire evaluation. So that's pretty cool. And they also outperform actually in terms of error, they outperform these these kind of baseline methods. So this is pretty cool as well. So not only are they faster, they also are less error prone. All of this pretty cool. Now let's just spend like a short time to dive into the code. The code is still quite a bit quite hacky. But that's research. So deal with it. So here you can see that the the top class is what this called this net 2d. So and that's 2d, I always I like to look at the forward pass before I look at the how the network is made, because you understand how things flow. So in the forward pass, you simply have this con this this convolution right here. What's called conv one, it's not really a convolution, right? This is this is simply an instance of this simple block and x is just passed through it. So this simple block right here, by the way, the data is prepared, as you can see, there is quite a bit of preparation going on. So you have a and you have you so a as you can see, is prepared as an s by s, that's the discretization of the grid by t in. So this is your D by D by 10, like this is 10 input time steps. And it is already expanded to a T tensor. So the T is going to be the output steps that we're going to consider. So here, a is going to be transformed repeatedly into a, a tensor that ultimately will have T output time steps. You can see you have to hold one of these things in memory for each training sample. And then you annotate actually x and y and t, these are like positional encodings for if you know transformer positional encodings, these are simply linear positional encodings for x, y, and t, you can catenate those and off you go. So where were we x was forward passed through this simple block 2d. What's the simple block 2d the simple block 2d is this thing right here. So again, let's look at the forward pass. So first of all, we're going to FC zero, which what looks like a fully connected layer, we're going to permute the axes, then we're going to through con zero, w zero, a batch norm, and a relu. So you can see this right here is what we saw in the diagram, x one and x two are the different paths through the network. This is the top path. If I go back to the paper quickly, this is the top path in this diagram. And the bottom path is this thing right here. And then there, the two are added. And then there's a batch norm, which is not in the diagram. And then there is a relu. So the bottom path is pretty simple. And you can see right here, by the way they restructure it, that this is going to be point wise. So this is not going to be in pixel space, this is going to be a point wise, only in the channel transformation. So these W's are implemented as one, one by one convolution, you see, it's a one D convolution and the kernel size is one. So all these does is for each point for each point in the grid space in the pixel space for each pixel, they're going to take this all of this pixels channels and transform this into a new vector of the same amount of channels. So you can see the input channels and output channels are always the same dimension. So actually, this entire network right here operates on this width, which is this latent dimension. It's only the first layer that transforms this from 13, which is 10 plus the three positional encodings to this latent dimension. And then the last network, this transforms it from the hidden dimension to 128 for some reason and then 128 to one, which is each pixel has a one dimensional output, which is this vorticity that you're trying to predict. And by pixel here, I mean an x, y, t entry. Okay. All right, so yeah, so exactly. So this goes from 13 to one, and then it is reshaped again, of course, to the to the appropriate size to give you all of the outputs. Okay, so you can see this is the input. This is the output down here. In between, we have four blocks of this upper path and lower path. So the upper path, sorry, the lower path we just saw is a one by one convolution. And the upper path is this conv zero. So this conv zero is this spectral con 3d fast. Okay. And it's parameterized by these modes. So the modes is how many of these Fourier modes you want to retain. We saw we throw away the top Fourier modes, whatever they are. And the modes here is whatever you want to retain in this case is set to four, which is actually eight, if you work it out, and we'll see why. So the spectral con 3d fast, again, let's look at the forward pass. So what does the forward pass do? It does a Fourier transform, a fast Fourier transform. And at the end, it does an inverse Fourier transform. Okay. So this is certainly, certainly we are now in the top part right here, Fourier transform and at the end, inverse Fourier transform. And now these are in the middle is implemented a bit weirdly, because of how the fast Fourier transform works, what you get, basically, you get an image out of it, not a get actually a 3d thing, but you get an image and the important Fourier modes are not like at the bottom or at the top, the important Fourier modes are actually in the corners right here. So what you what you want to cut away is all of this, all of this middle part if you want to throw away so this is equivalent to throwing away these high frequency things right here. So that's why this is implemented. So weirdly, you can see that here, first, we are going up to the modes in each of the x, y and t direction. But then we're also going from here, we're going to the last modes in this direction with all the others. This is corner, this is corner one, this is corner two, this is corner three, and this is corner four, sorry, the bottom two right here is corner four. It's a bit weird. And we don't have to actually do this with eight corners, which you might have guessed, because why don't we do it with modes three, you see modes one and two, they always appear negative and positive. And you would guess we'd need to do the same thing again, with negative modes three, but we don't because this thing here is one sided, which because this is con con because this is a has a property of of conjugacy. A lot of these entries of the Fourier transform would actually be sort of symmetric and the one sided only gives you one part of the symmetries such that it doesn't waste memory. And it does so for the last dimension. So this dimension right here doesn't have this corner property. It's a bit weird. And you need to know the exact implementation of the Fourier transforms. But you know, that's what it is. So you can see that this mole 3d here is a it's compel mole 3d, it simply multiplies the input which is the signal right here by these weights, the weights, as you can see is simply a weight matrix that is in channels out channels modes modes modes and two two because it's complex numbers, and you see in this multiplication that the this is a complex number multiplication. So the real parts, and the real part is this the imaginary part is this. And the operator is an Einstein operator. I just thought this was funny. It says, bixies, yokes is boxes. So I challenge everyone to make Einstein, Einstein some notation that spell cool words, big sees yokes is boxes. But the the important part here is, so a is going to be the signal, which is going to be a batch in channel and then x, y, t, b is going to be the weight that comes in the weight matrix, which is in channel out channels x, y, t. And you can see pretty clearly in the Einstein notation are also here that the input channels are multiplied away. So these are summed over. And what results is the output channel. So this is basically a matrix multiplication for each of the samples in the batch and for each location x, y, z, it's a multiplication summing over the input channels resulting in the output channels. This is pretty standard, pretty standard transform mapping vectors to vectors. It's complex, it's in Fourier space, but ultimately, it's just a multiplication. So this is the code, they simply do four of these layers, going to Fourier space, and then back again to Fourier space and then back again. Why do they do this? Because as we saw, they throw away these higher modes right here. And that also limits severely this applicability. So if you only throw away the higher modes, if you just do everything in Fourier space, you severely limit yourself. In fact, these Fourier methods, they are already not really good for problems that have like non periodic boundary conditions. So the periodic boundary conditions case is, as I understand, one of the easiest cases. And so the applicability would be limited. And the authors hope that by sort of doing this in the real space all the time, and also having these encoder and decoder networks, that they can retain sort of this information and be applicable to more than just periodic boundary conditions. Yeah, exactly. And that's basically it. I was ranting for so long, I think we are through to this paper. So maybe a quick summary, because this was a bit of a rant, right? So you want to predict these types of things. These types of things are well described by by their Fourier analysis. So transformations in the Fourier domain actually make more sense, because the evolutions of these things is more or less kind of these global signals. It's not localized like natural images, like there's the cat and there's something, these these this pattern right here, it will repeat, you know, as you go into infinity, these these sort of patterns will repeat and repeat. So the sort of global interactions between these periodic signals is much more important. That's why it makes sense to go to Fourier space to transform that in Fourier space, you can regularize by throwing away the higher modes, and you get the additional benefit that you are discretization independent. So you learn the function once and then you can input differently discretized signals. As you choose and the function stays the same because the Fourier transform, it will do as well as it can with the discretization that you give it. Once you're in Fourier space, you simply have a multiplication. And it's actually interesting, the filters here, the author shows some of the filters that are learned. So on top, you see filters in a CNN. And on the bottom, you see these filters, these Fourier filters learn these are actually as I understand it, these are transported back to the pixel space, so we can understand them. So you can see that the global kinds of patterns that these Fourier operators are sensitive to compared to the CNN filters, which just have like localized a certain pattern. So this is this is quite interesting. So it makes sense to go into Fourier space, there are a number of trade offs you have to do. You specifically you have memory requirements, and you can only predict signals that are similar to what you've seen in the training data set. And you could only solve things with periodic boundary conditions, but by means of architecture of these encoder and decoder networks at the beginning, like the P and the Q, and the fact that you always carry through and their residual way, the pixel space signal makes it such that you might get around this you might write it's not it's not a proof, but there is a possibility that you might get around this in total. This thing is way faster and more accurate than baselines, and has applicabilities and is sponsored by the nice people at the military. Alright, so this was long, I realize, but I invite you to check it out. The paper is technical, but well written. If you stick this kind of math part out in the middle, it's pretty cool. Alright, check out the code and I wish you a good time. Bye bye.
 AI has cracked a key mathematical puzzle for understanding our world.
 Just in from MIT technology review and look at this puzzle right here.
 It's got the bumps, it's got the valleys, the surfaces, it's got the braille, it's got
 the bits, the ones and the zeros, not only going up and down like in the matrix, but
 going in circles.
 It's got it all.
 This puzzle is really hard as you can see and AI has just cracked it.
 I'm being a bit hyperbolic of course.
 This is actually about a new paper that can solve, numerically solve a particular type
 of partial differential equations way faster than anything before it.
 So this is about this new paper and we'll get into the paper in a second.
 It's pretty cool, but as you can see MC Hammer, the infamous MC Hammer has tweeted this out
 and he has actually a pretty cool Twitter feed where he regularly tweets about scientific
 papers and so on.
 So pretty cool cross-domain overlap.
 I recommend that.
 So we'll get into the paper, we'll get into the code a little bit as well because I think
 it helps to understand what's going on.
 I want to start out by, this is the blog post by one of the authors and it's pretty good
 to get a basic overview of the paper and here is the motivational example.
 So the motivational example is the Navier-Stokes equation, which is an equation in fluid dynamics.
 So you're trying to predict how a fluid evolves over time given a certain parameters like
 its viscosity and a forcing function.
 So basically how sticky it is and how hard you stir it and then you want to know how
 it evolves over time.
 You can see on the left is given an initial condition and I think on the right is sort
 of a rollout after the 10th time step until the 50th time step.
 And the ground truth is obtained with a sort of classic numerical solver where you do little
 time steps and you calculate the interactions and then this takes a lot of time and compute.
 And on the right is the prediction of this new Fourier neural operator that this paper
 develops.
 And you can see it's almost equal and the gist of it is that the thing on the right
 simply takes one forward propagation through a neural network.
 So it takes like 0.00 something of a second to compute the thing on the right, whereas
 the thing on the left is quite hard to compute and as I understand can take minutes.
 So here you see the motivational example.
 These things are described by partial differential equations, which are sort of linearized ways
 of describing how the system evolves over one time step.
 And it'd be cool if we could solve these faster because this is applications in aerodynamics
 and other types of engineering fields.
 All right, so let's jump into the paper.
 As always, if you like content like this, consider sharing it out, telling your friends
 about it and subscribing, of course.
 So the paper is called Fourier Neural Operator for Parametric Partial Differential Equations.
 And it's by Tsong Yi Li, Nikola Kovatsky, Kamjar Aziza Deneshelli, Borygede Liu, Kaushik
 Patacharya, Andrew Stewart and Anima Anankumar of Caltech and Purdue University.
 So I feel the paper is both very cool and a bit overhyped.
 So we're going to see what it does.
 It's for a particular type of PDEs.
 And it has a lot of, let's say, engineering choices that make it possible to solve with
 neural networks, but also that limit its applicability to where the classical methods would be applicable
 where this thing isn't.
 So there are tradeoffs definitely to reach the sort of speed up that they reach.
 But we'll get into this.
 First, I actually want to scroll down right here all the way because there is something
 that you don't often see in the sort of machine learning field.
 And that is here in the acknowledgments section.
 And I just find it interesting.
 Don't regard this as anyone.
 But here we are supported by the LWLL grants, which I understand is DARPA.
 Beyond Limits, which is like a makes soft or makes AI or systems for things like gas
 and oil and so on with British Petroleum as a main sponsor.
 Raytheon, which of course is a giant military manufacturer.
 We have the Army Research Laboratory and so on.
 So you can see that this is kind of, I don't know, I don't see this often.
 This is sort of a good bouquet of sponsorships.
 Of course, there's also Microsoft, Google, and so on.
 Yeah, but it's just interesting to see that the Army is pretty heavily into these things.
 And of course they would be.
 I mean, rockets need to fly and they need to be aerodynamic and so on.
 So yeah, I'm not saying this is bad or good.
 I just thought it was interesting that Raytheon would be a sponsor of this.
 All right, so let's dive in.
 As we said, we're interested in these types of problems right here, where you have this
 thing called...
 So there is this quantity called the vorticity, which as I understand is a derivation of the
 viscosity.
 So it sort of tells you how the fluid is moving right now.
 And so this state right here.
 And then you apply a sort of constant forcing function and you want to know how that evolves
 over time.
 So you can see at time step 15, you get sort of this picture.
 So these move past each other and see this moves here, this moves here.
 And then at time step 20, you can see they are fairly moved.
 This blue thing moves in here as well.
 And they just sort of mix.
 And there are certain parameters that make the fluid more sticky or not so sticky.
 And the interesting regimes is, I guess, when it's not very sticky, so not too sticky, but
 also not sticky enough.
 And then these really complicated patterns occur.
 And to predict them would be very, very valuable.
 So you want something that takes in this initial state right here and outputs all of these
 these future states.
 And usually this is done by these classical numerical solvers.
 So the Navier-Stokes equation is described by a set of partial differential equations.
 And you can see this down here.
 So Navier-Stokes equation is described by this set of equations right here.
 Is there?
 Yep.
 And you can see that the that this this is fairly complex.
 It includes partial derivatives, gradients, and so on.
 So this is the this is this vorticity, and it includes that on on both sides.
 And this is this the yeah, this is two derivatives, maybe.
 Or is it just the delta?
 I don't even know.
 I'm not an expert in partial differential equations by any means.
 So anything coming from that direction, don't take me for granted.
 I'm going to give you sort of the under the thing of what I understand from this paper.
 And so with respect to that entire area, I'm not an expert, I just can understand that
 this is fairly complex.
 And what you usually do is you take the initial state and you just evolve it in time.
 So you take this time parameter, and you do you go one little little time step, and then
 you calculate because these are all sort of linear linear equations, you calculate this
 one little time step into the future, you update your state, right?
 It's sort of like, you know, you have your points here and how they move, and how they
 move is given by their gradients.
 So these are all sort of linearized things.
 Now, you don't want to move them too much per time step, because ultimately, if this
 thing moves, and this thing moves, then the movement of this arrow will change because
 this thing over here moves, right?
 So you want to compute this one little time step into the future, like to here and this
 to here, and then you want to recompute all of these arrows.
 So maybe now that points a little bit more here, and that points a little bit more here.
 And then you want to update it again.
 So you have these sort of these these numerical solvers that go little tiny time step by little
 tiny time step, it's not even this if here if you see t equals 20 or something, it's
 not 20 time step for these solvers, but these usually go like 1000 or 100 steps per time
 step that is here, or something like this, they need to take very tiny steps to be accurate.
 And that takes a long time.
 So the idea is, can't we simply can't we simply simply input this, let's say this thing or
 or like something at time 15, and directly predict the thing at time 30.
 And that's exactly what this paper does.
 And a lot of papers have done this before, but without much success.
 So this paper proposes to do this in the Fourier domain, and we'll see the path that they take
 right there.
 So they go into the will shortly go into sort of the the basics right here.
 So what you want what you're looking for is a function G that takes an A and gives a U.
 So what are A and U, A and U are both function spaces.
 So A, A and U here are functions.
 So A is a function, as you can see, A is a function, and U is a function, but you can
 characterize them as data points.
 So in this in this way, there is a functions and data points are sort of interchangeable,
 you can see an image like this as a data point, where it's an image, but you can also see
 it as a function where every x and y coordinate is mapped to a value, right.
 So when when they talk about functions, very often they talk about this type of function,
 where you have x, y and t, so t is also t is zero here, x, so the function would x,
 y, t map that to some value, right here, the vorticity.
 And you want to transform this function.
 So this function would be A, A would be the function at time, let's say zero or something
 or the times zero to 15, you would want to map that to the function, the function U that
 also takes an x and the y, let's leave t out for the moment, also takes an x and the y
 and let's say t, but t is set to 30, and maps that to a vorticity, right.
 So you want to input a function and output a function, but it's the same as inputting
 an image and outputting an image in as for from an engineering perspective, of course,
 from a math perspective, it's a little bit different.
 But other than that, it's a fairly standard machine learning problem.
 So you have this, these sets A and U, and you're looking for this function, G that maps
 A to U.
 So we study maps, which maps G, which arises the solution operators of parametric PDEs.
 Suppose we have observations, where A is an IID sequence from probability measure mu,
 transported on I and U is the A transported by G, it is possibly corrupted with noise,
 we aim to build an approximation of G by constructing a parametric map.
 This G right here.
 So it's a bit of a mathy way of saying we have a bunch of data points where we were
 a this is the initial state goes to U, which is the state at some point in time.
 And we know that there is a function G, this is this G with this inverse cross, we know
 that there is a true function that maps any A to U. So a single function G that can if
 I input the initial state can give me the output state.
 And what I want to do is I want to approximate this by a parametric version.
 So these here are the parameters.
 And of course, as you can guess by now, G is going to be this G right here is going
 to be a neural network that is parameterized by theta.
 So these would be the layers of the neural network.
 And we're going to input A into the neural network, and we're going to get out U.
 So that's basically that there is quite a bit of math right here.
 And the math here is to derive what they call a neural operator.
 So here is one layer of this neural network.
 As we said, we're going to input A. Now A first thing that we do A is going to be, let's
 say up projected.
 So A is going to be made into a latent representation v zero.
 So this is let's call that here P. So there is a function P, which is going to be a little
 layer of neural network.
 And it is going to produce this v zero.
 So v zero is going to be a latent state of the neural network.
 And then there is going to be a number of these layers that transform this to v1, v2,
 v3.
 I think there are four layers of these in their particular implementation, but there
 don't need to be four layers.
 You can choose that, as you can choose any depth of neural network.
 And then at the end, you're going to project that down to whatever output you want.
 So U.
 So this function here is called Q. And these are just going to be neural networks.
 So P and Q are going to be your very, very classic up projections and down projections
 of data point.
 We'll get into sampling.
 Let's go actually right now.
 So one thing right here, and they stress this, is that they work in function space, right?
 They don't work on the, let's say they don't map the data point to the data point.
 What you could do is simply have like a convolutional neural network, an image to image network,
 and so on.
 But what is the problem with that?
 So if you have your A, which is your initial state, and it has these bunch of fluid things
 right here.
 And what you do when you have an image is you sample this, right?
 You sample this at different, sorry, maybe a regular grid.
 I am terrible at regular.
 So you sample this into a certain amount of pixels, and your neural network will operate
 on this, right?
 This will give you some kind of a tensor, which is, let's say we have a, so this is
 a seven by seven grid.
 Okay, so your neural network is going to expect this as an input dimension.
 And whatever U is, of course, so you map this to U, which is also going to be some sort
 of image, okay, where you need to output pixels.
 So again, you have some set resolution, and your neural network can only operate at that
 particular resolution.
 What they're doing right here is the cool thing about is it can operate at any resolution.
 So once you've learned the network, you can input higher resolution images, or you can
 output higher resolution images, any any sort of, you can deal with more resolution, less
 resolution sampled irregularly, you can deal with a lot of things once the neural network
 is their neural network is learned.
 And how do they do it?
 They do it by only ever acting point wise in the spatial domain.
 So what they're going to do is they're going to take this a, and now we get into the more
 critical things.
 So here, a and u aren't just the beginning state and the end state.
 In fact, in this Navier-Stokes example, a is a tensor like this.
 So a is going to be a tensor with slices, and each slice describes one time step up
 to a given time.
 So this here could be t equals zero.
 So there is kind of the initial distribution, and then t equals one and so on up until t
 equals like 10.
 Let's say I think they do 10.
 So they let this thing evolve for 10 time steps.
 And I'm going to guess they do it using one of these classical methods.
 And that's the input.
 So the input isn't just the initial state, the input is actually here is what happened
 in the first time 10 time steps.
 And then the output isn't just the output at some particular time, but the output is
 actually also a slice right here.
 Each slice here describes the output at a particular time.
 So this would be t equals 11 up until t equals 50.
 So this is u.
 So the top one is sort of the conceptual thing, but the bottom one is what really happens.
 So they input 10 time steps, and they get out the 40 subsequent time steps, they predict
 them all at once.
 So and now you can see that in this particular case, how I can understand this is at each
 pixel here, I want to know what what is that pixels value after what after like certain
 amount of time steps, okay, like 11 or 50 right here or 40.
 And of course, the result is going to not only depend on the time zero, but on the entire
 evolution of time zero to time 10.
 So this here is an entire column for that pixel.
 And this is akin to that particular pixel having this many channels.
 So here I can just say, well, these are technically 10 channels or 11 or something like this,
 I probably screwed up this should be t equals zero to nine, and then 10 to 49.
 But so this is this is an entire stack.
 This is we can interpret this as input channels right here.
 And we can interpret these as output channels.
 Okay, so ultimately, one pixel is going to have input channels, all the time steps that
 happened up until the point where we want to predict and the output channels are going
 to be at the same time all the time steps of what we want to predict.
 Okay, so these projections now coming back to this, they simply work in the channels.
 So these P and Q, they are one by one convolutions.
 And the one by one convolution simply up project and down project these features, you see,
 these are one by one convolutions.
 Actually they could be dense layers.
 Let's check that in the code later.
 But for sure, what they do is they only work point wise.
 So they don't they don't mix the individual pixels together.
 In here, you simply get at like a D by D grid with each has 10 channels.
 And then you simply up project that to so here you have D by D times 10.
 And then you up project that using P to D by D times and here is a parameter that you
 choose.
 So this is sort of your latent dimension.
 Okay.
 And you are going to transform this tensor keeping it in this D by D by W dimensionality
 until you back projected using Q to D by D by in this case, 40.
 Okay, so but this, this and this, they only work point wise.
 And that means there is no particular dependence on the D right here.
 So the next data point could actually have a different D as long as this pipeline right
 here can handle different dimensions, because the P and Q only act point wise, you're good.
 So what do what do these magic layers here do?
 So these are these Fourier neural operators, okay, they transform one hidden state into
 the next note that we have four of these layers.
 So they don't need to be the same as the number of time steps we're trying to predict, you
 see.
 And it's pretty clear from here.
 So we these four hidden layers, they're simply transforming this entire volume right here,
 this entire input volume, they are transforming this as a sequence of latent states, and then
 outputting this entire volume.
 So this down here has nothing to do with the time steps that we're trying to predict.
 It is simply a sequence of computations of latent computations.
 And you know, that in a neural network, the deeper you make it, the sort of more complicated
 functions arise.
 Even though of course, the universal approximation theorem says that with one hidden layer, you
 can do anything.
 But in general, if you have deeper neural networks, the more you can kind of make more
 complicated things.
 And so four seems to be a good number of complicated for these particular problems.
 So here's what one of these layers does.
 It is very much like a residual network.
 So here you have the the V is the hidden representation at t plus one and t plus one is not as I said,
 is not the time step in the in the Navier-Stokes sense of time evolution of the PDE.
 This is simply the layer t plus one.
 So I don't know why they maybe Yeah, maybe t here makes still makes sense.
 Is it not because it's large t?
 Yeah, so they have large t right here.
 Okay, maybe.
 But in the engineering sense, it is not.
 This is simply the layer.
 And you can see it's formulated as a function.
 But again, don't be like the x right here.
 This is simply the x and y and t coordinates.
 So this, this, all of this here can be represented as one big tensor x, y, t, or x, y channels
 or something like this.
 Okay, don't.
 So don't, don't be confused by the fact that these are formulated as functions.
 So what we want to do is we have two different things.
 So one neural, this is one neural network layer, as you can see, at the very end is
 a nonlinearity.
 This is a point wise nonlinearity.
 And this is in the original pixel space or in the original spatial space, the D by D
 space, each of the things gets a nonlinear function slapped on top, as is normal.
 Then this part is normal as well.
 This is simply a linear transformation of the input.
 Again, this is point wise.
 Okay, so this is a linear transformation.
 So so far, so good.
 We have a linear transformation of the input and a nonlinearity.
 The important part is this thing here.
 So what this thing is, this is a kernel function that depends on the initial condition.
 So not only on the last hidden state, but the initial condition and sort of is then
 applied by the last hidden representation, like like here, and then only x is applied.
 So notice the difference right here.
 This is at a point x, we're getting this function value, which means we're getting the entry
 of that tensor.
 And then we're applying the linear transformation.
 This makes it point wise.
 Here, first, we compute this function by this by applying this kernel to the input function,
 so to the entire input tensor, and only then we are looking for the particular entry.
 So that means this thing here is a point wise transformation of that tensor, while this
 thing here, it takes in the whole tensor and outputs a sort of new tensor.
 So this is going to be the magic.
 Here where k, it goes, you can see it goes from from u space to u space, maps to bounded
 linear operators on u, and is parameterized by theta, maybe what's this?
 I don't know.
 I never know.
 So the this this kernel, we choose this to be a kernel integral transformation parameterized
 by neural network.
 So they define the kernel integral operator as this.
 And you can see this is an integral over the D, D is the input space of u and a actually.
 So this is a function that's dependent not only on where you are in the tensor, but on
 the initial input this a, and then that's convolved.
 So this here is a, a integral over the entire space.
 So that's convolved with v, you can see that this is a convolution.
 And it's fairly complicated.
 So this alone tells you nothing.
 But luckily, they say that they restrict this.
 So it's a bit annoying when things always depend on this a, that means that each of
 these functions right here, each of these arrows right here, these are the neural operators,
 actually let's go here.
 Each of these Fourier neural operators right here.
 They would always also depend on this a here, like this, and like this, and like this.
 This is a bit annoying for deep learning, because we sort of want one layer's representation
 to go into the next one.
 So they simply make an engineering choice and say, nope, nope, nope.
 So they say, we impose, right, we impose.
 If we remove the dependence on the function a, we impose that the kernel is simply a function
 of x, not only x and w, but only x minus w.
 So now you have a sort of proper kernel function in there that we can handle.
 We obtain that four is a convolution operator.
 Okay, it wasn't a convolution before it was just an integral.
 But now if you restrict your kernel functions to this, you get a convolution, we exploit
 the fact in the following section by parameterizing k directly in Fourier space and using the
 fast Fourier transform to efficiently compute four.
 This leads to fast architecture, which abstains state of the art results for PDE problems.
 So there's quite a bit of math right here to finally arrive at this thing here.
 So what is all this math for?
 This math is for saying what we want, we want to build our neural network like this.
 And what we do is we simplify and specify this kernel thing until the kernel looks something
 like this.
 So we restrict the kernel to be a convolution.
 And since a convolution in Fourier space is just a multiplication, what we can do is instead
 of taking the function V and convolving it with this kernel, what we can do is we take
 the Fourier transform of the function V, then multiply it in Fourier space by this thing.
 And this thing is now simply a matrix that's learned in as a bunch of parameters.
 And then we do the inverse Fourier transform.
 Now you might ask why is this relevant?
 Why can't we just do a convolution like we do normally?
 And the reason is, so when you do a Fourier transform, what do you do?
 You have some kind of signal like...
 And so on.
 So you take this signal and you transform this into Fourier space.
 And here we just go like one vector.
 So here, as you know, in Fourier space, you have these basis functions, which are sort
 of these different parameterization of sine waves, or you can do it with cosine waves,
 and they get faster and faster, and so on.
 So you know that you can decompose any signal into its basis functions in this kind of periodic
 function space.
 So this function right here might have, you know, one times this function, plus 0.1 times
 this function, plus two times this function, minus five times this function, and so on.
 So you can describe any of that.
 Now for these type of PDEs that we're looking for, the special thing about them is they
 are fairly well described if you simply cut away the sort of top Fourier modes and only
 work with these because they are, you know, sort of the individual tiny ripples you might
 not want to take into account.
 So you can truncate the lower Fourier modes, and that's what they do exactly here.
 And they learn.
 So instead of transforming this signal directly into the next hidden representation, they
 go to Fourier space, cut the top Fourier modes.
 They have a way of making the next representation in Fourier space.
 And this is this r here.
 And that is simply a weight matrix that they multiply with.
 And that is, you can prove that that is the same as convolving in the original space.
 So multiplying in Fourier space is the same as convolving in the original space.
 And so they multiply the green numbers right here by r.
 Then you get something out.
 So I should maybe, this is way too much.
 So the green numbers you multiply by r to obtain new green numbers.
 So maybe r is the, is 2, 2, 4.
 So the new green numbers would be 2, 0.4.
 Then you do the inverse Fourier transform.
 So you get back to a signal.
 Now with 2 times this, so it might be bigger.
 And 0.4 times, so I can't even draw, but you sort of get the idea.
 You put it into Fourier space.
 You apply the function r, which is a multiplying by a matrix that you learn in Fourier space.
 You get new Fourier coefficients, you map them back.
 And there you have your next layers representation.
 Almost.
 Okay.
 So this is this Fourier neural operator and is described right here.
 What you do is you take your representation, your hidden representation, put it through
 a Fourier transform, which you can do in a differentiable fashion.
 You get these Fourier modes, which describes how to decompose the signal into these periodic
 functions.
 You take away the top modes, which is your sort of regularization.
 You apply r, which is in a dense layer of neural, not even that.
 It's a multiplication, okay, by a weight matrix.
 And then you obtain this, these new Fourier modes.
 You do the inverse, and then you have the next representation.
 Almost.
 What you do is we saw this before, a point wise transformation in the original pixel
 space.
 So this is very much like a residual network, right?
 Residual networks, they also have this.
 They have the implemented as one by one convolutions.
 So and then at the end, you apply the non linearity.
 What is good about this?
 Two things.
 First of all, throwing away the top Fourier modes is very advantageous to these types
 of problems that we have right here.
 You can see that the little jiggles right here, they will be sort of sorted out by the
 larger scale movements of the fluid.
 So throwing away the top modes is a sort of a regularization.
 It helps with generalization.
 And it's very easy in Fourier space.
 So these things other than natural images are described well by these Fourier spaces.
 And that, again, is an engineering choice.
 So you cannot not apply these things to everything.
 You can apply them to where this type of assumption holds.
 Second of all, this is now fully independent of the discretization of the input.
 Okay?
 Because when I take a picture and I sample it in a three by three, I can do a Fourier
 transform and I'll get all of these numbers right here.
 Okay?
 It's just, you know, the Fourier transform does a good job as possible.
 When I sample it in a seven by seven grid, like I sample it super densely, I do the same
 for transform, I get the same numbers right here.
 Okay?
 And it's not exactly the same.
 So they always claim it's the same.
 It's not exactly the same, of course, if you don't sample densely enough, your Fourier
 transform isn't going to be as accurate, let's say.
 So ideally, you want the Fourier transform of the real signal or the real underlying
 signal.
 But since you sample this, you can't have this.
 So there is a bit of a difference, but it is independent.
 So that's true.
 The function R that you learn simply operates on these Fourier modes.
 And these are fairly independent of how regularly you sample, of course, more regular, better,
 but still fairly independent.
 Yeah, so that's good.
 So if you have what they're going to do is they're going to have something like the three
 by three during training and then sample more densely during during inference, which is
 something you can do but understand that this is just it's just a form of interpolation,
 right?
 So the inverse Fourier transform simply gives you whatever you want interpolating using
 the Fourier modes it has.
 And of course, given a certain number of Fourier modes, which is quite small for them, I think
 it's something like eight or 12 higher resolution at some point doesn't help you anymore, because
 you've cut off the high resolution Fourier modes, I guess what can help you is this,
 this thing right here.
 But this thing right here only acts point wise.
 So you see, this is now fully independent of the discretization of the signal, which
 is a cool thing.
 So the two cool things about this entire stuff is that first of all, independent of discretization,
 second of all, these types of problems that we are having here, lend themselves very well
 to be described in Fourier space.
 Yeah, so that's why I'm saying this is for a particular type of problem.
 And also, there are a bunch of other things you can see right here.
 You have this entire input tensor right here, and this entire output tensor right here.
 And these can be fairly large, right, and all the intermediate representations have
 to be kind of at D by D by W.
 So this is, you can't go infinite time right here, like you could with a classic solver,
 like a numerical solver, all you need is the last time step, right, you go, what's the
 t equals one, then at t equals 1.1, 1.2, and so on, you just count up and you
 just go always from the last time step to the next time step here.
 Since it's in neural network, during training, you need to keep all of these tensors, the
 intermediate things, I guess you can do gradient checkpointing, but this is engineering wise,
 you predict all the future time steps at the same time.
 So you can't really go infinite in time.
 And how do you train this thing?
 You train it by simply giving it one of these A, right, you have a bunch of A's, so you
 have a bunch of these input tensors, a data set.
 And where you always say here is a one of these Navier-Stokes equation, sorry, type
 of problems, I've sampled it somehow, and I've let it run for 10 time steps.
 And then I've let it run for longer, u, so I let it run for longer.
 And here are time steps of this t equals zero to t equals nine or 10, let's go 10.
 And here is t equals 11 to t equals 50.
 So you have a data set, and this data set is fully computed by a classic forward solver.
 So you can't replace the forward solvers right yet, because you need them for generating
 training data, right?
 So this becomes your training data, this becomes generally your x and this becomes your y.
 And now you're learning this neural network, this entire thing to give you x to y.
 So you see, you still need the classic solvers to produce the training data.
 That's the first thing.
 The second thing is, you can pretty clearly see that the good thing is that now we can
 input any a so the classic solvers, you need to rerun them for each initial condition.
 Now we simply train with a bunch of initial conditions trained in neural network to predict
 what happens then, and then it can generalize to other initial conditions.
 But you know about generalization that the problem is, we can we can only trust our neural
 network, if the problem we're considering is very similar to what we had in the data
 set, it doesn't arbitrarily generalize.
 Okay, so that is, you know, it is something to remember.
 So I said, all of these things have trade offs trade off one there is you have to predict
 all time steps at the same time, which is hard on your memory, right?
 It limits the size of things you can do trade off to you can only really trust your network
 if the problem you're considering is within your data set vicinity.
 There are other problems that we've mentioned problem three, we've made very specific choices
 with respect to how our kernel looks that it's only ever dependent on x minus y.
 So therefore it is a convolution.
 There's all these these channels, you know, engineering choice, more you cut off the top
 Fourier modes, which limits the types of signals you can analyze.
 The next choice is the number of intermediate computation steps right here, which limits
 the complexity you can assume, and so on.
 So there are just I'm not saying you don't have choices in the other numerical solvers
 you probably do, but just to remember there that that this is the case.
 So someone might say, well, can't you can't you just if you want to predict for longer
 time steps, you could make this t equals 11.
 And then simply, you know, not not go in slices of one, but maybe going slices of 100.
 So this could be t equals 111, this could be t equals 211, and so on.
 And that is completely completely valid.
 What they actually do is they subdivide the space further.
 So instead of doing like 40 time steps, they are doing like 80 time steps, but still times
 11 to 50, I believe.
 The problem with extrapolating like like this and leaving away time steps is that see here
 you have a supervision signal in your training for each of the times.
 And it it might be that the fact that so you know, time step 15 looks something like this.
 And I know I'm trimmed to M this time step 16 is just like a small evolution like this
 from right, it's it's like a small difference.
 And it could be that the neural networks, because they don't have internal dynamics,
 right, they don't internally like dynamically simulate this physical system, they simply
 learn to map things to things.
 And if if they are still related to each other a lot, then sort of they can make sense of
 it.
 So if one slice, so this could be the slice 15.
 This could be slice 16, if, if these are sort of related, you know, it can, it can make
 sense there is a relation between them.
 Also you can implement this as an RNN.
 And then also, from one step to the next, it sort of makes sense, you don't need an
 internal dynamic simulation.
 However, if you jump from time step 15 directly to time step 115, right, then it might look
 like it might look nothing like it, right, because it has evolved so much.
 And there can be quite chaotic dynamics.
 And that's the entire problem with PD is that the dynamics can be super complicated, and
 not easily predictable.
 So here, you don't really have a relation, right.
 And so since the neural network doesn't do internal dynamic simulation, it probably wouldn't
 I'm going to guess something like this wouldn't work too well, I could be wrong.
 But I'm going to guess classical solvers are still needed for this type of situation.
 So that's the other limiting factor is that you sort of are bound to data samples that
 can be statistically correlatively predicted from one another without having to do these
 physical, the real physical underlying simulations, though I have been proven wrong in the past.
 All right, so they talk a bit about how the fast Fourier transform plays into this.
 And there is actually an interesting thing, which we'll see at the code.
 And then they have three examples, like the Darcy flow burgers equation, and Navier Stokes
 equation.
 And they also do these Bayesian inverse problems, where I believe the what here what you have
 is sort of a thing at time step, you have the bottom thing given at some time step,
 and then you want to find out the original thing.
 And what you do is you have like an algorithm that is simply guessing.
 So you have a you given and you want to find out the a so the a is unknown.
 So you simply start with a zero and guess what you is going to be from that a zero.
 So you evolve your state a to you.
 And then if it's not entirely correct, you try again, you try a one.
 Okay, what does that give me now?
 You see you kind of play a game of guessing and you have an algorithm that does this guessing
 kind of smartly.
 So it says, Oh, now that's not the direction I want to go to, it's sort of a reinforcement
 learning algorithm a little bit.
 And the important part is it needs to do a lot of these forward evaluation, right, it
 needs to change a little bit, and then evaluate and see if the you that comes out is the same
 as the you that you want.
 So you want to find the initial state of any given evolved state.
 And if you need a lot of forward evaluations, it's going to be a problem if the if the forward
 evaluation is really slow, like these classical simulators.
 So these neural networks can really help right here, and I think they bring it down, they
 bring down the time it takes from 18 hours or so to two and a half minutes for this entire
 evaluation.
 So that's pretty cool.
 And they also outperform actually in terms of error, they outperform these these kind
 of baseline methods.
 So this is pretty cool as well.
 So not only are they faster, they also are less error prone.
 All of this pretty cool.
 Now let's just spend like a short time to dive into the code.
 The code is still quite a bit quite hacky.
 But that's research.
 So deal with it.
 So here you can see that the the top class is what this called this net 2d.
 So and that's 2d, I always I like to look at the forward pass before I look at the how
 the network is made, because you understand how things flow.
 So in the forward pass, you simply have this con this this convolution right here.
 What's called conv one, it's not really a convolution, right?
 This is this is simply an instance of this simple block and x is just passed through
 it.
 So this simple block right here, by the way, the data is prepared, as you can see, there
 is quite a bit of preparation going on.
 So you have a and you have you so a as you can see, is prepared as an s by s, that's
 the discretization of the grid by t in.
 So this is your D by D by 10, like this is 10 input time steps.
 And it is already expanded to a T tensor.
 So the T is going to be the output steps that we're going to consider.
 So here, a is going to be transformed repeatedly into a, a tensor that ultimately will have
 T output time steps.
 You can see you have to hold one of these things in memory for each training sample.
 And then you annotate actually x and y and t, these are like positional encodings for
 if you know transformer positional encodings, these are simply linear positional encodings
 for x, y, and t, you can catenate those and off you go.
 So where were we x was forward passed through this simple block 2d.
 What's the simple block 2d the simple block 2d is this thing right here.
 So again, let's look at the forward pass.
 So first of all, we're going to FC zero, which what looks like a fully connected layer, we're
 going to permute the axes, then we're going to through con zero, w zero, a batch norm,
 and a relu.
 So you can see this right here is what we saw in the diagram, x one and x two are the
 different paths through the network.
 This is the top path.
 If I go back to the paper quickly, this is the top path in this diagram.
 And the bottom path is this thing right here.
 And then there, the two are added.
 And then there's a batch norm, which is not in the diagram.
 And then there is a relu.
 So the bottom path is pretty simple.
 And you can see right here, by the way they restructure it, that this is going to be point
 wise.
 So this is not going to be in pixel space, this is going to be a point wise, only in
 the channel transformation.
 So these W's are implemented as one, one by one convolution, you see, it's a one D convolution
 and the kernel size is one.
 So all these does is for each point for each point in the grid space in the pixel space
 for each pixel, they're going to take this all of this pixels channels and transform
 this into a new vector of the same amount of channels.
 So you can see the input channels and output channels are always the same dimension.
 So actually, this entire network right here operates on this width, which is this latent
 dimension.
 It's only the first layer that transforms this from 13, which is 10 plus the three positional
 encodings to this latent dimension.
 And then the last network, this transforms it from the hidden dimension to 128 for some
 reason and then 128 to one, which is each pixel has a one dimensional output, which
 is this vorticity that you're trying to predict.
 And by pixel here, I mean an x, y, t entry.
 Okay.
 All right, so yeah, so exactly.
 So this goes from 13 to one, and then it is reshaped again, of course, to the to the appropriate
 size to give you all of the outputs.
 Okay, so you can see this is the input.
 This is the output down here.
 In between, we have four blocks of this upper path and lower path.
 So the upper path, sorry, the lower path we just saw is a one by one convolution.
 And the upper path is this conv zero.
 So this conv zero is this spectral con 3d fast.
 Okay.
 And it's parameterized by these modes.
 So the modes is how many of these Fourier modes you want to retain.
 We saw we throw away the top Fourier modes, whatever they are.
 And the modes here is whatever you want to retain in this case is set to four, which
 is actually eight, if you work it out, and we'll see why.
 So the spectral con 3d fast, again, let's look at the forward pass.
 So what does the forward pass do?
 It does a Fourier transform, a fast Fourier transform.
 And at the end, it does an inverse Fourier transform.
 Okay.
 So this is certainly, certainly we are now in the top part right here, Fourier transform
 and at the end, inverse Fourier transform.
 And now these are in the middle is implemented a bit weirdly, because of how the fast Fourier
 transform works, what you get, basically, you get an image out of it, not a get actually
 a 3d thing, but you get an image and the important Fourier modes are not like at the bottom or
 at the top, the important Fourier modes are actually in the corners right here.
 So what you what you want to cut away is all of this, all of this middle part if you want
 to throw away so this is equivalent to throwing away these high frequency things right here.
 So that's why this is implemented.
 So weirdly, you can see that here, first, we are going up to the modes in each of the
 x, y and t direction.
 But then we're also going from here, we're going to the last modes in this direction
 with all the others.
 This is corner, this is corner one, this is corner two, this is corner three, and this
 is corner four, sorry, the bottom two right here is corner four.
 It's a bit weird.
 And we don't have to actually do this with eight corners, which you might have guessed,
 because why don't we do it with modes three, you see modes one and two, they always appear
 negative and positive.
 And you would guess we'd need to do the same thing again, with negative modes three, but
 we don't because this thing here is one sided, which because this is con con because this
 is a has a property of of conjugacy.
 A lot of these entries of the Fourier transform would actually be sort of symmetric and the
 one sided only gives you one part of the symmetries such that it doesn't waste memory.
 And it does so for the last dimension.
 So this dimension right here doesn't have this corner property.
 It's a bit weird.
 And you need to know the exact implementation of the Fourier transforms.
 But you know, that's what it is.
 So you can see that this mole 3d here is a it's compel mole 3d, it simply multiplies
 the input which is the signal right here by these weights, the weights, as you can see
 is simply a weight matrix that is in channels out channels modes modes modes and two two
 because it's complex numbers, and you see in this multiplication that the this is a
 complex number multiplication.
 So the real parts, and the real part is this the imaginary part is this.
 And the operator is an Einstein operator.
 I just thought this was funny.
 It says, bixies, yokes is boxes.
 So I challenge everyone to make Einstein, Einstein some notation that spell cool words,
 big sees yokes is boxes.
 But the the important part here is, so a is going to be the signal, which is going to
 be a batch in channel and then x, y, t, b is going to be the weight that comes in the
 weight matrix, which is in channel out channels x, y, t.
 And you can see pretty clearly in the Einstein notation are also here that the input channels
 are multiplied away.
 So these are summed over.
 And what results is the output channel.
 So this is basically a matrix multiplication for each of the samples in the batch and for
 each location x, y, z, it's a multiplication summing over the input channels resulting
 in the output channels.
 This is pretty standard, pretty standard transform mapping vectors to vectors.
 It's complex, it's in Fourier space, but ultimately, it's just a multiplication.
 So this is the code, they simply do four of these layers, going to Fourier space, and
 then back again to Fourier space and then back again.
 Why do they do this?
 Because as we saw, they throw away these higher modes right here.
 And that also limits severely this applicability.
 So if you only throw away the higher modes, if you just do everything in Fourier space,
 you severely limit yourself.
 In fact, these Fourier methods, they are already not really good for problems that have like
 non periodic boundary conditions.
 So the periodic boundary conditions case is, as I understand, one of the easiest cases.
 And so the applicability would be limited.
 And the authors hope that by sort of doing this in the real space all the time, and also
 having these encoder and decoder networks, that they can retain sort of this information
 and be applicable to more than just periodic boundary conditions.
 Yeah, exactly.
 And that's basically it.
 I was ranting for so long, I think we are through to this paper.
 So maybe a quick summary, because this was a bit of a rant, right?
 So you want to predict these types of things.
 These types of things are well described by by their Fourier analysis.
 So transformations in the Fourier domain actually make more sense, because the evolutions of
 these things is more or less kind of these global signals.
 It's not localized like natural images, like there's the cat and there's something, these
 these this pattern right here, it will repeat, you know, as you go into infinity, these these
 sort of patterns will repeat and repeat.
 So the sort of global interactions between these periodic signals is much more important.
 That's why it makes sense to go to Fourier space to transform that in Fourier space,
 you can regularize by throwing away the higher modes, and you get the additional benefit
 that you are discretization independent.
 So you learn the function once and then you can input differently discretized signals.
 As you choose and the function stays the same because the Fourier transform, it will do
 as well as it can with the discretization that you give it.
 Once you're in Fourier space, you simply have a multiplication.
 And it's actually interesting, the filters here, the author shows some of the filters
 that are learned.
 So on top, you see filters in a CNN.
 And on the bottom, you see these filters, these Fourier filters learn these are actually
 as I understand it, these are transported back to the pixel space, so we can understand
 them.
 So you can see that the global kinds of patterns that these Fourier operators are sensitive
 to compared to the CNN filters, which just have like localized a certain pattern.
 So this is this is quite interesting.
 So it makes sense to go into Fourier space, there are a number of trade offs you have
 to do.
 You specifically you have memory requirements, and you can only predict signals that are
 similar to what you've seen in the training data set.
 And you could only solve things with periodic boundary conditions, but by means of architecture
 of these encoder and decoder networks at the beginning, like the P and the Q, and the fact
 that you always carry through and their residual way, the pixel space signal makes it such
 that you might get around this you might write it's not it's not a proof, but there is a
 possibility that you might get around this in total.
 This thing is way faster and more accurate than baselines, and has applicabilities and
 is sponsored by the nice people at the military.
 Alright, so this was long, I realize, but I invite you to check it out.
 The paper is technical, but well written.
 If you stick this kind of math part out in the middle, it's pretty cool.
 Alright, check out the code and I wish you a good time.
 Bye bye.