NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ML Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper nerf network nerf neural network deep learning tutorial what is deep learning introduction to deep learning deep learning explanation nerf network explanation neural rendering differentiable rendering differentiable neural rendering volume rendering nerf view synthesis view synthesis view synthesis nerf view synthesis neural novel view synthesis nerf
#nerf #neuralrendering #deeplearning

View Synthesis is a tricky problem, especially when only given a sparse set of images as an input. NeRF embeds an entire scene into the weights of a feedforward neural network, trained by backpropagation through a differential volume rendering procedure, and achieves state-of-the-art view synthesis. It includes directional dependence and is able to capture fine structural details, as well as reflection effects and transparency.

OUTLINE:
0:00 - Intro & Overview
4:50 - View Synthesis Task Description
5:50 - The fundamental difference to classic Deep Learning
7:00 - NeRF Core Concept
15:30 - Training the NeRF from sparse views
20:50 - Radiance Field Volume Rendering
23:20 - Resulting View Dependence
24:00 - Positional Encoding
28:00 - Hierarchical Volume Sampling
30:15 - Experimental Results
33:30 - Comments & Conclusion

Paper: https://arxiv.org/abs/2003.08934
Website & Code: https://www.matthewtancik.com/nerf

My Video on SIREN: https://youtu.be/Q5g3p9Zwjrk

Abstract:
We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x,y,z) and viewing direction (θ,ϕ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there. Look at these objects right here. What if I told you that I'm going to give you a bunch of pictures of these objects from different sides. And what you have to do is you have to come up with a system that generates me the picture as if the object was viewed from any direction. So something like this, right? Any direction, you can get me a picture of that object from just a few input pictures. This is a pretty daunting task. Specifically, look at the ship, for example, right here. You can see in the water, there's specularities that only appear if you view it from a very particular angle, right? Also the drum kit, you see that the microphone on the left, it has very specific structure to it. So this is not at all like a trivial task. There's very intricate things here. And this not only with toy data, but here you can see real world scenes. So this isn't some kind of abstract thing. You can actually use this in the real world. Now, don't look at these things too long. They tend to make me dizzy. But that's ultimately the goal. Input a few pictures and then being able to synthesize any kind of view. So the paper we're going to look at, it's a bit of an older paper, but I think it's pretty cool and it's relevant. And there is a bunch of follow up work to this. This is very popular right now. This is the paper introducing NERF, representing scenes as neural radiance fields for view synthesis. And it's by Ben Mildenhall, Pratul P. Srinivasan, Matthew Tanchik, Jonathan T. Barron, Ravi Ramamurthy and Ren Ng. This, as you can see, the task is called view synthesis. And what you can do with view synthesis or with this paper specifically is you can it can also it takes into account your viewing direction, which gives a much more realistic impression. We've already seen this with kind of the lighting here. But in order to really show you this on the left, you're going to see this novel view that is rendered. And on the right, it's sort of like a fake thing that you couldn't do in reality. But what we're going to do is we're going to keep the camera at the same position, but we're going to tell the scene that the camera is at a like switching around. And that makes you able to see just how different a pic like a room can look like if viewed from different directions. So the right one is really kind of physically impossible. It's just meant to show you how different things look differently if they think they are viewed from a different direction. Right. So the same thing here. And it just looks amazing. What you get automatically out of the systems are depth maps. These are notoriously hard to get, especially for complex scenes such as this one. Also, this one right here. It's it's very complex and it handles it fairly well. Sorry. You can even do something like AR right here since you now have a representation that tells you how far everything is away and you have it from different views. You can see. Yeah. And you can even get meshes. So I should be able to move that around here. This is now a mesh. It's not only view synthesis, but you can actually fill out the voxels, which is a slightly different task. And if you have pictures from all around, you can synthesize kind of any view in between, as you can see right here. So we're going to switch away from the fancy videos to the paper. Now the special thing about this paper and this is it's in the spirit of something like sirens. So sirens, we've I've made a video about it. And the special thing right here is it uses deep learning in a little bit of a different way than we would normally use it. So first of all, what does the abstract say? We present a novel, sorry, a method, where it is novel, that achieves state of the art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. So the task description is the view synthesis, right? Synthesizing novel views. Also, you're given a sparse set of input views. So you're given you have a scene. Let's say you have a tree or something like this. So here's a tree. I know beautiful and you're given a bunch of images. So maybe someone, you know, stood here and took a picture. So the picture kind of views in in this direction. It pictures depicts the tree and someone stood here and took a picture of the same tree. Maybe the same person, someone flew up here, took a picture of that tree. So you get a bunch of those. Maybe you get 20 or something around the tree, maybe more, maybe less. So from these pictures, you want to build a thing that can generate any view from anywhere. And the way they do it is by optimizing an underlying continuous volumetric scene function. This is a cryptic way, but it goes along the direction of the sirens and kind of a bigger trend in, I think in the AI in these in these neural rendering papers and so on, which is that we want to overfit a neural network to a single data point. This is really different from classic deep learning. If you ask someone, how would you go about this problem with deep learning? What they would tell you is, okay, I need a data set. I need a data set of these different scenes and the input. Now I have my X and my Y. So the input X is going to be always like, you know, 30 images of a scene and Y is going to be the scene itself or whatnot, like the tree or the mesh of the tree or something like this. And I need this many, many times. So I need a data set with 30 images of, I don't know, a house and the Y is the house and so on. So that's my training data set. And in my test data set, it can be something else, right? So it can be things that I now want to test. However, in this particular case, this is not the case here. It is one neural network that is fit to one scene. So what we have is a neural network that has a bunch of layers and all the neural network cares about is this particular scene, right? If we want to render a new scene, we take a new neural network. That's what I mean. We overfit a single neural network to this particular scene. We use the 30 images or so we got to train to completely overfit this neural network. And the goal is going to be that the tree itself, like the scene itself, is going to be in the weights of this neural network. So the weights of the neural network now represent the scene. And this has various advantages, right? If we already saw this with the sirens that very often this is a much, much better representation, more compact representation of the entire mesh than any other way. Like if you store it in voxels or something. But I hope this is a bit clear. Now, of course, the question is, what's the input and what's the output of this neural network? So the input is the following. Imagine you have a coordinate system here. So you get you get a coordinate system X, Y, and Z. Okay. And the neural network gets two things as an input. It gets as an input a position in that coordinate system, which we call we call X. And X is actually X, Y, Z is a three dimensional vector. Right. For example, right here, this is our X now. And also we get an D, which is a viewing direction. Okay. So the for example, if my camera is the top camera right here, the viewing direction would be this ray here. Well, everything's orange. I make that blue. So the viewing direction D would be that. Okay. So the angle here, we care about the angle. It's actually two angles you need to describe this viewing direction. So a position and the viewing direction and the output of the neural network. What does it output? The output of the neural network is going to be a color. See, like what color is at that particular location and the density. Is there even something at that particular location? Right. So the density tells you whether there is something or not. And if there is something, the color tells you what color it is. All right. This is a really different way. I want to stress that again of using neural networks. There is no longer images going in and you know something coming out. What goes in is a position and a direction. So you ask the neural network, hey, neural network, you in your entirety, you represent this scene. You represent if you're trained well, if you're overfit well, you're overfit on the tree. Now, I want to know at a particular location in this scene viewed from a particular angle. What am I going to see? So on this picture right here, I'm wondering for this pixel. If I send a ray to this location, what am I going to see? And the network will tell you you're probably not going to see anything because there's nothing there. Or if there is something there, you're going to see the color, I don't know, red. So how from this you can pretty easily get a picture, namely if I have my frame of the picture. For each pixel, I need to send a ray through the scene. So I send a ray through the scene. And what I need to do is I need simply need to query this model at each location. Here, here, here, here, here, here, here, and so on. At each location, I will ask the neural network, is there something there? And if there is, what kind of color am I going to see? And what you'll get is a bit of a curve. Thank you. Is a bit of a curve. So if here is your zero and you send the ray out into the scene, and this is the density going up, they have these graphs in the paper, by the way. I'm not smart enough to come up with them by myself. But they say, well, maybe at the beginning you're not going to see anything because there's nothing there. But then, you know, at some point you're going to see something. There is something there. You hit the tree, right? And you're inside the tree. And then you're out of the tree again. At the same time, at every point, it gives you color. Now here, it actually doesn't matter what the color is. It will still output a color, but it doesn't matter. And here it's going to say green, right? It's going to say at every point here, it's going to say green, green, green, green. And here, I guess it doesn't matter. It's probably going to say green as well. But in any case, what you can now do is you can simply look at where do I hit the first time the object, which is here, right? When the density goes up and what colors there. And now I know what I need to render at that particular pixel. Now you can simply do this for all pixels and you got yourself an image. And the neural network is powerful enough that for the same location, you can see this right here. It can give you different results depending on the different viewing directions. So that makes it such that it can kind of depend on where you view it from. It can capture these lighting effects, these reflections. And also it can capture transparency because imagine you have a curve that is not as clear as this one, but you have a curve that is something like here. So here is one wall of a glass and here is another wall of the glass. And they go up in density, but they're not fully dense. And the front of the glass is maybe blue and the back of the glass is red. And now if you integrate your ray along this and you integrate weighted by the density, you're going to get a mixture of preferably blue because that's in the front, but also a little bit of red. You can see that if a ray goes through here, you can handle transparency. And so this is a really powerful model right here. And again, there's no need for a data set other than the scene that is right in front of you. So the goal is going to be that if in the future we want to make augmented reality applications, we want to make games and so on, you are not actually going to store a mesh or kind of a voxel grid of some scene. What you're going to store is a neural network that can be queried from anywhere you want to look at the scene. And the neural network will tell you what you're going to see. It just happens that these things work extraordinarily well. So here's the process again, the task, you get a set of input images right here. You want to find out where they're taken from. So for each input image, you need to determine where was the camera and in which direction did it look. This is a known problem. You can see all these kind of classic structures from motion, slam and so on. They need to determine the camera positions from the pictures. And so that's a thing you can take from existing research. And then you want to render the new views. And yeah, here is, I think, where they get into it, where this is. Yeah, we represent, they say, a continuous scene as a 5D vector valued function. And this vector function is going to be a neural network. It has a five dimensional input and it has a the output is going to be a color, which is three dimensions and a density, which is one dimension. So the input is a 3D location and a 2D viewing direction. And the output is a color and a volume density. So in practice, we express direction as a 3D Cartesian unit vector. And they say we approximate this continuous 5D scene representation with an MLP network. So the network, as we said, this is the input, this is the output. And we optimize its weights to map from each input 5D coordinate to its corresponding volume density and directional emitted color. Now, the only question is, of course, we have these images. We don't actually have as a training set kind of the densities at that place. So everything needs to be sort of grounded into the images that we have. Now, luckily, the whole process that I've described here, which you see again here. So if you want to render an image, you take an image, you pick a pixel, you shoot a ray, and you sample along the ray and you ask your network what's there. The network will tell you if there's something there. And if so, what color you're going to see the density over time. And then you can render an image. Now, if you already have an image, right, which is we are given a set of these images, if you already have one, you can now calculate a loss. Namely, what do I see and what does the network tell me I should see? If the network is not trained yet, that's going to be a pretty big loss. And if you make the loss as something differentiable, then this whole process is in fact differentiable. That's the next cool thing about this. The whole process of sending the ray, sampling the position, integrating over it, and at the end coming up with a pixel color, that is a differentiable process. If, of course, if you do it correctly. But that means we can use those 30 images or 50 or whatever we have in order to construct a big loss. Every ray, so every pixel in every picture that we have defines a ray. So every ray essentially is a data point that we can fit to. So at the end, we get a pretty sizable data set for the network, which is going to be number of pixels times number of pictures. However, again, it is a different problem than having a data set of many of these scenes. So the whole process is differentiable, and that means you can just fit the neural network to this scene. You overfit it to these 30 images that you have, and that's going to be your network. And this network then is going to represent the scene in its weights. So the weights are the scene at the end. There is a bit of a so there are lots of engineering tricks here. So, for example, we encourage the representation to be multi view consistent by restricting the network to predict the volume density as a function of only the location X, while allowing the RGB color to be predicted as a function of both location and viewing direction. So the reasoning here is that the volume density is not dependent on the direction. Like either even if something is kind of transparent, it's going to be transparent. It's going to be the same transparent in from different direction. There's only very limited amount of materials where that is not the case. Right. So as a simplifying concept, we're going to see the transparency of the object is always the same, which is kind of where stuff is, is independent of where you look from. It's only how stuff looks that is dependent. So the RGB color is going to be a function of both location and viewing direction. And what they do is essentially so they input X right here. They so the the location, they yank this through a network, they get out two things. So they first get out this density and they also get out a hidden representation that hidden representation. They then concatenate with the viewing direction. And that goes through another stack of layers in order to give them the color. I think it's also, you know, you could do something with a transformer here and some causal masking, though I'm pretty sure someone has already done this, given that the paper is almost ancient at one year of age in the machine learning world. That's really old. So exactly. So this is the formula for new for rendering. This is a technique called volume rendering with radiance fields. So if you have a radiance field, a radiance field is a function that tells you exactly what we train in our network to do. Namely, you know, if I look from here and I look at that point, what do I see? What you want to do is you want to send a ray through the scene and you want to integrate along that race. You have kind of a far bound and a near bound. And you want to integrate from the near bound to the far bound. So that means you send the ray through the thing you want to integrate. This thing, this T thing right here, that tells you. You can see the density is in here along the ray from the beginning to the point where you are. That is the probability that the ray doesn't hit anything. Right. It's a probability that the ray goes on through that room. Basically, it's a probability of empty space. So or, you know, the inverse of that, like this distinguishes whether there is something or not, whether the ray continues up until the point T or not. So you have whether or not the ray is actually at that particular point. How dense that particular point is. So how much stuff there is in terms of occludance for your ray. So if this is high, your ray is going to stop and you're going to adopt the color that is there. You can see it's this is multiplied by the color at that particular place. So you send the ray. And as soon as your system determine, you know, there's something here, you're going to, since this is multiplied, the density is multiplied by the color, your your ray is going to adopt the color of whatever is there. And then after that, this quantity here is going to be small because this quantity is again an inner integral that tells you whether or not the ray even reaches that location. So the ray reaches the first location, at which point it's going to adopt the color. And after that, the it even though there is stuff right, even though the density is high, the ray is not reaching it. So the whole formula captures all of this. And as we said, with a bit of nuance, it like if this is not always zero one, it can handle transparency as well. And here they demonstrate again from the scene. So you have two different points in the same scene, but viewed from different locations. And on the right, they show you this is all the same point in the scene, but the circle represents kind of different angles at which you can view it from. And you can see that the color is really different depending on the angle where you look from. There are what do we have here? There are a lot of tricks. Oh, yeah, so they they approximate the integral with like quadrature, which also has existed. And they have a bunch of tricks. So the first trick to really get this to work is a novel like not a novel, but kind of the employment of a positional encoding that a positional encoding is not the same as you might know it from Transformers or something. The positional encoding here, it simply means that you send the input data point, which is this thing right here. XYZ, theta, phi, Greek letter. You send that to a higher dimensional space, right, in a very deterministic way. So if you have these low dimensional input, and especially if you want to represent this, this is really fine structure right here. You can see that this stuff right here, it's quite fine grained. OK, and so you need a way to handle fine differences between things. But you also need a way to handle, you know, course differences. And just a single floating point number probably isn't going to do it for a continuous function like this. So what you do is you send this to a higher dimensionality with these positional encodings that we know from Transformers. So these encodings right here, they will send. So what you do, and so in my video on attention is all you need, I explain those in detail. But you construct a hierarchy of sine waves or sine and cosine waves. But we can just do it with sine waves. So the lowest hierarchy is like this. And then the next thing in the hierarchy would be like double as fast. And then the next thing, well, this is four times as fast, isn't it? Well, you get the point, right? It's so I need up, down, up, wow. And then up, down, up, down, up. This is not a sine wave. But you I hope you get the point. And then you want to take a look, for example, your X, you take your X, you put it here like, OK, X is so this is like negative. I think they go from negative one to one. The coordinates they have and your high dimensional output is going to be, you know, this point, this point, this point and this point in the in their respective coordinate systems. Right. So that's you can. What this does is you can still clearly identify every point here. In fact, yeah, you can. You can identify every single point in your input space by, you know, looking at looking at the combination of where it is in the sine waves. But it gives the network a better chance to focus, for example, on details. If it wants to focus on details, it's going to look at this scale right here because tiny changes in the underlying X is going to result in a large change in this feature. If you want to focus on coarse grain stuff, then you look at this where you can, you know, you have to move pretty far to have a change. Whereas if you look at this scale for coarse grain things, it means almost nothing because, you know, if you want to make little difference between these two things, if you look at coarse grained structure, but they have, as you can see, like there's a lot of difference between those like this may be zero and this is maybe negative one. However, if you look at the two data points right here, sorry about that. So the same, let's say the orange distance and the blue distance, you can see that the two aren't so different in this representation. So it gives the network the choice at which scale it wants to look at for particular positions. So ultimately, you're going to map this five dimensional vector into a higher dimensional vector. And they consider like 10, 10 layers or four layers of these. How many of these different sine wave and cosine waves they construct. So again, they call it positional ketting. They say this is referred to as a positional encoding. However, transformers use it for a different goal of providing discrete representations as input to an architecture, yada, yada, yada. In contrast, we use these functions to map continuous input coordinates into a higher dimensional space to enable our MLP to more easily approximate a higher frequency functions. The second thing they do is they do hierarchical volume sampling. So when we said I send a ray through the scene and then I sample along, this either would take a lot of time or it would not be accurate enough. So what they do is they have actually two layers of neural network, one they call a course and one they call a fine. And as I understand it, here is a ray they first sample with the course one at rather coarse locations. And then they use that to evaluate where they should sample more. Let's say this thing right here has a real high density in the course network. They then sample around that a lot more, maybe one here, two, but a lot more, you know, sampling around where the course network things, the important stuff is. They optimize both networks at the same time. And that actually works out well. So here you see the loss. The loss is a combination now of the coarse network and the fine grain network. And you need to optimize both, even though the final view is only going to come from the fine grain network. You need to optimize both because the coarse grain network can tell you where the important stuff is. So the results you have already seen, there are a bunch of metrics that prove that this one is really good. And it can, as you can see, like you can handle fine grain structure right here in the microphone that others can't. And it also so they say it fits into a few. So one neural network of one scene fits into like a few megabytes. And this is so it fits into five megabytes. And this is a lot better than things that use like voxel grid representations, which I think this other thing they compare to uses over 15 gigabytes for the same scene. Which and this is interesting, which is even less memory than the input images alone for a single scene from any of our data sets. So this is really like it's it's really it's even smaller than the pictures. So so even if you maybe want to show this to another human, it'd be better. You send the train nerf than the pictures if space is a consideration, though I don't know how they measure the pictures. Like you can probably compress if it's different pictures from the same scene. I guess there's some compression potential if you want to transmit them as a never mind. So they also do ablations. And the only downside here is that it does take a long time to fit one of these neural networks. I don't exactly remember where they say it, but they say they calculate like, oh, here. So it's not too bad, but the optimization for a single scene typically take around 100 to 300 K iterations to converge on a single video of 100 GPU, which is about one to two days. So it's a single GPU. So it is, you know, you don't need a data center for it. But you're going to wait a while until you train one, though you only need to train it once and then you can render new views as you please. So the idea, I think, is going to be that let's say you make a video game or so. You're going to render this at your servers, then you transmit the neural network to the clients and the clients can just render it out right there. And yeah, there's a bunch of results and a bunch of ablations where they kind of leave away different parts and they show that especially kind of the positional encodings. I think this is the positional encodings are really important, as you can see on the right, there is no positional encodings. The view dependence is also quite important. You see if there's no view dependence, as you can see here, you do get the fine grain structure since you do have positional encodings. But you don't get these kind of light effects, right? This is this thing here is not a different color. It's simply the fact that the line light shines on it. And it's just not there here because, you know, all the network can do is output the same color for all directions. And most directions simply don't have that reflection. All right, so that is it. The code is available on this website that I've showed you. I'm certainly going to link it. Tell me what you think. I think this is pretty cool. I know this has given rise to a lot of work following up on this. I have very little overview over what's going on in the nerf space, but I think it's cool and I want to dive deeper into it. Thanks for being here. Bye bye.
 Hello there. Look at these objects right here. What if I told you that I'm going to give you a bunch of pictures of these objects from different sides.
 And what you have to do is you have to come up with a system that generates me the picture as if the object was viewed from any direction.
 So something like this, right? Any direction, you can get me a picture of that object from just a few input pictures.
 This is a pretty daunting task. Specifically, look at the ship, for example, right here. You can see in the water, there's specularities that only appear if you view it from a very particular angle, right?
 Also the drum kit, you see that the microphone on the left, it has very specific structure to it.
 So this is not at all like a trivial task. There's very intricate things here. And this not only with toy data, but here you can see real world scenes.
 So this isn't some kind of abstract thing. You can actually use this in the real world.
 Now, don't look at these things too long. They tend to make me dizzy. But that's ultimately the goal. Input a few pictures and then being able to synthesize any kind of view.
 So the paper we're going to look at, it's a bit of an older paper, but I think it's pretty cool and it's relevant. And there is a bunch of follow up work to this.
 This is very popular right now. This is the paper introducing NERF, representing scenes as neural radiance fields for view synthesis.
 And it's by Ben Mildenhall, Pratul P. Srinivasan, Matthew Tanchik, Jonathan T. Barron, Ravi Ramamurthy and Ren Ng.
 This, as you can see, the task is called view synthesis. And what you can do with view synthesis or with this paper specifically is you can it can also it takes into account your viewing direction, which gives a much more realistic impression.
 We've already seen this with kind of the lighting here. But in order to really show you this on the left, you're going to see this novel view that is rendered. And on the right, it's sort of like a fake thing that you couldn't do in reality.
 But what we're going to do is we're going to keep the camera at the same position, but we're going to tell the scene that the camera is at a like switching around. And that makes you able to see just how different a pic like a room can look like if viewed from different directions.
 So the right one is really kind of physically impossible. It's just meant to show you how different things look differently if they think they are viewed from a different direction. Right. So the same thing here.
 And it just looks amazing. What you get automatically out of the systems are depth maps. These are notoriously hard to get, especially for complex scenes such as this one. Also, this one right here.
 It's it's very complex and it handles it fairly well. Sorry. You can even do something like AR right here since you now have a representation that tells you how far everything is away and you have it from different views.
 You can see. Yeah. And you can even get meshes. So I should be able to move that around here. This is now a mesh. It's not only view synthesis, but you can actually fill out the voxels, which is a slightly different task.
 And if you have pictures from all around, you can synthesize kind of any view in between, as you can see right here. So we're going to switch away from the fancy videos to the paper.
 Now the special thing about this paper and this is it's in the spirit of something like sirens. So sirens, we've I've made a video about it.
 And the special thing right here is it uses deep learning in a little bit of a different way than we would normally use it.
 So first of all, what does the abstract say? We present a novel, sorry, a method, where it is novel, that achieves state of the art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views.
 So the task description is the view synthesis, right? Synthesizing novel views. Also, you're given a sparse set of input views. So you're given you have a scene. Let's say you have a tree or something like this. So here's a tree.
 I know beautiful and you're given a bunch of images. So maybe someone, you know, stood here and took a picture. So the picture kind of views in in this direction. It pictures depicts the tree and someone stood here and took a picture of the same tree.
 Maybe the same person, someone flew up here, took a picture of that tree. So you get a bunch of those. Maybe you get 20 or something around the tree, maybe more, maybe less.
 So from these pictures, you want to build a thing that can generate any view from anywhere. And the way they do it is by optimizing an underlying continuous volumetric scene function.
 This is a cryptic way, but it goes along the direction of the sirens and kind of a bigger trend in, I think in the AI in these in these neural rendering papers and so on, which is that we want to overfit a neural network to a single data point.
 This is really different from classic deep learning. If you ask someone, how would you go about this problem with deep learning? What they would tell you is, okay, I need a data set.
 I need a data set of these different scenes and the input.
 Now I have my X and my Y. So the input X is going to be always like, you know, 30 images of a scene and Y is going to be the scene itself or whatnot, like the tree or the mesh of the tree or something like this.
 And I need this many, many times. So I need a data set with 30 images of, I don't know, a house and the Y is the house and so on.
 So that's my training data set. And in my test data set, it can be something else, right? So it can be things that I now want to test.
 However, in this particular case, this is not the case here.
 It is one neural network that is fit to one scene. So what we have is a neural network that has a bunch of layers and all the neural network cares about is this particular scene, right?
 If we want to render a new scene, we take a new neural network. That's what I mean. We overfit a single neural network to this particular scene.
 We use the 30 images or so we got to train to completely overfit this neural network.
 And the goal is going to be that the tree itself, like the scene itself, is going to be in the weights of this neural network.
 So the weights of the neural network now represent the scene. And this has various advantages, right?
 If we already saw this with the sirens that very often this is a much, much better representation, more compact representation of the entire mesh than any other way.
 Like if you store it in voxels or something. But I hope this is a bit clear.
 Now, of course, the question is, what's the input and what's the output of this neural network?
 So the input is the following. Imagine you have a coordinate system here.
 So you get you get a coordinate system X, Y, and Z.
 Okay. And the neural network gets two things as an input.
 It gets as an input a position in that coordinate system, which we call we call X.
 And X is actually X, Y, Z is a three dimensional vector. Right.
 For example, right here, this is our X now.
 And also we get an D, which is a viewing direction.
 Okay. So the for example, if my camera is the top camera right here, the viewing direction would be this ray here.
 Well, everything's orange. I make that blue. So the viewing direction D would be that.
 Okay. So the angle here, we care about the angle.
 It's actually two angles you need to describe this viewing direction.
 So a position and the viewing direction and the output of the neural network.
 What does it output? The output of the neural network is going to be a color.
 See, like what color is at that particular location and the density.
 Is there even something at that particular location? Right.
 So the density tells you whether there is something or not.
 And if there is something, the color tells you what color it is.
 All right. This is a really different way.
 I want to stress that again of using neural networks.
 There is no longer images going in and you know something coming out.
 What goes in is a position and a direction.
 So you ask the neural network, hey, neural network, you in your entirety, you represent this scene.
 You represent if you're trained well, if you're overfit well, you're overfit on the tree.
 Now, I want to know at a particular location in this scene viewed from a particular angle.
 What am I going to see? So on this picture right here, I'm wondering for this pixel.
 If I send a ray to this location, what am I going to see?
 And the network will tell you you're probably not going to see anything because there's nothing there.
 Or if there is something there, you're going to see the color, I don't know, red.
 So how from this you can pretty easily get a picture, namely if I have my frame of the picture.
 For each pixel, I need to send a ray through the scene.
 So I send a ray through the scene.
 And what I need to do is I need simply need to query this model at each location.
 Here, here, here, here, here, here, here, and so on.
 At each location, I will ask the neural network, is there something there?
 And if there is, what kind of color am I going to see?
 And what you'll get is a bit of a curve. Thank you.
 Is a bit of a curve.
 So if here is your zero and you send the ray out into the scene,
 and this is the density going up, they have these graphs in the paper, by the way.
 I'm not smart enough to come up with them by myself.
 But they say, well, maybe at the beginning you're not going to see anything because there's nothing there.
 But then, you know, at some point you're going to see something.
 There is something there. You hit the tree, right?
 And you're inside the tree. And then you're out of the tree again.
 At the same time, at every point, it gives you color.
 Now here, it actually doesn't matter what the color is.
 It will still output a color, but it doesn't matter.
 And here it's going to say green, right?
 It's going to say at every point here, it's going to say green, green, green, green.
 And here, I guess it doesn't matter. It's probably going to say green as well.
 But in any case, what you can now do is you can simply look at where do I hit the first time the object,
 which is here, right? When the density goes up and what colors there.
 And now I know what I need to render at that particular pixel.
 Now you can simply do this for all pixels and you got yourself an image.
 And the neural network is powerful enough that for the same location, you can see this right here.
 It can give you different results depending on the different viewing directions.
 So that makes it such that it can kind of depend on where you view it from.
 It can capture these lighting effects, these reflections.
 And also it can capture transparency because imagine you have a curve that is not as clear as this one,
 but you have a curve that is something like here.
 So here is one wall of a glass and here is another wall of the glass.
 And they go up in density, but they're not fully dense.
 And the front of the glass is maybe blue and the back of the glass is red.
 And now if you integrate your ray along this and you integrate weighted by the density,
 you're going to get a mixture of preferably blue because that's in the front, but also a little bit of red.
 You can see that if a ray goes through here, you can handle transparency.
 And so this is a really powerful model right here.
 And again, there's no need for a data set other than the scene that is right in front of you.
 So the goal is going to be that if in the future we want to make augmented reality applications,
 we want to make games and so on, you are not actually going to store a mesh or kind of a voxel grid of some scene.
 What you're going to store is a neural network that can be queried from anywhere you want to look at the scene.
 And the neural network will tell you what you're going to see.
 It just happens that these things work extraordinarily well.
 So here's the process again, the task, you get a set of input images right here.
 You want to find out where they're taken from.
 So for each input image, you need to determine where was the camera and in which direction did it look.
 This is a known problem.
 You can see all these kind of classic structures from motion, slam and so on.
 They need to determine the camera positions from the pictures.
 And so that's a thing you can take from existing research.
 And then you want to render the new views.
 And yeah, here is, I think, where they get into it, where this is.
 Yeah, we represent, they say, a continuous scene as a 5D vector valued function.
 And this vector function is going to be a neural network.
 It has a five dimensional input and it has a the output is going to be a color,
 which is three dimensions and a density, which is one dimension.
 So the input is a 3D location and a 2D viewing direction.
 And the output is a color and a volume density.
 So in practice, we express direction as a 3D Cartesian unit vector.
 And they say we approximate this continuous 5D scene representation with an MLP network.
 So the network, as we said, this is the input, this is the output.
 And we optimize its weights to map from each input 5D coordinate to its corresponding volume density and directional emitted color.
 Now, the only question is, of course, we have these images.
 We don't actually have as a training set kind of the densities at that place.
 So everything needs to be sort of grounded into the images that we have.
 Now, luckily, the whole process that I've described here, which you see again here.
 So if you want to render an image, you take an image, you pick a pixel, you shoot a ray,
 and you sample along the ray and you ask your network what's there.
 The network will tell you if there's something there.
 And if so, what color you're going to see the density over time.
 And then you can render an image.
 Now, if you already have an image, right, which is we are given a set of these images,
 if you already have one, you can now calculate a loss.
 Namely, what do I see and what does the network tell me I should see?
 If the network is not trained yet, that's going to be a pretty big loss.
 And if you make the loss as something differentiable, then this whole process is in fact differentiable.
 That's the next cool thing about this.
 The whole process of sending the ray, sampling the position, integrating over it,
 and at the end coming up with a pixel color, that is a differentiable process.
 If, of course, if you do it correctly.
 But that means we can use those 30 images or 50 or whatever we have in order to construct a big loss.
 Every ray, so every pixel in every picture that we have defines a ray.
 So every ray essentially is a data point that we can fit to.
 So at the end, we get a pretty sizable data set for the network,
 which is going to be number of pixels times number of pictures.
 However, again, it is a different problem than having a data set of many of these scenes.
 So the whole process is differentiable, and that means you can just fit the neural network to this scene.
 You overfit it to these 30 images that you have, and that's going to be your network.
 And this network then is going to represent the scene in its weights.
 So the weights are the scene at the end.
 There is a bit of a so there are lots of engineering tricks here.
 So, for example, we encourage the representation to be multi view consistent
 by restricting the network to predict the volume density as a function of only the location X,
 while allowing the RGB color to be predicted as a function of both location and viewing direction.
 So the reasoning here is that the volume density is not dependent on the direction.
 Like either even if something is kind of transparent, it's going to be transparent.
 It's going to be the same transparent in from different direction.
 There's only very limited amount of materials where that is not the case.
 Right. So as a simplifying concept, we're going to see the transparency of the object is always the same,
 which is kind of where stuff is, is independent of where you look from.
 It's only how stuff looks that is dependent.
 So the RGB color is going to be a function of both location and viewing direction.
 And what they do is essentially so they input X right here.
 They so the the location, they yank this through a network, they get out two things.
 So they first get out this density and they also get out a hidden representation that hidden representation.
 They then concatenate with the viewing direction.
 And that goes through another stack of layers in order to give them the color.
 I think it's also, you know, you could do something with a transformer here and some causal masking,
 though I'm pretty sure someone has already done this, given that the paper is almost ancient at one year of age
 in the machine learning world. That's really old.
 So exactly. So this is the formula for new for rendering.
 This is a technique called volume rendering with radiance fields.
 So if you have a radiance field, a radiance field is a function that tells you exactly what we train in our network to do.
 Namely, you know, if I look from here and I look at that point, what do I see?
 What you want to do is you want to send a ray through the scene and you want to integrate along that race.
 You have kind of a far bound and a near bound.
 And you want to integrate from the near bound to the far bound.
 So that means you send the ray through the thing you want to integrate.
 This thing, this T thing right here, that tells you.
 You can see the density is in here along the ray from the beginning to the point where you are.
 That is the probability that the ray doesn't hit anything.
 Right. It's a probability that the ray goes on through that room.
 Basically, it's a probability of empty space.
 So or, you know, the inverse of that, like this distinguishes whether there is something or not,
 whether the ray continues up until the point T or not.
 So you have whether or not the ray is actually at that particular point.
 How dense that particular point is.
 So how much stuff there is in terms of occludance for your ray.
 So if this is high, your ray is going to stop and you're going to adopt the color that is there.
 You can see it's this is multiplied by the color at that particular place.
 So you send the ray.
 And as soon as your system determine, you know, there's something here, you're going to, since this is multiplied,
 the density is multiplied by the color, your your ray is going to adopt the color of whatever is there.
 And then after that, this quantity here is going to be small because this quantity is again an inner integral
 that tells you whether or not the ray even reaches that location.
 So the ray reaches the first location, at which point it's going to adopt the color.
 And after that, the it even though there is stuff right, even though the density is high, the ray is not reaching it.
 So the whole formula captures all of this.
 And as we said, with a bit of nuance, it like if this is not always zero one, it can handle transparency as well.
 And here they demonstrate again from the scene.
 So you have two different points in the same scene, but viewed from different locations.
 And on the right, they show you this is all the same point in the scene, but the circle represents kind of different angles
 at which you can view it from.
 And you can see that the color is really different depending on the angle where you look from.
 There are what do we have here?
 There are a lot of tricks.
 Oh, yeah, so they they approximate the integral with like quadrature, which also has existed.
 And they have a bunch of tricks.
 So the first trick to really get this to work is a novel like not a novel, but kind of the employment of a positional encoding
 that a positional encoding is not the same as you might know it from Transformers or something.
 The positional encoding here, it simply means that you send the input data point, which is this thing right here.
 XYZ, theta, phi, Greek letter.
 You send that to a higher dimensional space, right, in a very deterministic way.
 So if you have these low dimensional input, and especially if you want to represent this, this is really fine structure right here.
 You can see that this stuff right here, it's quite fine grained.
 OK, and so you need a way to handle fine differences between things.
 But you also need a way to handle, you know, course differences.
 And just a single floating point number probably isn't going to do it for a continuous function like this.
 So what you do is you send this to a higher dimensionality with these positional encodings that we know from Transformers.
 So these encodings right here, they will send.
 So what you do, and so in my video on attention is all you need, I explain those in detail.
 But you construct a hierarchy of sine waves or sine and cosine waves.
 But we can just do it with sine waves.
 So the lowest hierarchy is like this.
 And then the next thing in the hierarchy would be like double as fast.
 And then the next thing, well, this is four times as fast, isn't it?
 Well, you get the point, right?
 It's so I need up, down, up, wow.
 And then up, down, up, down, up.
 This is not a sine wave.
 But you I hope you get the point.
 And then you want to take a look, for example, your X, you take your X, you put it here like, OK,
 X is so this is like negative.
 I think they go from negative one to one.
 The coordinates they have and your high dimensional output is going to be, you know, this point, this point, this point and this point in the in their respective coordinate systems.
 Right. So that's you can.
 What this does is you can still clearly identify every point here.
 In fact, yeah, you can.
 You can identify every single point in your input space by, you know, looking at looking at the combination of where it is in the sine waves.
 But it gives the network a better chance to focus, for example, on details.
 If it wants to focus on details, it's going to look at this scale right here because tiny changes in the underlying X is going to result in a large change in this feature.
 If you want to focus on coarse grain stuff, then you look at this where you can, you know, you have to move pretty far to have a change.
 Whereas if you look at this scale for coarse grain things, it means almost nothing because, you know, if you want to make little difference between these two things,
 if you look at coarse grained structure, but they have, as you can see, like there's a lot of difference between those like this may be zero and this is maybe negative one.
 However, if you look at the two data points right here, sorry about that.
 So the same, let's say the orange distance and the blue distance, you can see that the two aren't so different in this representation.
 So it gives the network the choice at which scale it wants to look at for particular positions.
 So ultimately, you're going to map this five dimensional vector into a higher dimensional vector.
 And they consider like 10, 10 layers or four layers of these.
 How many of these different sine wave and cosine waves they construct.
 So again, they call it positional ketting. They say this is referred to as a positional encoding.
 However, transformers use it for a different goal of providing discrete representations as input to an architecture, yada, yada, yada.
 In contrast, we use these functions to map continuous input coordinates into a higher dimensional space to enable our MLP to more easily approximate a higher frequency functions.
 The second thing they do is they do hierarchical volume sampling.
 So when we said I send a ray through the scene and then I sample along, this either would take a lot of time or it would not be accurate enough.
 So what they do is they have actually two layers of neural network, one they call a course and one they call a fine.
 And as I understand it, here is a ray they first sample with the course one at rather coarse locations.
 And then they use that to evaluate where they should sample more.
 Let's say this thing right here has a real high density in the course network.
 They then sample around that a lot more, maybe one here, two, but a lot more, you know, sampling around where the course network things, the important stuff is.
 They optimize both networks at the same time.
 And that actually works out well.
 So here you see the loss. The loss is a combination now of the coarse network and the fine grain network.
 And you need to optimize both, even though the final view is only going to come from the fine grain network.
 You need to optimize both because the coarse grain network can tell you where the important stuff is.
 So the results you have already seen, there are a bunch of metrics that prove that this one is really good.
 And it can, as you can see, like you can handle fine grain structure right here in the microphone that others can't.
 And it also so they say it fits into a few. So one neural network of one scene fits into like a few megabytes.
 And this is so it fits into five megabytes.
 And this is a lot better than things that use like voxel grid representations, which I think this other thing they compare to uses over 15 gigabytes for the same scene.
 Which and this is interesting, which is even less memory than the input images alone for a single scene from any of our data sets.
 So this is really like it's it's really it's even smaller than the pictures.
 So so even if you maybe want to show this to another human, it'd be better.
 You send the train nerf than the pictures if space is a consideration, though I don't know how they measure the pictures.
 Like you can probably compress if it's different pictures from the same scene.
 I guess there's some compression potential if you want to transmit them as a never mind.
 So they also do ablations. And the only downside here is that it does take a long time to fit one of these neural networks.
 I don't exactly remember where they say it, but they say they calculate like, oh, here.
 So it's not too bad, but the optimization for a single scene typically take around 100 to 300 K iterations to converge on a single video of 100 GPU, which is about one to two days.
 So it's a single GPU. So it is, you know, you don't need a data center for it.
 But you're going to wait a while until you train one, though you only need to train it once and then you can render new views as you please.
 So the idea, I think, is going to be that let's say you make a video game or so.
 You're going to render this at your servers, then you transmit the neural network to the clients and the clients can just render it out right there.
 And yeah, there's a bunch of results and a bunch of ablations where they kind of leave away different parts and they show that especially kind of the positional encodings.
 I think this is the positional encodings are really important, as you can see on the right, there is no positional encodings.
 The view dependence is also quite important. You see if there's no view dependence, as you can see here, you do get the fine grain structure since you do have positional encodings.
 But you don't get these kind of light effects, right? This is this thing here is not a different color.
 It's simply the fact that the line light shines on it. And it's just not there here because, you know, all the network can do is output the same color for all directions.
 And most directions simply don't have that reflection.
 All right, so that is it. The code is available on this website that I've showed you. I'm certainly going to link it. Tell me what you think.
 I think this is pretty cool. I know this has given rise to a lot of work following up on this.
 I have very little overview over what's going on in the nerf space, but I think it's cool and I want to dive deeper into it.
 Thanks for being here. Bye bye.