Full Self-Driving is HARD! Analyzing Elon Musk re: Tesla Autopilot on Lex Fridman's Podcast
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper lex fridman elon musk elon musk tesla fsd when will fsd ship when will fsd be ready tesla fsd release tesla fsd release date how does tesla autopilot work does tesla use neural networks andrej karpathy self driving tesla self driving how good is tesla fsd how safe is tesla vector space podcast analysis elon musk self-driving how good is tesla autopilot
#tesla #fsd #elon

Watch the original podcast: https://www.youtube.com/watch?v=DxREm3s1scA

An analysis of Elon's appearance on Lex Fridman. Very interesting conversation and a good overview of past, current, and future versions of Tesla's Autopilot system.

OUTLINE:
0:00 - Intro
0:40 - Tesla Autopilot: How hard is it?
9:05 - Building an accurate understanding of the world
16:25 - History of Tesla's neural network stack
26:00 - When is full self-driving ready?
29:55 - FSD 11: Less code, more neural networks
37:00 - Auto-labelling is essential
39:05 - Tesla Bot & Discussion


Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hey, how's everyone doing today? We're going to analyze Elon Musk's appearance on the Lex Friedman podcast. Specifically, we're going to look at the part where Elon talks about the Tesla autopilot and to a certain degree, also the Tesla bot. We've previously analyzed the talk by Andrej Karpati about what kind of architectures and so on goes into the Tesla self-driving system. And this naturally progresses over time. So Elon's going to drop some more hints here. What exactly is going on under the hood? We're going to dive right in. Let me know if you enjoy talk analysis or not. Who knows? All I know is that whenever you put Elon Musk on something, you get insanely many clicks. So thank you for that. Autopilot. Tesla autopilot. I love how they go like autopilot and then both are like, yeah, as if they're saying like, yeah, like, like, like that's ever going to work. As you might know, autopilot is a bit behind schedule. It's been promised again and again and again, especially the full self-driving sort of autopilot. But there also has been insanely much progress. Like no one is pushing that. People have told me, you know, other car companies are doing it as well. Yeah, but no one's kind of pushing it quite like that. And sure, there are some risks to to go along with rolling out alpha and beta versions just to users. But I mean, come on. And so there is a natural skepticism. When I first drove a Tesla with the initial system based on Mobileye, I thought there's no way. So first, when I got in, I thought there's no way this car could maintain like stay in the lane and create a comfortable experience. OK, so I didn't know that the first system was based on on Mobileye, which is interesting because at one point during my PhD, we got visit from a researcher who also worked on Mobileye. I won't name the researcher here because I might be about to tell some stuff that would get them into trouble. But they showed us a video of themselves in a car. I remember this vividly. And the car was just kind of opened. The whole dashboard was opened. All the cables were like hanging out and going into some laptop that was just kind of dangling on sort of the the middle of the car, you know, where the stick, I don't know what what you call that stuff in. In English, it was like a super instable setup and, you know, a cable flying around everywhere. And then the camera kind of pans up and you can see that car is on the highway, like middle of the highway. Car is here, car is here and just driving itself. You see the steering wheel, no hands on it. And it was insane. Like when I when I saw this, I never expected technology to be this far already. And yes, I know in the 70s and 80s, people have done self-driving on highways. But still, for someone to trust the system enough to essentially sit there and let the system steer the car based on nothing but cameras was insane. This system is just the beginning, like the baseline for the Tesla system. I didn't know that. And I thought it was an interesting story to tell. I was already super impressed by the Mobilize system. Yet, as you will see, this has been surpassed a lot. What are some insights you've gained over those five, six years of autopilot about the problem of autonomous driving? So you leaped in having some sort of first principles kinds of intuitions, but nobody knows how difficult the problem is. I thought the self-driving problem would be hard, but it was harder than I thought. It's not like I thought it would be easy. I thought it would be very hard, but it was actually way harder than even that. So what it comes down to at the end of the day is to solve self-driving, you have to solve... You basically need to recreate what humans do to drive, which is humans drive with optical sensors, eyes, and biological neural nets. And so in order to... That's how the entire road system is designed to work, with basically passive optical and neural nets, biologically. And now that we need to... So actually for full self-driving to work, we have to recreate that in digital form. So we have to... So the argument here is, I guess, if you want to solve the self-driving problem, you need to essentially do what humans do. And I'm not exactly buying this argument, just because humans only drive with vision, especially just because humans have neural networks. We also must use neural networks. That seems a bit shady, but there is a point to it, right? That the whole road system and cars and whatnot are designed around human capabilities and vision and audio and stuff like this. And therefore, yes, it's good to drive if you have like a radar and a lidar and whatnot, that's additional sensors, but you're not going to get around building in the human sensors as well. So a car that just drives mainly on radar or lidar is probably good at avoiding obstacles that are just on the road somewhere, but it's not going to be able to see any signs. It's not going to be able to sort of make sense of the world visually, understand what's going on and things like this, which if something's speeding along, coming along, and you can anticipate it by vision, it's probably a lot better than you having to somehow detect it on the radar. So I think that's a fair point right here. But humans having neural network, therefore, we must have neural network. I'm not super sure that's valid. How much game theoretic kind of stuff needs to be involved at a four-way stop sign? As humans, when we drive, our actions affect the world. It changes how others behave. Most of the time, when driving, you're usually just responding to the scene as opposed to really asserting yourself in the scene. Do you think... I think these sort of control logic conundrums are not the hard part. What do you think is the hard part in this whole beautiful complex problem? So it's a lot of freaking software, man. A lot of smart lines of code. For sure, in order to have... Create an accurate vector space. So like you're coming from image space, which is... So I think Elon's gonna make the point here that... What Lex's concern is that there's a lot of game theoretic stuff. And he mentions the four-way crossroads. And then you sort of have to communicate who goes first, who goes last, and so on. And Elon says that that's not the big problem in self-driving. He's gonna make the point that once you do have an accurate representation of the world, once you know where every car is and so on, what every sign means, that you can figure this stuff out easily. And I think I agree. At least the number of situations you can broadly cover with programming heuristics is sort of countable. And I would guess that that would work. Though I'm not super sure if that goes all the way. Because there is game theoretic stuff. Like you can, you know, change a lane based on the fact that you know, kind of game theoretically, that other people won't sort of cut you off while you do it, because they'd crash their car and so on. Which you can't just know by looking at their speeds and the positions of the cars. Sort of the anticipation of how everyone else is going to react in certain situations is, I think, a big part of driving and also a big part of sort of predicting dangers. So I'm not super sure if you can just hard code all of that. But I think saying that, you know, the perception problem is conceptually the harder problem. Because for the perception problem, there isn't even an approach with regular programming, right? You have to sort of learn it then. Yes, if you make a mistake in the perception problem, that's going to have vast downstream effects. So I do agree here that probably the self-driving problem might at least at this time, largely be a computer vision, or let's say, not only vision, but sort of world understanding perception problem. After that, it becomes sort of easier. Once you have an accurate vector space, the control problem is similar to that of a video game, like a Grand Theft Auto or Cyberpunk. Oh, yeah. Yes, I want my traffic management system. I want my self-driving system to be the one from cyberpunk, please. Lord help us, please. Yeah, I mean, point taken, right? What Elon calls vector space right here, I guess you'd sort of call a scene understanding, a scene graph, you know, anything like this. Essentially, where are the objects in the scene, sort of what's their position, their momentum, I guess, you know, where are the signs, what do they mean, where are the traffic lights, all of this kind of stuff. Once you have that, the problem of sort of planning ahead what you should do becomes probably relatively easy, at least compared to that perception problem. Like when's the last time you looked right and left, you know, or and rearward, or even diagonally, you know, forward to actually refresh your vector space. So you're glancing around and what your mind is doing is trying to distill the relevant vectors, basically objects with a position and motion. And then editing that down to the least amount that's necessary for you to drive. It does seem to be able to edit it down or compress it even further into things like concepts. So it's not, it's like it goes beyond, the human mind seems to go sometimes beyond vector space, to sort of space of concepts, to where you'll see a thing, it's no longer represented spatially somehow. It's almost like a concept that you should be aware of. Like if this is a school zone, you'll remember that as a concept, which is a... That's a really good point. So Elon made the point essentially that what your brain is doing and therefore what, you know, the AI should be doing is take all that information and build what Elon calls this vector space, which is, as he said, sort of objects and their motions. But Lex goes a step further and says, well, you also know sort of that this is a school zone. And in a school zone, not only should I be driving slower, but there might be children around. So I need to be sort of careful. I in fact, adapt my attention and my vision on different things than if something like, then if it's a highway. And I think that is as of yet, probably not considered by these AI systems. I'm pretty sure they, the input feed is all the same, no matter whether it's a school zone or whether it is a highway. Of course, there's different things. Us humans have limited amounts of attention and Elon just pointed out, sort of all the ways in which your system is screwed up like blind spots and yada, yada, yada. And that might be the reason why we have to sort of focus our attention on different things. And, you know, depending on where we are. So it could be that the machines are just, you know, they don't care. They can always pay attention to everything. And therefore, this is not a concern to them. I'm not entirely convinced by this. The sort of guiding of attention and sort of the top down feedback loop to the lower systems, I think is as of yet, completely missing from the AI systems. I'm not sure actually. Maybe they do sort of feed, let's say they know they're in a school zone. They know, you know, the speed limit is such and such and, or there's a construction site. Maybe they feed sort of embeddings of this stuff into sort of the vision networks. And the vision networks might be able to adjust sort of their attention patterns. Not that probably they don't use attention. They probably use con nets or so. But it would be interesting to see if that was happening. I would be very surprised if it was though. So not sure. This might be a fundamental limitation. It might be that without this, the driving problem is essentially unsolvable or, or there's, there's major hurdles that can't be overcome. It could also be that just, you know, the machines can always pay attention to everything. And therefore it just doesn't matter. You saw that there were some kids about to cross the road in front of the truck. Now you can no longer see the kids, but you, you need to be able, but you would now know, okay, those kids are probably going to pass by the truck and cross the road, even though you cannot see them. So you have to have, um, memory, uh, you have to need to remember that there were kids there and you need to have some forward prediction of what their position will be. It's a really hard problem. I mean, yeah, exactly. So they're going to talk about occlusions here, occlusions, uh, detecting occluded objects and so on. But I think Elon's point is bigger than that. You need to have a forward predicting model in order to do the self driving, you know, solve the self driving problem to a realistic degree. And here I would, you know, challenge zero to your statement that once you have the vector space, the problem is sort of, you know, not that hard. I think this particular part of the remaining problem is actually quite hard in itself because it's not like you can just calculate the Nash equilibrium of self driving and then assume that everyone's acting rationally. You have to sort of take into account all the human factors right here and how you expect other humans to act, be that pedestrians or other drivers or anything like this. Yeah, I think this is another area, this sort of forward prediction where neuro-sensory prediction where neural net or in general machine learning is going to make a big difference. And then as I said, I'd be wondering if there is sort of a top down feedback loop that as you're predicting forward, you're going to change sort of the perception pipeline on the fly or not. But like, let's say you, you're parked at a light and you, and you saw, you use a pedestrian example that people were waiting to cross the, across the road and you can't, you can't quite see them because of an occlusion. But they might wait for a minute before the light changes for them to cross the road. You still need to remember that that's where they were and that they're probably going to cross the road type of thing. So even if that exceeds your time-based memory, it should not exceed your space memory. And I just think the data engine side of that, so getting the data to learn all of the concepts that you're saying now is an incredible process. It's this iterative process of just... And I just think... So what he said right there, I think is quite important as well. You know, you can probably understand it in the concept. If you do reinforcement learning, let's say you did reinforcement learning in this thing, typically in reinforcement learning, we have a finite amount of time where you can go back over time and still be able to do back propagation, especially if you're at like a high frame rate like these systems operate right here. That's not going to be a long time. It's not going to be a minute of real time. And therefore, yes, if you need to learn to remember something like there are pedestrians right there and they're still there a minute later because all the lights were red, that is going to be quite a bit of a problem and a challenge in itself. Sort of learning to remember things is a long-standing challenge in reinforcement learning. And you probably be better off sort of coding all the objects in this, what Elon calls the vector space. So understand the scene and then explicitly representing each object that's there rather than having the neural networks learn everything from perception. I think the data engine side of that, so getting the data to learn all the concepts that you're saying now is an incredible process. It's this iterative process of just... This is HydroNet, many... HydroNet. We're changing the name to something else. Okay. I'm sure it'll be equally as Rick and Morty like... There's a lot of... Yeah. We've re-architected the neural net in the cars so many times. It's crazy. Oh, so every time there's a new major version, you'll rename it to something more ridiculous or memorable and beautiful? Sorry. Not ridiculous, of course. If you see the full array of neural nets that are operating in the cars, it boggles the mind. There's so many layers, it's crazy. What is he actually saying here? It's hard to decipher Elon because obviously he's not a deep learning engineer, so he sort of probably gets the pitch from Andre and some diagrams or something like this. But as of now, we don't know if there are many neural nets, but it's unlikely because he says it's mind bogglingly many and you'd have to sort of train all of them. I couldn't really imagine how you'd put mind bogglingly many neural networks into a system like this. I'm going to guess that they have a couple and these are just kind of big and complicated. And that's exactly what we saw in Karpati's talk when he explained how they go vision only and so on. If you haven't seen this, watch my analysis of that. He's about to explain a bit more in depth of what's going on. We started off with simple neural nets that were basically image recognition on a single frame from a single camera and then trying to knit those together with C. I should say we're primarily running C here because C++ is too much overhead and we have our own C compiler. So to get maximum performance, we actually wrote our own C compiler and are continuing to optimize our C compiler for maximum efficiency. In fact, we've just recently done a new rev on a C compiler that will compile directly to our autopilot hardware. So you want to compile the whole thing down? I mean, he's going to talk about two things kind of interleaved right here that have on the surface not too much to do with each other. So apparently there is a C compiler that compiles directly to the hardware, which makes sense, right? These cars have the property that you have to be super duper efficient and power saving and whatnot. And running Python on top of that, the overhead of that might just be too much. You can in fact save a lot of energy, a lot of time and so on by building a compiler that uses the hardware as optimally as possible. Now that being said, this has little to do with how you build the neural network system other than the neural networks will be faster if you compile them down correctly. And so there's actually a lot of work done by some very talented software engineers at Tesla at a very foundational level to improve the efficiency of compute and how we use the trip accelerators, which are basically doing matrix math dot products like a bazillion dot products. And it's like what are neural nets, it's like compute wise like 99% dot products. So yeah, I mean, he's obviously correct right here, though it has to be said, you know, for anyone who's listening to this, your neural network isn't slow because you don't have the right compiler. It is true that if you do it correctly, you compile your network down to like a format that is optimal for some hardware and you run it with you know, the correct libraries and and you set up everything correctly, you can probably get like maybe if you if you did if you did it terribly wrong, and then you do it terribly right, you can get up to a 10x speed up I would guess maybe you know, 5x 10x speed up something like this best case. However, usually, usually, the first thing you should investigate is whether or not the architecture you're using is the correct one. You can get like many, many more times a speed up by simply changing the architecture to something more appropriate. So Elon says this here, because obviously, this is the last step. And you know, they need to they need to get every, every millisecond they can out of these systems. But just for most people listening, this is sort of the the sugar, the icing on the cake, you should first care about the cake and try to make your architecture, you know, more optimal, maybe use less layers or anything like this change from this operation to that operation analyze your bottlenecks. And only once you have everything through and you have the exact model you want, then you can care about doing all the engineering things. One of the things we're moving towards now is no post processing of the image through the image signal processor. So like, what happens for cameras is that almost all cameras is they there's a lot of post processing done in order to make pictures look pretty. And so we don't care about pictures looking pretty. We just want the data. So we're moving just roll photon counts. So the system will like the image that that the computer sees is actually much more than what you'd see if you represented on a camera. It's got much more data. And even in very low light conditions, you can see that there's a small photon count difference between, you know, this spot here and that spot there, which means that so it can see in the dark incredibly well, because it can detect these tiny differences in photon counts. That's much better than you could possibly imagine. So I mean, that is, again, like that is a third issue next to the the C compiler. And what the neural networks do is essentially saying that if you remove the post processing within the camera sensors that are usually built into, let's say cameras that you could buy on the market, then you get the raw data. And since you don't have to look at the pictures, the raw data is much more useful than the post process data, since it's a machine anyway, that analyzes the signal. And therefore, you might as well make it machine friendly. I think it is a good lesson for maybe other fields as well to think about, you know, what parts of the pipeline are just there to make it, you know, because because humans are involved and try to remove those. But you know, it doesn't really add to what's the what's the deal with the neural networks, which I think was the original question here. And then we also save 13 milliseconds on latency. So from removing the post processing an image? Yes. Yeah. It's like because we've got eight cameras and then there's roughly, I don't know, one and a half milliseconds or so, maybe one point six milliseconds of latency for each camera. And so like going to just basically bypassing the image processor gets us back 13 milliseconds of latency, which is important. Yeah, I think this, you know, besides getting the raw data, this is also again, they need to squeeze out sort of the last mile here or the last milliseconds here. And this is another thing they they can practically do. So getting rid of jitter is extremely important. And that affects your control decisions and all those kinds of things. OK. Yeah, the cars is going to fundamentally maneuver better with lower jitter. The cars will maneuver with superhuman ability and reaction time much faster than a human. I mean, I think over time, the autopilot full self driving will be capable of maneuvers that are far more than what James Bond could do in the best movie type of thing. That's exactly what I was imagining in my mind, as you said. It's like impossible maneuvers that a human couldn't do. Well, OK, it's two things. Impossible maneuvers are impossible and things that humans could do are things that humans could do. I have no doubt that at one point in the near future, self driving cars will be able to do things that humans couldn't do. The question is more, are there going to be things that humans do that the cars couldn't do? Right. Or can't do? Because that's the actual gap you're trying to close. You know, look at Boston Dynamics or so. If you hard code stuff and you have extremely, extremely good sensors and actuators, you can do many things that humans couldn't do. But on the other hand, it's the things that humans can do that the machines can't. Those are the problem. Well, let me ask sort of looking back the six years, looking out into the future, based on your current understanding, how hard do you think this full self driving problem, when do you think Tesla will solve level four FSD? I think Elon gets asked this question every year and every year he says next year. So I mean, it's looking quite likely that it will be next year. This is the thing with Elon Musk, he always promises things like next year or on ridiculously short amounts of time. And I wonder how long it's going to take for people to just, you know, stop believing him. I guess many people already did. But it's still, you know, a thing to consider that on one hand, obviously, if you do it too much, then people are simply going to say, oh, well, probably in five years if he says next year. But on the other hand, he's also able to sort of it's a motivating thing. It's a cool thing. It drives momentum. And that itself accelerates the development of these things, people being ready to just flip on a beta version and so on. It's a bit insane. But I do think his optimism and a little bit salesmanship also a lot of benefits besides the obvious negatives. So the interventions, you know, per million miles has been dropping dramatically at some point. And that trend looks like it happens next year is that the probability of an accident on FSD is less than that of the average human and then significantly less than that of the average human. So it certainly appears like we will get there next year. There's a lot of hedging going on here. But you know, you can this is this is actually a nice method, I think, of making these types of predictions, you see that the rate of disengagement is dropping at a certain speed, you can extrapolate maybe a little bit and say, look, you know, here's going to be the sort of threshold where we're better than a human. I think that's a quite a sober analysis if done correctly. And I also think people who are, you know, it's obviously good to be skeptical of fully self driving systems. But on the other hand, you also have to think if they're a lot better than humans, it makes makes total sense, right? It also makes total sense to have them and not engage them all the time, right? There might still be situations you want to drive yourself. The question is a little bit, can you just continue the trend? Or is there a sort of an okay, you solve the easy problems. And that is what makes the rates of disengagement go down now. But now come the more and more hard problems and sort of it gets exponentially harder to continue that trend, in which case, we're not going to be there for a long time. Then there's going to be a case of, okay, we'll not have to prove this to regulators and prove it to you know, and we want a standard that is not just equivalent to a human, but much better than the average human, I think it's got to be at least two or three times higher safety than a human. Probably more like 10, like knowing, you know, regulators and how the public perceives these types of things. Of course, right now they're cool, but then it's really easy to publicize in a few accidents that few stupid accidents that happen if you build machine learning systems for the real world, they are going to make stupid mistakes. It doesn't matter how accurate they are on average, they're going to make stupid mistakes that a human would never do and people are just going to point at it and never forget that one instance. And I think it's pretty easy to sort of scare people publicizing those kinds of things. And therefore, yeah, you have to be like massively better than humans. I agree here. There is some fundamental leap that really deserves the 11. I mean, that's a pretty cool number. Yeah. 11 would be a single stack for all, one stack to rule them all. But there are just some really fundamental neural net architecture changes that will allow for much more capability, but at first they're going to have issues. So we have this working on like sort of alpha software and it's good, but it's basically taking a whole bunch of C++ code and deleting a massive amount of C++ code and replacing it with a neural net. And Andrei makes this point a lot, which is like neural nets are kind of eating software. So it's interesting what Elon says right here. This upcoming version 11 of the Tesla software seems to have kind of a rewrite in what he calls the creation of the vector space. And specifically, he says you replace a whole bunch of C and C++ code with neural networks. And I guess what that means is that they used to have certain heuristics for what he calls creating the vector space, right? And remember, creating the vector space means seeing and understanding. So what objects exist? Where are they? How are they moving? And so on. And you want to get that out of your cameras and whatever other sensors you have. So it seems like until now, they had a bunch of neural networks that would do, you know, their stuff. I can imagine they had maybe single frame neural networks or kind of short frames, one after another neural networks that would recognize sort of bounding boxing the objects in the image. And then they would use sort of an algorithm heuristic algorithm that they wrote themselves to stitch that together over time. Maybe they use algorithms to do some kind of inferences like what he mentioned with the object tracking, and so on. And it seems to be that what they want to do is just end to end train one big neural network that just does it all. You input all of the sensor data, let's say from, you know, not only just right now, but you know, from the from the recent past, you just input it all in there. And the neural network will spit out this finished vector space, this finished scene understanding graph. And this obviously you can see where it comes from. This has been the story of deep learning so far, replacing more and more classical heuristics with an end to end learning system. And it also matches exactly with what Elon is saying, namely that right now, it doesn't seem to work quite well yet, but in time, it will get there. And again, this has been the story of deep learning in pretty much everything we've tackled since the beginning of deep learning. End to end systems ultimately came to be the heuristic systems, but it takes time, it takes work, it takes data, obviously massive amounts of compute. You know, over time, there's like, less and less conventional software, more and more neural net, which is still software, but it's, you know, still comes out the lines of software, but it's more more neural net stuff, and less, you know, heuristics, basically. If you're more more more matrix based stuff, and less heuristics based stuff. So by the way, the reason why this is the case, the reason why it works to replace heuristics with neural networks with data driven systems is that the world is always more complicated than you can encode in any heuristic. That's why we use machine learning in the first place, because we can't just program the algorithms that do image recognition, or speech recognition or whatnot. So the only representation of this really complex world, like the actual underlying world that is so complicated is the data. And therefore, our best chance to create systems that deal well with the world as such is systems that actually learn from data from the real world. And that's why it often works to replace the heuristics with data driven systems. If you have the data, and if you have the compute, which Tesla obviously does. We call it the giant bag of points. And it's like, so you go to pixel and something associated with that pixel, like this pixel is probably car, the pixel is probably lane line. Then you've got to assemble this giant bag of points in the C code and turn it into vectors. And it does a pretty good job of it, but we need another layer of neural nets on top of that to take the giant bag of points and distill that down to vector space in the neural net part of the software as opposed to the heuristics part of the software. So the translation of this is probably, if I understand Elon correctly, what they were doing so far is sort of semantic segmentation or pixel based pixel labeling. I can also imagine that they estimated things like depth maps and so on just from pixels. But then, as I said before, it was heuristics, it was sort of classical algorithms. And these aren't, I mean, classical, these are advanced algorithms, right, that take point clouds that take sort of segmentation maps and depth maps and all of that and turn them into objects. These are mostly heuristic based but very sophisticated algorithms. But it is clearly a good or a, let's say a modern move to ditch all of that and also teach the neural networks to just handle it until you have the semantic result that you want, namely the space of objects, the scene understanding graph. It's really outputting proper vectors to the CC++ control code, as opposed to the sort of constructing the vectors in C. We've done, I think, quite a good job of, but it's kind of hitting a local maximum on how well the C can do this. So this is really a big deal. And just all of the networks in the car need to... By the way, whenever you hear him talk about C and C++ code, just replace that with human authored code, right? The difference isn't necessarily the language you use, the difference is more like who writes the code. And when he says C and C++, it's humans, very smart humans, but still humans that write the code out of their thinking. And whenever he says neural networks, it's some sort of a data-driven systems, which obviously human author in the first place, but probably also is as well implemented in C and C++. The training, the amount of work done with... We've written all this custom software for training and labeling and to do auto labeling. Auto labeling is essential, especially when you've got surround video. It's very difficult to label surround video from scratch. It's extremely difficult. Like a human's such a long time to even label one video clip, like several hours. Or the auto label it, basically we just apply a heavy duty, like a lot of compute to the video clips to pre-assign and guess what all the things are that are going on in the surround video. And then there's like correcting it. Yeah. And then all the human has to do is like tweet, like say, adjust what is incorrect. This is like increase this productivity by effect a hundred or more. Yeah. So you've presented that... I mean, we've discussed this in the last video that I did about Karpotty's talk. And this to me is, I think too few people are currently doing something like this. Essentially it's active learning, right? It's sort of, if you're not sure about something, ask the human. It has a slight twist on it in that they probably always ask the human, but they suggest a label which is super powerful, especially in something like semantic segmentation where you need to annotate every pixel or you need to place bounding boxes around many objects. It's really different if you simply have to check and adjust a little bit versus if, you know, there's a data point and you have to place the labels yourself. I think we're going to see quite a bit more of that in sort of the near future. A lot of people are already doing something like this, but I think still too few are. It's not quite in Tesla's primary mission direction of accelerating sustainable energy, but it is an extremely useful thing that we can do for the world, which is to make a useful humanoid robot that is capable of interacting with the world. All right. The rest of them talking about AI is talking about the Tesla bot, which is a bit more far fetched I have to say. The Tesla bot just on its face is way more complicated than a car, especially if it is supposed to not only, you know, be on the factory floor in which case they just build like a robot arm, right? These are like the most useful things in a factory on a factory floor. But if it's actually to sort of interact with humans or in a human way navigate not only unknown terrain, but also society potentially. I mean, this is just this is just futurism at this point and that there's really nothing we can legitimately say about what's possible, what's not possible, where this is. And obviously they like we don't we don't have a prototype. We just have like a human in a suit to demonstrate the Tesla bot. So I will not comment much further on that with respect to the Tesla fully self driving system. I would say that obviously, you know, for Elon Musk, there's always kind of lovers and haters and I think you can acknowledge both sides. He is a bit of a salesperson. He sells these things very well. He always promises, you know, next year we'll be ready, next year we'll be ready. And then they never are or he over promises massively on you know, how much cost you can save and yada, yada, yada. But then on the other hand, he also delivers a lot more than other people deliver. Maybe that's just because a little bit of recklessness, but also the sort of optimism and momentum that he's able to to to come up and drive. And all of that together, I think just makes for like an interesting person. And I think the advances itself are remarkable. Even if you say other car companies are on the track and whatnot, Tesla has done more than all other car companies together for the adoption of electric vehicles. Yes, you can debate whether or not that in itself is a good thing. But just to say that it's not only salesmanship, there are also results. And I have no doubt that in the near future, we will see self driving cars. Sure, they're not going to be accident free, but I believe they will be much, much better than humans. And the question is simply is this next year in two years in five years? I cannot tell you, but I'm excited to see. I hope you like this talk analysis interview analysis. If you want more of these things, let me know. Otherwise, let me know what you think in the comments and I'll see you next time. Bye bye.
 Hey, how's everyone doing today?
 We're going to analyze Elon Musk's appearance on the Lex Friedman podcast.
 Specifically, we're going to look at the part where Elon talks about the Tesla
 autopilot and to a certain degree, also the Tesla bot.
 We've previously analyzed the talk by Andrej Karpati about what kind of
 architectures and so on goes into the Tesla self-driving system.
 And this naturally progresses over time.
 So Elon's going to drop some more hints here.
 What exactly is going on under the hood?
 We're going to dive right in.
 Let me know if you enjoy talk analysis or not.
 Who knows? All I know is that whenever you put Elon Musk on something,
 you get insanely many clicks.
 So thank you for that.
 Autopilot.
 Tesla autopilot.
 I love how they go like autopilot and then both are like, yeah,
 as if they're saying like, yeah, like, like, like that's ever going to work.
 As you might know, autopilot is a bit behind schedule.
 It's been promised again and again and again, especially the full
 self-driving sort of autopilot.
 But there also has been insanely much progress.
 Like no one is pushing that.
 People have told me, you know, other car companies are doing it as well.
 Yeah, but no one's kind of pushing it quite like that.
 And sure, there are some risks to to go along with rolling out
 alpha and beta versions just to users.
 But I mean, come on.
 And so there is a natural skepticism.
 When I first drove a Tesla with the initial system based on Mobileye,
 I thought there's no way.
 So first, when I got in, I thought there's no way this car could maintain
 like stay in the lane and create a comfortable experience.
 OK, so I didn't know that the first system was based on on Mobileye,
 which is interesting because at one point during my PhD,
 we got visit from a researcher who also worked on Mobileye.
 I won't name the researcher here because I might be about to tell some stuff
 that would get them into trouble.
 But they showed us a video of themselves in a car.
 I remember this vividly.
 And the car was just kind of opened.
 The whole dashboard was opened.
 All the cables were like hanging out and going into some laptop
 that was just kind of dangling on sort of the the middle of the car,
 you know, where the stick, I don't know what what you call that stuff in.
 In English, it was like a super instable setup and, you know,
 a cable flying around everywhere.
 And then the camera kind of pans up and you can see that car is on the highway,
 like middle of the highway.
 Car is here, car is here and just driving itself.
 You see the steering wheel, no hands on it.
 And it was insane.
 Like when I when I saw this, I never expected technology to be this far already.
 And yes, I know in the 70s and 80s,
 people have done self-driving on highways.
 But still, for someone to trust the system enough to essentially sit there
 and let the system steer the car based on nothing but cameras was insane.
 This system is just the beginning, like the baseline for the Tesla system.
 I didn't know that.
 And I thought it was an interesting story to tell.
 I was already super impressed by the Mobilize system.
 Yet, as you will see, this has been surpassed a lot.
 What are some insights you've gained over those five, six years of autopilot
 about the problem of autonomous driving?
 So you leaped in having some sort of first principles kinds of intuitions,
 but nobody knows how difficult the problem is.
 I thought the self-driving problem would be hard,
 but it was harder than I thought.
 It's not like I thought it would be easy.
 I thought it would be very hard, but it was actually way harder than even that.
 So what it comes down to at the end of the day is to solve self-driving,
 you have to solve...
 You basically need to recreate what humans do to drive,
 which is humans drive with optical sensors, eyes, and biological neural nets.
 And so in order to...
 That's how the entire road system is designed to work,
 with basically passive optical and neural nets, biologically.
 And now that we need to...
 So actually for full self-driving to work, we have to recreate that in digital form.
 So we have to...
 So the argument here is, I guess, if you want to solve the self-driving problem,
 you need to essentially do what humans do.
 And I'm not exactly buying this argument,
 just because humans only drive with vision,
 especially just because humans have neural networks.
 We also must use neural networks.
 That seems a bit shady, but there is a point to it, right?
 That the whole road system and cars and whatnot
 are designed around human capabilities and vision and audio and stuff like this.
 And therefore, yes, it's good to drive if you have like a radar and a lidar and whatnot,
 that's additional sensors,
 but you're not going to get around building in the human sensors as well.
 So a car that just drives mainly on radar or lidar
 is probably good at avoiding obstacles that are just on the road somewhere,
 but it's not going to be able to see any signs.
 It's not going to be able to sort of make sense of the world visually,
 understand what's going on and things like this,
 which if something's speeding along, coming along,
 and you can anticipate it by vision,
 it's probably a lot better than you having to somehow detect it on the radar.
 So I think that's a fair point right here.
 But humans having neural network, therefore, we must have neural network.
 I'm not super sure that's valid.
 How much game theoretic kind of stuff needs to be involved at a four-way stop sign?
 As humans, when we drive, our actions affect the world.
 It changes how others behave.
 Most of the time, when driving, you're usually just responding to the scene
 as opposed to really asserting yourself in the scene.
 Do you think...
 I think these sort of control logic conundrums are not the hard part.
 What do you think is the hard part in this whole beautiful complex problem?
 So it's a lot of freaking software, man.
 A lot of smart lines of code.
 For sure, in order to have...
 Create an accurate vector space.
 So like you're coming from image space, which is...
 So I think Elon's gonna make the point here that...
 What Lex's concern is that there's a lot of game theoretic stuff.
 And he mentions the four-way crossroads.
 And then you sort of have to communicate who goes first, who goes last, and so on.
 And Elon says that that's not the big problem in self-driving.
 He's gonna make the point that once you do have an accurate representation of the world,
 once you know where every car is and so on, what every sign means,
 that you can figure this stuff out easily.
 And I think I agree.
 At least the number of situations you can broadly cover with programming heuristics
 is sort of countable.
 And I would guess that that would work.
 Though I'm not super sure if that goes all the way.
 Because there is game theoretic stuff.
 Like you can, you know, change a lane based on the fact that you know,
 kind of game theoretically, that other people won't sort of cut you off while you do it,
 because they'd crash their car and so on.
 Which you can't just know by looking at their speeds and the positions of the cars.
 Sort of the anticipation of how everyone else is going to react in certain situations
 is, I think, a big part of driving and also a big part of sort of predicting dangers.
 So I'm not super sure if you can just hard code all of that.
 But I think saying that, you know, the perception problem is conceptually the harder problem.
 Because for the perception problem, there isn't even an approach with regular programming, right?
 You have to sort of learn it then.
 Yes, if you make a mistake in the perception problem,
 that's going to have vast downstream effects.
 So I do agree here that probably the self-driving problem might at least at this time,
 largely be a computer vision, or let's say, not only vision,
 but sort of world understanding perception problem.
 After that, it becomes sort of easier.
 Once you have an accurate vector space,
 the control problem is similar to that of a video game, like a Grand Theft Auto or Cyberpunk.
 Oh, yeah.
 Yes, I want my traffic management system.
 I want my self-driving system to be the one from cyberpunk, please.
 Lord help us, please.
 Yeah, I mean, point taken, right?
 What Elon calls vector space right here, I guess you'd sort of call a scene understanding,
 a scene graph, you know, anything like this.
 Essentially, where are the objects in the scene, sort of what's their position,
 their momentum, I guess, you know, where are the signs, what do they mean,
 where are the traffic lights, all of this kind of stuff.
 Once you have that, the problem of sort of planning ahead what you should do
 becomes probably relatively easy, at least compared to that perception problem.
 Like when's the last time you looked right and left, you know, or and rearward,
 or even diagonally, you know, forward to actually refresh your vector space.
 So you're glancing around and what your mind is doing is trying to distill
 the relevant vectors, basically objects with a position and motion.
 And then editing that down to the least amount that's necessary for you to drive.
 It does seem to be able to edit it down or compress it even further into things like concepts.
 So it's not, it's like it goes beyond, the human mind seems to go sometimes beyond vector space,
 to sort of space of concepts, to where you'll see a thing,
 it's no longer represented spatially somehow.
 It's almost like a concept that you should be aware of.
 Like if this is a school zone, you'll remember that as a concept, which is a...
 That's a really good point.
 So Elon made the point essentially that what your brain is doing and therefore what,
 you know, the AI should be doing is take all that information and build what Elon calls
 this vector space, which is, as he said, sort of objects and their motions.
 But Lex goes a step further and says, well, you also know sort of that this is a school zone.
 And in a school zone, not only should I be driving slower, but there might be children around.
 So I need to be sort of careful.
 I in fact, adapt my attention and my vision on different things than if something like,
 then if it's a highway.
 And I think that is as of yet, probably not considered by these AI systems.
 I'm pretty sure they, the input feed is all the same, no matter whether it's a school zone
 or whether it is a highway.
 Of course, there's different things.
 Us humans have limited amounts of attention and Elon just pointed out,
 sort of all the ways in which your system is screwed up like blind spots and yada, yada, yada.
 And that might be the reason why we have to sort of focus our attention on different things.
 And, you know, depending on where we are.
 So it could be that the machines are just, you know, they don't care.
 They can always pay attention to everything.
 And therefore, this is not a concern to them.
 I'm not entirely convinced by this.
 The sort of guiding of attention and sort of the top down feedback loop to the lower systems,
 I think is as of yet, completely missing from the AI systems.
 I'm not sure actually.
 Maybe they do sort of feed, let's say they know they're in a school zone.
 They know, you know, the speed limit is such and such and, or there's a construction site.
 Maybe they feed sort of embeddings of this stuff into sort of the vision networks.
 And the vision networks might be able to adjust sort of their attention patterns.
 Not that probably they don't use attention.
 They probably use con nets or so.
 But it would be interesting to see if that was happening.
 I would be very surprised if it was though.
 So not sure.
 This might be a fundamental limitation.
 It might be that without this, the driving problem is essentially unsolvable or, or there's,
 there's major hurdles that can't be overcome.
 It could also be that just, you know, the machines can always pay attention to everything.
 And therefore it just doesn't matter.
 You saw that there were some kids about to cross the road in front of the truck.
 Now you can no longer see the kids, but you, you need to be able, but you would now know,
 okay, those kids are probably going to pass by the truck and cross the road, even though
 you cannot see them.
 So you have to have, um, memory, uh, you have to need to remember that there were kids there
 and you need to have some forward prediction of what their position will be.
 It's a really hard problem.
 I mean, yeah, exactly.
 So they're going to talk about occlusions here, occlusions, uh, detecting occluded objects
 and so on.
 But I think Elon's point is bigger than that.
 You need to have a forward predicting model in order to do the self driving, you know,
 solve the self driving problem to a realistic degree.
 And here I would, you know, challenge zero to your statement that once you have the vector
 space, the problem is sort of, you know, not that hard.
 I think this particular part of the remaining problem is actually quite hard in itself because
 it's not like you can just calculate the Nash equilibrium of self driving and then assume
 that everyone's acting rationally.
 You have to sort of take into account all the human factors right here and how you expect
 other humans to act, be that pedestrians or other drivers or anything like this.
 Yeah, I think this is another area, this sort of forward prediction where neuro-sensory
 prediction where neural net or in general machine learning is going to make a big difference.
 And then as I said, I'd be wondering if there is sort of a top down feedback loop that as
 you're predicting forward, you're going to change sort of the perception pipeline on
 the fly or not.
 But like, let's say you, you're parked at a light and you, and you saw, you use a pedestrian
 example that people were waiting to cross the, across the road and you can't, you can't
 quite see them because of an occlusion.
 But they might wait for a minute before the light changes for them to cross the road.
 You still need to remember that that's where they were and that they're probably going
 to cross the road type of thing.
 So even if that exceeds your time-based memory, it should not exceed your space memory.
 And I just think the data engine side of that, so getting the data to learn all of the concepts
 that you're saying now is an incredible process.
 It's this iterative process of just...
 And I just think...
 So what he said right there, I think is quite important as well.
 You know, you can probably understand it in the concept.
 If you do reinforcement learning, let's say you did reinforcement learning in this thing,
 typically in reinforcement learning, we have a finite amount of time where you can go back
 over time and still be able to do back propagation, especially if you're at like a high frame
 rate like these systems operate right here.
 That's not going to be a long time.
 It's not going to be a minute of real time.
 And therefore, yes, if you need to learn to remember something like there are pedestrians
 right there and they're still there a minute later because all the lights were red, that
 is going to be quite a bit of a problem and a challenge in itself.
 Sort of learning to remember things is a long-standing challenge in reinforcement learning.
 And you probably be better off sort of coding all the objects in this, what Elon calls the
 vector space.
 So understand the scene and then explicitly representing each object that's there rather
 than having the neural networks learn everything from perception.
 I think the data engine side of that, so getting the data to learn all the concepts that you're
 saying now is an incredible process.
 It's this iterative process of just...
 This is HydroNet, many...
 HydroNet.
 We're changing the name to something else.
 Okay.
 I'm sure it'll be equally as Rick and Morty like...
 There's a lot of...
 Yeah.
 We've re-architected the neural net in the cars so many times.
 It's crazy.
 Oh, so every time there's a new major version, you'll rename it to something more ridiculous
 or memorable and beautiful?
 Sorry.
 Not ridiculous, of course.
 If you see the full array of neural nets that are operating in the cars, it boggles the
 mind.
 There's so many layers, it's crazy.
 What is he actually saying here?
 It's hard to decipher Elon because obviously he's not a deep learning engineer, so he sort
 of probably gets the pitch from Andre and some diagrams or something like this.
 But as of now, we don't know if there are many neural nets, but it's unlikely because
 he says it's mind bogglingly many and you'd have to sort of train all of them.
 I couldn't really imagine how you'd put mind bogglingly many neural networks into a system
 like this.
 I'm going to guess that they have a couple and these are just kind of big and complicated.
 And that's exactly what we saw in Karpati's talk when he explained how they go vision
 only and so on.
 If you haven't seen this, watch my analysis of that.
 He's about to explain a bit more in depth of what's going on.
 We started off with simple neural nets that were basically image recognition on a single
 frame from a single camera and then trying to knit those together with C. I should say
 we're primarily running C here because C++ is too much overhead and we have our own C
 compiler.
 So to get maximum performance, we actually wrote our own C compiler and are continuing
 to optimize our C compiler for maximum efficiency.
 In fact, we've just recently done a new rev on a C compiler that will compile directly
 to our autopilot hardware.
 So you want to compile the whole thing down?
 I mean, he's going to talk about two things kind of interleaved right here that have on
 the surface not too much to do with each other.
 So apparently there is a C compiler that compiles directly to the hardware, which makes sense,
 right?
 These cars have the property that you have to be super duper efficient and power saving
 and whatnot.
 And running Python on top of that, the overhead of that might just be too much.
 You can in fact save a lot of energy, a lot of time and so on by building a compiler that
 uses the hardware as optimally as possible.
 Now that being said, this has little to do with how you build the neural network system
 other than the neural networks will be faster if you compile them down correctly.
 And so there's actually a lot of work done by some very talented software engineers at
 Tesla at a very foundational level to improve the efficiency of compute and how we use the
 trip accelerators, which are basically doing matrix math dot products like a bazillion
 dot products.
 And it's like what are neural nets, it's like compute wise like 99% dot products.
 So yeah, I mean, he's obviously correct right here, though it has to be said, you know,
 for anyone who's listening to this, your neural network isn't slow because you don't have
 the right compiler.
 It is true that if you do it correctly, you compile your network down to like a format
 that is optimal for some hardware and you run it with you know, the correct libraries
 and and you set up everything correctly, you can probably get like maybe if you if you
 did if you did it terribly wrong, and then you do it terribly right, you can get up to
 a 10x speed up I would guess maybe you know, 5x 10x speed up something like this best case.
 However, usually, usually, the first thing you should investigate is whether or not the
 architecture you're using is the correct one.
 You can get like many, many more times a speed up by simply changing the architecture to
 something more appropriate.
 So Elon says this here, because obviously, this is the last step.
 And you know, they need to they need to get every, every millisecond they can out of these
 systems.
 But just for most people listening, this is sort of the the sugar, the icing on the cake,
 you should first care about the cake and try to make your architecture, you know, more
 optimal, maybe use less layers or anything like this change from this operation to that
 operation analyze your bottlenecks.
 And only once you have everything through and you have the exact model you want, then
 you can care about doing all the engineering things.
 One of the things we're moving towards now is no post processing of the image through
 the image signal processor.
 So like, what happens for cameras is that almost all cameras is they there's a lot of
 post processing done in order to make pictures look pretty.
 And so we don't care about pictures looking pretty.
 We just want the data.
 So we're moving just roll photon counts.
 So the system will like the image that that the computer sees is actually much more than
 what you'd see if you represented on a camera.
 It's got much more data.
 And even in very low light conditions, you can see that there's a small photon count
 difference between, you know, this spot here and that spot there, which means that so it
 can see in the dark incredibly well, because it can detect these tiny differences in photon
 counts.
 That's much better than you could possibly imagine.
 So I mean, that is, again, like that is a third issue next to the the C compiler.
 And what the neural networks do is essentially saying that if you remove the post processing
 within the camera sensors that are usually built into, let's say cameras that you could
 buy on the market, then you get the raw data.
 And since you don't have to look at the pictures, the raw data is much more useful than the
 post process data, since it's a machine anyway, that analyzes the signal.
 And therefore, you might as well make it machine friendly.
 I think it is a good lesson for maybe other fields as well to think about, you know, what
 parts of the pipeline are just there to make it, you know, because because humans are involved
 and try to remove those.
 But you know, it doesn't really add to what's the what's the deal with the neural networks,
 which I think was the original question here.
 And then we also save 13 milliseconds on latency.
 So from removing the post processing an image?
 Yes.
 Yeah.
 It's like because we've got eight cameras and then there's roughly, I don't know, one
 and a half milliseconds or so, maybe one point six milliseconds of latency for each camera.
 And so like going to just basically bypassing the image processor gets us back 13 milliseconds
 of latency, which is important.
 Yeah, I think this, you know, besides getting the raw data, this is also again, they need
 to squeeze out sort of the last mile here or the last milliseconds here.
 And this is another thing they they can practically do.
 So getting rid of jitter is extremely important.
 And that affects your control decisions and all those kinds of things.
 OK.
 Yeah, the cars is going to fundamentally maneuver better with lower jitter.
 The cars will maneuver with superhuman ability and reaction time much faster than a human.
 I mean, I think over time, the autopilot full self driving will be capable of maneuvers
 that are far more than what James Bond could do in the best movie type of thing.
 That's exactly what I was imagining in my mind, as you said.
 It's like impossible maneuvers that a human couldn't do.
 Well, OK, it's two things.
 Impossible maneuvers are impossible and things that humans could do are things that humans
 could do.
 I have no doubt that at one point in the near future, self driving cars will be able to
 do things that humans couldn't do.
 The question is more, are there going to be things that humans do that the cars couldn't
 do?
 Right.
 Or can't do?
 Because that's the actual gap you're trying to close.
 You know, look at Boston Dynamics or so.
 If you hard code stuff and you have extremely, extremely good sensors and actuators, you
 can do many things that humans couldn't do.
 But on the other hand, it's the things that humans can do that the machines can't.
 Those are the problem.
 Well, let me ask sort of looking back the six years, looking out into the future, based
 on your current understanding, how hard do you think this full self driving problem,
 when do you think Tesla will solve level four FSD?
 I think Elon gets asked this question every year and every year he says next year.
 So I mean, it's looking quite likely that it will be next year.
 This is the thing with Elon Musk, he always promises things like next year or on ridiculously
 short amounts of time.
 And I wonder how long it's going to take for people to just, you know, stop believing him.
 I guess many people already did.
 But it's still, you know, a thing to consider that on one hand, obviously, if you do it
 too much, then people are simply going to say, oh, well, probably in five years if he
 says next year.
 But on the other hand, he's also able to sort of it's a motivating thing.
 It's a cool thing.
 It drives momentum.
 And that itself accelerates the development of these things, people being ready to just
 flip on a beta version and so on.
 It's a bit insane.
 But I do think his optimism and a little bit salesmanship also a lot of benefits besides
 the obvious negatives.
 So the interventions, you know, per million miles has been dropping dramatically at some
 point.
 And that trend looks like it happens next year is that the probability of an accident
 on FSD is less than that of the average human and then significantly less than that of the
 average human.
 So it certainly appears like we will get there next year.
 There's a lot of hedging going on here.
 But you know, you can this is this is actually a nice method, I think, of making these types
 of predictions, you see that the rate of disengagement is dropping at a certain speed, you can extrapolate
 maybe a little bit and say, look, you know, here's going to be the sort of threshold where
 we're better than a human.
 I think that's a quite a sober analysis if done correctly.
 And I also think people who are, you know, it's obviously good to be skeptical of fully
 self driving systems.
 But on the other hand, you also have to think if they're a lot better than humans, it makes
 makes total sense, right?
 It also makes total sense to have them and not engage them all the time, right?
 There might still be situations you want to drive yourself.
 The question is a little bit, can you just continue the trend?
 Or is there a sort of an okay, you solve the easy problems.
 And that is what makes the rates of disengagement go down now.
 But now come the more and more hard problems and sort of it gets exponentially harder to
 continue that trend, in which case, we're not going to be there for a long time.
 Then there's going to be a case of, okay, we'll not have to prove this to regulators
 and prove it to you know, and we want a standard that is not just equivalent to a human, but
 much better than the average human, I think it's got to be at least two or three times
 higher safety than a human.
 Probably more like 10, like knowing, you know, regulators and how the public perceives these
 types of things.
 Of course, right now they're cool, but then it's really easy to publicize in a few accidents
 that few stupid accidents that happen if you build machine learning systems for the real
 world, they are going to make stupid mistakes.
 It doesn't matter how accurate they are on average, they're going to make stupid mistakes
 that a human would never do and people are just going to point at it and never forget
 that one instance.
 And I think it's pretty easy to sort of scare people publicizing those kinds of things.
 And therefore, yeah, you have to be like massively better than humans.
 I agree here.
 There is some fundamental leap that really deserves the 11.
 I mean, that's a pretty cool number.
 Yeah.
 11 would be a single stack for all, one stack to rule them all.
 But there are just some really fundamental neural net architecture changes that will
 allow for much more capability, but at first they're going to have issues.
 So we have this working on like sort of alpha software and it's good, but it's basically
 taking a whole bunch of C++ code and deleting a massive amount of C++ code and replacing
 it with a neural net.
 And Andrei makes this point a lot, which is like neural nets are kind of eating software.
 So it's interesting what Elon says right here.
 This upcoming version 11 of the Tesla software seems to have kind of a rewrite in what he
 calls the creation of the vector space.
 And specifically, he says you replace a whole bunch of C and C++ code with neural networks.
 And I guess what that means is that they used to have certain heuristics for what he calls
 creating the vector space, right?
 And remember, creating the vector space means seeing and understanding.
 So what objects exist?
 Where are they?
 How are they moving?
 And so on.
 And you want to get that out of your cameras and whatever other sensors you have.
 So it seems like until now, they had a bunch of neural networks that would do, you know,
 their stuff.
 I can imagine they had maybe single frame neural networks or kind of short frames, one
 after another neural networks that would recognize sort of bounding boxing the objects in the
 image.
 And then they would use sort of an algorithm heuristic algorithm that they wrote themselves
 to stitch that together over time.
 Maybe they use algorithms to do some kind of inferences like what he mentioned with
 the object tracking, and so on.
 And it seems to be that what they want to do is just end to end train one big neural
 network that just does it all.
 You input all of the sensor data, let's say from, you know, not only just right now, but
 you know, from the from the recent past, you just input it all in there.
 And the neural network will spit out this finished vector space, this finished scene
 understanding graph.
 And this obviously you can see where it comes from.
 This has been the story of deep learning so far, replacing more and more classical heuristics
 with an end to end learning system.
 And it also matches exactly with what Elon is saying, namely that right now, it doesn't
 seem to work quite well yet, but in time, it will get there.
 And again, this has been the story of deep learning in pretty much everything we've tackled
 since the beginning of deep learning.
 End to end systems ultimately came to be the heuristic systems, but it takes time, it takes
 work, it takes data, obviously massive amounts of compute.
 You know, over time, there's like, less and less conventional software, more and more
 neural net, which is still software, but it's, you know, still comes out the lines of software,
 but it's more more neural net stuff, and less, you know, heuristics, basically.
 If you're more more more matrix based stuff, and less heuristics based stuff.
 So by the way, the reason why this is the case, the reason why it works to replace heuristics
 with neural networks with data driven systems is that the world is always more complicated
 than you can encode in any heuristic.
 That's why we use machine learning in the first place, because we can't just program
 the algorithms that do image recognition, or speech recognition or whatnot.
 So the only representation of this really complex world, like the actual underlying
 world that is so complicated is the data.
 And therefore, our best chance to create systems that deal well with the world as such is systems
 that actually learn from data from the real world.
 And that's why it often works to replace the heuristics with data driven systems.
 If you have the data, and if you have the compute, which Tesla obviously does.
 We call it the giant bag of points.
 And it's like, so you go to pixel and something associated with that pixel, like this pixel
 is probably car, the pixel is probably lane line.
 Then you've got to assemble this giant bag of points in the C code and turn it into vectors.
 And it does a pretty good job of it, but we need another layer of neural nets on top of
 that to take the giant bag of points and distill that down to vector space in the neural net
 part of the software as opposed to the heuristics part of the software.
 So the translation of this is probably, if I understand Elon correctly, what they were
 doing so far is sort of semantic segmentation or pixel based pixel labeling.
 I can also imagine that they estimated things like depth maps and so on just from pixels.
 But then, as I said before, it was heuristics, it was sort of classical algorithms.
 And these aren't, I mean, classical, these are advanced algorithms, right, that take
 point clouds that take sort of segmentation maps and depth maps and all of that and turn
 them into objects.
 These are mostly heuristic based but very sophisticated algorithms.
 But it is clearly a good or a, let's say a modern move to ditch all of that and also
 teach the neural networks to just handle it until you have the semantic result that you
 want, namely the space of objects, the scene understanding graph.
 It's really outputting proper vectors to the CC++ control code, as opposed to the
 sort of constructing the vectors in C.
 We've done, I think, quite a good job of, but it's kind of hitting a local maximum on
 how well the C can do this.
 So this is really a big deal.
 And just all of the networks in the car need to...
 By the way, whenever you hear him talk about C and C++ code, just replace that with human
 authored code, right?
 The difference isn't necessarily the language you use, the difference is more like who writes
 the code.
 And when he says C and C++, it's humans, very smart humans, but still humans that write
 the code out of their thinking.
 And whenever he says neural networks, it's some sort of a data-driven systems, which
 obviously human author in the first place, but probably also is as well implemented in
 C and C++.
 The training, the amount of work done with...
 We've written all this custom software for training and labeling and to do auto labeling.
 Auto labeling is essential, especially when you've got surround video.
 It's very difficult to label surround video from scratch.
 It's extremely difficult.
 Like a human's such a long time to even label one video clip, like several hours.
 Or the auto label it, basically we just apply a heavy duty, like a lot of compute to the
 video clips to pre-assign and guess what all the things are that are going on in the surround
 video.
 And then there's like correcting it.
 Yeah.
 And then all the human has to do is like tweet, like say, adjust what is incorrect.
 This is like increase this productivity by effect a hundred or more.
 Yeah.
 So you've presented that...
 I mean, we've discussed this in the last video that I did about Karpotty's talk.
 And this to me is, I think too few people are currently doing something like this.
 Essentially it's active learning, right?
 It's sort of, if you're not sure about something, ask the human.
 It has a slight twist on it in that they probably always ask the human, but they suggest a label
 which is super powerful, especially in something like semantic segmentation where you need
 to annotate every pixel or you need to place bounding boxes around many objects.
 It's really different if you simply have to check and adjust a little bit versus if, you
 know, there's a data point and you have to place the labels yourself.
 I think we're going to see quite a bit more of that in sort of the near future.
 A lot of people are already doing something like this, but I think still too few are.
 It's not quite in Tesla's primary mission direction of accelerating sustainable energy,
 but it is an extremely useful thing that we can do for the world, which is to make a useful
 humanoid robot that is capable of interacting with the world.
 All right.
 The rest of them talking about AI is talking about the Tesla bot, which is a bit more far
 fetched I have to say.
 The Tesla bot just on its face is way more complicated than a car, especially if it is
 supposed to not only, you know, be on the factory floor in which case they just build
 like a robot arm, right?
 These are like the most useful things in a factory on a factory floor.
 But if it's actually to sort of interact with humans or in a human way navigate not only
 unknown terrain, but also society potentially.
 I mean, this is just this is just futurism at this point and that there's really nothing
 we can legitimately say about what's possible, what's not possible, where this is.
 And obviously they like we don't we don't have a prototype.
 We just have like a human in a suit to demonstrate the Tesla bot.
 So I will not comment much further on that with respect to the Tesla fully self driving
 system.
 I would say that obviously, you know, for Elon Musk, there's always kind of lovers and
 haters and I think you can acknowledge both sides.
 He is a bit of a salesperson.
 He sells these things very well.
 He always promises, you know, next year we'll be ready, next year we'll be ready.
 And then they never are or he over promises massively on you know, how much cost you can
 save and yada, yada, yada.
 But then on the other hand, he also delivers a lot more than other people deliver.
 Maybe that's just because a little bit of recklessness, but also the sort of optimism
 and momentum that he's able to to to come up and drive.
 And all of that together, I think just makes for like an interesting person.
 And I think the advances itself are remarkable.
 Even if you say other car companies are on the track and whatnot, Tesla has done more
 than all other car companies together for the adoption of electric vehicles.
 Yes, you can debate whether or not that in itself is a good thing.
 But just to say that it's not only salesmanship, there are also results.
 And I have no doubt that in the near future, we will see self driving cars.
 Sure, they're not going to be accident free, but I believe they will be much, much better
 than humans.
 And the question is simply is this next year in two years in five years?
 I cannot tell you, but I'm excited to see.
 I hope you like this talk analysis interview analysis.
 If you want more of these things, let me know.
 Otherwise, let me know what you think in the comments and I'll see you next time.
 Bye bye.