Self-driving from VISION ONLY - Tesla's self-driving progress by Andrej Karpathy (Talk Analysis)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper tesla elon musk karpathy full self driving tesla fsd karpathy talk tesla talk tesla computer vision how does tesla work computer vision driving lidar radar autonomous car autonomous car tesla autonomous driving driverless car driverless car tesla tesla machine learning tesla self driving tesla ai tesla research
#tesla #selfdriving #karpathy

Tesla is pushing the state-of-the-art in full self-driving, and interestingly, they explicitly switch from having multiple different sensors to a vision-only system. We discuss the highlights of Andrej Karpathy's talk about Tesla's FSD system, how to label petabytes of data, how to sample edge-cases, how to train a neural network that has to work in real-time, and why moving to having only cameras is superior to multi-sensor approaches.

OUTLINE:
0:00 - Intro & Overview
1:55 - Current Auto-Breaking system
3:20 - Full Self-Driving from vision only
4:55 - Auto-Labelling for collecting data
8:45 - How to get diverse data from edge-cases
12:15 - Neural network architecture
16:05 - Tesla's in-house supercomputer
17:00 - Owning the whole pipeline
18:20 - Example results from vision only
23:10 - Conclusion & Comments

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 All right, hello everyone. Today we're going to look at Andrej Karpathy's CVPR talk about full self-driving mode in Tesla and what Tesla has been doing to push that beyond its current state. So let's just say that autonomous driving is a hard problem. You have to control a car and pretty much anything could happen. However, we're able to teach it to pretty much any human on the planet. So the problem is definitely solvable. Now the current stack they have for full self-driving or that they intended to use, it seems like is what they call sensor fusion, which is where you take a bunch of different signals like camera signals, and radar signals and so on. And you try to fuse their signals together. This kind of works, it seems, but it runs into problems such as what do you do when the different sensors disagree. And it turns out solving that problem is quite hard. And that's why Tesla apparently is transitioning to a fully only vision stack. Everything is going to be vision based in Tesla full self-driving. Now today we're going to look at the best and important bits of the talk right here. Now I absolutely invite you to go watch the entire talk if you're interested. It is enjoyable in full length and it is on YouTube. Andrej gives a lot of good examples here and the amount of effort that went into engineering this into collecting the data, how this is deployed is astounding. Now keep in mind, this is the lead AI scientist for Tesla as it is going to be a bit of an ad. However, it is pretty cool to see that we are actually making a real push towards full self-driving. A lot of people have been super salty saying that Elon Musk has promised this like one or two years ago already. But come on, I mean, do you see anyone else doing fully self-driving at this level? No. So shut up. So the first thing right here is a couple of scenarios of what Tesla is already doing, which is sort of a driver assistance. So if the person is driving, but the system is relatively sure that the person is making a mistake, the system kicks in mostly to do automatic braking for the user. So I just I want to show you this one example right here. You start slowing and probably you know, does not actually enter the intersection. These are examples from pedal misapplication mitigation PMM. Here a person is un-parking from their driving spot and they are trying to turn and then they mess up and they accidentally floor it. So they floor it right there. So you see like the person wanted to brake but stepped on the gas, there are people right in front of the car. So be salty all you want. This right here is already worth it. As a human there is a lot of resistance against fully self-driving feeling that you're no longer in control anymore. But the matter of the fact is that these systems already are and in the near future will be even much more better than humans at driving is going to be much cleaner, much safer, much faster, less traffic jams and so on to let the machines take over the driving pretty much in the same way as it's much safer to let the machines take over the braking in these scenarios. The only times you're actually going to drive by hand is when you do it for fun. Now I drive a motorbike. It's a lot of fun to drive but in a car especially with other people or if I do it for work if I may be a little bit tired machines all the way. So the full self-driving beta is rolled out to a small handful of customers right now. And they do upload YouTube videos every now and then of what they're doing. And it seems to work fairly fairly well. Apparently they had had no crashes so far while driving about 1.7 million miles in full stealth driving. You can see on the screen in the middle right here that the predictions that the system gives is pretty good, though we've also seen some other prediction that are not so good throughout YouTube. Like there's this one video where the truck in front of the car has street lights on its back and the car just keeps thinking it's kind of red lights. However, we don't know if this is the legacy stack or not and if the car would actually break since the lights are not on red. But it's been a scare going around YouTube for a little bit. So here Andre shows a video of Waymo already doing this much earlier than Tesla having sort of an automatic car drive around an intersection and so on. This works if you're in a really defined zone, let's say a city that you know that you have accurate maps for this does not work if you want to do this anywhere in the world. To do this anywhere in the world, you need to rely on the car itself. That means you need a lot of data. So the data that this new system gets is just vision, it's eight cameras around the car. And that's it. And Andre makes a good case here that that is actually all you need humans are able to navigate from this and cars should be able to do the same. So an absolutely necessary ingredient to train such a system is a good clean label data set. If you just wanted to use humans to annotate every single frame of cars driving around, that would probably be prohibitively expensive even for Tesla. So they came up with what I think is a pretty cool method called auto labeling. Now I'm sure they're not the inventors of the system. But to use it on this scale is very smart. And it works out pretty nicely. Of course, we need to collect training data. A typical approach might be to use humans to annotate cars around us in three dimensions. What we found actually works really well is an auto labeling approach. So it's not pure humans just like annotating cars. It's an offline tracker, as we call it. And it's an auto labeling process for collecting data at the scale that is necessary. So we need to get millions of hard examples. So this is where the scale comes from is that it's not labeled purely by humans, although humans are involved, it's labeled automatically. So here's an example of some automatic labels we were able to derive for cars on the highway. And the way you do this is because you are offline and you are trying to just annotate a clip, you have a large number of benefits that you don't typically have with your app test time under strict latency requirements in the car. So you can take your time to fully figure out exactly all the objects in your app. You can use neural networks that are extremely heavy. They are not deployable for various reasons. You can use benefit of hindsight because you know the future, not just the past. You can use all kinds of expensive offline optimization and tracking techniques. You can use extra sensors. In this case, for example, actually radar was one of the sensors that we used for the auto labeling. But there's actually a massive difference between using radar at test time and using it in the offline tracker. The point here is that if you record data and you're trying to figure out at inference time, like while you're driving, what's happening, it's a lot harder than if you have the same data, but kind of at home in the lab. So what you want to do is you want to drive around and just record not even not predict or anything, just record data record from all your sensors, you can even stick expensive sensors on the cars where you collect the data. And then you take all that data and you use the biggest, heaviest processors you have to figure out what actually happened during that time. What he mentions here is the benefit of hindsight, which means that if you're in a car and you're driving and all of a sudden something obscures your vision, you will be sort of lost because all you have, okay, you can maybe guess that a car in front of you is still there. But who knows they might turn or something. Now, if you record the whole video sequence, you're able to see what happens beyond the obstruction of vision. And if you see the car is still there, you can make a good inference that the car was actually there the whole time. And therefore you can annotate that data with a label saying, hey, that car was there the whole time, you can also do active learning and shell out to actual human annotators what you're not sure about. So this benefit of hindsight is really important here when you're under the time constraint of not being able to see into the future, as well as the latency constraint and you have to have like an efficient neural network in the lab, you don't have any of this the method here, if you're developing something real time, I mean, this might seem obvious to you, I found it to be pretty cool. Yes, record, then figure out what happened, then use that as a labeled data set. So here's an example of how such a persistent track would look like after the neural network has been trained on data like this. Here's some examples of really tricky scenarios. I don't actually know exactly what this is. But basically, this car drops a bunch of debris on us, and we maintain a consistent track for the label. And of course, if you have millions of labels like this, the neural net, if it's a powerful enough neural net, we'll actually end up learning to persist these tracks in these kinds of scenarios. Here's another example. There's a car in front of us. I actually am not 100% sure what happens in this case. But as you'll see, there's some kind of a dust cloud that develops here and briefly occludes the car. But in the auto labeling tool, we are able to persist this track because we saw it before and we saw it after so we can actually stitch it up and use it as a training set for the neural network. So that's how they get clean labels in an automatic or semi automatic way. But they still need to get a lot of data from kind of edge cases because most of driving is quite uneventful, straight driving and was done 40 years ago or something like this. I think Schmidhuber in GTC 21 talk talked about autonomous cars on highways on controlled stretches of highways super duper early already. So what we really need to collect is edge cases. And for collecting these edge cases, Tesla has developed these what they call triggers. So these are kind of hand programmed rules of what data should go into the annotation pipeline. So imagine if all these cars driving around not only the people with full self driving, but the detection the actual recording of data is activated in all the Tesla cars driving around, they all send that data back to the server. Of course, that's way too much data. And also, it's very unbalanced in terms of how many critical situations are in there. Again, most of it will be sort of straight road, empty, just drive straight. So what they do is they filter this data for these trigger events. Now these trigger events can be as simple as whenever the radar and the vision mismatch. So whenever they disagree on something, that's an interesting example. But you know, it goes into very detailed such as we detect breaking lights, but the acceleration is positive. So with these triggers, they're able to source a diverse set of training samples and edge cases where the neural network can learn the tricky situations rather than just the long stretches of road. So I think it's safe to say that a good mark of quality on these systems is going to be how well these triggers are maintained, like how well do they represent the full driving experience of the end users of the cars. But so far from the results we got, it seems like they cover the road situations fairly well. And all of them are iteration and you're looking at what's coming back, you're tuning your trigger and you're sourcing data from all these scenarios. Basically, over the last four months, we've done quite extensive data engine, we've ended up doing seven shadow modes and seven loops around this data engine here, where on the top right is where you begin, you have some seed data set, you train your neural network on your data set and you deploy the neural network in the customer cars in shadow mode. And the network is silently making predictions. By the way, if you if you like squint really hard, I don't know if this is just a depiction of a neural network or if this is the actual architecture they're using. I don't think so. But there is like a stride of six in there and max pooling, you know, just just noting that for no particular reason. And then you have to have some mechanisms for sourcing inaccuracies of the neural net, you're just looking at its predictions. And then you're using one of these triggers, you're getting these scenarios where the network is probably misbehaving. Some of those clips end up going to unit tests to make sure that we even if we're failing right now, we make sure we pass later. And in addition, those examples are being auto labeled and incorporated into a training set. And then as a synchronous process, we're also always data cleaning the current training set. So we spin this loop over and over again, until the network basically becomes incredibly good. So in total, we've done seven rounds of shadow mode for this release. So shadow mode is what they call when they let the predictions run, but they don't hook them up to the control. So you're driving yourself, but the system predicts all the time. And whenever one of these trigger happens, that's an interesting data point that is going to send back to the server. Actually, let's be honest, it's probably going to send everything back to the server. So the data set they come up with is 1.5 petabytes. Crazy. So next is going to go into the architecture of the neural net. And this is also fairly interesting and not entirely standard on the top. All of them are processed by an image extractor, the layout of the synthetic visual cortex in order to efficiently process this information. Our architecture roughly looks like this. We have these images coming from multiple cameras on the top. All of them are processed by an image extractor, like a backbone, like a ResNet kind of style. Then there's a multi-can fusion that uses the information from all the eight to use. And this is a kind of a transformer that we use to fuse this information. And then we fuse information first across all the cameras and then across all of time. And that is also done either by a transformer, by a recurrent neural network, or just by three-dimensional convolutions. We've experimented with a lot of fusion strategies here to get this to work really well. And then what we have afterwards, after the fusion is done, is we have this branching structure that doesn't just consist of heads, but actually we've expanded this over the last year or so, where you now have heads that branch into trunks that branch into terminals. So there's a lot of branching structure. And the reason you want this branching structure is because there's a huge amount of outputs that you're interested in, and you can't afford to have a single neural network for every one of the individual outputs. You have to, of course, amortize the forward pass. So this is pretty interesting. The top part here, what they call the backbone is pretty standard. If you have a video, especially with multiple cameras, you want to extract information from each frame of each camera sort of individually, then you want to fuse that information across all the cameras for a single time step. And then you want to fuse that information with the information of all the other time steps. So so far, so good. That sort of gives you a representation of what happens in these frames in these cameras during that stretch of time. However, after that, usually, even if you have multiple predictions, what you would do is you would sort of have like one prediction head on top of that backbone. However, since they are in a car and have to decide real fast, it's not really feasible to have sort of these different columns for each of the prediction tasks. Because as he says, they're interested in a lot of different signals, think depth prediction, which means that for every pixel, you have to provide a depth estimation, think tracks of other cars, think pedestrians, think streetlights, think, okay, where are the lanes at, or navigation in general. So all these signals are things to predict. And it's not good enough to have like a separate head for each of the predictions. So what they do is they have, as you call these branching structures, where there are multiple heads, yes. And within these multiple heads, there are what they call trunks. And within the trunks, there are the individual like little what they call terminals. Essentially, it's a hierarchical prediction, I'm going to guess that the tasks that go together, sort of are grouped together. So maybe one head is for all the pixel prediction tasks, and another head is more for the classification tasks. And then within one head, you have a trunk that deals more with like object classification, and another trunk that deals more with like navigation classification. And the individual terminals then do the actual tasks. So this is a pretty cool way of getting a highly performant many output network all together such that its size and computational speed are still maintained. The other nice benefit of the branching structure is that it decouples at the terminals, it decouples all these signals. So if I'm someone working on velocity for a particular object type, or something like that, I have a small piece of neural network that I can actually fine tune without touching any of the other signals. And so I can work in isolation to some extent, and actually get something to work pretty well. And then once in a while, so basically the iteration scheme is that a lot of people are fine tuning and once in a while... You just gotta imagine the ML ops behind this. It's like, hey, where do you deploy your models? I do it on the Kubernetes, I have ML flow. Oh, no, I use the TensorFlow extended. Yeah, it's pretty cool. What do you do? Car. I deploy on car. So next, he's going into this in house supercomputer that they built or are building. And this is a massive thing. Absolutely massive. He says that in terms of flops, it's something like the fifth biggest computer in the world. Its storage speed is incredible. So I'm pretty sure you could even actually render Far Cry 2 on this thing, maybe. But in total, it has 5760 GPUs, not any GPUs, the most expensive a 180 gigabyte GPUs, it would be interesting to see what kind of algorithms they use on top of this to actually do the distributed training or whether it's all just kind of simple data parallelism, aggregating gradients, and so on. Of course, they have super fast interconnect, super fast storage, super fast everything. And it looks sweet. Like is this a stock photo of a server room? Or is this the actual server room? This effort basically is incredibly vertically integrated into the AI team. So as I showed you, we own the vehicle and the sensing and we source our own data and we annotate our own data and we train our on-prem cluster. And then we deploy all of the neural networks that we train on our in-house developed chip. So we have the FSD computer here that has two SOCs, has the chips here, and they have our own custom NPU neural processing unit here at roughly 36 times each. So these chips are specifically designed for the neural networks that we want to run for. Yeah, I mean, this is the dream, right? If you're an AI professional, owning the whole pipeline is going to boost your productivity by so much. You're not bound by the constraint of anything other than the limits on the final system, which is a car so fairly difficult. But in between of that, you have control over everything, you have control over how the data is collected, annotated, you have control over where it is deployed to on what architecture of chip because you make the chip. So I guess the lesson is if you're looking to change the world, you better own a good chunk of it. So now I'm just going to show some examples of what this new vision only stack could do. Remember, they used to do fusion of sensors, which means they essentially have radar, they have vision, maybe some other sensors, and they try to integrate this information from all of the sensors. They compare this to the new vision based system. Now check out what happens in terms of the depth and velocity predictions that we're able to achieve by putting all these pieces together and training these networks at scale. So the first example here, I have a video where this is on track testing. So this is an engineering car and we asked it to slam on the brakes as hard as it possibly can. So this is a very harsh breaking here in front of us, even though it doesn't look like that in the videos is very harsh breaking. So what you can see on the right here is you can see the outputs from the legacy stack, which had radar vision fusion and from the new stack, which has vision alone in blue. So in the orange legacy stack, you can actually see these track drops here when the car was breaking really harshly. And basically the issue is that the breaking was so harsh that the radar stack that we have actually ended up not associating car and dropping the track and then re initializing it all the time. And so it's as if the vehicle disappeared and reappeared like six times during the period of this breaking. And so this created a bunch of artifacts here, but we see that the new stack in blue is actually not subject to this behavior at all. It just gives a clean signal. In fact, here there's no smoothing, I believe on the blue signal here. This is the raw depth and velocity that comes out from the neural net, the final neural net that we released with about three weeks ago. And you can see there it's fairly smooth here. And of course you could go into the radar stack and you could adjust the height parameters of the tracker. Like why is it dropping tracks and so on, but then you are spending engineering efforts and focus on a stack that is like not really barking up the right tree. And so it's better to again, focus on the vision and make it work really well. And we see that it is much more robust when you train it at scale. So there you have it, proved by one example that the new thing works better. Isn't that every CVPR paper ever, but no, in any case, I can totally believe that the new stack, even though it drops a bunch of the sensors is better. Because ultimately, if your one sensor, if vision is so performant that in every single disagreement, you go with the vision thing, then why do you have the other sensors at all? The thing in front of it is just kind of breaking too fast. So the radar kind of loses it and then regains it and loses it and regains it. Now I have no idea how radar works. So I'm speaking from complete ignorance right here. But what I'm going to guess as far as I understand it is that radar just kind of gives you the velocities of stuff in front of you. And then there is a tracking algorithm on top of radar that tries to figure out which stuff is the same stuff. And this is very much what they do in this auto labeling, where they have sort of a track on something, right, and then they use hindsight, and then they have a tracking algorithm that decides which things are the same, even though we don't see them all the time. And here you can clearly see the benefit of shifting this from inference time, which is what you have to do with radar to the training time, which is what you can do with vision. So you can teach the vision system to sort of do this persistent tracking, whereas the radar system, you have to hand tune it to do this in real time. Now it makes the point that of course, you could go into the radar system, change the hyper parameters. But then he says, why bark up the wrong tree? Why waste time on a stack that isn't functioning? It's a bit of a chicken and an egg problem, right? If you were to put as much effort into the radar stack as you were into the vision system, I'm going to guess that these results would go away. And that is able to keep up maybe. But arguments for going vision only is a strong one. And I don't doubt that it is probably a good way forward. And basically what's happening here is that the radar is very trigger happy and it sees all these false stationary objects everywhere, like everything that like sticks out as a stationary target and radar by itself doesn't know what actually is a stationary car and what isn't. So it's waiting for vision to associate with it. And vision, if it's not held up to a high enough bar is noisy and contributes to error. And the sensor fusion stack just kind of like picks it up too late. And so again, you could fix all that, even though it's a very gross system with a lot of statements and so on, because the sensor fusion is complicated because the error modes for vision and radar are slightly are quite different. But here, when we just work with vision alone and we take out the radar, vision recognizes this object very early, gives the correct depth and velocity, and there's no issues. So we actually get an initial slowdown much earlier and really like simplify the stack a lot. Yeah, so here you can see the same failure mode in vision that it kind of gets a track but doesn't but get a track but doesn't. The important part is that once you get closer to the object, it is fairly consistent, right? As you can see right here, the vision stack recognizes this truck on the side much earlier than the radar stack did. Now, again, this might just be a function of the hyper parameters used, I'm sure you could just lower the threshold for the radar, but you'd run into different problems. During the Q&A, he makes a good point in that, yes, other sensors would be nice to have, but just the pure economics speak in favor of vision too. Like we develop cameras with much more rigor as a society than we do radar systems. And therefore, the camera sensors are just so much better nowadays and cheaper. So you can afford to build many of them into all kinds of things and collect data and make your systems better through that than to put kind of a lidar on top of the car and having to sort of fuse those signals with the vision signals, especially when they're in conflict with one another. So if you ask me, I'm a fan, I like what I see here, even though I know it's kind of an ad, I don't own the Tesla, but I think it's still pretty cool. So in the end, he talks a bit about what they do to validate this data, and how they roll it out and gives a bunch of more examples of tracking. And there's a Q&A at the end. So if you are interested in that, I absolutely welcome you to go watch the entire talk. It is on YouTube. And that was it from me. I hope you enjoyed this and I'll see you next time. Ciao.
 All right, hello everyone. Today we're going to look at Andrej Karpathy's CVPR talk about full
 self-driving mode in Tesla and what Tesla has been doing to push that beyond its current state. So
 let's just say that autonomous driving is a hard problem. You have to control a car and pretty much
 anything could happen. However, we're able to teach it to pretty much any human on the planet.
 So the problem is definitely solvable. Now the current stack they have for full self-driving or
 that they intended to use, it seems like is what they call sensor fusion, which is where you take
 a bunch of different signals like camera signals, and radar signals and so on. And you try to fuse
 their signals together. This kind of works, it seems, but it runs into problems such as what do
 you do when the different sensors disagree. And it turns out solving that problem is quite hard. And
 that's why Tesla apparently is transitioning to a fully only vision stack. Everything is going to be
 vision based in Tesla full self-driving. Now today we're going to look at the best and important
 bits of the talk right here. Now I absolutely invite you to go watch the entire talk if you're
 interested. It is enjoyable in full length and it is on YouTube. Andrej gives a lot of good examples
 here and the amount of effort that went into engineering this into collecting the data,
 how this is deployed is astounding. Now keep in mind, this is the lead AI scientist for Tesla
 as it is going to be a bit of an ad. However, it is pretty cool to see that we are actually making
 a real push towards full self-driving. A lot of people have been super salty saying that Elon Musk
 has promised this like one or two years ago already. But come on, I mean, do you see anyone
 else doing fully self-driving at this level? No. So shut up. So the first thing right here is a
 couple of scenarios of what Tesla is already doing, which is sort of a driver assistance. So if
 the person is driving, but the system is relatively sure that the person is making a mistake,
 the system kicks in mostly to do automatic braking for the user. So I just I want to show you this
 one example right here. You start slowing and probably you know, does not actually enter the
 intersection. These are examples from pedal misapplication mitigation PMM. Here a person
 is un-parking from their driving spot and they are trying to turn and then they mess up and they
 accidentally floor it. So they floor it right there. So you see like the person wanted to brake but
 stepped on the gas, there are people right in front of the car. So be salty all you want. This
 right here is already worth it. As a human there is a lot of resistance against fully self-driving
 feeling that you're no longer in control anymore. But the matter of the fact is that these systems
 already are and in the near future will be even much more better than humans at driving is going
 to be much cleaner, much safer, much faster, less traffic jams and so on to let the machines take
 over the driving pretty much in the same way as it's much safer to let the machines take over the
 braking in these scenarios. The only times you're actually going to drive by hand is when you do it
 for fun. Now I drive a motorbike. It's a lot of fun to drive but in a car especially with other
 people or if I do it for work if I may be a little bit tired machines all the way. So the full
 self-driving beta is rolled out to a small handful of customers right now. And they do upload YouTube
 videos every now and then of what they're doing. And it seems to work fairly fairly well. Apparently
 they had had no crashes so far while driving about 1.7 million miles in full stealth driving. You can
 see on the screen in the middle right here that the predictions that the system gives is pretty
 good, though we've also seen some other prediction that are not so good throughout YouTube. Like
 there's this one video where the truck in front of the car has street lights on its back and the
 car just keeps thinking it's kind of red lights. However, we don't know if this is the legacy stack
 or not and if the car would actually break since the lights are not on red. But it's been a scare
 going around YouTube for a little bit. So here Andre shows a video of Waymo already doing this
 much earlier than Tesla having sort of an automatic car drive around an intersection and so on. This
 works if you're in a really defined zone, let's say a city that you know that you have accurate
 maps for this does not work if you want to do this anywhere in the world. To do this anywhere in the
 world, you need to rely on the car itself. That means you need a lot of data. So the data that
 this new system gets is just vision, it's eight cameras around the car. And that's it. And Andre
 makes a good case here that that is actually all you need humans are able to navigate from this
 and cars should be able to do the same. So an absolutely necessary ingredient to train such
 a system is a good clean label data set. If you just wanted to use humans to annotate every single
 frame of cars driving around, that would probably be prohibitively expensive even for Tesla. So
 they came up with what I think is a pretty cool method called auto labeling. Now I'm sure they're
 not the inventors of the system. But to use it on this scale is very smart. And it works out pretty
 nicely. Of course, we need to collect training data. A typical approach might be to use humans
 to annotate cars around us in three dimensions. What we found actually works really well is an
 auto labeling approach. So it's not pure humans just like annotating cars. It's an offline tracker,
 as we call it. And it's an auto labeling process for collecting data at the scale that is necessary.
 So we need to get millions of hard examples. So this is where the scale comes from is that
 it's not labeled purely by humans, although humans are involved, it's labeled automatically.
 So here's an example of some automatic labels we were able to derive for cars on the highway.
 And the way you do this is because you are offline and you are trying to just annotate a clip,
 you have a large number of benefits that you don't typically have with your app test time
 under strict latency requirements in the car. So you can take your time to fully figure out
 exactly all the objects in your app. You can use neural networks that are extremely heavy. They are
 not deployable for various reasons. You can use benefit of hindsight because you know the future,
 not just the past. You can use all kinds of expensive offline optimization and tracking
 techniques. You can use extra sensors. In this case, for example, actually radar was one of the
 sensors that we used for the auto labeling. But there's actually a massive difference between
 using radar at test time and using it in the offline tracker. The point here is that if you
 record data and you're trying to figure out at inference time, like while you're driving,
 what's happening, it's a lot harder than if you have the same data, but kind of at home in the
 lab. So what you want to do is you want to drive around and just record not even not predict or
 anything, just record data record from all your sensors, you can even stick expensive sensors on
 the cars where you collect the data. And then you take all that data and you use the biggest,
 heaviest processors you have to figure out what actually happened during that time. What he
 mentions here is the benefit of hindsight, which means that if you're in a car and you're driving
 and all of a sudden something obscures your vision, you will be sort of lost because all you have,
 okay, you can maybe guess that a car in front of you is still there. But who knows they might turn
 or something. Now, if you record the whole video sequence, you're able to see what happens beyond
 the obstruction of vision. And if you see the car is still there, you can make a good inference
 that the car was actually there the whole time. And therefore you can annotate that data with a
 label saying, hey, that car was there the whole time, you can also do active learning and shell
 out to actual human annotators what you're not sure about. So this benefit of hindsight is really
 important here when you're under the time constraint of not being able to see into the future,
 as well as the latency constraint and you have to have like an efficient neural network in the lab,
 you don't have any of this the method here, if you're developing something real time, I mean,
 this might seem obvious to you, I found it to be pretty cool. Yes, record, then figure out what
 happened, then use that as a labeled data set. So here's an example of how such a persistent track
 would look like after the neural network has been trained on data like this. Here's some examples
 of really tricky scenarios. I don't actually know exactly what this is. But basically, this car
 drops a bunch of debris on us, and we maintain a consistent track for the label. And of course,
 if you have millions of labels like this, the neural net, if it's a powerful enough neural net,
 we'll actually end up learning to persist these tracks in these kinds of scenarios.
 Here's another example. There's a car in front of us. I actually am not 100% sure what happens in
 this case. But as you'll see, there's some kind of a dust cloud that develops here and briefly
 occludes the car. But in the auto labeling tool, we are able to persist this track because we saw
 it before and we saw it after so we can actually stitch it up and use it as a training set for the
 neural network. So that's how they get clean labels in an automatic or semi automatic way. But they
 still need to get a lot of data from kind of edge cases because most of driving is quite uneventful,
 straight driving and was done 40 years ago or something like this. I think Schmidhuber in GTC
 21 talk talked about autonomous cars on highways on controlled stretches of highways super duper
 early already. So what we really need to collect is edge cases. And for collecting these edge cases,
 Tesla has developed these what they call triggers. So these are kind of hand programmed rules
 of what data should go into the annotation pipeline. So imagine if all these cars driving
 around not only the people with full self driving, but the detection the actual recording of data is
 activated in all the Tesla cars driving around, they all send that data back to the server. Of
 course, that's way too much data. And also, it's very unbalanced in terms of how many critical
 situations are in there. Again, most of it will be sort of straight road, empty, just drive straight.
 So what they do is they filter this data for these trigger events. Now these trigger events can be as
 simple as whenever the radar and the vision mismatch. So whenever they disagree on something,
 that's an interesting example. But you know, it goes into very detailed such as we detect
 breaking lights, but the acceleration is positive. So with these triggers, they're able to source a
 diverse set of training samples and edge cases where the neural network can learn the tricky
 situations rather than just the long stretches of road. So I think it's safe to say that a good
 mark of quality on these systems is going to be how well these triggers are maintained, like how
 well do they represent the full driving experience of the end users of the cars. But so far from the
 results we got, it seems like they cover the road situations fairly well. And all of them are
 iteration and you're looking at what's coming back, you're tuning your trigger and you're sourcing
 data from all these scenarios. Basically, over the last four months, we've done quite extensive data
 engine, we've ended up doing seven shadow modes and seven loops around this data engine here,
 where on the top right is where you begin, you have some seed data set, you train your neural
 network on your data set and you deploy the neural network in the customer cars in shadow mode. And
 the network is silently making predictions. By the way, if you if you like squint really hard,
 I don't know if this is just a depiction of a neural network or if this is the actual architecture
 they're using. I don't think so. But there is like a stride of six in there and max pooling,
 you know, just just noting that for no particular reason. And then you have to have some mechanisms
 for sourcing inaccuracies of the neural net, you're just looking at its predictions. And then you're
 using one of these triggers, you're getting these scenarios where the network is probably
 misbehaving. Some of those clips end up going to unit tests to make sure that we even if we're
 failing right now, we make sure we pass later. And in addition, those examples are being auto labeled
 and incorporated into a training set. And then as a synchronous process, we're also always data
 cleaning the current training set. So we spin this loop over and over again, until the network
 basically becomes incredibly good. So in total, we've done seven rounds of shadow mode for this
 release. So shadow mode is what they call when they let the predictions run, but they don't hook
 them up to the control. So you're driving yourself, but the system predicts all the time. And whenever
 one of these trigger happens, that's an interesting data point that is going to send back to the
 server. Actually, let's be honest, it's probably going to send everything back to the server.
 So the data set they come up with is 1.5 petabytes. Crazy. So next is going to go into the architecture
 of the neural net. And this is also fairly interesting and not entirely standard on the top.
 All of them are processed by an image extractor, the layout of the synthetic visual cortex in order
 to efficiently process this information. Our architecture roughly looks like this. We have
 these images coming from multiple cameras on the top. All of them are processed by an image
 extractor, like a backbone, like a ResNet kind of style. Then there's a multi-can fusion that uses
 the information from all the eight to use. And this is a kind of a transformer that we use to fuse
 this information. And then we fuse information first across all the cameras and then across all
 of time. And that is also done either by a transformer, by a recurrent neural network,
 or just by three-dimensional convolutions. We've experimented with a lot of fusion strategies here
 to get this to work really well. And then what we have afterwards, after the fusion is done,
 is we have this branching structure that doesn't just consist of heads, but actually we've expanded
 this over the last year or so, where you now have heads that branch into trunks that branch into
 terminals. So there's a lot of branching structure. And the reason you want this branching structure
 is because there's a huge amount of outputs that you're interested in, and you can't afford to have
 a single neural network for every one of the individual outputs. You have to, of course,
 amortize the forward pass. So this is pretty interesting. The top part here, what they call
 the backbone is pretty standard. If you have a video, especially with multiple cameras,
 you want to extract information from each frame of each camera sort of individually,
 then you want to fuse that information across all the cameras for a single time step. And then you
 want to fuse that information with the information of all the other time steps. So so far, so good.
 That sort of gives you a representation of what happens in these frames in these cameras during
 that stretch of time. However, after that, usually, even if you have multiple predictions,
 what you would do is you would sort of have like one prediction head on top of that backbone.
 However, since they are in a car and have to decide real fast, it's not really feasible to
 have sort of these different columns for each of the prediction tasks. Because as he says,
 they're interested in a lot of different signals, think depth prediction, which means that for every
 pixel, you have to provide a depth estimation, think tracks of other cars, think pedestrians,
 think streetlights, think, okay, where are the lanes at, or navigation in general. So all these
 signals are things to predict. And it's not good enough to have like a separate head for each of
 the predictions. So what they do is they have, as you call these branching structures, where there
 are multiple heads, yes. And within these multiple heads, there are what they call trunks. And within
 the trunks, there are the individual like little what they call terminals. Essentially, it's a
 hierarchical prediction, I'm going to guess that the tasks that go together, sort of are grouped
 together. So maybe one head is for all the pixel prediction tasks, and another head is more for
 the classification tasks. And then within one head, you have a trunk that deals more with like object
 classification, and another trunk that deals more with like navigation classification. And the
 individual terminals then do the actual tasks. So this is a pretty cool way of getting a highly
 performant many output network all together such that its size and computational speed are still
 maintained. The other nice benefit of the branching structure is that it decouples at the terminals,
 it decouples all these signals. So if I'm someone working on velocity for a particular object type,
 or something like that, I have a small piece of neural network that I can actually fine tune
 without touching any of the other signals. And so I can work in isolation to some extent, and
 actually get something to work pretty well. And then once in a while, so basically the iteration
 scheme is that a lot of people are fine tuning and once in a while... You just gotta imagine the ML
 ops behind this. It's like, hey, where do you deploy your models? I do it on the Kubernetes,
 I have ML flow. Oh, no, I use the TensorFlow extended. Yeah, it's pretty cool. What do you do?
 Car. I deploy on car. So next, he's going into this in house supercomputer that they built or
 are building. And this is a massive thing. Absolutely massive. He says that in terms of
 flops, it's something like the fifth biggest computer in the world. Its storage speed is
 incredible. So I'm pretty sure you could even actually render Far Cry 2 on this thing, maybe.
 But in total, it has 5760 GPUs, not any GPUs, the most expensive a 180 gigabyte GPUs, it would be
 interesting to see what kind of algorithms they use on top of this to actually do the distributed
 training or whether it's all just kind of simple data parallelism, aggregating gradients, and so on.
 Of course, they have super fast interconnect, super fast storage, super fast everything. And it looks
 sweet. Like is this a stock photo of a server room? Or is this the actual server room? This effort
 basically is incredibly vertically integrated into the AI team. So as I showed you, we own the vehicle
 and the sensing and we source our own data and we annotate our own data and we train our on-prem
 cluster. And then we deploy all of the neural networks that we train on our in-house developed
 chip. So we have the FSD computer here that has two SOCs, has the chips here, and they have our
 own custom NPU neural processing unit here at roughly 36 times each. So these chips are
 specifically designed for the neural networks that we want to run for. Yeah, I mean, this is the dream,
 right? If you're an AI professional, owning the whole pipeline is going to boost your productivity
 by so much. You're not bound by the constraint of anything other than the limits on the final system,
 which is a car so fairly difficult. But in between of that, you have control over everything,
 you have control over how the data is collected, annotated, you have control over where it is
 deployed to on what architecture of chip because you make the chip. So I guess the lesson is if
 you're looking to change the world, you better own a good chunk of it. So now I'm just going to show
 some examples of what this new vision only stack could do. Remember, they used to do fusion of
 sensors, which means they essentially have radar, they have vision, maybe some other sensors, and
 they try to integrate this information from all of the sensors. They compare this to the new vision
 based system. Now check out what happens in terms of the depth and velocity predictions that we're
 able to achieve by putting all these pieces together and training these networks at scale.
 So the first example here, I have a video where this is on track testing. So this is an engineering
 car and we asked it to slam on the brakes as hard as it possibly can. So this is a very harsh
 breaking here in front of us, even though it doesn't look like that in the videos is very
 harsh breaking. So what you can see on the right here is you can see the outputs from the legacy
 stack, which had radar vision fusion and from the new stack, which has vision alone in blue. So in
 the orange legacy stack, you can actually see these track drops here when the car was breaking
 really harshly. And basically the issue is that the breaking was so harsh that the radar stack
 that we have actually ended up not associating car and dropping the track and then re initializing it
 all the time. And so it's as if the vehicle disappeared and reappeared like six times during
 the period of this breaking. And so this created a bunch of artifacts here, but we see that the new
 stack in blue is actually not subject to this behavior at all. It just gives a clean signal.
 In fact, here there's no smoothing, I believe on the blue signal here. This is the raw depth
 and velocity that comes out from the neural net, the final neural net that we released with about
 three weeks ago. And you can see there it's fairly smooth here. And of course you could go into the
 radar stack and you could adjust the height parameters of the tracker. Like why is it dropping
 tracks and so on, but then you are spending engineering efforts and focus on a stack that is
 like not really barking up the right tree. And so it's better to again, focus on the vision and
 make it work really well. And we see that it is much more robust when you train it at scale.
 So there you have it, proved by one example that the new thing works better. Isn't that every CVPR
 paper ever, but no, in any case, I can totally believe that the new stack, even though it drops
 a bunch of the sensors is better. Because ultimately, if your one sensor, if vision is so
 performant that in every single disagreement, you go with the vision thing, then why do you have the
 other sensors at all? The thing in front of it is just kind of breaking too fast. So the radar kind
 of loses it and then regains it and loses it and regains it. Now I have no idea how radar works. So
 I'm speaking from complete ignorance right here. But what I'm going to guess as far as I understand
 it is that radar just kind of gives you the velocities of stuff in front of you. And then
 there is a tracking algorithm on top of radar that tries to figure out which stuff is the same stuff.
 And this is very much what they do in this auto labeling, where they have sort of a track on
 something, right, and then they use hindsight, and then they have a tracking algorithm that decides
 which things are the same, even though we don't see them all the time. And here you can clearly
 see the benefit of shifting this from inference time, which is what you have to do with radar to
 the training time, which is what you can do with vision. So you can teach the vision system to sort
 of do this persistent tracking, whereas the radar system, you have to hand tune it to do this in
 real time. Now it makes the point that of course, you could go into the radar system, change the
 hyper parameters. But then he says, why bark up the wrong tree? Why waste time on a stack that
 isn't functioning? It's a bit of a chicken and an egg problem, right? If you were to put as much
 effort into the radar stack as you were into the vision system, I'm going to guess that these
 results would go away. And that is able to keep up maybe. But arguments for going vision only is a
 strong one. And I don't doubt that it is probably a good way forward. And basically what's happening
 here is that the radar is very trigger happy and it sees all these false stationary objects everywhere,
 like everything that like sticks out as a stationary target and radar by itself doesn't know
 what actually is a stationary car and what isn't. So it's waiting for vision to associate with it.
 And vision, if it's not held up to a high enough bar is noisy and contributes to error. And the
 sensor fusion stack just kind of like picks it up too late. And so again, you could fix all that,
 even though it's a very gross system with a lot of statements and so on, because the sensor fusion
 is complicated because the error modes for vision and radar are slightly are quite different. But
 here, when we just work with vision alone and we take out the radar, vision recognizes this object
 very early, gives the correct depth and velocity, and there's no issues. So we actually get an
 initial slowdown much earlier and really like simplify the stack a lot. Yeah, so here you can
 see the same failure mode in vision that it kind of gets a track but doesn't but get a track but
 doesn't. The important part is that once you get closer to the object, it is fairly consistent,
 right? As you can see right here, the vision stack recognizes this truck on the side much earlier
 than the radar stack did. Now, again, this might just be a function of the hyper parameters used,
 I'm sure you could just lower the threshold for the radar, but you'd run into different problems.
 During the Q&A, he makes a good point in that, yes, other sensors would be nice to have,
 but just the pure economics speak in favor of vision too. Like we develop cameras with much
 more rigor as a society than we do radar systems. And therefore, the camera sensors are just so much
 better nowadays and cheaper. So you can afford to build many of them into all kinds of things and
 collect data and make your systems better through that than to put kind of a lidar on top of the
 car and having to sort of fuse those signals with the vision signals, especially when they're in
 conflict with one another. So if you ask me, I'm a fan, I like what I see here, even though I know
 it's kind of an ad, I don't own the Tesla, but I think it's still pretty cool. So in the end,
 he talks a bit about what they do to validate this data, and how they roll it out and gives a
 bunch of more examples of tracking. And there's a Q&A at the end. So if you are interested in that,
 I absolutely welcome you to go watch the entire talk. It is on YouTube. And that was it from me.
 I hope you enjoyed this and I'll see you next time. Ciao.