 Hi, and welcome back to the PyTorch training video series. In this video, we're going to cover Autograd, which is PyTorch's tooling for rapidly and dynamically computing the gradients that drive backpropagation-based learning in your machine learning model. In particular, we're going to go over what Autograd does for you and why it makes PyTorch so flexible and powerful for machine learning? We'll walk through a basic code example to give you a feel for what AutoGrad is doing under the hood. Then we'll see AutoGrad's role in a training loop. After that, we'll talk about why and how to turn the AutoGrad feature off and on for a particular tensor or a particular context. We'll see the AutoGrad profiler in action and we'll look at the AutoGrad high-level API that was released with PyTorch 1.5. The Autograd feature of PyTorch is a large part of what makes PyTorch a fast and flexible framework for building deep learning projects. It does this by easing the computation of the partial derivatives, also called gradients, that drive backpropagation-based learning. I'm not going to belabor the math here, although if you'd like a refresher, go ahead and download the notebook and follow along in detail. The important concept here is that when we're training our model, we compute a loss function, which tells us how far our model's prediction is from the ideal. We then need to find the partial derivatives of the loss function with respect to the model's learning weights. These derivatives tell us in what direction we have to adjust the weights in order to minimize the loss function with respect to the model's learning weights. These derivatives tell us in what direction we have to adjust the weights in order to minimize the loss. This involves the iterative application of the chain rule of differential calculus over every path through the computation. AutoGrad makes this computation faster by tracing your computation at runtime. Every output tensor from your model's computation carries with it a history of the operations that led to it. This history allows the rapid computation of derivatives over the graph, all the way back to your model's learning weights. In addition, because this history is gathered at runtime, you'll get the correct derivatives even if your model has a dynamic structure with decision branches and loops. This offers a lot of flexibility over tools that depend on analysis of a static computation graph. Let's have a look at a simple example of autograd in action. First, we'll import PyTorch and matplotlib so we can graph some stuff. Next, we'll make a one-dimensional tensor holding a bunch of values between 0 and 2 pi, and we'll add the requires grad equals true flag. Note that when we print A, Pytorch lets us know that A wants gradients computed on any calculation it's involved in. Now we'll perform a computation. Here we'll just take the sign of all the values in A and we'll graph that and that looks right. If you notice the calls that attach here, I'll be covering those later in the section on turning autograd on and off. So printing the tensor B, we see that PyTorch tells us it has a grad function. This means that b came from a computation where at least one of the inputs required the calculation of gradients. The grad function tells us that b came from the sine operation. Let's perform a couple of more steps. We'll double b and add 1 to it. When we do this, we see that the output tensors again contain information about the operations that generated them in the Grad function property. By default, AutoGrad expects the final function in a gradient computation to be single-valued. This is the case when we're computing the derivatives over learning weights. The loss function has a single scalar value for its output. It doesn't strictly have to be single value, but we'll go over that in a bit. Here we'll just sum the elements of the tensor and call that the final output for this computation. We can actually use the grad function property of any output or intermediate tensor to walk back to the beginning of the computation history, using the grad functionction's Next Functions property. Here, you can see that D knows it came from an addition operation, which knows it came from a multiplication operation, and so on back to A. A does not have a Grad Function. It is an input or leaf node of this computation graph, and so represents the target variables for which we want to compute the gradients. So we've looked a little at the history tracking, but how do we actually compute gradients? That's easy. Just call the backward method on the output tensor like so. Looking back over the computation, we have a sine, the derivative of which is cosine. We have the multiplication by 2, which should add a factor of 2 to our gradient, and the addition, which should not change the derivative at all. Graphing the grad property on A, we see that, in fact, the computed gradients are twice the cosine. Be aware that gradients are only computed for inputs or leaf nodes of the computation. Intermediate tensors will not have gradients attached after the backward pass. So we've peeked under the hood at how AutoGrad computes gradients in a simple case. Next we'll examine its role in the training loop of a PyTorch model. To see how AutoGrad works in your training loop, let's build a small model and watch how it changes over a single training batch. We'll define and instantiate a model and create some stand-in tensors for the training input and ideal output. You may have seen that we did not specify requires Grad equals true for the model's layers. Within a subclass of torch.nn.module, the gradient tracking is managed for you. If we look at the layers of the model, you can see the randomly initialized weights and that they have no gradients computed yet. You might have noticed there's a grad function on the weights I sampled. There's no grad function on the weights themselves because they're a leaf node of the computation graph, but the slice operation counts as a differentiable operation, so my little slice of the weights is a grad function indicating that it came from the slice. So let's see how this changes after one training batch. For a loss function, we'll use the square of the Euclidean distance between our prediction and our ideal output. We'll also set up a basic optimizer using stochastic gradient descent. Note that we initialize the optimizer with the model's learning weights or parameters. The optimizer is responsible for adjusting the weights. So what happens when we call loss.backward? We can see that the weights have not changed, but that we do have gradients computed. Again, these gradients guide the optimizer in determining how to adjust the weights to minimize the loss score. In order to actually update the weights, we have to call optimizer.step. And we can see that the weights have changed. This is how learning happens in your PyTorch models. There's one more important step in the process. After you call optimizer.step, you need to call optimizer.zerograd. If you don't, the gradients will accumulate over every training batch. For example, if we run a training batch five times without calling zerograd, you can see the gradients turn up mostly with much larger magnitudes because they were accumulated over each batch. And you can see that calling zero grad resets the gradients. If your model is not learning or training is giving you strange results, the very first thing you should check is whether you're calling zero grad after each training step. Sometimes you'll want to control whether gradients get tracked for a calculation. There are multiple ways to do this, depending on the situation. The easiest is just to set the requires grad flag directly, like so. And we can see that b1 has a grad function, but b2 does not, because we turned off history tracking in a prior to computing b2. If you only need autograd turned off temporarily, you can use the torch.nograd context manager. When we run this cell, we can see that history is tracked for all computations except the one in the nograd context. nograd can also be used as a function or method decorator, causing history tracking to be turned off for computations inside the decorated function. there's a corresponding context manager torch.enablegrad for turning autograd on in a local context. it may also be used as a decorator. finally, you may have a tensor tracking history, but you need a copy that doesn't. In this case, the tensor object has a detach method, which creates a copy of the tensor that is detached from the computation history. We did this above when we graphed some of our tensors. MapPlotLib expects a numpy array, but the implicit tensor and numpy array conversion is not enabled our tensors. MapPlotLib expects a NumPy array, but the implicit tensor and NumPy array conversion is not enabled for tensors tracking history. Once we make our attached copy, we're good to go. There's one more important note about Autograb mechanics. You have to be careful about using in-place operations on tensors tracking gradients. Doing so may destroy information you need to correctly do your backward pass later. In fact, PyTorch will even give you a runtime error if you try to perform an in-place operation on an input tensor that requires gradients. AutoGrad tracks every step of your tensor computation. Combining that information with time measurements would be useful for profiling gradient-tracked computations, and in fact, this feature is part of Auto tensor computation. Combining that information with time measurements would be useful for profiling gradient track computations. And in fact, this feature is part of AutoGrad. The next code cell shows basic usage of the profiler. The AutoGrad profiler can also group results by code blocks or input shape and can export results for other tracing tools. The link documentation has full details. PyTorch 1.5 saw the introduction of the AutoGrad high-level API, which exposes some of the core operations underlying AutoGrad. In order to explain this best, I'm going to go into some more mathematical depth on what AutoGrad is doing under the hood. So say you have a function with n inputs and m outputs. We'll say y equals a function of x. The complete set of partial derivatives of the outputs with respect to the inputs is a matrix called the Jacobian. Now, if you have a second function, which we'll call l, which equals g of y, that takes an m-dimensional input that is the same dimensionality as the output of our first function, and returns a scalar output, you can express its gradients with respect to y as a column vector. It's really just a one-column Jacobian. To tie this back to what we've been talking about, imagine the first function as your pie torch model, with potentially many inputs and many learning weights and many outputs, and the second function as a loss function, with the model's output as input and the loss value as the scalar output. If we multiply the first function's Jacobian by the gradient of the second function and apply the chain rule, we get another column vector. This column vector represents the partial derivatives of the second function with respect to the inputs of the first function, or in the case of our machine learning model, the partial derivative of the loss with respect to the learning weights. Torch.autograd is an engine for computing these vector Jacobian products. This is how we accumulate the gradients over the learning weights during the backward pass. This is how we accumulate the gradients over the learning weights during the backward pass. For this reason, the backward call can also take an optional vector input. The vector represents a set of gradients over the output tensor, which are multiplied by the Jacobian of the autograd trace tensor that precedes it. Let's try a specific example with a small vector. If we tried to call y.backward now, we'd get a runtime error, and a message that gradients can only be implicitly computed for scalar outputs. For a multidimensional output, AutoGrad expects us to provide the gradients for those three outputs that it can multiply into the Jacobian. Note that the output gradients here are all related to the Jacobian. Note that the output gradients here are all related to the power of the 2, which we'd expect from the repeated doubling operation in the previous cell. There's an API on AutoGrad that gives you direct access to important differential matrix and vector operations. In particular, it allows you to calculate the Jacobian and Hessian matrices of a particular function for particular inputs. The Hessian is like the Jacobian, but expresses all partial second derivatives. Let's take the Jacobian of a single function and evaluate it for two single element inputs. If you look closely, the first output should equal 2 times e to the x, since the derivative of e to the x is the exponential itself, and the second value should be 3. Now, you can of course do this with higher order tensors. Here we've computed Jacobian of that same adding function with a different set of inputs. There's also a function to directly compute the vector Jacobian product if you provide the vector. Autograd's JVP method does the same matrix multiplication as VJP with the operands reversed. The VHP and HVP methods do the same for the vector Hessian project. For more information including important performance notes, see the documentation for the new AutoGrad functional API.
The Fundamentals of Autograd
Autograd is the automatic gradient computation framework used with PyTorch tensors to speed the backward pass during training. This video covers the fundamentals of Autograd, including: the advantages of runtime computations tracking, the role of Autograd in model training, how to determine when Autograd is and is not active; profiling with Autograd, and Autograd's high-level API.

Download the Autograd notebook here: https://pytorch-tutorial-assets.s3.amazonaws.com/youtube-series/Video+3+-+Autograd.ipynb