 Hello, my name is Brad Heinz. I'm a partner engineer working with the PyTorch team at Facebook. In this video, I'll be giving you an introduction to PyTorch, its features, key concepts, and associated tools and libraries. This overview assumes that you are new to doing machine learning with PyTorch. In this video, we're going to cover an overview of PyTorch and related projects, tensors, which are the core data abstraction of PyTorch, AutoGrad, which drives the eager mode computation that makes rapid iteration of your model possible. We'll talk about building a model with PyTorch modules. We'll talk about how to load your data efficiently to train your model. We'll demonstrate a basic training loop. And finally, we'll talk about deployment with TorchScript. Before we get started, you'll want to install PyTorch and TorchVision so that you can follow along with the demos and exercises. If you haven't installed the latest version of PyTorch yet, visit pytorch.org. The front page has an install wizard shown here. There are two important things to note here. First, CUDA drivers are not available for the Mac. Therefore, GPU acceleration is not going to be available by a PyTorch on the Mac. Second, if you're working on a Linux or Windows machine with one or more NVIDIA CUDA compatible GPUs attached, make sure the version of CUDA toolkit you install matches the CUDA drivers on your machine. So what is PyTorch? PyTorch.org tells us that PyTorch is an open source machine learning framework that accelerates the path from research prototyping to production deployment. Let's unpack that. First, PyTorch is software for machine learning. It contains a full toolkit for building and deploying ML applications, including deep learning primitives such as neural network layer types, activation functions, and gradient based optimizers. It has hardware acceleration on NVIDIA GPUs, and it has associated libraries for computer vision, text and natural language, and audio applications. TorchVision, the PyTorch library for computer vision, text and natural language, and audio applications. TorchVision, the PyTorch library for computer vision applications, also includes pre-trained models and package data sets that you can use to train your own models. PyTorch is built to enable fast iteration on your ML models and applications. You can work in regular idiomatic Python. There's no new domain-specific language to learn to build your computation graph. With AutoGrad, Pytorch's automatic differentiation engine, the backward pass over your model is done with a single function call and done correctly no matter which path through the code a computation took, offering you unparalleled flexibility in model design. Pytorch has the tooling to work at enterprise scale with tools like Torch Script, which is a way to create serializable and optimizable models from your PyTorch code, TorchServe, PyTorch's model serving solution, and multiple options for quantizing your model for performance. And finally, PyTorch is free and open source software, free to use and open to contributions from the community. Its open source nature fosters a rich ecosystem of community projects as well, supporting use cases from stochastic processes to graph-based neural networks. The PyTorch community is large and growing, with over 1,200 contributors to the project from around the world and over 50% year-on-year growth in research paper citations. PyTorch is in use at top-tier companies like these and provides the foundations for projects like Allen NLP, the open-source research library for deep learning with natural language, FastAI, which simplifies training fast and accurate neural nets using best modern practices, ClassyVision, an end-to-end framework for image and video classification, and Captum, an open-source extensible library that helps you understand and interpret your model's behavior. Now that you've been introduced to PyTorch, let's look under the hood. Tensors will be at the center of everything you do in PyTorch. Your model's inputs, outputs, and learning weights are all in the form of tensors. Now if tensor is not a part of your normal mathematical vocabulary, just know that in this context we're talking about a multi-dimensional arraydimensional array with a lot of extra bells and whistles. PyTorch tensors come bundled with over 300 mathematical and logical operations that can be performed on them. Though you access tensors through a Python API, the computation actually happens in compiled C++ code optimized for CPU and GPU. Let's look at some typical tensor manipulations in PyTorch. The first thing we'll need to do is import PyTorch with the importTorch call. Then we'll go ahead and create our first tensor. Here I'm going to create a two-dimensional tensor with five rows and three columns and fill it with zeros. I'm going to query it two-dimensional tensor with five rows and three columns and fill it with zeros. I'm going to query it for the data type of those zeros. And here you can see I got my requested matrix of 15 zeros, and the data type is 32-bit floating point. By default, PyTorch creates all tensors as 32-bit floating point. What if you wanted integers instead? You can always override the default. Here in the next cell, I create a tensor full of ones. I request that they be 16-bit integers. And note that when I print it without being asked, PyTorch tells me that these are 16-bit integers, because it's not the default and it might not be what I expect. It's common to initialize learning weights randomly, often with a specific seed for the random number generator so that you can reproduce your results on subsequent runs. Here we demonstrate seeding the PyTorch random number generator with a specific number, generating a random tensor, generating a second random tensor, which we expect to be different from the first, reseeding the random number generator with the same input, and then finally creating another random tensor, which we expect to match the first, since it was the first thing created after seeding the RNG. And sure enough, those are the results we get. First tensor and the third tensor do match, and the second one does not. Arithmetic with PyTorch tensors is intuitive. Tensors of similar shapes may be added, multiplied, etc. And operations between a scalar and a tensor will distribute over all the cells of the tensor. So let's look at a couple of examples. First, I'm just going to create a tensor full of 1s. Then I'm going to create another tensor full of 1s, but I'm going to multiply it by a scalar 2. And what's going to happen is all of those ones are going to become twos. The multiplication is distributed over every element of the tensor. Then I'll add the two tensors. I can do this because they're of the same shape. The operation happens element-wise between the two of them, and we get out now a tensor full of threes. When I query that tensor for its shape, it's the same shape as the two input tensors from the addition operation. Finally, I create two random tensors of different shapes and attempt to add them. I get a runtime error because there's no clean way to do element-wise arithmetic operations between two tensors of different shapes. Here's a small sample of the mathematical operations available on PyTorch tensors. I'm going to create a random tensor and adjust it so that its values are between minus 1 and 1. I can take the absolute value of it and see all the values turn positive. I can take the inverse sign of it, because the values are between minus 1 and 1, and get an angle back. I can do linear algebra operations like taking the determinant or doing singular value decomposition. And there are statistical and aggregate operations as well, means and standard deviations and minimums and maximums, etc. There's a good deal more to know about the power of PyTorch tensors, including how to set them up for parallel computation on GPU. We'll be going into more depth in another video. As an introduction to AutoGrad, PyTorch's automated differentiation engine, let's consider the basic mechanics of a single training pass. For this example, we'll use a simple recurrent neural network, or RNN. We start with four tensors, x, the input, h, the hidden state of the RNN that gives it its memory, and two sets of learning weights, one each for the input and the hidden state. Next, we'll multiply the weights by their respective tensors. mm here stands for matrix multiplication. After that, we add the outputs of the two matrix multiplications and pass the result through an activation function, here hyperbolic tangent. And finally, we compute the loss for this output. The loss is the difference between the correct output and the actual prediction of our model. So we've taken a training input, run it through a model, gotten an output, and determined the loss. This is the point in the training loop where we have to compute the derivatives of that loss with respect to every parameter of the model and use the gradients over the learning weights to decide how to adjust those weights in a way that reduces the loss. Even for a small model like this, that's a bunch of parameters and a lot of derivatives to compute. But here's the good news. You can do it in one line of code. Even for a small model like this, that's a bunch of parameters and a lot of derivatives to compute. But here's the good news. You can do it in one line of code. Each tensor generated by this computation knows how it came to be. For example, I2h carries metadata indicating that it came from the matrix multiplication of wx and x, and so it continues down the rest of the graph. This history tracking enables the backward method to rapidly calculate the gradients your model needs for learning. This history tracking is one of the things that enables flexibility and rapid iteration in your models. Even in a complex model with decision branches and loops, the computation history will track the particular path through the model that a particular input took and compute the backward derivatives correctly. In a later video, we'll show you how to do more tricks with AutoGrad, like using the AutoGrad profiler and taking second derivatives, and how to turn off AutoGrad when you don't need it. We've talked so far about tensors and automatic differentiation and some of the ways they interact with your PyTorch model. But what does that model look like in code? Let's build and run a simple one to get a feel for it. First, we're going to import PyTorch. We're also going to import torch.nn, which contains the neural network layers that we're going to compose into our model, as well as the parent class of the model itself. And we're going to import torch.nn.functional to give us activation functions and max pooling functions that we'll use to connect the layers. So, here we have a diagram of LENET5. It's one of the earliest convolutional neural networks and one of the drivers of the explosion in deep learning. It was built to read small images of handwritten numbers, the NNIST dataset, and correctly classify which digit was represented in the image. Here's the abridged version of how it works. Layer C1 is a convolutional layer, meaning that it scans the input image for features it learned during training. It outputs a map of where it saw each of its learned features in this image. This activation map is downsampled in layer S2. Layer C3 is another convolutional layer, this time scanning C1's activation map for combinations of features. It also puts out an activation map describing the spatial locations of these feature combinations, which is downsampled in layer S4. Finally, the fully connected layers at the end, F5, F6, and Output are a classifier that takes the final activation map and classifies it into one of 10 bins representing the 10 digits. So how do we express this simple neural network in code? Looking over this code, you should be able to spot some structural similarities with the diagram above. This demonstrates the structure of a typical PyTorch model. It inherits from torch.nn.module. And modules may be nested. In fact, even the COM2D and linear layers here are subclasses of torch.nn.module. Every model will have an init, where it constructs the layers that it will compose into its computation graph, and loads any data artifacts it might need. For example, an NLP model might load a vocabulary. A model will have a forward function. This is where the actual computation happens. An input is passed through the network layers and various functions to generate an output, a prediction. Other than that, you can build out your model class like any other Python class, adding whatever properties and methods you need to support your model's computation. So let's instantiate this and run an input through it. So there are a few important things happening here. We're creating an instance of limit. We are printing the net object. Now, subclass of torch.nn module will report the layers it has created and their shapes and parameters. This can provide a handy overview of a model if you want to get the gist of its processing. Below that, we create a dummy input representing a 32x32 image with one color channel. Normally you would load an image tile and convert it to a tensor of this shape. You may have noticed an extra dimension to our tensor. This is the batch dimension. PyTorch models assume they are working on batches of data. For example, a batch of 16 of our image tiles would have the shape 16 by 1 by 32 by 32. Since we're only using one image, we create a batch of one with shape 1 by 1 by 32 by 32. We ask the model for an inference by calling it like a function, net input. The output of this call represents the model's confidence that the input represents a particular digit. Since this instance of the model hasn't been trained, we shouldn't expect to see any signal in the output. Looking at the shape of the output, we can see that it also has a batch dimension, the size of which should always match the input batch dimension. Had we passed in an input batch of 16 instances, output would have a shape of 16 by 10. You've seen how a model is built and how to give it a batch of input and examine the output. The model didn't do much though, because it hasn't been trained yet. For that, we'll need to feed it a bunch of data. In order to train our model, we're going to need a way to feed it data in bulk. This is where the PyTorch dataset and data loader classes come into play. Let's see them in action. So here I'm declaring matplotlib inline, because we'll be rendering some images in the notebook. I'm importing PyTorch, and I'm also importing TorchVision and TorchVision transforms. These are going to give us our data sets and some transforms that we need to apply to the images to make them digestible by our PyTorch model. The first thing we need to do is transform our incoming images into a PyTorch tensor. Here, we specify two transformations for our input. TransformsToTensor takes images loaded by the pillow library and converts them into PyTorch tensors. Transforms.Normalize adjusts the values of the tensor so that their average is 0 and their standard deviation is 0.5. Most activation functions had their strongest gradients around the zero point, so centering our data there can speed learning. There are many more transforms available, including cropping, centering, rotation, reflection, and most of the other things you might do to an image. Next, we're going to create an instance of the CIFAR-10 data set. This is a set of 32 by 32 color image tiles representing 10 classes of objects, six of animals and four of vehicles. When you're in the cell above, it may take a minute or two for the data set to finish downloading for you. So be aware of that. So this is an example of creating a data set in PyTorch. Downloadable data sets, like CIFR 10 above, are subclasses of Torch, Utils, Data, DataSet. Data set classes in PyTorch include the downloadable data sets in TorchVision, TorchText, and TorchAudio, as well as utility dataset classes such as TorchVision.datasets.image folder, which will read a folder of labeled images. You can also create your own subclasses of dataset. When we instantiate our dataset, we need to tell it a few things. The file system path where we want the data to go, whether or not we're using this set for training, because data sets will be split between training and test subsets, whether we would like to download the data set if we haven't already, and the transformations that we want to apply to the images. Once you have your data set ready, you can give it to the data loader. Now, a data set subclass wraps access to the data and is specialized the type of the data is serving. The data loader knows nothing about the data, but organizes the input tensors served by the data set into batches with the parameters you specify. In the example above, we've asked a data loader to give us batches of four images from train set, randomizing their order with shuffle equals true, and we told it to spin up two workers to load data from disk. It's good practice to visualize the batches your data loader serves. Running the shell should show you a strip of four images, and you should see a correct label for each one. And so here are our four images, which do, in fact, look like a cat, a deer, and two trucks. We've looked under the hood at tensors and autograd, and we've seen how PyTorch models are constructed and how to efficiently feed them data. It's time to put all the pieces together and see how a model gets trained. So here we are back in our notebook. You'll see the imports here. All of these should look familiar from earlier in the video, except for torch.optim, which I'll be talking about soon. The first thing we'll need is training and test datasets. So if you haven't already, run the cell below and make sure the dataset is downloaded. It may take a minute if you haven't done so already. We'll run our check on the output from the data loader. And again, we should see a strip of four images, plain, plain, plain ship. That looks correct. So our data loader is good. This is the model we'll train. Now, if this model looks familiar, it's because it's a variant of Lynette, which we discussed earlier in this video. But it's adapted to take three-color images. The final ingredients we need are a loss function and an optimizer. The loss function, as discussed earlier in this video, is a measure of how far from our ideal output the model's prediction was. Cross entropy loss is a typical loss function for classification models like ours. The optimizer is what drives the learning. Here we've created an optimizer that implements stochastic gradient descent, one of the more straightforward optimization algorithms. Besides parameters of the algorithm, like the learning rate and momentum, we also pass in net dot parameters, which is a collection of all the learning weights in the model, which is what the optimizer adjusts. Finally, all of this is assembled into the training loop. Go ahead and run this cell, as it'll take a couple of minutes to execute. So here we're only doing two training epochs, as you can see from line one. That is two complete passes over the training data set. Each pass has an inner loop that iterates over the training data, serving batches of transformed images and their correct labels. Zeroing the gradients in line 9 is a very important step. When you run a batch, gradients are accumulated over that batch. And if we don't reset the gradients for every batch, they will keep accumulating and provide incorrect values and learning will stop. In line 12, we ask the model for its actual prediction on the batch. In the following line, line 13, we compute the loss, the difference between the outputs and the labels. In line 14, we do our backward pass and calculate the gradients that will direct the learning. In line 15, the optimizer performs one learning step. It uses the gradients from the backward call to nudge the learning weights in the direction it thinks will reduce the loss. So the remainder of the loop just does some light reporting on the epoch number and how many training instances have been completed and what the collective loss is over the training epoch. So note that the loss is monotonically descending, indicating that our model is continuing to improve its performance in the training dataset. As a final step, we should check that the model is actually doing general learning and not simply memorizing a dataset. This is called overfitting, and will often indicate that either your dataset is too small and doesn't have enough examples or that your model is too large. It's over specified for modeling the data you're feeding it. So our training is done. So anyways, the way we check for overfitting and guard against it is to test the model on data it hasn't trained on. That's why we have a test data set. So here I'm just going to run the test data through. We'll get an accuracy measure out. 55%. Okay. So that's not exactly state of the art, but it's much better than the 10% we'd expect to see from a random output. This demonstrates that some general learning did happen in the model. Now when you go to the trouble of building and training a non-trivial model, it's usually because you want to use it for something. You need to connect it to a system that feeds it inputs and processes the model's predictions. If you're keen on optimizing performance, you may want to do this without a dependency on the Python interpreter. The good news is that PyTorch accommodates you with TorchScript. TorchScript is a static, high-performance subset of Python. When you convert a model to TorchScript, the dynamic and Pythonic nature of your model is fully preserved. Control flow is preserved when converting to TorchScript, and you can still use convenient Python data structures like lists and dictionaries. Looking at the code on the right, you'll see a PyTorch model defined in Python. Below that, an instance of the model is created, and then we'll call torch.jit.script my module. That one line of code is all it takes to convert your Python model to Torch script. The serialized version of this gets saved in the final line, and it contains all the information about your model's computation graph and its learning weights. The TorchScript rendering of the model is shown at the right. TorchScript is meant to be consumed by the PyTorch Just-In-Time compiler, or JIT. The JIT seeks runtime optimizations, such as operation reordering and layer fusion, to maximize your model's performance on CPU or GPU hardware. So how do you load and execute a TorchScript model? You start by loading the serialized package with torch.jit.load, and then you can call it, just like any other model. What's more, you can do this in Python, or you can load it into the PyTorch C++ runtime to remove the interpreted language dependency. In subsequent videos, we'll go into more detail about TorchScript, best practices for deployment, and we'll cover TorchServe, PyTorch's model-serving solution. So that's our lightning-fast overview of PyTorch. The models and datasets we used here were quite simple, but PyTorch is used in production at large enterprises for powerful, real-world use cases, like translating between human languages, describing the content of video scenes, or generating realistic human voices. In the videos to follow, we'll give you access to that power. We'll go deeper on all the topics covered here with more complex use cases like the ones you'll see in the real world. Thank you for your time and attention, and I hope to see you around the PyTorch forums.
Introduction to PyTorch
In the first video of this series, we give a broad overview of the parts of the PyTorch toolchain, including: Tensors, automatic gradient computation, model building basics, data loading abstractions, model training, and deployment for inference. This video is meant as a survey, with each topic being covered in more depth in subsequent videos.

Download all notebooks here: https://pytorch-tutorial-assets.s3.amazonaws.com/youtube-series/video1.zip

Download individual notebooks here:

1. Tensors - 04:45 to 07:50
https://pytorch-tutorial-assets.s3.amazonaws.com/youtube-series/video1/1+-+PyTorch+Tensors.ipynb
 
2. Autograd  - 08:00 to 9:50
 
3. A simple model - 10:00 to 14:00
https://pytorch-tutorial-assets.s3.amazonaws.com/youtube-series/video1/2+-+A+Simple+PyTorch+model.ipynb
 
4. Datasets - 14:00 to 17:10
https://pytorch-tutorial-assets.s3.amazonaws.com/youtube-series/video1/3+-+Dataset+and+DataLoader.ipynb
 
5. Training loop - 17:10 to 21:00
https://pytorch-tutorial-assets.s3.amazonaws.com/youtube-series/video1/4+-+A+Simple+PyTorch+Training+Loop.ipynb