 Welcome to the next video in the PyTorch training series. This video will talk about deploying your PyTorch model for inference in production. In particular, this video will talk about putting your PyTorch model in evaluation mode, converting your model to TorchScript and performing inference, using TorchScript with C++, and deploying your model with TorchServe, which is PyTorch's model-serving solution. No matter which deployment method you use, the first thing you should always do is put your model into evaluation mode. Evaluation mode is the opposite of training mode. It turns off training-related behaviors that you don't want during inference time. In particular, it turns off autograd. You may recall from the earlier video on autograd that PyTorch tensors, including your model's learning weights, track their computation history to aid the rapid computation of backward gradients for learning. This can be expensive in terms of both memory and compute and is not something you want to inference time. Eval mode also changes the behavior of certain modules that contain training-specific functionality. In particular, dropout layers are only active during training time. Setting your model in eval mode makes dropout a no-op. Batch norm layers track running stats on the computed mean and variance during training, but this behavior is turned off for eval mode. Next, let's look at the procedure for putting your model in evaluation mode. First, you'll want to load your model. For a Python-based model, that will involve loading the model's state dictionary from disk and initializing your object with it. Then, you call the eval method on your model, and you're done! Your model has now turned off training-related behaviors for inference. It's worth noting that the eval method is actually just an alias for calling the train method with an argument of false. You may find this useful if your code already contains a flag that indicates whether you're doing training or inference. Once you're in eval mode, you can start sending your model batches of data for inference. In the rest of this video, we're going to talk about different methods for deploying your model for inference, but for all of them, making sure your model is in evaluation mode is your first step. So what is TorchScript? It's a statically typed subset of Python for representing PyTorch models. And it's meant to be consumed by the JIT, the PyTorch Just-In-Time Compiler, which performs runtime optimizations to improve the performance of your model. It also allows you to save your model and weights in a single file and load them as a script module object that you can call just as you would your original model. So how do you use TorchScript? Build, test, and train your model in Python as you normally would. When you want to export your model for production inference, you can use the torch.jit.trace or torch.jjet.script calls to convert your model to TorchScript. After that, you can call the.save method on your TorchScript model to save it to a single file that contains both the computation graph and the learning weights for your model. The just-in-time compiler executes your TorchScript model, performing runtime optimizations such as operator fusion and batching matrix multiplications. You can also write your own custom extensions to TorchScript in C++. The code on the right shows what TorchScript looks like, but in the general case you won't have to edit it yourself. It's generated from your Python code. Let's walk through the process of using TorchScript in more detail. The process starts with the model you've built in Python and trained to the point of readiness for deployment. The next step is to convert your model to TorchScript. There are two ways to do this, torch.jit.script and torch.jit.trace. It's important to note the differences between the two techniques for converting your model to TorchScript. torch.jit.script converts your model by directly inspecting your code and running it through the TorchScript. Torch.Jet.Script converts your model by directly inspecting your code and running it through the TorchScript compiler. It preserves control flow, which you'll need if your forward function has conditionals or loops, and it accommodates common Python data structures. However, due to limitations of Python operator support in the TorchScript compiler, some models won't be convertible using torch.jet.script. Torch.jet.trace takes a sample input and traces it through your computation graph to generate the TorchScript version of your model. This doesn't suffer the operator coverage limitations of torch.jet.script, but because it only traces a single path through your code, it won't respect conditionals or other control flow structures that might cause variable or non-deterministic runtime behavior. It's also possible to mix tracing and scripting when converting a model. See the documentation for the Torch.Jet module for notes on mixing the two techniques. It's worth looking at the docs to see the optional arguments for script and trace. There are extra options for checking the consistency and tolerances of your TorchScript model. Now we'll save our TorchScript model. This saves both your computation graph and your learning weights in a single file, which means you don't have to ship the Python file with your model's class definition when you want to deploy to production. When it's time to do inference, you call torch.jet.load on your model, and feed it batches of input in the same way you would the Python version of your model. Everything I've shown you up to now has involved manipulating your model in Python code, even after you've converted it to TorchScript. There are situations and environments though, where you may need high throughput or real-time inference and would like to do without the overhead of the Python interpreter. It may also be the case that your production environment is already centered around C++ code, and you'd like to continue using C++ as much as possible. You may recall from an earlier video in this series that the important tensor computations in PyTorch happen in LibTorch, a compiled and optimized C++ library. PyTorch also has a C++ front end to this library. This means that you can load your TorchScript model in C++ and run it with no Python runtime dependencies. The first thing you'll need to do is to go to pytorch.org and download the latest version of LibTorch, unzip the package, and place it where your make system can find it. This slide shows a minimal CMake file for a project using LibTorch. Note that you'll need to be using C++14 or higher to make use of LibTorch. In Python, you'd import Torch, use torch.jit.load to bring your TorchScript model into memory, and then call your model with an input batch. The process is not so different in C++. First, include torch-script.h. This is your one-stop include for working with TorchScript in C++. Next, declare a TorchJetScript module variable, then use TorchJetLoad to load it into memory. To get predictions from your model, call its forward method with an appropriate input. Here we've created a dummy input with Torch1s. You'd be bringing in your own inputs of whatever size your model requires. Once you have your output predictions as a tensor, you can manipulate them with the C++ equivalents of the tensor methods you're used to in PyTorch's Python front-end. The PyTorch.org Tutorials section includes content walking you through setting up a C++ project as well as multiple tutorials demonstrating aspects of the C++ front-end. Setting up a production model serving environment can be complex, especially if you're serving multiple models, working with multiple versions of models, require scalability, or want detailed logging or metrics. TorchServe is the PyTorch model serving solution that covers all these needs and more. TorchServe loads instances of your model or models in individual process spaces and distributes incoming requests to them. It has a number of features to make it useful for creating ML-based web services. It has data handlers covering common use cases, including image classification and segmentation, object detection, and text classification. It allows you to set version identifiers for models, and you can manage and simultaneously serve multiple versions of a model. It can optionally batch input requests from multiple sources, which can sometimes improve throughput. It can optionally batch input requests from multiple sources, which can sometimes improve throughput. It features robust logging and the ability to log your own metrics. And it has separate RESTful APIs for inference and model management, which may be secured with HTTPS. I'll wrap up this video by walking through setting up and running TorchServe with one of the examples available at github.com slash pytorch slash serve in the examples folder. We'll set up a pre-trained image classification model for inference. First, let's install TorchServe. I'll demonstrate the process for setting it up on a Linux or Mac system, but TorchServe also works on Windows if that's your preferred server environment. First, I'm going to create a new conda environment for TorchServe also works on Windows if that's your preferred server environment. First, I'm going to create a new conda environment for TorchServe. I'm going to clone the source repository because it has convenient scripts for correctly installing TorchServe dependencies. TorchServe dependencies. When you run the dependency install script on a machine with NVIDIA GPUs, you may need to specify what version of the CUDA drivers you have installed. Details are in the install procedure described in TorchServe's README on GitHub. Since I'm installing on a Mac, I can skip that. Now with the dependencies installed, I can either install from source or use pip or conda. I'm actually installing two programs, TorchServe and the TorchModelArchiver, which we'll get to in a minute. If you're installing with conda, don't forget to specify the PyTorch channel with "-cpytorch". Next thing your TorchServe environment needs is a model store directory. All your models served by TorchServe are stored in this folder. You can name it anything you like, but I'm going to keep that simple. Next, we'll need a model to serve. anything you like, but I'm going to keep that simple. Next, we'll need a model to serve. TorchServe expects models to be packaged in a model archive, which contains your model's code and weights along with any other files needed to support your model. For example, in a natural language application, you might have embeddings or vocabularies that you need to package with your model. A model archive is created by the model archiver, which was the package I installed alongside TorchServe above. First, we'll need to download some trained model weights. Next, let's create a model archive from these weights. Taking these arguments one at a time, every model has a name here, DenseNet 161. A model needs a version number. Here we just went with 1.0. We're going to be using a Python-based model. So we use the model file flag to bring in the Python file containing the model class. The serialized file argument specifies the file containing the model weights. If we were loading a TorchScript model, we'd skip the model file argument and just specify the serialized TorchScript file here. We're going to bring in an extra support file, a JSON file containing mappings of the model's trained category IDs to human readable labels. Finally, every model archive needs a handler to transform and prepare incoming data for inference. I'm going to use the built-in image classifier handler, but it's also possible to write your own handler and specify that file here. Now you can see we have a.mar file. This is our model archive. It belongs in the model store, so let's put it there. Now let's start TorchServe. We'll do so with four arguments. The start flag should be self-explanatory. By default, TorchServe stores its current configuration and loads its last config on startup, and the NCS flag suppresses this behavior. The model store flag lets us specify our model store folder, and optionally, we can tell TorchServe to start with the model loaded. We'll specify our new model archive for DenseNet 161. TorchServe puts out a lot of helpful information, all of which is also saved in log files. Let's have a look at the Logs folder now. Note that a log directory has been created alongside our model store. And here you can see we have logs for all TorchServe's behavior and metrics. Now that TorchServe is running, let's do some inference. We'll grab a sample image from the source repo for our input, and then we'll call curl. On the TorchServe side, the default image classifier model takes care of unpacking the image and converting it to a tensor, feeding it to the model, and processing the output. This shows a simple case of using the TorchServe inference API over HTTP, but you can also access it via gRPC or use the KF Serving API used by Kubeflow on Kubernetes. And here we have the top five classes identified by the model. If we want to learn about the status of the server or manage which models we're serving or how many worker processes are devoted to each worker, we can use the management API. Above, use the prediction API on its default port of 8080. The default for the management API is port 8081. Let's use this curl command to see how the server reports what models it's serving. The models endpoint enumerates models being served, which right now is just our DenseNet model. Let's get a little more detail on it. And here you can see it specifies things about this particular model, including how many workers are spun up, etc. We can be more specific if we have more than one version of the model by adding the version number to the URL. This shows the default configuration for a served model with 12 workers running. You can also use the Management API to alter that configuration. So let's change the number of workers. So I set both the Min and Max workers to 4. And now if I ask for the status of our model again, we should see the number of workers has changed. The Management API lets you register new models from a local model archive or from a URL. It lets you unregister models or set the default version of a model to serve or get the status of a model or models that you're serving. Finally, we can stop TorchServe with the stop flag. The TorchServe GitHub repo also has walkthroughs and examples for many common tasks, including specific server management tasks, setting up HTTPS, writing a custom handler, and more. And as always, everything I've described here and more is documented fully in the documentation and tutorials at pytorch.org.
Production Inference Deployment with PyTorch
After you've built and trained a PyTorch machine learning model, the next step is to deploy it someplace where it can be used to do inferences on new input. This video shows the fundamentals of PyTorch production deployment, including Setting your model to evaluation mode; TorchScript, PyTorch's optimized model representation format; using PyTorch's C++ front end to deploy without interpreted language overhead; and TorchServe, PyTorch's solution for scaled deployment of ML inference services.