Writing Extensions and Bindings for GPU made Easy
Singh, Krishna Kanta
As Deep Learning Engineer and Researcher we are always trying to optimize some bottleneck computation in our programs. Sometimes we are faced with situations when scientific libraries like NumPy, SciPy aren't just cutting it or worse there are no libraries that implement the esoteric function on our expensive GPU hardware. *Writing Custom C and Cuda Extension becomes an important skill and necessity for applications that require really fast computation. * In this talk, we go through a detailed example of image search on billions of items, we write custom C and Cuda kernel for distance computation and learn how to connect them seamlessly with our python codebase. We compare methods for writing these extensions and bindings for python in terms of both speed and ease of use.
 Let's get started. I guess you guys can hear me. So my talk is about writing extensions and bindings for GPU in Python. And essentially, what do I do? I am basically a data scientist or research scientist at Rakudin. We work for a nonprofit in Rakudin organization called RIT, which is Rakudin Institute of Technology. And so let me go over the goals of this talk or presentation, which are basically this. So our goals are basically trying to extend but firstly, we show how to extend Python using native CRC++ code. And then we will show how you can extend GPU CUDA code and use it within your Python code and basically writing RAPL code for that. So why did I try to solve this problem in my job and what is the motivation for it? So at Rakudin, I am basically working on fashion recommendation system and in fashion recommendation system, I had the task of actually finding out distances between two feature vectors or two fashion items. And if you actually do a native pair wise distance comparison, it takes invariably too much time to do it. And so essentially there are certain things called FAIS library from Facebook team, which actually does pair wise distance comparison on GPUs and it's pretty fast. But there are certain issues with that library, one that the word vector or the word vector size that it requires is 128 or larger than that. And we had a resource constraint on the memory requirement that we could save our word vectors. So that's why we had to device algorithm by ourselves that was less in so basically wanted to implement pair wise, pair wise distance computation and CUDA with significantly less memory and also wanted to be really fast. So initially I tried doing it using Python, but it was too slow, as I say, and then I tried using other libraries like Python or sci-fi, but in there, I found out that if I wanted to get more speed, I had to write code and there was no getting rid of the fact. So one of the things that I wanted to show here is if you actually look at the K-means algorithm, here, I what I did here is for comparison basis, basic, I have two datasets, one is of 1 million data points and one is only 400 data points. And this is the speed. So this is the speed of various technologies like Python C++, C++ using Eigen, Eigen is a library that is used for basically matrix manipulations and C++. You can think of it as a STD for matrix, doing matrix stuff. And it's a pretty, it is a template based library. It's pretty fast. And then we have CUDA, we have scikit-learn and sci-fi. These are pretty standard libraries and Python scikit-learn and sci-fi and they're used always for like, their ubiquitous for being used with scientific computing. So similarly, if you look at n equal to 100 points, you can see that Python is the speed in seconds is around 0.014 seconds. And if you compare it with C or C++, it's pretty low. And this is just, but if you look at CUDA, you would see that the speed is pretty high. So one of the things that CUDA is not really good at is if you're trying to run CUDA code for a very small dataset. The main reason for that is the bottleneck when you're trying to develop CUDA applications is the data transfer time between your CPU memory to your GPU memory. So that's why it takes a lot of time to do it. But you can easily see one of the simple things you can see here from this diagram is that this is that C++ is much better than Python in terms of even optimized libraries like sci-fi and also scikit-learn, which internally actually use something similar to CUDA. They actually end up calling C++ code internal. So, but if you look at n equal to a million data points, when you're trying to do k-means algorithm, where you said k to be anything like in these experiments, I said them to get on file. And so here you can see that the good thing about this diagram is you can see that you can't even see the CUDA stack one because CUDA actually takes milliseconds to compute this. But if you look at Python, it is taking around more than seven seconds to do this. And think of this because if you're trying to do pairwise things, you have to compute a million cross million matrix and you have to recompute it again and again because feature vectors changes according to how because we have to recompute the feature vectors again and again because the uploaders actually change the images all the time by adding certain things. So that it will be probably the expensive to do pairwise computation again and again. So that's why we actually, we actually, you can see the need of using C++ or CUDA or in any bottleneck. So here are like basically a list of the pros and cons of writing extensions. One is definitely speed. Another is you can use optimized C and C++ library functions, meta library functions like for C and C++, most of the blast extensions are written in C or C++ you can actually use them. So that's one of the good things about writing extensions. If you want to write them, you will have no GIL in Python most of the speed issues come because of the GIL block, you can actually parallelize your code pretty easily if you do not have GIL and you can have more memory. Why not to do extensions is pretty similar why you don't write C and C++. It's development so it's hard to write and there are memory set situations. Unless you want a 10x program. So here I just go over one of the simple, how do you write a simple CPU extension? So this is basically first you write a native code. This is can be thought of as a kernel code that will be written in C++ or CPP. And essentially what I have done here is written in a addition of two vectors, A and B and put the results in C. So this is just a for loop that looks over this. And you can actually do this in C Python as well. But it would be very, very too slow because loops in Python are real slow. And if we can vectorize somewhere we should vectorize. So then how do you do this? So I talk about this a little bit later. These technologies that are available for wrapping our code are C code and being able to we call it using Python code. So here I have shown the example using Cython. There are other in my GitHub repository, there are other examples of using Cython 5x11 and Ctypes. But here I've just shown you two space limitations. Just one example. So how do you write that? So what you do is you, in Cython what you do is you import C import numpy as NP. And this actually gives you numpy data. Like you can actually import direct numpy data types and call them in the Python function and then pass them on to your C functions. And you can actually then define CPU depth, adder CPU. And it actually takes in the argument of double, so essentially it's if you want to think of it, it's just a pointer to a memory buffer of numpy. And that is actually passed to adder CPU, which actually stores it in these data types of NP and C and then passes it to the native code. Then how do you set it up? So you have to, for distributing code it's pretty simple. You, by using the setup of the desktop, you can just add one line of extensions and add this sources of add and disco CPU and it will build like seamlessly as you just tried Python set of desktop. So after this I'll just go over what is a GPU and how GPU, how do you write codes on the desktop? So basically I like to think GPU as CPU with thousands of cores. If your CPU has four cores or 16 cores, the GPU would have like 1000 more cores than that. And that's why you can parallelize your any computation easily there. It is also called SINB, single instruction multiple data. So the same instruction set is run upon multiple data. That can be done using this. Yeah, but there is obviously no free lunch. As you can see like, see, GPUs are not great at control of it because they're controlled like the controller. If there is many if and else branches in your code, then GPUs are really bad at that because there is something called thread divergence, which happens. And GPUs would be very slow in that. But GPUs are very good at like single instruction multiple data set, because they have multiple units. So they can perform automatic really fast. Another thing that they're really shorter is the DRAM. So or the memory that a GPU has. So GPU for mostly has like 16 GB to 32 GB of RAM in the expensive case. But like, so you are mostly constrained by the amount of data you can put on your GPUs rather than being constrained upon how much computation you can do. So I go over some terminology before just how the code is written. So I go over some of the terms. So basically for terminology, essence, so if you read any GPU blogs on the video.com, they typically call it an there are some so basically GPUs are made up of essence, essence are streaming multi process and GPUs made up of two or three, generally two or four, essence. Then essence are then divided into grace. SM would have typically have four to eight grids. And this is confer configurable obviously, but there is a limit to how many grids you can have. So basically think of this a grid as a grid is a computational unit, you can think of it as a mini processor in some sense, a mini code. So you can have a two 16 grid something the 16 cores on one SN. And then every every grid has blocks within itself. So blocks can be 2D blocks or one block and blocks are blocks are unit of computation again. So yeah, like here, like I have shown like a grid, it contains many blocks. So these are truly blocks and they're indexed according to accordingly like zero, zero, one, zero, two, zero, and then zero, one, one, one, one block is then made up of something called threads threads. If are the smallest unit of multi like processing in GPUs, there are some conceptually, you can say that, but there is something called wraps. So wraps are actually memory aligned threads. These are 32 bit memory 32 threads which are aligned. And these are mostly required if you're doing real hardcore performance optimizations. Otherwise, generally wraps are not that important. So but threads are multi like so threads are basically your CPU friends as a thread computes a single unit, like it's a single program that it computes on a single data point. So if you can look here, so again, threads can be 2D or 1D. So CPU grid would is made up of blocks and blocks are then made up of threads. So the smallest unit is a thread which is now indexed by zero, zero. This would be in block zero, zero and in the grid, similarly, so that's how you index the place. So you need to get the thread index. So let's try to see this comment by some by a person called Mark human explains it in a comment. So GPUs are made up of grids, which are in terms of blocks and blocks are made up of this. So threads, you can then say that threads are somewhat equivalent to cores in CPU. So if you have six cores in CPU, but if you have 20,000 threads in your GPU, you would have essentially 20,000 cores in your GPU. Yeah, and the synchronization is done in a block level in in GPUs. So yeah, corporate cooperation is done at block levels. So what do I mean by corporation? It essentially means that thread zero can share your data with three, and this that's how cooperation is done in terms of because they have a shared memory, which they can write to and read from and perform combinations and begin right back. So that's how synchronization. They share the same share. And share memories are very important concept in GPUs. When you're writing performance optimized code like make for matrix transport, transpose, you will definitely be using share. So let's look at. Yeah, so we look at some code later, but so suppose that you do not want to write this code directly in the language, you would prefer to write it in Python. So there are some alternatives for that. One of the alternatives is number. So number is open source and pretty active development is there. It has both CPU and GPU support and it and it's pretty fast. So your code can actually run. You can vectorize it on CPUs or you can write it for which is really great. And it has access to libraries like QFFT, Q-Ryne, which are very important once you start doing actual. So PyCuda has full code API available and it has saved code writing because objects have limited lifetime attached to them. It is a concept called resource allocation. I don't remember the code form, but it's a concept called Rye and it's very prevalent with CNC++ users. So when an object goes out of, it's no longer needed, it's actually deleted. So that's what Rye is, once actually. Then you have Q by Q by is numpy for GPUs. So it has tried to actually replicate every numpy function on GPUs and it is supported by NPUs. So one of the issues with using all these libraries, I tried using all the number and Q by NPUs. One of them is it's harder to debug them. And if you actually want to write something for very, which are not, which cannot be written in NUMBA or Q by, it is actually some Q by you would have to actually write CUDA kernels and then actually import them to CUDA as strings. And then it's actually a key to writing custom kernels in CUDA. For NUMBA, one of the things is that if you want to write performance optimization code, then you cannot rely on NUMBA. You have to actually understand and get to the architecture of CUDA and then write CUDA code. There is no other way as I found out. So let's go over one of the similar examples of adding to... So let's go over a similar example of CUDA code. So here again, I'm trying to add two vectors and slowing the result in N. What I need here is the thread index. So as I said, thread is the basic unit of computation in GPUs. So every thread will now... So suppose that this A is a vector of size 256, then I can initialize my... So I can initialize 0, so blocks would be 0, and threads would be 256. So every thread is now responsible for one unit of data point. So like A0 is responsible for... So thread 0 is responsible for A0, thread 2 is responsible for A2 and similarly. So once you get the thread index, then you can use that index to actually get the data point, the index from A, and then you can add it. So then it is actually a computation which is done in parallel. So this is a pretty simple code. It gets actually... So this would be your main kernel function that would actually be run on the device code. That's why it's labeled as global. And yeah, I think that's about it. But how do you actually run this? So there is something called a driver function that actually calls the above kernel function. So firstly, what you would need to do is you need to allocate all these on your device memory. Device memory is your CPU memory. So you actually assign just a CUDA maloc code to... Oh, sorry. So CUDA maloc actually assigns... Can assign to both your host memory or your device memory as well. So here firstly, we are trying to assign memory on the device using CUDA maloc for dead-B and everything like dead-B and dead-C, which would actually be storing the computations on GPU. And then we actually copy the bytes from the host memory which was double A to a device memory which is dev underscore A. And that's why CUDA mem copy was to device. And then we actually initialize the number of threads to be 256 because we want it to be... Yeah, because we want to have 256 threads. Your A can be much longer than 256 as well. And then this formula comes into picture, which is n minus one divided by number of threads plus one. What this does essentially is it actually divides your data so that every thread... So it's done like your data is handled fully because it could be 512. Then you would need to have actually two blocks rather than just one. So that's what this code does. And then you actually do a kernel invocation which is done using this code. Basically, you say that this is the number of threads I require on my GPU. So the GPU assigns them to you and then you pass on your, basically, the pointers of the data where you have stored them and the computations. And then you actually then copy back the answer of dev C to C here. And this is here. And this is our device to host. And then you can use CUDA memory. So the wrapper is very similar to a wrapper in the CPU case, which is basically... I think it's very similar because we are dealing with pointers. So we just pass the pointers from the NumPy memory buffer here. And then we actually pass those pointers to this invocating code and get their pass-by reference here. The important part here that I will not be able to fully write down here is the setup dot pi by pi. So I struggled a lot with the setup dot pi by pi for GPUs because you need to actually add NVCC as a compiler to the setup dot pi by underscore compile function, which actually then understands, okay, that this is a CUDA code, CU code, then this should be compiled using NVCC. And if you just do this, it won't work. You would have to have some extra hacks, which are there in my data code. So you can have a full look there. But what this is saying is the sources like are my CUDA code and my wrapper that are in the CPU code as well. I need to add some certain libraries like CUDA lib64 and CUDA hard. And similarly, the runtime arguments. So essentially, if you have ever used the make command or ever compile C, C, C or C++, you actually need to pass out using minus the linker flags and the link libraries using minus L and minus I flag including. So this is similar to that. That's what we are doing here. You need to pass some other things to NVCC, certain flags like what architecture you want to compile it for, some compiler option like optimizations that you want. So in conclusion, what I had learned from this is if you need to do rapid prototyping or something and you want to write some custom CUDA commands, you should use NumPy and Cupid. If not, then it's better to write CUDA code on your own and do debugging upon that because when you're writing CUDA code, you would have to debug a lot to get better performance because one of the important things in getting better performance is finding out what number of blocks, what number of threads you need to use and then here as well because if it's a complex code, then you need to see how you would be sharing memory, how you would be writing the parallel code for parallel implementation. For more, you can look at the NVIDIA blog that there is where they go over transposing matrix in many forms and how it is done. So yeah, and for binding my conclusion were generally pretty simple. I preferred Scython in most of my work when I was deploying to production. Here the performance I found out to be much better but then the steep learning of because Scython is a purely new language. It has different syntax for functions and different syntax for types. If you are familiar with the C, you would be very like it's not that hard to get because you have C there, you have Cp there and then you have double basically it's an array notation in Scython. Pi bind is easiest to set up and it's the only option when you're working with C++ like it has native C++ support for native data source STD containers but what I found that the performance is not good by passing memory buffers and also that it's pretty tricky to solve certain issues in pi-v. Mainly I'm talking about like if you go to their docs, you can actually find out known issues that they have by dealing with data types from STD buffers and like this. Then C types are great for simple cases like they are pure Python. I did not explore this much but like I've seen many libraries using C types because they are just pointers that are passed. Yeah, so my conclusion for CUDA is that it has a steep learning curve but there are great performance returns in production. If you have some really critical performance intensive code that you have to write then you should basically learn CUDA and try to do it because GPUs are a very expensive resource at any company when you're trying to go to production and proper utilization of CUDA, I don't think most of the data scientists know and so I think that's something we need to learn to do. I'm still learning there's a lot to learn in CUDA. Yeah, for more example like my K-Name implementation and my pair-wise implementation you can check my GitHub which I'll throw in the link in the references. So yeah, that's about it. Is there any questions? Thank you.
 Let's get started. I guess you guys can hear me. So my talk is about writing extensions  and bindings for GPU in Python. And essentially, what do I do? I am basically a data scientist  or research scientist at Rakudin. We work for a nonprofit in Rakudin organization called  RIT, which is Rakudin Institute of Technology. And so let me go over the goals of this talk  or presentation, which are basically this. So our goals are basically trying to extend  but firstly, we show how to extend Python using native CRC++ code. And then we will  show how you can extend GPU CUDA code and use it within your Python code and basically  writing RAPL code for that. So why did I try to solve this problem in my job and what  is the motivation for it? So at Rakudin, I am basically working on fashion recommendation  system and in fashion recommendation system, I had the task of actually finding out distances  between two feature vectors or two fashion items. And if you actually do a native pair  wise distance comparison, it takes invariably too much time to do it. And so essentially  there are certain things called FAIS library from Facebook team, which actually does pair  wise distance comparison on GPUs and it's pretty fast. But there are certain issues  with that library, one that the word vector or the word vector size that it requires is  128 or larger than that. And we had a resource constraint on the memory requirement that  we could save our word vectors. So that's why we had to device algorithm by ourselves  that was less in so basically wanted to implement pair wise, pair wise distance computation  and CUDA with significantly less memory and also wanted to be really fast. So initially  I tried doing it using Python, but it was too slow, as I say, and then I tried using  other libraries like Python or sci-fi, but in there, I found out that if I wanted to  get more speed, I had to write code and there was no getting rid of the fact. So one of  the things that I wanted to show here is if you actually look at the K-means algorithm,  here, I what I did here is for comparison basis, basic, I have two datasets, one is  of 1 million data points and one is only 400 data points. And this is the speed. So this  is the speed of various technologies like Python C++, C++ using Eigen, Eigen is a library  that is used for basically matrix manipulations and C++. You can think of it as a STD for  matrix, doing matrix stuff. And it's a pretty, it is a template based library. It's pretty  fast. And then we have CUDA, we have scikit-learn and sci-fi. These are pretty standard libraries  and Python scikit-learn and sci-fi and they're used always for like, their ubiquitous for  being used with scientific computing. So similarly, if you look at n equal to 100 points, you  can see that Python is the speed in seconds is around 0.014 seconds. And if you compare  it with C or C++, it's pretty low. And this is just, but if you look at CUDA, you would  see that the speed is pretty high. So one of the things that CUDA is not really good at  is if you're trying to run CUDA code for a very small dataset. The main reason for that  is the bottleneck when you're trying to develop CUDA applications is the data transfer time  between your CPU memory to your GPU memory. So that's why it takes a lot of time to do  it. But you can easily see one of the simple things you can see here from this diagram  is that this is that C++ is much better than Python in terms of even optimized libraries  like sci-fi and also scikit-learn, which internally actually use something similar to CUDA. They  actually end up calling C++ code internal. So, but if you look at n equal to a million  data points, when you're trying to do k-means algorithm, where you said k to be anything  like in these experiments, I said them to get on file. And so here you can see that  the good thing about this diagram is you can see that you can't even see the CUDA stack  one because CUDA actually takes milliseconds to compute this. But if you look at Python,  it is taking around more than seven seconds to do this. And think of this because if you're  trying to do pairwise things, you have to compute a million cross million matrix and  you have to recompute it again and again because feature vectors changes according to how because  we have to recompute the feature vectors again and again because the uploaders actually change  the images all the time by adding certain things. So that it will be probably the expensive  to do pairwise computation again and again. So that's why we actually, we actually, you  can see the need of using C++ or CUDA or in any bottleneck. So here are like basically  a list of the pros and cons of writing extensions. One is definitely speed. Another is you can  use optimized C and C++ library functions, meta library functions like for C and C++,  most of the blast extensions are written in C or C++ you can actually use them. So that's  one of the good things about writing extensions. If you want to write them, you will have no  GIL in Python most of the speed issues come because of the GIL block, you can actually  parallelize your code pretty easily if you do not have GIL and you can have more memory.  Why not to do extensions is pretty similar why you don't write C and C++. It's development  so it's hard to write and there are memory set situations. Unless you want a 10x program.  So here I just go over one of the simple, how do you write a simple CPU extension? So  this is basically first you write a native code. This is can be thought of as a kernel  code that will be written in C++ or CPP. And essentially what I have done here is written  in a addition of two vectors, A and B and put the results in C. So this is just a for loop  that looks over this. And you can actually do this in C Python as well. But it would  be very, very too slow because loops in Python are real slow. And if we can vectorize somewhere  we should vectorize. So then how do you do this? So I talk about this a little bit later.  These technologies that are available for wrapping our code are C code and being able  to we call it using Python code. So here I have shown the example using Cython. There  are other in my GitHub repository, there are other examples of using Cython 5x11 and Ctypes.  But here I've just shown you two space limitations. Just one example. So how do you write that?  So what you do is you, in Cython what you do is you import C import numpy as NP. And  this actually gives you numpy data. Like you can actually import direct numpy data types  and call them in the Python function and then pass them on to your C functions. And you  can actually then define CPU depth, adder CPU. And it actually takes in the argument  of double, so essentially it's if you want to think of it, it's just a pointer to a memory  buffer of numpy. And that is actually passed to adder CPU, which actually stores it in  these data types of NP and C and then passes it to the native code. Then how do you set  it up? So you have to, for distributing code it's pretty simple. You, by using the setup  of the desktop, you can just add one line of extensions and add this sources of add and  disco CPU and it will build like seamlessly as you just tried Python set of desktop. So  after this I'll just go over what is a GPU and how GPU, how do you write codes on the  desktop? So basically I like to think GPU as CPU with thousands of cores. If your CPU  has four cores or 16 cores, the GPU would have like 1000 more cores than that. And that's  why you can parallelize your any computation easily there. It is also called SINB, single  instruction multiple data. So the same instruction set is run upon multiple data. That can be  done using this. Yeah, but there is obviously no free lunch. As you can see like, see,  GPUs are not great at control of it because they're controlled like the controller. If  there is many if and else branches in your code, then GPUs are really bad at that because  there is something called thread divergence, which happens. And GPUs would be very slow  in that. But GPUs are very good at like single instruction multiple data set, because they  have multiple units. So they can perform automatic really fast. Another thing that they're really  shorter is the DRAM. So or the memory that a GPU has. So GPU for mostly has like 16 GB  to 32 GB of RAM in the expensive case. But like, so you are mostly constrained by the  amount of data you can put on your GPUs rather than being constrained upon how much computation  you can do. So I go over some terminology before just how the code is written. So I  go over some of the terms. So basically for terminology, essence, so if you read any GPU  blogs on the video.com, they typically call it an there are some so basically GPUs are  made up of essence, essence are streaming multi process and GPUs made up of two or three,  generally two or four, essence. Then essence are then divided into grace. SM would have  typically have four to eight grids. And this is confer configurable obviously, but there  is a limit to how many grids you can have. So basically think of this a grid as a grid  is a computational unit, you can think of it as a mini processor in some sense, a mini  code. So you can have a two 16 grid something the 16 cores on one SN. And then every every  grid has blocks within itself. So blocks can be 2D blocks or one block and blocks are  blocks are unit of computation again. So yeah, like here, like I have shown like a grid,  it contains many blocks. So these are truly blocks and they're indexed according to accordingly  like zero, zero, one, zero, two, zero, and then zero, one, one, one, one block is then  made up of something called threads threads. If are the smallest unit of multi like processing  in GPUs, there are some conceptually, you can say that, but there is something called  wraps. So wraps are actually memory aligned threads. These are 32 bit memory 32 threads  which are aligned. And these are mostly required if you're doing real hardcore performance  optimizations. Otherwise, generally wraps are not that important. So but threads are multi  like so threads are basically your CPU friends as a thread computes a single unit, like it's  a single program that it computes on a single data point. So if you can look here, so again,  threads can be 2D or 1D. So CPU grid would is made up of blocks and blocks are then made  up of threads. So the smallest unit is a thread which is now indexed by zero, zero. This would  be in block zero, zero and in the grid, similarly, so that's how you index the place. So you  need to get the thread index. So let's try to see this comment by some by a person called  Mark human explains it in a comment. So GPUs are made up of grids, which are in terms of  blocks and blocks are made up of this. So threads, you can then say that threads are  somewhat equivalent to cores in CPU. So if you have six cores in CPU, but if you have  20,000 threads in your GPU, you would have essentially 20,000 cores in your GPU. Yeah,  and the synchronization is done in a block level in in GPUs. So yeah, corporate cooperation  is done at block levels. So what do I mean by corporation? It essentially means that  thread zero can share your data with three, and this that's how cooperation is done in  terms of because they have a shared memory, which they can write to and read from and  perform combinations and begin right back. So that's how synchronization. They share  the same share. And share memories are very important concept in GPUs. When you're writing  performance optimized code like make for matrix transport, transpose, you will definitely  be using share. So let's look at.  Yeah, so we look at some code later, but so suppose that you do not want to write this  code directly in the language, you would prefer to write it in Python. So there are some alternatives  for that. One of the alternatives is number. So number is open source and pretty active  development is there. It has both CPU and GPU support and it and it's pretty fast. So  your code can actually run. You can vectorize it on CPUs or you can write it for which is  really great. And it has access to libraries like QFFT, Q-Ryne, which are very important  once you start doing actual. So PyCuda has full code API available and it has saved code  writing because objects have limited lifetime attached to them. It is a concept called resource  allocation. I don't remember the code form, but it's a concept called Rye and it's very  prevalent with CNC++ users. So when an object goes out of, it's no longer needed, it's actually  deleted. So that's what Rye is, once actually. Then you have Q by Q by is numpy for GPUs.  So it has tried to actually replicate every numpy function on GPUs and it is supported  by NPUs. So one of the issues with using all these libraries, I tried using all the number  and Q by NPUs. One of them is it's harder to debug them. And if you actually want to  write something for very, which are not, which cannot be written in NUMBA or Q by, it is  actually some Q by you would have to actually write CUDA kernels and then actually import  them to CUDA as strings. And then it's actually a key to writing custom kernels in CUDA.  For NUMBA, one of the things is that if you want to write performance optimization code,  then you cannot rely on NUMBA. You have to actually understand and get to the architecture  of CUDA and then write CUDA code. There is no other way as I found out. So let's go over  one of the similar examples of adding to... So let's go over a similar example of CUDA  code. So here again, I'm trying to add two vectors and slowing the result in N. What  I need here is the thread index. So as I said, thread is the basic unit of computation in  GPUs. So every thread will now... So suppose that this A is a vector of size 256, then  I can initialize my... So I can initialize 0, so blocks would be 0, and threads would  be 256. So every thread is now responsible for one unit of data point. So like A0 is  responsible for... So thread 0 is responsible for A0, thread 2 is responsible for A2 and  similarly. So once you get the thread index, then you can use that index to actually get  the data point, the index from A, and then you can add it. So then it is actually a computation  which is done in parallel. So this is a pretty simple code. It gets actually... So this would  be your main kernel function that would actually be run on the device code. That's why it's  labeled as global. And yeah, I think that's about it. But how do you actually run this?  So there is something called a driver function that actually calls the above kernel function.  So firstly, what you would need to do is you need to allocate all these on your device  memory. Device memory is your CPU memory. So you actually assign just a CUDA maloc code  to... Oh, sorry. So CUDA maloc actually assigns... Can assign to both your host memory or your  device memory as well. So here firstly, we are trying to assign memory on the device using  CUDA maloc for dead-B and everything like dead-B and dead-C, which would actually be  storing the computations on GPU. And then we actually copy the bytes from the host memory  which was double A to a device memory which is dev underscore A. And that's why CUDA  mem copy was to device. And then we actually initialize the number of threads to be 256  because we want it to be... Yeah, because we want to have 256 threads. Your A can be  much longer than 256 as well. And then this formula comes into picture, which is n minus  one divided by number of threads plus one. What this does essentially is it actually  divides your data so that every thread... So it's done like your data is handled fully  because it could be 512. Then you would need to have actually two blocks rather than just  one. So that's what this code does. And then you actually do a kernel invocation which  is done using this code. Basically, you say that this is the number of threads I require  on my GPU. So the GPU assigns them to you and then you pass on your, basically, the  pointers of the data where you have stored them and the computations. And then you actually  then copy back the answer of dev C to C here. And this is here. And this is our device to  host. And then you can use CUDA memory. So the wrapper is very similar to a wrapper in  the CPU case, which is basically... I think it's very similar because we are dealing with  pointers. So we just pass the pointers from the NumPy memory buffer here. And then we  actually pass those pointers to this invocating code and get their pass-by reference here.  The important part here that I will not be able to fully write down here is the setup  dot pi by pi. So I struggled a lot with the setup dot pi by pi for GPUs because you need  to actually add NVCC as a compiler to the setup dot pi by underscore compile function,  which actually then understands, okay, that this is a CUDA code, CU code, then this should  be compiled using NVCC. And if you just do this, it won't work. You would have to have  some extra hacks, which are there in my data code. So you can have a full look there. But  what this is saying is the sources like are my CUDA code and my wrapper that are in the  CPU code as well. I need to add some certain libraries like CUDA lib64 and CUDA hard. And  similarly, the runtime arguments. So essentially, if you have ever used the make command or  ever compile C, C, C or C++, you actually need to pass out using minus the linker flags  and the link libraries using minus L and minus I flag including. So this is similar to that.  That's what we are doing here. You need to pass some other things to NVCC, certain flags  like what architecture you want to compile it for, some compiler option like optimizations  that you want. So in conclusion, what I had learned from this is if you need to do rapid  prototyping or something and you want to write some custom CUDA commands, you should use  NumPy and Cupid. If not, then it's better to write CUDA code on your own and do debugging  upon that because when you're writing CUDA code, you would have to debug a lot to get  better performance because one of the important things in getting better performance is finding  out what number of blocks, what number of threads you need to use and then here as well  because if it's a complex code, then you need to see how you would be sharing memory, how  you would be writing the parallel code for parallel implementation. For more, you can  look at the NVIDIA blog that there is where they go over transposing matrix in many forms  and how it is done. So yeah, and for binding my conclusion were generally pretty simple.  I preferred Scython in most of my work when I was deploying to production. Here the performance  I found out to be much better but then the steep learning of because Scython is a purely  new language. It has different syntax for functions and different syntax for types.  If you are familiar with the C, you would be very like it's not that hard to get because  you have C there, you have Cp there and then you have double basically it's an array notation  in Scython. Pi bind is easiest to set up and it's the only option when you're working  with C++ like it has native C++ support for native data source STD containers but what  I found that the performance is not good by passing memory buffers and also that it's  pretty tricky to solve certain issues in pi-v. Mainly I'm talking about like if you go to  their docs, you can actually find out known issues that they have by dealing with data  types from STD buffers and like this. Then C types are great for simple cases like they  are pure Python. I did not explore this much but like I've seen many libraries using C  types because they are just pointers that are passed.  Yeah, so my conclusion for CUDA is that it has a steep learning curve but there are great  performance returns in production. If you have some really critical performance intensive  code that you have to write then you should basically learn CUDA and try to do it because  GPUs are a very expensive resource at any company when you're trying to go to production  and proper utilization of CUDA, I don't think most of the data scientists know and so I  think that's something we need to learn to do. I'm still learning there's a lot to learn  in CUDA. Yeah, for more example like my K-Name implementation and my pair-wise implementation  you can check my GitHub which I'll throw in the link in the references. So yeah, that's  about it. Is there any questions? Thank you.
Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_0_3455.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_3456_7619.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_7620_7775.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_7776_8771.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_8772_10211.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_10212_11267.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_11268_12467.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_12468_13127.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_13128_13739.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_13740_16007.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_16008_16127.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_16128_19235.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_19236_20831.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_20832_21875.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_21876_27083.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_27084_27143.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_27144_27323.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_27324_27527.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_27528_28679.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_28680_30347.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_30348_30719.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_30720_31511.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_31512_37271.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_37272_38351.png Krishna_Kanta_Singh_-_Writing_Extensions_and_Bindi_Q3VBIOf2shg_38352_38597.png
Let's get started. I guess you guys can hear me. So my talk is about writing extensions and bindings for GPU in Python. And essentially, what do I do? I am basically a data scientist or research scientist at Rakudin. We work for a nonprofit in Rakudin organization called RIT, which is Rakudin Institute of Technology. And so let me go over the goals of this talk or presentation, which are basically this. So our goals are basically trying to extend but firstly, we show how to extend Python using native CRC++ code. And then we will show how you can extend GPU CUDA code and use it within your Python code and basically writing RAPL code for that. So why did I try to solve this problem in my job and what is the motivation for it? So at Rakudin, I am basically working on fashion recommendation system and in fashion recommendation system, I had the task of actually finding out distances between two feature vectors or two fashion items. And if you actually do a native pair wise distance comparison, it takes invariably too much time to do it. And so essentially there are certain things called FAIS library from Facebook team, which actually does pair wise distance comparison on GPUs and it's pretty fast. But there are certain issues with that library, one that the word vector or the word vector size that it requires is 128 or larger than that. And we had a resource constraint on the memory requirement that we could save our word vectors. So that's why we had to device algorithm by ourselves that was less in so basically wanted to implement pair wise, pair wise distance computation and CUDA with significantly less memory and also wanted to be really fast. So initially I tried doing it using Python, but it was too slow, as I say, and then I tried using other libraries like Python or sci-fi, but in there, I found out that if I wanted to get more speed, I had to write code and there was no getting rid of the fact. So one of the things that I wanted to show here is if you actually look at the K-means algorithm, here, I what I did here is for comparison basis, basic, I have two datasets, one is of 1 million data points and one is only 400 data points. And this is the speed. So this is the speed of various technologies like Python C++, C++ using Eigen, Eigen is a library that is used for basically matrix manipulations and C++. You can think of it as a STD for matrix, doing matrix stuff. And it's a pretty, it is a template based library. It's pretty fast. And then we have CUDA, we have scikit-learn and sci-fi. These are pretty standard libraries and Python scikit-learn and sci-fi and they're used always for like, their ubiquitous for being used with scientific computing. So similarly, if you look at n equal to 100 points, you can see that Python is the speed in seconds is around 0.014 seconds. And if you compare it with C or C++, it's pretty low. And this is just, but if you look at CUDA, you would see that the speed is pretty high. So one of the things that CUDA is not really good at is if you're trying to run CUDA code for a very small dataset. The main reason for that is the bottleneck when you're trying to develop CUDA applications is the data transfer time between your CPU memory to your GPU memory. So that's why it takes a lot of time to do it. But you can easily see one of the simple things you can see here from this diagram is that this is that C++ is much better than Python in terms of even optimized libraries like sci-fi and also scikit-learn, which internally actually use something similar to CUDA. They actually end up calling C++ code internal. So, but if you look at n equal to a million data points, when you're trying to do k-means algorithm, where you said k to be anything like in these experiments, I said them to get on file. And so here you can see that the good thing about this diagram is you can see that you can't even see the CUDA stack one because CUDA actually takes milliseconds to compute this. But if you look at Python, it is taking around more than seven seconds to do this. And think of this because if you're trying to do pairwise things, you have to compute a million cross million matrix and you have to recompute it again and again because feature vectors changes according to how because we have to recompute the feature vectors again and again because the uploaders actually change the images all the time by adding certain things. So that it will be probably the expensive to do pairwise computation again and again. So that's why we actually, we actually, you can see the need of using C++ or CUDA or in any bottleneck. So here are like basically a list of the pros and cons of writing extensions. One is definitely speed. Another is you can use optimized C and C++ library functions, meta library functions like for C and C++, most of the blast extensions are written in C or C++ you can actually use them. So that's one of the good things about writing extensions. If you want to write them, you will have no GIL in Python most of the speed issues come because of the GIL block, you can actually parallelize your code pretty easily if you do not have GIL and you can have more memory. Why not to do extensions is pretty similar why you don't write C and C++. It's development so it's hard to write and there are memory set situations. Unless you want a 10x program. So here I just go over one of the simple, how do you write a simple CPU extension? So this is basically first you write a native code. This is can be thought of as a kernel code that will be written in C++ or CPP. And essentially what I have done here is written in a addition of two vectors, A and B and put the results in C. So this is just a for loop that looks over this. And you can actually do this in C Python as well. But it would be very, very too slow because loops in Python are real slow. And if we can vectorize somewhere we should vectorize. So then how do you do this? So I talk about this a little bit later. These technologies that are available for wrapping our code are C code and being able to we call it using Python code. So here I have shown the example using Cython. There are other in my GitHub repository, there are other examples of using Cython 5x11 and Ctypes. But here I've just shown you two space limitations. Just one example. So how do you write that? So what you do is you, in Cython what you do is you import C import numpy as NP. And this actually gives you numpy data. Like you can actually import direct numpy data types and call them in the Python function and then pass them on to your C functions. And you can actually then define CPU depth, adder CPU. And it actually takes in the argument of double, so essentially it's if you want to think of it, it's just a pointer to a memory buffer of numpy. And that is actually passed to adder CPU, which actually stores it in these data types of NP and C and then passes it to the native code. Then how do you set it up? So you have to, for distributing code it's pretty simple. You, by using the setup of the desktop, you can just add one line of extensions and add this sources of add and disco CPU and it will build like seamlessly as you just tried Python set of desktop. So after this I'll just go over what is a GPU and how GPU, how do you write codes on the desktop? So basically I like to think GPU as CPU with thousands of cores. If your CPU has four cores or 16 cores, the GPU would have like 1000 more cores than that. And that's why you can parallelize your any computation easily there. It is also called SINB, single instruction multiple data. So the same instruction set is run upon multiple data. That can be done using this. Yeah, but there is obviously no free lunch. As you can see like, see, GPUs are not great at control of it because they're controlled like the controller. If there is many if and else branches in your code, then GPUs are really bad at that because there is something called thread divergence, which happens. And GPUs would be very slow in that. But GPUs are very good at like single instruction multiple data set, because they have multiple units. So they can perform automatic really fast. Another thing that they're really shorter is the DRAM. So or the memory that a GPU has. So GPU for mostly has like 16 GB to 32 GB of RAM in the expensive case. But like, so you are mostly constrained by the amount of data you can put on your GPUs rather than being constrained upon how much computation you can do. So I go over some terminology before just how the code is written. So I go over some of the terms. So basically for terminology, essence, so if you read any GPU blogs on the video.com, they typically call it an there are some so basically GPUs are made up of essence, essence are streaming multi process and GPUs made up of two or three, generally two or four, essence. Then essence are then divided into grace. SM would have typically have four to eight grids. And this is confer configurable obviously, but there is a limit to how many grids you can have. So basically think of this a grid as a grid is a computational unit, you can think of it as a mini processor in some sense, a mini code. So you can have a two 16 grid something the 16 cores on one SN. And then every every grid has blocks within itself. So blocks can be 2D blocks or one block and blocks are blocks are unit of computation again. So yeah, like here, like I have shown like a grid, it contains many blocks. So these are truly blocks and they're indexed according to accordingly like zero, zero, one, zero, two, zero, and then zero, one, one, one, one block is then made up of something called threads threads. If are the smallest unit of multi like processing in GPUs, there are some conceptually, you can say that, but there is something called wraps. So wraps are actually memory aligned threads. These are 32 bit memory 32 threads which are aligned. And these are mostly required if you're doing real hardcore performance optimizations. Otherwise, generally wraps are not that important. So but threads are multi like so threads are basically your CPU friends as a thread computes a single unit, like it's a single program that it computes on a single data point. So if you can look here, so again, threads can be 2D or 1D. So CPU grid would is made up of blocks and blocks are then made up of threads. So the smallest unit is a thread which is now indexed by zero, zero. This would be in block zero, zero and in the grid, similarly, so that's how you index the place. So you need to get the thread index. So let's try to see this comment by some by a person called Mark human explains it in a comment. So GPUs are made up of grids, which are in terms of blocks and blocks are made up of this. So threads, you can then say that threads are somewhat equivalent to cores in CPU. So if you have six cores in CPU, but if you have 20,000 threads in your GPU, you would have essentially 20,000 cores in your GPU. Yeah, and the synchronization is done in a block level in in GPUs. So yeah, corporate cooperation is done at block levels. So what do I mean by corporation? It essentially means that thread zero can share your data with three, and this that's how cooperation is done in terms of because they have a shared memory, which they can write to and read from and perform combinations and begin right back. So that's how synchronization. They share the same share. And share memories are very important concept in GPUs. When you're writing performance optimized code like make for matrix transport, transpose, you will definitely be using share. So let's look at. Yeah, so we look at some code later, but so suppose that you do not want to write this code directly in the language, you would prefer to write it in Python. So there are some alternatives for that. One of the alternatives is number. So number is open source and pretty active development is there. It has both CPU and GPU support and it and it's pretty fast. So your code can actually run. You can vectorize it on CPUs or you can write it for which is really great. And it has access to libraries like QFFT, Q-Ryne, which are very important once you start doing actual. So PyCuda has full code API available and it has saved code writing because objects have limited lifetime attached to them. It is a concept called resource allocation. I don't remember the code form, but it's a concept called Rye and it's very prevalent with CNC++ users. So when an object goes out of, it's no longer needed, it's actually deleted. So that's what Rye is, once actually. Then you have Q by Q by is numpy for GPUs. So it has tried to actually replicate every numpy function on GPUs and it is supported by NPUs. So one of the issues with using all these libraries, I tried using all the number and Q by NPUs. One of them is it's harder to debug them. And if you actually want to write something for very, which are not, which cannot be written in NUMBA or Q by, it is actually some Q by you would have to actually write CUDA kernels and then actually import them to CUDA as strings. And then it's actually a key to writing custom kernels in CUDA. For NUMBA, one of the things is that if you want to write performance optimization code, then you cannot rely on NUMBA. You have to actually understand and get to the architecture of CUDA and then write CUDA code. There is no other way as I found out. So let's go over one of the similar examples of adding to... So let's go over a similar example of CUDA code. So here again, I'm trying to add two vectors and slowing the result in N. What I need here is the thread index. So as I said, thread is the basic unit of computation in GPUs. So every thread will now... So suppose that this A is a vector of size 256, then I can initialize my... So I can initialize 0, so blocks would be 0, and threads would be 256. So every thread is now responsible for one unit of data point. So like A0 is responsible for... So thread 0 is responsible for A0, thread 2 is responsible for A2 and similarly. So once you get the thread index, then you can use that index to actually get the data point, the index from A, and then you can add it. So then it is actually a computation which is done in parallel. So this is a pretty simple code. It gets actually... So this would be your main kernel function that would actually be run on the device code. That's why it's labeled as global. And yeah, I think that's about it. But how do you actually run this? So there is something called a driver function that actually calls the above kernel function. So firstly, what you would need to do is you need to allocate all these on your device memory. Device memory is your CPU memory. So you actually assign just a CUDA maloc code to... Oh, sorry. So CUDA maloc actually assigns... Can assign to both your host memory or your device memory as well. So here firstly, we are trying to assign memory on the device using CUDA maloc for dead-B and everything like dead-B and dead-C, which would actually be storing the computations on GPU. And then we actually copy the bytes from the host memory which was double A to a device memory which is dev underscore A. And that's why CUDA mem copy was to device. And then we actually initialize the number of threads to be 256 because we want it to be... Yeah, because we want to have 256 threads. Your A can be much longer than 256 as well. And then this formula comes into picture, which is n minus one divided by number of threads plus one. What this does essentially is it actually divides your data so that every thread... So it's done like your data is handled fully because it could be 512. Then you would need to have actually two blocks rather than just one. So that's what this code does. And then you actually do a kernel invocation which is done using this code. Basically, you say that this is the number of threads I require on my GPU. So the GPU assigns them to you and then you pass on your, basically, the pointers of the data where you have stored them and the computations. And then you actually then copy back the answer of dev C to C here. And this is here. And this is our device to host. And then you can use CUDA memory. So the wrapper is very similar to a wrapper in the CPU case, which is basically... I think it's very similar because we are dealing with pointers. So we just pass the pointers from the NumPy memory buffer here. And then we actually pass those pointers to this invocating code and get their pass-by reference here. The important part here that I will not be able to fully write down here is the setup dot pi by pi. So I struggled a lot with the setup dot pi by pi for GPUs because you need to actually add NVCC as a compiler to the setup dot pi by underscore compile function, which actually then understands, okay, that this is a CUDA code, CU code, then this should be compiled using NVCC. And if you just do this, it won't work. You would have to have some extra hacks, which are there in my data code. So you can have a full look there. But what this is saying is the sources like are my CUDA code and my wrapper that are in the CPU code as well. I need to add some certain libraries like CUDA lib64 and CUDA hard. And similarly, the runtime arguments. So essentially, if you have ever used the make command or ever compile C, C, C or C++, you actually need to pass out using minus the linker flags and the link libraries using minus L and minus I flag including. So this is similar to that. That's what we are doing here. You need to pass some other things to NVCC, certain flags like what architecture you want to compile it for, some compiler option like optimizations that you want. So in conclusion, what I had learned from this is if you need to do rapid prototyping or something and you want to write some custom CUDA commands, you should use NumPy and Cupid. If not, then it's better to write CUDA code on your own and do debugging upon that because when you're writing CUDA code, you would have to debug a lot to get better performance because one of the important things in getting better performance is finding out what number of blocks, what number of threads you need to use and then here as well because if it's a complex code, then you need to see how you would be sharing memory, how you would be writing the parallel code for parallel implementation. For more, you can look at the NVIDIA blog that there is where they go over transposing matrix in many forms and how it is done. So yeah, and for binding my conclusion were generally pretty simple. I preferred Scython in most of my work when I was deploying to production. Here the performance I found out to be much better but then the steep learning of because Scython is a purely new language. It has different syntax for functions and different syntax for types. If you are familiar with the C, you would be very like it's not that hard to get because you have C there, you have Cp there and then you have double basically it's an array notation in Scython. Pi bind is easiest to set up and it's the only option when you're working with C++ like it has native C++ support for native data source STD containers but what I found that the performance is not good by passing memory buffers and also that it's pretty tricky to solve certain issues in pi-v. Mainly I'm talking about like if you go to their docs, you can actually find out known issues that they have by dealing with data types from STD buffers and like this. Then C types are great for simple cases like they are pure Python. I did not explore this much but like I've seen many libraries using C types because they are just pointers that are passed. Yeah, so my conclusion for CUDA is that it has a steep learning curve but there are great performance returns in production. If you have some really critical performance intensive code that you have to write then you should basically learn CUDA and try to do it because GPUs are a very expensive resource at any company when you're trying to go to production and proper utilization of CUDA, I don't think most of the data scientists know and so I think that's something we need to learn to do. I'm still learning there's a lot to learn in CUDA. Yeah, for more example like my K-Name implementation and my pair-wise implementation you can check my GitHub which I'll throw in the link in the references. So yeah, that's about it. Is there any questions? Thank you.
So my talk is about writing extensions and bindings for GPU in Python. And essentially, what let me go over the goals of this talk or implement pair wise, pair wise distance computation and CUDA with significantly less memory and also wanted to be really fast. So speed, I had to write code and there was no reason for that is the bottleneck when you're trying to develop CUDA applications is the data terms of even optimized libraries like sci-fi and them in the Python function and then pass them on stores it in these data types of NP and C and just go over what is a GPU and how GPU, how do write it in Python. So there are some So let's go over a similar example of CUDA code. our device to host. And then you can use CUDA the NumPy memory buffer here. And then we