Beyond CUDA: GPU Accelerated Python on Cross-Vendor Graphics Cards with Vulkan Kompute
Saucedo, Alejandro
This talk will provide practical insights on high performance GPU computing in Python using the Vulkan Kompute framework. We will cover the trends in GPU processing, the architecture of Vulkan Kompute, we will implement a simple parallel multiplication example, and we will then dive into a machine learning example building a logistic regression model from scratch which will run in the GPU.
 Welcome to my talk on beyond CUDA GPU accelerated Python on cross vendor graphics cards with Vulkan and compute. My name is Alejandro Saucedo. I am engineering director at selling technologies, chief scientist at the Institute for ethical AI, and member at large at the ACM. So we're going to be delving into a very interesting set of topics, primarily around the parallel and GPU computing ecosystem. We're going to talk about what is the Vulkan SDK, and how you can use the compute framework to build Python GPU accelerated applications together with a set of hands on examples, as well as some references, that you can actually delve into, if you're So, to start with, we want to cover the motivations of why parallel processing. And I'm going to be referencing a research survey. I recommend checking out as well in itself, as it basically collects insights from 227 papers in the parallel deep learning space, which provides a much more intuitive perspective of the adoption, and this trend of adoption on GPU for not just scientific computing, but also general purpose computing, and some key observations that are really interesting is to see how the emphasis around a lot of the functions and paradigms in this computational areas that can be abstracted into highly highly parallelizable steps, and others that may not in themselves be highly parallelizable, can actually be reduced into equivalent structures that that then can be processed with specialized hardware, like GPUs or even you know more than you might use. There's also concepts that are continuously evolving. You know, a simple one that may you may have come across is the concept of micro batching which is basically being able to not just process a single data point, but also being able to process several more at the same time to ensure that the computation is done at a single, or, you know, in a way intuitively on a single sort of parallel clock speed or clock iteration within that that perspective instead of just submitting each one separately, and very interesting and innovative ways of breaking down computation that can actually be processed in parallel to then be re aggregated. And the interesting thing about this paper is not only how the trend is moving towards this parallel processing, and you know GPUs and GPU type hardware, but also that it's expanding into distributed processing leveraging this parallel capable type of hardware. So really interesting area. Now, delving into the into the ecosystem, and the motivations of why, you know, we're even having a new sort of framework the Vulcan SDK which we're going to introduce in a bit is particularly given that there is this this year range and heterogeneity when it comes to the GPU and parallel capable processing hardware, like to be used etc. So this involves multiple different players multiple different architectures, different drivers, and consequently different frameworks available to take advantages of this parallel capabilities, as well as for example, you know, the increase in very powerful mobile capable components that now we're carrying in in our pockets as smartphones. So there is a sheer amount of need in this space due to the heterogeneity of the different hardware across different vendors. And the same perspective is one of the key motivations of why Vulcan came to be Vulcan is a cross industry initiative that brings in several of the leading industry players to create this open source and I think that's the one of the more exciting parts the open source and open sort of standard that focuses on not just interoperability but also performance, and this is reflected in the interface that is exposed by the Vulcan SDK. In regards to the Vulcan C++ SDK, there are several advantages and disadvantages as you would have with anything. Some of the very strong advantages is that it has a very low level interface with rich access to components that comes with this explicit and verbose API within its core that provides you a, you know, no language, sort of sophisticated verbose to your abstractions, pure direct access into not just the Vulcan SDK but also the hardware underneath right and this is very important for optimizations that are very important. There are a broad range of industry leading players that are contributing to the standards and to the SDKs and to the tooling, which has been very, very encouraging to see. And also, there is an emphasis towards this interoperability and high compatibility across different platforms, mobile suppliers, AMD and VDI Qualcomm etc. And there are very strong strengths right strong advantages. You know disadvantages you know, we also have that it's very low level with rich access to components which means that there is a lot of complexity, and you know a very rich interface that needs to be interacted with. And similarly with the C style API even though it provides you a what what you see is what you get with the hardware level. That means that there needs to be a lot of domain specific knowledge to be able to build the foundational layers required to start building the application components. Right with that. There's also the broad range of top players which even though it's a great thing, always with many opinions, many voices, there will be a lot of sort of interactions that will be pulling towards different directions even though they would have this sort of like best of a high compatibility in mind and trying to push for, you know what is what is best and again you know with a high compatibility across multiple different platforms that means that it needs to deal with a very rich interface that are not just rich interface but but rich set of flexible, you know, back ends that can actually interact with with this sheer number of different hardware that is underneath. And that's what the texture of the Vulcan SDK looks like so the Vulcan application is the overarching components that we will be seeing from applications you can actually spin up instances, these instances are the ones that then allow you to, you know, talk with the physical devices that C++ component that, you know, actually refers to the to the to the physical graphics card that you have in your computer. And then you can create what is referred to us logical devices or windows or views that then allow you to interact with the physical device right. So that logical device you know you can have multiple logical devices for one single physical device and we can have multiple physical devices for an instance, and multiple instances for an application. Right so this is where it starts getting a bit complex but to keep it simple, you know with this logical device the way that you interact with it with it with the graphics card is through a queue, and this queue would have multiple commands right and you know you can have multiple instructions that need to be executed. Right and ultimately this is how you would be able to submit instructions to the GPU. Once you actually want to run more complex components that's where the pipeline comes in. You know there's the compute pipeline the graphics pipeline but we're going to be just talking about the compute pipeline. This is all just to provide an intuition, you know we actually have another talk that you'll be able to reference and check out, as well as a documentation that explains all of this in detail this is just to give you an intuition once we delve into the depths of everything. With the pipeline. This is what you're able to say well I want to actually run a specific set of instructions as per a piece of code or an algorithm that is often referred to as a shader, a shader module. So this shader is basically going to look like a piece of C code that just so happens to run on the GPU itself. And this shader this piece of shader in the pipeline will also interact with data, right, and this data is referenced through the concept of the script or sets which again you know, you're not going to really have to like know all of these different things but it's still important to get an intuition that this is what's happening under the hood, right so the script or such as like a container that basically says well I'm going to be using this GPU visible GPU visible data, so that when you run a set of instructions it can actually reference this this different things. And you see that because we're interacting with this queue, the CPU in itself is able to send requests so you send send instructions to the GPU with its own specific memory address space, and even though there are shared memory spaces which means that you know the CPU may have access to see some of the data in the GPU memory. There is different address spaces for different computational areas in the in the actual hardware of the GPU which means that the actual code your C++ your Python will not be able to see that memory space. Right, and that means that you would be able to interact with your GPU with this SDK akin to what would be you interacting with a remote service, sending requests for the service to execute as if it was an API. And you know, even though this is not correct because this is your own machine. This is actually a good intuitive way to see how you're interacting with that GPU given that it is through this queue with, you know, asynchronous buffer that are executed and of course you can then have all the things that allow you to await until the execution has been finished, which we may actually talk about in later on. And, you know, being able to build the foundation code required to actually run a simple program in Vulkan, you know only takes, you know, from 500 to 2000 lines of C++ code, right. And this is the motivations well some of the motivations for compute itself the compute framework. So compute enables developers to get started interacting with the Vulkan SDK with dozens instead of thousands of lines of code. And the key thing to emphasize here is that the core principle is to augment the Vulkan interface instead of abstracting or hiding it. So bring your own Vulkan interface, which plays nicely with already existing Vulkan applications so you already have a Vulkan application to render graphics related stuff you'd be able to actually pass those Vulkan components, and it has non Vulkan naming convention and we're going to talk about that. It's just basically to avoid ambiguity as there are libraries that you know may have classes called buffer, and it's like is this buffer from Vulkan or for this other application right so things like that. So now, in regards to other features is that it has, you know, a C++ interface, but also the Python interface that we're going to be using today. It has explicit CPU and GPU memory ownership and this is important if you're using non compute Vulkan components right if you're already using Vulkan in another area of code role Vulkan. So we're planning access to GPU queues which is very important for optimizations, you know which we have some material that actually explores that in more detail. There is, you know, single header file for the C++ development and a Pi Pi module for easy installation with the Python, and it has integration with mobile apps through the Android and DK, as well as game engines, such as Godot, which is also part of the C++ and conference, and there is going to be a few links about that. So, how does the compute architecture look like. And this is basically relatively or conceptually simple. So everything starts with a compute manager, and the compute manager is the component that basically overseas and and manages all the explicitly and memory that is then created through your interaction with this, like compute application. So the manager in itself would handle this sort of like device and on the queue, which we talked about, you know, you don't need to go into that much depth, but it's still important. You can have sequences and sequences are basically basically single or batches of operations to run on the GPU. Right that's basically what a sequences, and you can have multiple sequences with multiple operations. Each sequence can have one or many operations, and each operation basically performs a specific action, and operation can have one or multiple tensors and tensors are basically abstractions into GPU and CPU memory, as well as the workflows related to move the memory around it. And optionally, you can also have what in the compute world is referred to as an algorithm, and an algorithm abstracts the concept of the pipeline, the Vulcan pipeline, the, the the nodes, and the specific shader code that then you can actually say well I want to run this, this code in the GPU with this specific data structures, and I want to run it in this type of way. Right so we will see what that actually looks like in more practical terms. But this is basically it right there's there's, there's not much more around this, and there is an almost one to one mapping between the compute components and the Vulcan and the explicit to reduce ambiguity. Now let's see what that looks like. So, in Python, you would basically create a simple manager, right, you would then create a set of tensors with, you know you can pass non pi arrays or Python lists. It's basically just to tensors that are going to be used in a multiplication and then the output where we're going to save all of the results. Right so that's what we're going to basically use you normally would actually initialize the tensors explicitly but there's a set of helper functions and you know take your CPU host memory list, and then copy it into GPU only memory. So all of this is is is handled to you. But again, it's not handled through magic. Every single thing can be actually accessed, and you can actually call those things yourself if you wish to which is very important for for for several optimizations. So we're going to define the shader code. This is basically the code that you're going to be running in the GPU. In this case is just basically a simple multiplication using the pie shader decorator. In this case we have the first buffer the second buffer, we're going to run a multiplication and store in the output. That's basically it. This is going to run in the GPU. We're going to actually run it through the manager so we're going to say we want to run this synchronously using this three tensors, and using this shader right and we can just basically pass that it runs synchronously we can also run it a synchronously and we can actually do a lot of optimizations which we're not going to delve into this but you can run it synchronously. Once it's finished running, then you can actually copy the data back, right make sure that the output is now visible in the CPU, and then you can print it right so you can actually see that our output is 246. Right so that's basically it. That's all there is to it, and this is all you need to basically do all of the crazy crazy stuff that happens underneath. But again, all of that crazy crazy stuff is really really interesting. And it is also very useful very rich and very relevant for once you're starting to do optimizations. So, ultimately not to hide but to augment. This is the key thing. So now let's actually delve into some of those optimizations that I've been mentioning. So what we're going to do right now is run a single command slash operation in through the manager. Right so we, we had the CPU running, we submitted that operation we waited then we came back, we then submitted the second one and then we came back. Right, what we can do as well is we can actually reuse multiple sequences, which means that we can pre record commands. So we can record a bunch of operations, and then run them so the CPU would basically then execute. So you can actually run specific, you know, operations that would run already, you know, in the GPU, and then you would actually wait until it's come back. You can actually run a synchronous. This patches or submissions of the of the of the of the commands, which means that you can actually not wait for the GPU to finish. So you can actually submit this, you know the CPU would continue doing other things while the GPU is doing other things, and then you can submit the other things as well as asynchronously and then do something else. There's also an await function that allows you to wait for the thing to to finish. And then finally, something that we're not covering in this presentation but it is also very interesting, is that you can also leverage GPU hardware concurrent to submit multiple batch operations that would then execute in different GPU queues that then would potentially run in parallel. And this is dependent on the hardware properties of your GPU and the families, you know normally for example in my NVIDIA 1650. So I have the ability to run hardware concurrent batches of GPU loads. If I submit them to one compute family queue and one graphics family queue we're not going to delve into that but if you're interested, you know there's there's a lot of like really relevant content in the documentation, as well as in the data stack. But today we're going to be actually covering something that delves into the world of machine learning. And what better thing to cover than the hell wall of machine learning, logistic regression, basically taking in a specific data point and classifying it as either, you know correct or false in this case is just a binary classification, and we're going to be letting the machine do the learning in the GPU. So what are we going to be doing in terms of intuition, we're going to have input data that is going to look like two numbers that are going to go through our model, our machine learning model, and are going to perform a prediction, which ultimately should be what we are expecting. And what it looks like, at least the training data that we're going to be using the training data that we're going to be using is going to be basically, you know when we see zero and zero we expect zero, who is zero and one we expect one we see one one we expect one, and we have like you know, a bunch of different training data that looks like this, extremely simple I know but just to make sure that you know the intuition comes through, as opposed to the machine learning I mean this is not a machine learning talk. And then you know what we're doing is we want to actually train a machine learning model basically learn the parameters that would allow us to ensure that every time that we have this inputs, we provide the respective outputs. Right. And you know we, there is sort of like a more in depth blog post that covers the underlying, you know functions, and the way that everything is broken down but we're going to skim through some of that. And if you're curious you can actually delve into that. We're still going to talk about what's actually going to be happening. We're going to be trying to find the parameters of this function. This is going to be the input, basically that sort of like specific x one and x two that you saw. In this case, because we want to actually leverage the hardware parallel capabilities, we're going to be able to submit multiple as micro batches so instead of just running one by one. We're going to be doing like five at the same time so the GPU actually runs five, and then comes back to us. So that's, that's how we're going to be able to do this. We're going to be learning this to parameters w and B. And we're going to be basically, you know this is the function that calculates that prediction. Right, we're not going to be delving into too many too much of the depths but there's a blog post that covers this in a bit more detail. And there's thousands of talks that talk about logistic regression in Python so more than welcome to check that key thing here is that this will be the shader code that we're going to be writing, right and even though we're not going to cover in much detail we're still going to be looking at what that what actually required to actually write that. And then the compute side so this is the shader but the compute side is what is going to be running this, we're going to have to create a bunch of tensors that represent our input data, our parameters, our predictions, etc etc the training data, we're going to have to initialize those tensors using, you know, in the sequence, we're going to initialize the algorithm, right, and then we're going to actually, so we're going to record it. And then we're going to actually like initialize them, and then we're going to iterate and learn, let the machine do the learning right so we're actually going to be running multiple different sort of like iterations to that data set, updating the parameters every single time, running micro batches that are running in the GPU in parallel. Yes, we're going to be basically doing that. And then, once we actually iterate those 100 times, we're going to be having learned those parameters. And basically what we're going to be doing in this high level logic. And you know, you know this is just an intuition right the key thing here is to see what's happening in the, in the compute and in the in the in the shader side. So the shader what it looks like is just a much more complex version of what we saw earlier. Right so we have all of the inputs so Xi, which is x one x two is each of them they're going to be an array because it's micro batches. So we have the expected outputs, we have the weights that are coming in and we have the weights that are output and calculated remember that the parameters that we're learning are w and B. So those are things that we actually want to take out. And you know we also taking the loss to be able to reuse where we're relevant, and the number of parameters to use right so in this case what we're going to be doing is we're going to be, you know, taking the specific input weights that are continuously in each of them. And then we're going to be able to take out the parameters that we're going to be able to use. And then we're going to be able to take out the parameters that are, these are the ones, the parameters that are we're going to update in each execution so we have to pass them every single time. We're going to be calculating the function right so as we saw, you know this is basically the function that we just had. And you know the blog post I break it down in very minute detail of how we actually go through each of of these steps and and we're going to be able to take out the parameters as the respective loss. But here the key thing is that we are able to ultimately calculate the, the, the, the parameters that that we now have. Right so now in the compute side, we're going to first create all of the specific tensor so as we saw we had some training data. This is the zero, zero equals zero. So we're going to be able to calculate the function zero should be that you know one one should be one etc etc. And then you know this is how we start with our weights so we start with just like a random initialization, which then you know we're going to be iterating towards. We're going to then similarly start with, you know, run a mission initialization for our other parameter which is going to be zero. And then the number of, you know, data points is going to be basically the actual size of this parameter right so we're going to start with with five. Right so that's what we're going to be doing we're going to be starting that in a variable called parameters so we can reference them. We're then going to just initialize it by creating our manager and initialize the tensors so what this does is it actually initializes them explicitly, whereas before we didn't do it we did, we did it implicitly with that, with a utility function so we initialize all of the parameters in the GPU so they're accessible in GPU memory. Then we're actually creating and recording this the operations in the sequence. So what are the operations, we're going to first actually sync the data to device for these two tensors, because remember the parameters are going to be updated every iteration. So we're going to be in the GPU device memory, we're going to then record that algorithm that we just actually wrote this is the the logistic regression shader that you saw so we're going to record that execution, and then we're going to record a sync to local. Right so we're going to record for all of the weights, the parameters and the, and the, and the, and the loss to actually be copied back to the host so that Python can see them. And then finally we're going to iterate 100 times by, you know, every iteration, running that shader or running the sequence which is basically all of the things that we just recorded, and then operating and then updating all of the, all of the weights. So I think there's an indentation missing here but basically what is happening here is we're just basically updating it by, you know, the specific learning rates which is just how fast do we want the actual parameters to be updated on each iteration. Again, the key thing here is just to see all of the features that you're able to use with compute and leveraging a simple machine learning use case as an example. And of course we're skimming through unconscious. Sorry for that for the people that you know, maybe are seeing this and going like well you know actually that's you know, 100% correct but you know in the blood post I break it down and you know much more in detail. This is just to show how you're able to actually interact with the GPU, and you know, optimize in different areas by you know pre recording components using the sequence etc etc. So in this case we're just iterating. In the finished you're able to then just print the calculated parameters which in this case, they are the first way the second way. The third way, and the actual be bias, which is ultimately what we ended up with. And so to emphasize you know we covered kind of like this high level example, but as I mentioned we have a blog post that covers this one in Python, one in C++. It breaks it down in in my new detail, and we have other tutorials other examples that cover, you know how to use this and actually the C++ as opposed to the Python one for integrating with your Android apps, as well as to game engines like the Godot engine, which we would recommend to check out. So more than anything what I recommend is to get involved. You know if you go to GitHub.com slash ethical ml slash Vulkan compute, you'd be able to actually check out what are some of the open issues. You know you can take one of the good first issues labeled good first issues. And also please you know there's open there's an issue number 52 which is open for just general discussion so you have ideas for improvements, or questions, and you can actually just post them in the chat, and we've had some really interesting suggestions. So the key things in the roadmap, you know one of the main motivations to build this framework is to actually integrated as a backend of an existing site scientific computing framework that is potentially even being used for, you know, mobile, you know machine learning or for other types of use cases so definitely really interesting on that and if someone is, you know, running a scientific computing library then you know being open to explore. Also creating more default operations, something like a fast forward transform or like a parallel some reduction, things like that would be really cool to have like out of the box operations that are, you know perhaps even written in C++ but also exposed as a new model. And then also adding examples if you try this in a new sort of shader, or a new sort of like algorithm, a new machine learning type of model, you know would love to actually contributed upstream and you know added to the repo because I think that that would be very cool. And so that's it. I think that's everything that we had sort of to cover today. Thank you very much for joining this talk on beyond kura GPU accelerated Python on cross vendor graphics card with Vulkan and compute. And I would like to take this opportunity to take this opportunity forward to explore and hear your thoughts, ideas and suggestions. And if you have any questions do please feel, feel free to reach out. Thank you very much.
 Welcome to my talk on beyond CUDA GPU accelerated Python on cross vendor graphics cards with  Vulkan and compute.  My name is Alejandro Saucedo.  I am engineering director at selling technologies, chief scientist at the Institute for ethical AI, and member at large at the ACM.  So we're going to be delving into a very interesting set of topics, primarily around the parallel and GPU computing ecosystem.  We're going to talk about what is the Vulkan SDK, and how you can use the compute framework to build Python GPU accelerated applications together with a set of hands on examples, as well as some references, that you can actually delve into, if you're  So, to start with, we want to cover the motivations of why parallel processing. And I'm going to be referencing a research survey.  I recommend checking out as well in itself, as it basically collects insights from 227 papers in the parallel deep learning space, which provides a much more intuitive perspective of the adoption, and this trend of adoption on GPU for not just  scientific computing, but also general purpose computing, and some key observations that are really interesting is to see how the emphasis around a lot of the functions and paradigms in this computational areas that can be abstracted into  highly highly parallelizable steps, and others that may not in themselves be highly parallelizable, can actually be reduced into equivalent structures that that then can be processed with specialized hardware, like GPUs or even you know more  than you might use. There's also concepts that are continuously evolving. You know, a simple one that may you may have come across is the concept of micro batching which is basically being able to not just process a single data point, but also being able to  process several more at the same time to ensure that the computation is done at a single, or, you know, in a way intuitively on a single sort of parallel clock speed or clock iteration within that that perspective instead of just submitting each one separately,  and very interesting and innovative ways of breaking down computation that can actually be processed in parallel to then be re aggregated. And the interesting thing about this paper is not only how the trend is moving towards this parallel processing, and you know GPUs and  GPU type hardware, but also that it's expanding into distributed processing leveraging this parallel capable type of hardware. So really interesting area.  Now, delving into the into the ecosystem, and the motivations of why, you know, we're even having a new sort of framework the Vulcan SDK which we're going to introduce in a bit is particularly given that there is this  this year range and heterogeneity when it comes to the GPU and parallel capable processing hardware, like to be used etc.  So this involves multiple different players multiple different architectures, different drivers, and consequently different frameworks available to take advantages of this parallel capabilities, as well as for example, you know, the increase in very powerful mobile capable  components that now we're carrying in in our pockets as smartphones. So there is a sheer amount of need in this space due to the heterogeneity of the different hardware across different vendors.  And the same perspective is one of the key motivations of why Vulcan came to be Vulcan is a cross industry initiative that brings in several of the leading industry players to create this open source and I think that's the one of the more  exciting parts the open source and open sort of standard that focuses on not just interoperability but also performance, and this is reflected in the interface that is exposed by the Vulcan SDK.  In regards to the Vulcan C++ SDK, there are several advantages and disadvantages as you would have with anything. Some of the very strong advantages is that it has a very low level interface with rich access to components that comes with this explicit and verbose  API within its core that provides you a, you know, no language, sort of sophisticated verbose to your abstractions, pure direct access into not just the Vulcan SDK but also the hardware underneath right and this is very important for optimizations that are  very important. There are a broad range of industry leading players that are contributing to the standards and to the SDKs and to the tooling, which has been very, very encouraging to see.  And also, there is an emphasis towards this interoperability and high compatibility across different platforms, mobile suppliers, AMD and VDI Qualcomm etc.  And there are very strong strengths right strong advantages. You know disadvantages you know, we also have that it's very low level with rich access to components which means that there is a lot of complexity, and you know a very rich interface that needs to be interacted  with. And similarly with the C style API even though it provides you a what what you see is what you get with the hardware level. That means that there needs to be a lot of domain specific knowledge to be able to build the foundational layers required to start building the application  components. Right with that.  There's also the broad range of top players which even though it's a great thing, always with many opinions, many voices, there will be a lot of sort of interactions that will be pulling towards different directions even though they would have this sort of like best  of a high compatibility in mind and trying to push for, you know what is what is best and again you know with a high compatibility across multiple different platforms that means that it needs to deal with a very rich interface that are not just rich  interface but but rich set of flexible, you know, back ends that can actually interact with with this sheer number of different hardware that is underneath.  And that's what the texture of the Vulcan SDK looks like so the Vulcan application is the overarching components that we will be seeing from applications you can actually spin up instances, these instances are the ones that then allow you to, you know, talk with  the physical devices that C++ component that, you know, actually refers to the to the to the physical graphics card that you have in your computer. And then you can create what is referred to us logical devices or windows or views that then allow you to interact with  the physical device right. So that logical device you know you can have multiple logical devices for one single physical device and we can have multiple physical devices for an instance, and multiple instances for an application.  Right so this is where it starts getting a bit complex but to keep it simple, you know with this logical device the way that you interact with it with it with the graphics card is through a queue, and this queue would have multiple commands right  and you know you can have multiple instructions that need to be executed. Right and ultimately this is how you would be able to submit instructions to the GPU.  Once you actually want to run more complex components that's where the pipeline comes in. You know there's the compute pipeline the graphics pipeline but we're going to be just talking about the compute pipeline.  This is all just to provide an intuition, you know we actually have another talk that you'll be able to reference and check out, as well as a documentation that explains all of this in detail this is just to give you an intuition once we delve into  the depths of everything.  With the pipeline. This is what you're able to say well I want to actually run a specific set of instructions as per a piece of code or an algorithm that is often referred to as a shader, a shader module.  So this shader is basically going to look like a piece of C code that just so happens to run on the GPU itself. And this shader this piece of shader in the pipeline will also interact with data, right, and this data is referenced through the concept of the script or  sets which again you know, you're not going to really have to like know all of these different things but it's still important to get an intuition that this is what's happening under the hood, right so the script or such as like a  container that basically says well I'm going to be using this GPU visible GPU visible data, so that when you run a set of instructions it can actually reference this this different things.  And you see that because we're interacting with this queue, the CPU in itself is able to send requests so you send send instructions to the GPU with its own specific memory address space, and even though there are shared  memory spaces which means that you know the CPU may have access to see some of the data in the GPU memory. There is different address spaces for different computational  areas in the in the actual hardware of the GPU which means that the actual code your C++ your Python will not be able to see that memory space.  Right, and that means that you would be able to interact with your GPU with this SDK akin to what would be you interacting with a remote service, sending requests for the service to execute as if it was an API.  And you know, even though this is not correct because this is your own machine. This is actually a good intuitive way to see how you're interacting with that GPU given that it is through this queue with, you know, asynchronous  buffer that are executed and of course you can then have all the things that allow you to await until the execution has been finished, which we may actually talk about in later on.  And, you know, being able to build the foundation code required to actually run a simple program in Vulkan, you know only takes, you know, from 500 to 2000 lines of C++ code, right.  And this is the motivations well some of the motivations for compute itself the compute framework. So compute enables developers to get started interacting with the Vulkan SDK with dozens instead of thousands of lines of code.  And the key thing to emphasize here is that the core principle is to augment the Vulkan interface instead of abstracting or hiding it.  So bring your own Vulkan interface, which plays nicely with already existing Vulkan applications so you already have a Vulkan application to render graphics related stuff you'd be able to actually pass those Vulkan components, and it has non Vulkan naming  convention and we're going to talk about that. It's just basically to avoid ambiguity as there are libraries that you know may have classes called buffer, and it's like is this buffer from Vulkan or for this other application right so things like that.  So now, in regards to other features is that it has, you know, a C++ interface, but also the Python interface that we're going to be using today.  It has explicit CPU and GPU memory ownership and this is important if you're using non compute Vulkan components right if you're already using Vulkan in another area of code role Vulkan.  So we're planning access to GPU queues which is very important for optimizations, you know which we have some material that actually explores that in more detail.  There is, you know, single header file for the C++ development and a Pi Pi module for easy installation with the Python, and it has integration with mobile apps through the Android and DK, as well as game engines, such as Godot, which is also part of the  C++ and conference, and there is going to be a few links about that.  So, how does the compute architecture look like. And this is basically relatively or conceptually simple. So everything starts with a compute manager, and the compute manager is the component that basically overseas and and manages all the explicitly  and memory that is then created through your interaction with this, like compute application.  So the manager in itself would handle this sort of like device and on the queue, which we talked about, you know, you don't need to go into that much depth, but it's still important.  You can have sequences and sequences are basically basically single or batches of operations to run on the GPU. Right that's basically what a sequences, and you can have multiple sequences with multiple operations.  Each sequence can have one or many operations, and each operation basically performs a specific action, and operation can have one or multiple tensors and tensors are basically abstractions into GPU and CPU memory, as well as the  workflows related to move the memory around it. And optionally, you can also have what in the compute world is referred to as an algorithm, and an algorithm abstracts the concept of the pipeline, the Vulcan pipeline, the, the  the nodes, and the specific shader code that then you can actually say well I want to run this, this code in the GPU with this specific data structures, and I want to run it in this type of way.  Right so we will see what that actually looks like in more practical terms. But this is basically it right there's there's, there's not much more around this, and there is an almost one to one mapping between the compute components and the Vulcan  and the explicit to reduce ambiguity.  Now let's see what that looks like. So, in Python, you would basically create a simple manager, right, you would then create a set of tensors with, you know you can pass non pi arrays or Python lists.  It's basically just to tensors that are going to be used in a multiplication and then the output where we're going to save all of the results. Right so that's what we're going to basically use you normally would actually initialize the tensors explicitly but there's a set of helper functions  and you know take your CPU host memory list, and then copy it into GPU only memory. So all of this is is is handled to you. But again, it's not handled through magic.  Every single thing can be actually accessed, and you can actually call those things yourself if you wish to which is very important for for for several optimizations.  So we're going to define the shader code. This is basically the code that you're going to be running in the GPU. In this case is just basically a simple multiplication using the pie shader decorator.  In this case we have the first buffer the second buffer, we're going to run a multiplication and store in the output. That's basically it. This is going to run in the GPU.  We're going to actually run it through the manager so we're going to say we want to run this synchronously using this three tensors, and using this shader right and we can just basically pass that it runs synchronously we can also run it a  synchronously and we can actually do a lot of optimizations which we're not going to delve into this but you can run it synchronously.  Once it's finished running, then you can actually copy the data back, right make sure that the output is now visible in the CPU, and then you can print it right so you can actually see that our output is 246.  Right so that's basically it. That's all there is to it, and this is all you need to basically do all of the crazy crazy stuff that happens underneath.  But again, all of that crazy crazy stuff is really really interesting. And it is also very useful very rich and very relevant for once you're starting to do optimizations.  So, ultimately not to hide but to augment. This is the key thing. So now let's actually delve into some of those optimizations that I've been mentioning.  So what we're going to do right now is run a single command slash operation in through the manager. Right so we, we had the CPU running, we submitted that operation we waited then we came back, we then submitted the second one and then we came back.  Right, what we can do as well is we can actually reuse multiple sequences, which means that we can pre record commands. So we can record a bunch of operations, and then run them so the CPU would basically then execute.  So you can actually run specific, you know, operations that would run already, you know, in the GPU, and then you would actually wait until it's come back. You can actually run a synchronous.  This patches or submissions of the of the of the of the commands, which means that you can actually not wait for the GPU to finish. So you can actually submit this, you know the CPU would continue doing other things while the GPU is doing other things, and then you can submit  the other things as well as asynchronously and then do something else. There's also an await function that allows you to wait for the thing to to finish. And then finally, something that we're not covering in this presentation but it is also very interesting, is that you can also leverage GPU hardware concurrent  to submit multiple batch operations that would then execute in different GPU queues that then would potentially run in parallel. And this is dependent on the hardware properties of your GPU and the families, you know normally for example in my NVIDIA 1650.  So I have the ability to run hardware concurrent batches of GPU loads. If I submit them to one compute family queue and one graphics family queue we're not going to delve into that but if you're interested, you know there's there's a lot of like really relevant content in the documentation, as well as in  the data stack. But today we're going to be actually covering something that delves into the world of machine learning. And what better thing to cover than the hell wall of machine learning, logistic regression, basically taking in a specific data point and classifying it as either, you know correct or false in this  case is just a binary classification, and we're going to be letting the machine do the learning in the GPU.  So what are we going to be doing in terms of intuition, we're going to have input data that is going to look like two numbers that are going to go through our model, our machine learning model, and are going to perform a prediction, which ultimately should be what we are expecting.  And what it looks like, at least the training data that we're going to be using the training data that we're going to be using is going to be basically, you know when we see zero and zero we expect zero, who is zero and one we expect one we see one one we expect one, and we have like you know, a bunch of  different training data that looks like this, extremely simple I know but just to make sure that you know the intuition comes through, as opposed to the machine learning I mean this is not a machine learning talk.  And then you know what we're doing is we want to actually train a machine learning model basically learn the parameters that would allow us to ensure that every time that we have this inputs, we provide the respective outputs.  Right. And you know we, there is sort of like a more in depth blog post that covers the underlying, you know functions, and the way that everything is broken down but we're going to skim through some of that.  And if you're curious you can actually delve into that. We're still going to talk about what's actually going to be happening. We're going to be trying to find the parameters of this function.  This is going to be the input, basically that sort of like specific x one and x two that you saw.  In this case, because we want to actually leverage the hardware parallel capabilities, we're going to be able to submit multiple as micro batches so instead of just running one by one.  We're going to be doing like five at the same time so the GPU actually runs five, and then comes back to us. So that's, that's how we're going to be able to do this.  We're going to be learning this to parameters w and B.  And we're going to be basically, you know this is the function that calculates that prediction. Right, we're not going to be delving into too many too much of the depths but there's a blog post that covers this in a bit more detail.  And there's thousands of talks that talk about logistic regression in Python so more than welcome to check that key thing here is that this will be the shader code that we're going to be writing, right and even though we're not going to cover in much detail we're  still going to be looking at what that what actually required to actually write that.  And then the compute side so this is the shader but the compute side is what is going to be running this, we're going to have to create a bunch of tensors that represent our input data, our parameters, our predictions, etc etc the training data,  we're going to have to initialize those tensors using, you know, in the sequence, we're going to initialize the algorithm, right, and then we're going to actually, so we're going to record it.  And then we're going to actually like initialize them, and then we're going to iterate and learn, let the machine do the learning right so we're actually going to be running multiple different sort of like iterations to that data set, updating the parameters every single  time, running micro batches that are running in the GPU in parallel. Yes, we're going to be basically doing that. And then, once we actually iterate those 100 times, we're going to be having learned those parameters.  And basically what we're going to be doing in this high level logic. And you know, you know this is just an intuition right the key thing here is to see what's happening in the, in the compute and in the in the in the shader side.  So the shader what it looks like is just a much more complex version of what we saw earlier. Right so we have all of the inputs so Xi, which is x one x two is each of them they're going to be an array because it's micro batches.  So we have the expected outputs, we have the weights that are coming in and we have the weights that are output and calculated remember that the parameters that we're learning are w and B.  So those are things that we actually want to take out. And you know we also taking the loss to be able to reuse where we're relevant, and the number of parameters to use right so in this case what we're going to be doing is we're going to be, you know, taking the specific input weights that are continuously  in each of them. And then we're going to be able to take out the parameters that we're going to be able to use.  And then we're going to be able to take out the parameters that are, these are the ones, the parameters that are we're going to update in each execution so we have to pass them every single time.  We're going to be calculating the function right so as we saw, you know this is basically the function that we just had. And you know the blog post I break it down in very minute detail of how we actually go through each of of these steps and and  we're going to be able to take out the parameters as the respective loss. But here the key thing is that we are able to ultimately calculate the, the, the, the parameters that that we now have.  Right so now in the compute side, we're going to first create all of the specific tensor so as we saw we had some training data. This is the zero, zero equals zero.  So we're going to be able to calculate the function zero should be that you know one one should be one etc etc. And then you know this is how we start with our weights so we start with just like a random initialization, which then you know we're going to be  iterating towards. We're going to then similarly start with, you know, run a mission initialization for our other parameter which is going to be zero.  And then the number of, you know, data points is going to be basically the actual size of this parameter right so we're going to start with with five.  Right so that's what we're going to be doing we're going to be starting that in a variable called parameters so we can reference them.  We're then going to just initialize it by creating our manager and initialize the tensors so what this does is it actually initializes them explicitly, whereas before we didn't do it we did, we did it implicitly with that, with a utility function so  we initialize all of the parameters in the GPU so they're accessible in GPU memory.  Then we're actually creating and recording this the operations in the sequence. So what are the operations, we're going to first actually sync the data to device for these two tensors, because remember the parameters are going to be updated every iteration.  So we're going to be in the GPU device memory, we're going to then record that algorithm that we just actually wrote this is the the logistic regression shader that you saw so we're going to record that execution, and then we're going to record a sync to local.  Right so we're going to record for all of the weights, the parameters and the, and the, and the, and the loss to actually be copied back to the host so that Python can see them.  And then finally we're going to iterate 100 times by, you know, every iteration, running that shader or running the sequence which is basically all of the things that we just recorded, and then operating and then updating all of the, all of the weights.  So I think there's an indentation missing here but basically what is happening here is we're just basically updating it by, you know, the specific learning rates which is just how fast do we want the actual parameters to be updated on each iteration.  Again, the key thing here is just to see all of the features that you're able to use with compute and leveraging a simple machine learning use case as an example.  And of course we're skimming through unconscious. Sorry for that for the people that you know, maybe are seeing this and going like well you know actually that's you know, 100% correct but you know in the blood post I break it down and you know much more  in detail. This is just to show how you're able to actually interact with the GPU, and you know, optimize in different areas by you know pre recording components using the sequence etc etc.  So in this case we're just iterating.  In the finished you're able to then just print the calculated parameters which in this case, they are the first way the second way. The third way, and the actual be bias, which is ultimately what we ended up with.  And so to emphasize you know we covered kind of like this high level example, but as I mentioned we have a blog post that covers this one in Python, one in C++. It breaks it down in in my new detail, and we have other tutorials other examples that cover, you know how to use this and actually  the C++ as opposed to the Python one for integrating with your Android apps, as well as to game engines like the Godot engine, which we would recommend to check out.  So more than anything what I recommend is to get involved. You know if you go to GitHub.com slash ethical ml slash Vulkan compute, you'd be able to actually check out what are some of the open issues.  You know you can take one of the good first issues labeled good first issues. And also please you know there's open there's an issue number 52 which is open for just general discussion so you have ideas for improvements, or questions, and you can actually just post them  in the chat, and we've had some really interesting suggestions. So the key things in the roadmap, you know one of the main motivations to build this framework is to actually integrated as a backend of an existing site scientific computing framework that is potentially  even being used for, you know, mobile, you know machine learning or for other types of use cases so definitely really interesting on that and if someone is, you know, running a scientific computing library then you know being open to explore.  Also creating more default operations, something like a fast forward transform or like a parallel some reduction, things like that would be really cool to have like out of the box operations that are, you know perhaps even written in C++ but also exposed as  a new model. And then also adding examples if you try this in a new sort of shader, or a new sort of like algorithm, a new machine learning type of model, you know would love to actually contributed upstream and you know added to the repo because I think that that would be very cool.  And so that's it. I think that's everything that we had sort of to cover today. Thank you very much for joining this talk on beyond kura GPU accelerated Python on cross vendor graphics card with Vulkan and compute.  And I would like to take this opportunity to take this opportunity forward to explore and hear your thoughts, ideas and suggestions. And if you have any questions do please feel, feel free to reach out. Thank you very much.
python_cuda_0_179.png python_cuda_180_491.png python_cuda_492_731.png python_cuda_732_1439.png python_cuda_1440_4907.png python_cuda_4908_6539.png python_cuda_6540_7487.png python_cuda_7488_11003.png python_cuda_11004_12587.png python_cuda_12588_13859.png python_cuda_13860_16259.png python_cuda_16260_16583.png python_cuda_16584_17843.png python_cuda_17844_19103.png python_cuda_19104_21959.png python_cuda_21960_23303.png python_cuda_23304_23831.png python_cuda_23832_25259.png python_cuda_25260_25703.png python_cuda_25704_26171.png python_cuda_26172_26807.png python_cuda_26808_28067.png python_cuda_28068_28631.png python_cuda_28632_29051.png python_cuda_29052_29615.png python_cuda_29616_30311.png python_cuda_30312_30431.png python_cuda_30432_31619.png python_cuda_31620_33611.png python_cuda_33612_35543.png python_cuda_35544_36623.png python_cuda_36624_37079.png python_cuda_37080_38159.png python_cuda_38160_39719.png python_cuda_39720_40211.png python_cuda_40212_40847.png python_cuda_40848_41243.png python_cuda_41244_41423.png python_cuda_41424_41603.png python_cuda_41604_43055.png python_cuda_43056_43655.png python_cuda_43656_43773.png
Welcome to my talk on beyond CUDA GPU accelerated Python on cross vendor graphics cards with Vulkan and compute. My name is Alejandro Saucedo. I am engineering director at selling technologies, chief scientist at the Institute for ethical AI, and member at large at the ACM. So we're going to be delving into a very interesting set of topics, primarily around the parallel and GPU computing ecosystem. We're going to talk about what is the Vulkan SDK, and how you can use the compute framework to build Python GPU accelerated applications together with a set of hands on examples, as well as some references, that you can actually delve into, if you're So, to start with, we want to cover the motivations of why parallel processing. And I'm going to be referencing a research survey. I recommend checking out as well in itself, as it basically collects insights from 227 papers in the parallel deep learning space, which provides a much more intuitive perspective of the adoption, and this trend of adoption on GPU for not just scientific computing, but also general purpose computing, and some key observations that are really interesting is to see how the emphasis around a lot of the functions and paradigms in this computational areas that can be abstracted into highly highly parallelizable steps, and others that may not in themselves be highly parallelizable, can actually be reduced into equivalent structures that that then can be processed with specialized hardware, like GPUs or even you know more than you might use. There's also concepts that are continuously evolving. You know, a simple one that may you may have come across is the concept of micro batching which is basically being able to not just process a single data point, but also being able to process several more at the same time to ensure that the computation is done at a single, or, you know, in a way intuitively on a single sort of parallel clock speed or clock iteration within that that perspective instead of just submitting each one separately, and very interesting and innovative ways of breaking down computation that can actually be processed in parallel to then be re aggregated. And the interesting thing about this paper is not only how the trend is moving towards this parallel processing, and you know GPUs and GPU type hardware, but also that it's expanding into distributed processing leveraging this parallel capable type of hardware. So really interesting area. Now, delving into the into the ecosystem, and the motivations of why, you know, we're even having a new sort of framework the Vulcan SDK which we're going to introduce in a bit is particularly given that there is this this year range and heterogeneity when it comes to the GPU and parallel capable processing hardware, like to be used etc. So this involves multiple different players multiple different architectures, different drivers, and consequently different frameworks available to take advantages of this parallel capabilities, as well as for example, you know, the increase in very powerful mobile capable components that now we're carrying in in our pockets as smartphones. So there is a sheer amount of need in this space due to the heterogeneity of the different hardware across different vendors. And the same perspective is one of the key motivations of why Vulcan came to be Vulcan is a cross industry initiative that brings in several of the leading industry players to create this open source and I think that's the one of the more exciting parts the open source and open sort of standard that focuses on not just interoperability but also performance, and this is reflected in the interface that is exposed by the Vulcan SDK. In regards to the Vulcan C++ SDK, there are several advantages and disadvantages as you would have with anything. Some of the very strong advantages is that it has a very low level interface with rich access to components that comes with this explicit and verbose API within its core that provides you a, you know, no language, sort of sophisticated verbose to your abstractions, pure direct access into not just the Vulcan SDK but also the hardware underneath right and this is very important for optimizations that are very important. There are a broad range of industry leading players that are contributing to the standards and to the SDKs and to the tooling, which has been very, very encouraging to see. And also, there is an emphasis towards this interoperability and high compatibility across different platforms, mobile suppliers, AMD and VDI Qualcomm etc. And there are very strong strengths right strong advantages. You know disadvantages you know, we also have that it's very low level with rich access to components which means that there is a lot of complexity, and you know a very rich interface that needs to be interacted with. And similarly with the C style API even though it provides you a what what you see is what you get with the hardware level. That means that there needs to be a lot of domain specific knowledge to be able to build the foundational layers required to start building the application components. Right with that. There's also the broad range of top players which even though it's a great thing, always with many opinions, many voices, there will be a lot of sort of interactions that will be pulling towards different directions even though they would have this sort of like best of a high compatibility in mind and trying to push for, you know what is what is best and again you know with a high compatibility across multiple different platforms that means that it needs to deal with a very rich interface that are not just rich interface but but rich set of flexible, you know, back ends that can actually interact with with this sheer number of different hardware that is underneath. And that's what the texture of the Vulcan SDK looks like so the Vulcan application is the overarching components that we will be seeing from applications you can actually spin up instances, these instances are the ones that then allow you to, you know, talk with the physical devices that C++ component that, you know, actually refers to the to the to the physical graphics card that you have in your computer. And then you can create what is referred to us logical devices or windows or views that then allow you to interact with the physical device right. So that logical device you know you can have multiple logical devices for one single physical device and we can have multiple physical devices for an instance, and multiple instances for an application. Right so this is where it starts getting a bit complex but to keep it simple, you know with this logical device the way that you interact with it with it with the graphics card is through a queue, and this queue would have multiple commands right and you know you can have multiple instructions that need to be executed. Right and ultimately this is how you would be able to submit instructions to the GPU. Once you actually want to run more complex components that's where the pipeline comes in. You know there's the compute pipeline the graphics pipeline but we're going to be just talking about the compute pipeline. This is all just to provide an intuition, you know we actually have another talk that you'll be able to reference and check out, as well as a documentation that explains all of this in detail this is just to give you an intuition once we delve into the depths of everything. With the pipeline. This is what you're able to say well I want to actually run a specific set of instructions as per a piece of code or an algorithm that is often referred to as a shader, a shader module. So this shader is basically going to look like a piece of C code that just so happens to run on the GPU itself. And this shader this piece of shader in the pipeline will also interact with data, right, and this data is referenced through the concept of the script or sets which again you know, you're not going to really have to like know all of these different things but it's still important to get an intuition that this is what's happening under the hood, right so the script or such as like a container that basically says well I'm going to be using this GPU visible GPU visible data, so that when you run a set of instructions it can actually reference this this different things. And you see that because we're interacting with this queue, the CPU in itself is able to send requests so you send send instructions to the GPU with its own specific memory address space, and even though there are shared memory spaces which means that you know the CPU may have access to see some of the data in the GPU memory. There is different address spaces for different computational areas in the in the actual hardware of the GPU which means that the actual code your C++ your Python will not be able to see that memory space. Right, and that means that you would be able to interact with your GPU with this SDK akin to what would be you interacting with a remote service, sending requests for the service to execute as if it was an API. And you know, even though this is not correct because this is your own machine. This is actually a good intuitive way to see how you're interacting with that GPU given that it is through this queue with, you know, asynchronous buffer that are executed and of course you can then have all the things that allow you to await until the execution has been finished, which we may actually talk about in later on. And, you know, being able to build the foundation code required to actually run a simple program in Vulkan, you know only takes, you know, from 500 to 2000 lines of C++ code, right. And this is the motivations well some of the motivations for compute itself the compute framework. So compute enables developers to get started interacting with the Vulkan SDK with dozens instead of thousands of lines of code. And the key thing to emphasize here is that the core principle is to augment the Vulkan interface instead of abstracting or hiding it. So bring your own Vulkan interface, which plays nicely with already existing Vulkan applications so you already have a Vulkan application to render graphics related stuff you'd be able to actually pass those Vulkan components, and it has non Vulkan naming convention and we're going to talk about that. It's just basically to avoid ambiguity as there are libraries that you know may have classes called buffer, and it's like is this buffer from Vulkan or for this other application right so things like that. So now, in regards to other features is that it has, you know, a C++ interface, but also the Python interface that we're going to be using today. It has explicit CPU and GPU memory ownership and this is important if you're using non compute Vulkan components right if you're already using Vulkan in another area of code role Vulkan. So we're planning access to GPU queues which is very important for optimizations, you know which we have some material that actually explores that in more detail. There is, you know, single header file for the C++ development and a Pi Pi module for easy installation with the Python, and it has integration with mobile apps through the Android and DK, as well as game engines, such as Godot, which is also part of the C++ and conference, and there is going to be a few links about that. So, how does the compute architecture look like. And this is basically relatively or conceptually simple. So everything starts with a compute manager, and the compute manager is the component that basically overseas and and manages all the explicitly and memory that is then created through your interaction with this, like compute application. So the manager in itself would handle this sort of like device and on the queue, which we talked about, you know, you don't need to go into that much depth, but it's still important. You can have sequences and sequences are basically basically single or batches of operations to run on the GPU. Right that's basically what a sequences, and you can have multiple sequences with multiple operations. Each sequence can have one or many operations, and each operation basically performs a specific action, and operation can have one or multiple tensors and tensors are basically abstractions into GPU and CPU memory, as well as the workflows related to move the memory around it. And optionally, you can also have what in the compute world is referred to as an algorithm, and an algorithm abstracts the concept of the pipeline, the Vulcan pipeline, the, the the nodes, and the specific shader code that then you can actually say well I want to run this, this code in the GPU with this specific data structures, and I want to run it in this type of way. Right so we will see what that actually looks like in more practical terms. But this is basically it right there's there's, there's not much more around this, and there is an almost one to one mapping between the compute components and the Vulcan and the explicit to reduce ambiguity. Now let's see what that looks like. So, in Python, you would basically create a simple manager, right, you would then create a set of tensors with, you know you can pass non pi arrays or Python lists. It's basically just to tensors that are going to be used in a multiplication and then the output where we're going to save all of the results. Right so that's what we're going to basically use you normally would actually initialize the tensors explicitly but there's a set of helper functions and you know take your CPU host memory list, and then copy it into GPU only memory. So all of this is is is handled to you. But again, it's not handled through magic. Every single thing can be actually accessed, and you can actually call those things yourself if you wish to which is very important for for for several optimizations. So we're going to define the shader code. This is basically the code that you're going to be running in the GPU. In this case is just basically a simple multiplication using the pie shader decorator. In this case we have the first buffer the second buffer, we're going to run a multiplication and store in the output. That's basically it. This is going to run in the GPU. We're going to actually run it through the manager so we're going to say we want to run this synchronously using this three tensors, and using this shader right and we can just basically pass that it runs synchronously we can also run it a synchronously and we can actually do a lot of optimizations which we're not going to delve into this but you can run it synchronously. Once it's finished running, then you can actually copy the data back, right make sure that the output is now visible in the CPU, and then you can print it right so you can actually see that our output is 246. Right so that's basically it. That's all there is to it, and this is all you need to basically do all of the crazy crazy stuff that happens underneath. But again, all of that crazy crazy stuff is really really interesting. And it is also very useful very rich and very relevant for once you're starting to do optimizations. So, ultimately not to hide but to augment. This is the key thing. So now let's actually delve into some of those optimizations that I've been mentioning. So what we're going to do right now is run a single command slash operation in through the manager. Right so we, we had the CPU running, we submitted that operation we waited then we came back, we then submitted the second one and then we came back. Right, what we can do as well is we can actually reuse multiple sequences, which means that we can pre record commands. So we can record a bunch of operations, and then run them so the CPU would basically then execute. So you can actually run specific, you know, operations that would run already, you know, in the GPU, and then you would actually wait until it's come back. You can actually run a synchronous. This patches or submissions of the of the of the of the commands, which means that you can actually not wait for the GPU to finish. So you can actually submit this, you know the CPU would continue doing other things while the GPU is doing other things, and then you can submit the other things as well as asynchronously and then do something else. There's also an await function that allows you to wait for the thing to to finish. And then finally, something that we're not covering in this presentation but it is also very interesting, is that you can also leverage GPU hardware concurrent to submit multiple batch operations that would then execute in different GPU queues that then would potentially run in parallel. And this is dependent on the hardware properties of your GPU and the families, you know normally for example in my NVIDIA 1650. So I have the ability to run hardware concurrent batches of GPU loads. If I submit them to one compute family queue and one graphics family queue we're not going to delve into that but if you're interested, you know there's there's a lot of like really relevant content in the documentation, as well as in the data stack. But today we're going to be actually covering something that delves into the world of machine learning. And what better thing to cover than the hell wall of machine learning, logistic regression, basically taking in a specific data point and classifying it as either, you know correct or false in this case is just a binary classification, and we're going to be letting the machine do the learning in the GPU. So what are we going to be doing in terms of intuition, we're going to have input data that is going to look like two numbers that are going to go through our model, our machine learning model, and are going to perform a prediction, which ultimately should be what we are expecting. And what it looks like, at least the training data that we're going to be using the training data that we're going to be using is going to be basically, you know when we see zero and zero we expect zero, who is zero and one we expect one we see one one we expect one, and we have like you know, a bunch of different training data that looks like this, extremely simple I know but just to make sure that you know the intuition comes through, as opposed to the machine learning I mean this is not a machine learning talk. And then you know what we're doing is we want to actually train a machine learning model basically learn the parameters that would allow us to ensure that every time that we have this inputs, we provide the respective outputs. Right. And you know we, there is sort of like a more in depth blog post that covers the underlying, you know functions, and the way that everything is broken down but we're going to skim through some of that. And if you're curious you can actually delve into that. We're still going to talk about what's actually going to be happening. We're going to be trying to find the parameters of this function. This is going to be the input, basically that sort of like specific x one and x two that you saw. In this case, because we want to actually leverage the hardware parallel capabilities, we're going to be able to submit multiple as micro batches so instead of just running one by one. We're going to be doing like five at the same time so the GPU actually runs five, and then comes back to us. So that's, that's how we're going to be able to do this. We're going to be learning this to parameters w and B. And we're going to be basically, you know this is the function that calculates that prediction. Right, we're not going to be delving into too many too much of the depths but there's a blog post that covers this in a bit more detail. And there's thousands of talks that talk about logistic regression in Python so more than welcome to check that key thing here is that this will be the shader code that we're going to be writing, right and even though we're not going to cover in much detail we're still going to be looking at what that what actually required to actually write that. And then the compute side so this is the shader but the compute side is what is going to be running this, we're going to have to create a bunch of tensors that represent our input data, our parameters, our predictions, etc etc the training data, we're going to have to initialize those tensors using, you know, in the sequence, we're going to initialize the algorithm, right, and then we're going to actually, so we're going to record it. And then we're going to actually like initialize them, and then we're going to iterate and learn, let the machine do the learning right so we're actually going to be running multiple different sort of like iterations to that data set, updating the parameters every single time, running micro batches that are running in the GPU in parallel. Yes, we're going to be basically doing that. And then, once we actually iterate those 100 times, we're going to be having learned those parameters. And basically what we're going to be doing in this high level logic. And you know, you know this is just an intuition right the key thing here is to see what's happening in the, in the compute and in the in the in the shader side. So the shader what it looks like is just a much more complex version of what we saw earlier. Right so we have all of the inputs so Xi, which is x one x two is each of them they're going to be an array because it's micro batches. So we have the expected outputs, we have the weights that are coming in and we have the weights that are output and calculated remember that the parameters that we're learning are w and B. So those are things that we actually want to take out. And you know we also taking the loss to be able to reuse where we're relevant, and the number of parameters to use right so in this case what we're going to be doing is we're going to be, you know, taking the specific input weights that are continuously in each of them. And then we're going to be able to take out the parameters that we're going to be able to use. And then we're going to be able to take out the parameters that are, these are the ones, the parameters that are we're going to update in each execution so we have to pass them every single time. We're going to be calculating the function right so as we saw, you know this is basically the function that we just had. And you know the blog post I break it down in very minute detail of how we actually go through each of of these steps and and we're going to be able to take out the parameters as the respective loss. But here the key thing is that we are able to ultimately calculate the, the, the, the parameters that that we now have. Right so now in the compute side, we're going to first create all of the specific tensor so as we saw we had some training data. This is the zero, zero equals zero. So we're going to be able to calculate the function zero should be that you know one one should be one etc etc. And then you know this is how we start with our weights so we start with just like a random initialization, which then you know we're going to be iterating towards. We're going to then similarly start with, you know, run a mission initialization for our other parameter which is going to be zero. And then the number of, you know, data points is going to be basically the actual size of this parameter right so we're going to start with with five. Right so that's what we're going to be doing we're going to be starting that in a variable called parameters so we can reference them. We're then going to just initialize it by creating our manager and initialize the tensors so what this does is it actually initializes them explicitly, whereas before we didn't do it we did, we did it implicitly with that, with a utility function so we initialize all of the parameters in the GPU so they're accessible in GPU memory. Then we're actually creating and recording this the operations in the sequence. So what are the operations, we're going to first actually sync the data to device for these two tensors, because remember the parameters are going to be updated every iteration. So we're going to be in the GPU device memory, we're going to then record that algorithm that we just actually wrote this is the the logistic regression shader that you saw so we're going to record that execution, and then we're going to record a sync to local. Right so we're going to record for all of the weights, the parameters and the, and the, and the, and the loss to actually be copied back to the host so that Python can see them. And then finally we're going to iterate 100 times by, you know, every iteration, running that shader or running the sequence which is basically all of the things that we just recorded, and then operating and then updating all of the, all of the weights. So I think there's an indentation missing here but basically what is happening here is we're just basically updating it by, you know, the specific learning rates which is just how fast do we want the actual parameters to be updated on each iteration. Again, the key thing here is just to see all of the features that you're able to use with compute and leveraging a simple machine learning use case as an example. And of course we're skimming through unconscious. Sorry for that for the people that you know, maybe are seeing this and going like well you know actually that's you know, 100% correct but you know in the blood post I break it down and you know much more in detail. This is just to show how you're able to actually interact with the GPU, and you know, optimize in different areas by you know pre recording components using the sequence etc etc. So in this case we're just iterating. In the finished you're able to then just print the calculated parameters which in this case, they are the first way the second way. The third way, and the actual be bias, which is ultimately what we ended up with. And so to emphasize you know we covered kind of like this high level example, but as I mentioned we have a blog post that covers this one in Python, one in C++. It breaks it down in in my new detail, and we have other tutorials other examples that cover, you know how to use this and actually the C++ as opposed to the Python one for integrating with your Android apps, as well as to game engines like the Godot engine, which we would recommend to check out. So more than anything what I recommend is to get involved. You know if you go to GitHub.com slash ethical ml slash Vulkan compute, you'd be able to actually check out what are some of the open issues. You know you can take one of the good first issues labeled good first issues. And also please you know there's open there's an issue number 52 which is open for just general discussion so you have ideas for improvements, or questions, and you can actually just post them in the chat, and we've had some really interesting suggestions. So the key things in the roadmap, you know one of the main motivations to build this framework is to actually integrated as a backend of an existing site scientific computing framework that is potentially even being used for, you know, mobile, you know machine learning or for other types of use cases so definitely really interesting on that and if someone is, you know, running a scientific computing library then you know being open to explore. Also creating more default operations, something like a fast forward transform or like a parallel some reduction, things like that would be really cool to have like out of the box operations that are, you know perhaps even written in C++ but also exposed as a new model. And then also adding examples if you try this in a new sort of shader, or a new sort of like algorithm, a new machine learning type of model, you know would love to actually contributed upstream and you know added to the repo because I think that that would be very cool. And so that's it. I think that's everything that we had sort of to cover today. Thank you very much for joining this talk on beyond kura GPU accelerated Python on cross vendor graphics card with Vulkan and compute. And I would like to take this opportunity to take this opportunity forward to explore and hear your thoughts, ideas and suggestions. And if you have any questions do please feel, feel free to reach out. Thank you very much.
primarily around the parallel and GPU computing with, we want to cover the motivations of why collects insights from 227 papers in the parallel into distributed processing leveraging this actually talk about in later on. And, you know, to actually run a simple program in Vulkan, you know, in the GPU, and then you would actually is we want to actually train a machine learning about logistic regression in Python so more than