HPC on OpenStack
Seren, Ümit
HPC systems have been traditionally operated as monolithic installations on bare-metal hardware primarily used by users with computational background to submit classic batch jobs. However the commoditization of compute resources and the introduction of new scientific fields such as life sciences to high performance computing has caused a shift in this paradigm. Today, an increasing number of biological software is made accessible through web portals. This improved ease of use has led towards a democratization of access to computational resources Users of those fields don’t have the same computational knowledge as traditional HPC users from physics or chemistry and additionally require different kinds of workloads and applications that don’t fit traditional non-interactive batch scheduling resource management systems. Additionally, cloud computing is becoming more and more relevant and various efforts to lift HPC into the Cloud were started.
 So, let's get started for the next talk. We have, umit, umit, that's how you pronounce it, sorry. Talking about HPC, how to set up HPC on OpenStack. Yeah. Let's give him an applause. Yes. Oh, sorry. Um, again, my name is Umit Seren and I'm an HPC systems engineer at the Vienna Bio Center Campus, which is also home for four life science institutes. In the next 20 minutes, I want to tell you a story of how we deployed our new 200 nodes LAM cluster on top of OpenStack. But before I start, maybe a show of hands. How many people here are currently running an HPC system? That's probably half. And how many people have experienced or have run an OpenStack instance running? That's even more surprising. Okay, let's get started. Anyone remembers the great talk by Damian Franser from last year's Boston about the Cloudster? If you run that talk, the Cloudster is a hypothetical system of heterogeneous hardware that not only allows you traditional HPC workloads, but also new workloads around big data and stateful applications. And when we embarked on our journey to deploy our HPC system on OpenStack, we had something along these lines in mind, a system that would not only allow traditional workloads, batch scheduling workloads, but also new use cases. So meet the Cloud Infrastructure Project, or short clip, that was started to implement this idea. So the idea was not only to consolidate the various compute clusters we had on campus and replace it with a new HPC system, but realize the idea of a software-defined data center strategy that would allow a system like Cloudster. We chose OpenStack Cloud Framework to implement this idea, and a team of four full-time engineers was formed and tasked to deliver this project in multiple phases. And we set three milestones. The first milestone would be to implement a traditionally batch-scaling system on top of OpenStack. Once this is done, we would implement new interactive applications, such as Jupyter and RStudio. And finally, we would extend the use of OpenStack to all kinds of long-running stateful applications, such as databases and web services. So on a high level, this is what we were aiming at. So at the bottom, you have a pool of heterogeneous compute nodes. And on top of that, we would install the OpenStack Cloud. And with that, Cloud Framework, we would provision different flavored VMs, and those would host our different services ranging all the way from a classical batch-scaling system like Slurm, but also Spark, Jupyter Hub, and RStudio. And the idea would be that the Cloud Framework would allow us to dynamically resize those services on top and reconfigure the whole system. So let's briefly also talk about the hardware. Like it was proposed in CloudStack, we have quite heterogeneous hardware. So we have around 200 compute nodes that are split in four types. Type 1 compute nodes are split in two subtypes. So 20% of those are high-frequency, low-core count Xilix, Skylix, and the remaining 80 percent are basically high-core, low-frequency Skylix again. A Type 2 compute nodes are large memory nodes. We have one 4TB machine and three 1.5TB machines. We have 12 ION nodes with NVMe flash storage for around 250TB of BGFS scratch storage. And then we have six GPU nodes with a mix of T100 and V100. And all these nodes are connected to an existing software-defined fabric from Cisco with 100G Ethernet. And additionally, we have existing data-centered infrastructures such as network attached storage of around 10 petabytes. As I mentioned before, we delivered this project in multiple phases. The project started at the end of 2017 with a small proof-of-concept to understand OpenStack. This lasted two months and on a pretty small scale. So this was one undercloud, one free controllers, and two refurbished hypervisors. This was followed by a deep analysis of OpenStack to understand the impact on the HPC workload. We developed, we evaluated different deployment methods, looked at tooling around it, also developed patterns around operations, and did extensive benchmarking. This was supposed to take eight months. We ended up working on this phase for one year. At the end of 2018, the hardware arrived and was installed. And the goal was to deploy the production cloud and the HPC payload on top of that within four months. This took us in the end 10 months. But in July 2019, our SLAM cluster went into production. And since then, we have been working on delivering the next milestone, namely those interactive applications such as Jupyter have an RStudio. And if you want to know more about this, just stay in the room because my colleague, Eric Biringruber, will talk about this right after. So now let's talk briefly about our experience with OpenStack, regarding the first deliverable, namely an HPC system. And first, I want to talk about how we deploy the cloud and how we operate it. There are a lot of different OpenStack installers out there. I think every major OpenStack distribution has its own. We decided to go with Red Hat OpenStack because of two reasons. First of all, we have an existing support contract. So we get vendor support. But more importantly, we wanted to deeply integrate OpenStack with our Cisco software-defined network fabric. And Cisco only provides installer for Canonical and Red Hat. Red Hat OpenStack uses a system called triple O that stands short for OpenStack on OpenStack. And triple O consists of two parts. You have a so-called undercloud, which is a single node installation of OpenStack with some of the services. And using this undercloud, and specifically we use Ironic for bare metal provisioning and heat the orchestration engine, we deploy our overcloud. This is the actual cloud that will run our HPC payload eventually. This approaches a couple of benefits. For us, the biggest benefit was that we as SUS admins only have to learn one set of API's to both deploy the cloud and as well as later on the payload on top of it. So for example, we use the image service glance. People who are familiar with OpenStack notice this is used to store OS images or VM images. We use the glance image service in the undercloud to store the OS images for the hypervisors and the controllers in the overcloud. And then we also use the glance image in the overcloud to store the VM images for our slurm payload. And tripleO supports two ways to install the cloud. One is through a web UI and the other one is through CLI using YAML files. Because we were new to tripleO, we decided to have a look at the UI. It looks pretty nice and you can configure the network and storage and then you can just kick it off and eventually you end up with a working cloud. However, this approach didn't work for us because we have three OpenStack environments with identical hardware to test different configurations. And obviously clicking three times through the UI is not scalable, doesn't allow for fast iterations and also isn't reproducible. So we decided just, okay, we scrap the web UI and just do everything from the CLI. And tripleO, as I mentioned before, uses a bunch of OpenStack services, but most importantly uses the ironic and heat to deploy the bare metal nodes in the overcloud. And once the bare metal nodes are deployed, a puppet agent will run on all the compute nodes and controller nodes and that in turn runs ansible to configure the services and install the nodes. You heard right, it's quite convoluted and it's also pretty complex. We also had to decide for a config management tool because we needed to configure other data center resources such as storage and network that is outside of tripleO. And we decided to go with ansible because that was the easiest one for us to use. Our deployment is done in three steps. So we have a set of playbooks that run on a Bashin VM that will deploy and install the undercloud in the free environments. So basically three underclouds in each of those environments. Once this is done, we have another set of playbooks that we will do some pre-validation and then feed the customized YAML files for the cloud configuration into the tripleO installer, drive the tripleO installer and deploy the overcloud and do some post-deployment validation. And then we have a third set of playbook that will do post-deployment customizations such as uploading VM images that we need later for the payload and pre-creating network constructs. And we can use the same set of playbooks that are by the way stored in two Git repos to scale up and down our cloud or to reconfigure it. So this approach works pretty well, but there are a couple of pitfalls and downsides you should be aware of. TripleO is very slow because it runs heat, runs puppet and then ansible. So if you want to change a single configuration in one of the services and need to restart the service, this will take on our production deployment 60 minutes. So obviously this is pretty painful, especially in the beginning when you haven't really settled on the final configuration of your cloud and you're iterating quickly. So we thought, okay, let's just use tripleO to get some working cloud running and then use ansible to modify or to find the final set of configurations. But this also doesn't work because when you do a stack update and you need to do a stack update when you scale up or down, the puppet agent will overwrite all your ansible modified files and some services can be done, which is actually not acceptable. So we decided to compromise, we decided, okay, in the beginning let's use ansible to find the final set of configurations. And once we settled on this final set of configuration, we would bring them back into the tripleO framework and use the tripleO exposed configuration stances. For everything else, we continue to use ansible. So we have a playbook to talk to the Cisco SDN controller and set up the network configuration for the free open stack environments. And the nice benefit is we also have a playbook that can move a node from one environment to another and back. So let's change gears and talk a bit also about operations. As I mentioned, we have free open stack environments and we have all the configuration for the cloud in Yammerl and Git. This allows us to, this is also for reproducible deployments, which is pretty nice. But what about software packages? How can we ensure that the deployment of software packages is also reproducible? That releases on a regular basis updates to OS packages but also to the open stack services and packages. We decided to use Red Hat Satellite, the upstream project is called Forman, to create free life cycle environments that correspond to our free open stack environments. And then we create content views. These are basically sets of RPM repos and create also versions for those content views that's basically freezing the versions of the packages at a certain time point. And then we can publish these versions and roll them out in those free life cycle environments. This way we can test, for example, new open stack packages in the depth environment and then roll them out in staging and production. As any other software, open stack has this fair share of bugs. And for some of those we came up with workarounds, for some we created tickets with Red Hat and others were fixed in upstream, newer upstream versions of either open stack or some of the packages. To keep track of all of this, we created gyro boards. So when we encounter a bug, we triage it, we document it. But most importantly, we put it in this gyro board, we call it Clip Release Engineering. And when we update open stack or one of the packages, we can come back to this gyro board, look at the tickets and maybe unroll the workarounds for some of the bugs that were fixed in the version that we deployed. So some concluding remarks about the deployment and operations. Open stack and triple low are pretty complex. So the only thing I want you to take out of this talk is that you really need a depth environment to test upgrades and also I would recommend to have some package management. Upgrades can break your cloud in different ways. For example, for us, going from a noncontaminarized control plane to a containerized was the biggest pain point. Sometimes you also need to customize the upstream containers and having a container built pipeline Jenkins is obviously very helpful. Triple OS also well supported installer for a common cloud deployment. If you have some exotic configuration, it's a bit challenging within this rigid triple of framework. I didn't talk about the next point, but obviously having monitoring and continuous test regression testing and performance testing is very important. And finally, infrared codes in in git is the way to go, but it requires really good developer discipline. So you need to do PRs and also release engineering. So now we have our clouds. How do we deploy our HPC system on top of that? We deploy our HPC or our slurm cluster in a two step process. We use the open stack heat orchestration engine in the overcloud to provision a bunch of VMs and generate an Ansible inventory. And then we have an Ansible playbook that will install and configure the slurm cluster using either self-developed Ansible rows, but we also heavily rely on stack HPCs very good Ansible rows. And we can use the same set of playbooks to scale up and down our slurm cluster or actually reconfigure the slurm cluster. We use satellites. So we register the VMs in satellites for package management. And because we have free environments, depth staging and production, we can test both different configurations of our slurm cluster and also newer package versions in depth. And if they work out, we can just roll them out in production. And this whole approach can be used for other complex system. So we can use it for provisioned Spark cluster, Kates cluster. We also use it for BGFS. So let's also briefly talk about performance. We deploy a virtualized slurm cluster. In order to get good performance, we need to apply a couple of tunings. And this is the non-exhaustive list of the tunings we apply. The most important thing is that every tuning comes with a caveat or downside. So for example, with new Maclean instances, you lose the ability to do live migrations and you cannot easily mix different VM flavors in the same hypervisor. This is not a big issue for the slurm cluster, but you should still be aware of it. We use static huge pages, but make sure that you leave enough memory for the host because otherwise you won't be able to update your cloud because the puppet agent on the node will be killed by the OM if there's not enough memory. Simple CPUs also helps with performance, but requires you to use SRV for good network performance. And finally, for GPUs and NVMe flash drives, you need to use PCI pass through. And in general, for network performance, SRV is the way to go. But keep in mind that those tunings have a smaller feature set compared to their fully virtualized counterparts. So with all these tunings, you can basically achieve their metal performance. And I don't want to bore you with plenty of charts. I only want to show you one. So these are here, you can see different bio-info workloads running in parallel on a node. Here on the left side, you can see basically if they run on a bare metal node without any isolation, and this is the standard deviation for the execution time, basically the jitter. And you can see that there's a lot of jitter, obviously because the two workloads affect each other. So if you spin up two optimized VMs on a hypervisor along the NUMMA boundaries and run the workloads in those VMs, they are well isolated and jitters reduced considerably. To be fair, slur on a bare metal node using C-groups does a fairly well job to do the same. So some important issues and pit-forced lessons that we learned regarding the deployment of our HPC payload on top of OpenStack. So Ansible is slow. If we run our slur on Playbook the second time, basically no modification end-to-end, this takes on the production instance an hour. So make sure to tag reoccurrents day-to-day operations like mounting new volumes or changing crosses with Ansible Tech so you can just run those tasks. And also you can use fat images to reduce the install time for the VMs. That is great for package management, but make sure to remove the upstream-centers images after you provision your VMs because otherwise all the versioning is for nothing. And also some issues you will only hit on the scale. So we tested our deployment on a small scale. Everything worked out. We decided to do the production deployment. And then we ran into a nasty Cisco SDN bug that basically forced us to scale in batches after 70 nodes. So be aware of those things that they can happen. So basically having isolated environments is pretty great. But this ends at shared infocomponents, especially if it's tightly integrated with OpenStack. So what do I mean with this? We once deployed the depth environments and we triggered a race condition in the Cisco SDN controller that basically caused an outage in the entire data center. And finally also be aware of unintended consequences of code changes. So once we wanted to scale up our SLAM cluster and we accidentally triggered a redeploy of the entire cluster, because months ago we changed one line in the heat configuration and heat sometimes will reprovision the VMs. And then we ran actually in the scaling issue, so the downtime was much, much longer than we planned for. I think I'm probably out of time. I will leave this, I will end the presentation on this last slide where I show you the bad ugly and the good parts and I will answer any questions you have. All right. Questions? I have actually two questions, first is what test you do before going to test from test to cross? Do you use rally or do you use something like rally or something else? And the second question, what is, how do you move one node from one environment to a different environment? So to the first question, for testing, we test both, so we have a testing for the cloud where we use tempest and rally for performance, but to be aware that the rally or tempest test that basically test your cloud functionality can leave some, you know, can leave some stray projects behind, right? And currently we are not running them. We so far manually test the payload itself. We test the answerable roles using molecule and then we are planning to use, for example, reframe to do some integration testing of also software packages. And the second question was about, yes, so the nice thing if you have a Cisco software defined network, basically it's like going to the switch and replugging the node and you just do this through the Cisco SDI has a rest endpoint and you can talk to the rest endpoint and just basically plug a compute node with the provisioning nick and the HSN, which is the high speed network from one environment, from the deaf environment, for example, to the production environment, and then you just reprovision the bare metal node. That's how we do it. That's pretty useful because we have different nodes and we can test, for example, new configuration firmware updates in depth and then move them back to production. Are you planning to replace Ansible with NSN faster because like one hour is really short? Yes. Well, we haven't really investigated if we can use something which is faster. We will probably have to spend a bit of time to optimize the playbook like with get effects and maybe, I mean, one aspect why this is specifically very long is that we mount, I think, 50 volumes. This is due to some, how we have the storage in the two institutes. We are unifying also the storage system. There will be only one mount point. So just mounting all the 50 volumes on all the 200 compute nodes, even if you do this in parallel, takes half an hour. So there are a couple of things we will do to cut down on the runtime for Ansible. But if anyone has a good suggestion for an alternative, which is agentless, we would be happy to hear about this. What share of your Ansible playbooks and roles did you have to write yourself? And what could you reuse from the existing repositories? So we could use, for example, the entire SLAM role. So we took the SLAM role. I think StackHPC has a couple of roles with which they deploy different HPC clusters. And we use the SLAM role from them a bit modified. For example, we have three partitions and we have three causes. We need to set limits. And SLAM only accepts absolute values for core limits. And we wanted to specify this in percentages. And so we modified this to calculate based on the number of cores, how many, so to calculate the absolute value. So these were a couple of modifications. Some of the small roles like SSSD and other stuff we developed on our own or used something and went from there, basically. Any more questions? All right. And let's thank the speaker.
 So, let's get started for the next talk.  We have, umit, umit, that's how you pronounce it, sorry.  Talking about HPC, how to set up HPC on OpenStack.  Yeah.  Let's give him an applause.  Yes.  Oh, sorry.  Um, again, my name is Umit Seren and I'm an HPC systems engineer at the Vienna Bio Center  Campus, which is also home for four life science institutes.  In the next 20 minutes, I want to tell you a story of how we deployed our new 200 nodes  LAM cluster on top of OpenStack.  But before I start, maybe a show of hands.  How many people here are currently running an HPC system?  That's probably half.  And how many people have experienced or have run an OpenStack instance running?  That's even more surprising.  Okay, let's get started.  Anyone remembers the great talk by Damian Franser from last year's Boston about the Cloudster?  If you run that talk, the Cloudster is a hypothetical system of heterogeneous hardware that not only  allows you traditional HPC workloads, but also new workloads around big data and stateful  applications.  And when we embarked on our journey to deploy our HPC system on OpenStack, we had something  along these lines in mind, a system that would not only allow traditional workloads, batch  scheduling workloads, but also new use cases.  So meet the Cloud Infrastructure Project, or short clip, that was started to implement  this idea.  So the idea was not only to consolidate the various compute clusters we had on campus  and replace it with a new HPC system, but realize the idea of a software-defined data  center strategy that would allow a system like Cloudster.  We chose OpenStack Cloud Framework to implement this idea, and a team of four full-time engineers  was formed and tasked to deliver this project in multiple phases.  And we set three milestones.  The first milestone would be to implement a traditionally batch-scaling system on top  of OpenStack.  Once this is done, we would implement new interactive applications, such as Jupyter  and RStudio.  And finally, we would extend the use of OpenStack to all kinds of long-running stateful applications,  such as databases and web services.  So on a high level, this is what we were aiming at.  So at the bottom, you have a pool of heterogeneous compute nodes.  And on top of that, we would install the OpenStack Cloud.  And with that, Cloud Framework, we would provision different flavored VMs, and those would host  our different services ranging all the way from a classical batch-scaling system like  Slurm, but also Spark, Jupyter Hub, and RStudio.  And the idea would be that the Cloud Framework would allow us to dynamically resize those  services on top and reconfigure the whole system.  So let's briefly also talk about the hardware.  Like it was proposed in CloudStack, we have quite heterogeneous hardware.  So we have around 200 compute nodes that are split in four types.  Type 1 compute nodes are split in two subtypes.  So 20% of those are high-frequency, low-core count Xilix, Skylix, and the remaining 80  percent are basically high-core, low-frequency Skylix again.  A Type 2 compute nodes are large memory nodes.  We have one 4TB machine and three 1.5TB machines.  We have 12 ION nodes with NVMe flash storage for around 250TB of BGFS scratch storage.  And then we have six GPU nodes with a mix of T100 and V100.  And all these nodes are connected to an existing software-defined fabric from Cisco with 100G  Ethernet.  And additionally, we have existing data-centered infrastructures such as network attached storage  of around 10 petabytes.  As I mentioned before, we delivered this project in multiple phases.  The project started at the end of 2017 with a small proof-of-concept to understand OpenStack.  This lasted two months and on a pretty small scale.  So this was one undercloud, one free controllers, and two refurbished hypervisors.  This was followed by a deep analysis of OpenStack to understand the impact on the HPC workload.  We developed, we evaluated different deployment methods, looked at tooling around it, also  developed patterns around operations, and did extensive benchmarking.  This was supposed to take eight months.  We ended up working on this phase for one year.  At the end of 2018, the hardware arrived and was installed.  And the goal was to deploy the production cloud and the HPC payload on top of that within  four months.  This took us in the end 10 months.  But in July 2019, our SLAM cluster went into production.  And since then, we have been working on delivering the next milestone, namely those interactive  applications such as Jupyter have an RStudio.  And if you want to know more about this, just stay in the room because my colleague, Eric  Biringruber, will talk about this right after.  So now let's talk briefly about our experience with OpenStack, regarding the first deliverable,  namely an HPC system.  And first, I want to talk about how we deploy the cloud and how we operate it.  There are a lot of different OpenStack installers out there.  I think every major OpenStack distribution has its own.  We decided to go with Red Hat OpenStack because of two reasons.  First of all, we have an existing support contract.  So we get vendor support.  But more importantly, we wanted to deeply integrate OpenStack with our Cisco software-defined  network fabric.  And Cisco only provides installer for Canonical and Red Hat.  Red Hat OpenStack uses a system called triple O that stands short for OpenStack on OpenStack.  And triple O consists of two parts.  You have a so-called undercloud, which is a single node installation of OpenStack with  some of the services.  And using this undercloud, and specifically we use Ironic for bare metal provisioning  and heat the orchestration engine, we deploy our overcloud.  This is the actual cloud that will run our HPC payload eventually.  This approaches a couple of benefits.  For us, the biggest benefit was that we as SUS admins only have to learn one set of API's  to both deploy the cloud and as well as later on the payload on top of it.  So for example, we use the image service glance.  People who are familiar with OpenStack notice this is used to store OS images or VM images.  We use the glance image service in the undercloud to store the OS images for the hypervisors  and the controllers in the overcloud.  And then we also use the glance image in the overcloud to store the VM images for our  slurm payload.  And tripleO supports two ways to install the cloud.  One is through a web UI and the other one is through CLI using YAML files.  Because we were new to tripleO, we decided to have a look at the UI.  It looks pretty nice and you can configure the network and storage and then you can just  kick it off and eventually you end up with a working cloud.  However, this approach didn't work for us because we have three OpenStack environments  with identical hardware to test different configurations.  And obviously clicking three times through the UI is not scalable, doesn't allow for  fast iterations and also isn't reproducible.  So we decided just, okay, we scrap the web UI and just do everything from the CLI.  And tripleO, as I mentioned before, uses a bunch of OpenStack services, but most importantly  uses the ironic and heat to deploy the bare metal nodes in the overcloud.  And once the bare metal nodes are deployed, a puppet agent will run on all the compute  nodes and controller nodes and that in turn runs ansible to configure the services and  install the nodes.  You heard right, it's quite convoluted and it's also pretty complex.  We also had to decide for a config management tool because we needed to configure other  data center resources such as storage and network that is outside of tripleO.  And we decided to go with ansible because that was the easiest one for us to use.  Our deployment is done in three steps.  So we have a set of playbooks that run on a Bashin VM that will deploy and install the  undercloud in the free environments.  So basically three underclouds in each of those environments.  Once this is done, we have another set of playbooks that we will do some pre-validation  and then feed the customized YAML files for the cloud configuration into the tripleO installer,  drive the tripleO installer and deploy the overcloud and do some post-deployment validation.  And then we have a third set of playbook that will do post-deployment customizations such  as uploading VM images that we need later for the payload and pre-creating network constructs.  And we can use the same set of playbooks that are by the way stored in two Git repos to  scale up and down our cloud or to reconfigure it.  So this approach works pretty well, but there are a couple of pitfalls and downsides you  should be aware of.  TripleO is very slow because it runs heat, runs puppet and then ansible.  So if you want to change a single configuration in one of the services and need to restart  the service, this will take on our production deployment 60 minutes.  So obviously this is pretty painful, especially in the beginning when you haven't really  settled on the final configuration of your cloud and you're iterating quickly.  So we thought, okay, let's just use tripleO to get some working cloud running and then  use ansible to modify or to find the final set of configurations.  But this also doesn't work because when you do a stack update and you need to do a stack  update when you scale up or down, the puppet agent will overwrite all your ansible modified  files and some services can be done, which is actually not acceptable.  So we decided to compromise, we decided, okay, in the beginning let's use ansible to find  the final set of configurations.  And once we settled on this final set of configuration, we would bring them back into the tripleO  framework and use the tripleO exposed configuration stances.  For everything else, we continue to use ansible.  So we have a playbook to talk to the Cisco SDN controller and set up the network configuration  for the free open stack environments.  And the nice benefit is we also have a playbook that can move a node from one environment  to another and back.  So let's change gears and talk a bit also about operations.  As I mentioned, we have free open stack environments and we have all the configuration for the  cloud in Yammerl and Git.  This allows us to, this is also for reproducible deployments, which is pretty nice.  But what about software packages?  How can we ensure that the deployment of software packages is also reproducible?  That releases on a regular basis updates to OS packages but also to the open stack services  and packages.  We decided to use Red Hat Satellite, the upstream project is called Forman, to create free life  cycle environments that correspond to our free open stack environments.  And then we create content views.  These are basically sets of RPM repos and create also versions for those content views  that's basically freezing the versions of the packages at a certain time point.  And then we can publish these versions and roll them out in those free life cycle environments.  This way we can test, for example, new open stack packages in the depth environment and  then roll them out in staging and production.  As any other software, open stack has this fair share of bugs.  And for some of those we came up with workarounds, for some we created tickets with Red Hat and  others were fixed in upstream, newer upstream versions of either open stack or some of the  packages.  To keep track of all of this, we created gyro boards.  So when we encounter a bug, we triage it, we document it.  But most importantly, we put it in this gyro board, we call it Clip Release Engineering.  And when we update open stack or one of the packages, we can come back to this gyro board,  look at the tickets and maybe unroll the workarounds for some of the bugs that were fixed in the  version that we deployed.  So some concluding remarks about the deployment and operations.  Open stack and triple low are pretty complex.  So the only thing I want you to take out of this talk is that you really need a depth  environment to test upgrades and also I would recommend to have some package management.  Upgrades can break your cloud in different ways.  For example, for us, going from a noncontaminarized control plane to a containerized was the  biggest pain point.  Sometimes you also need to customize the upstream containers and having a container  built pipeline Jenkins is obviously very helpful.  Triple OS also well supported installer for a common cloud deployment.  If you have some exotic configuration, it's a bit challenging within this rigid triple  of framework.  I didn't talk about the next point, but obviously having monitoring and continuous test regression  testing and performance testing is very important.  And finally, infrared codes in in git is the way to go, but it requires really good  developer discipline.  So you need to do PRs and also release engineering.  So now we have our clouds.  How do we deploy our HPC system on top of that?  We deploy our HPC or our slurm cluster in a two step process.  We use the open stack heat orchestration engine in the overcloud to provision a bunch of VMs  and generate an Ansible inventory.  And then we have an Ansible playbook that will install and configure the slurm cluster  using either self-developed Ansible rows, but we also heavily rely on stack HPCs very  good Ansible rows.  And we can use the same set of playbooks to scale up and down our slurm cluster or actually  reconfigure the slurm cluster.  We use satellites.  So we register the VMs in satellites for package management.  And because we have free environments, depth staging and production, we can test both different  configurations of our slurm cluster and also newer package versions in depth.  And if they work out, we can just roll them out in production.  And this whole approach can be used for other complex system.  So we can use it for provisioned Spark cluster, Kates cluster.  We also use it for BGFS.  So let's also briefly talk about performance.  We deploy a virtualized slurm cluster.  In order to get good performance, we need to apply a couple of tunings.  And this is the non-exhaustive list of the tunings we apply.  The most important thing is that every tuning comes with a caveat or downside.  So for example, with new Maclean instances, you lose the ability to do live migrations  and you cannot easily mix different VM flavors in the same hypervisor.  This is not a big issue for the slurm cluster, but you should still be aware of it.  We use static huge pages, but make sure that you leave enough memory for the host because  otherwise you won't be able to update your cloud because the puppet agent on the node  will be killed by the OM if there's not enough memory.  Simple CPUs also helps with performance, but requires you to use SRV for good network  performance.  And finally, for GPUs and NVMe flash drives, you need to use PCI pass through.  And in general, for network performance, SRV is the way to go.  But keep in mind that those tunings have a smaller feature set compared to their fully  virtualized counterparts.  So with all these tunings, you can basically achieve their metal performance.  And I don't want to bore you with plenty of charts.  I only want to show you one.  So these are here, you can see different bio-info workloads running in parallel on a  node.  Here on the left side, you can see basically if they run on a bare metal node without any  isolation, and this is the standard deviation for the execution time, basically the jitter.  And you can see that there's a lot of jitter, obviously because the two workloads affect  each other.  So if you spin up two optimized VMs on a hypervisor along the NUMMA boundaries and run  the workloads in those VMs, they are well isolated and jitters reduced considerably.  To be fair, slur on a bare metal node using C-groups does a fairly well job to do the  same.  So some important issues and pit-forced lessons that we learned regarding the deployment of  our HPC payload on top of OpenStack.  So Ansible is slow.  If we run our slur on Playbook the second time, basically no modification end-to-end,  this takes on the production instance an hour.  So make sure to tag reoccurrents day-to-day operations like mounting new volumes or changing  crosses with Ansible Tech so you can just run those tasks.  And also you can use fat images to reduce the install time for the VMs.  That is great for package management, but make sure to remove the upstream-centers images  after you provision your VMs because otherwise all the versioning is for nothing.  And also some issues you will only hit on the scale.  So we tested our deployment on a small scale.  Everything worked out.  We decided to do the production deployment.  And then we ran into a nasty Cisco SDN bug that basically forced us to scale in batches  after 70 nodes.  So be aware of those things that they can happen.  So basically having isolated environments is pretty great.  But this ends at shared infocomponents, especially if it's tightly integrated with OpenStack.  So what do I mean with this?  We once deployed the depth environments and we triggered a race condition in the Cisco  SDN controller that basically caused an outage in the entire data center.  And finally also be aware of unintended consequences of code changes.  So once we wanted to scale up our SLAM cluster and we accidentally triggered a redeploy of  the entire cluster, because months ago we changed one line in the heat configuration  and heat sometimes will reprovision the VMs.  And then we ran actually in the scaling issue, so the downtime was much, much longer than  we planned for.  I think I'm probably out of time.  I will leave this, I will end the presentation on this last slide where I show you the bad  ugly and the good parts and I will answer any questions you have.  All right.  Questions?  I have actually two questions, first is what test you do before going to test from test  to cross?  Do you use rally or do you use something like rally or something else?  And the second question, what is, how do you move one node from one environment to a different  environment?  So to the first question, for testing, we test both, so we have a testing for the cloud  where we use tempest and rally for performance, but to be aware that the rally or tempest  test that basically test your cloud functionality can leave some, you know, can leave some stray  projects behind, right?  And currently we are not running them.  We so far manually test the payload itself.  We test the answerable roles using molecule and then we are planning to use, for example,  reframe to do some integration testing of also software packages.  And the second question was about, yes, so the nice thing if you have a Cisco software  defined network, basically it's like going to the switch and replugging the node and  you just do this through the Cisco SDI has a rest endpoint and you can talk to the rest  endpoint and just basically plug a compute node with the provisioning nick and the HSN,  which is the high speed network from one environment, from the deaf environment, for  example, to the production environment, and then you just reprovision the bare metal node.  That's how we do it.  That's pretty useful because we have different nodes and we can test, for example, new configuration  firmware updates in depth and then move them back to production.  Are you planning to replace Ansible with NSN faster because like one hour is really  short?  Yes.  Well, we haven't really investigated if we can use something which is faster.  We will probably have to spend a bit of time to optimize the playbook like with get effects  and maybe, I mean, one aspect why this is specifically very long is that we mount, I  think, 50 volumes.  This is due to some, how we have the storage in the two institutes.  We are unifying also the storage system.  There will be only one mount point.  So just mounting all the 50 volumes on all the 200 compute nodes, even if you do this  in parallel, takes half an hour.  So there are a couple of things we will do to cut down on the runtime for Ansible.  But if anyone has a good suggestion for an alternative, which is agentless, we would  be happy to hear about this.  What share of your Ansible playbooks and roles did you have to write yourself?  And what could you reuse from the existing repositories?  So we could use, for example, the entire SLAM role.  So we took the SLAM role.  I think StackHPC has a couple of roles with which they deploy different HPC clusters.  And we use the SLAM role from them a bit modified.  For example, we have three partitions and we have three causes.  We need to set limits.  And SLAM only accepts absolute values for core limits.  And we wanted to specify this in percentages.  And so we modified this to calculate based on the number of cores, how many, so to calculate  the absolute value.  So these were a couple of modifications.  Some of the small roles like SSSD and other stuff we developed on our own or used something  and went from there, basically.  Any more questions?  All right.  And let's thank the speaker.
hpc_openstack_0_167.png hpc_openstack_168_515.png hpc_openstack_516_587.png hpc_openstack_588_1007.png hpc_openstack_1008_1103.png hpc_openstack_1104_1883.png hpc_openstack_1884_2687.png hpc_openstack_2688_4091.png hpc_openstack_4092_4979.png hpc_openstack_4980_6635.png hpc_openstack_6636_7067.png hpc_openstack_7068_7607.png hpc_openstack_7608_7751.png hpc_openstack_7752_8063.png hpc_openstack_8064_8231.png hpc_openstack_8232_8591.png hpc_openstack_8592_8939.png hpc_openstack_8940_11399.png hpc_openstack_11400_11627.png hpc_openstack_11628_12155.png hpc_openstack_12156_14723.png hpc_openstack_14724_17075.png hpc_openstack_17076_18167.png hpc_openstack_18168_18887.png hpc_openstack_18888_20015.png hpc_openstack_20016_20303.png hpc_openstack_20304_21731.png hpc_openstack_21732_21887.png hpc_openstack_21888_23459.png hpc_openstack_23460_25331.png hpc_openstack_25332_26435.png hpc_openstack_26436_29483.png hpc_openstack_29484_33695.png hpc_openstack_33696_34151.png hpc_openstack_34152_35459.png hpc_openstack_35460_35579.png hpc_openstack_35580_35687.png hpc_openstack_35688_35807.png
So, let's get started for the next talk. We have, umit, umit, that's how you pronounce it, sorry. Talking about HPC, how to set up HPC on OpenStack. Yeah. Let's give him an applause. Yes. Oh, sorry. Um, again, my name is Umit Seren and I'm an HPC systems engineer at the Vienna Bio Center Campus, which is also home for four life science institutes. In the next 20 minutes, I want to tell you a story of how we deployed our new 200 nodes LAM cluster on top of OpenStack. But before I start, maybe a show of hands. How many people here are currently running an HPC system? That's probably half. And how many people have experienced or have run an OpenStack instance running? That's even more surprising. Okay, let's get started. Anyone remembers the great talk by Damian Franser from last year's Boston about the Cloudster? If you run that talk, the Cloudster is a hypothetical system of heterogeneous hardware that not only allows you traditional HPC workloads, but also new workloads around big data and stateful applications. And when we embarked on our journey to deploy our HPC system on OpenStack, we had something along these lines in mind, a system that would not only allow traditional workloads, batch scheduling workloads, but also new use cases. So meet the Cloud Infrastructure Project, or short clip, that was started to implement this idea. So the idea was not only to consolidate the various compute clusters we had on campus and replace it with a new HPC system, but realize the idea of a software-defined data center strategy that would allow a system like Cloudster. We chose OpenStack Cloud Framework to implement this idea, and a team of four full-time engineers was formed and tasked to deliver this project in multiple phases. And we set three milestones. The first milestone would be to implement a traditionally batch-scaling system on top of OpenStack. Once this is done, we would implement new interactive applications, such as Jupyter and RStudio. And finally, we would extend the use of OpenStack to all kinds of long-running stateful applications, such as databases and web services. So on a high level, this is what we were aiming at. So at the bottom, you have a pool of heterogeneous compute nodes. And on top of that, we would install the OpenStack Cloud. And with that, Cloud Framework, we would provision different flavored VMs, and those would host our different services ranging all the way from a classical batch-scaling system like Slurm, but also Spark, Jupyter Hub, and RStudio. And the idea would be that the Cloud Framework would allow us to dynamically resize those services on top and reconfigure the whole system. So let's briefly also talk about the hardware. Like it was proposed in CloudStack, we have quite heterogeneous hardware. So we have around 200 compute nodes that are split in four types. Type 1 compute nodes are split in two subtypes. So 20% of those are high-frequency, low-core count Xilix, Skylix, and the remaining 80 percent are basically high-core, low-frequency Skylix again. A Type 2 compute nodes are large memory nodes. We have one 4TB machine and three 1.5TB machines. We have 12 ION nodes with NVMe flash storage for around 250TB of BGFS scratch storage. And then we have six GPU nodes with a mix of T100 and V100. And all these nodes are connected to an existing software-defined fabric from Cisco with 100G Ethernet. And additionally, we have existing data-centered infrastructures such as network attached storage of around 10 petabytes. As I mentioned before, we delivered this project in multiple phases. The project started at the end of 2017 with a small proof-of-concept to understand OpenStack. This lasted two months and on a pretty small scale. So this was one undercloud, one free controllers, and two refurbished hypervisors. This was followed by a deep analysis of OpenStack to understand the impact on the HPC workload. We developed, we evaluated different deployment methods, looked at tooling around it, also developed patterns around operations, and did extensive benchmarking. This was supposed to take eight months. We ended up working on this phase for one year. At the end of 2018, the hardware arrived and was installed. And the goal was to deploy the production cloud and the HPC payload on top of that within four months. This took us in the end 10 months. But in July 2019, our SLAM cluster went into production. And since then, we have been working on delivering the next milestone, namely those interactive applications such as Jupyter have an RStudio. And if you want to know more about this, just stay in the room because my colleague, Eric Biringruber, will talk about this right after. So now let's talk briefly about our experience with OpenStack, regarding the first deliverable, namely an HPC system. And first, I want to talk about how we deploy the cloud and how we operate it. There are a lot of different OpenStack installers out there. I think every major OpenStack distribution has its own. We decided to go with Red Hat OpenStack because of two reasons. First of all, we have an existing support contract. So we get vendor support. But more importantly, we wanted to deeply integrate OpenStack with our Cisco software-defined network fabric. And Cisco only provides installer for Canonical and Red Hat. Red Hat OpenStack uses a system called triple O that stands short for OpenStack on OpenStack. And triple O consists of two parts. You have a so-called undercloud, which is a single node installation of OpenStack with some of the services. And using this undercloud, and specifically we use Ironic for bare metal provisioning and heat the orchestration engine, we deploy our overcloud. This is the actual cloud that will run our HPC payload eventually. This approaches a couple of benefits. For us, the biggest benefit was that we as SUS admins only have to learn one set of API's to both deploy the cloud and as well as later on the payload on top of it. So for example, we use the image service glance. People who are familiar with OpenStack notice this is used to store OS images or VM images. We use the glance image service in the undercloud to store the OS images for the hypervisors and the controllers in the overcloud. And then we also use the glance image in the overcloud to store the VM images for our slurm payload. And tripleO supports two ways to install the cloud. One is through a web UI and the other one is through CLI using YAML files. Because we were new to tripleO, we decided to have a look at the UI. It looks pretty nice and you can configure the network and storage and then you can just kick it off and eventually you end up with a working cloud. However, this approach didn't work for us because we have three OpenStack environments with identical hardware to test different configurations. And obviously clicking three times through the UI is not scalable, doesn't allow for fast iterations and also isn't reproducible. So we decided just, okay, we scrap the web UI and just do everything from the CLI. And tripleO, as I mentioned before, uses a bunch of OpenStack services, but most importantly uses the ironic and heat to deploy the bare metal nodes in the overcloud. And once the bare metal nodes are deployed, a puppet agent will run on all the compute nodes and controller nodes and that in turn runs ansible to configure the services and install the nodes. You heard right, it's quite convoluted and it's also pretty complex. We also had to decide for a config management tool because we needed to configure other data center resources such as storage and network that is outside of tripleO. And we decided to go with ansible because that was the easiest one for us to use. Our deployment is done in three steps. So we have a set of playbooks that run on a Bashin VM that will deploy and install the undercloud in the free environments. So basically three underclouds in each of those environments. Once this is done, we have another set of playbooks that we will do some pre-validation and then feed the customized YAML files for the cloud configuration into the tripleO installer, drive the tripleO installer and deploy the overcloud and do some post-deployment validation. And then we have a third set of playbook that will do post-deployment customizations such as uploading VM images that we need later for the payload and pre-creating network constructs. And we can use the same set of playbooks that are by the way stored in two Git repos to scale up and down our cloud or to reconfigure it. So this approach works pretty well, but there are a couple of pitfalls and downsides you should be aware of. TripleO is very slow because it runs heat, runs puppet and then ansible. So if you want to change a single configuration in one of the services and need to restart the service, this will take on our production deployment 60 minutes. So obviously this is pretty painful, especially in the beginning when you haven't really settled on the final configuration of your cloud and you're iterating quickly. So we thought, okay, let's just use tripleO to get some working cloud running and then use ansible to modify or to find the final set of configurations. But this also doesn't work because when you do a stack update and you need to do a stack update when you scale up or down, the puppet agent will overwrite all your ansible modified files and some services can be done, which is actually not acceptable. So we decided to compromise, we decided, okay, in the beginning let's use ansible to find the final set of configurations. And once we settled on this final set of configuration, we would bring them back into the tripleO framework and use the tripleO exposed configuration stances. For everything else, we continue to use ansible. So we have a playbook to talk to the Cisco SDN controller and set up the network configuration for the free open stack environments. And the nice benefit is we also have a playbook that can move a node from one environment to another and back. So let's change gears and talk a bit also about operations. As I mentioned, we have free open stack environments and we have all the configuration for the cloud in Yammerl and Git. This allows us to, this is also for reproducible deployments, which is pretty nice. But what about software packages? How can we ensure that the deployment of software packages is also reproducible? That releases on a regular basis updates to OS packages but also to the open stack services and packages. We decided to use Red Hat Satellite, the upstream project is called Forman, to create free life cycle environments that correspond to our free open stack environments. And then we create content views. These are basically sets of RPM repos and create also versions for those content views that's basically freezing the versions of the packages at a certain time point. And then we can publish these versions and roll them out in those free life cycle environments. This way we can test, for example, new open stack packages in the depth environment and then roll them out in staging and production. As any other software, open stack has this fair share of bugs. And for some of those we came up with workarounds, for some we created tickets with Red Hat and others were fixed in upstream, newer upstream versions of either open stack or some of the packages. To keep track of all of this, we created gyro boards. So when we encounter a bug, we triage it, we document it. But most importantly, we put it in this gyro board, we call it Clip Release Engineering. And when we update open stack or one of the packages, we can come back to this gyro board, look at the tickets and maybe unroll the workarounds for some of the bugs that were fixed in the version that we deployed. So some concluding remarks about the deployment and operations. Open stack and triple low are pretty complex. So the only thing I want you to take out of this talk is that you really need a depth environment to test upgrades and also I would recommend to have some package management. Upgrades can break your cloud in different ways. For example, for us, going from a noncontaminarized control plane to a containerized was the biggest pain point. Sometimes you also need to customize the upstream containers and having a container built pipeline Jenkins is obviously very helpful. Triple OS also well supported installer for a common cloud deployment. If you have some exotic configuration, it's a bit challenging within this rigid triple of framework. I didn't talk about the next point, but obviously having monitoring and continuous test regression testing and performance testing is very important. And finally, infrared codes in in git is the way to go, but it requires really good developer discipline. So you need to do PRs and also release engineering. So now we have our clouds. How do we deploy our HPC system on top of that? We deploy our HPC or our slurm cluster in a two step process. We use the open stack heat orchestration engine in the overcloud to provision a bunch of VMs and generate an Ansible inventory. And then we have an Ansible playbook that will install and configure the slurm cluster using either self-developed Ansible rows, but we also heavily rely on stack HPCs very good Ansible rows. And we can use the same set of playbooks to scale up and down our slurm cluster or actually reconfigure the slurm cluster. We use satellites. So we register the VMs in satellites for package management. And because we have free environments, depth staging and production, we can test both different configurations of our slurm cluster and also newer package versions in depth. And if they work out, we can just roll them out in production. And this whole approach can be used for other complex system. So we can use it for provisioned Spark cluster, Kates cluster. We also use it for BGFS. So let's also briefly talk about performance. We deploy a virtualized slurm cluster. In order to get good performance, we need to apply a couple of tunings. And this is the non-exhaustive list of the tunings we apply. The most important thing is that every tuning comes with a caveat or downside. So for example, with new Maclean instances, you lose the ability to do live migrations and you cannot easily mix different VM flavors in the same hypervisor. This is not a big issue for the slurm cluster, but you should still be aware of it. We use static huge pages, but make sure that you leave enough memory for the host because otherwise you won't be able to update your cloud because the puppet agent on the node will be killed by the OM if there's not enough memory. Simple CPUs also helps with performance, but requires you to use SRV for good network performance. And finally, for GPUs and NVMe flash drives, you need to use PCI pass through. And in general, for network performance, SRV is the way to go. But keep in mind that those tunings have a smaller feature set compared to their fully virtualized counterparts. So with all these tunings, you can basically achieve their metal performance. And I don't want to bore you with plenty of charts. I only want to show you one. So these are here, you can see different bio-info workloads running in parallel on a node. Here on the left side, you can see basically if they run on a bare metal node without any isolation, and this is the standard deviation for the execution time, basically the jitter. And you can see that there's a lot of jitter, obviously because the two workloads affect each other. So if you spin up two optimized VMs on a hypervisor along the NUMMA boundaries and run the workloads in those VMs, they are well isolated and jitters reduced considerably. To be fair, slur on a bare metal node using C-groups does a fairly well job to do the same. So some important issues and pit-forced lessons that we learned regarding the deployment of our HPC payload on top of OpenStack. So Ansible is slow. If we run our slur on Playbook the second time, basically no modification end-to-end, this takes on the production instance an hour. So make sure to tag reoccurrents day-to-day operations like mounting new volumes or changing crosses with Ansible Tech so you can just run those tasks. And also you can use fat images to reduce the install time for the VMs. That is great for package management, but make sure to remove the upstream-centers images after you provision your VMs because otherwise all the versioning is for nothing. And also some issues you will only hit on the scale. So we tested our deployment on a small scale. Everything worked out. We decided to do the production deployment. And then we ran into a nasty Cisco SDN bug that basically forced us to scale in batches after 70 nodes. So be aware of those things that they can happen. So basically having isolated environments is pretty great. But this ends at shared infocomponents, especially if it's tightly integrated with OpenStack. So what do I mean with this? We once deployed the depth environments and we triggered a race condition in the Cisco SDN controller that basically caused an outage in the entire data center. And finally also be aware of unintended consequences of code changes. So once we wanted to scale up our SLAM cluster and we accidentally triggered a redeploy of the entire cluster, because months ago we changed one line in the heat configuration and heat sometimes will reprovision the VMs. And then we ran actually in the scaling issue, so the downtime was much, much longer than we planned for. I think I'm probably out of time. I will leave this, I will end the presentation on this last slide where I show you the bad ugly and the good parts and I will answer any questions you have. All right. Questions? I have actually two questions, first is what test you do before going to test from test to cross? Do you use rally or do you use something like rally or something else? And the second question, what is, how do you move one node from one environment to a different environment? So to the first question, for testing, we test both, so we have a testing for the cloud where we use tempest and rally for performance, but to be aware that the rally or tempest test that basically test your cloud functionality can leave some, you know, can leave some stray projects behind, right? And currently we are not running them. We so far manually test the payload itself. We test the answerable roles using molecule and then we are planning to use, for example, reframe to do some integration testing of also software packages. And the second question was about, yes, so the nice thing if you have a Cisco software defined network, basically it's like going to the switch and replugging the node and you just do this through the Cisco SDI has a rest endpoint and you can talk to the rest endpoint and just basically plug a compute node with the provisioning nick and the HSN, which is the high speed network from one environment, from the deaf environment, for example, to the production environment, and then you just reprovision the bare metal node. That's how we do it. That's pretty useful because we have different nodes and we can test, for example, new configuration firmware updates in depth and then move them back to production. Are you planning to replace Ansible with NSN faster because like one hour is really short? Yes. Well, we haven't really investigated if we can use something which is faster. We will probably have to spend a bit of time to optimize the playbook like with get effects and maybe, I mean, one aspect why this is specifically very long is that we mount, I think, 50 volumes. This is due to some, how we have the storage in the two institutes. We are unifying also the storage system. There will be only one mount point. So just mounting all the 50 volumes on all the 200 compute nodes, even if you do this in parallel, takes half an hour. So there are a couple of things we will do to cut down on the runtime for Ansible. But if anyone has a good suggestion for an alternative, which is agentless, we would be happy to hear about this. What share of your Ansible playbooks and roles did you have to write yourself? And what could you reuse from the existing repositories? So we could use, for example, the entire SLAM role. So we took the SLAM role. I think StackHPC has a couple of roles with which they deploy different HPC clusters. And we use the SLAM role from them a bit modified. For example, we have three partitions and we have three causes. We need to set limits. And SLAM only accepts absolute values for core limits. And we wanted to specify this in percentages. And so we modified this to calculate based on the number of cores, how many, so to calculate the absolute value. So these were a couple of modifications. Some of the small roles like SSSD and other stuff we developed on our own or used something and went from there, basically. Any more questions? All right. And let's thank the speaker.
I'm an HPC systems engineer at the Vienna Bio traditional HPC workloads, but also new workloads traditional workloads, batch scheduling was started to implement this idea. So the idea was not only to consolidate the various compute milestone would be to implement a traditionally all kinds of long-running stateful applications, such as databases and web services. So on a high those would host our different services ranging RStudio. And the idea would be that the Cloud from Cisco with 100G Ethernet. And additionally, have been working on delivering the next the cloud. One is through a web UI and the other with a working cloud. However, this approach back into the tripleO framework and use the production. As any other software, open stack has flavors in the same hypervisor. This is not a big performance, but to be aware that the rally or based on the number of cores, how many, so to