OpenStack Magnum at CERN
Trigazis, Spyros
Magnum is an OpenStack service offering Container Clusters as a service, withsupport for Docker Swarm, Kubernetes, Mesos or DC/OS. Users of the service candeploy clusters of thousands of nodes in minutes, and access them securelyusing their native APIs. In the first part of this talk we introduce the Magnum service. We present itsbasic and more advanced features, including support for heterogeneousclusters, cluster upgrades, cluster drivers and the ongoing effort to make itrock solid. Next, you can get idea if how Magnum is integrated in CERN’s cloudinfrastructure. How it started as a pilot service, the efforts done to addmissing features and how CERN’s users benefit from it. CERN’s use casesinclude end user physics analysis, web services, continuous integration,infrastructure services and more. Finally, such a complex service needs to be tested and benchmarked in depth,so the last section includes the results of a series of tests done in CERNcloud infrastructure and the CNCF testing infrastructure scaling the differentcontainer orchestration engines to thousands of nodes. Specifically, we reportresults of running several thousand containers in Kubernetes and scaling aservice to millions of requests a second.
 Okay. Go ahead and start. Hi everyone. I'm Spiros. I'm a software engineer at CERN. And I'm a developer in open stack magnum. So we'll talk about how we use open stack magnum at CERN and the use cases of containers we have and some scalability tests with our service and Kubernetes. So as i said, we are the open stack magnum. We offer an API service that offers Kubernetes Dockerfarm, mesos and DHS experimental as a service. So with two clicks you can have a cluster running and you can dock directly to the API of what you chose, Dockerfarm, Kubernetes or mesos. So what magnum does is it orchestrates compute instances that can be either VM server metals, it creates networks like tenant networks or public networks and load balancers. It also configures storage for container storage or for persistent storage. We also deploy the certificates you need to have a security service like TLS credentials for ATCD, TLS for the QBPI server. And of course you have the container native API. So if you use Docker, you do Docker run or Docker PS or whatever. And if you use Kubernetes, you use Cubsitl and mesos we use marathon and this U.S. has its own UI and its own API. So magnum mostly focuses on lifecycle operations. So the current available is create delete and scale up and down the cluster and you have more in progress. So this is the architecture of the service. So on the right side of the screen we have the magnum user that creates a cluster which has a special cluster driver. So if you are an operator, you can customize your driver and do modifications of how Kubernetes or Docker is deployed. And the orchestration service which is hit in OpenStack, it creates the cluster and then we pass our scripts with Cloud Meet inside the nodes, master nodes and worker nodes and we play the service. And then on the left side of the screen we have the native API of Kubernetes or Docker so we can use the tools or directly the rest API. So the OpenStack has two releases per year. So the next release is in two weeks and these are the plans that we have for Pyke which is the next release that will be in August. So we want to manage upgrades of clusters. So it's either upgrade Kubernetes only or upgrade Docker underneath or upgrade Docker and we plan to do it like rolling upgrades with node replacement. We also want to support heterogeneous clusters so to create an OpenStack always clusters in different availability zones or with different hardware or with different flavor. For example, have a bunch of big nodes and a bunch of small nodes. Also very soon we will release the Docker Swarm mode which is not available. We will use the legacy swarm for now. We're also working on providing a solution for container monitoring that on deployment time with Prometheus. So I just saw about the operators. We might do it with an operator maybe but it's to monitor Kubernetes itself to monitor something else. And we're going to improve the support for cluster drivers so we can allow different companies with different use cases, modify the drivers and customize the needs and extend a bit our bare metal support which is only limited to Kubernetes for now. So about infrastructure, this is a screen so taking this week. So we run at the moment 60 Magnum clusters if you can see but we have a very big infrastructure so we can create a bunch of them more. So the use cases at CERN. So for you that don't know what CERN is, is that we have a particle accelerator that accelerates particles in nearly the speed of lights and we smash them together and we take pictures of them and we start them as events. So the first use case is batch processing that is distributed system that tries to do event recreation from the metrics that the sensor did. We also have end user analysis with Jupyter notebooks because physicists want to analyze the data and to allow them to do it easier. We have these notebooks so they can do analysis on their browser. Also use cases for machine learning with TensorFlow, deep learning. So physicists are more into that. We just provide infrastructure. We also have infrastructure service, infrastructure management like moving data across the various data centers that used by CERN users and then web service, platform as a service, continuous integration like GitLab CI and many others. So this is a history of Magnum at CERN. We started looking into 2015 and in the beginning of 2016 we had the first pilot service and later last year we opened it to all users. So we modify as I said with cluster drivers a bit the upstream Magnum to support the app services such as CVMFSE use which mounts data from the LHC and we investigate how to do that with system containers with the comics if you want to have a look. So this is how it looks like for a CERN user to use Magnum. We have this public cluster template which is more swarm high availability, Kubernetes high availability and this is the workflow. So you do cluster create, you specify the node counts, you wait a bit depending on how many nodes you want, then you do list and you see that it's create complete. You do uncommadd, create config that fetches all the TLS credentials you want and then you talk to Docker or Kubernetes like you do normally in deployment. So how is this service, how good is this service that we offer? So we did two benchmarks. One is to benchmark the service. So does Magnum, is Magnum able to serve many users? How does it scale? How can it create all these clusters? And the second one is what are these resources good to use? Is the performance good or is it low? So we use the Kubernetes benchmark that the Google Cloud team released that creates some load bots and some HTTP servers that serve the static file and you scale up and down for that. So we did the test in two data centers, one at CERN and one at CNCF cluster in Las Vegas. So our deployment has 240 hypervisors with 13 cores, 100 hypervisors at CNCF. We use similar configuration for Magnum and Hit. Hit is the registration service. At CERN we used our production service so we have more controllers for AbitmQ which is heavily used when you create clusters. But in CNCF we use the upstream Ansible script so you can replicate what we did at CNCF using the Ansible scripts. And we have a bit of a small difference between CERN and CNCF. At CERN we have a flat network so all the VMs are in the same network but at CNCF cluster we had tenant networks. So this is the result at CERN for both tests. On the left is how to benchmark the service, how fast you can create VMs, so clusters. So for the two nodes clusters which are essentially three VMs, one master and two workers, in 2.5 minutes you can have it and we created 50 clusters at the same time. And as you can see the time scale is a bit stable until 100 nodes which is about five minutes but then we had some, we started to see that the scale is linear and for 1000 nodes we did like 25 minutes which is still pretty good. But as we noticed there is still room for improvement. And on the right is the Kubernetes benchmark. So in this example we managed to serve 7 million requests per second with 500 service, 500 engine servers serving static file and 9,500 load bots hammering this service and we had like pretty good latency about, not pretty good but reasonable 15 milliseconds. So about the Kubernetes test we did at CNCF we managed to have similar numbers but we did only, we didn't scale very much just to reach 1 million requests with 100 HTTP servers and 1000 load bots. And the deployment of clusters is very similar. For a small cluster we needed 3 minutes at certain we need 2.5 but then we must refine our ransom scripts to do a better deployment of rabbit MQ so we didn't have exact measurements about how it performed when we created many, many clusters. So we were only able to measure that we created successfully 200 clusters then our benchmark rally broke. So that's it. I hope you like the presentation and thank you. Thank you.
 Okay.  Go ahead and start.  Hi everyone. I'm Spiros. I'm a software engineer at CERN. And I'm a  developer in open stack magnum. So we'll talk about how we use  open stack magnum at CERN and the use cases of containers we  have and some scalability tests with our service and  Kubernetes. So as i said, we are the open stack  magnum. We offer an API service that offers Kubernetes  Dockerfarm, mesos and DHS experimental as a service. So  with two clicks you can have a cluster running and you can  dock directly to the API of what you chose, Dockerfarm,  Kubernetes or mesos. So what magnum does is it orchestrates  compute instances that can be either VM server metals, it  creates networks like tenant networks or public networks  and load balancers. It also configures storage for  container storage or for persistent storage. We also  deploy the certificates you need to have a security service  like TLS credentials for ATCD, TLS for the QBPI server.  And of course you have the container native API. So if you  use Docker, you do Docker run or Docker PS or whatever. And if  you use Kubernetes, you use Cubsitl and mesos we use  marathon and this U.S. has its own UI and its own API. So  magnum mostly focuses on lifecycle operations. So the  current available is create delete and scale up and down the  cluster and you have more in progress. So this is the  architecture of the service. So on the right side of the  screen we have the magnum user that creates a cluster which  has a special cluster driver. So if you are an operator, you  can customize your driver and do modifications of how  Kubernetes or Docker is deployed. And the orchestration  service which is hit in OpenStack, it creates the  cluster and then we pass our scripts with Cloud Meet inside  the nodes, master nodes and worker nodes and we play the  service. And then on the left side of the screen we have the  native API of Kubernetes or Docker so we can use the tools  or directly the rest API. So the OpenStack has two releases  per year. So the next release is in two weeks and these are  the plans that we have for Pyke which is the next release  that will be in August. So we want to manage upgrades of  clusters. So it's either upgrade Kubernetes only or  upgrade Docker underneath or upgrade Docker and we plan to do  it like rolling upgrades with node replacement. We also want  to support heterogeneous clusters so to create an OpenStack  always clusters in different availability zones or with  different hardware or with different flavor. For example,  have a bunch of big nodes and a bunch of small nodes.  Also very soon we will release the Docker Swarm mode which is  not available. We will use the legacy swarm for now. We're  also working on providing a solution for container  monitoring that on deployment time with Prometheus. So I just  saw about the operators. We might do it with an operator  maybe but it's to monitor Kubernetes itself to monitor  something else. And we're going to improve the support for  cluster drivers so we can allow different companies with  different use cases, modify the drivers and customize the  needs and extend a bit our bare metal support which is only  limited to Kubernetes for now. So about infrastructure, this  is a screen so taking this week. So we run at the moment  60 Magnum clusters if you can see but we have a very big  infrastructure so we can create a bunch of them more.  So the use cases at CERN. So for you that don't know what  CERN is, is that we have a particle accelerator that  accelerates particles in nearly the speed of lights and we  smash them together and we take pictures of them and we  start them as events. So the first use case is batch processing  that is distributed system that tries to do event recreation  from the metrics that the sensor did. We also have end user  analysis with Jupyter notebooks because physicists want to  analyze the data and to allow them to do it easier. We have  these notebooks so they can do analysis on their browser.  Also use cases for machine learning with TensorFlow, deep  learning. So physicists are more into that. We just provide  infrastructure. We also have infrastructure service,  infrastructure management like moving data across the various  data centers that used by CERN users and then web service,  platform as a service, continuous integration like  GitLab CI and many others. So this is a history of Magnum  at CERN. We started looking into 2015 and in the beginning  of 2016 we had the first pilot service and later last year  we opened it to all users. So we modify as I said with  cluster drivers a bit the upstream Magnum to support  the app services such as CVMFSE use which mounts data from  the LHC and we investigate how to do that with system  containers with the comics if you want to have a look.  So this is how it looks like for a CERN user to use Magnum.  We have this public cluster template which is more  swarm high availability, Kubernetes high availability  and this is the workflow. So you do cluster create, you specify  the node counts, you wait a bit depending on how many nodes  you want, then you do list and you see that it's  create complete. You do uncommadd, create config that  fetches all the TLS credentials you want and then you  talk to Docker or Kubernetes like you do normally in  deployment. So how is this service, how good is this  service that we offer? So we did two benchmarks.  One is to benchmark the service. So does Magnum, is Magnum  able to serve many users? How does it scale?  How can it create all these clusters? And the second one  is what are these resources good to use? Is the performance  good or is it low? So we use the Kubernetes benchmark  that the Google Cloud team released that creates some  load bots and some HTTP servers that serve the static  file and you scale up and down for that.  So we did the test in two data centers, one at CERN and  one at CNCF cluster in Las Vegas. So our deployment has  240 hypervisors with 13 cores, 100 hypervisors at CNCF.  We use similar configuration for Magnum and Hit.  Hit is the registration service. At CERN we used our  production service so we have more controllers for  AbitmQ which is heavily used when you create clusters.  But in CNCF we use the upstream Ansible script so you  can replicate what we did at CNCF using the Ansible  scripts. And we have a bit of a small difference between  CERN and CNCF. At CERN we have a flat network so all the  VMs are in the same network but at CNCF cluster we had  tenant networks. So this is the result at CERN for both  tests. On the left is how to benchmark the service, how  fast you can create VMs, so clusters.  So for the two nodes clusters which are essentially  three VMs, one master and two workers, in 2.5 minutes  you can have it and we created 50 clusters at the same  time. And as you can see the time scale is a bit  stable until 100 nodes which is about five minutes but  then we had some, we started to see that the scale is  linear and for 1000 nodes we did like 25 minutes which  is still pretty good. But as we noticed there is still  room for improvement. And on the right is the  Kubernetes benchmark. So in this example we managed to  serve 7 million requests per second with 500  service, 500 engine servers serving static file and  9,500 load bots hammering this service and we had  like pretty good latency about, not pretty good but reasonable  15 milliseconds. So about the Kubernetes test we did  at CNCF we managed to have similar numbers but we did  only, we didn't scale very much just to reach 1  million requests with 100 HTTP servers and 1000 load  bots. And the deployment of clusters is very similar.  For a small cluster we needed 3 minutes at certain  we need 2.5 but then we must refine our ransom scripts  to do a better deployment of rabbit MQ so we didn't  have exact measurements about how it performed when  we created many, many clusters. So we were only able  to measure that we created successfully 200 clusters  then our benchmark rally broke. So that's it.  I hope you like the presentation and thank you.  Thank you.
OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_0_59.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_60_107.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_108_911.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_912_2987.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_2988_3143.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_3144_3203.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_3204_4055.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_4056_6047.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_6048_6383.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_6384_7931.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_7932_8783.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_8784_9527.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_9528_10499.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_10500_11747.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_11748_12839.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_12840_13571.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_13572_14771.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_14772_15107.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_15108_15155.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_15156_15203.png OpenStack_Magnum_at_CERN_Scaling_container_clusters_to_thousands_of_nodes_15204_15226.png
Okay. Go ahead and start. Hi everyone. I'm Spiros. I'm a software engineer at CERN. And I'm a developer in open stack magnum. So we'll talk about how we use open stack magnum at CERN and the use cases of containers we have and some scalability tests with our service and Kubernetes. So as i said, we are the open stack magnum. We offer an API service that offers Kubernetes Dockerfarm, mesos and DHS experimental as a service. So with two clicks you can have a cluster running and you can dock directly to the API of what you chose, Dockerfarm, Kubernetes or mesos. So what magnum does is it orchestrates compute instances that can be either VM server metals, it creates networks like tenant networks or public networks and load balancers. It also configures storage for container storage or for persistent storage. We also deploy the certificates you need to have a security service like TLS credentials for ATCD, TLS for the QBPI server. And of course you have the container native API. So if you use Docker, you do Docker run or Docker PS or whatever. And if you use Kubernetes, you use Cubsitl and mesos we use marathon and this U.S. has its own UI and its own API. So magnum mostly focuses on lifecycle operations. So the current available is create delete and scale up and down the cluster and you have more in progress. So this is the architecture of the service. So on the right side of the screen we have the magnum user that creates a cluster which has a special cluster driver. So if you are an operator, you can customize your driver and do modifications of how Kubernetes or Docker is deployed. And the orchestration service which is hit in OpenStack, it creates the cluster and then we pass our scripts with Cloud Meet inside the nodes, master nodes and worker nodes and we play the service. And then on the left side of the screen we have the native API of Kubernetes or Docker so we can use the tools or directly the rest API. So the OpenStack has two releases per year. So the next release is in two weeks and these are the plans that we have for Pyke which is the next release that will be in August. So we want to manage upgrades of clusters. So it's either upgrade Kubernetes only or upgrade Docker underneath or upgrade Docker and we plan to do it like rolling upgrades with node replacement. We also want to support heterogeneous clusters so to create an OpenStack always clusters in different availability zones or with different hardware or with different flavor. For example, have a bunch of big nodes and a bunch of small nodes. Also very soon we will release the Docker Swarm mode which is not available. We will use the legacy swarm for now. We're also working on providing a solution for container monitoring that on deployment time with Prometheus. So I just saw about the operators. We might do it with an operator maybe but it's to monitor Kubernetes itself to monitor something else. And we're going to improve the support for cluster drivers so we can allow different companies with different use cases, modify the drivers and customize the needs and extend a bit our bare metal support which is only limited to Kubernetes for now. So about infrastructure, this is a screen so taking this week. So we run at the moment 60 Magnum clusters if you can see but we have a very big infrastructure so we can create a bunch of them more. So the use cases at CERN. So for you that don't know what CERN is, is that we have a particle accelerator that accelerates particles in nearly the speed of lights and we smash them together and we take pictures of them and we start them as events. So the first use case is batch processing that is distributed system that tries to do event recreation from the metrics that the sensor did. We also have end user analysis with Jupyter notebooks because physicists want to analyze the data and to allow them to do it easier. We have these notebooks so they can do analysis on their browser. Also use cases for machine learning with TensorFlow, deep learning. So physicists are more into that. We just provide infrastructure. We also have infrastructure service, infrastructure management like moving data across the various data centers that used by CERN users and then web service, platform as a service, continuous integration like GitLab CI and many others. So this is a history of Magnum at CERN. We started looking into 2015 and in the beginning of 2016 we had the first pilot service and later last year we opened it to all users. So we modify as I said with cluster drivers a bit the upstream Magnum to support the app services such as CVMFSE use which mounts data from the LHC and we investigate how to do that with system containers with the comics if you want to have a look. So this is how it looks like for a CERN user to use Magnum. We have this public cluster template which is more swarm high availability, Kubernetes high availability and this is the workflow. So you do cluster create, you specify the node counts, you wait a bit depending on how many nodes you want, then you do list and you see that it's create complete. You do uncommadd, create config that fetches all the TLS credentials you want and then you talk to Docker or Kubernetes like you do normally in deployment. So how is this service, how good is this service that we offer? So we did two benchmarks. One is to benchmark the service. So does Magnum, is Magnum able to serve many users? How does it scale? How can it create all these clusters? And the second one is what are these resources good to use? Is the performance good or is it low? So we use the Kubernetes benchmark that the Google Cloud team released that creates some load bots and some HTTP servers that serve the static file and you scale up and down for that. So we did the test in two data centers, one at CERN and one at CNCF cluster in Las Vegas. So our deployment has 240 hypervisors with 13 cores, 100 hypervisors at CNCF. We use similar configuration for Magnum and Hit. Hit is the registration service. At CERN we used our production service so we have more controllers for AbitmQ which is heavily used when you create clusters. But in CNCF we use the upstream Ansible script so you can replicate what we did at CNCF using the Ansible scripts. And we have a bit of a small difference between CERN and CNCF. At CERN we have a flat network so all the VMs are in the same network but at CNCF cluster we had tenant networks. So this is the result at CERN for both tests. On the left is how to benchmark the service, how fast you can create VMs, so clusters. So for the two nodes clusters which are essentially three VMs, one master and two workers, in 2.5 minutes you can have it and we created 50 clusters at the same time. And as you can see the time scale is a bit stable until 100 nodes which is about five minutes but then we had some, we started to see that the scale is linear and for 1000 nodes we did like 25 minutes which is still pretty good. But as we noticed there is still room for improvement. And on the right is the Kubernetes benchmark. So in this example we managed to serve 7 million requests per second with 500 service, 500 engine servers serving static file and 9,500 load bots hammering this service and we had like pretty good latency about, not pretty good but reasonable 15 milliseconds. So about the Kubernetes test we did at CNCF we managed to have similar numbers but we did only, we didn't scale very much just to reach 1 million requests with 100 HTTP servers and 1000 load bots. And the deployment of clusters is very similar. For a small cluster we needed 3 minutes at certain we need 2.5 but then we must refine our ransom scripts to do a better deployment of rabbit MQ so we didn't have exact measurements about how it performed when we created many, many clusters. So we were only able to measure that we created successfully 200 clusters then our benchmark rally broke. So that's it. I hope you like the presentation and thank you. Thank you.
the use cases of containers we have and some cluster running and you can dock directly to the have more in progress. So this is the of the screen we have the magnum user that orchestration service which is hit in OpenStack, nodes and worker nodes and we play the service. the native API of Kubernetes or Docker so we can that will be in August. So we want to manage support heterogeneous clusters so to create an very soon we will release the Docker Swarm mode to improve the support for cluster drivers so we cases, modify the drivers and customize the needs infrastructure so we can create a bunch of them that the sensor did. We also have end user they can do analysis on their browser. Also use platform as a service, continuous integration first pilot service and later last year we opened support the app services such as CVMFSE use which if you want to have a look. So this is how it you talk to Docker or Kubernetes like you do So does Magnum, is Magnum able to serve many users? How does it scale? How can it create all workers, in 2.5 minutes you can have it and we some, we started to see that the scale is linear managed to serve 7 million requests per second