Orchestrating Docker containers
Castelli, Flavio
As more and more users are starting to consider Docker in production environments, people have realized that having Docker alone is not enough. Instead, the community is gearing towards orchestration solutions: tools, frameworks and practices that deal with how containers are deployed on production and how administrators can monitor all this without going crazy. In this talk we are going to describe and detail the orchestration solution that we decided to support in SUSE, why we did it and how our users will benefit from it.
 Okay. So, hello everyone. I'm Flavio Castelli. I'm a Susan employee. I'm part of the Contairs team at SUSE. Here we do fancy stuff like Docker. We care about Docker both inside of Open SUSE and inside of SUSE. We take care of the official images that we have inside of Docker for Open SUSE. We take care of the images that we have. And now we care about other projects like portals or Zipperdocker that are going to be covered during this conference as well. And now we are entering into a new fancy landscape and we are going to take care also about orchestration. So, let's start with application containers. I'm going to assume you know about them at least to a certain degree. So, they are lightweight. So you can run a lot of them on a single machine. The containers are all these little boxes over there. The big green box is the main OS. So it's green. It's Open SUSE. Given they are lightweight, you can have lots of them on the single machine. They are also composable. You are encouraged to put them together in order to build stuff. So you can combine two of them to create an application. You can combine three of them. This is the whole microservice thing that is such a hype nowadays. But if you want to get serious, running everything on the same host is not going to work because it's really prone to failures. It's not going to scale. So you want to do multi-host deployments of containers. But how can you go from a situation where you have everything running on the same machine to managing a large cluster of hosts where you spread containers over them? So there are quite some challenges. So first of all, you have to figure out where each container should be placed on which host. And two containers stay on the same host? Maybe yes, maybe no, because of different reasons. So you have to also monitor the containers, make sure that they keep running, make sure that you monitor also the host. So if a host goes down, you react to this kind of failures. And of course, you have to handle those failures because you cannot let your container die. There are also interesting topics that you probably already faced when running containers on a single machine. So service discovery is one of those topics. But things get even more interesting when you start to run containers spread over different machines. You have also to figure out how to expose the service that are running inside of your containers to external entities like your customers. And of course, you have to handle secrets. It would be nice to handle those credentials, those certificates, all SSI keys and such. And of course, various data persistence. So you can go on your own and try to sort everything out by implementing stuff from scratch. Good luck with that. Otherwise, you can choose an orchestration container and work with that. Why I recommend to use an orchestration container? Well, first of all, you will enter this new, let's say, way of thinking about the deployment. You will stop to care about where a container is running. You will leave this task to the orchestration container. You will stop to care about containers failing. You want to receive a message on your page or whatsoever. And you won't have to SSH into machine to restart that manually or to instruct something to do that. The orchestration engine is going to take care of that as well. So you have just to do one thing. This thing is to declare how things should look like. This is called a desired state. All the orchestration engines, they have this concept of reconciliation. So you start by describing how things should look like. Like, I want to have five instances of this container running my web application running. At the same time, I want five of them. And then you just forget about that as a user. Behind the scene, the orchestration engine will look at what is the current state. Like, there are zero of those containers running. It will compute the actions that are required in order to reach this desired state. And then it will execute all those actions in order to reconcile the current state and to make your desired state happen. And this is a constant loop going on and on. So if one container dies, then the current state is different from the desired state. And henceforth, it will just fix itself. So they are really nice. But which one should you pick? There is quite some choice. So there is Kubernetes, there is Docker's war, there is Mizzus, there is Nomad, there is Lettuce. There are lots of them. So each one of them has its advantages, disadvantages, peculiarities, and such. So at SUSE, we had to figure out which one we would like to focus on, which one is very recommended by us. So our approach was simple. So as a team, we started looking to each one of those orchestration engines. We designed a use case. And then we used the same approach. We evaluated each orchestration engine trying to implement this scenario, which is a common scenario. We tried then to break the orchestration engine to see how reliable it is. And of course, while doing everything, we tried to make sure that we could reuse what we did in terms of open sourcing the stuff we discovered or the specific scenario. So talking about the scenario, we wanted to do something more than deploying a simple web application. So we focused on Portus. Portus is an open source project developed by us. It's a user interface and user interface and authorization service for the Docker registry. There is going to be a talk about Portus in the next days, I think on Sunday or Saturday. I highly recommend you to attend it. It's a really cool project. So it's a web application, technically speaking, which uses a database to store its data. So we have also to run MariaDB. And it's also interacting with the registry and the registry as well as to store its data. So this is not so easy to deploy because we have to keep track of the data. So we need stateful containers while the whole microservice thing is advocating for stateless containers. And Portus is using some crypto stuff. So we had to handle out certificates, we have to handle keys, we had to handle traditional secrets like the DB credentials. So that was not so easy and typical. So after quite some testing and experimenting, we made our decision and our decision is to go with Kubernetes. And this is what I'm going to talk during very many part of the talk. So Kubernetes abbreviated K8S, like internationalization, you know. So it's a project which has been created by Google. It's based on 10 years that Google had running containers in production. It's based on a paper called About Borg, which is the system that was in place inside of Google. So it's based on a really solid background. It has a really good design, of course. The guys from Google know how to run stuff and how to run better skills. So we trust them. It immediately attracted a lot of people. So it has a huge community, which is really active. It's really differentiated in terms of members. It's a project that is not controlled by Google. In fact, it's now donated to the cloud-native computing foundation, which is a foundation underneath Linux Foundation umbrella. So it's pretty famous. So there is a lot of interest from lots of people in using Kubernetes. And what is most important, it's an opinionated solution, which means that it has an answer to basically all the questions you might have. This can be a bad thing for some people, because maybe you want to be free to choose whatever they want. But it's also a really good thing for most of the person, I think, because basically, if you have to solve the problem like service discovery or persistent volume with some orchestration engines, you're up to your own. You have to look at, I don't know, 10 open source projects in order to figure out which one you want to trust, which one is going to work for you. And you have to make sure that it's going to keep working, because they are different projects collaborating, kind of collaborating together. So at each release, you might risk that something is not working properly. Kubernetes has everything inside of it, but it's still swappable. So there are some components that you can replace with whatever you want, which make it flexible. But at the same time, giving everything is inside of it, you have this trust in it that it's not going to break during each release. So the architecture of Kubernetes is following one. So you have two types of machines inside of the Kubernetes cluster. You have the master and you have the minions. The minions is where the containers are going to actually be running. The master instead is where all the decisions are made. As you can see, there is also there was yellow circles at CD. So at CD is a distributed key value storage, which is used to hold all the data about Kubernetes stuff. So the master is going to store and read data inside of it. The master basically has three demons running. There is the API server, which is your entry point into the cluster. This is how you interact with it. But it's also being used by all the other pieces to coordinate themselves, to get information, to react to changes and such. You have the scheduler, who is the one in charge to figure out, for example, where a certain container is going to be placed. You have the controller manager that is the component that constantly looks at the current state, figures out if it's different from your desired state and then computes the actions that needs to be done. Like, I want a new container to be running and then the scheduler will say, okay, this container has to be placed on this machine. And then on the minion, you have the KubeNet process, which will be aware about AF to run a container on my machine. So it will take care of running this container. To do that, the KubeNet will interact with some container engine. In this case, you can see there is Docker, but you can replace Docker with other container engines like Rocket, for example, which is an alternative container engine from CoreOS. And using this container engine, the KubeNet will create the pods. So the pod is a concept of Kubernetes, which is unique to it. Basically, you can decide that different containers that have a strong relationship, a strong bond that can be placed inside of the pod. A pod will remove some of the isolation features that are typical of containers. So for example, a peculiar thing, all the containers inside of the pod, they share the same network namespace, meaning they will have the same network card, they will have the same IP address. If a typical use case for that, for example, let's say that you have a web application like Ruby application. So you have, I don't know, Puma as a web service for it, but you don't want to expose Puma to the internet traffic. What you usually do, it's really typical, you put something in front of it, like engine X or Apache at a no. So you can have a container running your web application with Puma, and then you can have another container inside of the same pod, which is the official engine X container. And then you have the engine X container, which is going to be a balancer for the Puma, which is going to be listening on localos, because from a networking point of view, everything inside of the pod is on the same network interface. So you can easily scale up the pod without having to duplicate too much stuff. The last process of the minion is the Q proxy. This is required, as we will see later, to handle networking, which is a really complicated topic. It's worth a dedicated talk, but I will try to address that later on, but if you have more questions in the end, you're free to ask. So let's look at a concrete example. So I will focus on using Kubernetes. So I will focus as being a user who wants to run stuff inside of Kubernetes. I want to just skip the part about bringing up Kubernetes and administering it. I want to focus on a user. A user who wants to run a simple application. So we have a web application, which is using a database to store its data. So we could have a web container running on host one, and a database container running on host number two. So what we want is to react to failures. So if the host goes down, then of course, it means the database container will go down as well. We want Kubernetes to react to this event by bringing up a new database instance somewhere else, like on OS 3. So this is implemented in Kubernetes using a replication controller or there is also replication set. Basically, they allow you to describe the state. Say I want to have five to one replicas running at the same time. So if this number changes in the current state, Kubernetes will take care of bringing everything back to what you specified. So let's take a quick look. I will go back and forth. So right now here, can you read it? Is it big enough? Okay. So I have a Kubernetes cluster running on my machine. It's composed by three nodes. Just ignore this one. Everything is running on Leap so far. So we have three Minion nodes, and we are already running our web application and the database. So here you can see that I'm using kubectl, which is a common line tool to interact with Kubernetes. I'm getting the list of all the parts which are running. This is in real time. As you can see, I have one of them running on Minion zero, one of them running on Minion two. So now if I SSH into Minion zero, if I do a Docker PS, I can see that, well, can you still see that because it's at the bottom of the screen? So it's running my container here. This is the gas book application. This is the idea of the container. So if I kill the container manually, as you can see, Kubernetes immediately noticed that the gas book pod was broken. It was not working properly, and it immediately fixed that. So my application is still up and running. I could bring down the node, and of course that would trigger the move of the container somewhere else. But let's delete the pod instead. As you can see here, this one over one means that I want to have always one pod providing this gas book application running. So if I delete it, as you can see, this is the violation of my desired state. So a new pod is created this time on a different node, on Minion one. So as you can see, it's pretty robust. It's working. You might think we're done with that. That's not true. So let me introduce you to the survey discovery problem. So let's define two actors. You have a producer, which is a container which has a service running inside of it, like the database in our case, and you have a consumer, which is someone that is using this service. So in our case, it's the web application. So when you start the web application container, you have to find where the database is running. Now, according to what kind of network setup you have, you might need to know two kind of information, like the IP address of the database, and maybe even the port number. With Kubernetes, you don't need the port, but with our solution, you do. So how do you figure out where something is running? You might be tempted to use the DNS, like you usually do with normal stuff, but this is not going to work because there are the DNS implementation on the client side. Some of them are buggy, and they keep cache. They don't respect the time to expire stuff. So things are not going to work well in the containerized word, because you have to imagine that stuff keeps moving from one place to another. The IP address keeps changing and such. So the only way to do that is to use distributed key value storage, like console, LCD, or zookeeper, and then register the information about the producer inside of it, and then make this information available to all the containers that are interested in that. Usually this happens by setting some environment variables and then reading those environment variables back. So there are still some changes, of course. Even if you get everything to work when you start the web container, how are you going to handle a sudden move of the producer? So like it happened before, if I had killed the Mongo database, that would have moved to another place, and it would have another IP address. But the gas book application would be still looking for the Mongo database to the old IP address. So how can you react to these sudden changes? And how do you face multiple choices? You might have different backend services that you are consuming that are providing the same service. How can you choose between all of them? So the solution is to use a load balancer. You could be placing a load balancer in front of all the producers and then point the consumer to the load balancer. This is going to work. However, this is not so easy as it might seem. So first of all, you have to make the load balancer aware of all these containers. So to read the configuration changes from this key value storage and then generate a new configuration, reload the configuration. This is going to, depending on the load balancer, that can have some troubles when things keep changing and you keep reloading the configuration. So that load balancer can be unresponsive. So this is not really a good solution. And still, this is going to be the single point of failure of the whole thing because you have just one load balancer. Of course, you could place more of them. But what is the solution? With Kubernetes, we have this concept of service. Outside of Kubernetes, you are on your own. You have to figure out how to do stuff. Kubernetes has this concept of service. So with the service, you basically, well, you point the consumer to the service and the service is going to put the consumer in touch with the producer. So basically, the whole service is like a switchboard operator. And it's going to take care of making all the information flow to the right place. The nice thing is that it can also be used when you have to expose a service to the outside, like to the Internet. So you can have two types of services. You can have the internal ones, which are not visible outside of the cluster, like the database service. But you can also have services which are exposed to the public. So you can point all the Internet traffic, and this is going to flow straight to the right web container, regardless of where this is running. How is this done? So Kubernetes does that by defining a virtual IP address for each service. This IP address is created once, and it will always stay the same. It won't change at all. That means that, for example, using a nod on Kubernetes, you can have an internal DNS, which is not going to break. It's not affected by all these issues that I mentioned before. So you can have a DNS entry like, I don't know, mongo.kubernetes, that is going to be resolved to this virtual IP address. And then there is an internal log balancer, which is running basically on each node, which is Qproxy, which uses IP tables to route the connection to the right pod, providing the service. This is for the internal services. For the published ones, they are like the internal services, so it's done in the same way. Then each node, each minion inside of your cluster is going to have these services exposed on a fixed port. So now you just have to take your load balancers and point them to all the nodes of your cluster and to the same port on each one of them in order to expose the service. If you are running Kubernetes on some public cloud like AWS, there are even backends, which are already implemented, so you can reuse all the load balancer that are already provided by AWS in order to maintain even part of the infrastructure. So let me show you how this is going to work. So here we have our guest book. So we can write messages in there. These are stored inside of MongoDB. So as you can see, I have the Mongo database running on minion number two, the guest book running on minion number one. So now I'm going to force the move of the Mongo database to another node. So now the database is running on minion number zero. If I go there and I reload, as you can see, it's working. So the application is still reading data from the database, but as you may notice, something happened, like we lost all the messages, which brings us to the next problem, which is data persistency. So yeah, you can create 12 factors applications and respect all the manifesto and store data outside of it, but sometimes you still have to deal with state full containers. So we have to make sure that the data is persistent. If the container is moved to another host, or if it's a restart or whatsoever, the data is not lost. So with other solutions, like other orchestration engines, you are on your own. As I said in the beginning, there are, I don't know, 10 projects, 10 open source projects that try to address this problem. You try to find the one which works and you integrate that. But with Kubernetes, you have this concept of volumes. The nice thing is really nice. It's being built into Kubernetes. It just works out of the box, but it's also part of the grand design of Kubernetes, meaning that a volume is like any other type of resource. So it's taken into consideration by the scheduler when deciding where something has to be run or if something can even be run. There is the concept of, well, there is the administrator, the operator of Kubernetes is going to add some volumes to the cluster. Each volume is exposed as a persistent volume. And then as a user, you just have to write a claim, a persistent volume claim, meaning that you need your application to have, I don't know, two gigabytes of space and you want this space to be yours only. You don't want to have our application interacting with that, because for example, you might be fine having our application interacting with this volume. And then the scheduler will look at the persistent volumes that are available and will try to find a free one that can satisfy your claim. Then if it finds that, then it's fine. Your pod will be started. Otherwise, it will stay here until this condition is met. This is really nice. There are also different drivers available for you. So an FS, Sender, Glaster, FS, whatever, you can decide how to back up all, what to use in the backend of these volumes. It's really nice. So let's try to see how it works. So in the beginning, right now, as you see, there is no data persistency. Here, I defined already a volume, which is called Mongo Storage. It has two gigabytes of space and it's available, as you can see. Behind the scene, this is implemented using NFS, because it was pretty cheap for me to have everything running. So now, I have to change the definition I have for the Mongo Pod. So basically, I have to define... Well, this is... I didn't want to focus too much on that, but this is how you define stuff in Kubernetes. You have these YAML files, which you can use to define all your types of resources, and then you can consume them. The point is here, I'm going to say that I want to mount a volume under dataDB. And here, I define the volume claim. The volume claim, as some requirements. So I need the storage to offer me at least one gigabyte of space. And I'm fine with sharing this storage with someone else. So read, write, maybe this is a policy. I have to apply those changes to my pod. By applying those changes, I had to recreate the pod. So as you can see, the old one has been terminated. A new one is running. So the persistent volumes changed. The Mongo storage is no longer available. Now it's bound. And it's bound to this claim, which you can see here, persistent volume claim. It's called Mongo NFS. It's bound to the Mongo storage, and everything is fine. So now, let's go there. If I reload, everything is empty, of course, because I restarted everything. Now, let's create a message. Let's go back here. And let's delete again the pod running MongoDB. So it has been destroyed. Now it's running on Minion number zero. Let's reload the page. We have to wait for MongoDB to be warm enough. So yeah, as you can see, the data is still around. So this is how you can add data persistency in a really, really easy way. So even legacy application can keep working inside of Kubernetes with not many changes. One last thing is about Docker's worm. So I have been talking all the time about Kubernetes, but there is also Docker's worm out there. What about it? Well, things are a bit complicated. Before Docker, 1.12, which is in RC1 right now, I think. Swarm was a separate project. It was something that you can, could use on top of Docker. It has, it had nice concept. Basically, it was exposing the same API of Docker engine, but it had a couple of extra endpoints. By reusing the same API, it was possible to point any kind of tool written for the Docker engine against it and just use it. And that was pretty smart. It was possible to do some nice things. It was really, really easy to use worm. It was really easy to transition from using Docker on one machine to use worm. However, it was a really, really simple orchestrator. It didn't have a com set of desired state. It didn't have a com set of service. It didn't have a lot of things. The scheduling algorithms weren't so good. So that's one of the reasons why we just didn't decide to go for it. However, with 1.12, or since last Monday, they created the basically merged swarm into the Docker engine itself. So now the swarm project doesn't longer exist. There is the swarm kit project, which is a generic framework, but you can reuse to build orchestrators. This framework has been incorporated into the Docker engine. So now you don't have to add any extra tool. Everything comes out of the box. And they made a lot of changes compared to swarm. It's a huge step forward. They took inspiration from different orchestration engines like Kubernetes. They copied some concepts from it, which is great because now, while it's still really, really easy to deploy, you get a lot of things that were missing the past, like the desired state thing, the service discovery. They have an integrated load balancer using IPvS, which is something built into the kernel, which works really, really well. So it's really cool. However, there are still some things that are left to the user. So if you want data persistency, then you have to figure out how to do that. There are different plugins, like I said in the beginning. If you want to handle secrets, you are on your own. Well, for credentials, you could use environment variables. But for certificates and other things, you have to come up with something on your own. There are some operations that now are harder than before, like getting blocks from a running container on Kubernetes, for example, if I had to debug something, I could do something like that. Sorry. I could do get logs. By the way, right now, I'm running Kube City on my laptop, and it's just pointed to a remote server, which is really nice. So I can get logs. These are the logs in real time, if I want, of the pod. Let's say that I have an application which has some trouble, and I have to debug that live on production. So I can spawn a shell inside of, I can look into the container, basically. So by doing Kube City, I can get a terminal running inside of the container. As you can see, I'm really inside of that container, because bit number one is the Puma process. This is really, really nice to debug stuff. And if that was possible with Swarm as well, you could have done a Docker exec, a Docker logs, and it would have worked. Now that they moved to this new model, this is no longer possible, you have to figure out where the container that you want to debug is running in terms of the host. Then you have to SSH into that host, and then use the Docker engine which is running inside of it. So it's not so comfortable. Another thing, another drawback, is of course it's tied to the Docker engine. While with other orchestration engines like Kubernetes or Nomad, or even Mizzus, you can decide what kind of orchestration engine, sorry, container engine you want to use. Of course, it's in the early stage. They just announced that, so it will get better and better over the time. But I think it targets a different use case. With Kubernetes, you can do fancy stuff like isolate different tenants inside of the same cluster from a point, from a networking point of view if you want, or have resource control over them, like set quotas to each one of them. You can have a tire integration with other stuff, like, for example, if you run bet on top of an open stack, you could do auto scaling based on, even on the cloud itself, based on the load inside of the cluster. You can do quite some stuff. It's a different use case, I think. So with that, I think I'm done with early advance. I kind of rushed during it because I introduced different slides compared to the last time I did the talk. So do you have any questions about that? Have you ever used orchestration engines before? Okay. Containers? Okay, good. Okay, so if there's nothing, I think we're done. Thanks. Thank you.
 Okay. So, hello everyone. I'm Flavio Castelli. I'm a Susan employee. I'm part of the Contairs  team at SUSE. Here we do fancy stuff like Docker. We care about Docker both inside of  Open SUSE and inside of SUSE. We take care of the official images that we have inside  of Docker for Open SUSE. We take care of the images that we have. And now we care about  other projects like portals or Zipperdocker that are going to be covered during this  conference as well. And now we are entering into a new fancy landscape and we are going  to take care also about orchestration. So, let's start with application containers.  I'm going to assume you know about them at least to a certain degree. So, they are lightweight.  So you can run a lot of them on a single machine. The containers are all these little boxes  over there. The big green box is the main OS. So it's green. It's Open SUSE. Given they  are lightweight, you can have lots of them on the single machine. They are also composable.  You are encouraged to put them together in order to build stuff. So you can combine two  of them to create an application. You can combine three of them. This is the whole microservice  thing that is such a hype nowadays. But if you want to get serious, running everything  on the same host is not going to work because it's really prone to failures. It's not going  to scale. So you want to do multi-host deployments of containers. But how can you go from a situation  where you have everything running on the same machine to managing a large cluster of hosts  where you spread containers over them? So there are quite some challenges. So first  of all, you have to figure out where each container should be placed on which host.  And two containers stay on the same host? Maybe yes, maybe no, because of different reasons.  So you have to also monitor the containers, make sure that they keep running, make sure  that you monitor also the host. So if a host goes down, you react to this kind of failures.  And of course, you have to handle those failures because you cannot let your container die.  There are also interesting topics that you probably already faced when running containers  on a single machine. So service discovery is one of those topics. But things get even  more interesting when you start to run containers spread over different machines. You have also  to figure out how to expose the service that are running inside of your containers to external  entities like your customers. And of course, you have to handle secrets. It would be nice  to handle those credentials, those certificates, all SSI keys and such. And of course, various  data persistence. So you can go on your own and try to sort everything out by implementing  stuff from scratch. Good luck with that. Otherwise, you can choose an orchestration container  and work with that. Why I recommend to use an orchestration container? Well, first of  all, you will enter this new, let's say, way of thinking about the deployment. You will  stop to care about where a container is running. You will leave this task to the orchestration  container. You will stop to care about containers failing. You want to receive a message on  your page or whatsoever. And you won't have to SSH into machine to restart that manually  or to instruct something to do that. The orchestration engine is going to take care of that as well.  So you have just to do one thing. This thing is to declare how things should look like.  This is called a desired state. All the orchestration engines, they have this concept of reconciliation.  So you start by describing how things should look like. Like, I want to have five instances  of this container running my web application running. At the same time, I want five of  them. And then you just forget about that as a user. Behind the scene, the orchestration  engine will look at what is the current state. Like, there are zero of those containers running.  It will compute the actions that are required in order to reach this desired state. And then  it will execute all those actions in order to reconcile the current state and to make  your desired state happen. And this is a constant loop going on and on. So if one container dies,  then the current state is different from the desired state. And henceforth, it will just  fix itself. So they are really nice. But which one should you pick? There is quite some choice.  So there is Kubernetes, there is Docker's war, there is Mizzus, there is Nomad, there is Lettuce.  There are lots of them. So each one of them has its advantages, disadvantages, peculiarities,  and such. So at SUSE, we had to figure out which one we would like to focus on, which one is  very recommended by us. So our approach was simple. So as a team, we started looking to each one of  those orchestration engines. We designed a use case. And then we used the same approach. We  evaluated each orchestration engine trying to implement this scenario, which is a common scenario.  We tried then to break the orchestration engine to see how reliable it is. And of course,  while doing everything, we tried to make sure that we could reuse what we did in terms of  open sourcing the stuff we discovered or the specific scenario. So talking about the scenario,  we wanted to do something more than deploying a simple web application. So we focused on Portus.  Portus is an open source project developed by us. It's a user interface and user interface and  authorization service for the Docker registry. There is going to be a talk about Portus in the  next days, I think on Sunday or Saturday. I highly recommend you to attend it. It's a really cool  project. So it's a web application, technically speaking, which uses a database to store its  data. So we have also to run MariaDB. And it's also interacting with the registry and the registry  as well as to store its data. So this is not so easy to deploy because we have to keep track of  the data. So we need stateful containers while the whole microservice thing is advocating for  stateless containers. And Portus is using some crypto stuff. So we had to handle out certificates,  we have to handle keys, we had to handle traditional secrets like the DB credentials.  So that was not so easy and typical. So after quite some testing and experimenting,  we made our decision and our decision is to go with Kubernetes. And this is what I'm going to talk  during very many part of the talk. So Kubernetes abbreviated K8S, like internationalization,  you know. So it's a project which has been created by Google. It's based on 10 years that Google  had running containers in production. It's based on a paper called About Borg, which is the system  that was in place inside of Google. So it's based on a really solid background. It has a really good  design, of course. The guys from Google know how to run stuff and how to run better skills. So we  trust them. It immediately attracted a lot of people. So it has a huge community, which is really  active. It's really differentiated in terms of members. It's a project that is not controlled  by Google. In fact, it's now donated to the cloud-native computing foundation, which is a foundation  underneath Linux Foundation umbrella. So it's pretty famous. So there is a lot of interest  from lots of people in using Kubernetes. And what is most important, it's an opinionated solution,  which means that it has an answer to basically all the questions you might have. This can be a  bad thing for some people, because maybe you want to be free to choose whatever they want.  But it's also a really good thing for most of the person, I think, because basically, if you have  to solve the problem like service discovery or persistent volume with some orchestration engines,  you're up to your own. You have to look at, I don't know, 10 open source projects in order to  figure out which one you want to trust, which one is going to work for you. And you have to make sure  that it's going to keep working, because they are different projects collaborating, kind of  collaborating together. So at each release, you might risk that something is not working properly.  Kubernetes has everything inside of it, but it's still swappable. So there are some components  that you can replace with whatever you want, which make it flexible. But at the same time,  giving everything is inside of it, you have this trust in it that it's not going to break  during each release. So the architecture of Kubernetes is following one. So you have two  types of machines inside of the Kubernetes cluster. You have the master and you have the minions.  The minions is where the containers are going to actually be running. The master instead is  where all the decisions are made. As you can see, there is also there was yellow circles at CD.  So at CD is a distributed key value storage, which is used to hold all the data about  Kubernetes stuff. So the master is going to store and read data inside of it. The master  basically has three demons running. There is the API server, which is your entry point into the  cluster. This is how you interact with it. But it's also being used by all the other pieces to  coordinate themselves, to get information, to react to changes and such. You have the scheduler,  who is the one in charge to figure out, for example, where a certain container is going to be placed.  You have the controller manager that is the component that constantly looks at the current  state, figures out if it's different from your desired state and then computes the actions  that needs to be done. Like, I want a new container to be running and then the scheduler will say,  okay, this container has to be placed on this machine. And then on the minion, you have the  KubeNet process, which will be aware about AF to run a container on my machine. So it will take  care of running this container. To do that, the KubeNet will interact with some container engine.  In this case, you can see there is Docker, but you can replace Docker with other container  engines like Rocket, for example, which is an alternative container engine from CoreOS.  And using this container engine, the KubeNet will create the pods. So the pod is a concept of  Kubernetes, which is unique to it. Basically, you can decide that different containers that have a  strong relationship, a strong bond that can be placed inside of the pod. A pod will remove some  of the isolation features that are typical of containers. So for example, a peculiar thing,  all the containers inside of the pod, they share the same network namespace, meaning they will have  the same network card, they will have the same IP address. If a typical use case for that, for  example, let's say that you have a web application like Ruby application. So you have, I don't know,  Puma as a web service for it, but you don't want to expose Puma to the internet traffic.  What you usually do, it's really typical, you put something in front of it, like engine X or Apache  at a no. So you can have a container running your web application with Puma, and then you can have  another container inside of the same pod, which is the official engine X container. And then you  have the engine X container, which is going to be a balancer for the Puma, which is going to be  listening on localos, because from a networking point of view, everything inside of the pod is on  the same network interface. So you can easily scale up the pod without having to duplicate  too much stuff. The last process of the minion is the Q proxy. This is required, as we will see  later, to handle networking, which is a really complicated topic. It's worth a dedicated talk,  but I will try to address that later on, but if you have more questions in the end, you're free to  ask. So let's look at a concrete example. So I will focus on using Kubernetes. So I will focus  as being a user who wants to run stuff inside of Kubernetes. I want to just skip the part about  bringing up Kubernetes and administering it. I want to focus on a user. A user who wants to run  a simple application. So we have a web application, which is using a database to store its data.  So we could have a web container running on host one, and a database container running on host  number two. So what we want is to react to failures. So if the host goes down, then of course,  it means the database container will go down as well. We want Kubernetes to react to this event  by bringing up a new database instance somewhere else, like on OS 3. So this is implemented in  Kubernetes using a replication controller or there is also replication set. Basically, they  allow you to describe the state. Say I want to have five to one replicas running at the same time.  So if this number changes in the current state, Kubernetes will take care of bringing everything  back to what you specified. So let's take a quick look. I will go back and forth. So right now here,  can you read it? Is it big enough? Okay. So I have a Kubernetes cluster running on my machine.  It's composed by three nodes. Just ignore this one. Everything is running on Leap so far.  So we have three Minion nodes, and we are already running our web application and the database.  So here you can see that I'm using kubectl, which is a common line tool to interact with Kubernetes.  I'm getting the list of all the parts which are running. This is in real time. As you can see,  I have one of them running on Minion zero, one of them running on Minion two. So now if I SSH  into Minion zero, if I do a Docker PS, I can see that, well, can you still see that because it's  at the bottom of the screen? So it's running my container here. This is the gas book application.  This is the idea of the container. So if I kill the container manually, as you can see,  Kubernetes immediately noticed that the gas book pod was broken. It was not working properly,  and it immediately fixed that. So my application is still up and running. I could bring down the  node, and of course that would trigger the move of the container somewhere else. But  let's delete the pod instead. As you can see here, this one over one means that I want to have always  one pod providing this gas book application running. So if I delete it, as you can see,  this is the violation of my desired state. So a new pod is created this time on a different node,  on Minion one. So as you can see, it's pretty robust. It's working. You might think we're done  with that. That's not true. So let me introduce you to the survey discovery problem. So let's define  two actors. You have a producer, which is a container which has a service running inside of it,  like the database in our case, and you have a consumer, which is someone that is using this  service. So in our case, it's the web application. So when you start the web application container,  you have to find where the database is running. Now, according to what kind of network setup you have,  you might need to know two kind of information, like the IP address of the database, and maybe  even the port number. With Kubernetes, you don't need the port, but with our solution, you do.  So how do you figure out where something is running? You might be tempted to use the DNS,  like you usually do with normal stuff, but this is not going to work because there are the DNS  implementation on the client side. Some of them are buggy, and they keep cache. They don't respect  the time to expire stuff. So things are not going to work well in the containerized word, because  you have to imagine that stuff keeps moving from one place to another. The IP address keeps changing  and such. So the only way to do that is to use distributed key value storage, like console,  LCD, or zookeeper, and then register the information about the producer inside of it, and then make  this information available to all the containers that are interested in that. Usually this happens  by setting some environment variables and then reading those environment variables back.  So there are still some changes, of course. Even if you get everything to work when you start the  web container, how are you going to handle a sudden move of the producer? So like it happened  before, if I had killed the Mongo database, that would have moved to another place, and it would  have another IP address. But the gas book application would be still looking for the Mongo  database to the old IP address. So how can you react to these sudden changes? And how do you  face multiple choices? You might have different backend services that you are consuming that  are providing the same service. How can you choose between all of them? So the solution is to use a  load balancer. You could be placing a load balancer in front of all the producers and then  point the consumer to the load balancer. This is going to work. However, this is not so easy as it  might seem. So first of all, you have to make the load balancer aware of all these containers. So to  read the configuration changes from this key value storage and then generate a new configuration,  reload the configuration. This is going to, depending on the load balancer, that can have  some troubles when things keep changing and you keep reloading the configuration. So that load  balancer can be unresponsive. So this is not really a good solution. And still, this is going to be  the single point of failure of the whole thing because you have just one load balancer. Of course,  you could place more of them. But what is the solution? With Kubernetes, we have this concept  of service. Outside of Kubernetes, you are on your own. You have to figure out how to do stuff.  Kubernetes has this concept of service. So with the service, you basically, well, you point the  consumer to the service and the service is going to put the consumer in touch with the producer.  So basically, the whole service is like a switchboard operator. And it's going to take care of  making all the information flow to the right place. The nice thing is that it can also be used  when you have to expose a service to the outside, like to the Internet. So you can have two types  of services. You can have the internal ones, which are not visible outside of the cluster,  like the database service. But you can also have services which are exposed to the public. So you  can point all the Internet traffic, and this is going to flow straight to the right web container,  regardless of where this is running. How is this done? So Kubernetes does that by defining a virtual  IP address for each service. This IP address is created once, and it will always stay the same.  It won't change at all. That means that, for example, using a nod on Kubernetes, you can have an  internal DNS, which is not going to break. It's not affected by all these issues that I mentioned  before. So you can have a DNS entry like, I don't know, mongo.kubernetes, that is going to be  resolved to this virtual IP address. And then there is an internal log balancer, which is running  basically on each node, which is Qproxy, which uses IP tables to route the connection to the  right pod, providing the service. This is for the internal services. For the published ones,  they are like the internal services, so it's done in the same way. Then each node, each minion  inside of your cluster is going to have these services exposed on a fixed port. So now you just  have to take your load balancers and point them to all the nodes of your cluster and to the same  port on each one of them in order to expose the service. If you are running Kubernetes on some  public cloud like AWS, there are even backends, which are already implemented, so you can reuse  all the load balancer that are already provided by AWS in order to maintain even part of the  infrastructure. So let me show you how this is going to work. So here we have our guest book.  So we can write messages in there. These are stored inside of MongoDB. So  as you can see, I have the Mongo database running on minion number two, the guest book running on  minion number one. So now I'm going to force the move of the Mongo database to another node. So now  the database is running on minion number zero. If I go there and I reload, as you can see,  it's working. So the application is still reading data from the database, but as you may notice,  something happened, like we lost all the messages, which brings us to the next problem,  which is data persistency. So yeah, you can create 12 factors applications and respect  all the manifesto and store data outside of it, but sometimes you still have to deal with state  full containers. So we have to make sure that the data is persistent. If the container is moved to  another host, or if it's a restart or whatsoever, the data is not lost. So with other solutions,  like other orchestration engines, you are on your own. As I said in the beginning, there are, I don't  know, 10 projects, 10 open source projects that try to address this problem. You try to find the one  which works and you integrate that. But with Kubernetes, you have this concept of volumes. The  nice thing is really nice. It's being built into Kubernetes. It just works out of the box,  but it's also part of the grand design of Kubernetes, meaning that a volume is like any other type of  resource. So it's taken into consideration by the scheduler when deciding where something has to be  run or if something can even be run. There is the concept of, well, there is the administrator,  the operator of Kubernetes is going to add some volumes to the cluster. Each volume is exposed  as a persistent volume. And then as a user, you just have to write a claim, a persistent volume  claim, meaning that you need your application to have, I don't know, two gigabytes of space and you  want this space to be yours only. You don't want to have our application interacting with that,  because for example, you might be fine having our application interacting with this volume.  And then the scheduler will look at the persistent volumes that are available and will try to find  a free one that can satisfy your claim. Then if it finds that, then it's fine. Your pod will be  started. Otherwise, it will stay here until this condition is met. This is really nice. There are  also different drivers available for you. So an FS, Sender, Glaster, FS, whatever, you can decide  how to back up all, what to use in the backend of these volumes. It's really nice. So let's try to  see how it works. So in the beginning, right now, as you see, there is no data persistency.  Here, I defined already a volume, which is called Mongo Storage. It has two gigabytes of  space and it's available, as you can see. Behind the scene, this is implemented using  NFS, because it was pretty cheap for me to have everything running. So now, I have to change  the definition I have for the Mongo Pod. So basically, I have to define... Well, this is...  I didn't want to focus too much on that, but this is how you define stuff in Kubernetes. You have  these YAML files, which you can use to define all your types of resources, and then you can  consume them. The point is here, I'm going to say that I want to mount a volume under dataDB.  And here, I define the volume claim. The volume claim, as some requirements. So I need the storage  to offer me at least one gigabyte of space. And I'm fine with sharing this storage with someone  else. So read, write, maybe this is a policy. I have to apply those changes to my pod.  By applying those changes, I had to recreate the pod. So as you can see, the old one has been  terminated. A new one is running. So the persistent volumes changed. The Mongo storage is no longer  available. Now it's bound. And it's bound to this claim, which you can see here, persistent volume  claim. It's called Mongo NFS. It's bound to the Mongo storage, and everything is fine. So now,  let's go there. If I reload, everything is empty, of course, because I restarted everything. Now,  let's create a message. Let's go back here. And let's delete again the pod running MongoDB. So it  has been destroyed. Now it's running on Minion number zero. Let's reload the page. We have to  wait for MongoDB to be warm enough. So yeah, as you can see, the data is still around. So this is  how you can add data persistency in a really, really easy way. So even legacy application  can keep working inside of Kubernetes with not many changes. One last thing is about  Docker's worm. So I have been talking all the time about Kubernetes, but there is also  Docker's worm out there. What about it? Well, things are a bit complicated. Before Docker,  1.12, which is in RC1 right now, I think. Swarm was a separate project. It was something that you  can, could use on top of Docker. It has, it had nice concept. Basically, it was exposing the same  API of Docker engine, but it had a couple of extra endpoints. By reusing the same API,  it was possible to point any kind of tool written for the Docker engine against it and just use it.  And that was pretty smart. It was possible to do some nice things. It was really, really easy to  use worm. It was really easy to transition from using Docker on one machine to use worm. However,  it was a really, really simple orchestrator. It didn't have a com set of desired state. It  didn't have a com set of service. It didn't have a lot of things. The scheduling algorithms  weren't so good. So that's one of the reasons why we just didn't decide to go for it. However,  with 1.12, or since last Monday, they created the basically merged swarm into the Docker engine  itself. So now the swarm project doesn't longer exist. There is the swarm kit project,  which is a generic framework, but you can reuse to build orchestrators. This framework has been  incorporated into the Docker engine. So now you don't have to add any extra tool. Everything  comes out of the box. And they made a lot of changes compared to swarm. It's a huge step forward.  They took inspiration from different orchestration engines like Kubernetes. They copied some  concepts from it, which is great because now, while it's still really, really easy to deploy,  you get a lot of things that were missing the past, like the desired state thing,  the service discovery. They have an integrated load balancer using IPvS, which is something  built into the kernel, which works really, really well. So it's really cool. However,  there are still some things that are left to the user. So if you want data persistency, then you  have to figure out how to do that. There are different plugins, like I said in the beginning.  If you want to handle secrets, you are on your own. Well, for credentials, you could use environment  variables. But for certificates and other things, you have to come up with something on your own.  There are some operations that now are harder than before, like getting blocks from a running  container on Kubernetes, for example, if I had to debug something, I could do something like that.  Sorry.  I could do get logs. By the way, right now, I'm running Kube City on my laptop,  and it's just pointed to a remote server, which is really nice. So I can get logs.  These are the logs in real time, if I want, of the pod. Let's say that I have an application  which has some trouble, and I have to debug that live on production. So I can spawn a shell inside  of, I can look into the container, basically. So by doing Kube City, I can get a terminal  running inside of the container. As you can see, I'm really inside of that container,  because bit number one is the Puma process. This is really, really nice to debug stuff.  And if that was possible with Swarm as well, you could have done a Docker exec,  a Docker logs, and it would have worked. Now that they moved to this new model,  this is no longer possible, you have to figure out where the container that you want to debug  is running in terms of the host. Then you have to SSH into that host, and then use the Docker  engine which is running inside of it. So it's not so comfortable. Another thing, another drawback,  is of course it's tied to the Docker engine. While with other orchestration engines like Kubernetes  or Nomad, or even Mizzus, you can decide what kind of orchestration engine, sorry, container  engine you want to use. Of course, it's in the early stage. They just announced that, so it will  get better and better over the time. But I think it targets a different use case. With Kubernetes,  you can do fancy stuff like isolate different tenants inside of the same cluster from a  point, from a networking point of view if you want, or have resource control over them,  like set quotas to each one of them. You can have a tire integration with other stuff, like,  for example, if you run bet on top of an open stack, you could do auto scaling based on,  even on the cloud itself, based on the load inside of the cluster. You can do quite some stuff.  It's a different use case, I think. So with that, I think I'm done with early advance. I kind of  rushed during it because I introduced different slides compared to the last time I did the talk.  So do you have any questions about that? Have you ever used orchestration engines before?  Okay. Containers? Okay, good. Okay, so if there's nothing, I think we're done. Thanks.  Thank you.
flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_0_95.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_96_4235.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_4236_4499.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_4500_4715.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_4716_4931.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_4932_4979.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_4980_5531.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_5532_5651.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_5652_6275.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_6276_7691.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_7692_8447.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_8448_11495.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_11496_21287.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_21288_22775.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_22776_26891.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_26892_27011.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_27012_28115.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_28116_28307.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_28308_28547.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_28548_28643.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_28644_29147.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_29148_30335.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_30336_31415.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_31416_31955.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_31956_32195.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_32196_32927.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_32928_33203.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_33204_33947.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_33948_34691.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_34692_35315.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_35316_37007.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_37008_37127.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_37128_37655.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_37656_38147.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_38148_40067.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_40068_43655.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_43656_44591.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_44592_44639.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_44640_44987.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_44988_46235.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_46236_49463.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_49464_51947.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_51948_52871.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_52872_53507.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_53508_54659.png flavio_castelli__Orchestrating_Docker_containers_PEF96Yq6dVQ_54660_54775.png
Okay. So, hello everyone. I'm Flavio Castelli. I'm a Susan employee. I'm part of the Contairs team at SUSE. Here we do fancy stuff like Docker. We care about Docker both inside of Open SUSE and inside of SUSE. We take care of the official images that we have inside of Docker for Open SUSE. We take care of the images that we have. And now we care about other projects like portals or Zipperdocker that are going to be covered during this conference as well. And now we are entering into a new fancy landscape and we are going to take care also about orchestration. So, let's start with application containers. I'm going to assume you know about them at least to a certain degree. So, they are lightweight. So you can run a lot of them on a single machine. The containers are all these little boxes over there. The big green box is the main OS. So it's green. It's Open SUSE. Given they are lightweight, you can have lots of them on the single machine. They are also composable. You are encouraged to put them together in order to build stuff. So you can combine two of them to create an application. You can combine three of them. This is the whole microservice thing that is such a hype nowadays. But if you want to get serious, running everything on the same host is not going to work because it's really prone to failures. It's not going to scale. So you want to do multi-host deployments of containers. But how can you go from a situation where you have everything running on the same machine to managing a large cluster of hosts where you spread containers over them? So there are quite some challenges. So first of all, you have to figure out where each container should be placed on which host. And two containers stay on the same host? Maybe yes, maybe no, because of different reasons. So you have to also monitor the containers, make sure that they keep running, make sure that you monitor also the host. So if a host goes down, you react to this kind of failures. And of course, you have to handle those failures because you cannot let your container die. There are also interesting topics that you probably already faced when running containers on a single machine. So service discovery is one of those topics. But things get even more interesting when you start to run containers spread over different machines. You have also to figure out how to expose the service that are running inside of your containers to external entities like your customers. And of course, you have to handle secrets. It would be nice to handle those credentials, those certificates, all SSI keys and such. And of course, various data persistence. So you can go on your own and try to sort everything out by implementing stuff from scratch. Good luck with that. Otherwise, you can choose an orchestration container and work with that. Why I recommend to use an orchestration container? Well, first of all, you will enter this new, let's say, way of thinking about the deployment. You will stop to care about where a container is running. You will leave this task to the orchestration container. You will stop to care about containers failing. You want to receive a message on your page or whatsoever. And you won't have to SSH into machine to restart that manually or to instruct something to do that. The orchestration engine is going to take care of that as well. So you have just to do one thing. This thing is to declare how things should look like. This is called a desired state. All the orchestration engines, they have this concept of reconciliation. So you start by describing how things should look like. Like, I want to have five instances of this container running my web application running. At the same time, I want five of them. And then you just forget about that as a user. Behind the scene, the orchestration engine will look at what is the current state. Like, there are zero of those containers running. It will compute the actions that are required in order to reach this desired state. And then it will execute all those actions in order to reconcile the current state and to make your desired state happen. And this is a constant loop going on and on. So if one container dies, then the current state is different from the desired state. And henceforth, it will just fix itself. So they are really nice. But which one should you pick? There is quite some choice. So there is Kubernetes, there is Docker's war, there is Mizzus, there is Nomad, there is Lettuce. There are lots of them. So each one of them has its advantages, disadvantages, peculiarities, and such. So at SUSE, we had to figure out which one we would like to focus on, which one is very recommended by us. So our approach was simple. So as a team, we started looking to each one of those orchestration engines. We designed a use case. And then we used the same approach. We evaluated each orchestration engine trying to implement this scenario, which is a common scenario. We tried then to break the orchestration engine to see how reliable it is. And of course, while doing everything, we tried to make sure that we could reuse what we did in terms of open sourcing the stuff we discovered or the specific scenario. So talking about the scenario, we wanted to do something more than deploying a simple web application. So we focused on Portus. Portus is an open source project developed by us. It's a user interface and user interface and authorization service for the Docker registry. There is going to be a talk about Portus in the next days, I think on Sunday or Saturday. I highly recommend you to attend it. It's a really cool project. So it's a web application, technically speaking, which uses a database to store its data. So we have also to run MariaDB. And it's also interacting with the registry and the registry as well as to store its data. So this is not so easy to deploy because we have to keep track of the data. So we need stateful containers while the whole microservice thing is advocating for stateless containers. And Portus is using some crypto stuff. So we had to handle out certificates, we have to handle keys, we had to handle traditional secrets like the DB credentials. So that was not so easy and typical. So after quite some testing and experimenting, we made our decision and our decision is to go with Kubernetes. And this is what I'm going to talk during very many part of the talk. So Kubernetes abbreviated K8S, like internationalization, you know. So it's a project which has been created by Google. It's based on 10 years that Google had running containers in production. It's based on a paper called About Borg, which is the system that was in place inside of Google. So it's based on a really solid background. It has a really good design, of course. The guys from Google know how to run stuff and how to run better skills. So we trust them. It immediately attracted a lot of people. So it has a huge community, which is really active. It's really differentiated in terms of members. It's a project that is not controlled by Google. In fact, it's now donated to the cloud-native computing foundation, which is a foundation underneath Linux Foundation umbrella. So it's pretty famous. So there is a lot of interest from lots of people in using Kubernetes. And what is most important, it's an opinionated solution, which means that it has an answer to basically all the questions you might have. This can be a bad thing for some people, because maybe you want to be free to choose whatever they want. But it's also a really good thing for most of the person, I think, because basically, if you have to solve the problem like service discovery or persistent volume with some orchestration engines, you're up to your own. You have to look at, I don't know, 10 open source projects in order to figure out which one you want to trust, which one is going to work for you. And you have to make sure that it's going to keep working, because they are different projects collaborating, kind of collaborating together. So at each release, you might risk that something is not working properly. Kubernetes has everything inside of it, but it's still swappable. So there are some components that you can replace with whatever you want, which make it flexible. But at the same time, giving everything is inside of it, you have this trust in it that it's not going to break during each release. So the architecture of Kubernetes is following one. So you have two types of machines inside of the Kubernetes cluster. You have the master and you have the minions. The minions is where the containers are going to actually be running. The master instead is where all the decisions are made. As you can see, there is also there was yellow circles at CD. So at CD is a distributed key value storage, which is used to hold all the data about Kubernetes stuff. So the master is going to store and read data inside of it. The master basically has three demons running. There is the API server, which is your entry point into the cluster. This is how you interact with it. But it's also being used by all the other pieces to coordinate themselves, to get information, to react to changes and such. You have the scheduler, who is the one in charge to figure out, for example, where a certain container is going to be placed. You have the controller manager that is the component that constantly looks at the current state, figures out if it's different from your desired state and then computes the actions that needs to be done. Like, I want a new container to be running and then the scheduler will say, okay, this container has to be placed on this machine. And then on the minion, you have the KubeNet process, which will be aware about AF to run a container on my machine. So it will take care of running this container. To do that, the KubeNet will interact with some container engine. In this case, you can see there is Docker, but you can replace Docker with other container engines like Rocket, for example, which is an alternative container engine from CoreOS. And using this container engine, the KubeNet will create the pods. So the pod is a concept of Kubernetes, which is unique to it. Basically, you can decide that different containers that have a strong relationship, a strong bond that can be placed inside of the pod. A pod will remove some of the isolation features that are typical of containers. So for example, a peculiar thing, all the containers inside of the pod, they share the same network namespace, meaning they will have the same network card, they will have the same IP address. If a typical use case for that, for example, let's say that you have a web application like Ruby application. So you have, I don't know, Puma as a web service for it, but you don't want to expose Puma to the internet traffic. What you usually do, it's really typical, you put something in front of it, like engine X or Apache at a no. So you can have a container running your web application with Puma, and then you can have another container inside of the same pod, which is the official engine X container. And then you have the engine X container, which is going to be a balancer for the Puma, which is going to be listening on localos, because from a networking point of view, everything inside of the pod is on the same network interface. So you can easily scale up the pod without having to duplicate too much stuff. The last process of the minion is the Q proxy. This is required, as we will see later, to handle networking, which is a really complicated topic. It's worth a dedicated talk, but I will try to address that later on, but if you have more questions in the end, you're free to ask. So let's look at a concrete example. So I will focus on using Kubernetes. So I will focus as being a user who wants to run stuff inside of Kubernetes. I want to just skip the part about bringing up Kubernetes and administering it. I want to focus on a user. A user who wants to run a simple application. So we have a web application, which is using a database to store its data. So we could have a web container running on host one, and a database container running on host number two. So what we want is to react to failures. So if the host goes down, then of course, it means the database container will go down as well. We want Kubernetes to react to this event by bringing up a new database instance somewhere else, like on OS 3. So this is implemented in Kubernetes using a replication controller or there is also replication set. Basically, they allow you to describe the state. Say I want to have five to one replicas running at the same time. So if this number changes in the current state, Kubernetes will take care of bringing everything back to what you specified. So let's take a quick look. I will go back and forth. So right now here, can you read it? Is it big enough? Okay. So I have a Kubernetes cluster running on my machine. It's composed by three nodes. Just ignore this one. Everything is running on Leap so far. So we have three Minion nodes, and we are already running our web application and the database. So here you can see that I'm using kubectl, which is a common line tool to interact with Kubernetes. I'm getting the list of all the parts which are running. This is in real time. As you can see, I have one of them running on Minion zero, one of them running on Minion two. So now if I SSH into Minion zero, if I do a Docker PS, I can see that, well, can you still see that because it's at the bottom of the screen? So it's running my container here. This is the gas book application. This is the idea of the container. So if I kill the container manually, as you can see, Kubernetes immediately noticed that the gas book pod was broken. It was not working properly, and it immediately fixed that. So my application is still up and running. I could bring down the node, and of course that would trigger the move of the container somewhere else. But let's delete the pod instead. As you can see here, this one over one means that I want to have always one pod providing this gas book application running. So if I delete it, as you can see, this is the violation of my desired state. So a new pod is created this time on a different node, on Minion one. So as you can see, it's pretty robust. It's working. You might think we're done with that. That's not true. So let me introduce you to the survey discovery problem. So let's define two actors. You have a producer, which is a container which has a service running inside of it, like the database in our case, and you have a consumer, which is someone that is using this service. So in our case, it's the web application. So when you start the web application container, you have to find where the database is running. Now, according to what kind of network setup you have, you might need to know two kind of information, like the IP address of the database, and maybe even the port number. With Kubernetes, you don't need the port, but with our solution, you do. So how do you figure out where something is running? You might be tempted to use the DNS, like you usually do with normal stuff, but this is not going to work because there are the DNS implementation on the client side. Some of them are buggy, and they keep cache. They don't respect the time to expire stuff. So things are not going to work well in the containerized word, because you have to imagine that stuff keeps moving from one place to another. The IP address keeps changing and such. So the only way to do that is to use distributed key value storage, like console, LCD, or zookeeper, and then register the information about the producer inside of it, and then make this information available to all the containers that are interested in that. Usually this happens by setting some environment variables and then reading those environment variables back. So there are still some changes, of course. Even if you get everything to work when you start the web container, how are you going to handle a sudden move of the producer? So like it happened before, if I had killed the Mongo database, that would have moved to another place, and it would have another IP address. But the gas book application would be still looking for the Mongo database to the old IP address. So how can you react to these sudden changes? And how do you face multiple choices? You might have different backend services that you are consuming that are providing the same service. How can you choose between all of them? So the solution is to use a load balancer. You could be placing a load balancer in front of all the producers and then point the consumer to the load balancer. This is going to work. However, this is not so easy as it might seem. So first of all, you have to make the load balancer aware of all these containers. So to read the configuration changes from this key value storage and then generate a new configuration, reload the configuration. This is going to, depending on the load balancer, that can have some troubles when things keep changing and you keep reloading the configuration. So that load balancer can be unresponsive. So this is not really a good solution. And still, this is going to be the single point of failure of the whole thing because you have just one load balancer. Of course, you could place more of them. But what is the solution? With Kubernetes, we have this concept of service. Outside of Kubernetes, you are on your own. You have to figure out how to do stuff. Kubernetes has this concept of service. So with the service, you basically, well, you point the consumer to the service and the service is going to put the consumer in touch with the producer. So basically, the whole service is like a switchboard operator. And it's going to take care of making all the information flow to the right place. The nice thing is that it can also be used when you have to expose a service to the outside, like to the Internet. So you can have two types of services. You can have the internal ones, which are not visible outside of the cluster, like the database service. But you can also have services which are exposed to the public. So you can point all the Internet traffic, and this is going to flow straight to the right web container, regardless of where this is running. How is this done? So Kubernetes does that by defining a virtual IP address for each service. This IP address is created once, and it will always stay the same. It won't change at all. That means that, for example, using a nod on Kubernetes, you can have an internal DNS, which is not going to break. It's not affected by all these issues that I mentioned before. So you can have a DNS entry like, I don't know, mongo.kubernetes, that is going to be resolved to this virtual IP address. And then there is an internal log balancer, which is running basically on each node, which is Qproxy, which uses IP tables to route the connection to the right pod, providing the service. This is for the internal services. For the published ones, they are like the internal services, so it's done in the same way. Then each node, each minion inside of your cluster is going to have these services exposed on a fixed port. So now you just have to take your load balancers and point them to all the nodes of your cluster and to the same port on each one of them in order to expose the service. If you are running Kubernetes on some public cloud like AWS, there are even backends, which are already implemented, so you can reuse all the load balancer that are already provided by AWS in order to maintain even part of the infrastructure. So let me show you how this is going to work. So here we have our guest book. So we can write messages in there. These are stored inside of MongoDB. So as you can see, I have the Mongo database running on minion number two, the guest book running on minion number one. So now I'm going to force the move of the Mongo database to another node. So now the database is running on minion number zero. If I go there and I reload, as you can see, it's working. So the application is still reading data from the database, but as you may notice, something happened, like we lost all the messages, which brings us to the next problem, which is data persistency. So yeah, you can create 12 factors applications and respect all the manifesto and store data outside of it, but sometimes you still have to deal with state full containers. So we have to make sure that the data is persistent. If the container is moved to another host, or if it's a restart or whatsoever, the data is not lost. So with other solutions, like other orchestration engines, you are on your own. As I said in the beginning, there are, I don't know, 10 projects, 10 open source projects that try to address this problem. You try to find the one which works and you integrate that. But with Kubernetes, you have this concept of volumes. The nice thing is really nice. It's being built into Kubernetes. It just works out of the box, but it's also part of the grand design of Kubernetes, meaning that a volume is like any other type of resource. So it's taken into consideration by the scheduler when deciding where something has to be run or if something can even be run. There is the concept of, well, there is the administrator, the operator of Kubernetes is going to add some volumes to the cluster. Each volume is exposed as a persistent volume. And then as a user, you just have to write a claim, a persistent volume claim, meaning that you need your application to have, I don't know, two gigabytes of space and you want this space to be yours only. You don't want to have our application interacting with that, because for example, you might be fine having our application interacting with this volume. And then the scheduler will look at the persistent volumes that are available and will try to find a free one that can satisfy your claim. Then if it finds that, then it's fine. Your pod will be started. Otherwise, it will stay here until this condition is met. This is really nice. There are also different drivers available for you. So an FS, Sender, Glaster, FS, whatever, you can decide how to back up all, what to use in the backend of these volumes. It's really nice. So let's try to see how it works. So in the beginning, right now, as you see, there is no data persistency. Here, I defined already a volume, which is called Mongo Storage. It has two gigabytes of space and it's available, as you can see. Behind the scene, this is implemented using NFS, because it was pretty cheap for me to have everything running. So now, I have to change the definition I have for the Mongo Pod. So basically, I have to define... Well, this is... I didn't want to focus too much on that, but this is how you define stuff in Kubernetes. You have these YAML files, which you can use to define all your types of resources, and then you can consume them. The point is here, I'm going to say that I want to mount a volume under dataDB. And here, I define the volume claim. The volume claim, as some requirements. So I need the storage to offer me at least one gigabyte of space. And I'm fine with sharing this storage with someone else. So read, write, maybe this is a policy. I have to apply those changes to my pod. By applying those changes, I had to recreate the pod. So as you can see, the old one has been terminated. A new one is running. So the persistent volumes changed. The Mongo storage is no longer available. Now it's bound. And it's bound to this claim, which you can see here, persistent volume claim. It's called Mongo NFS. It's bound to the Mongo storage, and everything is fine. So now, let's go there. If I reload, everything is empty, of course, because I restarted everything. Now, let's create a message. Let's go back here. And let's delete again the pod running MongoDB. So it has been destroyed. Now it's running on Minion number zero. Let's reload the page. We have to wait for MongoDB to be warm enough. So yeah, as you can see, the data is still around. So this is how you can add data persistency in a really, really easy way. So even legacy application can keep working inside of Kubernetes with not many changes. One last thing is about Docker's worm. So I have been talking all the time about Kubernetes, but there is also Docker's worm out there. What about it? Well, things are a bit complicated. Before Docker, 1.12, which is in RC1 right now, I think. Swarm was a separate project. It was something that you can, could use on top of Docker. It has, it had nice concept. Basically, it was exposing the same API of Docker engine, but it had a couple of extra endpoints. By reusing the same API, it was possible to point any kind of tool written for the Docker engine against it and just use it. And that was pretty smart. It was possible to do some nice things. It was really, really easy to use worm. It was really easy to transition from using Docker on one machine to use worm. However, it was a really, really simple orchestrator. It didn't have a com set of desired state. It didn't have a com set of service. It didn't have a lot of things. The scheduling algorithms weren't so good. So that's one of the reasons why we just didn't decide to go for it. However, with 1.12, or since last Monday, they created the basically merged swarm into the Docker engine itself. So now the swarm project doesn't longer exist. There is the swarm kit project, which is a generic framework, but you can reuse to build orchestrators. This framework has been incorporated into the Docker engine. So now you don't have to add any extra tool. Everything comes out of the box. And they made a lot of changes compared to swarm. It's a huge step forward. They took inspiration from different orchestration engines like Kubernetes. They copied some concepts from it, which is great because now, while it's still really, really easy to deploy, you get a lot of things that were missing the past, like the desired state thing, the service discovery. They have an integrated load balancer using IPvS, which is something built into the kernel, which works really, really well. So it's really cool. However, there are still some things that are left to the user. So if you want data persistency, then you have to figure out how to do that. There are different plugins, like I said in the beginning. If you want to handle secrets, you are on your own. Well, for credentials, you could use environment variables. But for certificates and other things, you have to come up with something on your own. There are some operations that now are harder than before, like getting blocks from a running container on Kubernetes, for example, if I had to debug something, I could do something like that. Sorry. I could do get logs. By the way, right now, I'm running Kube City on my laptop, and it's just pointed to a remote server, which is really nice. So I can get logs. These are the logs in real time, if I want, of the pod. Let's say that I have an application which has some trouble, and I have to debug that live on production. So I can spawn a shell inside of, I can look into the container, basically. So by doing Kube City, I can get a terminal running inside of the container. As you can see, I'm really inside of that container, because bit number one is the Puma process. This is really, really nice to debug stuff. And if that was possible with Swarm as well, you could have done a Docker exec, a Docker logs, and it would have worked. Now that they moved to this new model, this is no longer possible, you have to figure out where the container that you want to debug is running in terms of the host. Then you have to SSH into that host, and then use the Docker engine which is running inside of it. So it's not so comfortable. Another thing, another drawback, is of course it's tied to the Docker engine. While with other orchestration engines like Kubernetes or Nomad, or even Mizzus, you can decide what kind of orchestration engine, sorry, container engine you want to use. Of course, it's in the early stage. They just announced that, so it will get better and better over the time. But I think it targets a different use case. With Kubernetes, you can do fancy stuff like isolate different tenants inside of the same cluster from a point, from a networking point of view if you want, or have resource control over them, like set quotas to each one of them. You can have a tire integration with other stuff, like, for example, if you run bet on top of an open stack, you could do auto scaling based on, even on the cloud itself, based on the load inside of the cluster. You can do quite some stuff. It's a different use case, I think. So with that, I think I'm done with early advance. I kind of rushed during it because I introduced different slides compared to the last time I did the talk. So do you have any questions about that? Have you ever used orchestration engines before? Okay. Containers? Okay, good. Okay, so if there's nothing, I think we're done. Thanks. Thank you.
have to also monitor the containers, make sure orchestration container. You will stop to care to make sure that we could reuse what we did in people. So it has a huge community, which is you have this trust in it that it's not going to is where all the decisions are made. As you can with some container engine. In this case, you can allow you to describe the state. Say I want to with our solution, you do. So how do you figure have to deal with state full containers. So we written for the Docker engine against it and just I have to debug that live on production. So I can