Keynote: So, I have all these Docker containers, now what?
Waite, Mandy
Mandy Waite - Keynote: So, I have all these Docker containers, now what? You've solved the issue of process-level reproducibility by packaging up your apps and execution environments into a number of Docker containers. But once you have a lot of containers running, you'll probably need to coordinate them across a cluster of machines while keeping them healthy and making sure they can find each other. Trying to do this imperatively can quickly turn into an unmanageable mess! Wouldn't it be helpful if you could declare to your cluster what you want it to do, and then have the cluster assign the resources to get it done and to recover from failures and scale on demand? Kubernetes is an open source, cross platform cluster management and container orchestration platform that simplifies the complex tasks of deploying and managing your applications in Docker containers. You declare a desired state, and Kubernetes does all the work needed to create and maintain it. In this talk, we’ll look at the basics of Kubernetes and at how to map common applications to these concepts. This will include a hands-on demonstration and visualization of the steps involved in getting an application up and running on Kubernetes.
 Thank you for coming and welcome. Thank you. Hey, thanks, everyone. You all hear me? I think you can. Good. Excellent. Thanks for having me at Python, EuroPython. My last big Python event was Python, PyConJP in Japan last year. I didn't get to speak, though, but it was really fun. Although most of the talks are in Japanese and my Japanese is getting better, it's not so great. My Spanish is really, really bad. But Spanish and Japanese are very similar, so maybe I should learn both together. No, really, seriously, they are. I know it doesn't seem to make sense. We're going to talk about containers and having containers and having lots of containers, because ultimately, everything is going to be containerized and we're going to have lots of containers. We won't know what to do with. And I'll ask you some questions, Ada, and see how far you are along with moving towards containerization. So basically, when we have lots of containers, what do we do then? And this is a problem we face at Google. So this is a data sensor. This is a Google data sensor in Iowa in the US. It's a place called Council Bluffs, and this is one of our bigger data sensors. And if I leave it out for a long enough, you'll probably be able to count all of the machines. How many there are. But this is a cluster. So clusters are one of the constructs we have internally, but these clusters are broken down into cells. So cells are smaller. We have many cells per cluster. And this will probably, a cell we're going to look at today, is going to have about 10,000 machines in it. So they're quite large. And this is a huge amount of compute power. Lots of compute power, but we need to make this available to our engineers, our software engineers, our developers. So how do we go about making this compute power available to our own developers? And it works something like this. This is what a developer does. Well, first in context, the one thing, given what you see there, we don't want the engineers to have to kind of select a rack, select a machine, and say, hey, I'm going to run it on that machine. I'm going to SSH, SFTP a binary over to the machine, SSH into the machine, stand up my process, my server or whatever, maybe log into many machines and do that multiple times. But that's not going to be possible. Huge amounts of machines, huge numbers of engineers, huge amounts of jobs to run. So how does it happen? So basically, we have a configuration file. In this case, it's called a Borg configuration file. I was in India recently, and nobody there had heard of Borg. How many of you are familiar with Borg and Star Trek? Okay, right. So we never used to be able to talk about Borg because Paramount Pictures own it. It was kind of like one of our worst kept secrets that we had this thing called Borg running internally. But now we talk about it all the time. So because it's fun, and it's really good to show this in the context of what we're going to talk about later, which is Kubernetes. So basically, this is a Borg configuration file, and what the developer does is he creates a job. Jason file, calls the job Hello World, says which cell he wants to run it in. Going back to what we said earlier, a cell is a few thousand machines. In this case, he's saying it's called IC, some random cell name we chose. And he specifies what binary to use. In this case, Hello World web server. So he wants to run Hello World on a web server. And this is going to be a fat binary. Statically links all of its dependencies with it. So effectively, we can run it pretty much anywhere without having to worry about the underlying operating system. And that includes the web server as well. So this thing is quite big, probably about 50 megabytes. So he specifies the path to his binary, or her binary. And unfortunately, we have too many male software engineers, not enough female software engineers. So let's encourage women to be software engineers. And arguments. We have to specify some arguments for our binary, pass them in via the environment. In this case, we want to specify what port to run on. This is parameterized. Then we have some requirements in terms of resources. Now, this is important. We'll circle back to this in a minute. So we can specify how much RAM, how much disk, how much CPU. And ultimately, we can say how many we want to run. So in this case, we want to run five replicas of this job, five tasks effectively. And why five? Why not do it at Google scale, 10,000? Makes more sense, right? We have all those machines. We saw how many machines we have. So let's run 10,000 copies of this. So once we finish this, our software engineer, she types in a command on the command line, passes in the config file. And that gets pushed out to somewhere. It gets pushed out to this ball scheduler. And what happens then is this. Over a period of time, in this case, about two minutes, 40 seconds, 10,000 tasks start. 10,000 instances of that job start. And it takes two minutes, 40 seconds, roughly. We do phase the rollout of all of these jobs to make sure we don't do them all at once. One of the key factors here is the size of the binary, 50 megabytes, 10 times 10,000. It's about 20 gigabytes per second of IO. We're going to be cashing that binary quite a lot, but we had to move it around between 10,000 machines. So there's a huge amount of IO going on. But eventually we get to a point where we have 10,000 running, or nearly 10,000. Maybe not quite 10,000. We'll talk about that in a second. And Borg looks like this. This is what Borg is to Google. It's not going to assimilate you, but I think we came up with a name because it's probably going to assimilate everybody eventually. So this is Borg. And Borg runs within a cell. So each cell has its own Borg master, its own Borg configuration. In this case, we have a Borg master, which is highly replicated. We have five copies of it for resilience. And we have lots of other things. These down here are our machines. These are our machines we saw in the racks. They're all running a thing called a Borglet. We have a scheduler. We have some configuration files in the binary. So what happens is the developer, the engineer, creates his or her binary. And they use a massively distributed parallel build system called, well, I won't say what it's called, but it's externally available now called Bazel. So we made this open source. So our own build system is now available open source called Bazel, B-A-Z-E-L, or if you're American, B-A-Z-E-L. It gets very confusing, believe me. If you go to Canada, it's so confusing. Like routes and routes. So basically, he or she creates the binary, pushes it out, and it gets stored in storage for the cell. And then they push their configuration file. Configuration file gets copied to the Borg master. We have a persistent pack source back store, consensus based. And what happens then is this scheduler, looking around, comes along and says, hey, what is the desired state? We should have this running. Do we have this running? And it sees 10,000 new tasks. And it says, hey, they're not running. We should have 10,000 of those. Let's make sure that's happening. Let's fix that. And so it goes about planning the running of these 10,000 tasks. And it creates a plan, and then it starts telling the Borg master what the Borg master makes decisions and tells them the Borglets on these machines to run this particular task. So they get communicated. The task will ultimately run inside a thin container wrapper. So it has a container around it. It's not just running the binary. It is containerized. A very lightweight shim container that's not Docker. It's not standards-based. The Borglet ultimately will pull the binary over from storage, and it will start running. And we'll see this. Lots of Hallowels. All over our data center. So now we're running multiple copies of that. And so that's what we had, 10,000. But if we look at it a little bit closer, we find there's 9,993 running. Not quite the 10,000 we expected. But this is a highly available service. We expect some lessening of the number of tasks we're running over time due to the way we operate. And that's interesting. So let's look at that in a little bit more detail. So failures. Things fail. But failure is kind of more of a generic term here. There are many reasons for failures. And one of the main reasons for failures, particularly for low priority jobs, is preemption. If we look at the top bar, which is our production jobs, we have very few failures. And most of them are down to a machine shutdown, where we've actually scheduled some maintenance on the machine, and we've taken the machine down. That task, any task running on that machine, we don't be rescheduled elsewhere in the cluster. We have a very small number of preemptions. Down here are non-production jobs, which are things like map producers, batch jobs. They get preempted all the time. They're happy to be preempted. And in fact, the calculation generally says that for about 10,000 tasks, about seven or eight of them will be not running at any given time because of preemption. They'll be about to be scheduled somewhere else, but they won't be running at that particular time. And we see other things here. We see, again, the blue bar, which is the machine shutdown, which is pretty much the same as production. And we have some other things as well, out of resources, very small number of machine failures. And for when you have as many clusters, the many machines as we have, machine failures are given. We expect that. We don't panic when machines go down. It's part of the normal running of our business. And another interesting thing is how we try to make efficient use of our resources. So we have CPUs, we have memory, we have disk.io, we have network.io. And sometimes it's quite possible for one task to be using lots of memory for very little CPU or vice versa, lots of CPU and very little memory. If you put one of those on a machine, then you may be wasting one of those resources. It's what's known as resource stranding. And these are the available resources, these white bars here. This example here is actually our virtual machines, which are tasks. Our virtual machines are actually containers, believe it or not. It's a Google computing engine. So these are all virtual machines, these bars, individual bars. And what we can see here is that some of these machines have available capacity, available RAM, available CPU, and if we look over here, we see a different situation where we have maybe some with available CPU and others with no available RAM and vice versa. This here and this here is called resource stranding. It means we're not actually making use of that resource. So we have spare memory capacity or spare CPU capacity that's being wasted effectively. So one of our challenges is like a Tetris puzzle to try to stack these things in a way where we get the best possible utilization out of our clusters. So we will mix and match them to make sure we have low CPU, high memory jobs running with high memory, low CPU jobs. And of course, we run multiple tasks per machine. That's extremely important. That can then come back to all this with Kubernetes shortly. And another interesting thing is this, which is going to be a huge challenge in the future when it comes to Kubernetes, but it's going to be really important to all of us. So we saw earlier that our developer, she specifies what resources she wants to use or he wants to use. 100 megabytes of RAM, 100 megabytes of disk, 0.1 CPU. And that will be this blue line up here. So everything's running. We'll match into this blue line. These are the resources that were requested by these jobs. In reality, though, it's like this. And so we have all of this wasted space, which we can't use because it's been allocated effectively for those running jobs. But we can use it. So what we do is we effectively estimate, based on the run patterns of the current jobs, how much they're going to use. And that's this blue line here. So this is our reservation. So this is how much we reserve specifically for those jobs. And what we can then do is reuse all that space. Now we can reuse that space for very low priority jobs. Again, those batch jobs, those map reduces. Things that we want to run, we want them to finish eventually, but we don't really care when it happens. It could be like running some kind of monthly report that nobody ever looks at that gets logged or running a map reduce across a huge amount of data that may be important at some point or just needs to be done, but we don't really care when it needs to be done. So all of that stuff, we can reuse it and we can run jobs within it. But that's really important. That's how we can get maximum utilization out of all of our machines in that data center. And so moving on to Kubernetes now, gradually, one of the observations is that if you have your developers spending time thinking about machines or thinking in terms of machines, and you're probably doing it wrong, because it's too low a level of abstraction. Now today, maybe it's fine, but in the future, this is not going to be the case. We need people to be thinking in terms of applications and not having to worry about the infrastructure in which they run. I mean, anybody who uses a platform as a service knows how important that is anyway. You don't care about the infrastructure. You want to write your work, configuration file, build a binary, and just say, run this for me. I don't care where you run it. I don't care about how you do it. Just run it for me and make sure it stays running. We get efficiency by sharing our resources and reclaiming unused allocations. And containers, the fact that we containerize everything, allows us to make our users much more productive. So everything we run runs on a container. Two billion containers a week, we estimate. We never really thought that was very important until Docker came along. Containers became the next big thing, right? Alex, then Docker. And Docker became huge. So now one of the things we talk about all the time now is we run containers all the time. We are pretty good at running containers. Which is why we created Kubernetes. If you're interested in more details of what I've just talked about, Borg, there's a paper here. G.gl, one capital C for N U O. That's the white paper in Borg. That's got all of the details, all of the graphics you just saw. It goes into much, much more detail, of course. So let's look in terms of a simple application and how we can do this externally with containers and through Kubernetes. So this is a very simple pattern. Generally when we give this sockets PHP in the middle, MySQL, Memcache, and we have a client. We have many of these pythons running. This could be many instances of flask, it could be some kind of event system, but we have the ability to run many, many concurrent requests. We're probably going to want to scale this thing on demand. We may not want to scale MySQL that much until we get to a point where we have to do replicas and sharding. Memcache, we've probably gone on a scale as well, but we're going to keep it simple for now. Just keep one MySQL, one Memcache, and a few of these pythons that are at the front end. So let's talk about containers. So how many of you are familiar with containers? How many of you have actually spun up a Docker container? Hey, lots of you. It's almost the same number, right? Again, last year we'd asked this, how many of you have heard of containers, lots of hands, how many of you have spun up a Docker container? Lots of many. So things have changed now. Docker is the future. Well, containers are the future. We now have this thing called the open container project. Docker have kindly made what they have into a spec, and we're all going to get behind it. We have this common specification from which we can write containers. Things like CoreOS with a rocket container, they're all going to fall in line and we'll have a common format for containers. Which is going to be great. But just for those of you who are not really familiar with containers, just a few slides, very few slides on containers. Just kind of give you some of the concepts. This is the way we used to do things in the old days. We have a machine, maybe next to our desk, in our bedrooms or in a colo or in a server room. And the machine would run our operating system. It would have all of the packages installed at provided libraries, things like open SSL. On top of that, we would run applications. And how many of you have had a situation where you're running one application and all of the other applications on the machine fail because that one application went mad? Use all of the CPU, use all of the RAM, it crashed the machine and took all of the other applications down. And this may have been a very low priority app, one that you didn't really care about taking down some really important ones. Now, this is never a good idea running multiple applications on one machine because there's no isolation between them. Whatever affects one application will probably affect all of the others. There's no namespacing. They all have one view of the machine in which they're running. They have one view of the CPU, one view of the memory, one view of the file system, one view of the network. They share libraries. And so you're in a situation where maybe one day you update a version of a package, it updates a library, and one of your applications says, hey, I'm not going to run anymore. That library is not compatible with me. So dependency hell. And if it's on Windows, it's DOL hell, and it's probably even worse. Applications are highly coupled to the operating system. This is a problem. And so we created virtual machines. And what we did basically is stuck a layer on top of the hardware called a hypervisor. And we now had an idealized piece of hardware on which we could run multiple operating systems. So now we have this thin layer, it looks like a piece of hardware to be running virtual machines. And that gives us some isolation. So now we can run applications in their own virtual machine. So each application is now isolated. If one application crashes, it doesn't affect the others. But it's extremely inefficient because we have this red bit at the bottom here. We have the operating system, the kernel. And you know when you install a virtual machine, you pretty much have to install the entire devian stack or the entire center stack or the entire Windows stack. So that's not very efficient at all. They're still the same type coupling between the operating system and the application. And as anybody who's tried to manage lots and lots of virtual machines to provide isolation, you know it's hard. So there are new ways containers. In this case, we move up a layer. So we move above the operating system and provide an idealized operating system. No longer idealized hardware and idealized operating system on which we can run apps and their dependent libraries. So the libraries here are part of the container. So the container has an application. It has all of its dependencies. It has its entire environment. So we can move this container around anywhere we want to. We can move it from one machine to another, from one runtime to another, from a laptop to a virtual machine, to a cloud, to a bare metal server, to a set-top box, ultimately, maybe even to a phone when we had Docker on Android and iOS. I'm sure it's going to happen, right? And let's look at the example. So we have our application, PHP in Apache. It should be Python in Apache. Sorry. See? I do apologize. So wherever you see PHP in this deck, read Python. We'll change it before I share the slides. So I'm trying to think of what could offend a Python audience most. It's probably talking about PHP, right? My god. Okay. So we have containers. We want to run these components of our application, Python and Apache, Memcache, MySQL. Not Apache. Obviously, Python and Vlask and Bottle and all of the other things we could potentially use. Memcache, MySQL. And MySQL has its own libraries. It doesn't have any common libraries with the others. So we can stack those libraries with the container in which MySQL runs. And Memcache, PHP, Python and Apache have their own, I keep saying it. And Python and whatever. Unicorn. Anything. They have their own dependencies. But they also have when we install them some shared dependencies as well. So some commonalities. So when we actually create the image, we can actually share some stuff between them. But that's not shared at run time. So when we create the container, they will have their own dependencies packaged together in the container. And underneath that, we have a server. And again, this could be a virtual machine. It could be a laptop. It could be a bare metal server. It could be anything, pretty much. And underneath it, we have the actual hardware. And all of this is being maintained by a Docker engine. So Docker is the thing that runs this. So when we talk about containers, mostly synonymous with Docker nowadays, but again, there are other container formats. And hopefully, they will all comply with a standard. And that's the Nirvana we're all heading towards. So Docker effectively controls the creation of these containers and the management of these containers. So at the end of it, we will have Python, and Flask, Angular, Memcache, MySQL, all running in containers. So why containers? So there's many important reasons for having containers. But you can see, just by looking at what we do, that's the only way we can do it. We can't do it any other way. This is perfect solution for the kind of scale that we want. But it's also perfect for smaller scale as well. Why? Because it's much more performance. It's much more performance in terms of the fact that we don't have to do all of that installation stuff. They are pretty much like they're running on bare metal. So the performance is pretty much the same as a virtual machine, but they're much quicker to get up and running. Which means you can swap them out quicker. You can do upgrades quicker. You can do pretty much everything quicker. Repeatability. So the whole problem where we have development, QA, build, test, production, where we want to have repeatable environments, where we have a situation where when we test something in QA and then run it in prod, it fails in prod where it works in QA. How many people have had that situation? Have you had any hands remembering those days? So what containers give us is the ability to have a consistent environment because the environment is packaged with a container. So basically when we run it in QA and when we run it in prod, it's exactly the same. It's exactly the same environment. So that's one of the great use cases of containers today. But much more is the portability of it, which we're going to talk about in a second. Quality of service. We can now do resource isolation as well. Using things like C groups and Linux and namespaces, we can actually isolate the resources. We can say we only want this to have 100 megabytes of RAM, 100 megabytes of disk, 0.1 CPU. And ultimately accounting. These things are easier to manage. They're easier to trace. They're easier to audit. They're small, composable units that can be tracked very easily. And ultimately, portability. You can move these things around from one cloud provider to another. Images specifically. You can't just pick up a running container and move it, but you can easily run the same container in a different cloud provider on a bare-measure machine on the laptop. You can move them from one machine to another as the shape of your cluster, if you have a cluster of machines, changes. You can move them around to be more efficient. So we can go back to what we had before with the efficient allocation of resources. We can do that if we have containers. And ultimately, this is a fundamentally different way of managing and building applications. So demo. I'm not going to do this demo. I left that slide in by mistake. This would have been a containers docker demo. And I don't think I want to bore you with that. It's very easy to find a tutorial on docker and get up and run them a bit. Now, let's not talk about that. Let's talk about Kubernetes instead. How many of you have heard of Kubernetes? How many of you can say Kubernetes? It's hard word to get your head around. Probably easier if you agree, because it's a Greek word. But if you want to help pronouncing it, I'll be outside in the Google booth after this talk. So I can definitely provide assistance on that. Maybe I'm saying it wrong. Maybe I've been saying it wrong all this time. So I'm happy to be corrected. So Kubernetes, let's talk about that. And we've given you an introduction to what we do at Google. So that should provide the context on why Kubernetes is necessary. What we often miss out, is that we don't provide that kind of context. So I'm hoping that the introduction to Borg has provided that for you. So Kubernetes is Greek word, means helmsman. It's the root of the word governor. So Arnold Schwarzenegger's governor comes from Kubernetes. And it's an orchestrator or scheduler for docker containers. Ultimately for other forms of containers, I think ChoraOS is already using it to schedule orchestrate rocket containers. It supports multiple cloud environments. So Metasphere, I always forget them. VMware, even Microsoft involved. You can run Kubernetes on Amazon. You can run it pretty much anywhere. You can run it on your laptop with Vagrant. So you can just create a four machine cluster, virtual machines with Vagrant up. And you'll have a Kubernetes cluster. And ultimately, eventually, we may have a situation where we can run Kubernetes across multiple cloud providers. It might be difficult. It might be possible. But it may be one day you'll have your fleet of machines will be running in Google, in Amazon, and Microsoft Azure as well. Possible. I'm not sure if it's going to happen. So this is kind of inspired and informed by everything that we saw previously, everything with Borg. And it's basically our experiences. Open source, ridden and go, like many good programs nowadays, but completely respect Python. I love Go. I love Python. I used to be a Java developer. I spent 15 years developing Java. No, 11 years. Now I moved to Google. I haven't wrote a line of Java code since. Now I write... Now I write... It's like Java program is anonymous, right? It's been four years since I wrote my last line of Java code. So now I write in Python, go, and write in Angular, and write in Java scripts, and all of those more interesting, useful languages. Java is getting better. Java rate is a big step forward. And ultimately, we want to be able to talk about managing applications and not machines, which is actually what we talked about earlier. And some very quick concepts. I'm not going to introduce them, but I want to show you the icons so that when you see them, you'll know what they mean. Angular, pod, service, volume, label, replication controller, node, are all of the key concepts. How many of you are familiar with sort stack? How many of you like the terminology in sort stack, like grains and such like it? I think it's really hard to get your head around. And I think one of the dangers about an abstraction is that you get too far away from the terms that are familiar to people. Most of these are familiar to people with service, the idea of a replication controller, a node, a label, a container. The pod is probably the most difficult one to get your head around. So let's talk about pods. Let's talk about nodes first and clusters. So we have a cluster, kind of maps back to what we talked about earlier with Borg, where we have a master. And the master has a scheduler and it has an API, an API server that can be used to talk to nodes. The nodes are all running a thing called a kubelet. And they have these things called pods running containers. We'll talk about pods shortly. They also have a proxy by which we can expose our running containers to the outside world. And we have many nodes. And a cluster, this is an abstraction. So a cluster could be different depending on which cloud provider you're using. And ultimately what you want to have is a fabric of machines that looks like a flat shape, which we can run containers. You don't care about it. You just care that they're all joined together. And it's one big flat space in which we can run stuff. And we'll let this thing, the scheduler, take care of running stuff for us ultimately. And so basically the options for clusters are laptops, multi-node clusters, hosted or even self-managed, on-prem or cloud-based using virtual machines or bare metal in virtual machines. Many, many options. There's a matrix down here, a short link. Hopefully we can share these slides afterwards. And the short link will give you a matrix of how you can run Kubernetes on what you want to run it on, CoreOS on Amazon. We have different ways of doing a networking. The networking is quite tough. Google Compute Engine makes it easy because of IP addressing. But often we have to put this other layer in called Flannel to actually provide that ability to give an IP address in a group of subnets to a running machine or a running pod. So let's talk about pods. How many of you are familiar with the concept of pods? OK, not so many of you. So in the diagram here we have a pod. It has a container. This is a container, this web server is a container. And it has a volume, like Docker containers can have volumes. A little bit different, but very similar. And so we want to run this web server. And the construct we use within Kubernetes is to create this thing called a pod. It's like a logical host. So like if you wanted to run Apache and something else alongside it, you would run it on a host machine. That's the same as a pod. So anything you would run together on the same machine will run in a pod. These are the atomic units of scheduling for Kubernetes. This is what Kubernetes schedules. We talked about jobs earlier when we looked at bulk. Kubernetes schedules pods. And your containers run within the pod. So thin wrapper around them. These are ephemeral. These are like, I've got this analogy. So everybody uses this pets versus cattle analogy. And I don't really like it from a vegetarian. So crops versus flowers. So pods are like crops. You don't care about them. You have a wheat field. You don't care about your individual plants that are growing. When you have flowers, you probably give them names and you water them and you talk to them as well. So you care about them. You don't care about your crops though. So pods are like crops. They can come and go. They can be replaced. They're all absolutely the same. You can take one and replace it with another. And ultimately, to make things simple now, you don't have to worry about a pod if you want to run a single container. You just say, run this container for me. It will create a pod for you. And you still have to think in terms of pods when you're doing monitoring, but you don't have to create a pod. You just say, run the container for me. It will create a pod for you. OK? So pods are an abstraction. Difficult to get ahead around. A little bit more information about them. Imagine this scenario where you want to have something that synchronizes with GitHub. This may be a push-to-deploy type scenario where whenever your developers do a merge into GitHub, you want those changes to be immediately pushed out into production or maybe into a staging service. So you have a thing called a Git synchronizer. And it's talking to Git, monitoring your project in Git. It pulls down any changes. And it writes them to somewhere on the disk. And your web server can then serve that latest content. Those things are tied together. They work together. And it makes sense for them to run side by side. And when one goes away, the other goes away. So we can run them both in the same pod. So now we're saying, on this logical host, this pod thing, let's run two containers. In this case, Git synchronizer and a Node.js app or a Python app. And we have a shared volume, a concept of a volume, which we'll talk about shortly. These are tightly coupled together. So when one, when a pod dies, they die together. It doesn't make any sense to have them running separately. It might do in your, in the way you architect things, but it doesn't have to. They share the network space and port space. They have the same concept of local host. They are completely ephemeral and think in terms of things you would run on a single machine. So a volume, what's a volume? I don't normally talk about volumes, but they are very important, so not talking about them seems a bit stupid, really. So a volume is basically bound to the pod that encloses it. And this is something where we can write data or read data from. Okay? And we have many options when it comes to volumes. Docker already has volumes. This is slightly different, but very similar. So to a container running in the pod, the volume looks like a directory. And what they are, what they're backed by and such like and when amounted is determined by the volume type. So the first type we have is an empty directory. So whenever we create a pod, it creates this space somewhere on disk, on the local disk, and they can basically share that volume between them. But it lives and dies with the pod. It only exists while the pod is there. So it could be your git synchronizer is writing stuff to this volume, being read by the Apache server or whatever server. And you don't care when the pod goes away, if that space goes away. It's just scratch data, just temporary data. There's nothing stored there that's important to you. And it can't even be backed by memory as well. So it could be tempFS file system. And that's great. It's really efficient, much faster as well. So that's what an empty directory is. That's the default you get for a, well, I don't know what it is, a default, actually. That's the specified type it is. So empty there is one of the options. The next one is host path, where we can actually map part of the file system of the node on which the pod is running into the pod. So this volume is actually effectively a snapshot of, not a snapshot, a link into the file system of the actual running machine. That's useful to read configuration data and stuff. But it's also kind of dangerous as well, because it may be that the state on the node may change in such a way that whenever you run a pod on one machine to another, you don't run it. Whenever the scheduler runs the pod on a different machine, it may see a different view of what's happening. So it no longer becomes completely isolated. So it's a kind of dangerous thing to do, but it might work for you. The other one is nfs and other similar services like GlusterFS. I can never say that. And if you're a G on it, I can't say some reason. So again, nfs, we can mount nfs paths on our pod and expose them to our containers as directories. Or we can also use a cloud provider, persistent storage, persistent block storage. Now we call them persistent storage in Google. Amazon call them elastic block storage, that kind of thing. So this is persistent disk. So basically they can write and read from the data from the disk, and it will always be there, whether the pod goes away or whatever. So what we're likely to do in this case is create a volume, a volume in the cloud provider. I call it a disk. We create a disk in the cloud provider which stores data and we'll mount it onto the pod. Whenever that pod goes away, the data is still there. The pod comes along and can mount it as well. And also with Google Cloud Platform, you can actually mount and read only on multiple pods as well. So some patterns for pods. The first one is the sidecar pattern. Because basically it's motorcycled inside car. I guess in this case the Node.js application or the Python app is the... You don't get offended when I say Node.js, right? The Node.js application is a bike and the git synchroniser is the sidecar in this case. That makes a lot of sense, right? Ambassador, in this case, something that acts on behalf of the actual running container. So this is a secondary container, a Redis proxy that effectively allows the PHP application to make calls and then have the Redis proxy call that to Shards. So we can just make... Have one service that the PHP application calls for reads and writes and the Redis proxy can do with the hard work of deciding whether to read from a master or read from a slave or write from a master. And the final one is an adapter pattern where in this case we have Redis running and we want to monitor it. We want to monitor all of our pods but we need a common format for monitoring. So in this case we actually adapt the output from the Redis monitoring using an adapter container. An adapter container will be plugged into the monitoring system. So it kind of adapts what's happening within the container. So these are kind of examples of where it makes sense to have a pod. I'm hoping it does make sense and I'll be interested to hear from you afterwards about whether pods make sense to you. So labels basically the single grouping mechanism within Kubernetes. This allows us to group things that we can build applications like a dashboard. So we have a running pod. We give it a label. Labels are key value pairs. So in this case type equals fe. Completely arbitrary metadata. Some of these things are meaningful to Kubernetes but mostly it can be anything that's meaningful to you. So we've put labels on pods and we can say I can build a dashboard application that uses the API to say give me the pods with this label. And I can show you all the status of that. And we can have different labels for different pods. In this case we have a version two pod. We have a different dashboard application with more different nodes. And that makes a lot more... Pods can have many labels. I surprise myself with my slides sometimes. Makes more sense with replication controllers because replication controllers are things that actually manage the running of pods. Now remember I said before that we created 10,000 tasks and we pushed them out to persistent storage in a ball master. And the scheduler comes along and says, yeah, these should be running but they're not. I'll fix that. So this is the same thing. The replication controller is responsible for managing your desired state. You say this is the way I want it to be. I want to have X number of these pods based on this container template. I want you to maintain that state for me. That is the job of the replication controller. So basically what they do is they work on a constituency of a label type. So in this case version equals V1 is what they select on. This replication controller is responsible for all pods with label version equals V1. And we tell it I want to have two of those. So this job is to make sure there's always two running. In this case we also have another replication controller. That has V2 of our pod. Version equals V2. I only want one of those. So make sure there's always one of those running. And the way it works is this is kind of like a control loop. The replication is one big control loop. Simple as that. It says look at the desired state. How many of we got running? We should have four running. We got four running. We got three running. That's not good. Let's start another one. We have four running. We have five running. That's not good. Let's take one away. It's basically monies the state to make sure we had the ones running. It also works with a template. So we provide a template, which is the pod template, which contains the container image definition. And how many we want to run. We pass that into the replication controller. It doesn't create the pods. But when we create the replication controller and we say we want two of these pods, it says, hmm, there's not two of these running. I should start them. So it starts them. That's how it works. And we can also plug in replication controllers after we've created the pods. And say, you're managing containers with this label. And finally, we get to services. And services are how we actually expose our running stuff. And we do this through this service here, which creates a virtual IP address, which has a constituency of pods based on the label selector. Again, we're going to have labels on here. We'll show it in the next slide. So basically, certain pods with a certain label are the constituency of this service. And when requests come in from clients, it will load balance them across the running pods, regardless of which node they're on. So they could be 10,000 nodes. We could have 10,000 pods run in different nodes. And it would load balance them across the running pods. At the moment, it only works around robbing. But eventually, it will have much more intelligence support for load balancing. This is used for exposing internal services within Kubernetes and also expose mining services to clients externally, which we'll see shortly. It not only provides a virtual IP address, but also a DNS name so we can do service discovery. And we want to move on. So this is a Canary example. So who understands the concept of Canary? A few of you. So basically, when you have a situation where you have a running application, you want to try out a new version of it. You may have one instance or two instances of that running application that are different. So some of your traffic will be pushed to the new versions. Some will go to the old versions. You can then do A, B testing against them to make sure that the new service works. If it doesn't, you can roll it back. If it does, you can push out the change for all of them. This is a similar situation where we have version equals V1, version equals V2, replication controllers and pods. But a service, all it cares about is labels type equals FE. And so the service has its constituency of all three of these pods. But these pods are managed by different replication controllers. So that's how it works. Virtual IP address exposes that to a client. And so we map the Kubernetes. It all looks kind of like this. We have pods. Remember all the symbols? I believe this is why it's important. So that's a pod and a volume and a service. And we have all a memcache vd drop down. A pod, a service and replication controller with a service. How does that look to a developer? So remember how it looks to a developer on Google? So this is how it looks. They specify a name. They can specify the image. This is a Docker image now. It could be a different image format in the future for a different type of container format. I left it in deliberately to stop, so you. Yeah, PHP gets back. You can specify resources. 128 maybe bits. Maybe bits. You can specify how much CPU. And the Kubernetes unfortunately had its own idea of slicing off a CPU. And I'm not going to get into it, but it's like 500 bits of a CPU in terms of Kubernetes. So you have to read the manual for that. Otherwise it won't make any sense. I'd rather have a percentage, but that doesn't work because you can't have a percentage of a core because you don't know how powerful it is. So that's how we specify CPU. The ports, protocol, TTP, and the replica is one or maybe $10,000. We covered that case as well. So that's how it works within a replication controller. There's other configuration files as well for services. Oh, sorry, do you? And scheduling at the moment, we saw the complexity of scheduling at Google. It's a bit simpler for Kubernetes currently. It's based on pod selection. So we want to have the pods run in that are based on selectors, and it's based on node capacity. So how much capacity does that node have? Is it capable of running my pod for me? If I have multiple nodes that can run my pod, I'm going to run it on the one that has the least resources consumed by running pods. And that's the priority. In the future, we'll have resource-aware scheduling. So we can do what we do back in Google where we try to make maximum utilization out of our CPU and memory. Kubernetes is 1.0 as of this week. Now it's on 21st of July at OSCON in Portland, Oregon. It's been open-sourced for over a year now. And we have a product called Google Container Engine, which I'm going to talk about shortly, not so much, but it is a good way of running Kubernetes. But it's not a product pitch. Hosted Kubernetes, I'm going to talk more about Container Engine shortly. And the roadmap for Kubernetes is there. It's kind of sparse at the moment because we've just gone through 1.0. So they're now deciding on the roadmap for the next releases, V1.1. And the one on the roadmap currently is autoscaling. The ability to autoscale your nodes dynamically based on the amount of work you have. Container Engine is a managed version of Kubernetes, and it manages uptime for you. You don't have to worry about the master in this case. It will take care of the master for you. You can't even see the master. You can't connect to the master. So one of the problems we have at the moment with Kubernetes is high availability. So we don't have that replicated master scenario we saw with Borg. So the only way to do it is to have multiple clusters to do high availability. But if we look after your master for you and make sure it's running, then you don't have to worry about it. We will make sure that your cluster is highly available. By making sure your master is always running. We can resize using things called Managed Instance Groups, which we'll look at in a minute. Centralize Login. We can pull all of our logins to one place in the Google Developer Console. And it also supports VPN. So you can actually have your pods inside your own network, your own private network. So demo. Very quickly. And we had to change the setup earlier to make all this work. And this is a cluster. We have kubectl get nodes. So we have two nodes running. So these are machines in our cluster. And I can look at them here. This is the Google Developers Console. And I can probably make that a bit smaller. If I go into VM instances here, I can see my running machines. I have a couple of extra machines as well. But these two in the middle are the nodes for our cluster. I have this thing called an instance group, which has two instances. This is the thing that manages the size of our cluster. And below here, we have container clusters. And we can see we have one cluster. OK. And if we go to... Wait a second. I've got very little screen real estate, so I can't see everything that's going on. So we go here, we can see a representation of what's running currently. So we have... These are pods. So this is a pod. This is a service. This is exposed internally. This is a service. And this is another service. Oh, MySQL is not running, which is a real pain. I'll have to run it. OK. I don't know how that happened. OK. So we have a front-end service. We have a memcache service. And we have MySQL. We don't have a pod running. So we need to have a pod running. So I'm going to start the pod very quickly. And then... OK. So I'm going to start the pod very quickly. OK. That's why it's not running. So we've just gone to 1.0. All my demos break. That's fine, though. Should be fine. I had it all running, but we had to reboot my machine because we were having problems with the display. And then we have a pod. Hey, pod. MySQL. Have you ever spun up MySQL so quickly? I bet you haven't. The next thing we want to do is run some PHPs. Unfortunately, there are PHPs currently. But I'm trying to get time to update them completely. But I had some problems with flask and Angular. So anybody else had problems with flask and Angular? No? OK. I should talk to your... Basically, on my badge here, it says that my Python skills are rated as free stars. So I probably should talk to you guys about doing it. OK. So I'm going to create a control. We have a file already created. And we're going to create that. And now we have pods and a replication controller. I need to make that smaller. So now we have some frontend pods and a frontend RC controller. So the next thing we want to do is actually look at the running application because my windows are all screwed up. We have it running. So this is the IP address of the service, as we can see here. And this is the application running. It's DevOps. Anybody who's been to DevOps, you don't want to go to DevOps. You want to go to the EuroPython. I told them to fix this beforehand. We have an update and we can roll that update out really easily to our cluster. Let's do that. Let's roll out an update to our cluster. And I'm going to close that down so we can see the visualization. And I will go and reverse my history for this. And I'm going to update to v2 of our frontend controller. So what's going to happen now is it creates a new controller. And now it's going to change those pods one by one to roll out our new version. So now we have three pods, the 2.0 and 2.1.0. We're going to get rid of one or 1.0. We're going to have a 1.0 and a 2. And then we're going to create a new 2.0 pod. And then we're going to get rid of the other 1.0 pod. And eventually we only have two 2.0 pods. And we get rid of the 1.0 controllers. We don't need that anymore. And if we go back to our app, nothing's working. And refresh. And we should get... Yay! APPLAUSE I'm hoping it works. I'm hoping my SQL is running properly. Yay! So, OK, that works. Brilliant. The other thing I can do as well, I should mention it. I'm probably getting close to my net time. What is the command? Is it RC or not? I don't remember. I never... Why should I get the SQL command? So I'm going to do B2. And I'm going to scale it to six replicas, the five replicas. And we go back to our biz. So now we want to add replicas to this. We can do that by scaling like that. And then we have five running pods. OK? It's as simple as that. So now we have five running. We can do that also within the developers console. And just to wrap up on the whole thing, just a quick talk about the last bits and pieces. That's how we visualized it. We visualized it using the API and a proxy. So kubicostal supports a proxy. We just pointed at some JSON. The JavaScript is all JS Plum. So if you want to know what we use, JS Plum. In terms of container engine, cluster scale in, we have this thing called a managed instance group. And that runs all of our nodes. And nodes run within the managed instance group. And we have this thing called an instance group manager that creates them and is responsible for making sure they're running. So that's actually monitoring the cluster of nodes. And we have a template by which we can create new nodes on demand so we can resize that managed instance group very easily. And yeah, I think that's about that for cluster scaling. We can also create clusters using tools such as the Google Developers Console, Google Deployment Manager, and Terraform. I was going to give an example, but it's very basic. Terraform will create a cluster for you, but it won't allow you to resize it. If you want to resize it, you have to replace it completely, which isn't really what you want to do. So you can create clusters with various different ways. And oh, that's a visualization. Some frequently asked questions are answered in the documentation. I could spend entire hours on all of these subjects. So if you have questions, I'll be outside all day on the Google booth. Come and see me. And Kubernetes is an open source, so we want your help making it even better. So please contribute to Kubernetes. If you have questions, go to IRC, IRC.free.net on hash Google containers. It's a very popular place. And also on Twitter, Kubernetes.io. You can tweet questions for me. If you don't answer them now, or you can find me on the booth. And ultimately, that's it. Thank you. We have time for one or two questions. At the beginning, you were talking about Borg and like five masters that you run in. And those figures are based on the data center or how does it? Based on the cell. So we break it up into a cell and each cell has its own Borg master. And that's about my limited knowledge of how the complexity of how Borg works. Not being a sweet one, sorry. But yeah, that's how it works. That's why we have multiple. So within Google, you're going to find multiple Borg clusters or Borg cells effectively. Hi. Thank you for the talk. Very interesting. When you compare VMs and containers, even if the user of a VM has root access, it's very difficult to escape from the IPvice or et cetera. How do you see the security in the current containers implementations? It's a work in progress. So this is about security with containers. I'm not really going to comment too much on it, but it's getting better all the time. Initially, we had problems with the kernel level and syscalls and such like being made back into the operating system. But it's getting better. So ultimately Docker and such like becoming more secure all the time. Ultimately, doing multi-tenant maybe currently with multiple customers applications run inside by side may not be the best idea. But we have to tackle that. So ultimately, we have to make sure people are more confident that they can run all of their jobs on containers securely. I don't think we're quite there yet, but we're working on it. So that's one challenge we need to crack. Is that a question? Yeah. One more question? Is that done? No, that's enough. OK. Come and find me outside. Come and find me outside. We can talk about PHP. It's Python, sorry. From our organization, we want to thank Mando to come in. Oh, wow. Give her a present. Thank you. Should I answer it? Should I put it? No. Oh, no. That is wonderful. Ah, fantastic. Exactly what I need. Thank you very much. Thank you. Thanks for having me. Thank you.
 Thank you for coming and welcome.  Thank you.  Hey, thanks, everyone.  You all hear me?  I think you can.  Good.  Excellent.  Thanks for having me at Python, EuroPython.  My last big Python event was Python, PyConJP in Japan last year.  I didn't get to speak, though, but it was really fun.  Although most of the talks are in Japanese and my Japanese is getting better, it's not  so great.  My Spanish is really, really bad.  But Spanish and Japanese are very similar, so maybe I should learn both together.  No, really, seriously, they are.  I know it doesn't seem to make sense.  We're going to talk about containers and having containers and having lots of containers,  because ultimately, everything is going to be containerized and we're going to have  lots of containers.  We won't know what to do with.  And I'll ask you some questions, Ada, and see how far you are along with moving towards  containerization.  So basically, when we have lots of containers, what do we do then?  And this is a problem we face at Google.  So this is a data sensor.  This is a Google data sensor in Iowa in the US.  It's a place called Council Bluffs, and this is one of our bigger data sensors.  And if I leave it out for a long enough, you'll probably be able to count all of the machines.  How many there are.  But this is a cluster.  So clusters are one of the constructs we have internally, but these clusters are broken  down into cells.  So cells are smaller.  We have many cells per cluster.  And this will probably, a cell we're going to look at today, is going to have about 10,000  machines in it.  So they're quite large.  And this is a huge amount of compute power.  Lots of compute power, but we need to make this available to our engineers, our software  engineers, our developers.  So how do we go about making this compute power available to our own developers?  And it works something like this.  This is what a developer does.  Well, first in context, the one thing, given what you see there, we don't want the engineers  to have to kind of select a rack, select a machine, and say, hey, I'm going to run it  on that machine.  I'm going to SSH, SFTP a binary over to the machine, SSH into the machine, stand up my  process, my server or whatever, maybe log into many machines and do that multiple times.  But that's not going to be possible.  Huge amounts of machines, huge numbers of engineers, huge amounts of jobs to run.  So how does it happen?  So basically, we have a configuration file.  In this case, it's called a Borg configuration file.  I was in India recently, and nobody there had heard of Borg.  How many of you are familiar with Borg and Star Trek?  Okay, right.  So we never used to be able to talk about Borg because Paramount Pictures own it.  It was kind of like one of our worst kept secrets that we had this thing called Borg  running internally.  But now we talk about it all the time.  So because it's fun, and it's really good to show this in the context of what we're  going to talk about later, which is Kubernetes.  So basically, this is a Borg configuration file, and what the developer does is he creates  a job.  Jason file, calls the job Hello World, says which cell he wants to run it in.  Going back to what we said earlier, a cell is a few thousand machines.  In this case, he's saying it's called IC, some random cell name we chose.  And he specifies what binary to use.  In this case, Hello World web server.  So he wants to run Hello World on a web server.  And this is going to be a fat binary.  Statically links all of its dependencies with it.  So effectively, we can run it pretty much anywhere without having to worry about the  underlying operating system.  And that includes the web server as well.  So this thing is quite big, probably about 50 megabytes.  So he specifies the path to his binary, or her binary.  And unfortunately, we have too many male software engineers, not enough female software engineers.  So let's encourage women to be software engineers.  And arguments.  We have to specify some arguments for our binary, pass them in via the environment.  In this case, we want to specify what port to run on.  This is parameterized.  Then we have some requirements in terms of resources.  Now, this is important.  We'll circle back to this in a minute.  So we can specify how much RAM, how much disk, how much CPU.  And ultimately, we can say how many we want to run.  So in this case, we want to run five replicas of this job, five tasks effectively.  And why five?  Why not do it at Google scale, 10,000?  Makes more sense, right?  We have all those machines.  We saw how many machines we have.  So let's run 10,000 copies of this.  So once we finish this, our software engineer, she types in a command on the command line,  passes in the config file.  And that gets pushed out to somewhere.  It gets pushed out to this ball scheduler.  And what happens then is this.  Over a period of time, in this case, about two minutes, 40 seconds, 10,000 tasks start.  10,000 instances of that job start.  And it takes two minutes, 40 seconds, roughly.  We do phase the rollout of all of these jobs to make sure we don't do them all at once.  One of the key factors here is the size of the binary, 50 megabytes, 10 times 10,000.  It's about 20 gigabytes per second of IO.  We're going to be cashing that binary quite a lot, but we had to move it around between  10,000 machines.  So there's a huge amount of IO going on.  But eventually we get to a point where we have 10,000 running, or nearly 10,000.  Maybe not quite 10,000.  We'll talk about that in a second.  And Borg looks like this.  This is what Borg is to Google.  It's not going to assimilate you, but I think we came up with a name because it's probably  going to assimilate everybody eventually.  So this is Borg.  And Borg runs within a cell.  So each cell has its own Borg master, its own Borg configuration.  In this case, we have a Borg master, which is highly replicated.  We have five copies of it for resilience.  And we have lots of other things.  These down here are our machines.  These are our machines we saw in the racks.  They're all running a thing called a Borglet.  We have a scheduler.  We have some configuration files in the binary.  So what happens is the developer, the engineer, creates his or her binary.  And they use a massively distributed parallel build system called, well, I won't say what  it's called, but it's externally available now called Bazel.  So we made this open source.  So our own build system is now available open source called Bazel, B-A-Z-E-L, or if you're  American, B-A-Z-E-L.  It gets very confusing, believe me.  If you go to Canada, it's so confusing.  Like routes and routes.  So basically, he or she creates the binary, pushes it out, and it gets stored in storage  for the cell.  And then they push their configuration file.  Configuration file gets copied to the Borg master.  We have a persistent pack source back store, consensus based.  And what happens then is this scheduler, looking around, comes along and says, hey, what is  the desired state?  We should have this running.  Do we have this running?  And it sees 10,000 new tasks.  And it says, hey, they're not running.  We should have 10,000 of those.  Let's make sure that's happening.  Let's fix that.  And so it goes about planning the running of these 10,000 tasks.  And it creates a plan, and then it starts telling the Borg master what the Borg master  makes decisions and tells them the Borglets on these machines to run this particular task.  So they get communicated.  The task will ultimately run inside a thin container wrapper.  So it has a container around it.  It's not just running the binary.  It is containerized.  A very lightweight shim container that's not Docker.  It's not standards-based.  The Borglet ultimately will pull the binary over from storage, and it will start running.  And we'll see this.  Lots of Hallowels.  All over our data center.  So now we're running multiple copies of that.  And so that's what we had, 10,000.  But if we look at it a little bit closer, we find there's 9,993 running.  Not quite the 10,000 we expected.  But this is a highly available service.  We expect some lessening of the number of tasks we're running over time due to the way  we operate.  And that's interesting.  So let's look at that in a little bit more detail.  So failures.  Things fail.  But failure is kind of more of a generic term here.  There are many reasons for failures.  And one of the main reasons for failures, particularly for low priority jobs, is preemption.  If we look at the top bar, which is our production jobs, we have very few failures.  And most of them are down to a machine shutdown, where we've actually scheduled some maintenance  on the machine, and we've taken the machine down.  That task, any task running on that machine, we don't be rescheduled elsewhere in the cluster.  We have a very small number of preemptions.  Down here are non-production jobs, which are things like map producers, batch jobs.  They get preempted all the time.  They're happy to be preempted.  And in fact, the calculation generally says that for about 10,000 tasks, about seven or  eight of them will be not running at any given time because of preemption.  They'll be about to be scheduled somewhere else, but they won't be running at that particular  time.  And we see other things here.  We see, again, the blue bar, which is the machine shutdown, which is pretty much the  same as production.  And we have some other things as well, out of resources, very small number of machine  failures.  And for when you have as many clusters, the many machines as we have, machine failures  are given.  We expect that.  We don't panic when machines go down.  It's part of the normal running of our business.  And another interesting thing is how we try to make efficient use of our resources.  So we have CPUs, we have memory, we have disk.io, we have network.io.  And sometimes it's quite possible for one task to be using lots of memory for very little  CPU or vice versa, lots of CPU and very little memory.  If you put one of those on a machine, then you may be wasting one of those resources.  It's what's known as resource stranding.  And these are the available resources, these white bars here.  This example here is actually our virtual machines, which are tasks.  Our virtual machines are actually containers, believe it or not.  It's a Google computing engine.  So these are all virtual machines, these bars, individual bars.  And what we can see here is that some of these machines have available capacity, available  RAM, available CPU, and if we look over here, we see a different situation where we have  maybe some with available CPU and others with no available RAM and vice versa.  This here and this here is called resource stranding.  It means we're not actually making use of that resource.  So we have spare memory capacity or spare CPU capacity that's being wasted effectively.  So one of our challenges is like a Tetris puzzle to try to stack these things in a way  where we get the best possible utilization out of our clusters.  So we will mix and match them to make sure we have low CPU, high memory jobs running  with high memory, low CPU jobs.  And of course, we run multiple tasks per machine.  That's extremely important.  That can then come back to all this with Kubernetes shortly.  And another interesting thing is this, which is going to be a huge challenge in the future  when it comes to Kubernetes, but it's going to be really important to all of us.  So we saw earlier that our developer, she specifies what resources she wants to use  or he wants to use.  100 megabytes of RAM, 100 megabytes of disk, 0.1 CPU.  And that will be this blue line up here.  So everything's running.  We'll match into this blue line.  These are the resources that were requested by these jobs.  In reality, though, it's like this.  And so we have all of this wasted space, which we can't use because it's been allocated  effectively for those running jobs.  But we can use it.  So what we do is we effectively estimate, based on the run patterns of the current jobs,  how much they're going to use.  And that's this blue line here.  So this is our reservation.  So this is how much we reserve specifically for those jobs.  And what we can then do is reuse all that space.  Now we can reuse that space for very low priority jobs.  Again, those batch jobs, those map reduces.  Things that we want to run, we want them to finish eventually, but we don't really care  when it happens.  It could be like running some kind of monthly report that nobody ever looks at that gets  logged or running a map reduce across a huge amount of data that may be important at some  point or just needs to be done, but we don't really care when it needs to be done.  So all of that stuff, we can reuse it and we can run jobs within it.  But that's really important.  That's how we can get maximum utilization out of all of our machines in that data center.  And so moving on to Kubernetes now, gradually, one of the observations is that if you have  your developers spending time thinking about machines or thinking in terms of machines,  and you're probably doing it wrong, because it's too low a level of abstraction.  Now today, maybe it's fine, but in the future, this is not going to be the case.  We need people to be thinking in terms of applications and not having to worry about  the infrastructure in which they run.  I mean, anybody who uses a platform as a service knows how important that is anyway.  You don't care about the infrastructure.  You want to write your work, configuration file, build a binary, and just say, run this  for me.  I don't care where you run it.  I don't care about how you do it.  Just run it for me and make sure it stays running.  We get efficiency by sharing our resources and reclaiming unused allocations.  And containers, the fact that we containerize everything, allows us to make our users much  more productive.  So everything we run runs on a container.  Two billion containers a week, we estimate.  We never really thought that was very important until Docker came along.  Containers became the next big thing, right?  Alex, then Docker.  And Docker became huge.  So now one of the things we talk about all the time now is we run containers all the  time.  We are pretty good at running containers.  Which is why we created Kubernetes.  If you're interested in more details of what I've just talked about, Borg, there's a paper  here.  G.gl, one capital C for N U O.  That's the white paper in Borg.  That's got all of the details, all of the graphics you just saw.  It goes into much, much more detail, of course.  So let's look in terms of a simple application and how we can do this externally with containers  and through Kubernetes.  So this is a very simple pattern.  Generally when we give this sockets PHP in the middle, MySQL, Memcache, and we have a  client.  We have many of these pythons running.  This could be many instances of flask, it could be some kind of event system, but we  have the ability to run many, many concurrent requests.  We're probably going to want to scale this thing on demand.  We may not want to scale MySQL that much until we get to a point where we have to do replicas  and sharding.  Memcache, we've probably gone on a scale as well, but we're going to keep it simple  for now.  Just keep one MySQL, one Memcache, and a few of these pythons that are at the front end.  So let's talk about containers.  So how many of you are familiar with containers?  How many of you have actually spun up a Docker container?  Hey, lots of you.  It's almost the same number, right?  Again, last year we'd asked this, how many of you have heard of containers, lots of hands,  how many of you have spun up a Docker container?  Lots of many.  So things have changed now.  Docker is the future.  Well, containers are the future.  We now have this thing called the open container project.  Docker have kindly made what they have into a spec, and we're all going to get behind  it.  We have this common specification from which we can write containers.  Things like CoreOS with a rocket container, they're all going to fall in line and we'll  have a common format for containers.  Which is going to be great.  But just for those of you who are not really familiar with containers, just a few slides,  very few slides on containers.  Just kind of give you some of the concepts.  This is the way we used to do things in the old days.  We have a machine, maybe next to our desk, in our bedrooms or in a colo or in a server  room.  And the machine would run our operating system.  It would have all of the packages installed at provided libraries, things like open SSL.  On top of that, we would run applications.  And how many of you have had a situation where you're running one application and all of the  other applications on the machine fail because that one application went mad?  Use all of the CPU, use all of the RAM, it crashed the machine and took all of the other  applications down.  And this may have been a very low priority app, one that you didn't really care about  taking down some really important ones.  Now, this is never a good idea running multiple applications on one machine because there's  no isolation between them.  Whatever affects one application will probably affect all of the others.  There's no namespacing.  They all have one view of the machine in which they're running.  They have one view of the CPU, one view of the memory, one view of the file system, one  view of the network.  They share libraries.  And so you're in a situation where maybe one day you update a version of a package, it  updates a library, and one of your applications says, hey, I'm not going to run anymore.  That library is not compatible with me.  So dependency hell.  And if it's on Windows, it's DOL hell, and it's probably even worse.  Applications are highly coupled to the operating system.  This is a problem.  And so we created virtual machines.  And what we did basically is stuck a layer on top of the hardware called a hypervisor.  And we now had an idealized piece of hardware on which we could run multiple operating systems.  So now we have this thin layer, it looks like a piece of hardware to be running virtual  machines.  And that gives us some isolation.  So now we can run applications in their own virtual machine.  So each application is now isolated.  If one application crashes, it doesn't affect the others.  But it's extremely inefficient because we have this red bit at the bottom here.  We have the operating system, the kernel.  And you know when you install a virtual machine, you pretty much have to install the entire  devian stack or the entire center stack or the entire Windows stack.  So that's not very efficient at all.  They're still the same type coupling between the operating system and the application.  And as anybody who's tried to manage lots and lots of virtual machines to provide isolation,  you know it's hard.  So there are new ways containers.  In this case, we move up a layer.  So we move above the operating system and provide an idealized operating system.  No longer idealized hardware and idealized operating system on which we can run apps  and their dependent libraries.  So the libraries here are part of the container.  So the container has an application.  It has all of its dependencies.  It has its entire environment.  So we can move this container around anywhere we want to.  We can move it from one machine to another, from one runtime to another, from a laptop  to a virtual machine, to a cloud, to a bare metal server, to a set-top box, ultimately,  maybe even to a phone when we had Docker on Android and iOS.  I'm sure it's going to happen, right?  And let's look at the example.  So we have our application, PHP in Apache.  It should be Python in Apache.  Sorry.  See?  I do apologize.  So wherever you see PHP in this deck, read Python.  We'll change it before I share the slides.  So I'm trying to think of what could offend a Python audience most.  It's probably talking about PHP, right?  My god.  Okay.  So we have containers.  We want to run these components of our application, Python and Apache, Memcache, MySQL.  Not Apache.  Obviously, Python and Vlask and Bottle and all of the other things we could potentially  use.  Memcache, MySQL.  And MySQL has its own libraries.  It doesn't have any common libraries with the others.  So we can stack those libraries with the container in which MySQL runs.  And Memcache, PHP, Python and Apache have their own, I keep saying it.  And Python and whatever.  Unicorn.  Anything.  They have their own dependencies.  But they also have when we install them some shared dependencies as well.  So some commonalities.  So when we actually create the image, we can actually share some stuff between them.  But that's not shared at run time.  So when we create the container, they will have their own dependencies packaged together  in the container.  And underneath that, we have a server.  And again, this could be a virtual machine.  It could be a laptop.  It could be a bare metal server.  It could be anything, pretty much.  And underneath it, we have the actual hardware.  And all of this is being maintained by a Docker engine.  So Docker is the thing that runs this.  So when we talk about containers, mostly synonymous with Docker nowadays, but again, there are  other container formats.  And hopefully, they will all comply with a standard.  And that's the Nirvana we're all heading towards.  So Docker effectively controls the creation of these containers and the management of  these containers.  So at the end of it, we will have Python, and Flask, Angular, Memcache, MySQL, all  running in containers.  So why containers?  So there's many important reasons for having containers.  But you can see, just by looking at what we do, that's the only way we can do it.  We can't do it any other way.  This is perfect solution for the kind of scale that we want.  But it's also perfect for smaller scale as well.  Why?  Because it's much more performance.  It's much more performance in terms of the fact that we don't have to do all of that  installation stuff.  They are pretty much like they're running on bare metal.  So the performance is pretty much the same as a virtual machine, but they're much quicker  to get up and running.  Which means you can swap them out quicker.  You can do upgrades quicker.  You can do pretty much everything quicker.  Repeatability.  So the whole problem where we have development, QA, build, test, production, where we want  to have repeatable environments, where we have a situation where when we test something  in QA and then run it in prod, it fails in prod where it works in QA.  How many people have had that situation?  Have you had any hands remembering those days?  So what containers give us is the ability to have a consistent environment because the  environment is packaged with a container.  So basically when we run it in QA and when we run it in prod, it's exactly the same.  It's exactly the same environment.  So that's one of the great use cases of containers today.  But much more is the portability of it, which we're going to talk about in a second.  Quality of service.  We can now do resource isolation as well.  Using things like C groups and Linux and namespaces, we can actually isolate the resources.  We can say we only want this to have 100 megabytes of RAM, 100 megabytes of disk, 0.1 CPU.  And ultimately accounting.  These things are easier to manage.  They're easier to trace.  They're easier to audit.  They're small, composable units that can be tracked very easily.  And ultimately, portability.  You can move these things around from one cloud provider to another.  Images specifically.  You can't just pick up a running container and move it, but you can easily run the same  container in a different cloud provider on a bare-measure machine on the laptop.  You can move them from one machine to another as the shape of your cluster, if you have  a cluster of machines, changes.  You can move them around to be more efficient.  So we can go back to what we had before with the efficient allocation of resources.  We can do that if we have containers.  And ultimately, this is a fundamentally different way of managing and building applications.  So demo.  I'm not going to do this demo.  I left that slide in by mistake.  This would have been a containers docker demo.  And I don't think I want to bore you with that.  It's very easy to find a tutorial on docker and get up and run them a bit.  Now, let's not talk about that.  Let's talk about Kubernetes instead.  How many of you have heard of Kubernetes?  How many of you can say Kubernetes?  It's hard word to get your head around.  Probably easier if you agree, because it's a Greek word.  But if you want to help pronouncing it, I'll be outside in the Google booth after this  talk.  So I can definitely provide assistance on that.  Maybe I'm saying it wrong.  Maybe I've been saying it wrong all this time.  So I'm happy to be corrected.  So Kubernetes, let's talk about that.  And we've given you an introduction to what we do at Google.  So that should provide the context on why Kubernetes is necessary.  What we often miss out, is that we don't provide that kind of context.  So I'm hoping that the introduction to Borg has provided that for you.  So Kubernetes is Greek word, means helmsman.  It's the root of the word governor.  So Arnold Schwarzenegger's governor comes from Kubernetes.  And it's an orchestrator or scheduler for docker containers.  Ultimately for other forms of containers, I think ChoraOS is already using it to schedule  orchestrate rocket containers.  It supports multiple cloud environments.  So Metasphere, I always forget them.  VMware, even Microsoft involved.  You can run Kubernetes on Amazon.  You can run it pretty much anywhere.  You can run it on your laptop with Vagrant.  So you can just create a four machine cluster, virtual machines with Vagrant up.  And you'll have a Kubernetes cluster.  And ultimately, eventually, we may have a situation where we can run Kubernetes across  multiple cloud providers.  It might be difficult.  It might be possible.  But it may be one day you'll have your fleet of machines will be running in Google, in  Amazon, and Microsoft Azure as well.  Possible.  I'm not sure if it's going to happen.  So this is kind of inspired and informed by everything that we saw previously, everything  with Borg.  And it's basically our experiences.  Open source, ridden and go, like many good programs nowadays, but completely respect  Python.  I love Go.  I love Python.  I used to be a Java developer.  I spent 15 years developing Java.  No, 11 years.  Now I moved to Google.  I haven't wrote a line of Java code since.  Now I write...  Now I write...  It's like Java program is anonymous, right?  It's been four years since I wrote my last line of Java code.  So now I write in Python, go, and write in Angular, and write in Java scripts, and all  of those more interesting, useful languages.  Java is getting better.  Java rate is a big step forward.  And ultimately, we want to be able to talk about managing applications and not machines,  which is actually what we talked about earlier.  And some very quick concepts.  I'm not going to introduce them, but I want to show you the icons so that when you see  them, you'll know what they mean.  Angular, pod, service, volume, label, replication controller, node, are all of the key concepts.  How many of you are familiar with sort stack?  How many of you like the terminology in sort stack, like grains and such like it?  I think it's really hard to get your head around.  And I think one of the dangers about an abstraction is that you get too far away from the terms  that are familiar to people.  Most of these are familiar to people with service, the idea of a replication controller,  a node, a label, a container.  The pod is probably the most difficult one to get your head around.  So let's talk about pods.  Let's talk about nodes first and clusters.  So we have a cluster, kind of maps back to what we talked about earlier with Borg, where  we have a master.  And the master has a scheduler and it has an API, an API server that can be used to talk  to nodes.  The nodes are all running a thing called a kubelet.  And they have these things called pods running containers.  We'll talk about pods shortly.  They also have a proxy by which we can expose our running containers to the outside world.  And we have many nodes.  And a cluster, this is an abstraction.  So a cluster could be different depending on which cloud provider you're using.  And ultimately what you want to have is a fabric of machines that looks like a flat shape,  which we can run containers.  You don't care about it.  You just care that they're all joined together.  And it's one big flat space in which we can run stuff.  And we'll let this thing, the scheduler, take care of running stuff for us ultimately.  And so basically the options for clusters are laptops, multi-node clusters, hosted or  even self-managed, on-prem or cloud-based using virtual machines or bare metal in virtual  machines.  Many, many options.  There's a matrix down here, a short link.  Hopefully we can share these slides afterwards.  And the short link will give you a matrix of how you can run Kubernetes on what you  want to run it on, CoreOS on Amazon.  We have different ways of doing a networking.  The networking is quite tough.  Google Compute Engine makes it easy because of IP addressing.  But often we have to put this other layer in called Flannel to actually provide that  ability to give an IP address in a group of subnets to a running machine or a running  pod.  So let's talk about pods.  How many of you are familiar with the concept of pods?  OK, not so many of you.  So in the diagram here we have a pod.  It has a container.  This is a container, this web server is a container.  And it has a volume, like Docker containers can have volumes.  A little bit different, but very similar.  And so we want to run this web server.  And the construct we use within Kubernetes is to create this thing called a pod.  It's like a logical host.  So like if you wanted to run Apache and something else alongside it, you would run it on a host  machine.  That's the same as a pod.  So anything you would run together on the same machine will run in a pod.  These are the atomic units of scheduling for Kubernetes.  This is what Kubernetes schedules.  We talked about jobs earlier when we looked at bulk.  Kubernetes schedules pods.  And your containers run within the pod.  So thin wrapper around them.  These are ephemeral.  These are like, I've got this analogy.  So everybody uses this pets versus cattle analogy.  And I don't really like it from a vegetarian.  So crops versus flowers.  So pods are like crops.  You don't care about them.  You have a wheat field.  You don't care about your individual plants that are growing.  When you have flowers, you probably give them names and you water them and you talk to them  as well.  So you care about them.  You don't care about your crops though.  So pods are like crops.  They can come and go.  They can be replaced.  They're all absolutely the same.  You can take one and replace it with another.  And ultimately, to make things simple now, you don't have to worry about a pod if you  want to run a single container.  You just say, run this container for me.  It will create a pod for you.  And you still have to think in terms of pods when you're doing monitoring, but you don't  have to create a pod.  You just say, run the container for me.  It will create a pod for you.  OK?  So pods are an abstraction.  Difficult to get ahead around.  A little bit more information about them.  Imagine this scenario where you want to have something that synchronizes with GitHub.  This may be a push-to-deploy type scenario where whenever your developers do a merge  into GitHub, you want those changes to be immediately pushed out into production or maybe into a  staging service.  So you have a thing called a Git synchronizer.  And it's talking to Git, monitoring your project in Git.  It pulls down any changes.  And it writes them to somewhere on the disk.  And your web server can then serve that latest content.  Those things are tied together.  They work together.  And it makes sense for them to run side by side.  And when one goes away, the other goes away.  So we can run them both in the same pod.  So now we're saying, on this logical host, this pod thing, let's run two containers.  In this case, Git synchronizer and a Node.js app or a Python app.  And we have a shared volume, a concept of a volume, which we'll talk about shortly.  These are tightly coupled together.  So when one, when a pod dies, they die together.  It doesn't make any sense to have them running separately.  It might do in your, in the way you architect things, but it doesn't have to.  They share the network space and port space.  They have the same concept of local host.  They are completely ephemeral and think in terms of things you would run on a single  machine.  So a volume, what's a volume?  I don't normally talk about volumes, but they are very important, so not talking about them  seems a bit stupid, really.  So a volume is basically bound to the pod that encloses it.  And this is something where we can write data or read data from.  Okay?  And we have many options when it comes to volumes.  Docker already has volumes.  This is slightly different, but very similar.  So to a container running in the pod, the volume looks like a directory.  And what they are, what they're backed by and such like and when amounted is determined  by the volume type.  So the first type we have is an empty directory.  So whenever we create a pod, it creates this space somewhere on disk, on the local disk,  and they can basically share that volume between them.  But it lives and dies with the pod.  It only exists while the pod is there.  So it could be your git synchronizer is writing stuff to this volume, being read by the Apache  server or whatever server.  And you don't care when the pod goes away, if that space goes away.  It's just scratch data, just temporary data.  There's nothing stored there that's important to you.  And it can't even be backed by memory as well.  So it could be tempFS file system.  And that's great.  It's really efficient, much faster as well.  So that's what an empty directory is.  That's the default you get for a, well, I don't know what it is, a default, actually.  That's the specified type it is.  So empty there is one of the options.  The next one is host path, where we can actually map part of the file system of the node on  which the pod is running into the pod.  So this volume is actually effectively a snapshot of, not a snapshot, a link into the file system  of the actual running machine.  That's useful to read configuration data and stuff.  But it's also kind of dangerous as well, because it may be that the state on the node  may change in such a way that whenever you run a pod on one machine to another, you don't  run it.  Whenever the scheduler runs the pod on a different machine, it may see a different view of what's  happening.  So it no longer becomes completely isolated.  So it's a kind of dangerous thing to do, but it might work for you.  The other one is nfs and other similar services like GlusterFS.  I can never say that.  And if you're a G on it, I can't say some reason.  So again, nfs, we can mount nfs paths on our pod and expose them to our containers as  directories.  Or we can also use a cloud provider, persistent storage, persistent block storage.  Now we call them persistent storage in Google.  Amazon call them elastic block storage, that kind of thing.  So this is persistent disk.  So basically they can write and read from the data from the disk, and it will always  be there, whether the pod goes away or whatever.  So what we're likely to do in this case is create a volume, a volume in the cloud provider.  I call it a disk.  We create a disk in the cloud provider which stores data and we'll mount it onto the pod.  Whenever that pod goes away, the data is still there.  The pod comes along and can mount it as well.  And also with Google Cloud Platform, you can actually mount and read only on multiple  pods as well.  So some patterns for pods.  The first one is the sidecar pattern.  Because basically it's motorcycled inside car.  I guess in this case the Node.js application or the Python app is the...  You don't get offended when I say Node.js, right?  The Node.js application is a bike and the git synchroniser is the sidecar in this case.  That makes a lot of sense, right?  Ambassador, in this case, something that acts on behalf of the actual running container.  So this is a secondary container, a Redis proxy that effectively allows the PHP application  to make calls and then have the Redis proxy call that to Shards.  So we can just make...  Have one service that the PHP application calls for reads and writes and the Redis proxy  can do with the hard work of deciding whether to read from a master or read from a slave  or write from a master.  And the final one is an adapter pattern where in this case we have Redis running and we  want to monitor it.  We want to monitor all of our pods but we need a common format for monitoring.  So in this case we actually adapt the output from the Redis monitoring using an adapter  container.  An adapter container will be plugged into the monitoring system.  So it kind of adapts what's happening within the container.  So these are kind of examples of where it makes sense to have a pod.  I'm hoping it does make sense and I'll be interested to hear from you afterwards about  whether pods make sense to you.  So labels basically the single grouping mechanism within Kubernetes.  This allows us to group things that we can build applications like a dashboard.  So we have a running pod.  We give it a label.  Labels are key value pairs.  So in this case type equals fe.  Completely arbitrary metadata.  Some of these things are meaningful to Kubernetes but mostly it can be anything that's meaningful  to you.  So we've put labels on pods and we can say I can build a dashboard application that uses  the API to say give me the pods with this label.  And I can show you all the status of that.  And we can have different labels for different pods.  In this case we have a version two pod.  We have a different dashboard application with more different nodes.  And that makes a lot more...  Pods can have many labels.  I surprise myself with my slides sometimes.  Makes more sense with replication controllers because replication controllers are things  that actually manage the running of pods.  Now remember I said before that we created 10,000 tasks and we pushed them out to persistent  storage in a ball master.  And the scheduler comes along and says, yeah, these should be running but they're not.  I'll fix that.  So this is the same thing.  The replication controller is responsible for managing your desired state.  You say this is the way I want it to be.  I want to have X number of these pods based on this container template.  I want you to maintain that state for me.  That is the job of the replication controller.  So basically what they do is they work on a constituency of a label type.  So in this case version equals V1 is what they select on.  This replication controller is responsible for all pods with label version equals V1.  And we tell it I want to have two of those.  So this job is to make sure there's always two running.  In this case we also have another replication controller.  That has V2 of our pod.  Version equals V2.  I only want one of those.  So make sure there's always one of those running.  And the way it works is this is kind of like a control loop.  The replication is one big control loop.  Simple as that.  It says look at the desired state.  How many of we got running?  We should have four running.  We got four running.  We got three running.  That's not good.  Let's start another one.  We have four running.  We have five running.  That's not good.  Let's take one away.  It's basically monies the state to make sure we had the ones running.  It also works with a template.  So we provide a template, which is the pod template, which contains the container image  definition.  And how many we want to run.  We pass that into the replication controller.  It doesn't create the pods.  But when we create the replication controller and we say we want two of these pods, it  says, hmm, there's not two of these running.  I should start them.  So it starts them.  That's how it works.  And we can also plug in replication controllers after we've created the pods.  And say, you're managing containers with this label.  And finally, we get to services.  And services are how we actually expose our running stuff.  And we do this through this service here, which creates a virtual IP address, which  has a constituency of pods based on the label selector.  Again, we're going to have labels on here.  We'll show it in the next slide.  So basically, certain pods with a certain label are the constituency of this service.  And when requests come in from clients, it will load balance them across the running  pods, regardless of which node they're on.  So they could be 10,000 nodes.  We could have 10,000 pods run in different nodes.  And it would load balance them across the running pods.  At the moment, it only works around robbing.  But eventually, it will have much more intelligence support for load balancing.  This is used for exposing internal services within Kubernetes and also expose mining services  to clients externally, which we'll see shortly.  It not only provides a virtual IP address, but also a DNS name so we can do service discovery.  And we want to move on.  So this is a Canary example.  So who understands the concept of Canary?  A few of you.  So basically, when you have a situation where you have a running application, you want to  try out a new version of it.  You may have one instance or two instances of that running application that are different.  So some of your traffic will be pushed to the new versions.  Some will go to the old versions.  You can then do A, B testing against them to make sure that the new service works.  If it doesn't, you can roll it back.  If it does, you can push out the change for all of them.  This is a similar situation where we have version equals V1, version equals V2, replication  controllers and pods.  But a service, all it cares about is labels type equals FE.  And so the service has its constituency of all three of these pods.  But these pods are managed by different replication controllers.  So that's how it works.  Virtual IP address exposes that to a client.  And so we map the Kubernetes.  It all looks kind of like this.  We have pods.  Remember all the symbols?  I believe this is why it's important.  So that's a pod and a volume and a service.  And we have all a memcache vd drop down.  A pod, a service and replication controller with a service.  How does that look to a developer?  So remember how it looks to a developer on Google?  So this is how it looks.  They specify a name.  They can specify the image.  This is a Docker image now.  It could be a different image format in the future for a different type of container format.  I left it in deliberately to stop, so you.  Yeah, PHP gets back.  You can specify resources.  128 maybe bits.  Maybe bits.  You can specify how much CPU.  And the Kubernetes unfortunately had its own idea of slicing off a CPU.  And I'm not going to get into it, but it's like 500 bits of a CPU in terms of Kubernetes.  So you have to read the manual for that.  Otherwise it won't make any sense.  I'd rather have a percentage, but that doesn't work because you can't have a percentage of  a core because you don't know how powerful it is.  So that's how we specify CPU.  The ports, protocol, TTP, and the replica is one or maybe $10,000.  We covered that case as well.  So that's how it works within a replication controller.  There's other configuration files as well for services.  Oh, sorry, do you?  And scheduling at the moment, we saw the complexity of scheduling at Google.  It's a bit simpler for Kubernetes currently.  It's based on pod selection.  So we want to have the pods run in that are based on selectors, and it's based on node  capacity.  So how much capacity does that node have?  Is it capable of running my pod for me?  If I have multiple nodes that can run my pod, I'm going to run it on the one that has the  least resources consumed by running pods.  And that's the priority.  In the future, we'll have resource-aware scheduling.  So we can do what we do back in Google where we try to make maximum utilization out of our  CPU and memory.  Kubernetes is 1.0 as of this week.  Now it's on 21st of July at OSCON in Portland, Oregon.  It's been open-sourced for over a year now.  And we have a product called Google Container Engine, which I'm going to talk about shortly,  not so much, but it is a good way of running Kubernetes.  But it's not a product pitch.  Hosted Kubernetes, I'm going to talk more about Container Engine shortly.  And the roadmap for Kubernetes is there.  It's kind of sparse at the moment because we've just gone through 1.0.  So they're now deciding on the roadmap for the next releases, V1.1.  And the one on the roadmap currently is autoscaling.  The ability to autoscale your nodes dynamically based on the amount of work you have.  Container Engine is a managed version of Kubernetes, and it manages uptime for you.  You don't have to worry about the master in this case.  It will take care of the master for you.  You can't even see the master.  You can't connect to the master.  So one of the problems we have at the moment with Kubernetes is high availability.  So we don't have that replicated master scenario we saw with Borg.  So the only way to do it is to have multiple clusters to do high availability.  But if we look after your master for you and make sure it's running, then you don't have  to worry about it.  We will make sure that your cluster is highly available.  By making sure your master is always running.  We can resize using things called Managed Instance Groups, which we'll look at in a  minute.  Centralize Login.  We can pull all of our logins to one place in the Google Developer Console.  And it also supports VPN.  So you can actually have your pods inside your own network, your own private network.  So demo.  Very quickly.  And we had to change the setup earlier to make all this work.  And this is a cluster.  We have kubectl get nodes.  So we have two nodes running.  So these are machines in our cluster.  And I can look at them here.  This is the Google Developers Console.  And I can probably make that a bit smaller.  If I go into VM instances here, I can see my running machines.  I have a couple of extra machines as well.  But these two in the middle are the nodes for our cluster.  I have this thing called an instance group, which has two instances.  This is the thing that manages the size of our cluster.  And below here, we have container clusters.  And we can see we have one cluster.  OK.  And if we go to...  Wait a second.  I've got very little screen real estate, so I can't see everything that's going on.  So we go here, we can see a representation of what's running currently.  So we have...  These are pods.  So this is a pod.  This is a service.  This is exposed internally.  This is a service.  And this is another service.  Oh, MySQL is not running, which is a real pain.  I'll have to run it.  OK.  I don't know how that happened.  OK.  So we have a front-end service.  We have a memcache service.  And we have MySQL.  We don't have a pod running.  So we need to have a pod running.  So I'm going to start the pod very quickly.  And then...  OK.  So I'm going to start the pod very quickly.  OK.  That's why it's not running.  So we've just gone to 1.0.  All my demos break.  That's fine, though.  Should be fine.  I had it all running, but we had to reboot my machine because we were having problems  with the display.  And then we have a pod.  Hey, pod.  MySQL.  Have you ever spun up MySQL so quickly?  I bet you haven't.  The next thing we want to do is run some PHPs.  Unfortunately, there are PHPs currently.  But I'm trying to get time to update them completely.  But I had some problems with flask and Angular.  So anybody else had problems with flask and Angular?  No?  OK.  I should talk to your...  Basically, on my badge here, it says that my Python skills are rated as free stars.  So I probably should talk to you guys about doing it.  OK.  So I'm going to create a control.  We have a file already created.  And we're going to create that.  And now we have pods and a replication controller.  I need to make that smaller.  So now we have some frontend pods and a frontend RC controller.  So the next thing we want to do is actually look at the running application because my  windows are all screwed up.  We have it running.  So this is the IP address of the service, as we can see here.  And this is the application running.  It's DevOps.  Anybody who's been to DevOps, you don't want to go to DevOps.  You want to go to the EuroPython.  I told them to fix this beforehand.  We have an update and we can roll that update out really easily to our cluster.  Let's do that.  Let's roll out an update to our cluster.  And I'm going to close that down so we can see the visualization.  And I will go and reverse my history for this.  And I'm going to update to v2 of our frontend controller.  So what's going to happen now is it creates a new controller.  And now it's going to change those pods one by one to roll out our new version.  So now we have three pods, the 2.0 and 2.1.0.  We're going to get rid of one or 1.0.  We're going to have a 1.0 and a 2.  And then we're going to create a new 2.0 pod.  And then we're going to get rid of the other 1.0 pod.  And eventually we only have two 2.0 pods.  And we get rid of the 1.0 controllers.  We don't need that anymore.  And if we go back to our app, nothing's working.  And refresh.  And we should get... Yay!  APPLAUSE  I'm hoping it works.  I'm hoping my SQL is running properly.  Yay!  So, OK, that works.  Brilliant.  The other thing I can do as well, I should mention it.  I'm probably getting close to my net time.  What is the command? Is it RC or not?  I don't remember.  I never... Why should I get the SQL command?  So I'm going to do B2.  And I'm going to scale it to six replicas, the five replicas.  And we go back to our biz.  So now we want to add replicas to this.  We can do that by scaling like that.  And then we have five running pods.  OK? It's as simple as that.  So now we have five running.  We can do that also within the developers console.  And just to wrap up on the whole thing,  just a quick talk about the last bits and pieces.  That's how we visualized it.  We visualized it using the API and a proxy.  So kubicostal supports a proxy.  We just pointed at some JSON.  The JavaScript is all JS Plum.  So if you want to know what we use, JS Plum.  In terms of container engine, cluster scale in,  we have this thing called a managed instance group.  And that runs all of our nodes.  And nodes run within the managed instance group.  And we have this thing called an instance group manager  that creates them and is responsible for making sure  they're running.  So that's actually monitoring the cluster of nodes.  And we have a template by which we can create new nodes  on demand so we can resize that managed instance group  very easily.  And yeah, I think that's about that for cluster scaling.  We can also create clusters using tools such as the  Google Developers Console, Google Deployment Manager,  and Terraform.  I was going to give an example, but it's very basic.  Terraform will create a cluster for you,  but it won't allow you to resize it.  If you want to resize it, you have to replace it completely,  which isn't really what you want to do.  So you can create clusters with various different ways.  And oh, that's a visualization.  Some frequently asked questions are answered in the documentation.  I could spend entire hours on all of these subjects.  So if you have questions, I'll be outside all day  on the Google booth.  Come and see me.  And Kubernetes is an open source, so we want your help  making it even better.  So please contribute to Kubernetes.  If you have questions, go to IRC, IRC.free.net  on hash Google containers.  It's a very popular place.  And also on Twitter, Kubernetes.io.  You can tweet questions for me.  If you don't answer them now, or you can find me on the booth.  And ultimately, that's it.  Thank you.  We have time for one or two questions.  At the beginning, you were talking about Borg  and like five masters that you run in.  And those figures are based on the data center or how does it?  Based on the cell.  So we break it up into a cell and each cell has its own Borg master.  And that's about my limited knowledge of how the complexity  of how Borg works.  Not being a sweet one, sorry.  But yeah, that's how it works.  That's why we have multiple.  So within Google, you're going to find multiple Borg clusters  or Borg cells effectively.  Hi.  Thank you for the talk.  Very interesting.  When you compare VMs and containers,  even if the user of a VM has root access,  it's very difficult to escape from the IPvice or et cetera.  How do you see the security in the current containers  implementations?  It's a work in progress.  So this is about security with containers.  I'm not really going to comment too much on it,  but it's getting better all the time.  Initially, we had problems with the kernel level and syscalls  and such like being made back into the operating system.  But it's getting better.  So ultimately Docker and such like becoming more secure all the time.  Ultimately, doing multi-tenant maybe currently with multiple customers  applications run inside by side may not be the best idea.  But we have to tackle that.  So ultimately, we have to make sure people are more confident that they can run  all of their jobs on containers securely.  I don't think we're quite there yet, but we're working on it.  So that's one challenge we need to crack.  Is that a question?  Yeah.  One more question?  Is that done?  No, that's enough.  OK.  Come and find me outside.  Come and find me outside.  We can talk about PHP.  It's Python, sorry.  From our organization, we want to thank Mando to come in.  Oh, wow.  Give her a present.  Thank you.  Should I answer it?  Should I put it?  No.  Oh, no.  That is wonderful.  Ah, fantastic.  Exactly what I need.  Thank you very much.  Thank you.  Thanks for having me.  Thank you.
Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__0_1355.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__1356_1451.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__1452_1811.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__1812_10571.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__10572_14075.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__14076_15467.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__15468_17111.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__17112_22307.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__22308_26999.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__27000_33227.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__33228_33299.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__33300_37019.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__37020_38195.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__38196_39695.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__39696_39791.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__39792_40271.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__40272_41063.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__41064_41183.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__41184_43403.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__43404_46871.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__46872_46955.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__46956_47027.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__47028_48143.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__48144_49343.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__49344_50123.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__50124_52175.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__52176_52427.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__52428_55103.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__55104_63935.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__63936_69551.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__69552_69671.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__69672_82715.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__82716_82847.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__82848_83759.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__83760_86447.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__86448_86531.png Mandy_Waite_-_Keynote__So__I_have_all_these_Docker_containers__now_what__86532_87435.png
Thank you for coming and welcome. Thank you. Hey, thanks, everyone. You all hear me? I think you can. Good. Excellent. Thanks for having me at Python, EuroPython. My last big Python event was Python, PyConJP in Japan last year. I didn't get to speak, though, but it was really fun. Although most of the talks are in Japanese and my Japanese is getting better, it's not so great. My Spanish is really, really bad. But Spanish and Japanese are very similar, so maybe I should learn both together. No, really, seriously, they are. I know it doesn't seem to make sense. We're going to talk about containers and having containers and having lots of containers, because ultimately, everything is going to be containerized and we're going to have lots of containers. We won't know what to do with. And I'll ask you some questions, Ada, and see how far you are along with moving towards containerization. So basically, when we have lots of containers, what do we do then? And this is a problem we face at Google. So this is a data sensor. This is a Google data sensor in Iowa in the US. It's a place called Council Bluffs, and this is one of our bigger data sensors. And if I leave it out for a long enough, you'll probably be able to count all of the machines. How many there are. But this is a cluster. So clusters are one of the constructs we have internally, but these clusters are broken down into cells. So cells are smaller. We have many cells per cluster. And this will probably, a cell we're going to look at today, is going to have about 10,000 machines in it. So they're quite large. And this is a huge amount of compute power. Lots of compute power, but we need to make this available to our engineers, our software engineers, our developers. So how do we go about making this compute power available to our own developers? And it works something like this. This is what a developer does. Well, first in context, the one thing, given what you see there, we don't want the engineers to have to kind of select a rack, select a machine, and say, hey, I'm going to run it on that machine. I'm going to SSH, SFTP a binary over to the machine, SSH into the machine, stand up my process, my server or whatever, maybe log into many machines and do that multiple times. But that's not going to be possible. Huge amounts of machines, huge numbers of engineers, huge amounts of jobs to run. So how does it happen? So basically, we have a configuration file. In this case, it's called a Borg configuration file. I was in India recently, and nobody there had heard of Borg. How many of you are familiar with Borg and Star Trek? Okay, right. So we never used to be able to talk about Borg because Paramount Pictures own it. It was kind of like one of our worst kept secrets that we had this thing called Borg running internally. But now we talk about it all the time. So because it's fun, and it's really good to show this in the context of what we're going to talk about later, which is Kubernetes. So basically, this is a Borg configuration file, and what the developer does is he creates a job. Jason file, calls the job Hello World, says which cell he wants to run it in. Going back to what we said earlier, a cell is a few thousand machines. In this case, he's saying it's called IC, some random cell name we chose. And he specifies what binary to use. In this case, Hello World web server. So he wants to run Hello World on a web server. And this is going to be a fat binary. Statically links all of its dependencies with it. So effectively, we can run it pretty much anywhere without having to worry about the underlying operating system. And that includes the web server as well. So this thing is quite big, probably about 50 megabytes. So he specifies the path to his binary, or her binary. And unfortunately, we have too many male software engineers, not enough female software engineers. So let's encourage women to be software engineers. And arguments. We have to specify some arguments for our binary, pass them in via the environment. In this case, we want to specify what port to run on. This is parameterized. Then we have some requirements in terms of resources. Now, this is important. We'll circle back to this in a minute. So we can specify how much RAM, how much disk, how much CPU. And ultimately, we can say how many we want to run. So in this case, we want to run five replicas of this job, five tasks effectively. And why five? Why not do it at Google scale, 10,000? Makes more sense, right? We have all those machines. We saw how many machines we have. So let's run 10,000 copies of this. So once we finish this, our software engineer, she types in a command on the command line, passes in the config file. And that gets pushed out to somewhere. It gets pushed out to this ball scheduler. And what happens then is this. Over a period of time, in this case, about two minutes, 40 seconds, 10,000 tasks start. 10,000 instances of that job start. And it takes two minutes, 40 seconds, roughly. We do phase the rollout of all of these jobs to make sure we don't do them all at once. One of the key factors here is the size of the binary, 50 megabytes, 10 times 10,000. It's about 20 gigabytes per second of IO. We're going to be cashing that binary quite a lot, but we had to move it around between 10,000 machines. So there's a huge amount of IO going on. But eventually we get to a point where we have 10,000 running, or nearly 10,000. Maybe not quite 10,000. We'll talk about that in a second. And Borg looks like this. This is what Borg is to Google. It's not going to assimilate you, but I think we came up with a name because it's probably going to assimilate everybody eventually. So this is Borg. And Borg runs within a cell. So each cell has its own Borg master, its own Borg configuration. In this case, we have a Borg master, which is highly replicated. We have five copies of it for resilience. And we have lots of other things. These down here are our machines. These are our machines we saw in the racks. They're all running a thing called a Borglet. We have a scheduler. We have some configuration files in the binary. So what happens is the developer, the engineer, creates his or her binary. And they use a massively distributed parallel build system called, well, I won't say what it's called, but it's externally available now called Bazel. So we made this open source. So our own build system is now available open source called Bazel, B-A-Z-E-L, or if you're American, B-A-Z-E-L. It gets very confusing, believe me. If you go to Canada, it's so confusing. Like routes and routes. So basically, he or she creates the binary, pushes it out, and it gets stored in storage for the cell. And then they push their configuration file. Configuration file gets copied to the Borg master. We have a persistent pack source back store, consensus based. And what happens then is this scheduler, looking around, comes along and says, hey, what is the desired state? We should have this running. Do we have this running? And it sees 10,000 new tasks. And it says, hey, they're not running. We should have 10,000 of those. Let's make sure that's happening. Let's fix that. And so it goes about planning the running of these 10,000 tasks. And it creates a plan, and then it starts telling the Borg master what the Borg master makes decisions and tells them the Borglets on these machines to run this particular task. So they get communicated. The task will ultimately run inside a thin container wrapper. So it has a container around it. It's not just running the binary. It is containerized. A very lightweight shim container that's not Docker. It's not standards-based. The Borglet ultimately will pull the binary over from storage, and it will start running. And we'll see this. Lots of Hallowels. All over our data center. So now we're running multiple copies of that. And so that's what we had, 10,000. But if we look at it a little bit closer, we find there's 9,993 running. Not quite the 10,000 we expected. But this is a highly available service. We expect some lessening of the number of tasks we're running over time due to the way we operate. And that's interesting. So let's look at that in a little bit more detail. So failures. Things fail. But failure is kind of more of a generic term here. There are many reasons for failures. And one of the main reasons for failures, particularly for low priority jobs, is preemption. If we look at the top bar, which is our production jobs, we have very few failures. And most of them are down to a machine shutdown, where we've actually scheduled some maintenance on the machine, and we've taken the machine down. That task, any task running on that machine, we don't be rescheduled elsewhere in the cluster. We have a very small number of preemptions. Down here are non-production jobs, which are things like map producers, batch jobs. They get preempted all the time. They're happy to be preempted. And in fact, the calculation generally says that for about 10,000 tasks, about seven or eight of them will be not running at any given time because of preemption. They'll be about to be scheduled somewhere else, but they won't be running at that particular time. And we see other things here. We see, again, the blue bar, which is the machine shutdown, which is pretty much the same as production. And we have some other things as well, out of resources, very small number of machine failures. And for when you have as many clusters, the many machines as we have, machine failures are given. We expect that. We don't panic when machines go down. It's part of the normal running of our business. And another interesting thing is how we try to make efficient use of our resources. So we have CPUs, we have memory, we have disk.io, we have network.io. And sometimes it's quite possible for one task to be using lots of memory for very little CPU or vice versa, lots of CPU and very little memory. If you put one of those on a machine, then you may be wasting one of those resources. It's what's known as resource stranding. And these are the available resources, these white bars here. This example here is actually our virtual machines, which are tasks. Our virtual machines are actually containers, believe it or not. It's a Google computing engine. So these are all virtual machines, these bars, individual bars. And what we can see here is that some of these machines have available capacity, available RAM, available CPU, and if we look over here, we see a different situation where we have maybe some with available CPU and others with no available RAM and vice versa. This here and this here is called resource stranding. It means we're not actually making use of that resource. So we have spare memory capacity or spare CPU capacity that's being wasted effectively. So one of our challenges is like a Tetris puzzle to try to stack these things in a way where we get the best possible utilization out of our clusters. So we will mix and match them to make sure we have low CPU, high memory jobs running with high memory, low CPU jobs. And of course, we run multiple tasks per machine. That's extremely important. That can then come back to all this with Kubernetes shortly. And another interesting thing is this, which is going to be a huge challenge in the future when it comes to Kubernetes, but it's going to be really important to all of us. So we saw earlier that our developer, she specifies what resources she wants to use or he wants to use. 100 megabytes of RAM, 100 megabytes of disk, 0.1 CPU. And that will be this blue line up here. So everything's running. We'll match into this blue line. These are the resources that were requested by these jobs. In reality, though, it's like this. And so we have all of this wasted space, which we can't use because it's been allocated effectively for those running jobs. But we can use it. So what we do is we effectively estimate, based on the run patterns of the current jobs, how much they're going to use. And that's this blue line here. So this is our reservation. So this is how much we reserve specifically for those jobs. And what we can then do is reuse all that space. Now we can reuse that space for very low priority jobs. Again, those batch jobs, those map reduces. Things that we want to run, we want them to finish eventually, but we don't really care when it happens. It could be like running some kind of monthly report that nobody ever looks at that gets logged or running a map reduce across a huge amount of data that may be important at some point or just needs to be done, but we don't really care when it needs to be done. So all of that stuff, we can reuse it and we can run jobs within it. But that's really important. That's how we can get maximum utilization out of all of our machines in that data center. And so moving on to Kubernetes now, gradually, one of the observations is that if you have your developers spending time thinking about machines or thinking in terms of machines, and you're probably doing it wrong, because it's too low a level of abstraction. Now today, maybe it's fine, but in the future, this is not going to be the case. We need people to be thinking in terms of applications and not having to worry about the infrastructure in which they run. I mean, anybody who uses a platform as a service knows how important that is anyway. You don't care about the infrastructure. You want to write your work, configuration file, build a binary, and just say, run this for me. I don't care where you run it. I don't care about how you do it. Just run it for me and make sure it stays running. We get efficiency by sharing our resources and reclaiming unused allocations. And containers, the fact that we containerize everything, allows us to make our users much more productive. So everything we run runs on a container. Two billion containers a week, we estimate. We never really thought that was very important until Docker came along. Containers became the next big thing, right? Alex, then Docker. And Docker became huge. So now one of the things we talk about all the time now is we run containers all the time. We are pretty good at running containers. Which is why we created Kubernetes. If you're interested in more details of what I've just talked about, Borg, there's a paper here. G.gl, one capital C for N U O. That's the white paper in Borg. That's got all of the details, all of the graphics you just saw. It goes into much, much more detail, of course. So let's look in terms of a simple application and how we can do this externally with containers and through Kubernetes. So this is a very simple pattern. Generally when we give this sockets PHP in the middle, MySQL, Memcache, and we have a client. We have many of these pythons running. This could be many instances of flask, it could be some kind of event system, but we have the ability to run many, many concurrent requests. We're probably going to want to scale this thing on demand. We may not want to scale MySQL that much until we get to a point where we have to do replicas and sharding. Memcache, we've probably gone on a scale as well, but we're going to keep it simple for now. Just keep one MySQL, one Memcache, and a few of these pythons that are at the front end. So let's talk about containers. So how many of you are familiar with containers? How many of you have actually spun up a Docker container? Hey, lots of you. It's almost the same number, right? Again, last year we'd asked this, how many of you have heard of containers, lots of hands, how many of you have spun up a Docker container? Lots of many. So things have changed now. Docker is the future. Well, containers are the future. We now have this thing called the open container project. Docker have kindly made what they have into a spec, and we're all going to get behind it. We have this common specification from which we can write containers. Things like CoreOS with a rocket container, they're all going to fall in line and we'll have a common format for containers. Which is going to be great. But just for those of you who are not really familiar with containers, just a few slides, very few slides on containers. Just kind of give you some of the concepts. This is the way we used to do things in the old days. We have a machine, maybe next to our desk, in our bedrooms or in a colo or in a server room. And the machine would run our operating system. It would have all of the packages installed at provided libraries, things like open SSL. On top of that, we would run applications. And how many of you have had a situation where you're running one application and all of the other applications on the machine fail because that one application went mad? Use all of the CPU, use all of the RAM, it crashed the machine and took all of the other applications down. And this may have been a very low priority app, one that you didn't really care about taking down some really important ones. Now, this is never a good idea running multiple applications on one machine because there's no isolation between them. Whatever affects one application will probably affect all of the others. There's no namespacing. They all have one view of the machine in which they're running. They have one view of the CPU, one view of the memory, one view of the file system, one view of the network. They share libraries. And so you're in a situation where maybe one day you update a version of a package, it updates a library, and one of your applications says, hey, I'm not going to run anymore. That library is not compatible with me. So dependency hell. And if it's on Windows, it's DOL hell, and it's probably even worse. Applications are highly coupled to the operating system. This is a problem. And so we created virtual machines. And what we did basically is stuck a layer on top of the hardware called a hypervisor. And we now had an idealized piece of hardware on which we could run multiple operating systems. So now we have this thin layer, it looks like a piece of hardware to be running virtual machines. And that gives us some isolation. So now we can run applications in their own virtual machine. So each application is now isolated. If one application crashes, it doesn't affect the others. But it's extremely inefficient because we have this red bit at the bottom here. We have the operating system, the kernel. And you know when you install a virtual machine, you pretty much have to install the entire devian stack or the entire center stack or the entire Windows stack. So that's not very efficient at all. They're still the same type coupling between the operating system and the application. And as anybody who's tried to manage lots and lots of virtual machines to provide isolation, you know it's hard. So there are new ways containers. In this case, we move up a layer. So we move above the operating system and provide an idealized operating system. No longer idealized hardware and idealized operating system on which we can run apps and their dependent libraries. So the libraries here are part of the container. So the container has an application. It has all of its dependencies. It has its entire environment. So we can move this container around anywhere we want to. We can move it from one machine to another, from one runtime to another, from a laptop to a virtual machine, to a cloud, to a bare metal server, to a set-top box, ultimately, maybe even to a phone when we had Docker on Android and iOS. I'm sure it's going to happen, right? And let's look at the example. So we have our application, PHP in Apache. It should be Python in Apache. Sorry. See? I do apologize. So wherever you see PHP in this deck, read Python. We'll change it before I share the slides. So I'm trying to think of what could offend a Python audience most. It's probably talking about PHP, right? My god. Okay. So we have containers. We want to run these components of our application, Python and Apache, Memcache, MySQL. Not Apache. Obviously, Python and Vlask and Bottle and all of the other things we could potentially use. Memcache, MySQL. And MySQL has its own libraries. It doesn't have any common libraries with the others. So we can stack those libraries with the container in which MySQL runs. And Memcache, PHP, Python and Apache have their own, I keep saying it. And Python and whatever. Unicorn. Anything. They have their own dependencies. But they also have when we install them some shared dependencies as well. So some commonalities. So when we actually create the image, we can actually share some stuff between them. But that's not shared at run time. So when we create the container, they will have their own dependencies packaged together in the container. And underneath that, we have a server. And again, this could be a virtual machine. It could be a laptop. It could be a bare metal server. It could be anything, pretty much. And underneath it, we have the actual hardware. And all of this is being maintained by a Docker engine. So Docker is the thing that runs this. So when we talk about containers, mostly synonymous with Docker nowadays, but again, there are other container formats. And hopefully, they will all comply with a standard. And that's the Nirvana we're all heading towards. So Docker effectively controls the creation of these containers and the management of these containers. So at the end of it, we will have Python, and Flask, Angular, Memcache, MySQL, all running in containers. So why containers? So there's many important reasons for having containers. But you can see, just by looking at what we do, that's the only way we can do it. We can't do it any other way. This is perfect solution for the kind of scale that we want. But it's also perfect for smaller scale as well. Why? Because it's much more performance. It's much more performance in terms of the fact that we don't have to do all of that installation stuff. They are pretty much like they're running on bare metal. So the performance is pretty much the same as a virtual machine, but they're much quicker to get up and running. Which means you can swap them out quicker. You can do upgrades quicker. You can do pretty much everything quicker. Repeatability. So the whole problem where we have development, QA, build, test, production, where we want to have repeatable environments, where we have a situation where when we test something in QA and then run it in prod, it fails in prod where it works in QA. How many people have had that situation? Have you had any hands remembering those days? So what containers give us is the ability to have a consistent environment because the environment is packaged with a container. So basically when we run it in QA and when we run it in prod, it's exactly the same. It's exactly the same environment. So that's one of the great use cases of containers today. But much more is the portability of it, which we're going to talk about in a second. Quality of service. We can now do resource isolation as well. Using things like C groups and Linux and namespaces, we can actually isolate the resources. We can say we only want this to have 100 megabytes of RAM, 100 megabytes of disk, 0.1 CPU. And ultimately accounting. These things are easier to manage. They're easier to trace. They're easier to audit. They're small, composable units that can be tracked very easily. And ultimately, portability. You can move these things around from one cloud provider to another. Images specifically. You can't just pick up a running container and move it, but you can easily run the same container in a different cloud provider on a bare-measure machine on the laptop. You can move them from one machine to another as the shape of your cluster, if you have a cluster of machines, changes. You can move them around to be more efficient. So we can go back to what we had before with the efficient allocation of resources. We can do that if we have containers. And ultimately, this is a fundamentally different way of managing and building applications. So demo. I'm not going to do this demo. I left that slide in by mistake. This would have been a containers docker demo. And I don't think I want to bore you with that. It's very easy to find a tutorial on docker and get up and run them a bit. Now, let's not talk about that. Let's talk about Kubernetes instead. How many of you have heard of Kubernetes? How many of you can say Kubernetes? It's hard word to get your head around. Probably easier if you agree, because it's a Greek word. But if you want to help pronouncing it, I'll be outside in the Google booth after this talk. So I can definitely provide assistance on that. Maybe I'm saying it wrong. Maybe I've been saying it wrong all this time. So I'm happy to be corrected. So Kubernetes, let's talk about that. And we've given you an introduction to what we do at Google. So that should provide the context on why Kubernetes is necessary. What we often miss out, is that we don't provide that kind of context. So I'm hoping that the introduction to Borg has provided that for you. So Kubernetes is Greek word, means helmsman. It's the root of the word governor. So Arnold Schwarzenegger's governor comes from Kubernetes. And it's an orchestrator or scheduler for docker containers. Ultimately for other forms of containers, I think ChoraOS is already using it to schedule orchestrate rocket containers. It supports multiple cloud environments. So Metasphere, I always forget them. VMware, even Microsoft involved. You can run Kubernetes on Amazon. You can run it pretty much anywhere. You can run it on your laptop with Vagrant. So you can just create a four machine cluster, virtual machines with Vagrant up. And you'll have a Kubernetes cluster. And ultimately, eventually, we may have a situation where we can run Kubernetes across multiple cloud providers. It might be difficult. It might be possible. But it may be one day you'll have your fleet of machines will be running in Google, in Amazon, and Microsoft Azure as well. Possible. I'm not sure if it's going to happen. So this is kind of inspired and informed by everything that we saw previously, everything with Borg. And it's basically our experiences. Open source, ridden and go, like many good programs nowadays, but completely respect Python. I love Go. I love Python. I used to be a Java developer. I spent 15 years developing Java. No, 11 years. Now I moved to Google. I haven't wrote a line of Java code since. Now I write... Now I write... It's like Java program is anonymous, right? It's been four years since I wrote my last line of Java code. So now I write in Python, go, and write in Angular, and write in Java scripts, and all of those more interesting, useful languages. Java is getting better. Java rate is a big step forward. And ultimately, we want to be able to talk about managing applications and not machines, which is actually what we talked about earlier. And some very quick concepts. I'm not going to introduce them, but I want to show you the icons so that when you see them, you'll know what they mean. Angular, pod, service, volume, label, replication controller, node, are all of the key concepts. How many of you are familiar with sort stack? How many of you like the terminology in sort stack, like grains and such like it? I think it's really hard to get your head around. And I think one of the dangers about an abstraction is that you get too far away from the terms that are familiar to people. Most of these are familiar to people with service, the idea of a replication controller, a node, a label, a container. The pod is probably the most difficult one to get your head around. So let's talk about pods. Let's talk about nodes first and clusters. So we have a cluster, kind of maps back to what we talked about earlier with Borg, where we have a master. And the master has a scheduler and it has an API, an API server that can be used to talk to nodes. The nodes are all running a thing called a kubelet. And they have these things called pods running containers. We'll talk about pods shortly. They also have a proxy by which we can expose our running containers to the outside world. And we have many nodes. And a cluster, this is an abstraction. So a cluster could be different depending on which cloud provider you're using. And ultimately what you want to have is a fabric of machines that looks like a flat shape, which we can run containers. You don't care about it. You just care that they're all joined together. And it's one big flat space in which we can run stuff. And we'll let this thing, the scheduler, take care of running stuff for us ultimately. And so basically the options for clusters are laptops, multi-node clusters, hosted or even self-managed, on-prem or cloud-based using virtual machines or bare metal in virtual machines. Many, many options. There's a matrix down here, a short link. Hopefully we can share these slides afterwards. And the short link will give you a matrix of how you can run Kubernetes on what you want to run it on, CoreOS on Amazon. We have different ways of doing a networking. The networking is quite tough. Google Compute Engine makes it easy because of IP addressing. But often we have to put this other layer in called Flannel to actually provide that ability to give an IP address in a group of subnets to a running machine or a running pod. So let's talk about pods. How many of you are familiar with the concept of pods? OK, not so many of you. So in the diagram here we have a pod. It has a container. This is a container, this web server is a container. And it has a volume, like Docker containers can have volumes. A little bit different, but very similar. And so we want to run this web server. And the construct we use within Kubernetes is to create this thing called a pod. It's like a logical host. So like if you wanted to run Apache and something else alongside it, you would run it on a host machine. That's the same as a pod. So anything you would run together on the same machine will run in a pod. These are the atomic units of scheduling for Kubernetes. This is what Kubernetes schedules. We talked about jobs earlier when we looked at bulk. Kubernetes schedules pods. And your containers run within the pod. So thin wrapper around them. These are ephemeral. These are like, I've got this analogy. So everybody uses this pets versus cattle analogy. And I don't really like it from a vegetarian. So crops versus flowers. So pods are like crops. You don't care about them. You have a wheat field. You don't care about your individual plants that are growing. When you have flowers, you probably give them names and you water them and you talk to them as well. So you care about them. You don't care about your crops though. So pods are like crops. They can come and go. They can be replaced. They're all absolutely the same. You can take one and replace it with another. And ultimately, to make things simple now, you don't have to worry about a pod if you want to run a single container. You just say, run this container for me. It will create a pod for you. And you still have to think in terms of pods when you're doing monitoring, but you don't have to create a pod. You just say, run the container for me. It will create a pod for you. OK? So pods are an abstraction. Difficult to get ahead around. A little bit more information about them. Imagine this scenario where you want to have something that synchronizes with GitHub. This may be a push-to-deploy type scenario where whenever your developers do a merge into GitHub, you want those changes to be immediately pushed out into production or maybe into a staging service. So you have a thing called a Git synchronizer. And it's talking to Git, monitoring your project in Git. It pulls down any changes. And it writes them to somewhere on the disk. And your web server can then serve that latest content. Those things are tied together. They work together. And it makes sense for them to run side by side. And when one goes away, the other goes away. So we can run them both in the same pod. So now we're saying, on this logical host, this pod thing, let's run two containers. In this case, Git synchronizer and a Node.js app or a Python app. And we have a shared volume, a concept of a volume, which we'll talk about shortly. These are tightly coupled together. So when one, when a pod dies, they die together. It doesn't make any sense to have them running separately. It might do in your, in the way you architect things, but it doesn't have to. They share the network space and port space. They have the same concept of local host. They are completely ephemeral and think in terms of things you would run on a single machine. So a volume, what's a volume? I don't normally talk about volumes, but they are very important, so not talking about them seems a bit stupid, really. So a volume is basically bound to the pod that encloses it. And this is something where we can write data or read data from. Okay? And we have many options when it comes to volumes. Docker already has volumes. This is slightly different, but very similar. So to a container running in the pod, the volume looks like a directory. And what they are, what they're backed by and such like and when amounted is determined by the volume type. So the first type we have is an empty directory. So whenever we create a pod, it creates this space somewhere on disk, on the local disk, and they can basically share that volume between them. But it lives and dies with the pod. It only exists while the pod is there. So it could be your git synchronizer is writing stuff to this volume, being read by the Apache server or whatever server. And you don't care when the pod goes away, if that space goes away. It's just scratch data, just temporary data. There's nothing stored there that's important to you. And it can't even be backed by memory as well. So it could be tempFS file system. And that's great. It's really efficient, much faster as well. So that's what an empty directory is. That's the default you get for a, well, I don't know what it is, a default, actually. That's the specified type it is. So empty there is one of the options. The next one is host path, where we can actually map part of the file system of the node on which the pod is running into the pod. So this volume is actually effectively a snapshot of, not a snapshot, a link into the file system of the actual running machine. That's useful to read configuration data and stuff. But it's also kind of dangerous as well, because it may be that the state on the node may change in such a way that whenever you run a pod on one machine to another, you don't run it. Whenever the scheduler runs the pod on a different machine, it may see a different view of what's happening. So it no longer becomes completely isolated. So it's a kind of dangerous thing to do, but it might work for you. The other one is nfs and other similar services like GlusterFS. I can never say that. And if you're a G on it, I can't say some reason. So again, nfs, we can mount nfs paths on our pod and expose them to our containers as directories. Or we can also use a cloud provider, persistent storage, persistent block storage. Now we call them persistent storage in Google. Amazon call them elastic block storage, that kind of thing. So this is persistent disk. So basically they can write and read from the data from the disk, and it will always be there, whether the pod goes away or whatever. So what we're likely to do in this case is create a volume, a volume in the cloud provider. I call it a disk. We create a disk in the cloud provider which stores data and we'll mount it onto the pod. Whenever that pod goes away, the data is still there. The pod comes along and can mount it as well. And also with Google Cloud Platform, you can actually mount and read only on multiple pods as well. So some patterns for pods. The first one is the sidecar pattern. Because basically it's motorcycled inside car. I guess in this case the Node.js application or the Python app is the... You don't get offended when I say Node.js, right? The Node.js application is a bike and the git synchroniser is the sidecar in this case. That makes a lot of sense, right? Ambassador, in this case, something that acts on behalf of the actual running container. So this is a secondary container, a Redis proxy that effectively allows the PHP application to make calls and then have the Redis proxy call that to Shards. So we can just make... Have one service that the PHP application calls for reads and writes and the Redis proxy can do with the hard work of deciding whether to read from a master or read from a slave or write from a master. And the final one is an adapter pattern where in this case we have Redis running and we want to monitor it. We want to monitor all of our pods but we need a common format for monitoring. So in this case we actually adapt the output from the Redis monitoring using an adapter container. An adapter container will be plugged into the monitoring system. So it kind of adapts what's happening within the container. So these are kind of examples of where it makes sense to have a pod. I'm hoping it does make sense and I'll be interested to hear from you afterwards about whether pods make sense to you. So labels basically the single grouping mechanism within Kubernetes. This allows us to group things that we can build applications like a dashboard. So we have a running pod. We give it a label. Labels are key value pairs. So in this case type equals fe. Completely arbitrary metadata. Some of these things are meaningful to Kubernetes but mostly it can be anything that's meaningful to you. So we've put labels on pods and we can say I can build a dashboard application that uses the API to say give me the pods with this label. And I can show you all the status of that. And we can have different labels for different pods. In this case we have a version two pod. We have a different dashboard application with more different nodes. And that makes a lot more... Pods can have many labels. I surprise myself with my slides sometimes. Makes more sense with replication controllers because replication controllers are things that actually manage the running of pods. Now remember I said before that we created 10,000 tasks and we pushed them out to persistent storage in a ball master. And the scheduler comes along and says, yeah, these should be running but they're not. I'll fix that. So this is the same thing. The replication controller is responsible for managing your desired state. You say this is the way I want it to be. I want to have X number of these pods based on this container template. I want you to maintain that state for me. That is the job of the replication controller. So basically what they do is they work on a constituency of a label type. So in this case version equals V1 is what they select on. This replication controller is responsible for all pods with label version equals V1. And we tell it I want to have two of those. So this job is to make sure there's always two running. In this case we also have another replication controller. That has V2 of our pod. Version equals V2. I only want one of those. So make sure there's always one of those running. And the way it works is this is kind of like a control loop. The replication is one big control loop. Simple as that. It says look at the desired state. How many of we got running? We should have four running. We got four running. We got three running. That's not good. Let's start another one. We have four running. We have five running. That's not good. Let's take one away. It's basically monies the state to make sure we had the ones running. It also works with a template. So we provide a template, which is the pod template, which contains the container image definition. And how many we want to run. We pass that into the replication controller. It doesn't create the pods. But when we create the replication controller and we say we want two of these pods, it says, hmm, there's not two of these running. I should start them. So it starts them. That's how it works. And we can also plug in replication controllers after we've created the pods. And say, you're managing containers with this label. And finally, we get to services. And services are how we actually expose our running stuff. And we do this through this service here, which creates a virtual IP address, which has a constituency of pods based on the label selector. Again, we're going to have labels on here. We'll show it in the next slide. So basically, certain pods with a certain label are the constituency of this service. And when requests come in from clients, it will load balance them across the running pods, regardless of which node they're on. So they could be 10,000 nodes. We could have 10,000 pods run in different nodes. And it would load balance them across the running pods. At the moment, it only works around robbing. But eventually, it will have much more intelligence support for load balancing. This is used for exposing internal services within Kubernetes and also expose mining services to clients externally, which we'll see shortly. It not only provides a virtual IP address, but also a DNS name so we can do service discovery. And we want to move on. So this is a Canary example. So who understands the concept of Canary? A few of you. So basically, when you have a situation where you have a running application, you want to try out a new version of it. You may have one instance or two instances of that running application that are different. So some of your traffic will be pushed to the new versions. Some will go to the old versions. You can then do A, B testing against them to make sure that the new service works. If it doesn't, you can roll it back. If it does, you can push out the change for all of them. This is a similar situation where we have version equals V1, version equals V2, replication controllers and pods. But a service, all it cares about is labels type equals FE. And so the service has its constituency of all three of these pods. But these pods are managed by different replication controllers. So that's how it works. Virtual IP address exposes that to a client. And so we map the Kubernetes. It all looks kind of like this. We have pods. Remember all the symbols? I believe this is why it's important. So that's a pod and a volume and a service. And we have all a memcache vd drop down. A pod, a service and replication controller with a service. How does that look to a developer? So remember how it looks to a developer on Google? So this is how it looks. They specify a name. They can specify the image. This is a Docker image now. It could be a different image format in the future for a different type of container format. I left it in deliberately to stop, so you. Yeah, PHP gets back. You can specify resources. 128 maybe bits. Maybe bits. You can specify how much CPU. And the Kubernetes unfortunately had its own idea of slicing off a CPU. And I'm not going to get into it, but it's like 500 bits of a CPU in terms of Kubernetes. So you have to read the manual for that. Otherwise it won't make any sense. I'd rather have a percentage, but that doesn't work because you can't have a percentage of a core because you don't know how powerful it is. So that's how we specify CPU. The ports, protocol, TTP, and the replica is one or maybe $10,000. We covered that case as well. So that's how it works within a replication controller. There's other configuration files as well for services. Oh, sorry, do you? And scheduling at the moment, we saw the complexity of scheduling at Google. It's a bit simpler for Kubernetes currently. It's based on pod selection. So we want to have the pods run in that are based on selectors, and it's based on node capacity. So how much capacity does that node have? Is it capable of running my pod for me? If I have multiple nodes that can run my pod, I'm going to run it on the one that has the least resources consumed by running pods. And that's the priority. In the future, we'll have resource-aware scheduling. So we can do what we do back in Google where we try to make maximum utilization out of our CPU and memory. Kubernetes is 1.0 as of this week. Now it's on 21st of July at OSCON in Portland, Oregon. It's been open-sourced for over a year now. And we have a product called Google Container Engine, which I'm going to talk about shortly, not so much, but it is a good way of running Kubernetes. But it's not a product pitch. Hosted Kubernetes, I'm going to talk more about Container Engine shortly. And the roadmap for Kubernetes is there. It's kind of sparse at the moment because we've just gone through 1.0. So they're now deciding on the roadmap for the next releases, V1.1. And the one on the roadmap currently is autoscaling. The ability to autoscale your nodes dynamically based on the amount of work you have. Container Engine is a managed version of Kubernetes, and it manages uptime for you. You don't have to worry about the master in this case. It will take care of the master for you. You can't even see the master. You can't connect to the master. So one of the problems we have at the moment with Kubernetes is high availability. So we don't have that replicated master scenario we saw with Borg. So the only way to do it is to have multiple clusters to do high availability. But if we look after your master for you and make sure it's running, then you don't have to worry about it. We will make sure that your cluster is highly available. By making sure your master is always running. We can resize using things called Managed Instance Groups, which we'll look at in a minute. Centralize Login. We can pull all of our logins to one place in the Google Developer Console. And it also supports VPN. So you can actually have your pods inside your own network, your own private network. So demo. Very quickly. And we had to change the setup earlier to make all this work. And this is a cluster. We have kubectl get nodes. So we have two nodes running. So these are machines in our cluster. And I can look at them here. This is the Google Developers Console. And I can probably make that a bit smaller. If I go into VM instances here, I can see my running machines. I have a couple of extra machines as well. But these two in the middle are the nodes for our cluster. I have this thing called an instance group, which has two instances. This is the thing that manages the size of our cluster. And below here, we have container clusters. And we can see we have one cluster. OK. And if we go to... Wait a second. I've got very little screen real estate, so I can't see everything that's going on. So we go here, we can see a representation of what's running currently. So we have... These are pods. So this is a pod. This is a service. This is exposed internally. This is a service. And this is another service. Oh, MySQL is not running, which is a real pain. I'll have to run it. OK. I don't know how that happened. OK. So we have a front-end service. We have a memcache service. And we have MySQL. We don't have a pod running. So we need to have a pod running. So I'm going to start the pod very quickly. And then... OK. So I'm going to start the pod very quickly. OK. That's why it's not running. So we've just gone to 1.0. All my demos break. That's fine, though. Should be fine. I had it all running, but we had to reboot my machine because we were having problems with the display. And then we have a pod. Hey, pod. MySQL. Have you ever spun up MySQL so quickly? I bet you haven't. The next thing we want to do is run some PHPs. Unfortunately, there are PHPs currently. But I'm trying to get time to update them completely. But I had some problems with flask and Angular. So anybody else had problems with flask and Angular? No? OK. I should talk to your... Basically, on my badge here, it says that my Python skills are rated as free stars. So I probably should talk to you guys about doing it. OK. So I'm going to create a control. We have a file already created. And we're going to create that. And now we have pods and a replication controller. I need to make that smaller. So now we have some frontend pods and a frontend RC controller. So the next thing we want to do is actually look at the running application because my windows are all screwed up. We have it running. So this is the IP address of the service, as we can see here. And this is the application running. It's DevOps. Anybody who's been to DevOps, you don't want to go to DevOps. You want to go to the EuroPython. I told them to fix this beforehand. We have an update and we can roll that update out really easily to our cluster. Let's do that. Let's roll out an update to our cluster. And I'm going to close that down so we can see the visualization. And I will go and reverse my history for this. And I'm going to update to v2 of our frontend controller. So what's going to happen now is it creates a new controller. And now it's going to change those pods one by one to roll out our new version. So now we have three pods, the 2.0 and 2.1.0. We're going to get rid of one or 1.0. We're going to have a 1.0 and a 2. And then we're going to create a new 2.0 pod. And then we're going to get rid of the other 1.0 pod. And eventually we only have two 2.0 pods. And we get rid of the 1.0 controllers. We don't need that anymore. And if we go back to our app, nothing's working. And refresh. And we should get... Yay! APPLAUSE I'm hoping it works. I'm hoping my SQL is running properly. Yay! So, OK, that works. Brilliant. The other thing I can do as well, I should mention it. I'm probably getting close to my net time. What is the command? Is it RC or not? I don't remember. I never... Why should I get the SQL command? So I'm going to do B2. And I'm going to scale it to six replicas, the five replicas. And we go back to our biz. So now we want to add replicas to this. We can do that by scaling like that. And then we have five running pods. OK? It's as simple as that. So now we have five running. We can do that also within the developers console. And just to wrap up on the whole thing, just a quick talk about the last bits and pieces. That's how we visualized it. We visualized it using the API and a proxy. So kubicostal supports a proxy. We just pointed at some JSON. The JavaScript is all JS Plum. So if you want to know what we use, JS Plum. In terms of container engine, cluster scale in, we have this thing called a managed instance group. And that runs all of our nodes. And nodes run within the managed instance group. And we have this thing called an instance group manager that creates them and is responsible for making sure they're running. So that's actually monitoring the cluster of nodes. And we have a template by which we can create new nodes on demand so we can resize that managed instance group very easily. And yeah, I think that's about that for cluster scaling. We can also create clusters using tools such as the Google Developers Console, Google Deployment Manager, and Terraform. I was going to give an example, but it's very basic. Terraform will create a cluster for you, but it won't allow you to resize it. If you want to resize it, you have to replace it completely, which isn't really what you want to do. So you can create clusters with various different ways. And oh, that's a visualization. Some frequently asked questions are answered in the documentation. I could spend entire hours on all of these subjects. So if you have questions, I'll be outside all day on the Google booth. Come and see me. And Kubernetes is an open source, so we want your help making it even better. So please contribute to Kubernetes. If you have questions, go to IRC, IRC.free.net on hash Google containers. It's a very popular place. And also on Twitter, Kubernetes.io. You can tweet questions for me. If you don't answer them now, or you can find me on the booth. And ultimately, that's it. Thank you. We have time for one or two questions. At the beginning, you were talking about Borg and like five masters that you run in. And those figures are based on the data center or how does it? Based on the cell. So we break it up into a cell and each cell has its own Borg master. And that's about my limited knowledge of how the complexity of how Borg works. Not being a sweet one, sorry. But yeah, that's how it works. That's why we have multiple. So within Google, you're going to find multiple Borg clusters or Borg cells effectively. Hi. Thank you for the talk. Very interesting. When you compare VMs and containers, even if the user of a VM has root access, it's very difficult to escape from the IPvice or et cetera. How do you see the security in the current containers implementations? It's a work in progress. So this is about security with containers. I'm not really going to comment too much on it, but it's getting better all the time. Initially, we had problems with the kernel level and syscalls and such like being made back into the operating system. But it's getting better. So ultimately Docker and such like becoming more secure all the time. Ultimately, doing multi-tenant maybe currently with multiple customers applications run inside by side may not be the best idea. But we have to tackle that. So ultimately, we have to make sure people are more confident that they can run all of their jobs on containers securely. I don't think we're quite there yet, but we're working on it. So that's one challenge we need to crack. Is that a question? Yeah. One more question? Is that done? No, that's enough. OK. Come and find me outside. Come and find me outside. We can talk about PHP. It's Python, sorry. From our organization, we want to thank Mando to come in. Oh, wow. Give her a present. Thank you. Should I answer it? Should I put it? No. Oh, no. That is wonderful. Ah, fantastic. Exactly what I need. Thank you very much. Thank you. Thanks for having me. Thank you.
But now we talk about it all the time. So because lessening of the number of tasks we're running running a map reduce across a huge amount of data how we can do this externally with containers and We're probably going to want to scale this thing what they have into a spec, and we're all going of give you some of the concepts. This is the way and all of the other applications on the machine management of these containers. So at the end of quicker to get up and running. Which means you if you have a cluster of machines, changes. You you'll have a Kubernetes cluster. And ultimately, we use within Kubernetes is to create this thing makes a lot of sense, right? Ambassador, in this the PHP application to make calls and then have container will be plugged into the monitoring controller is responsible for managing your desired state. You say this is the way I want it is a managed version of Kubernetes, and it Managed Instance Groups, which we'll look at in a and a replication controller. I need to make that creates them and is responsible for making sure the cluster of nodes. And we have a template by which we can create new nodes on demand so we can what you want to do. So you can create clusters an open source, so we want your help making it it's getting better. So ultimately Docker and