cgroupv2: Linux's new unified control group hierarchy
Down, Chris
cgroupv1 (or just "cgroups") has helped revolutionise the way that we manageand use containers over the past 8 years. A complete overhaul is coming -cgroupv2. This talk will go into why a new control group system was needed,the changes from cgroupv1, and practical uses that you can apply to improvethe level of control you have over the processes on your servers. We will go over: * Design decisions and deviations for cgroupv2 compared to v1 * Pitfalls and caveats you may encounter when migrating to cgroupv2 * Discussion of the internals of cgroupv2 * Practical information about how we are using cgroupv2 inside Facebook
 Okay, so hi. My name is Chris. I already mentioned I work as a production engineer at Facebook London working as a member of Web Foundation. I'm going to be giving a whistle-tock tour of the new version of Control Groups added in Linux 4.5. Don't worry if you haven't the faintest clue what Control Groups are yet. I guess since we're in the containers, there are probably some of you have at least some kind of idea. But I'll go into kind of what they are, where you may have used them, and a comparison of the old and the new in the next few slides. So like I said in this talk, I'm going to be giving kind of an introduction to Control Groups. I'm going to go over what they are, where you may have used them, where you might have encountered them before. If you already know something called Control Groups like you've fished around previously or you've used them before, you almost suddenly have been interacting with version one of Control Groups. Control Groups has existed in the kernel since a long, long time ago, since around 2008. And it's been kind of building our love of containers. It's one of the building blocks of containers as we know them. So it's definitely got a whole bunch of good things in it. It's also got like a whole bunch of problems, usability foibles, other kinds of things which are not so great. So I want to go into like what those are and when you might encounter them and how we try to improve that in Sigur V2. Sigur V1 is currently now kind of mostly in maintenance mode, whereas Sigur V2 is more in active development. They share the same core. It's mostly the user-facing API which is kind of different. And yeah, I'm also going to go over kind of what's still to be done and what's already been done. The core of Sigur V2 is already stable since kernel 4.5, but we have a whole bunch of work we want to be done. A lot of the core work has been to enable future work, to enable new features in Sigur V2 which we already work on. So first a little bit about me. I've been working at Facebook for about three years. Like I said, I work in this team called Web Foundation. Technically, Web Foundation is the team responsible for the web servers at Facebook, but web servers are kind of not usually extremely complicated. Like they're stateless, they serve you capuchess and that's about it. So we actually ended up becoming kind of a team which deals with the general reliability at Facebook. This means we get involved with production, incident discussions and all sorts of... We generally act as kind of the guardians of reliability at Facebook. So like I said, I work at Facebook London. We have an office there. We also have an office in Dublin, in Europe, and in Tel Aviv. Yeah, and we have a whole bunch of different kinds of people in Web Foundation. We have people who are experts at Linux and Pots of Linux stack which is basically my expertise. We also have people who are experts in our cache, architecture, RPC, and say like the web push and stuff like this, especially a cross-functional group of experts which all come together to deal with when shit really hits the fan. So that kind of brings up the question, why do we as a team care about C-groups? And why does Facebook as a company care about C-groups? So we have many, many hundreds of thousands of servers and we have a bunch of services which run on these servers. Some of them co-located on the same servers. Some of them spread throughout servers. It's all kind of all over the place. So most kind of outages at Facebook are a few, there are a few kinds of outages, but like one very common one is failure in multiple systems. And there are a few things you need to do to mitigate that. Another one of course is actively mapping out your dependencies and making sure you have an understanding of your dependencies. But another big one is making sure that when two services run on a single machine, you don't end up with a situation where one service completely overrides the other and results in it becoming completely useless. So that's one of my main concerns about C-groups 2 and one of the reasons why I'm super interested in it. So a typical use case for this is like you say on a normal web server, we have three types of running processes. Same goes for most other kinds of web servers, other kinds of servers as well. So you have your core workload. Your core workload is the thing which if you were to describe to somebody else what your server was actually doing, you would say it does this thing. It serves web requests. It does load balancing. But usually you don't end up only with this on your server. You usually end up with a whole bunch of other stuff, especially if your company has been around for a while or you have some ideas about how you should architect stuff. You end up with a whole bunch of non-core services which can probably be used interchangeably with kind of system services. This might be stuff which just comes with Linux like also the kernel workers or it could also be like say, Kobra's Demons or it could be like stuff which is related to your business needs say metric collections so we can work out what is going on with our server like whether we're managing to serve users correctly. But it's really, really bad if then this metric collection decides it's going to take up all the available memory in your server and then you can't actually serve users. Sure, it can tell me it did that but I don't give a fuck if it like completely fucked up the whole web server, right? Same goes for cron jobs and chef. I care about my server being up to date but I would rather have some system which actually acts in a reasonable manner if chef starts taking a bunch of memory. I would rather it got degraded service and still manage to run than it took all of the memory or took all the CPU and then everything was fucked. Like I would much rather have that outcome. Then we can have this third class like ad-hoc queries and debugging. These are typically things you don't know that you need and don't end up getting run on the majority of servers. These are like things which you end up realizing you need only when an incident is already happening. We kind of want people to be able to dynamically be able to determine the importance of those things as the incident is going on, whether they want it to take precedence over the core workload of the machine or not. C-groups is a very good use case. Sorry, this is a very good use case for C-groups. Like I mentioned, if you had some interaction with C-groups, you've almost certainly been interacting with version one. Version two has been in development for like five years now. It only just got stable in the Linux kernel. But even on recent kernels, version one is typically what's mounted by default. The reason for that is as you'd imagine with a different version number, the actual changes are backwards incompatible. I'm going to be going over kind of why we've made them backwards incompatible in a moment. The fact that we typically boot with your init system only mounting the version one hierarchy is a testament to why I'm doing this talk. I know we have a whole group full of container experts in this room. This is also kind of a sell to you about why you should give a fuck about C-group V2 and why you should take the time to care about it and invest in it in your products. So in the previous slide, we talked about multiple processes fitting into each of these C-groups as well. So a C-group can be set as tightly or as flexibly as you like. It can be all of the processes which are related to one service or it can be a single process. It can be however many processes you like from zero to however many you like. The idea here is we don't impose a structure on you. One of the guiding principles behind V2 has been we want you to be able to choose how to use it. We don't want to have a hierarchy which imposes how you should do things. We want the easy way to be the correct way. So a C-group is a control group. They're one and the same. They're a system for resource management on links. Like I mentioned, a resource here means CPU, I owe memory and management here can mean for example accounting. We know how much memory some processes in some particular C-group are using. It can also be limiting. We hit a particular threshold and we take some violent action to curb that. It can also, in V2 there's also been some work towards throttling which I'll go in as well. Generally, um-killing or whatever is quite violent. So we want to have some remediative actions instead of just straight up killing stuff. So the way that C-group V2 and V1 work are essentially you have this hierarchy at CISFS C-group. It's just a bunch of files and directories. We don't have a system call interface. There may well be one in future but we don't currently have one. The reason this is kind of a good thing is it's really easy from some user space application even if it's not written in C or C++ and you don't have access to C library functions or system calls easily. You can easily go and find out the state of your system when it comes to C-groups. You can just look at some file like I would hope that all of you are using languages which can open a file, make a directory, remove a directory. I would hope whatever the new hip kids are using still supports those kind of things. So each resource interface is provided by a controller. You'll probably hear me use the word controller, resource and domain kind of interchangeably. The idea is you write files that apply to some particular controller. This controller takes the values which you provide and it provides them to the kernel and the kernel makes some kind of decisions based on what values you gave it. So it's essentially the backing store for all of the stuff which you input to C-groups. So as mentioned previously, workload isolation is a really big use case for C-groups. So you might have many background jobs and a machine but you don't want them to override the main workload. That's the case which we talked about first. There are also some other kind of other cases like say you have a tier which runs asynchronous jobs in the background. We have that at Facebook. And some jobs have higher priority than others. Priority is a very abstract concept but priority usually has something to do with resources. It usually means either you expect it has more memory or less CPU or something like this. It's very much up to you what exactly that priority means. You also have shared environments say you're a VPS provider or something like that and you don't want some particular user to be able to use that container and steal all the resources from all your other customers which are now going to go and leave you a bad review. So yeah, there are kind of a lot of use cases for C-groups here. And you might ask, hey, my favorite application X, it already does this. Why should I give a single shit about C-groups? Well, the answer is if you have been using any kind of software in the last eight years, I would really pray that it does it through C-groups. It probably does it transparently to you and you don't ever have to talk to C-groups directly. But the backing for all of these is C-groups. That's how they do resource limiting. So let's go concretely over how this works in version one. So in V1, CISFS C-group contains the names of all the resources which have a controller. So this might be CPU, memory, PIDs, that kind of stuff. And inside these resource directories, there's another set of directories which are the C-groups themselves. So these C-groups exist in the context of this resource and you put processes into those C-groups. For example, we have the PIDs one at the bottom. So because this is to do with PID resources, it contains files in those directories and those files are related to how many PIDs you can have in a C-group, for example. And we also have the concept of having its own hierarchy for resource distribution. So in C-group V1, you have a resource and then you have a C-group hierarchy underneath each resource. So even if C-group 3 here was called the same as C-group 1, say they were both called FOO, from the kernel's perspective, they have absolutely no relation to each other. This is kind of important because if you look at how, for example, system D sets out C-groups typically, you often end up with quite similar looking hierarchies in different resources and you might even be inclined to believe they have some relation to each other. Well, from system D's perspective, I'm sure they do, but from the kernel's perspective, they do not. And that results in a whole slew of like subtle issues which cause problems at scale which I'll go into in a moment. C-groups are nested inside each other in this example. So when a C-group is nested inside another, typically what it means is it can control some limited amount up to the maximal amount of its parent. So if you have a memory C-group and then you have a child of a C-group which is also another C-group, then you can limit up to the maximum of its parent. So the resource that controls these C-groups or the C-group hierarchy that they're in determines what kind of files there are. I already mentioned that if you're in the memory hierarchy, you can only access files related to memory. Like for example, that's this file memory.limitinbytes. You can read the limitinbytes from it or you can write to it and set a separate limitinbytes. And one PID is an exactly one C-group per resource in C-group V1. So PID 2 here is explicitly assigned to resources A and C in C-group 1 and 5 respectively, but because we don't assign it to anything in resource B, it actually goes to this root C-group. The root C-group is kind of a special concept in C-groups. It's essentially unmanaged territory. How exactly it's managed is up to the controller. But the idea is you don't really have the opportunity to set really any limits in the root C-group because it's just the starting point for resource distribution of this resource across your whole machine. So yeah, you do get some kind of accounting, but in terms of limiting, it's basically useless. So here is a concrete look at how this looks in C-group V1. So you have C-to-Fest C-group and then you have the resources, which are like block I, M, M, and PIDs. And then you have the C-group names and we have nested C-groups here, A inside BG for two resources and B inside ad hoc for two resources. So once again, just as a reiteration because this is really important that you get this, from the Kahnels perspective, naming has no meaning. Like if it's in a different resource, even if it has the same name, it has no meaning. And that has also some weird implications. So here's how it looks in C-group V2 by comparison. So in C-group V2, we actually don't see the resources anymore. If you remember how it looked in version one, we have resources under C-to-Fest C-group, but now we actually have the C-groups themselves under C-to-Fest C-group. So how do these C-groups understand which resources they are supposed to apply to if they're not inside a resource hierarchy? Well, the answer is kind of, C-groups are global now. They're essentially a global set of C-groups and you enable resources inside the C-groups. This means that we have one hierarchy to rule them all. And the idea is like, you write to a special file, you tell us what particular controllers you want to enable, and we enable them for your C-group. We don't require you to create disparate hierarchies each time. You instead create one hierarchy and enable C-group controllers at will. So this is how the previous example now looks in C-group V2. So in C-group V2, like I mentioned, we now have the C-groups directly at the bottom, but you write to this special file, C-group.subtreeControl, and that enables the, in the children of that C-group, that those controllers are enabled. Essentially if you were not to enable them there, but they were enabled the next level up, it means they would compete freely for those resources that you didn't enable. So here's the version one hierarchy again for comparison. And as you can see here, we have resources first. And remember that in version one, C-groups with the same name again don't have any relation to each other. In C-group V2, we have this unified hierarchy, and you enable resources for C-groups children by writing plus memory, plus PIDs, plus CPU, plus IO, that kind of stuff to C-group.subtreeControl. And when you do this, the files appear in that directory instantaneously. Oh, and another thing to mention is, in real life, you also need to enable the memory PIDs and IO controllers at the top level for this to work, but for the sake of simplicity, I'm not out of them here. So the fundamental differences are obviously the unified hierarchy. Resources apply to C-groups now instead of C-groups applying to some, some distance across the hierarchy for a resource. This is very important for some kinds of common operations in Linux. Like, for example, you have PageCache writebacks, and PageCache writebacks transcend one particular resource. They happen across a whole bunch of different resources, and we need to be able to consider these actions together in Unity to be able to form reasonable limiting or other actions. We also have granularity at the thread group ID, not the thread ID level. This is a contentious point, but it's kind of important. In C-group V1, you could essentially put different threads from the same process into separate C-groups. This has a whole bunch of weird implications, like, for example, people would put, say, different threads from the same process into different memory C-groups. I don't know how the fuck that's supposed to work. Like, you have, like, basically the entire shared memory between these two fucking C-groups. I don't know. People have done insane shit with C-groups. This is the main thing is, like, we want to guide towards a reasonable implementation, because it's not that these people are stupid. That's not the problem. It's just that C-groups in V1 were, like, quite over-complicated, so it made people do some insane shit. So now limiting the PID level kind of gets us, like, a more reasonable approximation of what people generally want. Also, without extensive cooperation, even for the resources where maybe it would, in theory, make sense to have different threads from the same process in different resources, it often ends up being, like, you have to have some way to communicate which thread from your process is doing what. And there's no standardized way to do that in Linux, right? You can set the calm of your thread to, like, some value and then look at, like, the value of somewhere, but it's not standardized, and it's, like, really, really fucking hard to reason about. So yeah, it usually doesn't act in any reasonable way. In general, like, there's been a focus on simplicity and clarity in V2 over, like, ultimate flexibility. V1 was invented at the kind of the dawn of containerization. People didn't know what they wanted. They just know that they wanted something and they wanted it now. So V1 was kind of a solution to this problem, and V2 was kind of more like a more developed approach to the problems we know we're having for sure now. So another new feature in V2 is the addition of this no internal process constraint. So this means that C groups with processes and controllers enabled cannot create child C groups. Essentially, this means that in simpler words, these red C groups either have to have no processes or they have to have no controllers enabled in that part of the hierarchy. This is for a few reasons. It's kind of hard to reason about how that should act usually. In V1, this was allowed, and the problem is now you have two different types of objects kind of competing against each other. So say you have processes in I, and then you have some child C groups underneath I here. Now you have processes which are in I competing against C groups which are its children, and it's difficult. You have to make some kind of judgment about how we will treat processes compared to C groups. Maybe we can consider each one its own C group. Maybe we can consider them like I prime like a separate C group, but it's quite hard to reason about. For most cases, the better solution is just create another C group. So this is another guide towards like helping people create a sane hierarchy. The root C group is a special case. The controllers themselves have to decide how they're going to handle resources in the root. So clearly breaking the API is kind of a big deal. The C group API is a pretty fucking big deal. Like the fact that we want to create V2 instead of just improving V1 obviously needs some good reasoning there. So V1 works okay for basic situations, but it gets kind of exponentially complicated when you're getting more and more complex. As I mentioned in V1, design kind of often followed implementation. And trying to rework kernel APIs after the fact is really, really, really hard. Like you can't change the kernel API, the fundamental nature of the kernel APIs which people rely on day to day in production. That's just not something you can do. Even for stuff which was designed up front, like I mentioned, like generally the use cases for containers and C groups were not really that well fleshed out yet. They originally started as like only for CPU and then it grew and grew and grew kind of naturally. It was generally hard to gauge at the time how C groups would be used. So now this is an opportunity to redesign them and work them how we actually think they should be. So to fix these kind of fundamental issues, you need to have an API break and that's why V2 was created. So I want to go over some of the actual practical improvements because I've talked a lot about like the theoretical, how we've designed it, but I also want to go over why we've designed it in that way and what that actually means. So okay, pop quiz. When you write to a file in Linux, what happens? It's not a direct question, that's where. You get a file descriptor. Okay, that was possibly before what I wanted. Okay, so does it write directly to the disk? Okay, so where does your data go? I got about five different answers. I'm not sure what and how they were. Okay, so it goes, you write a dirty page into the page cache. You have like some sort of dirty pages now and your right syscall came back and it said everything's great. And from your application's perspective, now you can go on pretending, yes, it was written to disk and my right syscall succeeded. So it must be written to disk and you can have all this class of wonderful belief. But ultimately, it's not been written to this, right? It's still sitting in memory somewhere and if you shut down the machine right now, shit's going to go hey, what? So yeah, so there are multiple operations here to be considered. First, you have your right syscall, which then goes and writes it at dirty pages, returns to you. And then later, some kernel worker like PD flush comes and it says, okay, now is the time by some magical standards like the I know dirty ratio. I've decided I'm going to flush out these to disk. In v1, we don't have any tracking of this. So if you wrote dirty pages, we don't know where they came from afterwards when we flush them to disk. So that IO goes to the root cgroup and we can't account it to your process or your cgroup simply because it wasn't tracked. It wasn't tracked where this page comes from. In v2, it is tracked and we can actually count these towards your limit. So we can also make reasonable decisions about like, oh, you have IO contention and this is what I should do based on that or you have memory contention and this is what I should do based on that when you're trying to do a page cache write back. V2 is also generally kind of better integrated with subsystems. So most of the actions we could do based on thresholds in version one were kind of crude or in the case of the memory subsystem, quite violent. You set like a limit with memory.limit invites and what happens is your application has like a one minute spike in memory usage and then the umkiller goes along and goes oh, I'm going to kill you. And that's like the standard method of dealing with things. Usually like processes don't particularly like being kill-nined. I don't know, like maybe like there's some kind of statistic processes which like that. But I mean, ultimately it's not a very good way of limiting results usage. What you really want to do is like tell it okay, calm the fuck down and like stop allocating memory. Or in the better case where you can't tell the calm the fuck down, you want to tell the operating system hey, that guy's going fucking nuts though. I would like to now like take some action against this guy. But that action doesn't have to be like slay him where he stands. Yeah, I would like to think in human society we've come past the point where like the penalty for any kind of failure is instantaneous death. So yeah, so now in secret v2 we have like generally kind of better thresholds here. We have a new thing called memory.high which instead of killing a process, we still have memory.max which is very analogous to memory.limit invites which uncilled your process. But we also have memory.high and what memory.high does is when you pass this threshold, we start to do throttling and reclaim for every single memory allocation. So when you go over memory.high and you want to do another malloc or you want to get grab some more memory, then what we do is we break into a separate path in the kernel and we say hey, I would like to dial back this usage. So what I'm going to do is go to the tail of the inactive list and start reclaiming pages. So if you fail to reclaim any pages, it's still kind of good because you took a while to scan the page cache. It took you a while to scan the page cache. We've slowed down your application now and we've done it in a way which is kind of transparent to your application. But if you do manage to reclaim pages, then we also win because now you've retained some pages and you managed to get some memory free again. In fact, this is like a generally a much-sane way of doing things and this is like using this on web servers was like a big win when you see like these spikes in resource users. So notifications, I don't know how notifications are like one of the more edge cases for C-groups since it usually ends up being people like systemd which end up using them, but notifications are essentially a way to say hey, something in my C-group changed state. It could be like oh, I have no more processes in my C-groups so all of them have finished. It could be like oh, one of my processes ran out of memory and I'm going to take some action based on that. Ultimately, it's a way to get information about what is happening in your C-group. Systemd uses this for example to track which processes are running and the state of your system and the state of the services which are running. The problem is on v1 for release notifications which are the notifications which are sent when your C-group has no more processes which for example means like oh, we exited. You have to designate what's called a release agent and this release agent is just like giving a call dump utility. You tell it like here's the path to my executable and when you have no more processes go and execute this thing with these arguments. The problem is now if you're using C-groups as a utility where say you have C-groups expiring like a thousand times a second, you're now like forking a thousand times a second as well which is pretty bad. It's generally pretty expensive and it doesn't make a whole lot of sense since the rest of the C-group API used slightly more sane methods using eventfd. So now we have a notify support everywhere. CIS events C-group looks like a false system. It kind of makes sense that it supports iNotify. We still have eventfd support so you can like poll and find out what the answer is. The idea now is you can have one process to monitor everything you like. You don't have to fork a new process every time an event is created. This is just like a straight upgrade really. So utility controllers are another thing. Utility controllers are controllers essentially most controllers in C-group are related to some particular resource. Say you have memory, CPU, IO. There are other ones however like say Perf or Freezer which I'll also go into in a second which are not related to some resource but you put the processes in that C-group based on some actions you want to take to them as a group. So the idea is like basically you want to group them together so some user space utility can take some action based on that group. Perf is a tool for performance monitoring and tracing analytics I guess. Quite a few people have probably heard of it. And the way it works is when you say I want to have a certain set of C-groups is you give it a C-group path and it says okay here is like some particular set of processes which I'm going to map into my own C-group hierarchy. So now you have CISFS C-group Perf and inside there is a completely separate C-group hierarchy which only relates to Perf. This usually doesn't make a whole lot of sense. Usually what people want to do is monitor an existing C-group hierarchy not create a new one. So people ended up resorting to like tons of hacks like oh I want to like copy over this hierarchy to the other one. They would write like a tool to copy it from one hierarchy to another and like you end up with all these race conditions and like horrible and it was like really, really bad. So now having a unified hierarchy means we don't have to sync. You only have one hierarchy. So there's no way this could possibly go wrong. So in V1 there's also like a lot of inconsistency between controllers. This usually comes in kind of two forms. One you have inconsistent APIs between controllers which do exactly the same thing. So you have like the CPU, both CPU and I owe are essentially weight based or share based. You give a certain amount of some resource based on a relative amount to some other particular C-group. But the APIs were completely different. You had to learn two APIs to do one thing which is really, really not ideal. So there's been a lot of focus on trying to unify the APIs and also unify the naming. Like generally like it was a bit of a crap shoot in V1 and now we have the opportunity to rethink those names and generally standardize them a bit more. So V2 is generally more intuitive up front. Oh, another one is kind of inconsistent C-group semantics. For example, root C-groups, for example, sorry, most C-groups inherit their parent's limits. So when you create a child C-group of some particular C-group, it usually can only use up to its parent's limits. But some controllers didn't do that. Some controllers did their own thing. For some controllers, this whole idea of a hierarchy was like an imaginary thing and it just created a new C-group and didn't care where it was. It was all a bit of a crap shoot. So now with one unified hierarchy, it's more difficult to fuck it up, I guess. So V1's over-flexibility also contributed to like a whole bunch of API problems. For example, when memory limits were first created, they only limited a few, like a couple types of memory. And they were in this file memory.limitedbytes. Then as more and more memory types were added, they ended up getting like their own files one by one. So you ended up with memory.limitedbytes, memory.kmem.limitedbytes, memory.kmem.tcp.limitedbytes, memory.msw.limitedbytes. And the really bad thing about this is now, yes, I have very granular control, but it's not very useful. Because now, say I want to set a limit on the maximum number of TCP buffers, which is set with memory.kmem.tcp.limitedbytes. Now, say I have like 10 gigabytes of page cache free, and I set a limit on the number of TCP buffers. If I said like, oh, you should only get say X amount of TCP buffers. If you go one over, we'll kill you. Even if you had 10 gigabytes of page cache free or something like that. It's not a reasonable way of operating. Usually, like most people don't want, like most people don't care, like you allocated one TCP buffer too many. They want to give some kind of idea about the overall memory use and like reasonable memory use. Generally, unified limits are only reasonable way to approach that. So, yeah, again, another trade-off kind of in favor of usability over ultimate flexibility. Generally, if you do want to limit these things, say you wanted to limit some particular kind of resource, like the PID controller is a very good example. The PID, in the early days of C groups it was considered, maybe we could limit the number of PIDs by like limiting certain types of kind of memory. But that turns out to be really, really fucking hard. Like, so another, the way that was fixed was we now have a PID controller and that specifically controls this resource. So if you do want to do very specific kinds of limiting like some TCP buffer or something else, then you should do that through a new controller. That's the reasonable way to do that. So, if you go to Facebook.com right now, there is a one in 10 chance you are going to hit a server with C group B2. We are running a C group B2 pool in like the tens of thousands of machines now. And we are investing heavily in C group B2 for like a number of reasons. My main concern, like I mentioned at the beginning, is limiting the failure domains between services. I really care a lot about making sure that we don't have cascading failures or anything like that on a machine. And also being able to kind of manage the resource allocation in your data center, especially at Facebook sale, is really important. Like, if we can suck at this, that little bit more resource efficiency at the data center, then that's a really big win. We run C group B2 managed with system D. One of my teammates, Dav DeCapico, was sitting back there, did a talk about this at systemd.com for last year called, I believe it was called, deploying system D at scale, correct me if I'm wrong? Yes, it was. So yes, we're a really big contributor to the core of C group B2. We have two of the core maintainers working at Facebook. And we will continue to drive innovation here. Like this is a big thing we're working on right now. So I already mentioned that C group B2 is kind of new. C group B2 has been usable for a little while now, and the core is all very stable. But that doesn't mean that there isn't still work to be done here. Like a lot of this is kind of building the building blocks for future work. So the core APIs are stable, but there's definitely functionality to be worked on. When thinking about C groups, like most people think of kind of three things, I guess, which is IO memory, CPU, those are like pretty much the biggest ones. And two out of three of those are merged currently. The PID controller is also merged. For the CPU controller, like it's very important, but there have been some disagreements with the CPU subsystem team about how to merge it. They have some disagreements about the no internal process constraint and also like having process granularity instead of thread granularity and stuff like this. Yeah, there's a very juicy drama filled thread at that link. As all Linux kernel mining list threads are, it's probably still better than usual. We also may end up with some thread based API for some particular kinds of thread operations that make sense. That would be like in the works. Another kind of big bet that we're doing right now is one thing that Linux has never really had is a good metric for memory pressure. Like we have many kind of related metrics like the amount of memory have free or the amount of memory used. You can also look at stuff like certain kinds of page scans or like, yeah, it's, but it's all very heuristic. And ultimately, none of these metrics prove that you actually are encountering memory pressure because they can also happen in a bunch of normal scenarios. So our proposed measure is to track page refolting. So the essential way this will work is we track pages which are consistently refalted back into the inactive queue. And essentially the way to explain how this whole thing works. So you have the inactive set. Those are pages which the kernel considers are probably not being actively used by any process. Then you have the active set which you consider is more likely to be used by some process. And it's essentially one big list. And the way it works is when you have a page fault, you go to the head of the inactive list. If you got, if that page was accessed again, then it gets moved to the head of the active list, which means it's protected from reclaim. And when we do a reclaim, we go from the tail of the inactive list. So we take the pages which we consider the least likely to be used of a reclaim. So what happens if we keep on faulting pages in and they get so far to the end that we keep on reclaiming them and then they fault back in again immediately? That probably means that we have too many pages at once for our system to handle. It probably means that we can't, we end up pushing them so fast off the edge that we simply don't have the resources to be able to deal with this number of pages. That is probably not a bad metric for memory pressure. And it's one which is currently being worked on as part of the security effort as well. Because we do want to have those metrics around memory pressure, not just memory usage, which is kind of only tangential to the thing which you really want to know. So, sorry question. I thought I heard a question. All right. So we also have tracking now of page caps. Right, Max? What this first point means is we have the tracking of memory in IO, but we don't have tracking for CPU yet. Like you spend some time on CPU waiting for network packets to come in on the network. We, and we don't know who it's for yet because it didn't get rooted yet. And we can't account for that yet. And we also can't account for the CPU we spent like doing a page cache writeback yet. That's something which is going to take quite a bit of effort. But it's something which we're definitely working on at this point. Yeah. And as for the second point, I already mentioned like the, the refalting. Generally, the idea here is that we want better metrics for, for memory pressure because right now we only have something tangential. Those of you who use V1 probably also know about freezer. Freezer is like an alternative to umkilling, for example. Like you can freeze some set of processes in their state and then go and decide like, oh, I want to raise the memory limit or I want to kill them or I want to stop some new processes. It was essentially a way of freezing them in time and having some other process come and decide what to do about it. In, in Sigur V1 this didn't work at all basically. Like if you, if you used freezer, one of the most common things you might want to do is go and like get a stack trace and look at what they were doing when like they kept on allocating memory or whatever it was that you froze them for. But like it's a very common situation that you would end up with like say GDP. If you would try to attach GDP, it would end up in D state, which is not really the ideal result if you want to like find out the stack of some process which froze like that's generally the complete opposite of what, what I would like. The reason is like the freezer implementation in V1 doesn't guarantee that we stop anywhere reasonable. We, we often stop in a stack which makes absolutely no sense in the kernel. So in V2, the idea is to have like a more kind of sigstop style mechanism. Sigstop is very well defined and where it stops is very well defined as well. So it's, it's kind of a more reasonable solution to use for stopping processes. So the, the, the V2 implementation of freezer will be more along those kind of semantics. So I talked a lot here about trying to sell you a secret V2, but I should actually tell you how to get it probably at some point during this talk. So hopefully you're interested in trying it out yourself. So here's what you need to get started with version two. First you need a kernel above 4.5. Before that we do have a developer flag, which you can go find. I won't tell you because it'll like eat you if you try and use it and you don't know what you're doing. But I really wouldn't recommend using it before 4.5. 4.5 is the first point. We have a stable API. Once that's done, more or less two things to do. You need to turn off all of the controllers for V1 and you need to turn on and mount the, the, the file system for V2. Typically you want your init system to do this. For system D you use it with this flag. You basically put both of these on the kernel command line. But if you're crazy or you want to try it yourself, you can also manually mount it and cry when things break. So yeah, so if you're interested in hearing more about C groups, come talk to me. I'm happy to go over any of what I've been talking about on V1 or V2. And yeah, I'll be happy to go over any questions you might have. And if you've used V1 in the past, which I guess many of you have, and you found it lacking in some areas, please do try out V2 and let us know what you think. Thanks. Thank you.
 Okay, so hi. My name is Chris. I already mentioned I work as a production engineer at Facebook  London working as a member of Web Foundation. I'm going to be giving a whistle-tock tour  of the new version of Control Groups added in Linux 4.5. Don't worry if you haven't  the faintest clue what Control Groups are yet. I guess since we're in the containers,  there are probably some of you have at least some kind of idea. But I'll go into kind of  what they are, where you may have used them, and a comparison of the old and the new in  the next few slides. So like I said in this talk, I'm going to be giving kind of an introduction  to Control Groups. I'm going to go over what they are, where you may have used them, where  you might have encountered them before. If you already know something called Control  Groups like you've fished around previously or you've used them before, you almost suddenly  have been interacting with version one of Control Groups. Control Groups has existed  in the kernel since a long, long time ago, since around 2008. And it's been kind of  building our love of containers. It's one of the building blocks of containers as we  know them. So it's definitely got a whole bunch of good things in it. It's also got  like a whole bunch of problems, usability foibles, other kinds of things which are not  so great. So I want to go into like what those are and when you might encounter them and  how we try to improve that in Sigur V2. Sigur V1 is currently now kind of mostly in maintenance  mode, whereas Sigur V2 is more in active development. They share the same core. It's mostly the  user-facing API which is kind of different. And yeah, I'm also going to go over kind of  what's still to be done and what's already been done. The core of Sigur V2 is already  stable since kernel 4.5, but we have a whole bunch of work we want to be done. A lot of  the core work has been to enable future work, to enable new features in Sigur V2 which we  already work on. So first a little bit about me. I've been working at Facebook for about  three years. Like I said, I work in this team called Web Foundation. Technically, Web Foundation  is the team responsible for the web servers at Facebook, but web servers are kind of not  usually extremely complicated. Like they're stateless, they serve you capuchess and that's  about it. So we actually ended up becoming kind of a team which deals with the general  reliability at Facebook. This means we get involved with production, incident discussions  and all sorts of... We generally act as kind of the guardians of reliability at Facebook.  So like I said, I work at Facebook London. We have an office there. We also have an office  in Dublin, in Europe, and in Tel Aviv. Yeah, and we have a whole bunch of different kinds  of people in Web Foundation. We have people who are experts at Linux and Pots of Linux  stack which is basically my expertise. We also have people who are experts in our cache,  architecture, RPC, and say like the web push and stuff like this, especially a cross-functional  group of experts which all come together to deal with when shit really hits the fan.  So that kind of brings up the question, why do we as a team care about C-groups? And why  does Facebook as a company care about C-groups? So we have many, many hundreds of thousands  of servers and we have a bunch of services which run on these servers. Some of them co-located  on the same servers. Some of them spread throughout servers. It's all kind of all over the place.  So most kind of outages at Facebook are a few, there are a few kinds of outages, but  like one very common one is failure in multiple systems. And there are a few things you need  to do to mitigate that. Another one of course is actively mapping out your dependencies  and making sure you have an understanding of your dependencies. But another big one  is making sure that when two services run on a single machine, you don't end up with  a situation where one service completely overrides the other and results in it becoming completely  useless. So that's one of my main concerns about C-groups 2 and one of the reasons why  I'm super interested in it. So a typical use case for this is like you say on a normal  web server, we have three types of running processes. Same goes for most other kinds  of web servers, other kinds of servers as well. So you have your core workload. Your  core workload is the thing which if you were to describe to somebody else what your server  was actually doing, you would say it does this thing. It serves web requests. It does  load balancing. But usually you don't end up only with this on your server. You usually  end up with a whole bunch of other stuff, especially if your company has been around  for a while or you have some ideas about how you should architect stuff. You end up with  a whole bunch of non-core services which can probably be used interchangeably with  kind of system services. This might be stuff which just comes with Linux like also the  kernel workers or it could also be like say, Kobra's Demons or it could be like stuff  which is related to your business needs say metric collections so we can work out what  is going on with our server like whether we're managing to serve users correctly. But it's  really, really bad if then this metric collection decides it's going to take up all the available  memory in your server and then you can't actually serve users. Sure, it can tell me  it did that but I don't give a fuck if it like completely fucked up the whole web  server, right? Same goes for cron jobs and chef. I care about my server being up to date  but I would rather have some system which actually acts in a reasonable manner if chef  starts taking a bunch of memory. I would rather it got degraded service and still manage to  run than it took all of the memory or took all the CPU and then everything was fucked.  Like I would much rather have that outcome. Then we can have this third class like ad-hoc  queries and debugging. These are typically things you don't know that you need and don't  end up getting run on the majority of servers. These are like things which you end up realizing  you need only when an incident is already happening. We kind of want people to be able  to dynamically be able to determine the importance of those things as the incident is going on,  whether they want it to take precedence over the core workload of the machine or not.  C-groups is a very good use case. Sorry, this is a very good use case for C-groups. Like  I mentioned, if you had some interaction with C-groups, you've almost certainly been interacting  with version one. Version two has been in development for like five years now. It only  just got stable in the Linux kernel. But even on recent kernels, version one is typically  what's mounted by default. The reason for that is as you'd imagine with a different  version number, the actual changes are backwards incompatible. I'm going to be going over kind  of why we've made them backwards incompatible in a moment.  The fact that we typically boot with your init system only mounting the version one  hierarchy is a testament to why I'm doing this talk. I know we have a whole group full  of container experts in this room. This is also kind of a sell to you about why you should  give a fuck about C-group V2 and why you should take the time to care about it and invest  in it in your products.  So in the previous slide, we talked about multiple processes fitting into each of these  C-groups as well. So a C-group can be set as tightly or as flexibly as you like. It  can be all of the processes which are related to one service or it can be a single process.  It can be however many processes you like from zero to however many you like. The idea  here is we don't impose a structure on you. One of the guiding principles behind V2 has  been we want you to be able to choose how to use it. We don't want to have a hierarchy  which imposes how you should do things. We want the easy way to be the correct way.  So a C-group is a control group. They're one and the same. They're a system for resource  management on links. Like I mentioned, a resource here means CPU, I owe memory and management  here can mean for example accounting. We know how much memory some processes in some particular  C-group are using. It can also be limiting. We hit a particular threshold and we take  some violent action to curb that.  It can also, in V2 there's also been some work towards throttling which I'll go in  as well. Generally, um-killing or whatever is quite violent. So we want to have some  remediative actions instead of just straight up killing stuff. So the way that C-group  V2 and V1 work are essentially you have this hierarchy at CISFS C-group. It's just a bunch  of files and directories. We don't have a system call interface. There may well be one  in future but we don't currently have one. The reason this is kind of a good thing is  it's really easy from some user space application even if it's not written in C or C++ and you  don't have access to C library functions or system calls easily. You can easily go and  find out the state of your system when it comes to C-groups. You can just look at some  file like I would hope that all of you are using languages which can open a file, make  a directory, remove a directory. I would hope whatever the new hip kids are using still  supports those kind of things.  So each resource interface is provided by a controller. You'll probably hear me use the  word controller, resource and domain kind of interchangeably. The idea is you write  files that apply to some particular controller. This controller takes the values which you  provide and it provides them to the kernel and the kernel makes some kind of decisions  based on what values you gave it. So it's essentially the backing store for all of the  stuff which you input to C-groups.  So as mentioned previously, workload isolation is a really big use case for C-groups. So  you might have many background jobs and a machine but you don't want them to override  the main workload. That's the case which we talked about first. There are also some other  kind of other cases like say you have a tier which runs asynchronous jobs in the background.  We have that at Facebook. And some jobs have higher priority than others. Priority is a  very abstract concept but priority usually has something to do with resources. It usually  means either you expect it has more memory or less CPU or something like this. It's very  much up to you what exactly that priority means. You also have shared environments say  you're a VPS provider or something like that and you don't want some particular user to  be able to use that container and steal all the resources from all your other customers  which are now going to go and leave you a bad review. So yeah, there are kind of a lot  of use cases for C-groups here.  And you might ask, hey, my favorite application X, it already does this. Why should I give  a single shit about C-groups? Well, the answer is if you have been using any kind of software  in the last eight years, I would really pray that it does it through C-groups. It probably  does it transparently to you and you don't ever have to talk to C-groups directly. But  the backing for all of these is C-groups. That's how they do resource limiting.  So let's go concretely over how this works in version one. So in V1, CISFS C-group contains  the names of all the resources which have a controller. So this might be CPU, memory,  PIDs, that kind of stuff. And inside these resource directories, there's another set  of directories which are the C-groups themselves. So these C-groups exist in the context of  this resource and you put processes into those C-groups. For example, we have the PIDs one  at the bottom. So because this is to do with PID resources, it contains files in those  directories and those files are related to how many PIDs you can have in a C-group, for  example. And we also have the concept of having its own hierarchy for resource distribution.  So in C-group V1, you have a resource and then you have a C-group hierarchy underneath  each resource. So even if C-group 3 here was called the same as C-group 1, say they were  both called FOO, from the kernel's perspective, they have absolutely no relation to each other.  This is kind of important because if you look at how, for example, system D sets out C-groups  typically, you often end up with quite similar looking hierarchies in different resources  and you might even be inclined to believe they have some relation to each other. Well,  from system D's perspective, I'm sure they do, but from the kernel's perspective, they  do not. And that results in a whole slew of like subtle issues which cause problems at  scale which I'll go into in a moment. C-groups are nested inside each other in this example.  So when a C-group is nested inside another, typically what it means is it can control some  limited amount up to the maximal amount of its parent. So if you have a memory C-group  and then you have a child of a C-group which is also another C-group, then you can limit  up to the maximum of its parent. So the resource that controls these C-groups  or the C-group hierarchy that they're in determines what kind of files there are. I already mentioned  that if you're in the memory hierarchy, you can only access files related to memory. Like  for example, that's this file memory.limitinbytes. You can read the limitinbytes from it or you  can write to it and set a separate limitinbytes. And one PID is an exactly one C-group per  resource in C-group V1. So PID 2 here is explicitly assigned to resources A and C in C-group 1  and 5 respectively, but because we don't assign it to anything in resource B, it actually  goes to this root C-group. The root C-group is kind of a special concept in C-groups.  It's essentially unmanaged territory. How exactly it's managed is up to the controller.  But the idea is you don't really have the opportunity to set really any limits in the  root C-group because it's just the starting point for resource distribution of this resource  across your whole machine. So yeah, you do get some kind of accounting, but in terms  of limiting, it's basically useless. So here is a concrete look at how this looks in C-group  V1. So you have C-to-Fest C-group and then you have the resources, which are like block  I, M, M, and PIDs. And then you have the C-group names and we have nested C-groups here, A  inside BG for two resources and B inside ad hoc for two resources. So once again, just  as a reiteration because this is really important that you get this, from the Kahnels perspective,  naming has no meaning. Like if it's in a different resource, even if it has the same name, it  has no meaning. And that has also some weird implications.  So here's how it looks in C-group V2 by comparison. So in C-group V2, we actually don't see the  resources anymore. If you remember how it looked in version one, we have resources under  C-to-Fest C-group, but now we actually have the C-groups themselves under C-to-Fest C-group.  So how do these C-groups understand which resources they are supposed to apply to if  they're not inside a resource hierarchy? Well, the answer is kind of, C-groups are global  now. They're essentially a global set of C-groups and you enable resources inside the C-groups.  This means that we have one hierarchy to rule them all. And the idea is like, you write  to a special file, you tell us what particular controllers you want to enable, and we enable  them for your C-group. We don't require you to create disparate hierarchies each time.  You instead create one hierarchy and enable C-group controllers at will.  So this is how the previous example now looks in C-group V2. So in C-group V2, like I mentioned,  we now have the C-groups directly at the bottom, but you write to this special file, C-group.subtreeControl,  and that enables the, in the children of that C-group, that those controllers are enabled.  Essentially if you were not to enable them there, but they were enabled the next level  up, it means they would compete freely for those resources that you didn't enable. So  here's the version one hierarchy again for comparison. And as you can see here, we have  resources first. And remember that in version one, C-groups with the same name again don't  have any relation to each other. In C-group V2, we have this unified hierarchy, and you  enable resources for C-groups children by writing plus memory, plus PIDs, plus CPU, plus IO,  that kind of stuff to C-group.subtreeControl. And when you do this, the files appear in  that directory instantaneously. Oh, and another thing to mention is, in real life, you also  need to enable the memory PIDs and IO controllers at the top level for this to work, but for  the sake of simplicity, I'm not out of them here. So the fundamental differences are obviously  the unified hierarchy. Resources apply to C-groups now instead of C-groups applying to some,  some distance across the hierarchy for a resource. This is very important for some kinds of common  operations in Linux. Like, for example, you have PageCache writebacks, and PageCache writebacks  transcend one particular resource. They happen across a whole bunch of different resources,  and we need to be able to consider these actions together in Unity to be able to form reasonable  limiting or other actions. We also have granularity at the thread group ID, not the thread ID  level. This is a contentious point, but it's kind of important. In C-group V1, you could  essentially put different threads from the same process into separate C-groups. This  has a whole bunch of weird implications, like, for example, people would put, say, different  threads from the same process into different memory C-groups. I don't know how the fuck  that's supposed to work. Like, you have, like, basically the entire shared memory between  these two fucking C-groups. I don't know. People have done insane shit with C-groups.  This is the main thing is, like, we want to guide towards a reasonable implementation,  because it's not that these people are stupid. That's not the problem. It's just that C-groups  in V1 were, like, quite over-complicated, so it made people do some insane shit. So now  limiting the PID level kind of gets us, like, a more reasonable approximation of what people  generally want. Also, without extensive cooperation, even for the resources where maybe it would,  in theory, make sense to have different threads from the same process in different resources,  it often ends up being, like, you have to have some way to communicate which thread  from your process is doing what. And there's no standardized way to do that in Linux, right?  You can set the calm of your thread to, like, some value and then look at, like, the value  of somewhere, but it's not standardized, and it's, like, really, really fucking hard to  reason about. So yeah, it usually doesn't act in any reasonable way. In general, like,  there's been a focus on simplicity and clarity in V2 over, like, ultimate flexibility. V1  was invented at the kind of the dawn of containerization. People didn't know what they wanted. They  just know that they wanted something and they wanted it now. So V1 was kind of a solution  to this problem, and V2 was kind of more like a more developed approach to the problems  we know we're having for sure now. So another new feature in V2 is the addition of this  no internal process constraint. So this means that C groups with processes and controllers  enabled cannot create child C groups. Essentially, this means that in simpler words, these red  C groups either have to have no processes or they have to have no controllers enabled  in that part of the hierarchy. This is for a few reasons. It's kind of hard to reason  about how that should act usually. In V1, this was allowed, and the problem is now you  have two different types of objects kind of competing against each other. So say you have  processes in I, and then you have some child C groups underneath I here. Now you have processes  which are in I competing against C groups which are its children, and it's difficult.  You have to make some kind of judgment about how we will treat processes compared to C  groups. Maybe we can consider each one its own C group. Maybe we can consider them like  I prime like a separate C group, but it's quite hard to reason about. For most cases,  the better solution is just create another C group. So this is another guide towards  like helping people create a sane hierarchy. The root C group is a special case. The controllers  themselves have to decide how they're going to handle resources in the root.  So clearly breaking the API is kind of a big deal. The C group API is a pretty fucking  big deal. Like the fact that we want to create V2 instead of just improving V1 obviously  needs some good reasoning there. So V1 works okay for basic situations, but it gets kind  of exponentially complicated when you're getting more and more complex. As I mentioned in V1,  design kind of often followed implementation. And trying to rework kernel APIs after the  fact is really, really, really hard. Like you can't change the kernel API, the fundamental  nature of the kernel APIs which people rely on day to day in production. That's just  not something you can do. Even for stuff which was designed up front, like I mentioned, like  generally the use cases for containers and C groups were not really that well fleshed  out yet. They originally started as like only for CPU and then it grew and grew and grew  kind of naturally. It was generally hard to gauge at the time how C groups would be used.  So now this is an opportunity to redesign them and work them how we actually think they  should be. So to fix these kind of fundamental issues, you need to have an API break and  that's why V2 was created. So I want to go over some of the actual practical improvements  because I've talked a lot about like the theoretical, how we've designed it, but I  also want to go over why we've designed it in that way and what that actually means.  So okay, pop quiz. When you write to a file in Linux, what happens? It's not a direct  question, that's where. You get a file descriptor. Okay, that was possibly before what I wanted.  Okay, so does it write directly to the disk? Okay, so where does your data go? I got about  five different answers. I'm not sure what and how they were. Okay, so it goes, you write  a dirty page into the page cache. You have like some sort of dirty pages now and your  right syscall came back and it said everything's great. And from your application's perspective,  now you can go on pretending, yes, it was written to disk and my right syscall succeeded.  So it must be written to disk and you can have all this class of wonderful belief. But ultimately,  it's not been written to this, right? It's still sitting in memory somewhere and if you  shut down the machine right now, shit's going to go hey, what? So yeah, so there are multiple  operations here to be considered. First, you have your right syscall, which then goes and  writes it at dirty pages, returns to you. And then later, some kernel worker like PD flush  comes and it says, okay, now is the time by some magical standards like the I know dirty  ratio. I've decided I'm going to flush out these to disk. In v1, we don't have any tracking  of this. So if you wrote dirty pages, we don't know where they came from afterwards when  we flush them to disk. So that IO goes to the root cgroup and we can't account it to  your process or your cgroup simply because it wasn't tracked. It wasn't tracked where  this page comes from. In v2, it is tracked and we can actually count these towards your  limit. So we can also make reasonable decisions about like, oh, you have IO contention and  this is what I should do based on that or you have memory contention and this is what  I should do based on that when you're trying to do a page cache write back. V2 is also  generally kind of better integrated with subsystems. So most of the actions we could do based on  thresholds in version one were kind of crude or in the case of the memory subsystem, quite  violent. You set like a limit with memory.limit invites and what happens is your application  has like a one minute spike in memory usage and then the umkiller goes along and goes  oh, I'm going to kill you. And that's like the standard method of dealing with things.  Usually like processes don't particularly like being kill-nined. I don't know, like maybe  like there's some kind of statistic processes which like that. But I mean, ultimately it's  not a very good way of limiting results usage. What you really want to do is like tell it  okay, calm the fuck down and like stop allocating memory. Or in the better case where you can't  tell the calm the fuck down, you want to tell the operating system hey, that guy's going  fucking nuts though. I would like to now like take some action against this guy. But that  action doesn't have to be like slay him where he stands. Yeah, I would like to think in  human society we've come past the point where like the penalty for any kind of failure is  instantaneous death. So yeah, so now in secret v2 we have like generally kind of better thresholds  here. We have a new thing called memory.high which instead of killing a process, we still  have memory.max which is very analogous to memory.limit invites which uncilled your process.  But we also have memory.high and what memory.high does is when you pass this threshold, we start  to do throttling and reclaim for every single memory allocation. So when you go over memory.high  and you want to do another malloc or you want to get grab some more memory, then what we  do is we break into a separate path in the kernel and we say hey, I would like to dial  back this usage. So what I'm going to do is go to the tail of the inactive list and start  reclaiming pages. So if you fail to reclaim any pages, it's still kind of good because  you took a while to scan the page cache. It took you a while to scan the page cache.  We've slowed down your application now and we've done it in a way which is kind of transparent  to your application. But if you do manage to reclaim pages, then we also win because  now you've retained some pages and you managed to get some memory free again. In fact, this  is like a generally a much-sane way of doing things and this is like using this on web  servers was like a big win when you see like these spikes in resource users.  So notifications, I don't know how notifications are like one of the more edge cases for C-groups  since it usually ends up being people like systemd which end up using them, but notifications  are essentially a way to say hey, something in my C-group changed state. It could be like  oh, I have no more processes in my C-groups so all of them have finished. It could be  like oh, one of my processes ran out of memory and I'm going to take some action based on  that. Ultimately, it's a way to get information about what is happening in your C-group. Systemd  uses this for example to track which processes are running and the state of your system and  the state of the services which are running. The problem is on v1 for release notifications  which are the notifications which are sent when your C-group has no more processes which  for example means like oh, we exited. You have to designate what's called a release  agent and this release agent is just like giving a call dump utility. You tell it like  here's the path to my executable and when you have no more processes go and execute  this thing with these arguments. The problem is now if you're using C-groups as a utility  where say you have C-groups expiring like a thousand times a second, you're now like  forking a thousand times a second as well which is pretty bad. It's generally pretty  expensive and it doesn't make a whole lot of sense since the rest of the C-group API  used slightly more sane methods using eventfd. So now we have a notify support everywhere.  CIS events C-group looks like a false system. It kind of makes sense that it supports iNotify.  We still have eventfd support so you can like poll and find out what the answer is. The idea  now is you can have one process to monitor everything you like. You don't have to fork  a new process every time an event is created. This is just like a straight upgrade really.  So utility controllers are another thing. Utility controllers are controllers essentially  most controllers in C-group are related to some particular resource. Say you have memory,  CPU, IO. There are other ones however like say Perf or Freezer which I'll also go into  in a second which are not related to some resource but you put the processes in that  C-group based on some actions you want to take to them as a group. So the idea is like  basically you want to group them together so some user space utility can take some action  based on that group. Perf is a tool for performance monitoring and tracing analytics I guess. Quite  a few people have probably heard of it. And the way it works is when you say I want to  have a certain set of C-groups is you give it a C-group path and it says okay here is  like some particular set of processes which I'm going to map into my own C-group hierarchy.  So now you have CISFS C-group Perf and inside there is a completely separate C-group hierarchy  which only relates to Perf. This usually doesn't make a whole lot of sense. Usually what people  want to do is monitor an existing C-group hierarchy not create a new one. So people  ended up resorting to like tons of hacks like oh I want to like copy over this hierarchy  to the other one. They would write like a tool to copy it from one hierarchy to another  and like you end up with all these race conditions and like horrible and it was like really,  really bad. So now having a unified hierarchy means we don't have to sync. You only have  one hierarchy. So there's no way this could possibly go wrong. So in V1 there's also like  a lot of inconsistency between controllers. This usually comes in kind of two forms. One  you have inconsistent APIs between controllers which do exactly the same thing. So you have  like the CPU, both CPU and I owe are essentially weight based or share based. You give a certain  amount of some resource based on a relative amount to some other particular C-group. But  the APIs were completely different. You had to learn two APIs to do one thing which is  really, really not ideal. So there's been a lot of focus on trying to unify the APIs  and also unify the naming. Like generally like it was a bit of a crap shoot in V1 and  now we have the opportunity to rethink those names and generally standardize them a bit  more. So V2 is generally more intuitive up front. Oh, another one is kind of inconsistent  C-group semantics. For example, root C-groups, for example, sorry, most C-groups inherit  their parent's limits. So when you create a child C-group of some particular C-group,  it usually can only use up to its parent's limits. But some controllers didn't do that.  Some controllers did their own thing. For some controllers, this whole idea of a hierarchy  was like an imaginary thing and it just created a new C-group and didn't care where it was.  It was all a bit of a crap shoot. So now with one unified hierarchy, it's more difficult  to fuck it up, I guess. So V1's over-flexibility also contributed to like a whole bunch of  API problems. For example, when memory limits were first created, they only limited a few,  like a couple types of memory. And they were in this file memory.limitedbytes. Then as  more and more memory types were added, they ended up getting like their own files one  by one. So you ended up with memory.limitedbytes, memory.kmem.limitedbytes, memory.kmem.tcp.limitedbytes,  memory.msw.limitedbytes. And the really bad thing about this is now, yes, I have very  granular control, but it's not very useful. Because now, say I want to set a limit on  the maximum number of TCP buffers, which is set with memory.kmem.tcp.limitedbytes. Now,  say I have like 10 gigabytes of page cache free, and I set a limit on the number of TCP  buffers. If I said like, oh, you should only get say X amount of TCP buffers. If you go  one over, we'll kill you. Even if you had 10 gigabytes of page cache free or something  like that. It's not a reasonable way of operating. Usually, like most people don't want, like  most people don't care, like you allocated one TCP buffer too many. They want to give  some kind of idea about the overall memory use and like reasonable memory use. Generally,  unified limits are only reasonable way to approach that. So, yeah, again, another trade-off  kind of in favor of usability over ultimate flexibility. Generally, if you do want to  limit these things, say you wanted to limit some particular kind of resource, like the  PID controller is a very good example. The PID, in the early days of C groups it was  considered, maybe we could limit the number of PIDs by like limiting certain types of  kind of memory. But that turns out to be really, really fucking hard. Like, so another, the  way that was fixed was we now have a PID controller and that specifically controls this resource.  So if you do want to do very specific kinds of limiting like some TCP buffer or something  else, then you should do that through a new controller. That's the reasonable way to do  that. So, if you go to Facebook.com right now, there  is a one in 10 chance you are going to hit a server with C group B2. We are running a  C group B2 pool in like the tens of thousands of machines now. And we are investing heavily  in C group B2 for like a number of reasons. My main concern, like I mentioned at the beginning,  is limiting the failure domains between services. I really care a lot about making sure that  we don't have cascading failures or anything like that on a machine. And also being able  to kind of manage the resource allocation in your data center, especially at Facebook  sale, is really important. Like, if we can suck at this, that little bit more resource  efficiency at the data center, then that's a really big win.  We run C group B2 managed with system D. One of my teammates, Dav DeCapico, was sitting  back there, did a talk about this at systemd.com for last year called, I believe it was called,  deploying system D at scale, correct me if I'm wrong? Yes, it was. So yes, we're a really  big contributor to the core of C group B2. We have two of the core maintainers working  at Facebook. And we will continue to drive innovation here. Like this is a big thing  we're working on right now. So I already mentioned that C group B2 is kind of new. C group B2  has been usable for a little while now, and the core is all very stable. But that doesn't  mean that there isn't still work to be done here. Like a lot of this is kind of building  the building blocks for future work. So the core APIs are stable, but there's definitely  functionality to be worked on. When thinking about C groups, like most people think of  kind of three things, I guess, which is IO memory, CPU, those are like pretty much the  biggest ones. And two out of three of those are merged currently. The PID controller is  also merged. For the CPU controller, like it's very important, but there have been some  disagreements with the CPU subsystem team about how to merge it. They have some disagreements  about the no internal process constraint and also like having process granularity instead  of thread granularity and stuff like this. Yeah, there's a very juicy drama filled thread  at that link. As all Linux kernel mining list threads are, it's probably still better  than usual. We also may end up with some thread based API for some particular kinds of thread  operations that make sense. That would be like in the works. Another kind of big bet  that we're doing right now is one thing that Linux has never really had is a good metric  for memory pressure. Like we have many kind of related metrics like the amount of memory  have free or the amount of memory used. You can also look at stuff like certain kinds of  page scans or like, yeah, it's, but it's all very heuristic. And ultimately, none of these  metrics prove that you actually are encountering memory pressure because they can also happen  in a bunch of normal scenarios. So our proposed measure is to track page refolting. So the  essential way this will work is we track pages which are consistently refalted back into  the inactive queue. And essentially the way to explain how this whole thing works. So  you have the inactive set. Those are pages which the kernel considers are probably not  being actively used by any process. Then you have the active set which you consider is  more likely to be used by some process. And it's essentially one big list. And the way  it works is when you have a page fault, you go to the head of the inactive list. If you  got, if that page was accessed again, then it gets moved to the head of the active list,  which means it's protected from reclaim. And when we do a reclaim, we go from the tail  of the inactive list. So we take the pages which we consider the least likely to be used  of a reclaim. So what happens if we keep on faulting pages in and they get so far to the  end that we keep on reclaiming them and then they fault back in again immediately? That  probably means that we have too many pages at once for our system to handle. It probably  means that we can't, we end up pushing them so fast off the edge that we simply don't  have the resources to be able to deal with this number of pages. That is probably not  a bad metric for memory pressure. And it's one which is currently being worked on as  part of the security effort as well. Because we do want to have those metrics around memory  pressure, not just memory usage, which is kind of only tangential to the thing which you  really want to know. So, sorry question. I thought I heard a question. All right. So  we also have tracking now of page caps. Right, Max? What this first point means is we have  the tracking of memory in IO, but we don't have tracking for CPU yet. Like you spend  some time on CPU waiting for network packets to come in on the network. We, and we don't  know who it's for yet because it didn't get rooted yet. And we can't account for that yet.  And we also can't account for the CPU we spent like doing a page cache writeback yet. That's  something which is going to take quite a bit of effort. But it's something which we're  definitely working on at this point. Yeah. And as for the second point, I already mentioned  like the, the refalting. Generally, the idea here is that we want better metrics for, for  memory pressure because right now we only have something tangential. Those of you who  use V1 probably also know about freezer. Freezer is like an alternative to umkilling, for example.  Like you can freeze some set of processes in their state and then go and decide like,  oh, I want to raise the memory limit or I want to kill them or I want to stop some new  processes. It was essentially a way of freezing them in time and having some other process  come and decide what to do about it. In, in Sigur V1 this didn't work at all basically.  Like if you, if you used freezer, one of the most common things you might want to do is  go and like get a stack trace and look at what they were doing when like they kept on  allocating memory or whatever it was that you froze them for. But like it's a very common  situation that you would end up with like say GDP. If you would try to attach GDP, it  would end up in D state, which is not really the ideal result if you want to like find  out the stack of some process which froze like that's generally the complete opposite  of what, what I would like. The reason is like the freezer implementation in V1 doesn't  guarantee that we stop anywhere reasonable. We, we often stop in a stack which makes absolutely  no sense in the kernel. So in V2, the idea is to have like a more kind of sigstop style  mechanism. Sigstop is very well defined and where it stops is very well defined as well.  So it's, it's kind of a more reasonable solution to use for stopping processes. So the, the,  the V2 implementation of freezer will be more along those kind of semantics. So I talked  a lot here about trying to sell you a secret V2, but I should actually tell you how to  get it probably at some point during this talk. So hopefully you're interested in trying  it out yourself. So here's what you need to get started with version two. First you need  a kernel above 4.5. Before that we do have a developer flag, which you can go find. I  won't tell you because it'll like eat you if you try and use it and you don't know what  you're doing. But I really wouldn't recommend using it before 4.5. 4.5 is the first point.  We have a stable API. Once that's done, more or less two things to do. You need to turn  off all of the controllers for V1 and you need to turn on and mount the, the, the file  system for V2. Typically you want your init system to do this. For system D you use it  with this flag. You basically put both of these on the kernel command line. But if you're  crazy or you want to try it yourself, you can also manually mount it and cry when things  break. So yeah, so if you're interested in hearing more about C groups, come talk to  me. I'm happy to go over any of what I've been talking about on V1 or V2. And yeah,  I'll be happy to go over any questions you might have. And if you've used V1 in the  past, which I guess many of you have, and you found it lacking in some areas, please  do try out V2 and let us know what you think. Thanks.  Thank you.
cgroupv2_Linux_s_new_unified_control_group_hierarchy_0_131.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_132_755.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_756_2687.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_2688_3407.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_3408_4199.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_4200_5579.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_5580_5807.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_5808_6443.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_6444_8471.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_8472_9275.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_9276_9539.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_9540_9587.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_9588_9647.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_9648_10667.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_10668_13403.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_13404_14759.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_14760_15383.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_15384_16043.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_16044_16343.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_16344_18059.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_18060_18563.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_18564_19643.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_19644_20423.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_20424_20963.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_20964_21719.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_21720_22451.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_22452_22727.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_22728_23147.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_23148_23195.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_23196_23423.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_23424_27047.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_27048_29075.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_29076_30755.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_30756_30983.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_30984_33839.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_33840_34307.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_34308_37475.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_37476_38891.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_38892_40079.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_40080_42335.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_42336_43511.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_43512_44351.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_44352_47219.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_47220_48911.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_48912_50531.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_50532_53507.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_53508_56711.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_56712_57983.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_57984_58595.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_58596_58763.png cgroupv2_Linux_s_new_unified_control_group_hierarchy_58764_58881.png
Okay, so hi. My name is Chris. I already mentioned I work as a production engineer at Facebook London working as a member of Web Foundation. I'm going to be giving a whistle-tock tour of the new version of Control Groups added in Linux 4.5. Don't worry if you haven't the faintest clue what Control Groups are yet. I guess since we're in the containers, there are probably some of you have at least some kind of idea. But I'll go into kind of what they are, where you may have used them, and a comparison of the old and the new in the next few slides. So like I said in this talk, I'm going to be giving kind of an introduction to Control Groups. I'm going to go over what they are, where you may have used them, where you might have encountered them before. If you already know something called Control Groups like you've fished around previously or you've used them before, you almost suddenly have been interacting with version one of Control Groups. Control Groups has existed in the kernel since a long, long time ago, since around 2008. And it's been kind of building our love of containers. It's one of the building blocks of containers as we know them. So it's definitely got a whole bunch of good things in it. It's also got like a whole bunch of problems, usability foibles, other kinds of things which are not so great. So I want to go into like what those are and when you might encounter them and how we try to improve that in Sigur V2. Sigur V1 is currently now kind of mostly in maintenance mode, whereas Sigur V2 is more in active development. They share the same core. It's mostly the user-facing API which is kind of different. And yeah, I'm also going to go over kind of what's still to be done and what's already been done. The core of Sigur V2 is already stable since kernel 4.5, but we have a whole bunch of work we want to be done. A lot of the core work has been to enable future work, to enable new features in Sigur V2 which we already work on. So first a little bit about me. I've been working at Facebook for about three years. Like I said, I work in this team called Web Foundation. Technically, Web Foundation is the team responsible for the web servers at Facebook, but web servers are kind of not usually extremely complicated. Like they're stateless, they serve you capuchess and that's about it. So we actually ended up becoming kind of a team which deals with the general reliability at Facebook. This means we get involved with production, incident discussions and all sorts of... We generally act as kind of the guardians of reliability at Facebook. So like I said, I work at Facebook London. We have an office there. We also have an office in Dublin, in Europe, and in Tel Aviv. Yeah, and we have a whole bunch of different kinds of people in Web Foundation. We have people who are experts at Linux and Pots of Linux stack which is basically my expertise. We also have people who are experts in our cache, architecture, RPC, and say like the web push and stuff like this, especially a cross-functional group of experts which all come together to deal with when shit really hits the fan. So that kind of brings up the question, why do we as a team care about C-groups? And why does Facebook as a company care about C-groups? So we have many, many hundreds of thousands of servers and we have a bunch of services which run on these servers. Some of them co-located on the same servers. Some of them spread throughout servers. It's all kind of all over the place. So most kind of outages at Facebook are a few, there are a few kinds of outages, but like one very common one is failure in multiple systems. And there are a few things you need to do to mitigate that. Another one of course is actively mapping out your dependencies and making sure you have an understanding of your dependencies. But another big one is making sure that when two services run on a single machine, you don't end up with a situation where one service completely overrides the other and results in it becoming completely useless. So that's one of my main concerns about C-groups 2 and one of the reasons why I'm super interested in it. So a typical use case for this is like you say on a normal web server, we have three types of running processes. Same goes for most other kinds of web servers, other kinds of servers as well. So you have your core workload. Your core workload is the thing which if you were to describe to somebody else what your server was actually doing, you would say it does this thing. It serves web requests. It does load balancing. But usually you don't end up only with this on your server. You usually end up with a whole bunch of other stuff, especially if your company has been around for a while or you have some ideas about how you should architect stuff. You end up with a whole bunch of non-core services which can probably be used interchangeably with kind of system services. This might be stuff which just comes with Linux like also the kernel workers or it could also be like say, Kobra's Demons or it could be like stuff which is related to your business needs say metric collections so we can work out what is going on with our server like whether we're managing to serve users correctly. But it's really, really bad if then this metric collection decides it's going to take up all the available memory in your server and then you can't actually serve users. Sure, it can tell me it did that but I don't give a fuck if it like completely fucked up the whole web server, right? Same goes for cron jobs and chef. I care about my server being up to date but I would rather have some system which actually acts in a reasonable manner if chef starts taking a bunch of memory. I would rather it got degraded service and still manage to run than it took all of the memory or took all the CPU and then everything was fucked. Like I would much rather have that outcome. Then we can have this third class like ad-hoc queries and debugging. These are typically things you don't know that you need and don't end up getting run on the majority of servers. These are like things which you end up realizing you need only when an incident is already happening. We kind of want people to be able to dynamically be able to determine the importance of those things as the incident is going on, whether they want it to take precedence over the core workload of the machine or not. C-groups is a very good use case. Sorry, this is a very good use case for C-groups. Like I mentioned, if you had some interaction with C-groups, you've almost certainly been interacting with version one. Version two has been in development for like five years now. It only just got stable in the Linux kernel. But even on recent kernels, version one is typically what's mounted by default. The reason for that is as you'd imagine with a different version number, the actual changes are backwards incompatible. I'm going to be going over kind of why we've made them backwards incompatible in a moment. The fact that we typically boot with your init system only mounting the version one hierarchy is a testament to why I'm doing this talk. I know we have a whole group full of container experts in this room. This is also kind of a sell to you about why you should give a fuck about C-group V2 and why you should take the time to care about it and invest in it in your products. So in the previous slide, we talked about multiple processes fitting into each of these C-groups as well. So a C-group can be set as tightly or as flexibly as you like. It can be all of the processes which are related to one service or it can be a single process. It can be however many processes you like from zero to however many you like. The idea here is we don't impose a structure on you. One of the guiding principles behind V2 has been we want you to be able to choose how to use it. We don't want to have a hierarchy which imposes how you should do things. We want the easy way to be the correct way. So a C-group is a control group. They're one and the same. They're a system for resource management on links. Like I mentioned, a resource here means CPU, I owe memory and management here can mean for example accounting. We know how much memory some processes in some particular C-group are using. It can also be limiting. We hit a particular threshold and we take some violent action to curb that. It can also, in V2 there's also been some work towards throttling which I'll go in as well. Generally, um-killing or whatever is quite violent. So we want to have some remediative actions instead of just straight up killing stuff. So the way that C-group V2 and V1 work are essentially you have this hierarchy at CISFS C-group. It's just a bunch of files and directories. We don't have a system call interface. There may well be one in future but we don't currently have one. The reason this is kind of a good thing is it's really easy from some user space application even if it's not written in C or C++ and you don't have access to C library functions or system calls easily. You can easily go and find out the state of your system when it comes to C-groups. You can just look at some file like I would hope that all of you are using languages which can open a file, make a directory, remove a directory. I would hope whatever the new hip kids are using still supports those kind of things. So each resource interface is provided by a controller. You'll probably hear me use the word controller, resource and domain kind of interchangeably. The idea is you write files that apply to some particular controller. This controller takes the values which you provide and it provides them to the kernel and the kernel makes some kind of decisions based on what values you gave it. So it's essentially the backing store for all of the stuff which you input to C-groups. So as mentioned previously, workload isolation is a really big use case for C-groups. So you might have many background jobs and a machine but you don't want them to override the main workload. That's the case which we talked about first. There are also some other kind of other cases like say you have a tier which runs asynchronous jobs in the background. We have that at Facebook. And some jobs have higher priority than others. Priority is a very abstract concept but priority usually has something to do with resources. It usually means either you expect it has more memory or less CPU or something like this. It's very much up to you what exactly that priority means. You also have shared environments say you're a VPS provider or something like that and you don't want some particular user to be able to use that container and steal all the resources from all your other customers which are now going to go and leave you a bad review. So yeah, there are kind of a lot of use cases for C-groups here. And you might ask, hey, my favorite application X, it already does this. Why should I give a single shit about C-groups? Well, the answer is if you have been using any kind of software in the last eight years, I would really pray that it does it through C-groups. It probably does it transparently to you and you don't ever have to talk to C-groups directly. But the backing for all of these is C-groups. That's how they do resource limiting. So let's go concretely over how this works in version one. So in V1, CISFS C-group contains the names of all the resources which have a controller. So this might be CPU, memory, PIDs, that kind of stuff. And inside these resource directories, there's another set of directories which are the C-groups themselves. So these C-groups exist in the context of this resource and you put processes into those C-groups. For example, we have the PIDs one at the bottom. So because this is to do with PID resources, it contains files in those directories and those files are related to how many PIDs you can have in a C-group, for example. And we also have the concept of having its own hierarchy for resource distribution. So in C-group V1, you have a resource and then you have a C-group hierarchy underneath each resource. So even if C-group 3 here was called the same as C-group 1, say they were both called FOO, from the kernel's perspective, they have absolutely no relation to each other. This is kind of important because if you look at how, for example, system D sets out C-groups typically, you often end up with quite similar looking hierarchies in different resources and you might even be inclined to believe they have some relation to each other. Well, from system D's perspective, I'm sure they do, but from the kernel's perspective, they do not. And that results in a whole slew of like subtle issues which cause problems at scale which I'll go into in a moment. C-groups are nested inside each other in this example. So when a C-group is nested inside another, typically what it means is it can control some limited amount up to the maximal amount of its parent. So if you have a memory C-group and then you have a child of a C-group which is also another C-group, then you can limit up to the maximum of its parent. So the resource that controls these C-groups or the C-group hierarchy that they're in determines what kind of files there are. I already mentioned that if you're in the memory hierarchy, you can only access files related to memory. Like for example, that's this file memory.limitinbytes. You can read the limitinbytes from it or you can write to it and set a separate limitinbytes. And one PID is an exactly one C-group per resource in C-group V1. So PID 2 here is explicitly assigned to resources A and C in C-group 1 and 5 respectively, but because we don't assign it to anything in resource B, it actually goes to this root C-group. The root C-group is kind of a special concept in C-groups. It's essentially unmanaged territory. How exactly it's managed is up to the controller. But the idea is you don't really have the opportunity to set really any limits in the root C-group because it's just the starting point for resource distribution of this resource across your whole machine. So yeah, you do get some kind of accounting, but in terms of limiting, it's basically useless. So here is a concrete look at how this looks in C-group V1. So you have C-to-Fest C-group and then you have the resources, which are like block I, M, M, and PIDs. And then you have the C-group names and we have nested C-groups here, A inside BG for two resources and B inside ad hoc for two resources. So once again, just as a reiteration because this is really important that you get this, from the Kahnels perspective, naming has no meaning. Like if it's in a different resource, even if it has the same name, it has no meaning. And that has also some weird implications. So here's how it looks in C-group V2 by comparison. So in C-group V2, we actually don't see the resources anymore. If you remember how it looked in version one, we have resources under C-to-Fest C-group, but now we actually have the C-groups themselves under C-to-Fest C-group. So how do these C-groups understand which resources they are supposed to apply to if they're not inside a resource hierarchy? Well, the answer is kind of, C-groups are global now. They're essentially a global set of C-groups and you enable resources inside the C-groups. This means that we have one hierarchy to rule them all. And the idea is like, you write to a special file, you tell us what particular controllers you want to enable, and we enable them for your C-group. We don't require you to create disparate hierarchies each time. You instead create one hierarchy and enable C-group controllers at will. So this is how the previous example now looks in C-group V2. So in C-group V2, like I mentioned, we now have the C-groups directly at the bottom, but you write to this special file, C-group.subtreeControl, and that enables the, in the children of that C-group, that those controllers are enabled. Essentially if you were not to enable them there, but they were enabled the next level up, it means they would compete freely for those resources that you didn't enable. So here's the version one hierarchy again for comparison. And as you can see here, we have resources first. And remember that in version one, C-groups with the same name again don't have any relation to each other. In C-group V2, we have this unified hierarchy, and you enable resources for C-groups children by writing plus memory, plus PIDs, plus CPU, plus IO, that kind of stuff to C-group.subtreeControl. And when you do this, the files appear in that directory instantaneously. Oh, and another thing to mention is, in real life, you also need to enable the memory PIDs and IO controllers at the top level for this to work, but for the sake of simplicity, I'm not out of them here. So the fundamental differences are obviously the unified hierarchy. Resources apply to C-groups now instead of C-groups applying to some, some distance across the hierarchy for a resource. This is very important for some kinds of common operations in Linux. Like, for example, you have PageCache writebacks, and PageCache writebacks transcend one particular resource. They happen across a whole bunch of different resources, and we need to be able to consider these actions together in Unity to be able to form reasonable limiting or other actions. We also have granularity at the thread group ID, not the thread ID level. This is a contentious point, but it's kind of important. In C-group V1, you could essentially put different threads from the same process into separate C-groups. This has a whole bunch of weird implications, like, for example, people would put, say, different threads from the same process into different memory C-groups. I don't know how the fuck that's supposed to work. Like, you have, like, basically the entire shared memory between these two fucking C-groups. I don't know. People have done insane shit with C-groups. This is the main thing is, like, we want to guide towards a reasonable implementation, because it's not that these people are stupid. That's not the problem. It's just that C-groups in V1 were, like, quite over-complicated, so it made people do some insane shit. So now limiting the PID level kind of gets us, like, a more reasonable approximation of what people generally want. Also, without extensive cooperation, even for the resources where maybe it would, in theory, make sense to have different threads from the same process in different resources, it often ends up being, like, you have to have some way to communicate which thread from your process is doing what. And there's no standardized way to do that in Linux, right? You can set the calm of your thread to, like, some value and then look at, like, the value of somewhere, but it's not standardized, and it's, like, really, really fucking hard to reason about. So yeah, it usually doesn't act in any reasonable way. In general, like, there's been a focus on simplicity and clarity in V2 over, like, ultimate flexibility. V1 was invented at the kind of the dawn of containerization. People didn't know what they wanted. They just know that they wanted something and they wanted it now. So V1 was kind of a solution to this problem, and V2 was kind of more like a more developed approach to the problems we know we're having for sure now. So another new feature in V2 is the addition of this no internal process constraint. So this means that C groups with processes and controllers enabled cannot create child C groups. Essentially, this means that in simpler words, these red C groups either have to have no processes or they have to have no controllers enabled in that part of the hierarchy. This is for a few reasons. It's kind of hard to reason about how that should act usually. In V1, this was allowed, and the problem is now you have two different types of objects kind of competing against each other. So say you have processes in I, and then you have some child C groups underneath I here. Now you have processes which are in I competing against C groups which are its children, and it's difficult. You have to make some kind of judgment about how we will treat processes compared to C groups. Maybe we can consider each one its own C group. Maybe we can consider them like I prime like a separate C group, but it's quite hard to reason about. For most cases, the better solution is just create another C group. So this is another guide towards like helping people create a sane hierarchy. The root C group is a special case. The controllers themselves have to decide how they're going to handle resources in the root. So clearly breaking the API is kind of a big deal. The C group API is a pretty fucking big deal. Like the fact that we want to create V2 instead of just improving V1 obviously needs some good reasoning there. So V1 works okay for basic situations, but it gets kind of exponentially complicated when you're getting more and more complex. As I mentioned in V1, design kind of often followed implementation. And trying to rework kernel APIs after the fact is really, really, really hard. Like you can't change the kernel API, the fundamental nature of the kernel APIs which people rely on day to day in production. That's just not something you can do. Even for stuff which was designed up front, like I mentioned, like generally the use cases for containers and C groups were not really that well fleshed out yet. They originally started as like only for CPU and then it grew and grew and grew kind of naturally. It was generally hard to gauge at the time how C groups would be used. So now this is an opportunity to redesign them and work them how we actually think they should be. So to fix these kind of fundamental issues, you need to have an API break and that's why V2 was created. So I want to go over some of the actual practical improvements because I've talked a lot about like the theoretical, how we've designed it, but I also want to go over why we've designed it in that way and what that actually means. So okay, pop quiz. When you write to a file in Linux, what happens? It's not a direct question, that's where. You get a file descriptor. Okay, that was possibly before what I wanted. Okay, so does it write directly to the disk? Okay, so where does your data go? I got about five different answers. I'm not sure what and how they were. Okay, so it goes, you write a dirty page into the page cache. You have like some sort of dirty pages now and your right syscall came back and it said everything's great. And from your application's perspective, now you can go on pretending, yes, it was written to disk and my right syscall succeeded. So it must be written to disk and you can have all this class of wonderful belief. But ultimately, it's not been written to this, right? It's still sitting in memory somewhere and if you shut down the machine right now, shit's going to go hey, what? So yeah, so there are multiple operations here to be considered. First, you have your right syscall, which then goes and writes it at dirty pages, returns to you. And then later, some kernel worker like PD flush comes and it says, okay, now is the time by some magical standards like the I know dirty ratio. I've decided I'm going to flush out these to disk. In v1, we don't have any tracking of this. So if you wrote dirty pages, we don't know where they came from afterwards when we flush them to disk. So that IO goes to the root cgroup and we can't account it to your process or your cgroup simply because it wasn't tracked. It wasn't tracked where this page comes from. In v2, it is tracked and we can actually count these towards your limit. So we can also make reasonable decisions about like, oh, you have IO contention and this is what I should do based on that or you have memory contention and this is what I should do based on that when you're trying to do a page cache write back. V2 is also generally kind of better integrated with subsystems. So most of the actions we could do based on thresholds in version one were kind of crude or in the case of the memory subsystem, quite violent. You set like a limit with memory.limit invites and what happens is your application has like a one minute spike in memory usage and then the umkiller goes along and goes oh, I'm going to kill you. And that's like the standard method of dealing with things. Usually like processes don't particularly like being kill-nined. I don't know, like maybe like there's some kind of statistic processes which like that. But I mean, ultimately it's not a very good way of limiting results usage. What you really want to do is like tell it okay, calm the fuck down and like stop allocating memory. Or in the better case where you can't tell the calm the fuck down, you want to tell the operating system hey, that guy's going fucking nuts though. I would like to now like take some action against this guy. But that action doesn't have to be like slay him where he stands. Yeah, I would like to think in human society we've come past the point where like the penalty for any kind of failure is instantaneous death. So yeah, so now in secret v2 we have like generally kind of better thresholds here. We have a new thing called memory.high which instead of killing a process, we still have memory.max which is very analogous to memory.limit invites which uncilled your process. But we also have memory.high and what memory.high does is when you pass this threshold, we start to do throttling and reclaim for every single memory allocation. So when you go over memory.high and you want to do another malloc or you want to get grab some more memory, then what we do is we break into a separate path in the kernel and we say hey, I would like to dial back this usage. So what I'm going to do is go to the tail of the inactive list and start reclaiming pages. So if you fail to reclaim any pages, it's still kind of good because you took a while to scan the page cache. It took you a while to scan the page cache. We've slowed down your application now and we've done it in a way which is kind of transparent to your application. But if you do manage to reclaim pages, then we also win because now you've retained some pages and you managed to get some memory free again. In fact, this is like a generally a much-sane way of doing things and this is like using this on web servers was like a big win when you see like these spikes in resource users. So notifications, I don't know how notifications are like one of the more edge cases for C-groups since it usually ends up being people like systemd which end up using them, but notifications are essentially a way to say hey, something in my C-group changed state. It could be like oh, I have no more processes in my C-groups so all of them have finished. It could be like oh, one of my processes ran out of memory and I'm going to take some action based on that. Ultimately, it's a way to get information about what is happening in your C-group. Systemd uses this for example to track which processes are running and the state of your system and the state of the services which are running. The problem is on v1 for release notifications which are the notifications which are sent when your C-group has no more processes which for example means like oh, we exited. You have to designate what's called a release agent and this release agent is just like giving a call dump utility. You tell it like here's the path to my executable and when you have no more processes go and execute this thing with these arguments. The problem is now if you're using C-groups as a utility where say you have C-groups expiring like a thousand times a second, you're now like forking a thousand times a second as well which is pretty bad. It's generally pretty expensive and it doesn't make a whole lot of sense since the rest of the C-group API used slightly more sane methods using eventfd. So now we have a notify support everywhere. CIS events C-group looks like a false system. It kind of makes sense that it supports iNotify. We still have eventfd support so you can like poll and find out what the answer is. The idea now is you can have one process to monitor everything you like. You don't have to fork a new process every time an event is created. This is just like a straight upgrade really. So utility controllers are another thing. Utility controllers are controllers essentially most controllers in C-group are related to some particular resource. Say you have memory, CPU, IO. There are other ones however like say Perf or Freezer which I'll also go into in a second which are not related to some resource but you put the processes in that C-group based on some actions you want to take to them as a group. So the idea is like basically you want to group them together so some user space utility can take some action based on that group. Perf is a tool for performance monitoring and tracing analytics I guess. Quite a few people have probably heard of it. And the way it works is when you say I want to have a certain set of C-groups is you give it a C-group path and it says okay here is like some particular set of processes which I'm going to map into my own C-group hierarchy. So now you have CISFS C-group Perf and inside there is a completely separate C-group hierarchy which only relates to Perf. This usually doesn't make a whole lot of sense. Usually what people want to do is monitor an existing C-group hierarchy not create a new one. So people ended up resorting to like tons of hacks like oh I want to like copy over this hierarchy to the other one. They would write like a tool to copy it from one hierarchy to another and like you end up with all these race conditions and like horrible and it was like really, really bad. So now having a unified hierarchy means we don't have to sync. You only have one hierarchy. So there's no way this could possibly go wrong. So in V1 there's also like a lot of inconsistency between controllers. This usually comes in kind of two forms. One you have inconsistent APIs between controllers which do exactly the same thing. So you have like the CPU, both CPU and I owe are essentially weight based or share based. You give a certain amount of some resource based on a relative amount to some other particular C-group. But the APIs were completely different. You had to learn two APIs to do one thing which is really, really not ideal. So there's been a lot of focus on trying to unify the APIs and also unify the naming. Like generally like it was a bit of a crap shoot in V1 and now we have the opportunity to rethink those names and generally standardize them a bit more. So V2 is generally more intuitive up front. Oh, another one is kind of inconsistent C-group semantics. For example, root C-groups, for example, sorry, most C-groups inherit their parent's limits. So when you create a child C-group of some particular C-group, it usually can only use up to its parent's limits. But some controllers didn't do that. Some controllers did their own thing. For some controllers, this whole idea of a hierarchy was like an imaginary thing and it just created a new C-group and didn't care where it was. It was all a bit of a crap shoot. So now with one unified hierarchy, it's more difficult to fuck it up, I guess. So V1's over-flexibility also contributed to like a whole bunch of API problems. For example, when memory limits were first created, they only limited a few, like a couple types of memory. And they were in this file memory.limitedbytes. Then as more and more memory types were added, they ended up getting like their own files one by one. So you ended up with memory.limitedbytes, memory.kmem.limitedbytes, memory.kmem.tcp.limitedbytes, memory.msw.limitedbytes. And the really bad thing about this is now, yes, I have very granular control, but it's not very useful. Because now, say I want to set a limit on the maximum number of TCP buffers, which is set with memory.kmem.tcp.limitedbytes. Now, say I have like 10 gigabytes of page cache free, and I set a limit on the number of TCP buffers. If I said like, oh, you should only get say X amount of TCP buffers. If you go one over, we'll kill you. Even if you had 10 gigabytes of page cache free or something like that. It's not a reasonable way of operating. Usually, like most people don't want, like most people don't care, like you allocated one TCP buffer too many. They want to give some kind of idea about the overall memory use and like reasonable memory use. Generally, unified limits are only reasonable way to approach that. So, yeah, again, another trade-off kind of in favor of usability over ultimate flexibility. Generally, if you do want to limit these things, say you wanted to limit some particular kind of resource, like the PID controller is a very good example. The PID, in the early days of C groups it was considered, maybe we could limit the number of PIDs by like limiting certain types of kind of memory. But that turns out to be really, really fucking hard. Like, so another, the way that was fixed was we now have a PID controller and that specifically controls this resource. So if you do want to do very specific kinds of limiting like some TCP buffer or something else, then you should do that through a new controller. That's the reasonable way to do that. So, if you go to Facebook.com right now, there is a one in 10 chance you are going to hit a server with C group B2. We are running a C group B2 pool in like the tens of thousands of machines now. And we are investing heavily in C group B2 for like a number of reasons. My main concern, like I mentioned at the beginning, is limiting the failure domains between services. I really care a lot about making sure that we don't have cascading failures or anything like that on a machine. And also being able to kind of manage the resource allocation in your data center, especially at Facebook sale, is really important. Like, if we can suck at this, that little bit more resource efficiency at the data center, then that's a really big win. We run C group B2 managed with system D. One of my teammates, Dav DeCapico, was sitting back there, did a talk about this at systemd.com for last year called, I believe it was called, deploying system D at scale, correct me if I'm wrong? Yes, it was. So yes, we're a really big contributor to the core of C group B2. We have two of the core maintainers working at Facebook. And we will continue to drive innovation here. Like this is a big thing we're working on right now. So I already mentioned that C group B2 is kind of new. C group B2 has been usable for a little while now, and the core is all very stable. But that doesn't mean that there isn't still work to be done here. Like a lot of this is kind of building the building blocks for future work. So the core APIs are stable, but there's definitely functionality to be worked on. When thinking about C groups, like most people think of kind of three things, I guess, which is IO memory, CPU, those are like pretty much the biggest ones. And two out of three of those are merged currently. The PID controller is also merged. For the CPU controller, like it's very important, but there have been some disagreements with the CPU subsystem team about how to merge it. They have some disagreements about the no internal process constraint and also like having process granularity instead of thread granularity and stuff like this. Yeah, there's a very juicy drama filled thread at that link. As all Linux kernel mining list threads are, it's probably still better than usual. We also may end up with some thread based API for some particular kinds of thread operations that make sense. That would be like in the works. Another kind of big bet that we're doing right now is one thing that Linux has never really had is a good metric for memory pressure. Like we have many kind of related metrics like the amount of memory have free or the amount of memory used. You can also look at stuff like certain kinds of page scans or like, yeah, it's, but it's all very heuristic. And ultimately, none of these metrics prove that you actually are encountering memory pressure because they can also happen in a bunch of normal scenarios. So our proposed measure is to track page refolting. So the essential way this will work is we track pages which are consistently refalted back into the inactive queue. And essentially the way to explain how this whole thing works. So you have the inactive set. Those are pages which the kernel considers are probably not being actively used by any process. Then you have the active set which you consider is more likely to be used by some process. And it's essentially one big list. And the way it works is when you have a page fault, you go to the head of the inactive list. If you got, if that page was accessed again, then it gets moved to the head of the active list, which means it's protected from reclaim. And when we do a reclaim, we go from the tail of the inactive list. So we take the pages which we consider the least likely to be used of a reclaim. So what happens if we keep on faulting pages in and they get so far to the end that we keep on reclaiming them and then they fault back in again immediately? That probably means that we have too many pages at once for our system to handle. It probably means that we can't, we end up pushing them so fast off the edge that we simply don't have the resources to be able to deal with this number of pages. That is probably not a bad metric for memory pressure. And it's one which is currently being worked on as part of the security effort as well. Because we do want to have those metrics around memory pressure, not just memory usage, which is kind of only tangential to the thing which you really want to know. So, sorry question. I thought I heard a question. All right. So we also have tracking now of page caps. Right, Max? What this first point means is we have the tracking of memory in IO, but we don't have tracking for CPU yet. Like you spend some time on CPU waiting for network packets to come in on the network. We, and we don't know who it's for yet because it didn't get rooted yet. And we can't account for that yet. And we also can't account for the CPU we spent like doing a page cache writeback yet. That's something which is going to take quite a bit of effort. But it's something which we're definitely working on at this point. Yeah. And as for the second point, I already mentioned like the, the refalting. Generally, the idea here is that we want better metrics for, for memory pressure because right now we only have something tangential. Those of you who use V1 probably also know about freezer. Freezer is like an alternative to umkilling, for example. Like you can freeze some set of processes in their state and then go and decide like, oh, I want to raise the memory limit or I want to kill them or I want to stop some new processes. It was essentially a way of freezing them in time and having some other process come and decide what to do about it. In, in Sigur V1 this didn't work at all basically. Like if you, if you used freezer, one of the most common things you might want to do is go and like get a stack trace and look at what they were doing when like they kept on allocating memory or whatever it was that you froze them for. But like it's a very common situation that you would end up with like say GDP. If you would try to attach GDP, it would end up in D state, which is not really the ideal result if you want to like find out the stack of some process which froze like that's generally the complete opposite of what, what I would like. The reason is like the freezer implementation in V1 doesn't guarantee that we stop anywhere reasonable. We, we often stop in a stack which makes absolutely no sense in the kernel. So in V2, the idea is to have like a more kind of sigstop style mechanism. Sigstop is very well defined and where it stops is very well defined as well. So it's, it's kind of a more reasonable solution to use for stopping processes. So the, the, the V2 implementation of freezer will be more along those kind of semantics. So I talked a lot here about trying to sell you a secret V2, but I should actually tell you how to get it probably at some point during this talk. So hopefully you're interested in trying it out yourself. So here's what you need to get started with version two. First you need a kernel above 4.5. Before that we do have a developer flag, which you can go find. I won't tell you because it'll like eat you if you try and use it and you don't know what you're doing. But I really wouldn't recommend using it before 4.5. 4.5 is the first point. We have a stable API. Once that's done, more or less two things to do. You need to turn off all of the controllers for V1 and you need to turn on and mount the, the, the file system for V2. Typically you want your init system to do this. For system D you use it with this flag. You basically put both of these on the kernel command line. But if you're crazy or you want to try it yourself, you can also manually mount it and cry when things break. So yeah, so if you're interested in hearing more about C groups, come talk to me. I'm happy to go over any of what I've been talking about on V1 or V2. And yeah, I'll be happy to go over any questions you might have. And if you've used V1 in the past, which I guess many of you have, and you found it lacking in some areas, please do try out V2 and let us know what you think. Thanks. Thank you.
going to go over what they are, where you may love of containers. It's one of the building way. So a C-group is a control group. They're one and the same. They're a system for resource jobs in the background. We have that at Facebook. read the limitinbytes from it or you can write to apply to if they're not inside a resource process into separate C-groups. This has a whole about how we will treat processes compared to C this is like using this on web servers was like a Ultimately, it's a way to get information about what is happening in your C-group. Systemd uses and it just created a new C-group and didn't care in the past, which I guess many of you have, and