Systemd @ Facebook - a year later
Cavalca, Davide
We'll be talking about what we learned throughout the past year running systemd in production at Facebook: new challenges that have come up, how the integration process went and the areas of improvement we discovered. We'll also discuss our efforts building a monitoring solution for system services based on systemd.
 All right. Let's get started. Hello, everybody. My name is Davide. I'm a production engineer at Facebook. I'll be talking about what we've been doing with system D for the past year. Before we begin, here's the agenda for today. I'll start with a quick recap of the stories so far. We'll talk about how we are keeping system D updated in our fleet and how we are tracking upstream changes. We'll focus on a couple of things we've been working on lately around resource management and service monitoring. Then we'll discuss a few case studies that hopefully showcase a bunch of interesting problems we've seen and close with a couple of words about advocacy. I was at system D account about a year ago. At the time, we were moving the fleet from center six to center seven. My team works on the operating system team. My team is responsible for the bare metal experience of the fleet. We maintain what keeps the physical machines at Facebook running. We maintain the operating system with the centers. We take care of packaging. We take care of configuration management using Chef among other things. Our fleet is made of hundreds of thousands of physical machines spread around various data centers in the world. All these machines run centers. That's what runs the website and everything else. About a year ago, we were moving from six to seven. On six, we had five or six different crazy ways of supervising services. On seven, we have system D. I'm happy to say that now we're everywhere running center seven and everywhere running system D. Which makes me personally very happy because we managed to get rid of all of these crazy ways of doing service supervision. As part of this, we got to migrate a lot of services to system D. We got to see a lot of people that started building their services with system D mine and started leveraging more and more features. One thing we also did to help this, we started integrating lip system D in our internal build system so that people are now able to use features like the socket activation and all these features directly in our demons that make people's life a bit easier. This talk will focus primarily on the bare metal. Zil and Zolton are going to give a talk later today about containers. If you're interested about system D in containers, I recommend you attend the later talk. Let's talk a bit about how we are managing system D on the fleet. In general, at Facebook, we manage machines using Chef and we have a system to do package updates on machines in a controlled fashion. We can say this package has been to this version and then 1% of the fleet gets this other version and 2% and 5% and so on and so forth. That's the system we use for managing updates of system packages. About a year ago, we were on system D 231 and we went from 31, 32, 32, 33, 33, 34. Right now, we have about half the fleet on 233 and the other half on 234 and I'll explain why in a few slides. We also started testing 235. In general, we try to run, we run CentOS 7 because we want a stable base and we want to be able to pull security updates in an automated fashion but we also want to have a modern user space. We backport a lot of core system components from Fedora or Hyde. These components are system D, of course, but there's also a lot of the ancillary ecosystem. We backport things like D-Bus and Util Linux and Proc.ps and a lot of these basic system tools. The experience you get using the system is somewhat similar to the experience you would have using a Fedora system, at least from the point of view of a developer. We publish these backports on GitHub and you can get them on that GitHub org. These are just the spec files and whatever patches we have. System D there is on 235. This is something that came out from last year's conference because people were interested in this so we made an effort to get them published. Then, Yana Reddick was kind enough to make a copper from these packages. If you happen to run CentOS 7 and happen to want a modern system D, you can get binary packages directly from there and those are also mostly up to date. If you need the earlier versions, you can go back in the GitHub history. Now, of course, having a lot of machines, the updates aren't always smooth and things can happen. I'm going to go over a few interesting things that happen during package updates that might make things a little bit more interesting. The first thing is not actually a system D-specific thing. It's something that happens in general when you are dealing with a large fleet of write-up systems and have two or eight packages on all of them. It's issues generally around RPM. Machines can get in bad states for various reasons. You can have issues around power loss. You can have people running kill-dash-9 on things or processes running kill-dash-9 on things that can leave the system in weirder than interesting states. This can result in found situations. Some things we found that we always get this every time we update a major package on the fleet, we always get a sizable number of these things. We get issues like the obligated packages. You end up with a machine that does both system D-233 and system D-234 installed at the same time, which is not ideal. You fix this using something called package cleanup, which is a tool part of Yamutils. Of course, you don't want to do this by hand. What we do is we have a small shell script that runs at the beginning of every time we run Chef on the machine. Chef runs every 15 minutes to convert the machine to the steady state. Before Chef runs, we want to make sure the machine is not terribly messed up. We run this package cleanup wrapper that just runs package cleanup in various ways and tries to resolve the transactions back and forth. That's one thing. The other issue you can have is general issues around RPM-DB corruption. This happened especially if you happen to kill RPM in the middle of a transaction. There's a very good chance you'll end up with a database in bad state. There's not really a single recipe to fix this. A lot of time, you have to try various solutions. We wrote a tool called DCRPM that takes care of these and tries a few remediations back and forth. We go to the database until the machine is in a non-terrible state. The package cleanup stuff is not terribly interesting because it's just a shell script. DCRPM is actually somewhat interesting. We're looking at trying to get it open sourced. We just finish it revriting it in a way that should be more maintainable. With these two remediations together, you tend to get a system in a reasonable state unless it's really broken because of, say, hardware issues. Then you can end up with other interesting problems. I think it was 2.31, 2.32. We saw that we had a lot of machines that would come up, they would do the upgrade, and then suddenly basically nothing would really work. You run system CTL and it would fail and the machine would be really sad. It turns out for reasons unclear to this day, we ended up with P1 running 2.34, but the library system elapsed from 2.33. They're dynamically linked and nothing really works at that point. My fix for this, which I'm not proud of but works, is in the sequence of remediations running LDD on the binary for P1, grabbing for stuff missing and forcing every install of all the packages to the right version. Surprisingly these works. It's a fairly curbed solution and it's not great, but these together put something in a state where we can update, we can update system D and other major packages and we do similar things for other packages. The first two are generic. The last thing is specific to system D, but these together put us in a state where we can update system D on the fleet in a reasonable fashion. Then there's the other side of the coin. We have to track what's going on in upstream and upstream exchanges. One change that happened a few releases ago was the change of the build system from Auto2 to Mason. If you're not familiar with this, Mason is this new Python based build system and system D transition from one to the other. They had both supported in one version and then they dropped AutoTools. That's fine. We expected to do a bunch of work. Luckily the Fedora package already did all the work because the Fedora packaging was moved to Mason almost immediately. We had time to basically rebuild our backboard on top of the new packaging and it was mostly okay. The annoying bit there was that Mason on CentOS didn't actually work at all. We had to back to Mason and Ninja and a few other Python things to keep it happy and fed. But that worked well. One good thing, though, we got out of this was some improvements on the Compadlibs. Compadlibs is something that's really old and cranky about that. Unfortunately, we have to deal with. CentOS 7 ships with system D219, I believe, like a fairly old version of system D. That version was before the library split. All the versions of system D shipped with split libraries. Newer versions would just live system D and all the symbols are there. It happens that packages like Apache or Samba, a lot of system packages you really don't want to rebuild, link to the old libraries. If you want to use the new system D, you need the old libraries in one way or another. I'm starting to up support for this because they're different. We used to have these pretty nasty patches to reinstate them and plumb them in the build system. That worked. It was quite a pain to keep it forward ported, but it was fine. With the move to Mason, that patch had to be tossed because it wasn't really workable. We started approaching alternative solutions. The thing we came out with was stealing some of the code from that patch, making it into a standalone project and leveraging the subprojects feature in Mason. Now we have a standalone thing that when it builds, it picks up a copy of the latest system D, builds it, builds the compile libraries and links that are the same and gives you this. This works. We published this and it's available on GitHub on that org in case you need it. We also published the RPMs that we use in the other repo in the RPMs backports. You can also directly get spec files. We're going to keep this mostly up to date and in sync with system D even if you don't really need to because these symbols are so old that they're not really expected to change. Okay. Then there's interesting things that happened because we have a special environment. So when we were old, I said before we were on 2.3.5 and 2.3.4.5. The reason is because we were doing 2.3.4 and it was fine. We would lose a few machines, but you always lose a few machines. That's okay. Then we bumped the shard to do the update to go to like 50%. We lost a lot of machines and that's not great. We started looking at what was happening. You go on these machines and SSH takes five minutes to let you in. You get on the machine and system D just hangs. You do system CTL and it stops. These things are not good. You run a trace against PID 1 and you see it's hanging trying to open a TTY, trying to open TTY 0, which is really not good. We pinned this down that it ran deep on the reexec. It went for 2.3.3 and 2.3.4 and 2.3.4 got stuck trying to open this TTY. After some digging, it turns out our container manager, Tupperware, also happens to fiddle with the TTY. The code in the kernel that deals with this is pretty old. There's probably a race there. We don't know yet what these is. We're still trying to figure it out. The current theory is that there's a user free bug in the line discipline code on the TTY subsystem that is triggered sometimes if you call DmonRia's economic machine that's also running our container manager that happens to do things to TTY 0. In some cases, we have an artificial repo of this, but the actual repo is pretty hard to get. The fix is that Tupperware probably should not fiddle with TTYs at all. That's like old code we should not have. We can fix that if we just make use of TTY. There's a bug in the kernel that our folks are trying to figure out. That's an example of something you're unlikely to get if you have a few machines or a lot of machines. When you have hundreds of thousands of machines, these kinds of crazy things unfortunately come up. Let's talk about interesting things we're doing with system D. The first thing is resource management. We're interested in resource management because we run a lot of things on our machines. We want to be able to control and make sure the machine is doing what it's supposed to do. The process that is supposed to run that has actual work, say the web server, has the resources it needs and doesn't get content from random auxiliary services, say the thing that controls power on the machine. We do this using SIGURP 2. I'm not going to talk about SIGURP 2 because my co-worker Chris is going to do a talk on this tomorrow. If you're interested in SIGURP 2, you should attend that. For the purpose of this SIGURP 2 is a kernel API that lets you set resource limits on processes. By resource limits, I mean things like memory, CPU, and IO. System D leverages this and lets you apply limits to your processes and to your services. It also lets you bucket your services and partition them in slices and apply limits to the slices as a whole. We use this and we use this to bucket the services and apply these limits. That's part of the picture. That's the implementation. You also need a way to tell what's going on. We do that with a small demo that runs on the boxes that picks metrics from SIGURP and shows them in our monitoring. We can get data around the limit is set here and the current memory usage is there. This is actually working. This is thrashing. There's a problem here. The other thing we do is that we have an API in Chef that lets people define and set the changes without having to go on individual hosts and do system CTL edit and change things. The way we do this is that we have an API in Chef and this translates to system D override files. It applies the changes and you can change the memory limit for this thing here or move the service from this bucket to this other bucket. This is something that's very much in progress and we're still learning what the right things to do are. This is the general hierarchy we're using right now where there's a system bucket and that's stuff that runs on the box that needs to run on the box but it's not critical to what the box is supposed to do. There's workload which is what the box is supposed to do, say MySQL or HHVM, the web server or whatever. Then we have another bucket that's called TBD inventively. That's meant for stuff that we need to run unrestricted, say the hardware folks have to do a stress test on the box and that doesn't need to be limited because it's a stress test. The initial idea was okay, we'll cap system to 4 gigabytes of RAM, we'll leave workload unlimited, it will be great. It's really not. First because our working set is way larger than 4 gigs so if you do that, the machine immediately dies. Also because it turns out it's not quite easy to tell what a workload is. In a lot of cases, say your workload is the web server but the web server gets configuration data from another demon. If that demon is in system and gets kept and becomes really slow, the web server is very angry so you need to figure something out there. The way we're addressing that is by making a sub bucket under workload and moving some of these demons there so we can control them better. The other thing we're doing is shifting the focus from doing memory limits and hard limiting in general to doing protection. Doing things like system DL is like memory low or memory.low in security problems that give you protection. Guarantee that this service will have at least this amount of memory available rather than a hard limit to this other service. This isn't always the solution and all of this stuff is hard. It requires you to understand in fair detail how your service works, what the dependencies are, how memory management in the kernel works and how these apply together with C groups. There's a lot of work to do to make this more simple and easier and we hope to be able to get to the point where we can give people a tool that they can run and give them an idea of, okay, these are sane defaults for your service based on what we see. Start with that and try. Another thing we are interested in doing is service monitoring. That stands from the fact that, as I said before, we had like five or six ways to do service monitoring, service management for bare metal, but they were all like fairly blind. We didn't have a good way to pull metrics from services. But systemD does because systemD knows a lot of things about you because it's supervising you directly. It stores them and crucially it exposes them over debuts. You get properties like all the timestamp properties. When did I start? How long have I been running? When was I restarted the last time? 235 added a thing that's really awesome and restarts. That's by far the main thing that people ask me that they would like to know. Is my service flapping? People are really interested in that because if your service is flapping, it's very likely that something is wrong. These kind of things are useful and they're easy to get out of systemD. SystemD also gives you status events where you can hook to the service on the bus and send your notification whenever the service changes. It goes from inactive to active and so on and so forth. It gives you basically a view of the state machine and what's going on. The downside of this is that these are exposed to debuts. You need to talk to debuts to get them out. We started looking at ways to do this. By the way, you don't need to talk to debuts. You can use systemCTL show and it will dump them all. The problem of doing that is that you don't want to run systemCTL show in a tight loop on every machine because that's going to take up a lot of resources and it's fairly brittle. You really want a programmatic way to do this. If you look online, there's a few toy projects I'd say on GitHub that already do this to some extent and all of them use either Python or libDebuss. Which would be great except after two weeks trying to get that to build in our internal build system, we essentially gave up. I'm still not quite sure what was going on there. It's definitely not an issue with debuts itself. It's an issue with how we do things at Facebook. The long story short, it was not resulting in minor that would run. We started looking at alternatives. Obviously, alternatives would be using SDBus, which is included in systemD for a while. The downside there is that SDBus is a plain C API. There are not really any wrappers for it. I really didn't want to write this in C because then I would have to maintain it. Most people in inFread Facebook are like Python people and not C people. There's also C people, but they're harder to come by. I don't know so far a prototype because I don't quite know if this would work. Writing a prototype in C is not a great idea, at least from my standpoint. I started looking at things and I found that the chorus folks had also bindings in Go. These bindings don't rely on libDebuss. They're standalone and they work. I don't know Go, but I figured I could pick it up. I did that and I wrote a POC of these in Go. This is basically a small demon that runs on a box. It hooks using the DBS API to systemD. It pulls for units, properties, and uses subscriptions to get events. Then it collects this data internally, messages it a bit, and then it should sit out to a few of our monitoring systems so we can get pretty pictures and data and things like that. I was fine for a POC. I was a bit uneasy of doing these in a language I didn't know very well and basically as a hack. Then Alvaro came along and Alvaro works on Instagram and they have the same problem because they use systemD to manage their services. He started looking at these and Alvaro is a better coder than I am and he knows how to use Syson. He wrote a wrapper using Syson on top of Sdbus. If you don't know about Syson, it's this magical Python thing that lets you call a C API from Python and expose what you get back in terms of Python objects. Using this system, we can talk to systemD through Dbus and talk to it and we get to interact with real Python objects that translate that internally to Dbus calls, which is kind of neat, especially for prototyping because you get a nice wrapper where you can directly focus things and get properties and see if it makes sense. We're likely going to use this and rewrite my prototype on this. We're going to open source this. I actually have the repo mostly ready, so it should be out in a couple of weeks. SystemD mount itself, we have to figure out how to release it, but we expect to release it at some point ideally this year. Hopefully, people will find this useful and I would love to get feedback on whether something like this could be useful or if you have ideas of how you deal with this problem or ways we could do this better. All right. Now, let's move on to interesting stories and case studies. By far, the main thing that still causes random issues that we don't quite understand is Dbus in production. The problem with Dbus is that when the demon gets sad or angry, systemD also gets sad or angry and the problems with the Dbus cannot really be resolved without rebooting the machine because Dbus doesn't really support takeover in place. Most of the time when bad things happen, your only resort goes booting the machine, which is fine for a laptop but not fine for a server. The other problem is that if you get the system to a state where systemD is still mostly working but the connection is severed, so system CTL is not working, either if it's hanging or if our product failing, like connection to bus failed, for us, that manifests a chef failing on the box because chef cannot manage services anymore, so it will yell at you, which is a problem because then people get all the chef failures but they're not really actionable for them because they see, oh, what is this Dbus stuff? The other thing we found is that it's actually surprisingly easy to dust Dbus and put it in a bad state. We had a co-worker who was doing tests with user services and he wrote a thing that when you would log in on the box, your batch RC will start a user service that will do some stuff and then start other things. That semi-reliably managed to crash it, not on every machine on like a good chunk of machines. Unfortunately, a lot of these problems are hard to pin down and like I would love to be able to get like, okay, here's a repro and file a bug upstream, but I don't have that. I have on X percent of machines, sometimes this happens and it's really fucking hard to find it. So we are right now we're mitigating this by basically rebooting machines when necessary and trying to keep Dbus up to date and get bug fixes. We're also looking at alternatives. There's this Dbus broker project that's an initial replacement of the Dbus demon coming up to the bus one project that we started looking at and testing on a small number of machines and it looks promising, but I don't have any hard data on whether it works well at our scale, but I'm definitely interested to see how this fares and if it's going to behave in a more reliable way or in an easier way to remediate in case of issues. Let's move on. RPM macros, yes. So we, this is one that's more like people problem than technological problems. So we, a lot of the stuff we build at Facebook are not, they're not like large complicated tools. They're like one binary. You have to ship one binary, one system DC unit and that's it. So we have a tool that you feed it, your build config and it spits out an RPM, it makes your spec file, it takes your system service at the right macros and here's your RPM, which is fine. The macros that you get by default by Fedora restart your service on package update, which is also fine, except a lot of people are used to the old design pattern of front send five, front send six, when you do a chef, you do package upgrade and then notify your service and ask chef to restart it. If you're doing that and you're also static on upgrade, you're starting twice, which doesn't seem to be a bad thing, except maybe your service takes quite a while to start or takes up a lot of resources when it's starting or starts talking to the world. So this fix for this was really easy. Like you add an object in the tool that you, so you can disable the default restart behavior and you're done. The trick was like socializing this and understanding that this was actually the problem and figuring out how to propagate it. And there's a few, there's a few things like this that I'll have later that are not necessarily technical issues, but more like getting things to be better understood. Another area of interest for us is the journal and logging. So the journal for us is set up to only log in memory, log in a small in memory buffer and then feed everything to syslog. And we do this because we have a ton of infrastructure based on syslog, our own security. So we need syslog to keep going. And people are used to syslog. People want to be able to tell the chef, see their stuff. On the other hand, we found pretty quickly that when people actually start understanding what the journal is and start playing with it, they really like it. People start using journal CTL and like, oh, I can filter by things. This actually works. So people started asking us, okay, can I use the journal? Because I'd like to get more data than just that, the shit-y 10 megabyte buffer. You can, of course, you can enlarge the buffer, you can make it store on disk. When you do that, you end up with a double writing problem. Because you're writing both syslog and the journal. They're both write to disk. And some of our tools are really chatty. So you can end up writing like megabytes or more per second, which is not ideal if you're IO constrained. And it's also an all or nothing proposition. Because you either do this system-wide or you don't do it at all. So what we'd really like to get is some way to do this on a per unit basis. So people could say, for these applications, I want the journal data to have these buffers and go on disk and be persistent for these to be transient. But we haven't found a good way to do that. So for now, what we're telling our customers to do is either take the hit of the double writing or set things in such a way that they silence some of the stuff that ends up ending up in syslog. But that causes other problems. So it's not really a tenable solution in the long run. Now another fun, for some value of fun problem we had was loops. So I didn't know this. I discovered this when I hit this problem. But if there's a loop, system-D breaks the loop between, a dependency loop between units, it will break it by removing an arbitrary unit from the loop. And it tells you, it puts a log line like that. And we found this out because we would machine and it would come up and it would be missing ten files and directories that were supposed to be created by system-D temp files. And then Chef would fail because this directory that's supposed to be here is not here. And it's like, what the hell? So you look at the logs and you see that line, which is, okay, that's interesting. Why? The way we debug this was using system-D analyze. And system-D analyze gives you both book chart style plots, like time zero, the system booth, time one, system-D starts, time two, the service starts. And it gives you dependency graphs, so nice, like octopus style graphs with dependencies. That's completely useless for us because if you plot it, you end up like the size of this roaming graph paper to get it out. But you can make it plot as small subset. And that part is useful. Because in this case, that line gave us enough clue to know that, like, SMC proxy, which is a thing we ran, was responsible somehow. So we started using analyze from there, and we figured that it has something to do with mouse, found that somebody added to FSTAB an entry to ask to mount this network file system and make it require SMC proxy, which is fine, but if you don't say it was actually a network file system, so this ended up ordered in an order in conflict because it's like, you need the file system, which needs the service, which needs the network, but the network needs this, and this doesn't work. So fixing this is adding net dev to the FSTAB entry and we're done. But the process of finding it was a bit more interesting than that. And that's one thing that we also, like, I wasn't personally aware of this was a thing, so it took a while to dig through the logs and see that this line was a problem and this is what was going on here. On a somewhat similar vein, another fun problem we had was around training units and system here on. So we have these machines that do builds and testing for mobile apps, and the way they do that is that there's a lot of phones plugged in USB, essentially. And then we run stuff that does things on the phone. And the way the team that manages does it, they use system Iran to start these processes and they run these using device allow policy so they only talk to the thing they're supposed to talk. And Iran is a lot because we do a lot of tests, we have a lot of phones. And every time it just runs, it enqueues a unit, runs a unit, does this thing, disposes of the unit, all good. But like, USB cables aren't great, sometimes stuff breaks, so this can also fail. When unit fails, system D lets them stick around, they stick around in like failed, and they're there. Not a problem if it's like one, two, three, five. Wait a bit. If you end up with like 10,000 units, there's something in PID 1 that reads this and ends up taking a shit ton of CPU. Like we saw with 10K failed units, this goes around 50% CPU usage on the box. With 30K, it goes around 100%, and that's when people started noticing because it was kind of interesting. So the fix is putting a cron job that caused reset failed. Yes. Yes, that's also what I told people to try. But for now, the way this was fixed was because people wanted to get a sense of how bad the problem was. So what they did was they added a monitoring counter that would tell them how many failed units are there, and then they added their remediation if it's past the threshold called reset failed. But yeah, ignoring the result is likely the better option. I need to check if there's a way to do that with system Iran directly or if they need to write a unit. I just thought about that we maybe want to improve system d run so that system d run can still extract the failed stuff, but then system d run automatically makes sure that the thing goes away. Oh, yeah, that would be awesome. I think that would take care of the case here. Yeah, because in this case, it's failures that are not really actionable because if it's failed, it's not because of something we can control. On to more fun, this time, C group related. If you have a service by default, when you terminate it, it kills everything in the C group. You can change this. You can tell it don't kill everything in the C group, only kill the main process and leave everything else to end off by itself. We try to discourage people from doing, but people are stubborn and sometimes do their own thing. If you combine this and when that happens, if you end up with leftover processes, the C group stays around because there are services in the C group, of course, that's processing in the C group. What happens if you then reload, you change the configuration of that service that before was in this slice, you make it go to this other slice and then reload system D. Ideally, you would expect it would apply the change. Turns out it doesn't if there's already a C group running with these processes in, which makes sense. Fixing this is easy. You kill everything or you change it to the control group. This was kind of surprising. It caused a bit of customization for people because they really wanted to use the kill mode process for reasons not particularly interesting here. The fact that the old unit was sticking around wasn't quite as evident. You basically find out because your own system city health status and it shows a stopped, but it also shows the C group is still there. Then you poke in the C group and you see it is there and there's processes running. Then the only result is just to kill everything. Yes. Yes. In this case, yes. The question was is there a reason if we're not using kill mode mix? In this case, the reason was that the thing we were starting was not actually the process. It was a thing that was spawned off a bunch of children and do hand-wavy supervision of these children and they wanted that they could replace the supervisor but not the children. It was something like that. The solution was redesigned is to not do that because you shouldn't do supervision in your application. I don't know if we tried mix specifically, but I remember the folks that had this system were fairly adamant. They wanted to use kill mode process the way that it designed. For the specific thing, there's not much we can do because the current behavior actually makes sense. It's not a problem. It's just something that's not evident at all. Finally, for the most stupid and irritating bug I found, the system.us is logic that translates the device names and paths to unit names and uses shell control characters. Slash, dev, slash something becomes dev dash something and then escape. Slash, escape is also a shell control character. If you take that and write system CTL status something and shout out to that without escaping it, you're going to have a bad time. Chef did that, which made it interesting when we were trying to write a cookbook to manage swap devices in some special way and it would fail because of our weird shell escape stuff. We fixed it. The fix was really trivial and it was basically make sure we call shell escape. It was somewhat embarrassing because it took a while to figure out that was the actual thing that was going on and we were like, we should have cut this before. As part of this, we wrote, we have a cookbook that's open source. We open source all of our cookbooks there on GitHub. We had a small wrapper to wrap the system.uscape function to get a unit name from path so you don't have to shout out from it. This closes the gallery of horrors. I want to spend a few minutes to talk about how you do these things that try to not make people too angry. If you look at things from the point of view of the service owner, the application owner, you kind of like your system to be stable and sort of static. On the other end, you want to get new features and you want to know that your system is stable but it's also getting updates. The approach we found that works reasonably well for us is trying to communicate things as much as possible to people and make sure people understand why you're doing this. Announcing core package updates widely and announcing changes that are happening and why they're happening and what's going on with them and giving people a venue to comment on whether this feature is interesting. I like to know more. I like to use it. This is going to cause me trouble. Doing these updates is a way that people can react to them and give you early feedback. When we did one of the system version upgrades, it happened to cause issues to the same folks that did the phone stuff because they were doing other secret things and it was fine. We could spin them to the old version for now, resolve it and then move on. The other thing we found is that documentation is critical. While upstream documentation is great and very detailed, it is not great if you don't know what you're doing and you're starting from the place of wanting to learn. We started writing internally snippets of like, okay, this is the suggested way to do things, like a basic unit to do these or that. We found that doing this with the custom use case in mind works better. We also encourage people to follow what's going on with upstream because we don't have all the answers. People can read the source code because this is an open source project. They can go on the mailing list. They can look at things. Finally, we found a talking and giving tech talks internally, both as a company and both going to a team and sitting at a team meeting and talk to them for an hour about what they're doing, what is the system they're saying, what you can do with it. It goes a long way making people happier and more amenable to using this and leveraging it and then actually liking it and building cool things with it. With this, thank you very much. I'm happy to take any questions you might have. Minor clarification. He said all of our cookbooks are open source. They are not and our lawyers will get angry. Some of our cookbooks are open source. Some of our core cookbooks. Yes. Thank you, Bill. I was curious. You were mentioning one of the things that you're having issues with in terms of Chef integrating with system D is with system control and things hanging. Is that because of Damon reloads or something else with the lockup? Aside from D bus issues. So Chef shells out to system CTL for basically every interaction it does with system D. So if I have a unit that says I want this service to be started, I want this service to be enabled, I'm adding a new unit or I'm changing a unit I need to reload system D. All of these things end up calling system CTL. So he sent any of these interactions if the system is unhealthy and the system returns an error or hangs, we'll lead to Chef failing. What I'm curious is if you know what the root cause is on the system D, say, PID one side of why those things are hanging. We know it sometimes. The kernel TTY raises one example where we found out what it was. In other cases, we don't. For some we pin it down to D bus. Most of the ones that are unknown, we're fairly sure it's something that we did the bus columns, but we haven't managed to report in a way that we can actually poke at it more and get some actual data out of it. Yes. Regarding the issues with RPM database and inconsistent packages, have you looked into deploying using atomic or OS tree or similar stuff? No. I'll tell you why. The main thing is that for system packages specifically, we want to try to stay as close as possible to how upstream is built and how upstream works because it makes it a lot easier to keep things not too alien so people can understand it and when you interact with people outside. We might well look at that in the future, but I think in the short midterm, I think we're going to stick with RPM and just deal with the things. The other thing we're doing is trying to make RPM better and we're actually engaging with the RPM developers and the YAM developers there. Hopefully we'll have something there soon. Well, my time is up. Thank you very much, folks. Thank you.
 All right. Let's get started. Hello, everybody. My name is Davide. I'm a production engineer  at Facebook. I'll be talking about what we've been doing with system D for the past year.  Before we begin, here's the agenda for today. I'll start with a quick recap of the stories  so far. We'll talk about how we are keeping system D updated in our fleet and how we are  tracking upstream changes. We'll focus on a couple of things we've been working on lately  around resource management and service monitoring. Then we'll discuss a few case studies that  hopefully showcase a bunch of interesting problems we've seen and close with a couple  of words about advocacy. I was at system D account about a year ago. At the time, we  were moving the fleet from center six to center seven. My team works on the operating  system team. My team is responsible for the bare metal experience of the fleet. We maintain  what keeps the physical machines at Facebook running. We maintain the operating system  with the centers. We take care of packaging. We take care of configuration management using  Chef among other things. Our fleet is made of hundreds of thousands of physical machines  spread around various data centers in the world. All these machines run centers. That's  what runs the website and everything else. About a year ago, we were moving from six  to seven. On six, we had five or six different crazy ways of supervising services. On seven,  we have system D. I'm happy to say that now we're everywhere running center seven and  everywhere running system D. Which makes me personally very happy because we managed to  get rid of all of these crazy ways of doing service supervision. As part of this, we got  to migrate a lot of services to system D. We got to see a lot of people that started  building their services with system D mine and started leveraging more and more features.  One thing we also did to help this, we started integrating lip system D in our internal build  system so that people are now able to use features like the socket activation and all  these features directly in our demons that make people's life a bit easier. This talk  will focus primarily on the bare metal. Zil and Zolton are going to give a talk later  today about containers. If you're interested about system D in containers, I recommend  you attend the later talk. Let's talk a bit about how we are managing system D on the  fleet. In general, at Facebook, we manage machines using Chef and we have a system to  do package updates on machines in a controlled fashion. We can say this package has been  to this version and then 1% of the fleet gets this other version and 2% and 5% and so on  and so forth. That's the system we use for managing updates of system packages. About  a year ago, we were on system D 231 and we went from 31, 32, 32, 33, 33, 34. Right now,  we have about half the fleet on 233 and the other half on 234 and I'll explain why in  a few slides. We also started testing 235. In general, we try to run, we run CentOS 7  because we want a stable base and we want to be able to pull security updates in an  automated fashion but we also want to have a modern user space. We backport a lot of  core system components from Fedora or Hyde. These components are system D, of course,  but there's also a lot of the ancillary ecosystem. We backport things like D-Bus and Util Linux  and Proc.ps and a lot of these basic system tools. The experience you get using the system  is somewhat similar to the experience you would have using a Fedora system, at least  from the point of view of a developer. We publish these backports on GitHub and you can get  them on that GitHub org. These are just the spec files and whatever patches we have. System  D there is on 235. This is something that came out from last year's conference because  people were interested in this so we made an effort to get them published. Then, Yana  Reddick was kind enough to make a copper from these packages. If you happen to run CentOS  7 and happen to want a modern system D, you can get binary packages directly from there  and those are also mostly up to date. If you need the earlier versions, you can go back  in the GitHub history. Now, of course, having a lot of machines, the updates aren't always  smooth and things can happen. I'm going to go over a few interesting things that happen  during package updates that might make things a little bit more interesting. The first thing  is not actually a system D-specific thing. It's something that happens in general when  you are dealing with a large fleet of write-up systems and have two or eight packages on  all of them. It's issues generally around RPM. Machines can get in bad states for various  reasons. You can have issues around power loss. You can have people running kill-dash-9  on things or processes running kill-dash-9 on things that can leave the system in weirder  than interesting states. This can result in found situations. Some things we found that  we always get this every time we update a major package on the fleet, we always get a sizable  number of these things. We get issues like the obligated packages. You end up with a  machine that does both system D-233 and system D-234 installed at the same time, which is  not ideal. You fix this using something called package cleanup, which is a tool part of Yamutils.  Of course, you don't want to do this by hand. What we do is we have a small shell script  that runs at the beginning of every time we run Chef on the machine. Chef runs every 15  minutes to convert the machine to the steady state. Before Chef runs, we want to make sure  the machine is not terribly messed up. We run this package cleanup wrapper that just  runs package cleanup in various ways and tries to resolve the transactions back and forth.  That's one thing. The other issue you can have is general issues around RPM-DB corruption.  This happened especially if you happen to kill RPM in the middle of a transaction. There's  a very good chance you'll end up with a database in bad state. There's not really a single  recipe to fix this. A lot of time, you have to try various solutions. We wrote a tool  called DCRPM that takes care of these and tries a few remediations back and forth. We  go to the database until the machine is in a non-terrible state. The package cleanup  stuff is not terribly interesting because it's just a shell script. DCRPM is actually  somewhat interesting. We're looking at trying to get it open sourced. We just finish it  revriting it in a way that should be more maintainable. With these two remediations together,  you tend to get a system in a reasonable state unless it's really broken because of, say,  hardware issues. Then you can end up with other interesting problems. I think it was  2.31, 2.32. We saw that we had a lot of machines that would come up, they would do the upgrade,  and then suddenly basically nothing would really work. You run system CTL and it would  fail and the machine would be really sad. It turns out for reasons unclear to this day,  we ended up with P1 running 2.34, but the library system elapsed from 2.33. They're dynamically  linked and nothing really works at that point. My fix for this, which I'm not proud of but  works, is in the sequence of remediations running LDD on the binary for P1, grabbing  for stuff missing and forcing every install of all the packages to the right version.  Surprisingly these works. It's a fairly curbed solution and it's not great, but these together  put something in a state where we can update, we can update system D and other major packages  and we do similar things for other packages. The first two are generic. The last thing  is specific to system D, but these together put us in a state where we can update system  D on the fleet in a reasonable fashion. Then there's the other side of the coin. We have  to track what's going on in upstream and upstream exchanges. One change that happened a few  releases ago was the change of the build system from Auto2 to Mason. If you're not familiar  with this, Mason is this new Python based build system and system D transition from  one to the other. They had both supported in one version and then they dropped AutoTools.  That's fine. We expected to do a bunch of work. Luckily the Fedora package already did all  the work because the Fedora packaging was moved to Mason almost immediately. We had time to  basically rebuild our backboard on top of the new packaging and it was mostly okay. The  annoying bit there was that Mason on CentOS didn't actually work at all. We had to back  to Mason and Ninja and a few other Python things to keep it happy and fed. But that worked  well. One good thing, though, we got out of this was some improvements on the Compadlibs.  Compadlibs is something that's really old and cranky about that. Unfortunately, we have  to deal with. CentOS 7 ships with system D219, I believe, like a fairly old version of system  D. That version was before the library split. All the versions of system D shipped with  split libraries. Newer versions would just live system D and all the symbols are there.  It happens that packages like Apache or Samba, a lot of system packages you really don't  want to rebuild, link to the old libraries. If you want to use the new system D, you need  the old libraries in one way or another. I'm starting to up support for this because they're  different. We used to have these pretty nasty patches to reinstate them and plumb them in  the build system. That worked. It was quite a pain to keep it forward ported, but it was  fine. With the move to Mason, that patch had to be tossed because it wasn't really workable.  We started approaching alternative solutions. The thing we came out with was stealing some  of the code from that patch, making it into a standalone project and leveraging the subprojects  feature in Mason. Now we have a standalone thing that when it builds, it picks up a copy  of the latest system D, builds it, builds the compile libraries and links that are the  same and gives you this. This works. We published this and it's available on GitHub on that  org in case you need it. We also published the RPMs that we use in the other repo in  the RPMs backports. You can also directly get spec files. We're going to keep this mostly  up to date and in sync with system D even if you don't really need to because these symbols  are so old that they're not really expected to change.  Okay. Then there's interesting things that happened because we have a special environment.  So when we were old, I said before we were on 2.3.5 and 2.3.4.5. The reason is because  we were doing 2.3.4 and it was fine. We would lose a few machines, but you always lose a  few machines. That's okay. Then we bumped the shard to do the update to go to like 50%.  We lost a lot of machines and that's not great. We started looking at what was happening.  You go on these machines and SSH takes five minutes to let you in. You get on the machine  and system D just hangs. You do system CTL and it stops. These things are not good. You  run a trace against PID 1 and you see it's hanging trying to open a TTY, trying to open  TTY 0, which is really not good. We pinned this down that it ran deep on the reexec.  It went for 2.3.3 and 2.3.4 and 2.3.4 got stuck trying to open this TTY. After some  digging, it turns out our container manager, Tupperware, also happens to fiddle with the  TTY. The code in the kernel that deals with this is pretty old. There's probably a race  there. We don't know yet what these is. We're still trying to figure it out. The current  theory is that there's a user free bug in the line discipline code on the TTY subsystem  that is triggered sometimes if you call DmonRia's economic machine that's also running our container  manager that happens to do things to TTY 0. In some cases, we have an artificial repo  of this, but the actual repo is pretty hard to get. The fix is that Tupperware probably  should not fiddle with TTYs at all. That's like old code we should not have. We can fix  that if we just make use of TTY. There's a bug in the kernel that our folks are trying  to figure out. That's an example of something you're unlikely to get if you have a few machines  or a lot of machines. When you have hundreds of thousands of machines, these kinds of crazy  things unfortunately come up. Let's talk about interesting things we're doing with system  D. The first thing is resource management. We're interested in resource management because  we run a lot of things on our machines. We want to be able to control and make sure the  machine is doing what it's supposed to do. The process that is supposed to run that has  actual work, say the web server, has the resources it needs and doesn't get content from random  auxiliary services, say the thing that controls power on the machine. We do this using SIGURP  2. I'm not going to talk about SIGURP 2 because my co-worker Chris is going to do a talk on  this tomorrow. If you're interested in SIGURP 2, you should attend that. For the purpose  of this SIGURP 2 is a kernel API that lets you set resource limits on processes. By resource  limits, I mean things like memory, CPU, and IO. System D leverages this and lets you apply  limits to your processes and to your services. It also lets you bucket your services and  partition them in slices and apply limits to the slices as a whole. We use this and  we use this to bucket the services and apply these limits. That's part of the picture.  That's the implementation. You also need a way to tell what's going on. We do that with  a small demo that runs on the boxes that picks metrics from SIGURP and shows them in our  monitoring. We can get data around the limit is set here and the current memory usage is  there. This is actually working. This is thrashing. There's a problem here. The other thing we  do is that we have an API in Chef that lets people define and set the changes without  having to go on individual hosts and do system CTL edit and change things. The way we do  this is that we have an API in Chef and this translates to system D override files. It  applies the changes and you can change the memory limit for this thing here or move  the service from this bucket to this other bucket. This is something that's very much  in progress and we're still learning what the right things to do are. This is the general  hierarchy we're using right now where there's a system bucket and that's stuff that runs  on the box that needs to run on the box but it's not critical to what the box is supposed  to do. There's workload which is what the box is supposed to do, say MySQL or HHVM,  the web server or whatever. Then we have another bucket that's called TBD inventively. That's  meant for stuff that we need to run unrestricted, say the hardware folks have to do a stress  test on the box and that doesn't need to be limited because it's a stress test. The  initial idea was okay, we'll cap system to 4 gigabytes of RAM, we'll leave workload unlimited,  it will be great. It's really not. First because our working set is way larger than 4 gigs  so if you do that, the machine immediately dies. Also because it turns out it's not quite  easy to tell what a workload is. In a lot of cases, say your workload is the web server  but the web server gets configuration data from another demon. If that demon is in system  and gets kept and becomes really slow, the web server is very angry so you need to figure  something out there. The way we're addressing that is by making a sub bucket under workload  and moving some of these demons there so we can control them better. The other thing we're  doing is shifting the focus from doing memory limits and hard limiting in general to doing  protection. Doing things like system DL is like memory low or memory.low in security  problems that give you protection. Guarantee that this service will have at least this  amount of memory available rather than a hard limit to this other service. This isn't always  the solution and all of this stuff is hard. It requires you to understand in fair detail  how your service works, what the dependencies are, how memory management in the kernel works  and how these apply together with C groups. There's a lot of work to do to make this more  simple and easier and we hope to be able to get to the point where we can give people  a tool that they can run and give them an idea of, okay, these are sane defaults for  your service based on what we see. Start with that and try. Another thing we are interested  in doing is service monitoring. That stands from the fact that, as I said before, we had  like five or six ways to do service monitoring, service management for bare metal, but they  were all like fairly blind. We didn't have a good way to pull metrics from services.  But systemD does because systemD knows a lot of things about you because it's supervising  you directly. It stores them and crucially it exposes them over debuts. You get properties  like all the timestamp properties. When did I start? How long have I been running? When  was I restarted the last time? 235 added a thing that's really awesome and restarts.  That's by far the main thing that people ask me that they would like to know. Is my service  flapping? People are really interested in that because if your service is flapping, it's  very likely that something is wrong. These kind of things are useful and they're easy  to get out of systemD. SystemD also gives you status events where you can hook to the  service on the bus and send your notification whenever the service changes. It goes from  inactive to active and so on and so forth. It gives you basically a view of the state  machine and what's going on. The downside of this is that these are exposed to debuts.  You need to talk to debuts to get them out. We started looking at ways to do this. By  the way, you don't need to talk to debuts. You can use systemCTL show and it will dump  them all. The problem of doing that is that you don't want to run systemCTL show in a  tight loop on every machine because that's going to take up a lot of resources and it's  fairly brittle. You really want a programmatic way to do this. If you look online, there's  a few toy projects I'd say on GitHub that already do this to some extent and all of  them use either Python or libDebuss. Which would be great except after two weeks trying  to get that to build in our internal build system, we essentially gave up. I'm still  not quite sure what was going on there. It's definitely not an issue with debuts itself.  It's an issue with how we do things at Facebook. The long story short, it was not resulting  in minor that would run. We started looking at alternatives. Obviously, alternatives would  be using SDBus, which is included in systemD for a while. The downside there is that SDBus  is a plain C API. There are not really any wrappers for it. I really didn't want to write  this in C because then I would have to maintain it. Most people in inFread Facebook are like  Python people and not C people. There's also C people, but they're harder to come by. I  don't know so far a prototype because I don't quite know if this would work. Writing a prototype  in C is not a great idea, at least from my standpoint. I started looking at things and  I found that the chorus folks had also bindings in Go. These bindings don't rely on libDebuss.  They're standalone and they work. I don't know Go, but I figured I could pick it up.  I did that and I wrote a POC of these in Go. This is basically a small demon that runs  on a box. It hooks using the DBS API to systemD. It pulls for units, properties, and uses  subscriptions to get events. Then it collects this data internally, messages it a bit, and  then it should sit out to a few of our monitoring systems so we can get pretty pictures and  data and things like that. I was fine for a POC. I was a bit uneasy of doing these in  a language I didn't know very well and basically as a hack. Then Alvaro came along and Alvaro  works on Instagram and they have the same problem because they use systemD to manage  their services. He started looking at these and Alvaro is a better coder than I am and  he knows how to use Syson. He wrote a wrapper using Syson on top of Sdbus. If you don't  know about Syson, it's this magical Python thing that lets you call a C API from Python  and expose what you get back in terms of Python objects. Using this system, we can talk to  systemD through Dbus and talk to it and we get to interact with real Python objects that  translate that internally to Dbus calls, which is kind of neat, especially for prototyping  because you get a nice wrapper where you can directly focus things and get properties and  see if it makes sense. We're likely going to use this and rewrite my prototype on this.  We're going to open source this. I actually have the repo mostly ready, so it should be  out in a couple of weeks. SystemD mount itself, we have to figure out how to release it, but  we expect to release it at some point ideally this year. Hopefully, people will find this  useful and I would love to get feedback on whether something like this could be useful  or if you have ideas of how you deal with this problem or ways we could do this better.  All right. Now, let's move on to interesting stories and case studies. By far, the main  thing that still causes random issues that we don't quite understand is Dbus in production.  The problem with Dbus is that when the demon gets sad or angry, systemD also gets sad or  angry and the problems with the Dbus cannot really be resolved without rebooting the machine  because Dbus doesn't really support takeover in place. Most of the time when bad things  happen, your only resort goes booting the machine, which is fine for a laptop but not  fine for a server. The other problem is that if you get the system to a state where systemD  is still mostly working but the connection is severed, so system CTL is not working,  either if it's hanging or if our product failing, like connection to bus failed, for us, that  manifests a chef failing on the box because chef cannot manage services anymore, so it  will yell at you, which is a problem because then people get all the chef failures but  they're not really actionable for them because they see, oh, what is this Dbus stuff? The  other thing we found is that it's actually surprisingly easy to dust Dbus and put it  in a bad state. We had a co-worker who was doing tests with user services and he wrote  a thing that when you would log in on the box, your batch RC will start a user service  that will do some stuff and then start other things. That semi-reliably managed to crash  it, not on every machine on like a good chunk of machines. Unfortunately, a lot of these  problems are hard to pin down and like I would love to be able to get like, okay, here's  a repro and file a bug upstream, but I don't have that. I have on X percent of machines,  sometimes this happens and it's really fucking hard to find it.  So we are right now we're mitigating this by basically rebooting machines when necessary  and trying to keep Dbus up to date and get bug fixes. We're also looking at alternatives.  There's this Dbus broker project that's an initial replacement of the Dbus demon coming  up to the bus one project that we started looking at and testing on a small number of  machines and it looks promising, but I don't have any hard data on whether it works well  at our scale, but I'm definitely interested to see how this fares and if it's going to  behave in a more reliable way or in an easier way to remediate in case of issues.  Let's move on. RPM macros, yes. So we, this is one that's more like people problem than  technological problems. So we, a lot of the stuff we build at Facebook are not, they're  not like large complicated tools. They're like one binary. You have to ship one binary,  one system DC unit and that's it. So we have a tool that you feed it, your build config  and it spits out an RPM, it makes your spec file, it takes your system service at the  right macros and here's your RPM, which is fine. The macros that you get by default by  Fedora restart your service on package update, which is also fine, except a lot of people  are used to the old design pattern of front send five, front send six, when you do a chef,  you do package upgrade and then notify your service and ask chef to restart it. If you're  doing that and you're also static on upgrade, you're starting twice, which doesn't seem  to be a bad thing, except maybe your service takes quite a while to start or takes up a  lot of resources when it's starting or starts talking to the world. So this fix for this  was really easy. Like you add an object in the tool that you, so you can disable the  default restart behavior and you're done. The trick was like socializing this and understanding  that this was actually the problem and figuring out how to propagate it. And there's a few,  there's a few things like this that I'll have later that are not necessarily technical issues,  but more like getting things to be better understood. Another area of interest for us  is the journal and logging. So the journal for us is set up to only log in memory, log  in a small in memory buffer and then feed everything to syslog. And we do this because  we have a ton of infrastructure based on syslog, our own security. So we need syslog to keep  going. And people are used to syslog. People want to be able to tell the chef, see their  stuff. On the other hand, we found pretty quickly that when people actually start understanding  what the journal is and start playing with it, they really like it. People start using  journal CTL and like, oh, I can filter by things. This actually works. So people started  asking us, okay, can I use the journal? Because I'd like to get more data than just that,  the shit-y 10 megabyte buffer. You can, of course, you can enlarge the buffer, you can  make it store on disk. When you do that, you end up with a double writing problem. Because  you're writing both syslog and the journal. They're both write to disk. And some of our  tools are really chatty. So you can end up writing like megabytes or more per second,  which is not ideal if you're IO constrained. And it's also an all or nothing proposition.  Because you either do this system-wide or you don't do it at all. So what we'd really  like to get is some way to do this on a per unit basis. So people could say, for these  applications, I want the journal data to have these buffers and go on disk and be persistent  for these to be transient. But we haven't found a good way to do that. So for now, what  we're telling our customers to do is either take the hit of the double writing or set  things in such a way that they silence some of the stuff that ends up ending up in syslog.  But that causes other problems. So it's not really a tenable solution in the long run.  Now another fun, for some value of fun problem we had was loops. So I didn't know this. I  discovered this when I hit this problem. But if there's a loop, system-D breaks the loop  between, a dependency loop between units, it will break it by removing an arbitrary  unit from the loop. And it tells you, it puts a log line like that. And we found this out  because we would machine and it would come up and it would be missing ten files and directories  that were supposed to be created by system-D temp files. And then Chef would fail because  this directory that's supposed to be here is not here. And it's like, what the hell?  So you look at the logs and you see that line, which is, okay, that's interesting. Why? The  way we debug this was using system-D analyze. And system-D analyze gives you both book chart  style plots, like time zero, the system booth, time one, system-D starts, time two, the service  starts. And it gives you dependency graphs, so nice, like octopus style graphs with dependencies.  That's completely useless for us because if you plot it, you end up like the size of this  roaming graph paper to get it out. But you can make it plot as small subset. And that  part is useful. Because in this case, that line gave us enough clue to know that, like,  SMC proxy, which is a thing we ran, was responsible somehow. So we started using analyze from there,  and we figured that it has something to do with mouse, found that somebody added to FSTAB  an entry to ask to mount this network file system and make it require SMC proxy, which  is fine, but if you don't say it was actually a network file system, so this ended up ordered  in an order in conflict because it's like, you need the file system, which needs the  service, which needs the network, but the network needs this, and this doesn't work.  So fixing this is adding net dev to the FSTAB entry and we're done. But the process of finding  it was a bit more interesting than that. And that's one thing that we also, like, I wasn't  personally aware of this was a thing, so it took a while to dig through the logs and see  that this line was a problem and this is what was going on here.  On a somewhat similar vein, another fun problem we had was around training units and system  here on. So we have these machines that do builds and testing for mobile apps, and the  way they do that is that there's a lot of phones plugged in USB, essentially. And then  we run stuff that does things on the phone. And the way the team that manages does it,  they use system Iran to start these processes and they run these using device allow policy  so they only talk to the thing they're supposed to talk. And Iran is a lot because we do a  lot of tests, we have a lot of phones. And every time it just runs, it enqueues a unit,  runs a unit, does this thing, disposes of the unit, all good. But like, USB cables aren't  great, sometimes stuff breaks, so this can also fail. When unit fails, system D lets them  stick around, they stick around in like failed, and they're there. Not a problem if it's like  one, two, three, five. Wait a bit. If you end up with like 10,000 units, there's something  in PID 1 that reads this and ends up taking a shit ton of CPU. Like we saw with 10K failed  units, this goes around 50% CPU usage on the box. With 30K, it goes around 100%, and that's  when people started noticing because it was kind of interesting. So the fix is putting  a cron job that caused reset failed. Yes.  Yes, that's also what I told people to try. But for now, the way this was fixed was because  people wanted to get a sense of how bad the problem was. So what they did was they added  a monitoring counter that would tell them how many failed units are there, and then they  added their remediation if it's past the threshold called reset failed. But yeah, ignoring the  result is likely the better option. I need to check if there's a way to do that with  system Iran directly or if they need to write a unit.  I just thought about that we maybe want to improve system d run so that system d run  can still extract the failed stuff, but then system d run automatically makes sure that  the thing goes away.  Oh, yeah, that would be awesome. I think that would take care of the case here. Yeah, because  in this case, it's failures that are not really actionable because if it's failed, it's not  because of something we can control.  On to more fun, this time, C group related. If you have a service by default, when you  terminate it, it kills everything in the C group. You can change this. You can tell  it don't kill everything in the C group, only kill the main process and leave everything  else to end off by itself. We try to discourage people from doing, but people are stubborn  and sometimes do their own thing. If you combine this and when that happens, if you end up  with leftover processes, the C group stays around because there are services in the C  group, of course, that's processing in the C group.  What happens if you then reload, you change the configuration of that service that before  was in this slice, you make it go to this other slice and then reload system D. Ideally,  you would expect it would apply the change. Turns out it doesn't if there's already a  C group running with these processes in, which makes sense. Fixing this is easy. You kill  everything or you change it to the control group. This was kind of surprising. It caused  a bit of customization for people because they really wanted to use the kill mode process  for reasons not particularly interesting here. The fact that the old unit was sticking around  wasn't quite as evident. You basically find out because your own system city health status  and it shows a stopped, but it also shows the C group is still there. Then you poke  in the C group and you see it is there and there's processes running. Then the only  result is just to kill everything. Yes. Yes. In this case, yes. The question was is there  a reason if we're not using kill mode mix? In this case, the reason was that the thing  we were starting was not actually the process. It was a thing that was spawned off a bunch  of children and do hand-wavy supervision of these children and they wanted that they could  replace the supervisor but not the children. It was something like that. The solution was  redesigned is to not do that because you shouldn't do supervision in your application. I don't  know if we tried mix specifically, but I remember the folks that had this system were fairly  adamant. They wanted to use kill mode process the way that it designed. For the specific  thing, there's not much we can do because the current behavior actually makes sense.  It's not a problem. It's just something that's not evident at all. Finally, for the most  stupid and irritating bug I found, the system.us is logic that translates the device names and  paths to unit names and uses shell control characters. Slash, dev, slash something becomes  dev dash something and then escape. Slash, escape is also a shell control character.  If you take that and write system CTL status something and shout out to that without escaping  it, you're going to have a bad time. Chef did that, which made it interesting when we  were trying to write a cookbook to manage swap devices in some special way and it would  fail because of our weird shell escape stuff. We fixed it. The fix was really trivial and  it was basically make sure we call shell escape. It was somewhat embarrassing because it took  a while to figure out that was the actual thing that was going on and we were like,  we should have cut this before. As part of this, we wrote, we have a cookbook that's open  source. We open source all of our cookbooks there on GitHub. We had a small wrapper to  wrap the system.uscape function to get a unit name from path so you don't have to shout  out from it. This closes the gallery of horrors. I want to spend a few minutes to talk about  how you do these things that try to not make people too angry. If you look at things from  the point of view of the service owner, the application owner, you kind of like your system  to be stable and sort of static. On the other end, you want to get new features and you  want to know that your system is stable but it's also getting updates. The approach we  found that works reasonably well for us is trying to communicate things as much as possible  to people and make sure people understand why you're doing this. Announcing core package  updates widely and announcing changes that are happening and why they're happening and  what's going on with them and giving people a venue to comment on whether this feature  is interesting. I like to know more. I like to use it. This is going to cause me trouble.  Doing these updates is a way that people can react to them and give you early feedback.  When we did one of the system version upgrades, it happened to cause issues to the same folks  that did the phone stuff because they were doing other secret things and it was fine.  We could spin them to the old version for now, resolve it and then move on. The other  thing we found is that documentation is critical. While upstream documentation is great and  very detailed, it is not great if you don't know what you're doing and you're starting  from the place of wanting to learn. We started writing internally snippets of like, okay,  this is the suggested way to do things, like a basic unit to do these or that. We found  that doing this with the custom use case in mind works better. We also encourage people  to follow what's going on with upstream because we don't have all the answers. People can  read the source code because this is an open source project. They can go on the mailing  list. They can look at things. Finally, we found a talking and giving tech talks internally,  both as a company and both going to a team and sitting at a team meeting and talk to  them for an hour about what they're doing, what is the system they're saying, what you  can do with it. It goes a long way making people happier and more amenable to using  this and leveraging it and then actually liking it and building cool things with it. With  this, thank you very much. I'm happy to take any questions you might have.  Minor clarification. He said all of our cookbooks are open source. They are not and our lawyers  will get angry. Some of our cookbooks are open source. Some of our core cookbooks. Yes.  Thank you, Bill. I was curious. You were mentioning one of the things that you're having issues  with in terms of Chef integrating with system D is with system control and things hanging.  Is that because of Damon reloads or something else with the lockup? Aside from D bus issues.  So Chef shells out to system CTL for basically every interaction it does with system D. So  if I have a unit that says I want this service to be started, I want this service to be enabled,  I'm adding a new unit or I'm changing a unit I need to reload system D. All of these things  end up calling system CTL. So he sent any of these interactions if the system is unhealthy  and the system returns an error or hangs, we'll lead to Chef failing.  What I'm curious is if you know what the root cause is on the system D, say, PID one side  of why those things are hanging. We know it sometimes. The kernel TTY raises  one example where we found out what it was. In other cases, we don't. For some we pin  it down to D bus. Most of the ones that are unknown, we're fairly sure it's something  that we did the bus columns, but we haven't managed to report in a way that we can actually  poke at it more and get some actual data out of it. Yes.  Regarding the issues with RPM database and inconsistent packages, have you looked into  deploying using atomic or OS tree or similar stuff?  No. I'll tell you why. The main thing is that for system packages specifically, we want  to try to stay as close as possible to how upstream is built and how upstream works because  it makes it a lot easier to keep things not too alien so people can understand it and  when you interact with people outside. We might well look at that in the future, but  I think in the short midterm, I think we're going to stick with RPM and just deal with  the things. The other thing we're doing is trying to make RPM better and we're actually  engaging with the RPM developers and the YAM developers there. Hopefully we'll have something  there soon. Well, my time is up. Thank you very much, folks.  Thank you.
2017_systemd___Facebook___a_year_later_102kzouLJ2c_0_47.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_48_155.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_156_395.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_396_995.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_996_1607.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_1608_2159.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_2160_3419.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_3420_3971.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_3972_6527.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_6528_11603.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_11604_15599.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_15600_18491.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_18492_19127.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_19128_21311.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_21312_24551.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_24552_24995.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_24996_28919.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_28920_29735.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_29736_31703.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_31704_31931.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_31932_35195.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_35196_37583.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_37584_38759.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_38760_39167.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_39168_39935.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_39936_40139.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_40140_43475.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_43476_47003.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_47004_50903.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_50904_52787.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_52788_52931.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_52932_53759.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_53760_54515.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_54516_55283.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_55284_55667.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_55668_55955.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_55956_60395.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_60396_60479.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_60480_60527.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_60528_60587.png 2017_systemd___Facebook___a_year_later_102kzouLJ2c_60588_60626.png
All right. Let's get started. Hello, everybody. My name is Davide. I'm a production engineer at Facebook. I'll be talking about what we've been doing with system D for the past year. Before we begin, here's the agenda for today. I'll start with a quick recap of the stories so far. We'll talk about how we are keeping system D updated in our fleet and how we are tracking upstream changes. We'll focus on a couple of things we've been working on lately around resource management and service monitoring. Then we'll discuss a few case studies that hopefully showcase a bunch of interesting problems we've seen and close with a couple of words about advocacy. I was at system D account about a year ago. At the time, we were moving the fleet from center six to center seven. My team works on the operating system team. My team is responsible for the bare metal experience of the fleet. We maintain what keeps the physical machines at Facebook running. We maintain the operating system with the centers. We take care of packaging. We take care of configuration management using Chef among other things. Our fleet is made of hundreds of thousands of physical machines spread around various data centers in the world. All these machines run centers. That's what runs the website and everything else. About a year ago, we were moving from six to seven. On six, we had five or six different crazy ways of supervising services. On seven, we have system D. I'm happy to say that now we're everywhere running center seven and everywhere running system D. Which makes me personally very happy because we managed to get rid of all of these crazy ways of doing service supervision. As part of this, we got to migrate a lot of services to system D. We got to see a lot of people that started building their services with system D mine and started leveraging more and more features. One thing we also did to help this, we started integrating lip system D in our internal build system so that people are now able to use features like the socket activation and all these features directly in our demons that make people's life a bit easier. This talk will focus primarily on the bare metal. Zil and Zolton are going to give a talk later today about containers. If you're interested about system D in containers, I recommend you attend the later talk. Let's talk a bit about how we are managing system D on the fleet. In general, at Facebook, we manage machines using Chef and we have a system to do package updates on machines in a controlled fashion. We can say this package has been to this version and then 1% of the fleet gets this other version and 2% and 5% and so on and so forth. That's the system we use for managing updates of system packages. About a year ago, we were on system D 231 and we went from 31, 32, 32, 33, 33, 34. Right now, we have about half the fleet on 233 and the other half on 234 and I'll explain why in a few slides. We also started testing 235. In general, we try to run, we run CentOS 7 because we want a stable base and we want to be able to pull security updates in an automated fashion but we also want to have a modern user space. We backport a lot of core system components from Fedora or Hyde. These components are system D, of course, but there's also a lot of the ancillary ecosystem. We backport things like D-Bus and Util Linux and Proc.ps and a lot of these basic system tools. The experience you get using the system is somewhat similar to the experience you would have using a Fedora system, at least from the point of view of a developer. We publish these backports on GitHub and you can get them on that GitHub org. These are just the spec files and whatever patches we have. System D there is on 235. This is something that came out from last year's conference because people were interested in this so we made an effort to get them published. Then, Yana Reddick was kind enough to make a copper from these packages. If you happen to run CentOS 7 and happen to want a modern system D, you can get binary packages directly from there and those are also mostly up to date. If you need the earlier versions, you can go back in the GitHub history. Now, of course, having a lot of machines, the updates aren't always smooth and things can happen. I'm going to go over a few interesting things that happen during package updates that might make things a little bit more interesting. The first thing is not actually a system D-specific thing. It's something that happens in general when you are dealing with a large fleet of write-up systems and have two or eight packages on all of them. It's issues generally around RPM. Machines can get in bad states for various reasons. You can have issues around power loss. You can have people running kill-dash-9 on things or processes running kill-dash-9 on things that can leave the system in weirder than interesting states. This can result in found situations. Some things we found that we always get this every time we update a major package on the fleet, we always get a sizable number of these things. We get issues like the obligated packages. You end up with a machine that does both system D-233 and system D-234 installed at the same time, which is not ideal. You fix this using something called package cleanup, which is a tool part of Yamutils. Of course, you don't want to do this by hand. What we do is we have a small shell script that runs at the beginning of every time we run Chef on the machine. Chef runs every 15 minutes to convert the machine to the steady state. Before Chef runs, we want to make sure the machine is not terribly messed up. We run this package cleanup wrapper that just runs package cleanup in various ways and tries to resolve the transactions back and forth. That's one thing. The other issue you can have is general issues around RPM-DB corruption. This happened especially if you happen to kill RPM in the middle of a transaction. There's a very good chance you'll end up with a database in bad state. There's not really a single recipe to fix this. A lot of time, you have to try various solutions. We wrote a tool called DCRPM that takes care of these and tries a few remediations back and forth. We go to the database until the machine is in a non-terrible state. The package cleanup stuff is not terribly interesting because it's just a shell script. DCRPM is actually somewhat interesting. We're looking at trying to get it open sourced. We just finish it revriting it in a way that should be more maintainable. With these two remediations together, you tend to get a system in a reasonable state unless it's really broken because of, say, hardware issues. Then you can end up with other interesting problems. I think it was 2.31, 2.32. We saw that we had a lot of machines that would come up, they would do the upgrade, and then suddenly basically nothing would really work. You run system CTL and it would fail and the machine would be really sad. It turns out for reasons unclear to this day, we ended up with P1 running 2.34, but the library system elapsed from 2.33. They're dynamically linked and nothing really works at that point. My fix for this, which I'm not proud of but works, is in the sequence of remediations running LDD on the binary for P1, grabbing for stuff missing and forcing every install of all the packages to the right version. Surprisingly these works. It's a fairly curbed solution and it's not great, but these together put something in a state where we can update, we can update system D and other major packages and we do similar things for other packages. The first two are generic. The last thing is specific to system D, but these together put us in a state where we can update system D on the fleet in a reasonable fashion. Then there's the other side of the coin. We have to track what's going on in upstream and upstream exchanges. One change that happened a few releases ago was the change of the build system from Auto2 to Mason. If you're not familiar with this, Mason is this new Python based build system and system D transition from one to the other. They had both supported in one version and then they dropped AutoTools. That's fine. We expected to do a bunch of work. Luckily the Fedora package already did all the work because the Fedora packaging was moved to Mason almost immediately. We had time to basically rebuild our backboard on top of the new packaging and it was mostly okay. The annoying bit there was that Mason on CentOS didn't actually work at all. We had to back to Mason and Ninja and a few other Python things to keep it happy and fed. But that worked well. One good thing, though, we got out of this was some improvements on the Compadlibs. Compadlibs is something that's really old and cranky about that. Unfortunately, we have to deal with. CentOS 7 ships with system D219, I believe, like a fairly old version of system D. That version was before the library split. All the versions of system D shipped with split libraries. Newer versions would just live system D and all the symbols are there. It happens that packages like Apache or Samba, a lot of system packages you really don't want to rebuild, link to the old libraries. If you want to use the new system D, you need the old libraries in one way or another. I'm starting to up support for this because they're different. We used to have these pretty nasty patches to reinstate them and plumb them in the build system. That worked. It was quite a pain to keep it forward ported, but it was fine. With the move to Mason, that patch had to be tossed because it wasn't really workable. We started approaching alternative solutions. The thing we came out with was stealing some of the code from that patch, making it into a standalone project and leveraging the subprojects feature in Mason. Now we have a standalone thing that when it builds, it picks up a copy of the latest system D, builds it, builds the compile libraries and links that are the same and gives you this. This works. We published this and it's available on GitHub on that org in case you need it. We also published the RPMs that we use in the other repo in the RPMs backports. You can also directly get spec files. We're going to keep this mostly up to date and in sync with system D even if you don't really need to because these symbols are so old that they're not really expected to change. Okay. Then there's interesting things that happened because we have a special environment. So when we were old, I said before we were on 2.3.5 and 2.3.4.5. The reason is because we were doing 2.3.4 and it was fine. We would lose a few machines, but you always lose a few machines. That's okay. Then we bumped the shard to do the update to go to like 50%. We lost a lot of machines and that's not great. We started looking at what was happening. You go on these machines and SSH takes five minutes to let you in. You get on the machine and system D just hangs. You do system CTL and it stops. These things are not good. You run a trace against PID 1 and you see it's hanging trying to open a TTY, trying to open TTY 0, which is really not good. We pinned this down that it ran deep on the reexec. It went for 2.3.3 and 2.3.4 and 2.3.4 got stuck trying to open this TTY. After some digging, it turns out our container manager, Tupperware, also happens to fiddle with the TTY. The code in the kernel that deals with this is pretty old. There's probably a race there. We don't know yet what these is. We're still trying to figure it out. The current theory is that there's a user free bug in the line discipline code on the TTY subsystem that is triggered sometimes if you call DmonRia's economic machine that's also running our container manager that happens to do things to TTY 0. In some cases, we have an artificial repo of this, but the actual repo is pretty hard to get. The fix is that Tupperware probably should not fiddle with TTYs at all. That's like old code we should not have. We can fix that if we just make use of TTY. There's a bug in the kernel that our folks are trying to figure out. That's an example of something you're unlikely to get if you have a few machines or a lot of machines. When you have hundreds of thousands of machines, these kinds of crazy things unfortunately come up. Let's talk about interesting things we're doing with system D. The first thing is resource management. We're interested in resource management because we run a lot of things on our machines. We want to be able to control and make sure the machine is doing what it's supposed to do. The process that is supposed to run that has actual work, say the web server, has the resources it needs and doesn't get content from random auxiliary services, say the thing that controls power on the machine. We do this using SIGURP 2. I'm not going to talk about SIGURP 2 because my co-worker Chris is going to do a talk on this tomorrow. If you're interested in SIGURP 2, you should attend that. For the purpose of this SIGURP 2 is a kernel API that lets you set resource limits on processes. By resource limits, I mean things like memory, CPU, and IO. System D leverages this and lets you apply limits to your processes and to your services. It also lets you bucket your services and partition them in slices and apply limits to the slices as a whole. We use this and we use this to bucket the services and apply these limits. That's part of the picture. That's the implementation. You also need a way to tell what's going on. We do that with a small demo that runs on the boxes that picks metrics from SIGURP and shows them in our monitoring. We can get data around the limit is set here and the current memory usage is there. This is actually working. This is thrashing. There's a problem here. The other thing we do is that we have an API in Chef that lets people define and set the changes without having to go on individual hosts and do system CTL edit and change things. The way we do this is that we have an API in Chef and this translates to system D override files. It applies the changes and you can change the memory limit for this thing here or move the service from this bucket to this other bucket. This is something that's very much in progress and we're still learning what the right things to do are. This is the general hierarchy we're using right now where there's a system bucket and that's stuff that runs on the box that needs to run on the box but it's not critical to what the box is supposed to do. There's workload which is what the box is supposed to do, say MySQL or HHVM, the web server or whatever. Then we have another bucket that's called TBD inventively. That's meant for stuff that we need to run unrestricted, say the hardware folks have to do a stress test on the box and that doesn't need to be limited because it's a stress test. The initial idea was okay, we'll cap system to 4 gigabytes of RAM, we'll leave workload unlimited, it will be great. It's really not. First because our working set is way larger than 4 gigs so if you do that, the machine immediately dies. Also because it turns out it's not quite easy to tell what a workload is. In a lot of cases, say your workload is the web server but the web server gets configuration data from another demon. If that demon is in system and gets kept and becomes really slow, the web server is very angry so you need to figure something out there. The way we're addressing that is by making a sub bucket under workload and moving some of these demons there so we can control them better. The other thing we're doing is shifting the focus from doing memory limits and hard limiting in general to doing protection. Doing things like system DL is like memory low or memory.low in security problems that give you protection. Guarantee that this service will have at least this amount of memory available rather than a hard limit to this other service. This isn't always the solution and all of this stuff is hard. It requires you to understand in fair detail how your service works, what the dependencies are, how memory management in the kernel works and how these apply together with C groups. There's a lot of work to do to make this more simple and easier and we hope to be able to get to the point where we can give people a tool that they can run and give them an idea of, okay, these are sane defaults for your service based on what we see. Start with that and try. Another thing we are interested in doing is service monitoring. That stands from the fact that, as I said before, we had like five or six ways to do service monitoring, service management for bare metal, but they were all like fairly blind. We didn't have a good way to pull metrics from services. But systemD does because systemD knows a lot of things about you because it's supervising you directly. It stores them and crucially it exposes them over debuts. You get properties like all the timestamp properties. When did I start? How long have I been running? When was I restarted the last time? 235 added a thing that's really awesome and restarts. That's by far the main thing that people ask me that they would like to know. Is my service flapping? People are really interested in that because if your service is flapping, it's very likely that something is wrong. These kind of things are useful and they're easy to get out of systemD. SystemD also gives you status events where you can hook to the service on the bus and send your notification whenever the service changes. It goes from inactive to active and so on and so forth. It gives you basically a view of the state machine and what's going on. The downside of this is that these are exposed to debuts. You need to talk to debuts to get them out. We started looking at ways to do this. By the way, you don't need to talk to debuts. You can use systemCTL show and it will dump them all. The problem of doing that is that you don't want to run systemCTL show in a tight loop on every machine because that's going to take up a lot of resources and it's fairly brittle. You really want a programmatic way to do this. If you look online, there's a few toy projects I'd say on GitHub that already do this to some extent and all of them use either Python or libDebuss. Which would be great except after two weeks trying to get that to build in our internal build system, we essentially gave up. I'm still not quite sure what was going on there. It's definitely not an issue with debuts itself. It's an issue with how we do things at Facebook. The long story short, it was not resulting in minor that would run. We started looking at alternatives. Obviously, alternatives would be using SDBus, which is included in systemD for a while. The downside there is that SDBus is a plain C API. There are not really any wrappers for it. I really didn't want to write this in C because then I would have to maintain it. Most people in inFread Facebook are like Python people and not C people. There's also C people, but they're harder to come by. I don't know so far a prototype because I don't quite know if this would work. Writing a prototype in C is not a great idea, at least from my standpoint. I started looking at things and I found that the chorus folks had also bindings in Go. These bindings don't rely on libDebuss. They're standalone and they work. I don't know Go, but I figured I could pick it up. I did that and I wrote a POC of these in Go. This is basically a small demon that runs on a box. It hooks using the DBS API to systemD. It pulls for units, properties, and uses subscriptions to get events. Then it collects this data internally, messages it a bit, and then it should sit out to a few of our monitoring systems so we can get pretty pictures and data and things like that. I was fine for a POC. I was a bit uneasy of doing these in a language I didn't know very well and basically as a hack. Then Alvaro came along and Alvaro works on Instagram and they have the same problem because they use systemD to manage their services. He started looking at these and Alvaro is a better coder than I am and he knows how to use Syson. He wrote a wrapper using Syson on top of Sdbus. If you don't know about Syson, it's this magical Python thing that lets you call a C API from Python and expose what you get back in terms of Python objects. Using this system, we can talk to systemD through Dbus and talk to it and we get to interact with real Python objects that translate that internally to Dbus calls, which is kind of neat, especially for prototyping because you get a nice wrapper where you can directly focus things and get properties and see if it makes sense. We're likely going to use this and rewrite my prototype on this. We're going to open source this. I actually have the repo mostly ready, so it should be out in a couple of weeks. SystemD mount itself, we have to figure out how to release it, but we expect to release it at some point ideally this year. Hopefully, people will find this useful and I would love to get feedback on whether something like this could be useful or if you have ideas of how you deal with this problem or ways we could do this better. All right. Now, let's move on to interesting stories and case studies. By far, the main thing that still causes random issues that we don't quite understand is Dbus in production. The problem with Dbus is that when the demon gets sad or angry, systemD also gets sad or angry and the problems with the Dbus cannot really be resolved without rebooting the machine because Dbus doesn't really support takeover in place. Most of the time when bad things happen, your only resort goes booting the machine, which is fine for a laptop but not fine for a server. The other problem is that if you get the system to a state where systemD is still mostly working but the connection is severed, so system CTL is not working, either if it's hanging or if our product failing, like connection to bus failed, for us, that manifests a chef failing on the box because chef cannot manage services anymore, so it will yell at you, which is a problem because then people get all the chef failures but they're not really actionable for them because they see, oh, what is this Dbus stuff? The other thing we found is that it's actually surprisingly easy to dust Dbus and put it in a bad state. We had a co-worker who was doing tests with user services and he wrote a thing that when you would log in on the box, your batch RC will start a user service that will do some stuff and then start other things. That semi-reliably managed to crash it, not on every machine on like a good chunk of machines. Unfortunately, a lot of these problems are hard to pin down and like I would love to be able to get like, okay, here's a repro and file a bug upstream, but I don't have that. I have on X percent of machines, sometimes this happens and it's really fucking hard to find it. So we are right now we're mitigating this by basically rebooting machines when necessary and trying to keep Dbus up to date and get bug fixes. We're also looking at alternatives. There's this Dbus broker project that's an initial replacement of the Dbus demon coming up to the bus one project that we started looking at and testing on a small number of machines and it looks promising, but I don't have any hard data on whether it works well at our scale, but I'm definitely interested to see how this fares and if it's going to behave in a more reliable way or in an easier way to remediate in case of issues. Let's move on. RPM macros, yes. So we, this is one that's more like people problem than technological problems. So we, a lot of the stuff we build at Facebook are not, they're not like large complicated tools. They're like one binary. You have to ship one binary, one system DC unit and that's it. So we have a tool that you feed it, your build config and it spits out an RPM, it makes your spec file, it takes your system service at the right macros and here's your RPM, which is fine. The macros that you get by default by Fedora restart your service on package update, which is also fine, except a lot of people are used to the old design pattern of front send five, front send six, when you do a chef, you do package upgrade and then notify your service and ask chef to restart it. If you're doing that and you're also static on upgrade, you're starting twice, which doesn't seem to be a bad thing, except maybe your service takes quite a while to start or takes up a lot of resources when it's starting or starts talking to the world. So this fix for this was really easy. Like you add an object in the tool that you, so you can disable the default restart behavior and you're done. The trick was like socializing this and understanding that this was actually the problem and figuring out how to propagate it. And there's a few, there's a few things like this that I'll have later that are not necessarily technical issues, but more like getting things to be better understood. Another area of interest for us is the journal and logging. So the journal for us is set up to only log in memory, log in a small in memory buffer and then feed everything to syslog. And we do this because we have a ton of infrastructure based on syslog, our own security. So we need syslog to keep going. And people are used to syslog. People want to be able to tell the chef, see their stuff. On the other hand, we found pretty quickly that when people actually start understanding what the journal is and start playing with it, they really like it. People start using journal CTL and like, oh, I can filter by things. This actually works. So people started asking us, okay, can I use the journal? Because I'd like to get more data than just that, the shit-y 10 megabyte buffer. You can, of course, you can enlarge the buffer, you can make it store on disk. When you do that, you end up with a double writing problem. Because you're writing both syslog and the journal. They're both write to disk. And some of our tools are really chatty. So you can end up writing like megabytes or more per second, which is not ideal if you're IO constrained. And it's also an all or nothing proposition. Because you either do this system-wide or you don't do it at all. So what we'd really like to get is some way to do this on a per unit basis. So people could say, for these applications, I want the journal data to have these buffers and go on disk and be persistent for these to be transient. But we haven't found a good way to do that. So for now, what we're telling our customers to do is either take the hit of the double writing or set things in such a way that they silence some of the stuff that ends up ending up in syslog. But that causes other problems. So it's not really a tenable solution in the long run. Now another fun, for some value of fun problem we had was loops. So I didn't know this. I discovered this when I hit this problem. But if there's a loop, system-D breaks the loop between, a dependency loop between units, it will break it by removing an arbitrary unit from the loop. And it tells you, it puts a log line like that. And we found this out because we would machine and it would come up and it would be missing ten files and directories that were supposed to be created by system-D temp files. And then Chef would fail because this directory that's supposed to be here is not here. And it's like, what the hell? So you look at the logs and you see that line, which is, okay, that's interesting. Why? The way we debug this was using system-D analyze. And system-D analyze gives you both book chart style plots, like time zero, the system booth, time one, system-D starts, time two, the service starts. And it gives you dependency graphs, so nice, like octopus style graphs with dependencies. That's completely useless for us because if you plot it, you end up like the size of this roaming graph paper to get it out. But you can make it plot as small subset. And that part is useful. Because in this case, that line gave us enough clue to know that, like, SMC proxy, which is a thing we ran, was responsible somehow. So we started using analyze from there, and we figured that it has something to do with mouse, found that somebody added to FSTAB an entry to ask to mount this network file system and make it require SMC proxy, which is fine, but if you don't say it was actually a network file system, so this ended up ordered in an order in conflict because it's like, you need the file system, which needs the service, which needs the network, but the network needs this, and this doesn't work. So fixing this is adding net dev to the FSTAB entry and we're done. But the process of finding it was a bit more interesting than that. And that's one thing that we also, like, I wasn't personally aware of this was a thing, so it took a while to dig through the logs and see that this line was a problem and this is what was going on here. On a somewhat similar vein, another fun problem we had was around training units and system here on. So we have these machines that do builds and testing for mobile apps, and the way they do that is that there's a lot of phones plugged in USB, essentially. And then we run stuff that does things on the phone. And the way the team that manages does it, they use system Iran to start these processes and they run these using device allow policy so they only talk to the thing they're supposed to talk. And Iran is a lot because we do a lot of tests, we have a lot of phones. And every time it just runs, it enqueues a unit, runs a unit, does this thing, disposes of the unit, all good. But like, USB cables aren't great, sometimes stuff breaks, so this can also fail. When unit fails, system D lets them stick around, they stick around in like failed, and they're there. Not a problem if it's like one, two, three, five. Wait a bit. If you end up with like 10,000 units, there's something in PID 1 that reads this and ends up taking a shit ton of CPU. Like we saw with 10K failed units, this goes around 50% CPU usage on the box. With 30K, it goes around 100%, and that's when people started noticing because it was kind of interesting. So the fix is putting a cron job that caused reset failed. Yes. Yes, that's also what I told people to try. But for now, the way this was fixed was because people wanted to get a sense of how bad the problem was. So what they did was they added a monitoring counter that would tell them how many failed units are there, and then they added their remediation if it's past the threshold called reset failed. But yeah, ignoring the result is likely the better option. I need to check if there's a way to do that with system Iran directly or if they need to write a unit. I just thought about that we maybe want to improve system d run so that system d run can still extract the failed stuff, but then system d run automatically makes sure that the thing goes away. Oh, yeah, that would be awesome. I think that would take care of the case here. Yeah, because in this case, it's failures that are not really actionable because if it's failed, it's not because of something we can control. On to more fun, this time, C group related. If you have a service by default, when you terminate it, it kills everything in the C group. You can change this. You can tell it don't kill everything in the C group, only kill the main process and leave everything else to end off by itself. We try to discourage people from doing, but people are stubborn and sometimes do their own thing. If you combine this and when that happens, if you end up with leftover processes, the C group stays around because there are services in the C group, of course, that's processing in the C group. What happens if you then reload, you change the configuration of that service that before was in this slice, you make it go to this other slice and then reload system D. Ideally, you would expect it would apply the change. Turns out it doesn't if there's already a C group running with these processes in, which makes sense. Fixing this is easy. You kill everything or you change it to the control group. This was kind of surprising. It caused a bit of customization for people because they really wanted to use the kill mode process for reasons not particularly interesting here. The fact that the old unit was sticking around wasn't quite as evident. You basically find out because your own system city health status and it shows a stopped, but it also shows the C group is still there. Then you poke in the C group and you see it is there and there's processes running. Then the only result is just to kill everything. Yes. Yes. In this case, yes. The question was is there a reason if we're not using kill mode mix? In this case, the reason was that the thing we were starting was not actually the process. It was a thing that was spawned off a bunch of children and do hand-wavy supervision of these children and they wanted that they could replace the supervisor but not the children. It was something like that. The solution was redesigned is to not do that because you shouldn't do supervision in your application. I don't know if we tried mix specifically, but I remember the folks that had this system were fairly adamant. They wanted to use kill mode process the way that it designed. For the specific thing, there's not much we can do because the current behavior actually makes sense. It's not a problem. It's just something that's not evident at all. Finally, for the most stupid and irritating bug I found, the system.us is logic that translates the device names and paths to unit names and uses shell control characters. Slash, dev, slash something becomes dev dash something and then escape. Slash, escape is also a shell control character. If you take that and write system CTL status something and shout out to that without escaping it, you're going to have a bad time. Chef did that, which made it interesting when we were trying to write a cookbook to manage swap devices in some special way and it would fail because of our weird shell escape stuff. We fixed it. The fix was really trivial and it was basically make sure we call shell escape. It was somewhat embarrassing because it took a while to figure out that was the actual thing that was going on and we were like, we should have cut this before. As part of this, we wrote, we have a cookbook that's open source. We open source all of our cookbooks there on GitHub. We had a small wrapper to wrap the system.uscape function to get a unit name from path so you don't have to shout out from it. This closes the gallery of horrors. I want to spend a few minutes to talk about how you do these things that try to not make people too angry. If you look at things from the point of view of the service owner, the application owner, you kind of like your system to be stable and sort of static. On the other end, you want to get new features and you want to know that your system is stable but it's also getting updates. The approach we found that works reasonably well for us is trying to communicate things as much as possible to people and make sure people understand why you're doing this. Announcing core package updates widely and announcing changes that are happening and why they're happening and what's going on with them and giving people a venue to comment on whether this feature is interesting. I like to know more. I like to use it. This is going to cause me trouble. Doing these updates is a way that people can react to them and give you early feedback. When we did one of the system version upgrades, it happened to cause issues to the same folks that did the phone stuff because they were doing other secret things and it was fine. We could spin them to the old version for now, resolve it and then move on. The other thing we found is that documentation is critical. While upstream documentation is great and very detailed, it is not great if you don't know what you're doing and you're starting from the place of wanting to learn. We started writing internally snippets of like, okay, this is the suggested way to do things, like a basic unit to do these or that. We found that doing this with the custom use case in mind works better. We also encourage people to follow what's going on with upstream because we don't have all the answers. People can read the source code because this is an open source project. They can go on the mailing list. They can look at things. Finally, we found a talking and giving tech talks internally, both as a company and both going to a team and sitting at a team meeting and talk to them for an hour about what they're doing, what is the system they're saying, what you can do with it. It goes a long way making people happier and more amenable to using this and leveraging it and then actually liking it and building cool things with it. With this, thank you very much. I'm happy to take any questions you might have. Minor clarification. He said all of our cookbooks are open source. They are not and our lawyers will get angry. Some of our cookbooks are open source. Some of our core cookbooks. Yes. Thank you, Bill. I was curious. You were mentioning one of the things that you're having issues with in terms of Chef integrating with system D is with system control and things hanging. Is that because of Damon reloads or something else with the lockup? Aside from D bus issues. So Chef shells out to system CTL for basically every interaction it does with system D. So if I have a unit that says I want this service to be started, I want this service to be enabled, I'm adding a new unit or I'm changing a unit I need to reload system D. All of these things end up calling system CTL. So he sent any of these interactions if the system is unhealthy and the system returns an error or hangs, we'll lead to Chef failing. What I'm curious is if you know what the root cause is on the system D, say, PID one side of why those things are hanging. We know it sometimes. The kernel TTY raises one example where we found out what it was. In other cases, we don't. For some we pin it down to D bus. Most of the ones that are unknown, we're fairly sure it's something that we did the bus columns, but we haven't managed to report in a way that we can actually poke at it more and get some actual data out of it. Yes. Regarding the issues with RPM database and inconsistent packages, have you looked into deploying using atomic or OS tree or similar stuff? No. I'll tell you why. The main thing is that for system packages specifically, we want to try to stay as close as possible to how upstream is built and how upstream works because it makes it a lot easier to keep things not too alien so people can understand it and when you interact with people outside. We might well look at that in the future, but I think in the short midterm, I think we're going to stick with RPM and just deal with the things. The other thing we're doing is trying to make RPM better and we're actually engaging with the RPM developers and the YAM developers there. Hopefully we'll have something there soon. Well, my time is up. Thank you very much, folks. Thank you.
Facebook. I'll be talking about what we've been doing with system D for the past year. Before we machines at Facebook running. We maintain the understand is Dbus in production. The problem of infrastructure based on syslog, our own machine and it would come up and it would be a monitoring counter that would tell them how