OpenBSD rc.d(8)
Jacoutot, Antoine
OpenBSD has always used the traditional static BSD initialization script: /etc/rc. While dependable, it did not allow for easy integration with monitoring, configuration management software and/or any kind of tools requiring automated service handling. rc.d(8) was developed to abstract service management while pertaining the existing behavior like predictive and sequential start-up ordering (dependency-less). The way OpenBSD boots hasn't changed much since its inception. This presentation will describe the OpenBSD rc.d(8) framework and rc.subr(8) daemon control routines. While it resembles other implementations, it was written from scratch to match the project objectives (simple, ordered, non-intrusive). I will describe the internals of rc.subr and talk about the implications that it had on the traditional BSD start-up sequence. I will detail our requirements as well as the existing implementations and alternatives and explain why it was decided to write one from scratch. I will show how we managed to plug ourselves into the existent without having to transform it. I will describe how to use the rc.d control scripts and how start-up scripts look like. I will then introduce rcctl(8): an all-in-one utility for managing rc(8) daemons and services and look at how it helped orchestration and configuration management tools to work on OpenBSD (feature wise it is kind of a merge between the service(8) and chkconfig(8) utilities and a sysconfig editor as found in Red Hat).
 So first of all, thank you very much everyone for attending to this talk. Good morning. My name is Antoine Jacoutot and I'm going to talk about our RCD implementation on Open BSD today. I just want to apologize if I have a weird voice from time to time. I have caught a cold for a couple of days. If it's weird, that would be why. Or maybe it's because I mentioned system D. Also make some weird stuff in my voice. Anyway, a little bit by myself. I've been an Open BSD developer for about a decade now. I'm known as Jacoutot or AJA for the lazy ones. I work mostly on things that make my life easier when I put Open BSD in the real world. So I'm more on the assessment engineering side of things more than a regular developer. So these are some of the things that I've done. Maintain way too many ports. Among them the GNOME desktop for which I'm a foundation member and I maintain the French Open BSD mirror. I just want to say a quick thank you to my fellow developer who I developed the RCD system with, Robert Nagy and Ingo Schwartzy. Robert and I came up with the original implementation and Ingo gave us tremendous help in actually having a good shell foo. So the stuff we're going to talk about today, we're going to talk a little bit about the historical and current boot process of Open BSD. We're going to have a quick look at the alternative that we looked at when we first started the implementation, how to use RCD, the internals of RCD and I will introduce a small tool called RC control which is a high level tool to handle all the RC configuration and demons. So all in all we'll see if RCD had any implication on the traditional system that we used before it. Just a word of advice disclaimer. If you guys came up for a highly technical talk then I'm afraid you're going to be disappointed. RCD on Open BSD is just one very, very tiny small shell script. So it's not really interesting by itself. What's interesting is the story behind it and what it does. So that was the trigger why I got fed up about and wanted to create something that was a way to handle Jim in a generic way. A few years ago before RCD I was working at deploying Open BSD based infrastructure all over the world and especially in a area where Linux knowledge was very low, let alone BSD. And I just got tired because the weekend on this region are not the same. I got tired to be working up on Sundays at 6 a.m. by my ticket system sending an SMS that someone wanted to reload something and was not able to or wanted to start something and couldn't find ETC in it deep. So yeah, that was the trigger for it. Of course, the main reason after that that we wanted a generic way to handle services and demons was because we were using a lot of automation tools. So automatically enabling services or starting services if you don't have a generic interface is very hard. You really don't want to have a huge amount of Ruby code editing your RC Conflocal or anything. So yeah. So the way Open BSD boots hasn't changed much since its initial inception. It's actually pretty much how it still boots now. In a very general way you have the boot order that will fire up the kernel and the kernel will run ETC RC using SH. So yes, it's old but it's still very dependable. It's predictive, it's sequential, there is no service dependencies, et cetera. Basically what it does is it's a big RC script that runs things sequentially. To be able to control what we want or do not want to start, we have two configuration files, the ETC RC Conf which contains the operating system default configuration and we have ETC RC Conflocal where you can put the overrides compared to RC Conf. The way that we start a demon or service. Okay. I'm going to explain what we mean by demon or service in RC. First a demon is the regular demon that will fire up fork and run in the background. What a service is more like a kernel facility like I want to start PF or I want to enable the counting, et cetera. Anything that is not basically running a demon. So to start a demon, you edit to RC Conflocal and just put the demon name underscore flags equals. If you say anything but no, then the demon will be enabled. And service is just yes, no. So that's how it works. It worked before RCD and that is how it also works now. So nothing changed in that regard. So the requirement, as I said, nothing changed and nothing needed to change. We knew that convincing the open BSD crowd to have an RCD system would be quite challenging. So we didn't want to disrupt anything, really. Actually we even lied a bit. I will talk about that a bit later. So yeah. So the current paradigm couldn't change. We have to preserve existing behavior. So basically what we did is that we plugged on our RCD system on top of the traditional ATC RC initialization. There is one important thing is that we only wanted our RCD to handle demons. We didn't want to handle like starting a network or mounting filesystem or things like that. These were completely out of scope from the very beginning. And of course we wanted something small, simple, robust because as I said I'm not a developer, I'm stupid so I'm only able to do very simple things. And of course easily debugable. So at the time we started to look at different alternatives, not all RCD based but at least initialization based. Yeah. Well you don't see system here. The reason is not that obvious. At the time system was actually in its infancy. I think I'm not sure they even had a release yet. And anyway we're not looking to replace any D-chrone, C-slug, and what else. So we had a quick look at the SMF from Solaris, and Lumos, and Long D from MacOS. These are actually very nice but very complex system. It was way too complex for me. I'm stupid. But it was nice to have a look at it. They actually fix some real-world problem but these were out of scope for what we wanted to do. Then we had a deep loop at OpenRC which is actually quite sane RCD implementation on Linux. That's the one that's used by Gentle Linux and Alpine if I remember correctly. The problem is that it was way, way, way different than how things were done on OpenBSD. Of course it handled mounting files and setting TTYs and blah, blah, blah. So all that seemed that we didn't want to do. The init script were kind of too big for what we wanted to do. We wanted to try to have init script that were really, really small. And it introduced a whole bunch of new commands which was something that we really didn't want to have. And then I had a quick look at runit and demon tools. Demon tools felt just too weird to me. And runit was nice but it was a little bit like OpenRC introducing way too many commands. One of the most interesting Linux RCD implementation was the one from Slackware. But this one was actually too static for init. It works basically the same way OpenBSD used to work. You have a big ETC RC.M that will run sequentially the init script that you give it. The problem is that if you, I don't know, install the package that installed an init script then you have to manually add that to RC.M or RC.local or whatever. And there is no factorization between the RC script so they're all huge. And of course the obvious choice would have been free BSD or net BSD RCD and RC order. That is actually what we initially wanted to bring into OpenBSD. So we looked at it. The reason we didn't go for it is that there was a lot of small differences in the way we were doing things. Like enabling a demon was demon and the score enable equals yes. That's something we didn't do. That's something we didn't want to do because again it was forbidden for button to change anything. It was also used to do other stuff than just handle demons. And RC order provided dynamic startup of the script. I mean dependency. That's something we didn't want to have either. So basically if we had gone for the free BSD and net BSD RCD it would have meant that we would have to strip it quite a bit so it was not really worth it. So that's the reason we just went for our own implementation. I really want to enhance the fact that we did not want to start from scratch. It just happened because we couldn't find something that was good for our needs. So really it's not a non-invented here syndrome. It's really we looked at a lot of different ways to do it. So on and on we started with something small that needed to be targeted to the OS requirement. We didn't want to create a process supervisor that is not the goal of RCD from our point of view. Process supervisors are great. There is some great real use case for that but it was not in the scope of RCD. So no crazy stuff like even driven or socket activated demons or things like that. No parallelization either. We don't really care about the boot time of the machine. In these days of an age you have half of your machines or the end on the cloud somewhere. These boot super fast whether you have parallelized in it or whatever. And otherwise you just run on bare metal and then you have to wait 15 minutes for the post to finish anyway. And as I mentioned no automatic startup ordering. The reason we didn't want to have service dependencies and automatic ordering is that it introduced complexity and that was not really an all point of view necessary. When I put a system in production I usually put a system production that does one or two things. I'm not expecting to put something that will run 20 different services so I don't need to actually order things for 20 different demons. So basically manually adding two scripts in the order that you want to run them was enough for us. So the initial landing, initial implementation was done in October 2010. So it consisted of RCD.subber which contained all the basic functions to the RCD framework. And of course the RCD script coming from packages only. That time and that's where we lied, we said that it was designed for ports only. That was the only way that we could actually bring something into the base system without being knocked out. So of course base was the ultimate goal but we had to come up with a good proof of concept that what we had developed was actually useful and well done. So as I mentioned, I am not a developer so I didn't want to create a whole bunch of new commands to handle demons so we basically kept using the tool that we were already using manually. So namely kill or pkill or pgrep, et cetera, the traditional tool that are already there to manage processes. We did not want to rely on PID files because well you always have the chance to have like a dangling PID file that's not here or there or you have also a race with a PID file or when you have a demon you can change the PID so if you have to handle that and things like that. So it was not really, I mean we didn't want to go that way. So as I mentioned no start, stop demon or anything that would actually run the demon for you and as long as 95% of the ecosystem would be properly handled by RCD, well that was good enough for us. For the gigantic weird application that needs all kinds of stuff to properly start or stop and it's just better to write a home script for that. And for better or worse we did it in shell because the entire unit system was already in shell. So at that time the RCD script were called from RC local. RC local is a file that's typically modified by the admin of the machine so it has nothing to do with the base system and that's where you traditionally use to manually add the demon that you wanted to start. So it looks like a good fit to actually start the RCD script from there. And that's basically the only amount of code that we added to this file which is completely stupid. It's just a Y loop and if there is a script and it's executable then you just start it. So it was really, really small. We couldn't do a lot more than stop, start, reload but that was enough for a good proof of concept. So at that time RC.subber which provides all subroutines to RCD was 54 line of shell code. So to my point of view that's pretty small. I committed it when we committed it then we got this kind of answer. So that's what I mentioned earlier. We knew it would be a challenge. 54 line of shell code as a bloated interface. Of course we also had very nice review from people actually using OpenBSD in the real world and really needed some kind of RCD system. So yeah, that was amusing but that actually proved that we were right to be very, very cautious. And one release after that, we were allowed to handle base system demons. So why did everyone change its mind about moving RCD to base and not just handling ports and steam and? That's true. But I'm a liar. I said so already. So the main reason was that the processes were not started in isolation. Basically they inherited all whatever ETC RC was doing before, all the environment, all the stuff. So that was very bad and of course that can lead to unexpected things. And the RCD were using SU for environment sanitation. To give you an example of how it used to work, if you run the first command versus the second one, the first command will output this in your PHP info. So that is a lot of information I don't want to see in my web server. If you're like some people who also love to export like AWS private keys in their environment, then they will appear there as well. So yeah, that's what basically convinced all the rest of the OpenBSD crowd to go for RCD. A little word about the startup sequence, I've been talking about ETC RC all the time. So let's see a little bit what it does exactly. And let's see where RCD is actually used within that script. So ETC RC basically enabled your swap at boot time. It will run FSCK on the file system. It will mount your root directory, set up TTY flags, keyboard encoding, blah, blah, blah. It will start the network. Network is not handled by RCD. It's a script on itself called ETC net start. And then in the middle of the script, then RCD will start starting demons. What we call the early demons like syslogd, pflog, and tpd. Then it will run the RPC demon, port map, yp, blah, blah, blah. And then we're back to doing some sequential stuff like enabling quotas, cleaning TMP, setting up the default secure level, blah, blah, blah. And then RCD again is back to start all the rest of the demons like SSH, RelayD, GHCP, all that stuff. So basically RCD in itself is a very, very small subset of the boot sequence. What it is today, it's still small. RC.subbers only 224 line of shell code. We were able to remove 150 line of code from ETC RC itself. The only small code that we had to add to it was just sourcing RC subbers to get the functions. We created a very small function called startDemon. And that's what ETC RC will use to start the RC descript. Basically, you pass it all the RC descript that you want to run. And if the script is actually enabled in RCConf, then it will start it. If not, then it will not do anything. Then we have a small while to also start the package script. I will talk about that a bit later because there is a different way between starting the base system demons and the external packages RC descript. So on and on, it was a big fit again for only 70 line of shell code. So how was it used? It's worked tried to keep it very simple. We have four plus one action available. Well we have five action, but the restart action really is start and stop. So we have a start, obviously, with start to demon according to your configuration, with the flags you pass it, it can run as a different user under a different login class, et cetera. We have the stop action, which basically send a sick term to the demon. Reload will send a sick up. And we have the check action, which will pick up for the process and tell you whether it's running or not. These are the very basic. We'll see exactly in details how it's done. Because matching demons is kind of hard. So all of these actions, they need to be run as a privileged user except for the check action. Although it can happen that the check action also need privilege, right? We'll see about that a bit later. All these actions are fully configurable and overwriteable. These are only default. Like when I say to stop action, send a sick term to the process, you can completely override that in the RCD script itself. Same thing for the reload action. Because kill minus up doesn't always do what you think it does. And since it was the main user interface, we really tried to keep as less action as possible. We didn't want to do like four start or whatever. So that's how a minimal RCD script looks like. You have the she-bank and then the demon variable, which is the only mandatory variable, which is a path to the demon you want to run. Then we are sourcing RC.subber and we are passing it whatever action we want. Start, stop, whatever. It is small, but it is actually, I would say, like one half of the RCD script in OpenBSD are just like that. So that's enough for most of the stuff. Of course, for packages, it's a bit different because we have some weird stuff in there. Alongside the actions, we have two optional flags that we can pass. Minus D for debug mode. By default, RCD will redirect to dev-null the output of the script. So in minus D mode, it will output everything on the console. And we have minus F, which is similar to one start or four start. I don't remember which one. I mean, it's used to start a demon when that particular demon is not enabled. It's an op for the demon coming from the package script. The reason for that is that for all base system demons, we already have default in RCConf that are set to know or that are set to whatever because by default, this demon must run. The system has no knowledge of a default configuration for a package script because a package script is just an RCD script that's installed under ETCRCD. So if you run ETCRCD, my package script start, it will start because there is no default set to know. But it's only used for forcing a start of base system demons. So the way we enable demons are the traditional way for the base system demons. So that didn't change. It's just the foobar underscore flags. For the package script, we also use foobar underscore flags to actually append flags to the demon when we want to start. But to actually start a demon, so to tell the OS that we want to start that demon, we have to append it to a variable called package script, which is located in RC.conf. All the script that are listed in the package script variable will be run in order on start and a reverse order on stop. So we have several RCD variables available in RCD script. All these variables can be overridden in RC.conf.local. We have, first one is the demon class, which is the BSD login class that the demon will run under in default to demon, which is a preexisting class in login.conf on openBSD. That's pretty convenient. When we started working on RCD, we wanted to find a nice way to increase the niceness or inject environment variable in the process or something like that. And so at first, we're thinking about putting demon underscore, nice demon underscore, and stuff like that. And we just realized we actually had a facility for that. Login.conf is perfect. The demons are starting with SU, and SU can pass a BSD login class to the process that it will run. So nothing to develop here. It was already there. So the well-known now demon flags, which default to no or empty in RC.conf. So these are the flags that you want to pass to your demon on startup. So I don't know, minus F, ATC, blah, blah, blah. Then we have a demon R table, which allows to run the demon in a particular routing domain. Routing tables allows multiple lookup tables for routes, and they're usually used for police-based routing. I'm not a network guy, so... Then we have a demon timeout. We realized at the very beginning of RCD that if you make a typo in your demon flag, like for example, you forget to pass the minus D to demonize the demon, then you can fuck up your startup sequence quite badly. So we introduced a default timeout. If the process hasn't returned anything after 30 seconds, then we consider it failed, and we keep... We put it in the background and keep running. It's also used for the opposite. There's quite a few processes, like Squid or MySQL, when you have a huge database that can take actually a long time to stop, especially Squid actually on OpenBSD. So sometimes when you want to properly stop the demon, that you may need to increase that timeout. Otherwise, it will be a forced kill. And the last variable we have is the demon user. So it defaults to route, and that's the user that the demon will run under. So that is what SU will use as a user. So all these variables, they are overwriteable, obviously, by the RCD script itself, which usually contains the default for that particular demon. It is overwriteable by RCConf, which is the default OS configuration, and it is overwriteable by STCRCconf local, which is the locally modified configuration. The way that you override a particular variable, so here as an example for flags, is that instead of having demons flags equal, you substitute the demon part with the RCD script name. So the first line here, demon flags, is what's under the NetEsnMPD RCD script. So by default, it will start with minus you, blah, blah, blah. If you want to please Henning and remove IPv6 support, then you edit RCConf local and you will do IPv6, and minus A is for looking support or I don't remember. So that's the way to override any variable. Underscore class, underscore flags, underscore users, et cetera. Actually a light again. Yes. The way you override the demon class is a bit different. Here we'll not be using environment variable. RCD will automatically check if there is a login class that has the same name as your RCD script. So if you have a login class that's called asshgd, for example, or netesonMPD, and that example, it will use that from login.conf. If there is no matching, then it will fall back to the default login class, which is demon. So that was a simple way to add the BSD login class support. So here's an example of an RCConf local. So APMD flags equal minus A. APMD being a base system demon means that we will be starting it with minus A. We will be starting hotplug with no flags, and we'll pass some flags to the same D demon. But same D is an external package. So just putting that line here will not start it. We have to put it in here as well in the package script variable, and that's how the system knows that it has to run that script. And we don't want to run NTPD for whatever reason. Yes, it's bloated. So there are also a few special cases that we have to take into consideration. We have a notion of what we call meta RCD script. Those aren't really RCD scripts. They're just regular shell scripts that basically run other RCD scripts in the order that you pass them. It's used typically for, yeah, like Samba, you can start SMBD and NNBD, but there are also applications like Bacula where you want to run the director and the client on the same machine and etc. Or there's a RAFRA group where it has like seven or eight different demons. You don't want to have to enter all of them. You just have a meta script that will run all them in the sequence. Another special case, and this one is hard, is having multiple instances of the same demon. It works most of the time, but it depends on a lot of different details. The way we do it right now is that we link the original RCD script and we rename it to something. The reason we're linking and not copying is that package script are part of the base system, so if they get updated, then at least your link will be updated as well. Sorry. So that works for most of the demon, but there's a lot of package out there that changed their process like using setproc title or stuff like that. And then it's very hard to actually match the correct one in the process list. If a demon changed like some sample, you pass foo bar, minus VEP, blah, blah, blah, foo foo, and setproc title to just foo bar, then if you start two of those, there is no way that RCD will know which one it has to stop. That's because, again, we're not a process supervisor, so that's a drawback of the implementation. Sorry? No. Can you define a secret script? Yes. Well, I'm actually going to talk about that in a few. So let's have a look now at the internal of RCD, so RC.subber, that's the entry point of the whole framework. That's where we define the default function that will be used by all RCD script. All this function can be overwritten by the RCD script, as I mentioned earlier. If you don't want to send the hub to reload your process, but because that doesn't work and needs some other procedure, then that's where you're going to do it in the RCD script. Sorry. So the first function that RC.subber provides is RC on the score start. What it does? It's RC exact, demon, demon flags, and the bg variable. RC exact can be two things. It's basically the SU line where you pass it to a demon class, then the shell, and the user you want to run the command as, and then the command itself. If we define a different routing table, then we prepend a call to root minus t before running the SU line. RCvg, it's a small variable. If you set that to yes, then it appends enough ascent. That's for demons that are not able to fork themselves into the background. There's actually quite a lot around there, and I don't think systemd will help with that. So that's an example of how SSHG is called by RCD. We have RC stop, which by default will run pkil against the demon root table on the pxp. When you use minus xf, because we want to match the entire process and arguments and match exactly that. The pxp is the important part here. Pxp is what we use to actually match the process that we want to handle. By default, it's set to the demon variable, so the path to the demon. If there are demon flags, then space and the demon flags. That's how by default we match demons that we want to handle. One little thing that I want to mention is that actually down, the base systemd script are not handled by RCD. They're just being sent to sikterm, because all the base systemd demons on openBSD are supposed to behave properly when you send them to sikterm. So there is no need to do any specific shutdown procedure. Then we have RC reload, which by default will send a hub to the process, so the pxp. So that function is responsible for reloading a demon. OpenBSD reloading for us means reconfigure yourself. If minus hub does something else, like just reload but without reconfiguring or does any other crazy stuff, then we consider that it's not a reload and we disable that function. We'll see how a bit later. Then we have RC check, which is just a pgrep against the pxp, so the process list. That returns a code of zero when the demon is running and one when it's not. Then we have a few optional functions, RCP. RCP will be run by the start action, but before running RC start. Typically, it's used to create non-privileged owned directory to store whatever socket file or things like that. Then we have the opposite. We also have an RC post, which is run by the stop action after RC stop. That's typically used when a demon is not able to properly terminate itself and leaves a whole kind of stupid files around. For example, putting the system back into a Christian state, like Cups is a good example. When you start Cups, it will actually move away the traditional LPD command that you have in your path and replace them with the Cups ones. Of course, on stop, it will do the opposite. It will put back the original LPD commands. The last function that RC.subber provides is the RC command. That's the main entry point where you actually pass the start, stop, reload, etc. action. That is, and must be, the last command that an RC will call. Let's see how this command works. When you pass the start action to RC command, first, it will check that the demon is enabled. It will basically run RC check. It will check if it's not already running. It will run RC pre. It will run RC start. Then it will put all the demon variable under this directory, for an RC script name. I'm going to talk about that in five minutes. There is a reason we're doing that, unfortunately. Then, of course, it will wait up to the time of seconds to make sure that the demon is probably started. RC command stop. RC check that the demon is running. Then we run RC stop. Then we wait up to the time out again. Then we run RC post. Then we remove the file that we created with RC start in Varvoo and RCD. We start, as I said, just Fubar stop and Fubar start. The RC command reload. We check that the demon is running. If it is running, then we run RC reload on it. RC command check is just running RC check. Nothing really interesting here. As I mentioned, there is some action that some demon cannot handle. A good example is reload. There are a lot of demons that are not able to reload themselves. They have to stop first and start or restart. In this situation, what we do is that we transform, we use the function name that we put into a variable and set it to no. If your demon cannot reload itself, you put RC on the score reload equals no into the RC descript. If it cannot, I don't know, start, which would be stupid, you put RC on the score start. We have the RC user check variable. As I mentioned earlier, check is the only action that does not require a higher privilege to be run. That is not true in some situations. If I remember correctly, there is a couple of stuff like Samba, for example. I think we use SMB control, something like that, to check whether it's properly running or not. That has to be run as would. In which case, we add this to the RC descript so that when you try to run it, a regular user will tell you that you cannot do that. A little word about the file that we put under the var run RC script name. This file is used to actually have the current status of the demon that you want to manage. The reason is that RCD is dynamic and real-time. That means that if you run a script with RCD, it will check real-time in the process list. It will check the configuration as it is now and check the process, et cetera, et cetera. Say you want to change the flags of your process. You want to start NTPD with minus S, for example. You will edit ETC RC conflocal and set NTPD underscore flags equals minus S. Then you will try to restart NTPD. Of course, RCD will try to match the process that has a minus S in it. When this process doesn't exist yet. That's why we're storing all the information about the currently state of the demon running. It's not great. Here is a complete template of everything that can be done with an RCD script. I'm not going to review everything because we went there already. We can see the three mandatory variables. Just one thing I want to mention is that the demon is, as I said, the path to the demon that you want to start, but we now also force people to add the flags that demonize the demon in it. If your demon needs minus D to demonize, if we put that in demon flags instead of demon, if you want to change the flags of the demon, there is a good chance that you will forget about putting back minus D. You will just think about putting back the new flags that you want to add. It's just a precaution. That's basically the extent of the RCD system. At that point, we were quite happy with how things were working, but there was still one thing missing. At the time, we were using a lot of automation tools like Puppet and things like that. While it was nice to tell Puppet to be able to run ETC, RCD, blah, blah, there was no easy way to enable a demon or disable a demon or change its flags or whatever. We needed a tool that would handle that. I came up with a tool called RC Control. At first, it was meant to be an RC Conflocal Editor. RC Control enabled, blah, blah, blah, would actually modify RC Conflocal with the proper configuration. But it was so easy to actually make it like the service command on Linux and FreeBSD and NetBSD, that we actually implemented that as well. You can also handle your demons using RC Control, RC Control, Restart, whatever. It's like a big mix of the service command, the check config and the SysConfig Editor that you have in Red Hat Land before SystemD. It's an alternative, of course. You can still use VI or Emacs or whatever you want to edit RC Conflocal. It's not mandatory to use this tool. It's just useful. It wasn't must have to any automation tool or monitoring tool, things like that. I also want to mention that we're not service, the command service, compatible in the way that we handle things. We don't do RC Control SSHD start. We do RC Control start SSHD. The reason is that I mentioned that initially we only wanted to be an RC Conflocal Editor. Before RC Control, the user was still kind of confused because when you tell them, they will ask you, how do I enable multicast on my machine? You just put RC Conflocal, multicast equals yes. Okay, that's easy. So for SSHD, it's the same. No, no, no. It's different because that's a demon. So it's not how it works. You have to put SSHD on the score flags. Okay. So why can't I do multicast equals and nothing? That should also enable multicast. No, it's different. That's a service. So on and on, it was kind of confusing. And RC Control provided a unified interface for that. RC Control enabled multicast or RC Control enabled SSHD that will do the perusing and you don't have to worry about whether you need to append underscore flags or whatever. It's also useful because if you enable a script from external packages, it would also append it to the packet script variable to start it. So it provided us with a unified interface. It was developed to abstract all these specific cases that are mentioned. It also replaced a whole bunch of code in a pop-up solve and ansible that were manually dealing with RC Conflocal. So you have an example here. That's what Jasper at OpenBSD did for pop-up implemented the original RC Conflocal implementation and when RC Control came up, then it was able to delete most of the code. So it will abstract all the stuff like daemon versus service. You don't need to care whether it's a daemon or a service. You don't need to care whether it's a meta script or a regular script or whatever. By developing RC Control, there is something that looked very fragile to us at some point is that RC Control will source RC.subber to get all the RCD functions that it needs. It will then run the RC script which will source RC.conf and RC Conflocal which, and then the RC script will also resource RC.subber. So it looked like a bit of crazy sourcing. So what we did is that we implemented a function called RCParseConf which would basically parse the configuration in RC.conf and RC Conflocal instead of sourcing. Some people were not happy about it because they were doing some crazy stuff in RC Conflocal, mainly putting crazy shell code in there which you were not supposed to do that, but of course, since we were sourcing, that worked perfectly for them. So we went from sourcing to parsing and we introduced a small variable as well to be able to only parse for the RCD functions instead of parsing the entire framework each time we need to have a function from RC.subber. The way RC Control works is, I think, pretty easy. You give it the action that you want and then you give it the name of the service. So you can get service, you can get the service flags, you can get the entire service configuration, whether it's started, whether it's not started. The second line is basically like running the RCD script by yourself, like RCctl start SSHG will in effect run etc, rcd, sshg start. And you can order a daemon. This only works for packet script daemon because all the base system daemon are hot coded sequentially in etcrc. You can get information about what service or started, etc, etc. So here you can see a few examples. You can enable 3D daemon at once if you want. I want to enable multicast message bus in CubsD. That's completely transparent. I want to change the ntpd flags. I want to restart 3D daemon at once. That's way more convenient than having to run three times a different script. You can have a list whether you're running daemon. And we also have a small command called RCctl ls failed, which will list the daemon that are enabled in the configuration but that are not running. This is run every day by the DAIRL script on OpenBSD. So it's not like a supervision system or whatever, but it can be useful from time to time to make sure that all the daemon that are supposed to run are in fact running. So to conclude, RCD is really a compromise between features and isabuse. I think very simple. It's not suitable for all the crazy situation out there. It does has is drawback for sure. But it's OK. It was a design decision from the very beginning. So all the obvious deficiencies were actually known from the very beginning. So it's not a replacement. It's not a process control framework. It's not a service supervisor. Service supervisors are really fine. They actually use some of them, but only with specific daemons. So yeah, it's a compromise. I think we succeeded in making it horribly simple and robust. We preserved the original paradigm. All the old dudes were still happy with the way things worked. It's been built on decked old components. We're just using very basic tools, P-Kill and all the suite, SU. We provided a consistent interface with RC control. And it's also easy to integrate into other operating systems. Actually, we came up with a small homemade Linux distribution where we used the OpenBSD RCD system in it. It worked just fine. I think there was just a couple of things to change because we're using KSH on OpenBSD and there's Bash on Linux. I think there's one or two substitutions that were not working with Bash. But that's pretty much it. It worked out on the box. So that's it. Thank you very much. Any questions? Yeah. I think all the designs are just so much better than the other ones. The problem is that if you store a pit, then you have to end all the pit-changing. And sometimes it's racy. Sometimes it can be racy. Sometimes you have demons that restore themselves. I'm not, that's, it's something that's on my to-do list, that's been on my to-do list for forever, basically. That problem we want to be able to solve. It's just using PID is not the right answer for us. And I don't have the right answer yet. I'm still thinking about it. I had a few ideas a few months ago which I talked about with my fellow OpenBSD developers. But I wanted to inject some stuff in the process list. As Bob Beck said, the kernel is not a fact system. Yeah. Yes. Yeah. Yeah. Well, the problem is that if you do that, I mean most demons are super chatty. So you will overflow your whole console with messages that you don't want to have to see. So that's why it's not a default. It's easy to enable when you need it. And if something doesn't start, then you can manually run with minus d and c, and c y. Yeah, but that's what I'm saying. At start, we will end up with a huge amount of stuff. Not by default. But that's not a default. Yeah. There are a few stuff that throw out. And yeah, but I remember that there was a few, maybe that changed since now, but I remember there was a few. And actually it doesn't, it doesn't, I mean the fact that it was super quiet on startup was also a requirement because that's how the thing were done before. And we were actually telling whether the RCD script run okay or not. But we had to revert that because people didn't like too much information. So yeah, outputting the entire stuff on the console by default. I'll just get screen back. Yeah. Yeah. Yeah. Yeah, yeah, yeah. Something doesn't, doesn't, doesn't start. And it will tell you it failed. Yeah. Yeah. Yeah. But to be honest, most of the time when something doesn't start, it's logged and see it's logged. Yeah. Yeah. But I mean, that's, that's how you want those stuff. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. That was actually someone actually suggested that a few months ago. But actually the guy who suggested that didn't realize that his failure was logged in, see it's logged. Pretty reticulous. It wasn't going to help anything. Yes. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. No. No. Yeah. We could, we could add a couple of checks. Yeah. We could add a couple of check, but I mean, I don't think we'll be able to cover all the crazy stuff. So what it already does a couple of stuff is sort your fight, it removed some quotes where they're not needed stuff like that. But we don't want to do too much because again, it's shell. So parsing and all is already tricky. But you can't have it all. I mean, it's super small, super simple. That's, yeah. Okay. So, right on. Thank you. Yeah.
 So first of all, thank you very much everyone for attending to this talk.  Good morning.  My name is Antoine Jacoutot and I'm going to talk about our RCD implementation on Open  BSD today.  I just want to apologize if I have a weird voice from time to time.  I have caught a cold for a couple of days.  If it's weird, that would be why.  Or maybe it's because I mentioned system D. Also make some weird stuff in my voice.  Anyway, a little bit by myself.  I've been an Open BSD developer for about a decade now.  I'm known as Jacoutot or AJA for the lazy ones.  I work mostly on things that make my life easier when I put Open BSD in the real world.  So I'm more on the assessment engineering side of things more than a regular developer.  So these are some of the things that I've done.  Maintain way too many ports.  Among them the GNOME desktop for which I'm a foundation member and I maintain the French  Open BSD mirror.  I just want to say a quick thank you to my fellow developer who I developed the RCD system  with, Robert Nagy and Ingo Schwartzy.  Robert and I came up with the original implementation and Ingo gave us tremendous help in actually  having a good shell foo.  So the stuff we're going to talk about today, we're going to talk a little bit about the  historical and current boot process of Open BSD.  We're going to have a quick look at the alternative that we looked at when we first started the  implementation, how to use RCD, the internals of RCD and I will introduce a small tool called  RC control which is a high level tool to handle all the RC configuration and demons.  So all in all we'll see if RCD had any implication on the traditional system that we used before  it.  Just a word of advice disclaimer.  If you guys came up for a highly technical talk then I'm afraid you're going to be disappointed.  RCD on Open BSD is just one very, very tiny small shell script.  So it's not really interesting by itself.  What's interesting is the story behind it and what it does.  So that was the trigger why I got fed up about and wanted to create something that was a way  to handle Jim in a generic way.  A few years ago before RCD I was working at deploying Open BSD based infrastructure all  over the world and especially in a area where Linux knowledge was very low, let alone BSD.  And I just got tired because the weekend on this region are not the same.  I got tired to be working up on Sundays at 6 a.m. by my ticket system sending an SMS that  someone wanted to reload something and was not able to or wanted to start something and  couldn't find ETC in it deep.  So yeah, that was the trigger for it.  Of course, the main reason after that that we wanted a generic way to handle services  and demons was because we were using a lot of automation tools.  So automatically enabling services or starting services if you don't have a generic interface  is very hard.  You really don't want to have a huge amount of Ruby code editing your RC Conflocal or  anything.  So yeah.  So the way Open BSD boots hasn't changed much since its initial inception.  It's actually pretty much how it still boots now.  In a very general way you have the boot order that will fire up the kernel and the kernel  will run ETC RC using SH.  So yes, it's old but it's still very dependable.  It's predictive, it's sequential, there is no service dependencies, et cetera.  Basically what it does is it's a big RC script that runs things sequentially.  To be able to control what we want or do not want to start, we have two configuration  files, the ETC RC Conf which contains the operating system default configuration and  we have ETC RC Conflocal where you can put the overrides compared to RC Conf.  The way that we start a demon or service.  Okay.  I'm going to explain what we mean by demon or service in RC.  First a demon is the regular demon that will fire up fork and run in the background.  What a service is more like a kernel facility like I want to start PF or I want to enable  the counting, et cetera.  Anything that is not basically running a demon.  So to start a demon, you edit to RC Conflocal and just put the demon name underscore flags  equals.  If you say anything but no, then the demon will be enabled.  And service is just yes, no.  So that's how it works.  It worked before RCD and that is how it also works now.  So nothing changed in that regard.  So the requirement, as I said, nothing changed and nothing needed to change.  We knew that convincing the open BSD crowd to have an RCD system would be quite challenging.  So we didn't want to disrupt anything, really.  Actually we even lied a bit.  I will talk about that a bit later.  So yeah.  So the current paradigm couldn't change.  We have to preserve existing behavior.  So basically what we did is that we plugged on our RCD system on top of the traditional  ATC RC initialization.  There is one important thing is that we only wanted our RCD to handle demons.  We didn't want to handle like starting a network or mounting filesystem or things like that.  These were completely out of scope from the very beginning.  And of course we wanted something small, simple, robust because as I said I'm not a developer,  I'm stupid so I'm only able to do very simple things.  And of course easily debugable.  So at the time we started to look at different alternatives, not all RCD based but at least  initialization based.  Yeah.  Well you don't see system here.  The reason is not that obvious.  At the time system was actually in its infancy.  I think I'm not sure they even had a release yet.  And anyway we're not looking to replace any D-chrone, C-slug, and what else.  So we had a quick look at the SMF from Solaris, and Lumos, and Long D from MacOS.  These are actually very nice but very complex system.  It was way too complex for me.  I'm stupid.  But it was nice to have a look at it.  They actually fix some real-world problem but these were out of scope for what we wanted  to do.  Then we had a deep loop at OpenRC which is actually quite sane RCD implementation on  Linux.  That's the one that's used by Gentle Linux and Alpine if I remember correctly.  The problem is that it was way, way, way different than how things were done on OpenBSD.  Of course it handled mounting files and setting TTYs and blah, blah, blah.  So all that seemed that we didn't want to do.  The init script were kind of too big for what we wanted to do.  We wanted to try to have init script that were really, really small.  And it introduced a whole bunch of new commands which was something that we really didn't  want to have.  And then I had a quick look at runit and demon tools.  Demon tools felt just too weird to me.  And runit was nice but it was a little bit like OpenRC introducing way too many commands.  One of the most interesting Linux RCD implementation was the one from Slackware.  But this one was actually too static for init.  It works basically the same way OpenBSD used to work.  You have a big ETC RC.M that will run sequentially the init script that you give it.  The problem is that if you, I don't know, install the package that installed an init  script then you have to manually add that to RC.M or RC.local or whatever.  And there is no factorization between the RC script so they're all huge.  And of course the obvious choice would have been free BSD or net BSD RCD and RC order.  That is actually what we initially wanted to bring into OpenBSD.  So we looked at it.  The reason we didn't go for it is that there was a lot of small differences in the way  we were doing things.  Like enabling a demon was demon and the score enable equals yes.  That's something we didn't do.  That's something we didn't want to do because again it was forbidden for button to change  anything.  It was also used to do other stuff than just handle demons.  And RC order provided dynamic startup of the script.  I mean dependency.  That's something we didn't want to have either.  So basically if we had gone for the free BSD and net BSD RCD it would have meant that we  would have to strip it quite a bit so it was not really worth it.  So that's the reason we just went for our own implementation.  I really want to enhance the fact that we did not want to start from scratch.  It just happened because we couldn't find something that was good for our needs.  So really it's not a non-invented here syndrome.  It's really we looked at a lot of different ways to do it.  So on and on we started with something small that needed to be targeted to the OS requirement.  We didn't want to create a process supervisor that is not the goal of RCD from our point  of view.  Process supervisors are great.  There is some great real use case for that but it was not in the scope of RCD.  So no crazy stuff like even driven or socket activated demons or things like that.  No parallelization either.  We don't really care about the boot time of the machine.  In these days of an age you have half of your machines or the end on the cloud somewhere.  These boot super fast whether you have parallelized in it or whatever.  And otherwise you just run on bare metal and then you have to wait 15 minutes for the post  to finish anyway.  And as I mentioned no automatic startup ordering.  The reason we didn't want to have service dependencies and automatic ordering is that  it introduced complexity and that was not really an all point of view necessary.  When I put a system in production I usually put a system production that does one or two  things.  I'm not expecting to put something that will run 20 different services so I don't need  to actually order things for 20 different demons.  So basically manually adding two scripts in the order that you want to run them was enough  for us.  So the initial landing, initial implementation was done in October 2010.  So it consisted of RCD.subber which contained all the basic functions to the RCD framework.  And of course the RCD script coming from packages only.  That time and that's where we lied, we said that it was designed for ports only.  That was the only way that we could actually bring something into the base system without  being knocked out.  So of course base was the ultimate goal but we had to come up with a good proof of concept  that what we had developed was actually useful and well done.  So as I mentioned, I am not a developer so I didn't want to create a whole bunch of new  commands to handle demons so we basically kept using the tool that we were already using  manually.  So namely kill or pkill or pgrep, et cetera, the traditional tool that are already there  to manage processes.  We did not want to rely on PID files because well you always have the chance to have like  a dangling PID file that's not here or there or you have also a race with a PID file or  when you have a demon you can change the PID so if you have to handle that and things  like that.  So it was not really, I mean we didn't want to go that way.  So as I mentioned no start, stop demon or anything that would actually run the demon  for you and as long as 95% of the ecosystem would be properly handled by RCD, well that  was good enough for us.  For the gigantic weird application that needs all kinds of stuff to properly start or stop  and it's just better to write a home script for that.  And for better or worse we did it in shell because the entire unit system was already  in shell.  So at that time the RCD script were called from RC local.  RC local is a file that's typically modified by the admin of the machine so it has nothing  to do with the base system and that's where you traditionally use to manually add the  demon that you wanted to start.  So it looks like a good fit to actually start the RCD script from there.  And that's basically the only amount of code that we added to this file which is completely  stupid.  It's just a Y loop and if there is a script and it's executable then you just start it.  So it was really, really small.  We couldn't do a lot more than stop, start, reload but that was enough for a good proof  of concept.  So at that time RC.subber which provides all subroutines to RCD was 54 line of shell code.  So to my point of view that's pretty small.  I committed it when we committed it then we got this kind of answer.  So that's what I mentioned earlier.  We knew it would be a challenge.  54 line of shell code as a bloated interface.  Of course we also had very nice review from people actually using OpenBSD in the real  world and really needed some kind of RCD system.  So yeah, that was amusing but that actually proved that we were right to be very, very  cautious.  And one release after that, we were allowed to handle base system demons.  So why did everyone change its mind about moving RCD to base and not just handling ports  and steam and?  That's true.  But I'm a liar.  I said so already.  So the main reason was that the processes were not started in isolation.  Basically they inherited all whatever ETC RC was doing before, all the environment, all  the stuff.  So that was very bad and of course that can lead to unexpected things.  And the RCD were using SU for environment sanitation.  To give you an example of how it used to work, if you run the first command versus the second  one, the first command will output this in your PHP info.  So that is a lot of information I don't want to see in my web server.  If you're like some people who also love to export like AWS private keys in their environment,  then they will appear there as well.  So yeah, that's what basically convinced all the rest of the OpenBSD crowd to go for RCD.  A little word about the startup sequence, I've been talking about ETC RC all the time.  So let's see a little bit what it does exactly.  And let's see where RCD is actually used within that script.  So ETC RC basically enabled your swap at boot time.  It will run FSCK on the file system.  It will mount your root directory, set up TTY flags, keyboard encoding, blah, blah, blah.  It will start the network.  Network is not handled by RCD.  It's a script on itself called ETC net start.  And then in the middle of the script, then RCD will start starting demons.  What we call the early demons like syslogd, pflog, and tpd.  Then it will run the RPC demon, port map, yp, blah, blah, blah.  And then we're back to doing some sequential stuff like enabling quotas, cleaning TMP,  setting up the default secure level, blah, blah, blah.  And then RCD again is back to start all the rest of the demons like SSH, RelayD, GHCP,  all that stuff.  So basically RCD in itself is a very, very small subset of the boot sequence.  What it is today, it's still small.  RC.subbers only 224 line of shell code.  We were able to remove 150 line of code from ETC RC itself.  The only small code that we had to add to it was just sourcing RC subbers to get the  functions.  We created a very small function called startDemon.  And that's what ETC RC will use to start the RC descript.  Basically, you pass it all the RC descript that you want to run.  And if the script is actually enabled in RCConf, then it will start it.  If not, then it will not do anything.  Then we have a small while to also start the package script.  I will talk about that a bit later because there is a different way between starting  the base system demons and the external packages RC descript.  So on and on, it was a big fit again for only 70 line of shell code.  So how was it used?  It's worked tried to keep it very simple.  We have four plus one action available.  Well we have five action, but the restart action really is start and stop.  So we have a start, obviously, with start to demon according to your configuration, with  the flags you pass it, it can run as a different user under a different login class, et cetera.  We have the stop action, which basically send a sick term to the demon.  Reload will send a sick up.  And we have the check action, which will pick up for the process and tell you whether it's  running or not.  These are the very basic.  We'll see exactly in details how it's done.  Because matching demons is kind of hard.  So all of these actions, they need to be run as a privileged user except for the check  action.  Although it can happen that the check action also need privilege, right?  We'll see about that a bit later.  All these actions are fully configurable and overwriteable.  These are only default.  Like when I say to stop action, send a sick term to the process, you can completely override  that in the RCD script itself.  Same thing for the reload action.  Because kill minus up doesn't always do what you think it does.  And since it was the main user interface, we really tried to keep as less action as  possible.  We didn't want to do like four start or whatever.  So that's how a minimal RCD script looks like.  You have the she-bank and then the demon variable, which is the only mandatory variable, which  is a path to the demon you want to run.  Then we are sourcing RC.subber and we are passing it whatever action we want.  Start, stop, whatever.  It is small, but it is actually, I would say, like one half of the RCD script in OpenBSD  are just like that.  So that's enough for most of the stuff.  Of course, for packages, it's a bit different because we have some weird stuff in there.  Alongside the actions, we have two optional flags that we can pass.  Minus D for debug mode.  By default, RCD will redirect to dev-null the output of the script.  So in minus D mode, it will output everything on the console.  And we have minus F, which is similar to one start or four start.  I don't remember which one.  I mean, it's used to start a demon when that particular demon is not enabled.  It's an op for the demon coming from the package script.  The reason for that is that for all base system demons, we already have default in RCConf  that are set to know or that are set to whatever because by default, this demon must run.  The system has no knowledge of a default configuration for a package script because a package script  is just an RCD script that's installed under ETCRCD.  So if you run ETCRCD, my package script start, it will start because there is no default  set to know.  But it's only used for forcing a start of base system demons.  So the way we enable demons are the traditional way for the base system demons.  So that didn't change.  It's just the foobar underscore flags.  For the package script, we also use foobar underscore flags to actually append flags  to the demon when we want to start.  But to actually start a demon, so to tell the OS that we want to start that demon, we  have to append it to a variable called package script, which is located in RC.conf.  All the script that are listed in the package script variable will be run in order on start  and a reverse order on stop.  So we have several RCD variables available in RCD script.  All these variables can be overridden in RC.conf.local.  We have, first one is the demon class, which is the BSD login class that the demon will  run under in default to demon, which is a preexisting class in login.conf on openBSD.  That's pretty convenient.  When we started working on RCD, we wanted to find a nice way to increase the niceness  or inject environment variable in the process or something like that.  And so at first, we're thinking about putting demon underscore, nice demon underscore, and  stuff like that.  And we just realized we actually had a facility for that.  Login.conf is perfect.  The demons are starting with SU, and SU can pass a BSD login class to the process that  it will run.  So nothing to develop here.  It was already there.  So the well-known now demon flags, which default to no or empty in RC.conf.  So these are the flags that you want to pass to your demon on startup.  So I don't know, minus F, ATC, blah, blah, blah.  Then we have a demon R table, which allows to run the demon in a particular routing domain.  Routing tables allows multiple lookup tables for routes, and they're usually used for police-based  routing.  I'm not a network guy, so...  Then we have a demon timeout.  We realized at the very beginning of RCD that if you make a typo in your demon flag, like  for example, you forget to pass the minus D to demonize the demon, then you can fuck  up your startup sequence quite badly.  So we introduced a default timeout.  If the process hasn't returned anything after 30 seconds, then we consider it failed, and  we keep...  We put it in the background and keep running.  It's also used for the opposite.  There's quite a few processes, like Squid or MySQL, when you have a huge database that  can take actually a long time to stop, especially Squid actually on OpenBSD.  So sometimes when you want to properly stop the demon, that you may need to increase that  timeout.  Otherwise, it will be a forced kill.  And the last variable we have is the demon user.  So it defaults to route, and that's the user that the demon will run under.  So that is what SU will use as a user.  So all these variables, they are overwriteable, obviously, by the RCD script itself, which  usually contains the default for that particular demon.  It is overwriteable by RCConf, which is the default OS configuration, and it is overwriteable  by STCRCconf local, which is the locally modified configuration.  The way that you override a particular variable, so here as an example for flags, is that instead  of having demons flags equal, you substitute the demon part with the RCD script name.  So the first line here, demon flags, is what's under the NetEsnMPD RCD script.  So by default, it will start with minus you, blah, blah, blah.  If you want to please Henning and remove IPv6 support, then you edit RCConf local and you  will do IPv6, and minus A is for looking support or I don't remember.  So that's the way to override any variable.  Underscore class, underscore flags, underscore users, et cetera.  Actually a light again.  Yes.  The way you override the demon class is a bit different.  Here we'll not be using environment variable.  RCD will automatically check if there is a login class that has the same name as your  RCD script.  So if you have a login class that's called asshgd, for example, or netesonMPD, and that  example, it will use that from login.conf.  If there is no matching, then it will fall back to the default login class, which is  demon.  So that was a simple way to add the BSD login class support.  So here's an example of an RCConf local.  So APMD flags equal minus A. APMD being a base system demon means that we will be starting  it with minus A. We will be starting hotplug with no flags, and we'll pass some flags to  the same D demon.  But same D is an external package.  So just putting that line here will not start it.  We have to put it in here as well in the package script variable, and that's how the system  knows that it has to run that script.  And we don't want to run NTPD for whatever reason.  Yes, it's bloated.  So there are also a few special cases that we have to take into consideration.  We have a notion of what we call meta RCD script.  Those aren't really RCD scripts.  They're just regular shell scripts that basically run other RCD scripts in the order that you  pass them.  It's used typically for, yeah, like Samba, you can start SMBD and NNBD, but there are  also applications like Bacula where you want to run the director and the client on the  same machine and etc.  Or there's a RAFRA group where it has like seven or eight different demons.  You don't want to have to enter all of them.  You just have a meta script that will run all them in the sequence.  Another special case, and this one is hard, is having multiple instances of the same demon.  It works most of the time, but it depends on a lot of different details.  The way we do it right now is that we link the original RCD script and we rename it to  something.  The reason we're linking and not copying is that package script are part of the base  system, so if they get updated, then at least your link will be updated as well.  Sorry.  So that works for most of the demon, but there's a lot of package out there that changed their  process like using setproc title or stuff like that.  And then it's very hard to actually match the correct one in the process list.  If a demon changed like some sample, you pass foo bar, minus VEP, blah, blah, blah, foo  foo, and setproc title to just foo bar, then if you start two of those, there is no way  that RCD will know which one it has to stop.  That's because, again, we're not a process supervisor, so that's a drawback of the implementation.  Sorry?  No.  Can you define a secret script?  Yes.  Well, I'm actually going to talk about that in a few.  So let's have a look now at the internal of RCD, so RC.subber, that's the entry point  of the whole framework.  That's where we define the default function that will be used by all RCD script.  All this function can be overwritten by the RCD script, as I mentioned earlier.  If you don't want to send the hub to reload your process, but because that doesn't work  and needs some other procedure, then that's where you're going to do it in the RCD script.  Sorry.  So the first function that RC.subber provides is RC on the score start.  What it does?  It's RC exact, demon, demon flags, and the bg variable.  RC exact can be two things.  It's basically the SU line where you pass it to a demon class, then the shell, and the  user you want to run the command as, and then the command itself.  If we define a different routing table, then we prepend a call to root minus t before running  the SU line.  RCvg, it's a small variable.  If you set that to yes, then it appends enough ascent.  That's for demons that are not able to fork themselves into the background.  There's actually quite a lot around there, and I don't think systemd will help with  that.  So that's an example of how SSHG is called by RCD.  We have RC stop, which by default will run pkil against the demon root table on the pxp.  When you use minus xf, because we want to match the entire process and arguments and  match exactly that.  The pxp is the important part here.  Pxp is what we use to actually match the process that we want to handle.  By default, it's set to the demon variable, so the path to the demon.  If there are demon flags, then space and the demon flags.  That's how by default we match demons that we want to handle.  One little thing that I want to mention is that actually down, the base systemd script  are not handled by RCD.  They're just being sent to sikterm, because all the base systemd demons on openBSD are  supposed to behave properly when you send them to sikterm.  So there is no need to do any specific shutdown procedure.  Then we have RC reload, which by default will send a hub to the process, so the pxp.  So that function is responsible for reloading a demon.  OpenBSD reloading for us means reconfigure yourself.  If minus hub does something else, like just reload but without reconfiguring or does any  other crazy stuff, then we consider that it's not a reload and we disable that function.  We'll see how a bit later.  Then we have RC check, which is just a pgrep against the pxp, so the process list.  That returns a code of zero when the demon is running and one when it's not.  Then we have a few optional functions, RCP.  RCP will be run by the start action, but before running RC start.  Typically, it's used to create non-privileged owned directory to store whatever socket file  or things like that.  Then we have the opposite.  We also have an RC post, which is run by the stop action after RC stop.  That's typically used when a demon is not able to properly terminate itself and leaves  a whole kind of stupid files around.  For example, putting the system back into a Christian state, like Cups is a good example.  When you start Cups, it will actually move away the traditional LPD command that you  have in your path and replace them with the Cups ones.  Of course, on stop, it will do the opposite.  It will put back the original LPD commands.  The last function that RC.subber provides is the RC command.  That's the main entry point where you actually pass the start, stop, reload, etc. action.  That is, and must be, the last command that an RC will call.  Let's see how this command works.  When you pass the start action to RC command, first, it will check that the demon is enabled.  It will basically run RC check.  It will check if it's not already running.  It will run RC pre.  It will run RC start.  Then it will put all the demon variable under this directory, for an RC script name.  I'm going to talk about that in five minutes.  There is a reason we're doing that, unfortunately.  Then, of course, it will wait up to the time of seconds to make sure that the demon is  probably started.  RC command stop.  RC check that the demon is running.  Then we run RC stop.  Then we wait up to the time out again.  Then we run RC post.  Then we remove the file that we created with RC start in Varvoo and RCD.  We start, as I said, just Fubar stop and Fubar start.  The RC command reload.  We check that the demon is running.  If it is running, then we run RC reload on it.  RC command check is just running RC check.  Nothing really interesting here.  As I mentioned, there is some action that some demon cannot handle.  A good example is reload.  There are a lot of demons that are not able to reload themselves.  They have to stop first and start or restart.  In this situation, what we do is that we transform, we use the function name that we put into  a variable and set it to no.  If your demon cannot reload itself, you put RC on the score reload equals no into the  RC descript.  If it cannot, I don't know, start, which would be stupid, you put RC on the score start.  We have the RC user check variable.  As I mentioned earlier, check is the only action that does not require a higher privilege  to be run.  That is not true in some situations.  If I remember correctly, there is a couple of stuff like Samba, for example.  I think we use SMB control, something like that, to check whether it's properly running  or not.  That has to be run as would.  In which case, we add this to the RC descript so that when you try to run it, a regular user  will tell you that you cannot do that.  A little word about the file that we put under the var run RC script name.  This file is used to actually have the current status of the demon that you want to manage.  The reason is that RCD is dynamic and real-time.  That means that if you run a script with RCD, it will check real-time in the process list.  It will check the configuration as it is now and check the process, et cetera, et cetera.  Say you want to change the flags of your process.  You want to start NTPD with minus S, for example.  You will edit ETC RC conflocal and set NTPD underscore flags equals minus S. Then you  will try to restart NTPD.  Of course, RCD will try to match the process that has a minus S in it.  When this process doesn't exist yet.  That's why we're storing all the information about the currently state of the demon running.  It's not great.  Here is a complete template of everything that can be done with an RCD script.  I'm not going to review everything because we went there already.  We can see the three mandatory variables.  Just one thing I want to mention is that the demon is, as I said, the path to the demon  that you want to start, but we now also force people to add the flags that demonize the  demon in it.  If your demon needs minus D to demonize, if we put that in demon flags instead of demon,  if you want to change the flags of the demon, there is a good chance that you will forget  about putting back minus D. You will just think about putting back the new flags that  you want to add.  It's just a precaution.  That's basically the extent of the RCD system.  At that point, we were quite happy with how things were working, but there was still one  thing missing.  At the time, we were using a lot of automation tools like Puppet and things like that.  While it was nice to tell Puppet to be able to run ETC, RCD, blah, blah, there was no  easy way to enable a demon or disable a demon or change its flags or whatever.  We needed a tool that would handle that.  I came up with a tool called RC Control.  At first, it was meant to be an RC Conflocal Editor.  RC Control enabled, blah, blah, blah, would actually modify RC Conflocal with the proper  configuration.  But it was so easy to actually make it like the service command on Linux and FreeBSD and  NetBSD, that we actually implemented that as well.  You can also handle your demons using RC Control, RC Control, Restart, whatever.  It's like a big mix of the service command, the check config and the SysConfig Editor  that you have in Red Hat Land before SystemD.  It's an alternative, of course.  You can still use VI or Emacs or whatever you want to edit RC Conflocal.  It's not mandatory to use this tool.  It's just useful.  It wasn't must have to any automation tool or monitoring tool, things like that.  I also want to mention that we're not service, the command service, compatible in the way  that we handle things.  We don't do RC Control SSHD start.  We do RC Control start SSHD.  The reason is that I mentioned that initially we only wanted to be an RC Conflocal Editor.  Before RC Control, the user was still kind of confused because when you tell them, they  will ask you, how do I enable multicast on my machine?  You just put RC Conflocal, multicast equals yes.  Okay, that's easy.  So for SSHD, it's the same.  No, no, no.  It's different because that's a demon.  So it's not how it works.  You have to put SSHD on the score flags.  Okay.  So why can't I do multicast equals and nothing?  That should also enable multicast.  No, it's different.  That's a service.  So on and on, it was kind of confusing.  And RC Control provided a unified interface for that.  RC Control enabled multicast or RC Control enabled SSHD that will do the perusing and  you don't have to worry about whether you need to append underscore flags or whatever.  It's also useful because if you enable a script from external packages, it would also append  it to the packet script variable to start it.  So it provided us with a unified interface.  It was developed to abstract all these specific cases that are mentioned.  It also replaced a whole bunch of code in a pop-up solve and ansible that were manually  dealing with RC Conflocal.  So you have an example here.  That's what Jasper at OpenBSD did for pop-up implemented the original RC Conflocal implementation  and when RC Control came up, then it was able to delete most of the code.  So it will abstract all the stuff like daemon versus service.  You don't need to care whether it's a daemon or a service.  You don't need to care whether it's a meta script or a regular script or whatever.  By developing RC Control, there is something that looked very fragile to us at some point  is that RC Control will source RC.subber to get all the RCD functions that it needs.  It will then run the RC script which will source RC.conf and RC Conflocal which, and  then the RC script will also resource RC.subber.  So it looked like a bit of crazy sourcing.  So what we did is that we implemented a function called RCParseConf which would basically parse  the configuration in RC.conf and RC Conflocal instead of sourcing.  Some people were not happy about it because they were doing some crazy stuff in RC Conflocal,  mainly putting crazy shell code in there which you were not supposed to do that, but of course,  since we were sourcing, that worked perfectly for them.  So we went from sourcing to parsing and we introduced a small variable as well to be  able to only parse for the RCD functions instead of parsing the entire framework each time  we need to have a function from RC.subber.  The way RC Control works is, I think, pretty easy.  You give it the action that you want and then you give it the name of the service.  So you can get service, you can get the service flags, you can get the entire service configuration,  whether it's started, whether it's not started.  The second line is basically like running the RCD script by yourself, like RCctl start  SSHG will in effect run etc, rcd, sshg start.  And you can order a daemon.  This only works for packet script daemon because all the base system daemon are hot  coded sequentially in etcrc.  You can get information about what service or started, etc, etc.  So here you can see a few examples.  You can enable 3D daemon at once if you want.  I want to enable multicast message bus in CubsD.  That's completely transparent.  I want to change the ntpd flags.  I want to restart 3D daemon at once.  That's way more convenient than having to run three times a different script.  You can have a list whether you're running daemon.  And we also have a small command called RCctl ls failed, which will list the daemon that  are enabled in the configuration but that are not running.  This is run every day by the DAIRL script on OpenBSD.  So it's not like a supervision system or whatever, but it can be useful from time to time to  make sure that all the daemon that are supposed to run are in fact running.  So to conclude, RCD is really a compromise between features and isabuse.  I think very simple.  It's not suitable for all the crazy situation out there.  It does has is drawback for sure.  But it's OK.  It was a design decision from the very beginning.  So all the obvious deficiencies were actually known from the very beginning.  So it's not a replacement.  It's not a process control framework.  It's not a service supervisor.  Service supervisors are really fine.  They actually use some of them, but only with specific daemons.  So yeah, it's a compromise.  I think we succeeded in making it horribly simple and robust.  We preserved the original paradigm.  All the old dudes were still happy with the way things worked.  It's been built on decked old components.  We're just using very basic tools, P-Kill and all the suite, SU.  We provided a consistent interface with RC control.  And it's also easy to integrate into other operating systems.  Actually, we came up with a small homemade Linux distribution where we used the OpenBSD  RCD system in it.  It worked just fine.  I think there was just a couple of things to change because we're using KSH on OpenBSD  and there's Bash on Linux.  I think there's one or two substitutions that were not working with Bash.  But that's pretty much it.  It worked out on the box.  So that's it.  Thank you very much.  Any questions?  Yeah.  I think all the designs are just so much better than the other ones.  The problem is that if you store a pit, then you have to end all the pit-changing.  And sometimes it's racy.  Sometimes it can be racy.  Sometimes you have demons that restore themselves.  I'm not, that's, it's something that's on my to-do list, that's been on my to-do list  for forever, basically.  That problem we want to be able to solve.  It's just using PID is not the right answer for us.  And I don't have the right answer yet.  I'm still thinking about it.  I had a few ideas a few months ago which I talked about with my fellow OpenBSD developers.  But I wanted to inject some stuff in the process list.  As Bob Beck said, the kernel is not a fact system.  Yeah.  Yes.  Yeah.  Yeah.  Well, the problem is that if you do that, I mean most demons are super chatty.  So you will overflow your whole console with messages that you don't want to have to see.  So that's why it's not a default.  It's easy to enable when you need it.  And if something doesn't start, then you can manually run with minus d and c, and c y.  Yeah, but that's what I'm saying.  At start, we will end up with a huge amount of stuff.  Not by default.  But that's not a default.  Yeah.  There are a few stuff that throw out.  And yeah, but I remember that there was a few, maybe that changed since now, but I remember  there was a few.  And actually it doesn't, it doesn't, I mean the fact that it was super quiet on startup  was also a requirement because that's how the thing were done before.  And we were actually telling whether the RCD script run okay or not.  But we had to revert that because people didn't like too much information.  So yeah, outputting the entire stuff on the console by default.  I'll just get screen back.  Yeah.  Yeah.  Yeah.  Yeah, yeah, yeah.  Something doesn't, doesn't, doesn't start.  And it will tell you it failed.  Yeah.  Yeah.  Yeah.  But to be honest, most of the time when something doesn't start, it's logged and see it's logged.  Yeah.  Yeah.  But I mean, that's, that's how you want those stuff.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  That was actually someone actually suggested that a few months ago.  But actually the guy who suggested that didn't realize that his failure was logged in, see  it's logged.  Pretty reticulous.  It wasn't going to help anything.  Yes.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  No.  No.  Yeah.  We could, we could add a couple of checks.  Yeah.  We could add a couple of check, but I mean, I don't think we'll be able to cover all the  crazy stuff.  So what it already does a couple of stuff is sort your fight, it removed some quotes where  they're not needed stuff like that.  But we don't want to do too much because again, it's shell.  So parsing and all is already tricky.  But you can't have it all.  I mean, it's super small, super simple.  That's, yeah.  Okay.  So, right on.  Thank you.  Yeah.
OpenBSD_rc_d_8__pjHDvO_haQY_0_2015.png OpenBSD_rc_d_8__pjHDvO_haQY_2016_3347.png OpenBSD_rc_d_8__pjHDvO_haQY_3348_3551.png OpenBSD_rc_d_8__pjHDvO_haQY_3552_4283.png OpenBSD_rc_d_8__pjHDvO_haQY_4284_6131.png OpenBSD_rc_d_8__pjHDvO_haQY_6132_7331.png OpenBSD_rc_d_8__pjHDvO_haQY_7332_8015.png OpenBSD_rc_d_8__pjHDvO_haQY_8016_9023.png OpenBSD_rc_d_8__pjHDvO_haQY_9024_16787.png OpenBSD_rc_d_8__pjHDvO_haQY_16788_19427.png OpenBSD_rc_d_8__pjHDvO_haQY_19428_19883.png OpenBSD_rc_d_8__pjHDvO_haQY_19884_19931.png OpenBSD_rc_d_8__pjHDvO_haQY_19932_21143.png OpenBSD_rc_d_8__pjHDvO_haQY_21144_22139.png OpenBSD_rc_d_8__pjHDvO_haQY_22140_23519.png OpenBSD_rc_d_8__pjHDvO_haQY_23520_24239.png OpenBSD_rc_d_8__pjHDvO_haQY_24240_24287.png OpenBSD_rc_d_8__pjHDvO_haQY_24288_24443.png OpenBSD_rc_d_8__pjHDvO_haQY_24444_24635.png OpenBSD_rc_d_8__pjHDvO_haQY_24636_24827.png OpenBSD_rc_d_8__pjHDvO_haQY_24828_25175.png OpenBSD_rc_d_8__pjHDvO_haQY_25176_25259.png OpenBSD_rc_d_8__pjHDvO_haQY_25260_25451.png OpenBSD_rc_d_8__pjHDvO_haQY_25452_25655.png OpenBSD_rc_d_8__pjHDvO_haQY_25656_25799.png OpenBSD_rc_d_8__pjHDvO_haQY_25800_27311.png OpenBSD_rc_d_8__pjHDvO_haQY_27312_30875.png OpenBSD_rc_d_8__pjHDvO_haQY_30876_32111.png OpenBSD_rc_d_8__pjHDvO_haQY_32112_40187.png OpenBSD_rc_d_8__pjHDvO_haQY_40188_41231.png OpenBSD_rc_d_8__pjHDvO_haQY_41232_42671.png OpenBSD_rc_d_8__pjHDvO_haQY_42672_42875.png OpenBSD_rc_d_8__pjHDvO_haQY_42876_43103.png OpenBSD_rc_d_8__pjHDvO_haQY_43104_45083.png OpenBSD_rc_d_8__pjHDvO_haQY_45084_47555.png OpenBSD_rc_d_8__pjHDvO_haQY_47556_50519.png OpenBSD_rc_d_8__pjHDvO_haQY_50520_55955.png OpenBSD_rc_d_8__pjHDvO_haQY_55956_56207.png OpenBSD_rc_d_8__pjHDvO_haQY_56208_56267.png OpenBSD_rc_d_8__pjHDvO_haQY_56268_57995.png OpenBSD_rc_d_8__pjHDvO_haQY_57996_58307.png OpenBSD_rc_d_8__pjHDvO_haQY_58308_59471.png OpenBSD_rc_d_8__pjHDvO_haQY_59472_59987.png OpenBSD_rc_d_8__pjHDvO_haQY_59988_60203.png OpenBSD_rc_d_8__pjHDvO_haQY_60204_60707.png OpenBSD_rc_d_8__pjHDvO_haQY_60708_60923.png OpenBSD_rc_d_8__pjHDvO_haQY_60924_61451.png OpenBSD_rc_d_8__pjHDvO_haQY_61452_61859.png OpenBSD_rc_d_8__pjHDvO_haQY_61860_61955.png OpenBSD_rc_d_8__pjHDvO_haQY_61956_62123.png OpenBSD_rc_d_8__pjHDvO_haQY_62124_62171.png OpenBSD_rc_d_8__pjHDvO_haQY_62172_62231.png OpenBSD_rc_d_8__pjHDvO_haQY_62232_63107.png OpenBSD_rc_d_8__pjHDvO_haQY_63108_63443.png OpenBSD_rc_d_8__pjHDvO_haQY_63444_63731.png OpenBSD_rc_d_8__pjHDvO_haQY_63732_68615.png OpenBSD_rc_d_8__pjHDvO_haQY_68616_73895.png OpenBSD_rc_d_8__pjHDvO_haQY_73896_74087.png OpenBSD_rc_d_8__pjHDvO_haQY_74088_74243.png OpenBSD_rc_d_8__pjHDvO_haQY_74244_74543.png OpenBSD_rc_d_8__pjHDvO_haQY_74544_77399.png OpenBSD_rc_d_8__pjHDvO_haQY_77400_78587.png OpenBSD_rc_d_8__pjHDvO_haQY_78588_79523.png OpenBSD_rc_d_8__pjHDvO_haQY_79524_79679.png OpenBSD_rc_d_8__pjHDvO_haQY_79680_80807.png OpenBSD_rc_d_8__pjHDvO_haQY_80808_81551.png OpenBSD_rc_d_8__pjHDvO_haQY_81552_81599.png OpenBSD_rc_d_8__pjHDvO_haQY_81600_82895.png OpenBSD_rc_d_8__pjHDvO_haQY_82896_82967.png OpenBSD_rc_d_8__pjHDvO_haQY_82968_83951.png OpenBSD_rc_d_8__pjHDvO_haQY_83952_84503.png OpenBSD_rc_d_8__pjHDvO_haQY_84504_84755.png OpenBSD_rc_d_8__pjHDvO_haQY_84756_85031.png OpenBSD_rc_d_8__pjHDvO_haQY_85032_85403.png OpenBSD_rc_d_8__pjHDvO_haQY_85404_85559.png OpenBSD_rc_d_8__pjHDvO_haQY_85560_85823.png OpenBSD_rc_d_8__pjHDvO_haQY_85824_86363.png OpenBSD_rc_d_8__pjHDvO_haQY_86364_89171.png OpenBSD_rc_d_8__pjHDvO_haQY_89172_89423.png OpenBSD_rc_d_8__pjHDvO_haQY_89424_89711.png OpenBSD_rc_d_8__pjHDvO_haQY_89712_90287.png OpenBSD_rc_d_8__pjHDvO_haQY_90288_90443.png OpenBSD_rc_d_8__pjHDvO_haQY_90444_90851.png OpenBSD_rc_d_8__pjHDvO_haQY_90852_91053.png
So first of all, thank you very much everyone for attending to this talk. Good morning. My name is Antoine Jacoutot and I'm going to talk about our RCD implementation on Open BSD today. I just want to apologize if I have a weird voice from time to time. I have caught a cold for a couple of days. If it's weird, that would be why. Or maybe it's because I mentioned system D. Also make some weird stuff in my voice. Anyway, a little bit by myself. I've been an Open BSD developer for about a decade now. I'm known as Jacoutot or AJA for the lazy ones. I work mostly on things that make my life easier when I put Open BSD in the real world. So I'm more on the assessment engineering side of things more than a regular developer. So these are some of the things that I've done. Maintain way too many ports. Among them the GNOME desktop for which I'm a foundation member and I maintain the French Open BSD mirror. I just want to say a quick thank you to my fellow developer who I developed the RCD system with, Robert Nagy and Ingo Schwartzy. Robert and I came up with the original implementation and Ingo gave us tremendous help in actually having a good shell foo. So the stuff we're going to talk about today, we're going to talk a little bit about the historical and current boot process of Open BSD. We're going to have a quick look at the alternative that we looked at when we first started the implementation, how to use RCD, the internals of RCD and I will introduce a small tool called RC control which is a high level tool to handle all the RC configuration and demons. So all in all we'll see if RCD had any implication on the traditional system that we used before it. Just a word of advice disclaimer. If you guys came up for a highly technical talk then I'm afraid you're going to be disappointed. RCD on Open BSD is just one very, very tiny small shell script. So it's not really interesting by itself. What's interesting is the story behind it and what it does. So that was the trigger why I got fed up about and wanted to create something that was a way to handle Jim in a generic way. A few years ago before RCD I was working at deploying Open BSD based infrastructure all over the world and especially in a area where Linux knowledge was very low, let alone BSD. And I just got tired because the weekend on this region are not the same. I got tired to be working up on Sundays at 6 a.m. by my ticket system sending an SMS that someone wanted to reload something and was not able to or wanted to start something and couldn't find ETC in it deep. So yeah, that was the trigger for it. Of course, the main reason after that that we wanted a generic way to handle services and demons was because we were using a lot of automation tools. So automatically enabling services or starting services if you don't have a generic interface is very hard. You really don't want to have a huge amount of Ruby code editing your RC Conflocal or anything. So yeah. So the way Open BSD boots hasn't changed much since its initial inception. It's actually pretty much how it still boots now. In a very general way you have the boot order that will fire up the kernel and the kernel will run ETC RC using SH. So yes, it's old but it's still very dependable. It's predictive, it's sequential, there is no service dependencies, et cetera. Basically what it does is it's a big RC script that runs things sequentially. To be able to control what we want or do not want to start, we have two configuration files, the ETC RC Conf which contains the operating system default configuration and we have ETC RC Conflocal where you can put the overrides compared to RC Conf. The way that we start a demon or service. Okay. I'm going to explain what we mean by demon or service in RC. First a demon is the regular demon that will fire up fork and run in the background. What a service is more like a kernel facility like I want to start PF or I want to enable the counting, et cetera. Anything that is not basically running a demon. So to start a demon, you edit to RC Conflocal and just put the demon name underscore flags equals. If you say anything but no, then the demon will be enabled. And service is just yes, no. So that's how it works. It worked before RCD and that is how it also works now. So nothing changed in that regard. So the requirement, as I said, nothing changed and nothing needed to change. We knew that convincing the open BSD crowd to have an RCD system would be quite challenging. So we didn't want to disrupt anything, really. Actually we even lied a bit. I will talk about that a bit later. So yeah. So the current paradigm couldn't change. We have to preserve existing behavior. So basically what we did is that we plugged on our RCD system on top of the traditional ATC RC initialization. There is one important thing is that we only wanted our RCD to handle demons. We didn't want to handle like starting a network or mounting filesystem or things like that. These were completely out of scope from the very beginning. And of course we wanted something small, simple, robust because as I said I'm not a developer, I'm stupid so I'm only able to do very simple things. And of course easily debugable. So at the time we started to look at different alternatives, not all RCD based but at least initialization based. Yeah. Well you don't see system here. The reason is not that obvious. At the time system was actually in its infancy. I think I'm not sure they even had a release yet. And anyway we're not looking to replace any D-chrone, C-slug, and what else. So we had a quick look at the SMF from Solaris, and Lumos, and Long D from MacOS. These are actually very nice but very complex system. It was way too complex for me. I'm stupid. But it was nice to have a look at it. They actually fix some real-world problem but these were out of scope for what we wanted to do. Then we had a deep loop at OpenRC which is actually quite sane RCD implementation on Linux. That's the one that's used by Gentle Linux and Alpine if I remember correctly. The problem is that it was way, way, way different than how things were done on OpenBSD. Of course it handled mounting files and setting TTYs and blah, blah, blah. So all that seemed that we didn't want to do. The init script were kind of too big for what we wanted to do. We wanted to try to have init script that were really, really small. And it introduced a whole bunch of new commands which was something that we really didn't want to have. And then I had a quick look at runit and demon tools. Demon tools felt just too weird to me. And runit was nice but it was a little bit like OpenRC introducing way too many commands. One of the most interesting Linux RCD implementation was the one from Slackware. But this one was actually too static for init. It works basically the same way OpenBSD used to work. You have a big ETC RC.M that will run sequentially the init script that you give it. The problem is that if you, I don't know, install the package that installed an init script then you have to manually add that to RC.M or RC.local or whatever. And there is no factorization between the RC script so they're all huge. And of course the obvious choice would have been free BSD or net BSD RCD and RC order. That is actually what we initially wanted to bring into OpenBSD. So we looked at it. The reason we didn't go for it is that there was a lot of small differences in the way we were doing things. Like enabling a demon was demon and the score enable equals yes. That's something we didn't do. That's something we didn't want to do because again it was forbidden for button to change anything. It was also used to do other stuff than just handle demons. And RC order provided dynamic startup of the script. I mean dependency. That's something we didn't want to have either. So basically if we had gone for the free BSD and net BSD RCD it would have meant that we would have to strip it quite a bit so it was not really worth it. So that's the reason we just went for our own implementation. I really want to enhance the fact that we did not want to start from scratch. It just happened because we couldn't find something that was good for our needs. So really it's not a non-invented here syndrome. It's really we looked at a lot of different ways to do it. So on and on we started with something small that needed to be targeted to the OS requirement. We didn't want to create a process supervisor that is not the goal of RCD from our point of view. Process supervisors are great. There is some great real use case for that but it was not in the scope of RCD. So no crazy stuff like even driven or socket activated demons or things like that. No parallelization either. We don't really care about the boot time of the machine. In these days of an age you have half of your machines or the end on the cloud somewhere. These boot super fast whether you have parallelized in it or whatever. And otherwise you just run on bare metal and then you have to wait 15 minutes for the post to finish anyway. And as I mentioned no automatic startup ordering. The reason we didn't want to have service dependencies and automatic ordering is that it introduced complexity and that was not really an all point of view necessary. When I put a system in production I usually put a system production that does one or two things. I'm not expecting to put something that will run 20 different services so I don't need to actually order things for 20 different demons. So basically manually adding two scripts in the order that you want to run them was enough for us. So the initial landing, initial implementation was done in October 2010. So it consisted of RCD.subber which contained all the basic functions to the RCD framework. And of course the RCD script coming from packages only. That time and that's where we lied, we said that it was designed for ports only. That was the only way that we could actually bring something into the base system without being knocked out. So of course base was the ultimate goal but we had to come up with a good proof of concept that what we had developed was actually useful and well done. So as I mentioned, I am not a developer so I didn't want to create a whole bunch of new commands to handle demons so we basically kept using the tool that we were already using manually. So namely kill or pkill or pgrep, et cetera, the traditional tool that are already there to manage processes. We did not want to rely on PID files because well you always have the chance to have like a dangling PID file that's not here or there or you have also a race with a PID file or when you have a demon you can change the PID so if you have to handle that and things like that. So it was not really, I mean we didn't want to go that way. So as I mentioned no start, stop demon or anything that would actually run the demon for you and as long as 95% of the ecosystem would be properly handled by RCD, well that was good enough for us. For the gigantic weird application that needs all kinds of stuff to properly start or stop and it's just better to write a home script for that. And for better or worse we did it in shell because the entire unit system was already in shell. So at that time the RCD script were called from RC local. RC local is a file that's typically modified by the admin of the machine so it has nothing to do with the base system and that's where you traditionally use to manually add the demon that you wanted to start. So it looks like a good fit to actually start the RCD script from there. And that's basically the only amount of code that we added to this file which is completely stupid. It's just a Y loop and if there is a script and it's executable then you just start it. So it was really, really small. We couldn't do a lot more than stop, start, reload but that was enough for a good proof of concept. So at that time RC.subber which provides all subroutines to RCD was 54 line of shell code. So to my point of view that's pretty small. I committed it when we committed it then we got this kind of answer. So that's what I mentioned earlier. We knew it would be a challenge. 54 line of shell code as a bloated interface. Of course we also had very nice review from people actually using OpenBSD in the real world and really needed some kind of RCD system. So yeah, that was amusing but that actually proved that we were right to be very, very cautious. And one release after that, we were allowed to handle base system demons. So why did everyone change its mind about moving RCD to base and not just handling ports and steam and? That's true. But I'm a liar. I said so already. So the main reason was that the processes were not started in isolation. Basically they inherited all whatever ETC RC was doing before, all the environment, all the stuff. So that was very bad and of course that can lead to unexpected things. And the RCD were using SU for environment sanitation. To give you an example of how it used to work, if you run the first command versus the second one, the first command will output this in your PHP info. So that is a lot of information I don't want to see in my web server. If you're like some people who also love to export like AWS private keys in their environment, then they will appear there as well. So yeah, that's what basically convinced all the rest of the OpenBSD crowd to go for RCD. A little word about the startup sequence, I've been talking about ETC RC all the time. So let's see a little bit what it does exactly. And let's see where RCD is actually used within that script. So ETC RC basically enabled your swap at boot time. It will run FSCK on the file system. It will mount your root directory, set up TTY flags, keyboard encoding, blah, blah, blah. It will start the network. Network is not handled by RCD. It's a script on itself called ETC net start. And then in the middle of the script, then RCD will start starting demons. What we call the early demons like syslogd, pflog, and tpd. Then it will run the RPC demon, port map, yp, blah, blah, blah. And then we're back to doing some sequential stuff like enabling quotas, cleaning TMP, setting up the default secure level, blah, blah, blah. And then RCD again is back to start all the rest of the demons like SSH, RelayD, GHCP, all that stuff. So basically RCD in itself is a very, very small subset of the boot sequence. What it is today, it's still small. RC.subbers only 224 line of shell code. We were able to remove 150 line of code from ETC RC itself. The only small code that we had to add to it was just sourcing RC subbers to get the functions. We created a very small function called startDemon. And that's what ETC RC will use to start the RC descript. Basically, you pass it all the RC descript that you want to run. And if the script is actually enabled in RCConf, then it will start it. If not, then it will not do anything. Then we have a small while to also start the package script. I will talk about that a bit later because there is a different way between starting the base system demons and the external packages RC descript. So on and on, it was a big fit again for only 70 line of shell code. So how was it used? It's worked tried to keep it very simple. We have four plus one action available. Well we have five action, but the restart action really is start and stop. So we have a start, obviously, with start to demon according to your configuration, with the flags you pass it, it can run as a different user under a different login class, et cetera. We have the stop action, which basically send a sick term to the demon. Reload will send a sick up. And we have the check action, which will pick up for the process and tell you whether it's running or not. These are the very basic. We'll see exactly in details how it's done. Because matching demons is kind of hard. So all of these actions, they need to be run as a privileged user except for the check action. Although it can happen that the check action also need privilege, right? We'll see about that a bit later. All these actions are fully configurable and overwriteable. These are only default. Like when I say to stop action, send a sick term to the process, you can completely override that in the RCD script itself. Same thing for the reload action. Because kill minus up doesn't always do what you think it does. And since it was the main user interface, we really tried to keep as less action as possible. We didn't want to do like four start or whatever. So that's how a minimal RCD script looks like. You have the she-bank and then the demon variable, which is the only mandatory variable, which is a path to the demon you want to run. Then we are sourcing RC.subber and we are passing it whatever action we want. Start, stop, whatever. It is small, but it is actually, I would say, like one half of the RCD script in OpenBSD are just like that. So that's enough for most of the stuff. Of course, for packages, it's a bit different because we have some weird stuff in there. Alongside the actions, we have two optional flags that we can pass. Minus D for debug mode. By default, RCD will redirect to dev-null the output of the script. So in minus D mode, it will output everything on the console. And we have minus F, which is similar to one start or four start. I don't remember which one. I mean, it's used to start a demon when that particular demon is not enabled. It's an op for the demon coming from the package script. The reason for that is that for all base system demons, we already have default in RCConf that are set to know or that are set to whatever because by default, this demon must run. The system has no knowledge of a default configuration for a package script because a package script is just an RCD script that's installed under ETCRCD. So if you run ETCRCD, my package script start, it will start because there is no default set to know. But it's only used for forcing a start of base system demons. So the way we enable demons are the traditional way for the base system demons. So that didn't change. It's just the foobar underscore flags. For the package script, we also use foobar underscore flags to actually append flags to the demon when we want to start. But to actually start a demon, so to tell the OS that we want to start that demon, we have to append it to a variable called package script, which is located in RC.conf. All the script that are listed in the package script variable will be run in order on start and a reverse order on stop. So we have several RCD variables available in RCD script. All these variables can be overridden in RC.conf.local. We have, first one is the demon class, which is the BSD login class that the demon will run under in default to demon, which is a preexisting class in login.conf on openBSD. That's pretty convenient. When we started working on RCD, we wanted to find a nice way to increase the niceness or inject environment variable in the process or something like that. And so at first, we're thinking about putting demon underscore, nice demon underscore, and stuff like that. And we just realized we actually had a facility for that. Login.conf is perfect. The demons are starting with SU, and SU can pass a BSD login class to the process that it will run. So nothing to develop here. It was already there. So the well-known now demon flags, which default to no or empty in RC.conf. So these are the flags that you want to pass to your demon on startup. So I don't know, minus F, ATC, blah, blah, blah. Then we have a demon R table, which allows to run the demon in a particular routing domain. Routing tables allows multiple lookup tables for routes, and they're usually used for police-based routing. I'm not a network guy, so... Then we have a demon timeout. We realized at the very beginning of RCD that if you make a typo in your demon flag, like for example, you forget to pass the minus D to demonize the demon, then you can fuck up your startup sequence quite badly. So we introduced a default timeout. If the process hasn't returned anything after 30 seconds, then we consider it failed, and we keep... We put it in the background and keep running. It's also used for the opposite. There's quite a few processes, like Squid or MySQL, when you have a huge database that can take actually a long time to stop, especially Squid actually on OpenBSD. So sometimes when you want to properly stop the demon, that you may need to increase that timeout. Otherwise, it will be a forced kill. And the last variable we have is the demon user. So it defaults to route, and that's the user that the demon will run under. So that is what SU will use as a user. So all these variables, they are overwriteable, obviously, by the RCD script itself, which usually contains the default for that particular demon. It is overwriteable by RCConf, which is the default OS configuration, and it is overwriteable by STCRCconf local, which is the locally modified configuration. The way that you override a particular variable, so here as an example for flags, is that instead of having demons flags equal, you substitute the demon part with the RCD script name. So the first line here, demon flags, is what's under the NetEsnMPD RCD script. So by default, it will start with minus you, blah, blah, blah. If you want to please Henning and remove IPv6 support, then you edit RCConf local and you will do IPv6, and minus A is for looking support or I don't remember. So that's the way to override any variable. Underscore class, underscore flags, underscore users, et cetera. Actually a light again. Yes. The way you override the demon class is a bit different. Here we'll not be using environment variable. RCD will automatically check if there is a login class that has the same name as your RCD script. So if you have a login class that's called asshgd, for example, or netesonMPD, and that example, it will use that from login.conf. If there is no matching, then it will fall back to the default login class, which is demon. So that was a simple way to add the BSD login class support. So here's an example of an RCConf local. So APMD flags equal minus A. APMD being a base system demon means that we will be starting it with minus A. We will be starting hotplug with no flags, and we'll pass some flags to the same D demon. But same D is an external package. So just putting that line here will not start it. We have to put it in here as well in the package script variable, and that's how the system knows that it has to run that script. And we don't want to run NTPD for whatever reason. Yes, it's bloated. So there are also a few special cases that we have to take into consideration. We have a notion of what we call meta RCD script. Those aren't really RCD scripts. They're just regular shell scripts that basically run other RCD scripts in the order that you pass them. It's used typically for, yeah, like Samba, you can start SMBD and NNBD, but there are also applications like Bacula where you want to run the director and the client on the same machine and etc. Or there's a RAFRA group where it has like seven or eight different demons. You don't want to have to enter all of them. You just have a meta script that will run all them in the sequence. Another special case, and this one is hard, is having multiple instances of the same demon. It works most of the time, but it depends on a lot of different details. The way we do it right now is that we link the original RCD script and we rename it to something. The reason we're linking and not copying is that package script are part of the base system, so if they get updated, then at least your link will be updated as well. Sorry. So that works for most of the demon, but there's a lot of package out there that changed their process like using setproc title or stuff like that. And then it's very hard to actually match the correct one in the process list. If a demon changed like some sample, you pass foo bar, minus VEP, blah, blah, blah, foo foo, and setproc title to just foo bar, then if you start two of those, there is no way that RCD will know which one it has to stop. That's because, again, we're not a process supervisor, so that's a drawback of the implementation. Sorry? No. Can you define a secret script? Yes. Well, I'm actually going to talk about that in a few. So let's have a look now at the internal of RCD, so RC.subber, that's the entry point of the whole framework. That's where we define the default function that will be used by all RCD script. All this function can be overwritten by the RCD script, as I mentioned earlier. If you don't want to send the hub to reload your process, but because that doesn't work and needs some other procedure, then that's where you're going to do it in the RCD script. Sorry. So the first function that RC.subber provides is RC on the score start. What it does? It's RC exact, demon, demon flags, and the bg variable. RC exact can be two things. It's basically the SU line where you pass it to a demon class, then the shell, and the user you want to run the command as, and then the command itself. If we define a different routing table, then we prepend a call to root minus t before running the SU line. RCvg, it's a small variable. If you set that to yes, then it appends enough ascent. That's for demons that are not able to fork themselves into the background. There's actually quite a lot around there, and I don't think systemd will help with that. So that's an example of how SSHG is called by RCD. We have RC stop, which by default will run pkil against the demon root table on the pxp. When you use minus xf, because we want to match the entire process and arguments and match exactly that. The pxp is the important part here. Pxp is what we use to actually match the process that we want to handle. By default, it's set to the demon variable, so the path to the demon. If there are demon flags, then space and the demon flags. That's how by default we match demons that we want to handle. One little thing that I want to mention is that actually down, the base systemd script are not handled by RCD. They're just being sent to sikterm, because all the base systemd demons on openBSD are supposed to behave properly when you send them to sikterm. So there is no need to do any specific shutdown procedure. Then we have RC reload, which by default will send a hub to the process, so the pxp. So that function is responsible for reloading a demon. OpenBSD reloading for us means reconfigure yourself. If minus hub does something else, like just reload but without reconfiguring or does any other crazy stuff, then we consider that it's not a reload and we disable that function. We'll see how a bit later. Then we have RC check, which is just a pgrep against the pxp, so the process list. That returns a code of zero when the demon is running and one when it's not. Then we have a few optional functions, RCP. RCP will be run by the start action, but before running RC start. Typically, it's used to create non-privileged owned directory to store whatever socket file or things like that. Then we have the opposite. We also have an RC post, which is run by the stop action after RC stop. That's typically used when a demon is not able to properly terminate itself and leaves a whole kind of stupid files around. For example, putting the system back into a Christian state, like Cups is a good example. When you start Cups, it will actually move away the traditional LPD command that you have in your path and replace them with the Cups ones. Of course, on stop, it will do the opposite. It will put back the original LPD commands. The last function that RC.subber provides is the RC command. That's the main entry point where you actually pass the start, stop, reload, etc. action. That is, and must be, the last command that an RC will call. Let's see how this command works. When you pass the start action to RC command, first, it will check that the demon is enabled. It will basically run RC check. It will check if it's not already running. It will run RC pre. It will run RC start. Then it will put all the demon variable under this directory, for an RC script name. I'm going to talk about that in five minutes. There is a reason we're doing that, unfortunately. Then, of course, it will wait up to the time of seconds to make sure that the demon is probably started. RC command stop. RC check that the demon is running. Then we run RC stop. Then we wait up to the time out again. Then we run RC post. Then we remove the file that we created with RC start in Varvoo and RCD. We start, as I said, just Fubar stop and Fubar start. The RC command reload. We check that the demon is running. If it is running, then we run RC reload on it. RC command check is just running RC check. Nothing really interesting here. As I mentioned, there is some action that some demon cannot handle. A good example is reload. There are a lot of demons that are not able to reload themselves. They have to stop first and start or restart. In this situation, what we do is that we transform, we use the function name that we put into a variable and set it to no. If your demon cannot reload itself, you put RC on the score reload equals no into the RC descript. If it cannot, I don't know, start, which would be stupid, you put RC on the score start. We have the RC user check variable. As I mentioned earlier, check is the only action that does not require a higher privilege to be run. That is not true in some situations. If I remember correctly, there is a couple of stuff like Samba, for example. I think we use SMB control, something like that, to check whether it's properly running or not. That has to be run as would. In which case, we add this to the RC descript so that when you try to run it, a regular user will tell you that you cannot do that. A little word about the file that we put under the var run RC script name. This file is used to actually have the current status of the demon that you want to manage. The reason is that RCD is dynamic and real-time. That means that if you run a script with RCD, it will check real-time in the process list. It will check the configuration as it is now and check the process, et cetera, et cetera. Say you want to change the flags of your process. You want to start NTPD with minus S, for example. You will edit ETC RC conflocal and set NTPD underscore flags equals minus S. Then you will try to restart NTPD. Of course, RCD will try to match the process that has a minus S in it. When this process doesn't exist yet. That's why we're storing all the information about the currently state of the demon running. It's not great. Here is a complete template of everything that can be done with an RCD script. I'm not going to review everything because we went there already. We can see the three mandatory variables. Just one thing I want to mention is that the demon is, as I said, the path to the demon that you want to start, but we now also force people to add the flags that demonize the demon in it. If your demon needs minus D to demonize, if we put that in demon flags instead of demon, if you want to change the flags of the demon, there is a good chance that you will forget about putting back minus D. You will just think about putting back the new flags that you want to add. It's just a precaution. That's basically the extent of the RCD system. At that point, we were quite happy with how things were working, but there was still one thing missing. At the time, we were using a lot of automation tools like Puppet and things like that. While it was nice to tell Puppet to be able to run ETC, RCD, blah, blah, there was no easy way to enable a demon or disable a demon or change its flags or whatever. We needed a tool that would handle that. I came up with a tool called RC Control. At first, it was meant to be an RC Conflocal Editor. RC Control enabled, blah, blah, blah, would actually modify RC Conflocal with the proper configuration. But it was so easy to actually make it like the service command on Linux and FreeBSD and NetBSD, that we actually implemented that as well. You can also handle your demons using RC Control, RC Control, Restart, whatever. It's like a big mix of the service command, the check config and the SysConfig Editor that you have in Red Hat Land before SystemD. It's an alternative, of course. You can still use VI or Emacs or whatever you want to edit RC Conflocal. It's not mandatory to use this tool. It's just useful. It wasn't must have to any automation tool or monitoring tool, things like that. I also want to mention that we're not service, the command service, compatible in the way that we handle things. We don't do RC Control SSHD start. We do RC Control start SSHD. The reason is that I mentioned that initially we only wanted to be an RC Conflocal Editor. Before RC Control, the user was still kind of confused because when you tell them, they will ask you, how do I enable multicast on my machine? You just put RC Conflocal, multicast equals yes. Okay, that's easy. So for SSHD, it's the same. No, no, no. It's different because that's a demon. So it's not how it works. You have to put SSHD on the score flags. Okay. So why can't I do multicast equals and nothing? That should also enable multicast. No, it's different. That's a service. So on and on, it was kind of confusing. And RC Control provided a unified interface for that. RC Control enabled multicast or RC Control enabled SSHD that will do the perusing and you don't have to worry about whether you need to append underscore flags or whatever. It's also useful because if you enable a script from external packages, it would also append it to the packet script variable to start it. So it provided us with a unified interface. It was developed to abstract all these specific cases that are mentioned. It also replaced a whole bunch of code in a pop-up solve and ansible that were manually dealing with RC Conflocal. So you have an example here. That's what Jasper at OpenBSD did for pop-up implemented the original RC Conflocal implementation and when RC Control came up, then it was able to delete most of the code. So it will abstract all the stuff like daemon versus service. You don't need to care whether it's a daemon or a service. You don't need to care whether it's a meta script or a regular script or whatever. By developing RC Control, there is something that looked very fragile to us at some point is that RC Control will source RC.subber to get all the RCD functions that it needs. It will then run the RC script which will source RC.conf and RC Conflocal which, and then the RC script will also resource RC.subber. So it looked like a bit of crazy sourcing. So what we did is that we implemented a function called RCParseConf which would basically parse the configuration in RC.conf and RC Conflocal instead of sourcing. Some people were not happy about it because they were doing some crazy stuff in RC Conflocal, mainly putting crazy shell code in there which you were not supposed to do that, but of course, since we were sourcing, that worked perfectly for them. So we went from sourcing to parsing and we introduced a small variable as well to be able to only parse for the RCD functions instead of parsing the entire framework each time we need to have a function from RC.subber. The way RC Control works is, I think, pretty easy. You give it the action that you want and then you give it the name of the service. So you can get service, you can get the service flags, you can get the entire service configuration, whether it's started, whether it's not started. The second line is basically like running the RCD script by yourself, like RCctl start SSHG will in effect run etc, rcd, sshg start. And you can order a daemon. This only works for packet script daemon because all the base system daemon are hot coded sequentially in etcrc. You can get information about what service or started, etc, etc. So here you can see a few examples. You can enable 3D daemon at once if you want. I want to enable multicast message bus in CubsD. That's completely transparent. I want to change the ntpd flags. I want to restart 3D daemon at once. That's way more convenient than having to run three times a different script. You can have a list whether you're running daemon. And we also have a small command called RCctl ls failed, which will list the daemon that are enabled in the configuration but that are not running. This is run every day by the DAIRL script on OpenBSD. So it's not like a supervision system or whatever, but it can be useful from time to time to make sure that all the daemon that are supposed to run are in fact running. So to conclude, RCD is really a compromise between features and isabuse. I think very simple. It's not suitable for all the crazy situation out there. It does has is drawback for sure. But it's OK. It was a design decision from the very beginning. So all the obvious deficiencies were actually known from the very beginning. So it's not a replacement. It's not a process control framework. It's not a service supervisor. Service supervisors are really fine. They actually use some of them, but only with specific daemons. So yeah, it's a compromise. I think we succeeded in making it horribly simple and robust. We preserved the original paradigm. All the old dudes were still happy with the way things worked. It's been built on decked old components. We're just using very basic tools, P-Kill and all the suite, SU. We provided a consistent interface with RC control. And it's also easy to integrate into other operating systems. Actually, we came up with a small homemade Linux distribution where we used the OpenBSD RCD system in it. It worked just fine. I think there was just a couple of things to change because we're using KSH on OpenBSD and there's Bash on Linux. I think there's one or two substitutions that were not working with Bash. But that's pretty much it. It worked out on the box. So that's it. Thank you very much. Any questions? Yeah. I think all the designs are just so much better than the other ones. The problem is that if you store a pit, then you have to end all the pit-changing. And sometimes it's racy. Sometimes it can be racy. Sometimes you have demons that restore themselves. I'm not, that's, it's something that's on my to-do list, that's been on my to-do list for forever, basically. That problem we want to be able to solve. It's just using PID is not the right answer for us. And I don't have the right answer yet. I'm still thinking about it. I had a few ideas a few months ago which I talked about with my fellow OpenBSD developers. But I wanted to inject some stuff in the process list. As Bob Beck said, the kernel is not a fact system. Yeah. Yes. Yeah. Yeah. Well, the problem is that if you do that, I mean most demons are super chatty. So you will overflow your whole console with messages that you don't want to have to see. So that's why it's not a default. It's easy to enable when you need it. And if something doesn't start, then you can manually run with minus d and c, and c y. Yeah, but that's what I'm saying. At start, we will end up with a huge amount of stuff. Not by default. But that's not a default. Yeah. There are a few stuff that throw out. And yeah, but I remember that there was a few, maybe that changed since now, but I remember there was a few. And actually it doesn't, it doesn't, I mean the fact that it was super quiet on startup was also a requirement because that's how the thing were done before. And we were actually telling whether the RCD script run okay or not. But we had to revert that because people didn't like too much information. So yeah, outputting the entire stuff on the console by default. I'll just get screen back. Yeah. Yeah. Yeah. Yeah, yeah, yeah. Something doesn't, doesn't, doesn't start. And it will tell you it failed. Yeah. Yeah. Yeah. But to be honest, most of the time when something doesn't start, it's logged and see it's logged. Yeah. Yeah. But I mean, that's, that's how you want those stuff. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. That was actually someone actually suggested that a few months ago. But actually the guy who suggested that didn't realize that his failure was logged in, see it's logged. Pretty reticulous. It wasn't going to help anything. Yes. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. No. No. Yeah. We could, we could add a couple of checks. Yeah. We could add a couple of check, but I mean, I don't think we'll be able to cover all the crazy stuff. So what it already does a couple of stuff is sort your fight, it removed some quotes where they're not needed stuff like that. But we don't want to do too much because again, it's shell. So parsing and all is already tricky. But you can't have it all. I mean, it's super small, super simple. That's, yeah. Okay. So, right on. Thank you. Yeah.
Antoine Jacoutot and I'm going to talk about our started the implementation, how to use RCD, the internals of RCD and I will introduce a small on the traditional system that we used before it. services and demons was because we were using a yeah. So the way Open BSD boots hasn't changed much since its initial inception. It's actually dependable. It's predictive, it's sequential, It worked before RCD and that is how it also preserve existing behavior. So basically what we we had a quick look at the SMF from Solaris, and Linux RCD implementation was the one from factorization between the RC script so they're enhance the fact that we did not want to start from scratch. It just happened because we I mentioned no automatic startup ordering. The write a home script for that. And for better or system has no knowledge of a default it in here as well in the package script that basically run other RCD scripts in the order xf, because we want to match the entire process transform, we use the function name that we put list. It will check the configuration as it is SysConfig Editor that you have in Red Hat Land have to any automation tool or monitoring tool, That's a service. So on and on, it was kind of It was developed to abstract all these specific functions that it needs. It will then run the RC start. And you can order a daemon. This only script on OpenBSD. So it's not like a supervision with RC control. And it's also easy to integrate we used the OpenBSD RCD system in it. It worked