The Anykernel and Rump Kernels
Kantee, Antti
This talk will introduce the concepts of the anykernel and rump kernels, motivate their existence, and show a few cool tricks that are unique benefits. We will not go deep into technical detail - basic knowledge of operating systems is enough to follow the talk. The anykernel is a new way of looking at kernel architecture. Conventional models such as the monolithic kernel, microkernel and exokernel dictate the execution model for kernel drivers. In contrast, the anykernel states that any of these should be possible as a runtime choice. A rump kernel is a lightweight virtualized driver execution environment. Typically, OS virtualization means that the application execution environment is virtualized. In contrast, the rump kernel is designed for ultralightweight kernel driver virtualization. Benefits of using rump kernels include millisecond bootstrap times and a small memory footprint. A production quality implementation of the concepts is available in NetBSD 6. The implementation lends itself to various use cases, such as isolating kernel drivers into separate servers on an on-demand basis and reusing kernel drivers in a library fashion such as for accessing file system images. In addition to providing a basic overview of the concepts and benefits, the talk will introduce recent developments. An example of such is a script for building rump kernels for non-NetBSD platforms which allows NetBSD kernel drivers such as the TCP/IP stack to be leveraged on platforms beyond NetBSD. As a demonstration, the NetBSD kernel FFS driver will be run in a browser. This is accomplished by compiling the driver with a C-to-javascript compiler and running a rump kernel on top of a hypervisor provided by the javascript environment.
 Good luck. Okay, thank you. So, as everyone can probably clearly see, the title of this talk is the Anycernel and Rump Kernels. The basis for this talk will be formed by what I did for my doctoral work, which I finished a while ago. But when I say doctoral work, don't worry, it doesn't mean that I'll have a lot of theory, I'll have a lot of Greek equations, I'll have a lot of stuff like that. This will be a very high-level, undetailed talk. And if you want to plow down further into details, my dissertation is linked from the talk website, so you can find a few hundred pages of details, approximately this much here's a physical copy. But even that isn't written in Greek, it's more written in a hackery fashion, so I discuss more the implementation and those kind of issues. So that's kind of the basis for my talk. Well, the implementation that I did for my doctoral work is for NetBSD, it's written in production quality code, it's part of the NetBSD6 release. And what I've been doing after graduating, I've kind of been working on making it more accessible to people. Let's make power accessible to the laptop. Yeah, so I'll show a lot of demos today, and everything I show, if you have a sensible open source system installed like, let's say, Linux or freeBSD, you should be able to repeat on your system. And I'll even tell you how you can do it, or at least I'll give you the pointer. So just to recap the main idea, the main idea of this talk is to introduce the subject, tell you what you can do with this stuff, and not so much go into details, but give you an understanding of if you want to know the details where you can find them and how you can play with this stuff. Okay, so let's just simply start off with the demo. Like I said, if you have any sensible open source operating system, you should be able to repeat what I'll show you. Well, unfortunately, this isn't a sensible open source operating system. This happens to be Windows. No, don't worry, I'm not at the wrong conference, but let's not allow this inconvenience to get to me. So what I have here is I have a directory which has a file system image. At least file claims that it's a Unix fast file system image. Now let's assume I'd want to read this image. Well, Windows doesn't at least natively support this file system, so am I at a luck? Do I need to install a virtual machine with NetBSD on it to be able to read this? Well, let's see. So, yeah, I have a file system image, then I just have a funny test file which just contains the text on file. Then I have a short bit of application code which does a few very simple things. It bootstraps a virtual kernel on this line here. It does a bunch of other things, creates a directory within the Rump kernel namespace, mounts a file system within the Rump kernel, changes the current process in the Rump kernel context into this directory, and then it calls this routine myread file twice. Once it passes function pointers to host system calls open and read, once it passes them to Rump kernel function pointers, and finally it reboots. Yeah. And what myread does, it opens a file here, it reads it, and then it prints the contents. Okay, so let's compile it. And then let's run it. Now, what this program will do is it will bootstrap a virtual kernel, mount a file system in it, read a file using the virtual kernel, and reboot. So how long do you think this process will take? Ten seconds, five seconds, a second? Well, it actually depends on if SIGWIN is merciful. So that's how long. And most of the time we spent loading the thing into cache. So if I run it a number of times, sometimes SIGWIN makes it run faster, sometimes it doesn't. But anyway, it's a pretty instant operation for... Let's try to adjust this a bit. Okay, that's better. So anyway, bootstrapping the virtual Rump kernel, mounting a file system, reading a file, rebooting the Rump kernel, that's a pretty instantaneous operation. And what happens here is I'm actually using unmodified NetBSD kernel code in this virtualized kernel. So what else can we do with this? Well, this is something which is running in user space, so can we run GDB on it? Sure, why not? I mean, let's put a breakpoint somewhere like into the kernel's FFS file system read method. Let's run this thing. And we hit the breakpoint, we can examine the stack trace just as we could in a normal application. We can single step it. We can print the values of arguments and set the right, set the right, set the right. So that works. Well, okay, so now we have a virtualized kernel running in user space with unmodified kernel code working very fast. So of course, now immediately already if you're a kernel developer, you'll maybe start thinking, hey, could I actually use this stuff for my advantage? Can I actually develop and debug code with this? And yes, you can. And iteration is really, really fast. Since running it takes a fraction of a second. And since it's virtualized, let's just do a small experiment. So what would happen if this is the same read method we put the breakpoint into previously? So let's just put panic there, because panics are always fun. Everyone likes kernel panics. Let's recompile the file system component, install it, then let's rebuild this and run it. And now we got a kernel panic. But of course, our host system wasn't affected since everything is running in a process. Yeah. So what happened here? So this is kind of now, you know, I showed a demo in the beginning to give you some concrete idea of what the talk is about. And now I'm actually going to explain what the hell happened. So this is kind of some sort of fancy picture of what was going on. So this entire screen represents one process. So we have the application layer on the top, which was the source code I showed you. That's the one which decided to where we bootstrap the Rumpkernel. We mounted the file system, we read the file system contents and so forth and so forth. And you can see there's the blue arrow which goes directly from the application to the host, and there's the green arrow which goes through the Rumpkernel. Now, we called the MyRead file routine twice. Once it read the host, read the file using host calls, and once it read it using Rumpkernel system calls. And most of the stuff in here is unmodified netBSD kernel code. Actually, it can even be unmodified netBSD kernel binaries on some systems. Like if you host this, I'm hosting this on Siegwin so it won't work because the object formats are different. But if you're, for example, hosting a Rumpkernel on a Linux system, you can load netBSD kernel code into it just fine. And the Rumpkernel runs on top of a very lightweight hypervisor, so you could say that all netBSD kernel drivers have been virtualized now. And the hypervisor is the guy who makes the calls into the host. So this Rumpkernel bit is completely self-contained. It can only access the world through the hypervisor. And the hypervisor, it's not a hypervisor in the normal virtualization sense, like where you virtualize an operating system where it's a, the hypervisor takes care of low-level fiddly bits like maintaining page tables and so forth. This guy can pretty much just do create a thread, allocate a page of memory, and then access some IO routines. And yeah, this is a single, or in this case it was a single process, runs on a host. It doesn't have to be a comical host, but this host happened to be comical, so that's why I depicted it as comical. Okay, so that's what happened now. I'll backtrack even more and I'll explain the actual concept. So someone asked me before to talk what is an any kernel. So now I'll actually tell you what's an any kernel. And any kernel is just a kind of tongue-in-cheek term I invented. You know how operating systems, you usually wonder, is this a microkernel operating system? Is this a monolithic kernel operating system? Is this an exo kernel operating system? Does this use partitioned kernels or whatever? And my view on this problem is that it's irrelevant because you can write your operating system so that you can choose at runtime which model you want to use. And one very important thing to understand is I'm talking about running the same code, the same drivers, line by line. You don't need to rewrite drivers. I'm not talking about the actual possibility to interconnect some components. Like can you attach or can you mount a file system using a user space server or a internal file system? That's not what I'm talking about. That's the kind of request transport method. I'm talking about the actual driver implementations. So simply put, the any kernel is a way to write kernel drivers where you're not restricting yourself to a monolithic kernel, a microkernel or whatever. They all just work. And well, if this sounds like it wouldn't be possible, well, you saw it running. You're welcome to look at the code and tell me that it doesn't work. So that's the concept of the any kernel. Well, moving on from any kernels, the other part of the topic of the talk was Rump kernels. Now, what is a Rump kernel? A Rump kernel is simply a virtualized instance of a driver when it's not running inside the monolithic kernel. So why don't I just call it a driver then? Well, a driver requires some amount of supporting functionality to work. So someone needs to schedule the driver. The driver needs to allocate memory. The driver needs to do locking. The driver needs to do this and that or the driver needs to be able to do this and that to actually function properly. And that's where the Rump kernel comes in. The Rump kernel provides the necessary set of support bits for running the driver on top of the lightweight hypervisor we saw or which was depicted in the previous picture. A Rump kernel is component oriented in the sense that as you saw, if you were paying attention when I linked the example, there was just one C file and then there was a huge amount of libraries. And the libraries were the components that the Rump kernel was built from. If you looked very carefully, you saw that there was a VFS component, there was the driver component, there was a disk driver component and a few bits like that. So the idea is pretty much the same as with kernel modules. You compile a bunch of code into libraries or modules or whatever you want to call them. When you want to instantiate a Rump kernel, you just link stuff together. Do you need to link them together statically? No, you can do it dynamically too. There's actually a utility in NETBSD called Rump Server which does all of this for you. So to recap, the any kernel is the possibility of running kernel drivers in any configuration feasible, the same code. That's the whole point. You don't need to go rewriting drivers. And the Rump kernel is the lightweight virtualized instance. Okay. So let's do another demonstration because demonstrations are fun. They give an opportunity for things to go wrong. So we did file systems. Let's do TCP IP networking this time. So what I'll do here is I'll run two NETBSD kernel TCP IP stacks and have them talk to each other. This whole thing runs completely unprivileged and how it works is pretty much as the first example. You have a Rump application layer. You have a Rump kernel which makes, sorry, you have an application layer which makes some sort of calls into the Rump kernel. Then drivers do stuff. In this case, the driver is the TCP IP stack. And the bottom layer of the TCP IP stack is the interface driver here. And here I've implemented the interface driver which allows multiple different Rump kernels running on the same host to communicate with each other using shared memory. So this is something that would be and could be pretty neat for TCP IP implementation testing. Let me just adjust this. Yesterday I wanted to check that all of these terminals are the right size. Then I checked it. Then they told me that they'll change some things. Now they're all the wrong size. Okay. So in this directory we have, well, again, a make file. Then we have a source file called net worker which just has two routines. Both bootstrap, a Rump kernel, open a socket. One starts listening on it and, sorry, well, this one obviously doesn't start listening on it. This one does a connect to the server. This happens to be the address I configure for the server. Does a connect and then it just reads whatever the server sends and exits. And then the server conversely opens a socket, starts listening on it and accepts a connection and then it writes a test string into the socket and then exits. Of course the text string is conveniently non-terminated so the client doesn't have to bother with non-termination. Very good networking programming practice. But this is demonstration code. So, okay, let's compile it. So now you see a different set of components linked together. For example, you see you have the network driver component and you have the TCP IP stack. You'll notice that there's a very distinct lack of anything related to file systems in this example because this Rump kernel has absolutely zero file system support. Where's my other terminal? It's here in a very weird size. Now it's even a weirder size. That's why I love doing demos on Windows because I can blame everything on Windows. Okay, so let's run the server. Then we run the client and then it worked. So what actually happened here? So we have this, I told you it uses shared memory. The shared memory handle is a file name which is memory mapped into both Rump kernels and if you examine it, for example, dump the hex, you can clearly see the network traffic which was going on. Well, even I would admit that that's a bit annoying to read. Now conveniently, I wrote a tool which can convert this into the pick up format so you can actually open it in Wireshark. And there you can see the network or the exchange between the two stacks. So there's some initial bootstrapping spam which TCP IP stacks usually do when you configure an address and then you actually see the TCP exchange here. The SIN was sent and the SIN act and the ACK and here the server wrote the data to the client and we can see that we have this. Strange sensation that we saw BSD networking stacks running on Windows earlier too. And well, these two processes are completely independent. So I can again do something like run GDB on the server. Let's put a breakpoint into TCP input. Let's run the server. Then let's run the client. Then of course we get nothing back from the server because the server is now stuck in the breakpoint. We can trace it. We see that we came somewhere through an IP soft interrupt. You know, we can single step this server. The client is waiting here or maybe timing out. We can again print values like we can see that the source address of the IP packet that the server is processing is what we expect and so is the destination address. So this example was about showing how you can run the unmodified NetBSD kernel TCP IP stack without any privileges in user space. Now how far does this scale? I showed you two. Well, I've personally done up to, was it 255? So I bootstrapped 255 TCP IP stacks in Rump kernels and I configured their routing tables so that they were just, they had a linear topology. So you know, this guy could talk to this guy, who could talk to this guy, who could talk to this guy, who could talk to this guy. So why 255? Well, that's the maximum value you can give the IP time to live. So you know, I didn't have an imagination to come up with a more and more difficult test. So my test was start this number of TCP IP stacks, configure networking for them, configure routing and send a packet from the other end to the other end and wait for it to come back. So how long do you think this whole process took? Starting the clock from when I started running the TCP IP stacks and stopping the clock when I received the response. Well, I actually did two versions. One version I did with a shell script using tools which you can find from a standard NetBSD installation. The script is actually documented on some page in the dissertation. So if you're interested what went on there, you can find out that took somewhere around eight seconds. Then I did another version. I optimized the process a bit and I got this kind of, I got this amount of TCP IP stacks bootstrapped and a thing sent from one end to the other end in about a second. So if you're doing massive network testing and whenever you want to restart your cluster, it takes a while, then you're maybe doing something wrong. Okay. Would anyone have any questions at any point or everyone's busy trying out the stuff I showed for themselves? No, I can put the source code somewhere on the demo source code. All of the other, the driver source code is obviously out there but I can put the demo source code on the net if someone kind of wants to start experimenting and use this as a basis for their experiments. Okay. So I've shown you file systems. I've shown you TCP IP networking. What about hardware? Is hardware supported? I don't like hardware. It's very difficult because you can't virtualize it. You actually have to own the hardware. My hardware budget for this project was five euros I used it to buy a Bluetooth USB thing because I wanted to support USB hardware drivers because that allowed me to test a lot of other things like it allowed to test me Bluetooth, it allowed me to test the scuzzy mid layer, it allowed me to test parts of net 802.11 support, et cetera, et cetera. Now, would hardware be impossible to support? No. I think if you really wanted to and you started hacking now, you could have something limping along by the end of the conference or maybe you can start after my talk, I don't know. So things you just probably need to do is somehow map the device registers into the Rumpkernel, figure out how you deliver interrupts there and figure out what you want to do with the DMA. It shouldn't be difficult but I just haven't done it. Okay. So both examples I've shown you so far have had a one process model. So the application was in the same package as the Rumpkernel. Now, this is very convenient for things like let's say you want to use unmodified net BSD kernel drivers on an embedded system because you can't be or because it's just a lot of effort to implement them yourself. So hey, we have these drivers which can which are ported on top of this convenient hypervisor. Let's just slurp them into our embedded system. Our embedded system doesn't even support virtual memory, doesn't support the multiprocess model. Well, that doesn't matter. Rumpkernels will run just fine. But for other purposes, let's say this little thing called network configuration is typically implemented in the sense that you have a server which in the normal cases is your kernel. You run one command. It does something to the network configuration. You run another command. It maybe modifies the routing tables a bit, et cetera, et cetera. So it's quite inconvenient if the command and the Rumpkernel run in single process because when the command exits, the Rumpkernel goes away too. So you won't get very far with this approach. But does this model actually work where you have a separate server and a separate client and they somehow communicate with each other? The short answer to that is yes. The long answer is a bit longer. That's why they call it the long answer. Okay. So third demo. Everyone's still awake. So now this time we have two processes. We have a server process and we have a client process and they're going to talk to each other. I'll just show you the code for the server because it's called simple server and it's really simple. So what it does, it bootstraps a Rumpkernel. It initializes some listening address where the kernel is waiting for clients to send communications and then it just pauses. So if we run that, then if we run the client, actually, we need to tell the client that the server is listening here. This could also be a... It doesn't need to be TCP. It can use local domain transport and actually this thing can run over anything. I do have factual knowledge about that. But let's see, there's like getPID which is very interesting. So every time we run it, we are in a different process. If we kill the kernel, restart the kernel, the numbers start from... This is a very interesting example I can see. Let's do something to modify the state a bit. Let's create a directory called mount. If we try to create it again, it fails because it exists already. We can... There's something called readFile which can read a file. So let's try to read something from the mount point. Well, opening the file fails because it doesn't exist anymore. This demo client, we can mount the kernel file system into there and after that we can actually do readFile. And as usual, in my demos, GDB usage is mandatory. So this time let's put a break point into copy out. If you know a bit about how things work between an application and the kernel, the kernel always copies everything in and out from application, the application address space. Now in this case, the application is in different process. So we again run it and actually now we need to execute... Oh, now we rerun it. So let's just remount the file system and try to read. And now this obviously hangs because the server is hanging in a break point and you can actually look at this on your own time. So that's remote system calls. This is kind of another way to put it. You have a kernel server somewhere and you have applications which communicate with the server using some method of transport. For example, local domain sockets are extremely convenient for kernel testing because you have unlimited namespace between... Or for binding the socket so you're not limited to the amount of TCP IP ports. You don't have to do any negotiation if you run... I don't know. If you want to run 10,000 tests in parallel, you can run 10,000 tests in parallel. Well, whatever. So usually when I present this, a lot of people tell me, aha, but you're reinventing plan nine. So then I ask them, how can I reinvent plan nine using unmodified unique system calls? So that's again something to think about. Now let's talk about something else. The any kernel is interesting in the sense that it allows the user or the operating system administrator to decide if now is a good time to trade performance for stability or so forth. So one argument you hear often is performance doesn't matter anymore. But for some reason you never hear anyone say it doesn't matter if we need to buy 10% more hardware into this server form because maybe performance does matter after all. There are some cases where stability is more important than performance. There are cases where performance is more important than stability. And this just allows you to choose yourself. The evil operating system implementers don't have that power over you anymore. So you can choose whatever. Well, then let's talk quickly about the performance of Rump kernels. How fast do they run? Well, all the code runs as native code. There's no kind of the any kernel guarantees us that there's no privileged instructions in a driver code that's been all abstracted away. So it runs the same speed as native code. So it boils down to how fast you can do IO. If, for example, you run a file system driver, how efficient is your communication with the disk driver? And in some tests I've run the performance was worse. In some cases it was better because in the disk driver I was using in user space, the caching behavior was slightly different from the one in the kernel. So you actually got better performance. Well, another way to look at this problem is kind of not from a micro optimization perspective, but from an algorithmic perspective. Developing kernel code with literally zero iteration time. First of all, it allows you to try out wild ideas. So you know, what if we do this crazy thing or what if we do this crazy thing and very quickly see how it affects things. Second of all, since you don't run anything, everything in a monolithic kernel anymore, you don't have to worry necessarily about like, hey, let's do this hack into the TCP IP stack. So what kind of effect does it have on every possible user of the TCP IP stack? You can just have your own virtual TCP IP stack dedicated to a certain application and hack the hell out of that. So that's, you know, the bigger picture on performance. Now, just to give an example of things you can do, these partitioned operating systems, which are kind of a, well, kind of a recent trend, tend to run operating system components one per CPU and do message passing between them. And the part of the idea for this is that you don't need any atomic bus locks because you just have one non-conflicting thread running on one CPU. Then when it wants to do something else, it sends a message to another CPU and then something else runs there. So I kind of read about this, then I got interested, could you actually do this with existing drivers, kind of optimize a way locking completely if you limited yourself to one processor? And turns out you can. It's implemented in a source file called locksup, which I at some point realized maybe wasn't the best name, but it's not. Well, then something interesting, or I don't know interesting, but something I did just for fun recently. So, you know, we have these drivers, you can compile them into virtualized environments, you can run them, they're pretty flexible. So I thought, hmm, could we compile these so that they run natively on a web browser? So then I went out in search for a C2JavaScript compiler and some amount of hacking later I had the answer for that tool. Yes, you can. So I don't know if you've seen this, but this is available on the Internet and it's just a RAM kernel compiled into JavaScript. Now, this is pretty much the same set of drivers we saw in the first demo. You can execute things, well, there's a very small command interface. You can execute various commands. And if you actually a, no, wrong button, I forget how this web stuff works. So if you can run it, for example, run the fire bug and then you can take a while. So let's, well, I had a break point in there already. Actually, let's put a break point here into FFS read because that's what I showed. Showed in the, which one is the play button? This one. So this is what I showed in the, this is good. So now we hit the break point and if you memorize the stack trace from the first example, which of course everyone did, you will no doubt notice that the stack trace is very similar because this is exactly the same unmodified NetBSD kernel driver, which instead of being compiled into assembly, it has been compiled into JavaScript and is now being run in a browser and it's very useless otherwise, but it's fun. Okay. So I promise to tell you how you can run this stuff on, if you have a sensible, sensible open source operating system in just in user space. You don't need to install any virtual machines. You don't need to do anything like that. You just get a script called buildrump.sh, which is hosted on GitHub. You get the NetBSD source tree. You run the script and some minutes later you will be rewarded with a lot of libraries. I'll give you the URL at the end of the talk, although I'm fairly certain that you are imaginative enough to be able to find it yourself without my help. So then there's this kind of if our kernel or our Rump kernel is from NetBSD and we try to call it from, let's say Linux or I don't know, Sigwin or whatever. There's this huge potential for type conflicts. You know, some constants might have different values when you pass them to the system calls. They just don't agree. And this is kind of something I've not entirely solved yet. You can actually fairly easily brute force your way around this. Like depending on what you want to do, you just, you apply brute force, which means that you create copies of types. You create copies of constants. You know, like it's half a day of hacking to or maybe a whole day. I can't quite remember. Or maybe it was a night of hacking if you want to reimplement IF config as a kernel component. That kind of solves the type problem in a brute force way, but it's not very nice. Well, anyway, moving on a bit. What this slide is trying to say that if your operating system isn't an any kernel, it sucks. Well, I believe there are so many both internal and external benefits that every operating system, at least any every journal purpose operating system should try their hands at being an any kernel. It's the question, well, the interesting question is how can this possibly work? What proof do I have that it works? Well, that's the proof. Is there any grand theoretical model? Is there any great wisdom behind it? Well, like I like to say, the main reason operating systems work anyway is luck. So this kind of taps into that same principle. It just happens to work. And before you try it, you can't know for sure, but at least at least this thing has been implemented and working in net BST since 2007. So maybe that proves something. I actually tried to come up with an analogy for this. I came with an analogy that trying this is kind of like throwing spaghetti on the wall. And if it sticks, you can take a photograph and call it art. So then I of course wanted to try this. And that night I had a very long discussion with my girlfriend. See, she told me that you're not supposed to throw it on the ceiling. Sorry, the wall, you're supposed to throw it on the ceiling. And that's not for creating art. That's how you test if spaghetti is done. And then she added, everyone knows that. So, well, anyway, so if you don't remember anything from this talk except you test spaghetti is done by throwing it on the ceiling, then I've done something useful. So in addition to converting all of the other operating systems into any kernels, there's a bunch of things you could do with the net BST implementation should you be so inclined. First of all, you can test the build drum script on a platform of your choosing. And if you do so, please report the result. If it works, I'm still collecting a list of hosts where it's been known to work. So if it works, you know, send an addition to the list of hosts where it's known to work. If it doesn't work, please, please at least file a bug report on GitHub or if you can manage a patch, it's even better. Something else you can do, you can package RAMP kernels for your or the net BST kernel drivers for your favorite operating systems, packaging system so that, for example, if you happen to need to read file systems, which there are only net BST kernel drivers available for, you can actually do it afterwards. Well, one of the funny things you can do with this is you can very easily run while grind on kernel code. There's a few rough edges still like examining the string length, which leads into interesting results because string length is a routine which actually the optimized version looks beyond the end of the string for each word. So there's a few recipes which you can maybe use to kind of improve the while grind, report it, and then maybe document them somewhere. Well, one very useful thing would be to use autocon for the hypervisor. So now the hypervisor runs on net BST, runs on JavaScript, runs on Linux, runs on Solaris, runs on FreeBSD, et cetera, et cetera, but I've been very lazy so I haven't done anything fancy. I've just put some if-defs in there and with different versions of different operating systems that approach is kind of starting to come to an end. So if you like autocon, you could maybe apply autocons to the hypervisor and then you can tell all your friends that you used autocon to make a kernel more portable. Then they will maybe look at you in a slightly funny way. But then there's things like authentication support for remote clients. Shortly put, currently every remotely connecting process gets root privileges, which is good for testing, but if you want to use this for something else, it might not be what you want. And then there's the whole issue with type incompatibilities, which if you run this on Linux, I'm sure you'll understand sooner or later. I have a theory that it could be solved automatically with a toolchain driven approach, but that requires someone with maybe more toolchain knowledge and I don't know, maybe it could be if you know someone who's looking for a good master's thesis project, that could be something to look at. Well, that's a pretty accurate conclusion. And you? Yes, question? For Walgreen, you can just use dash F no built in. Yes, but see, okay, so what he said is then it uses the C library string length function, but see the rom kernel is self-contained, so it doesn't use the C libraries string length function. Yes, but you might run into some issues if you try to use the C library past the hypervisor. So everything into the C library should go through the hypervisor. And actual question is what why does it? Why does it use threads? So the actual question was why does it use threads? Well, because the drivers require threads. Actually, you don't need threads in all circumstances, like for example, file systems run just fine without threads. And you can kind of probably fake threads. Let's assume you probably need some sort of concept of multiple stacks, but you don't maybe necessarily need to use threads as we understand them. But yeah, the underlying reason why does it use threads is that because the kernel drivers depend on threads. For example, to deliver network packets. So there's no model for interrupts. Everything always runs completely to finish, and that's for simplicity reasons. Yes? Yes? Sorry? Yes? What new kinds errors? Okay. Okay. So two questions. First question. So if you use remote system calls and it could theoretically introduce some new kinds of failures. For example, well, the thing you said TCP packets could be lost. But it uses TCP stream so that the stream isn't lost. I mean, the stream itself will eventually always, unless the network breaks, unless the kernel crashes. Especially when I was testing this, I, for weeks and weeks, I ran an unmodified Firefox against Rumpkernel TCP IP stack because I wanted to know if it works. And then I thought, hey, it would be cool if I could actually kill the TCP IP stack and restart it and have my web browser notice nothing. So I also added reconnection support to the remote protocol. So if Firefox is loading a page, you see kill the TCP IP stack. Well, then obviously packets don't flow anymore. But if you restart the TCP IP stack, reconfigure it and hit control R, then things work again. Yes. So he said that system calls are always expected to return and the transport code must guarantee that. Well, that's what the regular application expects anyway. Now, the system call might not return, for example, if the system crashes. I mean, that's, you know, but yeah, adding distribution adds a whole other array of problems. So, you know, it works. But if you want to take the problems of distribution into account, then maybe you need to do a bit more work. But that's kind of a use case specific. I mean, you can run an application here and a Rumpkernel in Japan. But I haven't found very many good uses for that because the latency is quite huge. So things tend to go slow. Well, does anyone else have questions? Okay. So you are allowed to ask a question. The second question was how do I deal with signals? So signals are always something that the host provides where you run the application. So, okay, so there's a kind of, well, I don't know how to answer this in a time which will not make the moderator more upset. So maybe we can discuss it afterwards. Okay.
 Good luck. Okay, thank you. So, as everyone can probably clearly see, the title of this  talk is the Anycernel and Rump Kernels. The basis for this talk will be formed by what  I did for my doctoral work, which I finished a while ago. But when I say doctoral work,  don't worry, it doesn't mean that I'll have a lot of theory, I'll have a lot of Greek  equations, I'll have a lot of stuff like that. This will be a very high-level, undetailed  talk. And if you want to plow down further into details, my dissertation is linked from  the talk website, so you can find a few hundred pages of details, approximately this much  here's a physical copy. But even that isn't written in Greek, it's more written in a  hackery fashion, so I discuss more the implementation and those kind of issues. So that's kind of  the basis for my talk. Well, the implementation that I did for my doctoral work is for NetBSD,  it's written in production quality code, it's part of the NetBSD6 release. And what I've  been doing after graduating, I've kind of been working on making it more accessible  to people. Let's make power accessible to the laptop.  Yeah, so I'll show a lot of demos today, and everything I show, if you have a sensible  open source system installed like, let's say, Linux or freeBSD, you should be able to repeat  on your system. And I'll even tell you how you can do it, or at least I'll give you the  pointer. So just to recap the main idea, the main idea of this talk is to introduce the  subject, tell you what you can do with this stuff, and not so much go into details, but  give you an understanding of if you want to know the details where you can find them and  how you can play with this stuff. Okay, so let's just simply start off with the demo.  Like I said, if you have any sensible open source operating system, you should be able  to repeat what I'll show you. Well, unfortunately, this isn't a sensible open source operating  system. This happens to be Windows. No, don't worry, I'm not at the wrong conference, but  let's not allow this inconvenience to get to me. So what I have here is I have a directory  which has a file system image. At least file claims that it's a Unix fast file system image.  Now let's assume I'd want to read this image. Well, Windows doesn't at least natively support  this file system, so am I at a luck? Do I need to install a virtual machine with NetBSD  on it to be able to read this? Well, let's see. So, yeah, I have a file system image,  then I just have a funny test file which just contains the text on file. Then I have a short  bit of application code which does a few very simple things. It bootstraps a virtual kernel  on this line here. It does a bunch of other things, creates a directory within the Rump  kernel namespace, mounts a file system within the Rump kernel, changes the current process  in the Rump kernel context into this directory, and then it calls this routine myread file  twice. Once it passes function pointers to host system calls open and read, once it passes  them to Rump kernel function pointers, and finally it reboots. Yeah. And what myread does,  it opens a file here, it reads it, and then it prints the contents. Okay, so let's compile  it. And then let's run it. Now, what this program will do is it will bootstrap a virtual  kernel, mount a file system in it, read a file using the virtual kernel, and reboot.  So how long do you think this process will take? Ten seconds, five seconds, a second?  Well, it actually depends on if SIGWIN is merciful. So that's how long. And most of the time we  spent loading the thing into cache. So if I run it a number of times, sometimes SIGWIN  makes it run faster, sometimes it doesn't. But anyway, it's a pretty instant operation  for... Let's try to adjust this a bit. Okay, that's better. So anyway, bootstrapping the  virtual Rump kernel, mounting a file system, reading a file, rebooting the Rump kernel,  that's a pretty instantaneous operation. And what happens here is I'm actually using unmodified  NetBSD kernel code in this virtualized kernel. So what else can we do with this? Well, this  is something which is running in user space, so can we run GDB on it? Sure, why not? I  mean, let's put a breakpoint somewhere like into the kernel's FFS file system read method.  Let's run this thing. And we hit the breakpoint, we can examine the stack trace just as we  could in a normal application. We can single step it. We can print the values of arguments  and set the right, set the right, set the right. So that works. Well, okay, so now we  have a virtualized kernel running in user space with unmodified kernel code working  very fast. So of course, now immediately already if you're a kernel developer, you'll maybe  start thinking, hey, could I actually use this stuff for my advantage? Can I actually  develop and debug code with this? And yes, you can. And iteration is really, really fast.  Since running it takes a fraction of a second. And since it's virtualized, let's just do  a small experiment. So what would happen if this is the same read method we put the breakpoint  into previously? So let's just put panic there, because panics are always fun. Everyone likes  kernel panics. Let's recompile the file system component, install it, then let's rebuild  this and run it. And now we got a kernel panic. But of course, our host system wasn't affected  since everything is running in a process. Yeah. So what happened here? So this is kind  of now, you know, I showed a demo in the beginning to give you some concrete idea of what the  talk is about. And now I'm actually going to explain what the hell happened. So this  is kind of some sort of fancy picture of what was going on. So this entire screen represents  one process. So we have the application layer on the top, which was the source code I showed  you. That's the one which decided to where we bootstrap the Rumpkernel. We mounted the  file system, we read the file system contents and so forth and so forth. And you can see  there's the blue arrow which goes directly from the application to the host, and there's  the green arrow which goes through the Rumpkernel. Now, we called the MyRead file routine twice.  Once it read the host, read the file using host calls, and once it read it using Rumpkernel  system calls. And most of the stuff in here is unmodified netBSD kernel code. Actually,  it can even be unmodified netBSD kernel binaries on some systems. Like if you host this, I'm  hosting this on Siegwin so it won't work because the object formats are different. But if you're,  for example, hosting a Rumpkernel on a Linux system, you can load netBSD kernel code into  it just fine. And the Rumpkernel runs on top of a very lightweight hypervisor, so you could  say that all netBSD kernel drivers have been virtualized now. And the hypervisor is the  guy who makes the calls into the host. So this Rumpkernel bit is completely self-contained.  It can only access the world through the hypervisor. And the hypervisor, it's not a hypervisor  in the normal virtualization sense, like where you virtualize an operating system where it's  a, the hypervisor takes care of low-level fiddly bits like maintaining page tables and  so forth. This guy can pretty much just do create a thread, allocate a page of memory,  and then access some IO routines. And yeah, this is a single, or in this case it was a  single process, runs on a host. It doesn't have to be a comical host, but this host  happened to be comical, so that's why I depicted it as comical. Okay, so that's what happened  now. I'll backtrack even more and I'll explain the actual concept. So someone asked me before  to talk what is an any kernel. So now I'll actually tell you what's an any kernel. And  any kernel is just a kind of tongue-in-cheek term I invented. You know how operating systems,  you usually wonder, is this a microkernel operating system? Is this a monolithic kernel  operating system? Is this an exo kernel operating system? Does this use partitioned kernels  or whatever? And my view on this problem is that it's irrelevant because you can write  your operating system so that you can choose at runtime which model you want to use. And  one very important thing to understand is I'm talking about running the same code, the  same drivers, line by line. You don't need to rewrite drivers. I'm not talking about  the actual possibility to interconnect some components. Like can you attach or can you  mount a file system using a user space server or a internal file system? That's not what  I'm talking about. That's the kind of request transport method. I'm talking about the actual  driver implementations. So simply put, the any kernel is a way to write kernel drivers  where you're not restricting yourself to a monolithic kernel, a microkernel or whatever.  They all just work. And well, if this sounds like it wouldn't be possible, well, you saw  it running. You're welcome to look at the code and tell me that it doesn't work. So  that's the concept of the any kernel. Well, moving on from any kernels, the other part  of the topic of the talk was Rump kernels. Now, what is a Rump kernel? A Rump kernel  is simply a virtualized instance of a driver when it's not running inside the monolithic  kernel. So why don't I just call it a driver then? Well, a driver requires some amount  of supporting functionality to work. So someone needs to schedule the driver. The driver needs  to allocate memory. The driver needs to do locking. The driver needs to do this and that  or the driver needs to be able to do this and that to actually function properly. And  that's where the Rump kernel comes in. The Rump kernel provides the necessary set of  support bits for running the driver on top of the lightweight hypervisor we saw or which  was depicted in the previous picture. A Rump kernel is component oriented in the sense  that as you saw, if you were paying attention when I linked the example, there was just  one C file and then there was a huge amount of libraries. And the libraries were the components  that the Rump kernel was built from. If you looked very carefully, you saw that there  was a VFS component, there was the driver component, there was a disk driver component  and a few bits like that. So the idea is pretty much the same as with kernel modules. You  compile a bunch of code into libraries or modules or whatever you want to call them.  When you want to instantiate a Rump kernel, you just link stuff together. Do you need  to link them together statically? No, you can do it dynamically too. There's actually  a utility in NETBSD called Rump Server which does all of this for you. So to recap, the  any kernel is the possibility of running kernel drivers in any configuration feasible, the  same code. That's the whole point. You don't need to go rewriting drivers. And the Rump  kernel is the lightweight virtualized instance. Okay. So let's do another demonstration because  demonstrations are fun. They give an opportunity for things to go wrong. So we did file systems.  Let's do TCP IP networking this time. So what I'll do here is I'll run two NETBSD kernel  TCP IP stacks and have them talk to each other. This whole thing runs completely unprivileged  and how it works is pretty much as the first example. You have a Rump application layer.  You have a Rump kernel which makes, sorry, you have an application layer which makes  some sort of calls into the Rump kernel. Then drivers do stuff. In this case, the driver  is the TCP IP stack. And the bottom layer of the TCP IP stack is the interface driver  here. And here I've implemented the interface driver which allows multiple different Rump  kernels running on the same host to communicate with each other using shared memory. So this  is something that would be and could be pretty neat for TCP IP implementation testing. Let  me just adjust this. Yesterday I wanted to check that all of these terminals are the  right size. Then I checked it. Then they told me that they'll change some things. Now they're  all the wrong size. Okay. So in this directory we have, well, again, a make file. Then we  have a source file called net worker which just has two routines. Both bootstrap, a Rump  kernel, open a socket. One starts listening on it and, sorry, well, this one obviously  doesn't start listening on it. This one does a connect to the server. This happens to be  the address I configure for the server. Does a connect and then it just reads whatever  the server sends and exits. And then the server conversely opens a socket, starts listening  on it and accepts a connection and then it writes a test string into the socket and then  exits. Of course the text string is conveniently non-terminated so the client doesn't have  to bother with non-termination. Very good networking programming practice. But this  is demonstration code. So, okay, let's compile it. So now you see a different set of components  linked together. For example, you see you have the network driver component and you have  the TCP IP stack. You'll notice that there's a very distinct lack of anything related to  file systems in this example because this Rump kernel has absolutely zero file system  support. Where's my other terminal? It's here in a very weird size. Now it's even a  weirder size. That's why I love doing demos on Windows because I can blame everything  on Windows. Okay, so let's run the server. Then we run the client and then it worked.  So what actually happened here? So we have this, I told you it uses shared memory. The  shared memory handle is a file name which is memory mapped into both Rump kernels and  if you examine it, for example, dump the hex, you can clearly see the network traffic which  was going on. Well, even I would admit that that's a bit annoying to read. Now conveniently,  I wrote a tool which can convert this into the pick up format so you can actually open  it in Wireshark. And there you can see the network or the exchange between the two stacks.  So there's some initial bootstrapping spam which TCP IP stacks usually do when you configure  an address and then you actually see the TCP exchange here. The SIN was sent and the SIN  act and the ACK and here the server wrote the data to the client and we can see that  we have this. Strange sensation that we saw BSD networking stacks running on Windows earlier  too. And well, these two processes are completely independent. So I can again do something like  run GDB on the server. Let's put a breakpoint into TCP input. Let's run the server. Then  let's run the client. Then of course we get nothing back from the server because the server  is now stuck in the breakpoint. We can trace it. We see that we came somewhere through  an IP soft interrupt. You know, we can single step this server. The client is waiting here  or maybe timing out. We can again print values like we can see that the source address of  the IP packet that the server is processing is what we expect and so is the destination  address. So this example was about showing how you can run the unmodified NetBSD kernel  TCP IP stack without any privileges in user space. Now how far does this scale? I showed  you two. Well, I've personally done up to, was it 255? So I bootstrapped 255 TCP IP stacks  in Rump kernels and I configured their routing tables so that they were just, they had a  linear topology. So you know, this guy could talk to this guy, who could talk to this guy,  who could talk to this guy, who could talk to this guy. So why 255? Well, that's the maximum  value you can give the IP time to live. So you know, I didn't have an imagination to  come up with a more and more difficult test. So my test was start this number of TCP IP  stacks, configure networking for them, configure routing and send a packet from the other end  to the other end and wait for it to come back. So how long do you think this whole process  took? Starting the clock from when I started running the TCP IP stacks and stopping the  clock when I received the response. Well, I actually did two versions. One version I  did with a shell script using tools which you can find from a standard NetBSD installation.  The script is actually documented on some page in the dissertation. So if you're interested  what went on there, you can find out that took somewhere around eight seconds. Then  I did another version. I optimized the process a bit and I got this kind of, I got this  amount of TCP IP stacks bootstrapped and a thing sent from one end to the other end in  about a second. So if you're doing massive network testing and whenever you want to restart  your cluster, it takes a while, then you're maybe doing something wrong. Okay. Would anyone  have any questions at any point or everyone's busy trying out the stuff I showed for themselves?  No, I can put the source code somewhere on the demo source code. All of the other, the  driver source code is obviously out there but I can put the demo source code on the  net if someone kind of wants to start experimenting and use this as a basis for their experiments.  Okay. So I've shown you file systems. I've shown you TCP IP networking. What about hardware?  Is hardware supported? I don't like hardware. It's very difficult because you can't virtualize  it. You actually have to own the hardware. My hardware budget for this project was five  euros I used it to buy a Bluetooth USB thing because I wanted to support USB hardware drivers  because that allowed me to test a lot of other things like it allowed to test me Bluetooth,  it allowed me to test the scuzzy mid layer, it allowed me to test parts of net 802.11  support, et cetera, et cetera. Now, would hardware be impossible to support? No. I think  if you really wanted to and you started hacking now, you could have something limping along  by the end of the conference or maybe you can start after my talk, I don't know. So things  you just probably need to do is somehow map the device registers into the Rumpkernel,  figure out how you deliver interrupts there and figure out what you want to do with the  DMA. It shouldn't be difficult but I just haven't done it. Okay. So both examples I've  shown you so far have had a one process model. So the application was in the same package  as the Rumpkernel. Now, this is very convenient for things like let's say you want to use  unmodified net BSD kernel drivers on an embedded system because you can't be or because it's  just a lot of effort to implement them yourself. So hey, we have these drivers which can which  are ported on top of this convenient hypervisor. Let's just slurp them into our embedded system.  Our embedded system doesn't even support virtual memory, doesn't support the multiprocess  model. Well, that doesn't matter. Rumpkernels will run just fine. But for other purposes,  let's say this little thing called network configuration is typically implemented in  the sense that you have a server which in the normal cases is your kernel. You run one  command. It does something to the network configuration. You run another command. It  maybe modifies the routing tables a bit, et cetera, et cetera. So it's quite inconvenient  if the command and the Rumpkernel run in single process because when the command exits, the  Rumpkernel goes away too. So you won't get very far with this approach. But does this  model actually work where you have a separate server and a separate client and they somehow  communicate with each other? The short answer to that is yes. The long answer is a bit longer.  That's why they call it the long answer. Okay. So third demo. Everyone's still awake. So  now this time we have two processes. We have a server process and we have a client process  and they're going to talk to each other. I'll just show you the code for the server because  it's called simple server and it's really simple. So what it does, it bootstraps a Rumpkernel.  It initializes some listening address where the kernel is waiting for clients to send  communications and then it just pauses. So if we run that, then if we run the client,  actually, we need to tell the client that the server is listening here. This could also  be a... It doesn't need to be TCP. It can use local domain transport and actually this  thing can run over anything. I do have factual knowledge about that. But let's see, there's  like getPID which is very interesting. So every time we run it, we are in a different  process. If we kill the kernel, restart the kernel, the numbers start from... This is  a very interesting example I can see. Let's do something to modify the state a bit. Let's  create a directory called mount. If we try to create it again, it fails because it exists  already. We can... There's something called readFile which can read a file. So let's try  to read something from the mount point. Well, opening the file fails because it doesn't exist  anymore. This demo client, we can mount the kernel file system into there and after that  we can actually do readFile. And as usual, in my demos, GDB usage is mandatory. So this  time let's put a break point into copy out. If you know a bit about how things work between  an application and the kernel, the kernel always copies everything in and out from application,  the application address space. Now in this case, the application is in different process.  So we again run it and actually now we need to execute... Oh, now we rerun it. So let's  just remount the file system and try to read. And now this obviously hangs because the server  is hanging in a break point and you can actually look at this on your own time. So that's remote  system calls. This is kind of another way to put it. You have a kernel server somewhere  and you have applications which communicate with the server using some method of transport.  For example, local domain sockets are extremely convenient for kernel testing because you  have unlimited namespace between... Or for binding the socket so you're not limited to  the amount of TCP IP ports. You don't have to do any negotiation if you run... I don't  know. If you want to run 10,000 tests in parallel, you can run 10,000 tests in parallel. Well,  whatever. So usually when I present this, a lot of people tell me, aha, but you're  reinventing plan nine. So then I ask them, how can I reinvent plan nine using unmodified  unique system calls? So that's again something to think about. Now let's talk about something  else. The any kernel is interesting in the sense that it allows the user or the operating  system administrator to decide if now is a good time to trade performance for stability  or so forth. So one argument you hear often is performance doesn't matter anymore. But  for some reason you never hear anyone say it doesn't matter if we need to buy 10% more  hardware into this server form because maybe performance does matter after all. There are  some cases where stability is more important than performance. There are cases where performance  is more important than stability. And this just allows you to choose yourself. The evil  operating system implementers don't have that power over you anymore. So you can choose  whatever. Well, then let's talk quickly about the performance of Rump kernels. How fast  do they run? Well, all the code runs as native code. There's no kind of the any kernel guarantees  us that there's no privileged instructions in a driver code that's been all abstracted  away. So it runs the same speed as native code. So it boils down to how fast you can  do IO. If, for example, you run a file system driver, how efficient is your communication  with the disk driver? And in some tests I've run the performance was worse. In some cases  it was better because in the disk driver I was using in user space, the caching behavior  was slightly different from the one in the kernel. So you actually got better performance.  Well, another way to look at this problem is kind of not from a micro optimization perspective,  but from an algorithmic perspective. Developing kernel code with literally zero iteration  time. First of all, it allows you to try out wild ideas. So you know, what if we do this  crazy thing or what if we do this crazy thing and very quickly see how it affects things.  Second of all, since you don't run anything, everything in a monolithic kernel anymore,  you don't have to worry necessarily about like, hey, let's do this hack into the TCP  IP stack. So what kind of effect does it have on every possible user of the TCP IP stack?  You can just have your own virtual TCP IP stack dedicated to a certain application and  hack the hell out of that. So that's, you know, the bigger picture on performance. Now, just  to give an example of things you can do, these partitioned operating systems, which are kind  of a, well, kind of a recent trend, tend to run operating system components one per CPU  and do message passing between them. And the part of the idea for this is that you don't  need any atomic bus locks because you just have one non-conflicting thread running on  one CPU. Then when it wants to do something else, it sends a message to another CPU and  then something else runs there. So I kind of read about this, then I got interested,  could you actually do this with existing drivers, kind of optimize a way locking completely  if you limited yourself to one processor? And turns out you can. It's implemented in  a source file called locksup, which I at some point realized maybe wasn't the best name,  but it's not. Well, then something interesting, or I don't know interesting, but something  I did just for fun recently. So, you know, we have these drivers, you can compile them  into virtualized environments, you can run them, they're pretty flexible. So I thought,  hmm, could we compile these so that they run natively on a web browser? So then I went  out in search for a C2JavaScript compiler and some amount of hacking later I had the answer  for that tool. Yes, you can. So I don't know if you've seen this, but this is available  on the Internet and it's just a RAM kernel compiled into JavaScript. Now, this is pretty  much the same set of drivers we saw in the first demo. You can execute things, well,  there's a very small command interface. You can execute various commands. And if you actually  a, no, wrong button, I forget how this web stuff works. So if you can run it, for example,  run the fire bug and then you can take a while. So let's, well, I had a break point in there  already. Actually, let's put a break point here into FFS read because that's what I showed.  Showed in the, which one is the play button? This one. So this is what I showed in the,  this is good. So now we hit the break point and if you memorize the stack trace from the  first example, which of course everyone did, you will no doubt notice that the stack trace  is very similar because this is exactly the same unmodified NetBSD kernel driver, which  instead of being compiled into assembly, it has been compiled into JavaScript and is now  being run in a browser and it's very useless otherwise, but it's fun. Okay. So I promise  to tell you how you can run this stuff on, if you have a sensible, sensible open source  operating system in just in user space. You don't need to install any virtual machines.  You don't need to do anything like that. You just get a script called buildrump.sh, which  is hosted on GitHub. You get the NetBSD source tree. You run the script and some minutes  later you will be rewarded with a lot of libraries. I'll give you the URL at the end of the talk,  although I'm fairly certain that you are imaginative enough to be able to find it yourself without  my help. So then there's this kind of if our kernel or our Rump kernel is from NetBSD  and we try to call it from, let's say Linux or I don't know, Sigwin or whatever. There's  this huge potential for type conflicts. You know, some constants might have different  values when you pass them to the system calls. They just don't agree. And this is kind of  something I've not entirely solved yet. You can actually fairly easily brute force your  way around this. Like depending on what you want to do, you just, you apply brute force,  which means that you create copies of types. You create copies of constants. You know,  like it's half a day of hacking to or maybe a whole day. I can't quite remember. Or maybe  it was a night of hacking if you want to reimplement IF config as a kernel component. That kind  of solves the type problem in a brute force way, but it's not very nice. Well, anyway,  moving on a bit. What this slide is trying to say that if your operating system isn't  an any kernel, it sucks. Well, I believe there are so many both internal and external benefits  that every operating system, at least any every journal purpose operating system should  try their hands at being an any kernel. It's the question, well, the interesting question  is how can this possibly work? What proof do I have that it works? Well, that's the  proof. Is there any grand theoretical model? Is there any great wisdom behind it? Well,  like I like to say, the main reason operating systems work anyway is luck. So this kind of  taps into that same principle. It just happens to work. And before you try it, you can't  know for sure, but at least at least this thing has been implemented and working in  net BST since 2007. So maybe that proves something. I actually tried to come up with  an analogy for this. I came with an analogy that trying this is kind of like throwing  spaghetti on the wall. And if it sticks, you can take a photograph and call it art. So  then I of course wanted to try this. And that night I had a very long discussion with my  girlfriend. See, she told me that you're not supposed to throw it on the ceiling. Sorry,  the wall, you're supposed to throw it on the ceiling. And that's not for creating art.  That's how you test if spaghetti is done. And then she added, everyone knows that. So,  well, anyway, so if you don't remember anything from this talk except you test spaghetti is  done by throwing it on the ceiling, then I've done something useful. So in addition to converting  all of the other operating systems into any kernels, there's a bunch of things you could  do with the net BST implementation should you be so inclined. First of all, you can  test the build drum script on a platform of your choosing. And if you do so, please report  the result. If it works, I'm still collecting a list of hosts where it's been known to work.  So if it works, you know, send an addition to the list of hosts where it's known to work.  If it doesn't work, please, please at least file a bug report on GitHub or if you can  manage a patch, it's even better. Something else you can do, you can package RAMP kernels  for your or the net BST kernel drivers for your favorite operating systems, packaging  system so that, for example, if you happen to need to read file systems, which there  are only net BST kernel drivers available for, you can actually do it afterwards. Well,  one of the funny things you can do with this is you can very easily run while grind on  kernel code. There's a few rough edges still like examining the string length, which leads  into interesting results because string length is a routine which actually the optimized  version looks beyond the end of the string for each word. So there's a few recipes which  you can maybe use to kind of improve the while grind, report it, and then maybe document  them somewhere. Well, one very useful thing would be to use autocon for the hypervisor.  So now the hypervisor runs on net BST, runs on JavaScript, runs on Linux, runs on Solaris,  runs on FreeBSD, et cetera, et cetera, but I've been very lazy so I haven't done anything  fancy. I've just put some if-defs in there and with different versions of different operating  systems that approach is kind of starting to come to an end. So if you like autocon,  you could maybe apply autocons to the hypervisor and then you can tell all your friends that  you used autocon to make a kernel more portable. Then they will maybe look at you in a slightly  funny way. But then there's things like authentication support for remote clients. Shortly put, currently  every remotely connecting process gets root privileges, which is good for testing, but  if you want to use this for something else, it might not be what you want. And then there's  the whole issue with type incompatibilities, which if you run this on Linux, I'm sure you'll  understand sooner or later. I have a theory that it could be solved automatically with  a toolchain driven approach, but that requires someone with maybe more toolchain knowledge  and I don't know, maybe it could be if you know someone who's looking for a good master's  thesis project, that could be something to look at. Well, that's a pretty accurate conclusion.  And you? Yes, question?  For Walgreen, you can just use dash F no built in. Yes, but see, okay, so what he said is  then it uses the C library string length function, but see the rom kernel is self-contained,  so it doesn't use the C libraries string length function. Yes, but you might run into some  issues if you try to use the C library past the hypervisor. So everything into the C library  should go through the hypervisor. And actual question is what why does it? Why does it use  threads? So the actual question was why does it use threads? Well, because the drivers  require threads. Actually, you don't need threads in all circumstances, like for example,  file systems run just fine without threads. And you can kind of probably fake threads.  Let's assume you probably need some sort of concept of multiple stacks, but you don't maybe  necessarily need to use threads as we understand them. But yeah, the underlying reason why  does it use threads is that because the kernel drivers depend on threads. For example, to  deliver network packets. So there's no model for interrupts. Everything always runs completely  to finish, and that's for simplicity reasons. Yes?  Yes? Sorry? Yes? What new kinds errors?  Okay. Okay. So two questions. First question. So if you use remote system calls and it could  theoretically introduce some new kinds of failures. For example, well, the thing you said TCP  packets could be lost. But it uses TCP stream so that the stream isn't lost. I mean, the  stream itself will eventually always, unless the network breaks, unless the kernel crashes.  Especially when I was testing this, I, for weeks and weeks, I ran an unmodified Firefox  against Rumpkernel TCP IP stack because I wanted to know if it works. And then I thought,  hey, it would be cool if I could actually kill the TCP IP stack and restart it and have my  web browser notice nothing. So I also added reconnection support to the remote protocol.  So if Firefox is loading a page, you see kill the TCP IP stack. Well, then obviously packets  don't flow anymore. But if you restart the TCP IP stack, reconfigure it and hit control  R, then things work again.  Yes. So he said that system calls are always expected to return and the transport code  must guarantee that. Well, that's what the regular application expects anyway. Now,  the system call might not return, for example, if the system crashes. I mean, that's, you  know, but yeah, adding distribution adds a whole other array of problems. So, you know,  it works. But if you want to take the problems of distribution into account, then maybe you  need to do a bit more work. But that's kind of a use case specific. I mean, you can run  an application here and a Rumpkernel in Japan. But I haven't found very many good uses for  that because the latency is quite huge. So things tend to go slow.  Well, does anyone else have questions? Okay. So you are allowed to ask a question.  The second question was how do I deal with signals? So signals are always something that  the host provides where you run the application. So, okay, so there's a kind of, well, I don't  know how to answer this in a time which will not make the moderator more upset. So maybe  we can discuss it afterwards. Okay.
The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_0_1811.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_1812_2003.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_2004_4679.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_4680_16823.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_16824_17219.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_17220_17351.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_17352_17975.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_17976_18047.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_18048_19847.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_19848_20507.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_20508_20783.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_20784_20831.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_20832_20915.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_20916_20987.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_20988_21239.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_21240_21731.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_21732_25691.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_25692_25739.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_25740_29855.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_29856_36503.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_36504_36563.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_36564_37355.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_37356_40091.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_40092_41099.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_41100_41243.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_41244_41579.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_41580_42239.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_42240_43631.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_43632_43679.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_43680_43763.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_43764_43811.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_43812_46343.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_46344_48179.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_48180_49583.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_49584_49763.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_49764_49991.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_49992_50087.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_50088_50459.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_50460_50651.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_50652_51263.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_51264_51359.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_51360_51443.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_51444_51731.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_51732_52667.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_52668_54455.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_54456_54827.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_54828_54983.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_54984_58403.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_58404_58607.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_58608_58811.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_58812_59075.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_59076_59819.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_59820_60395.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_60396_61931.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_61932_62363.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_62364_62519.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_62520_63875.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_63876_64115.png The_Anykernel_and_Rump_Kernels_HaXvYBJW4ok_2_64116_78827.png
Good luck. Okay, thank you. So, as everyone can probably clearly see, the title of this talk is the Anycernel and Rump Kernels. The basis for this talk will be formed by what I did for my doctoral work, which I finished a while ago. But when I say doctoral work, don't worry, it doesn't mean that I'll have a lot of theory, I'll have a lot of Greek equations, I'll have a lot of stuff like that. This will be a very high-level, undetailed talk. And if you want to plow down further into details, my dissertation is linked from the talk website, so you can find a few hundred pages of details, approximately this much here's a physical copy. But even that isn't written in Greek, it's more written in a hackery fashion, so I discuss more the implementation and those kind of issues. So that's kind of the basis for my talk. Well, the implementation that I did for my doctoral work is for NetBSD, it's written in production quality code, it's part of the NetBSD6 release. And what I've been doing after graduating, I've kind of been working on making it more accessible to people. Let's make power accessible to the laptop. Yeah, so I'll show a lot of demos today, and everything I show, if you have a sensible open source system installed like, let's say, Linux or freeBSD, you should be able to repeat on your system. And I'll even tell you how you can do it, or at least I'll give you the pointer. So just to recap the main idea, the main idea of this talk is to introduce the subject, tell you what you can do with this stuff, and not so much go into details, but give you an understanding of if you want to know the details where you can find them and how you can play with this stuff. Okay, so let's just simply start off with the demo. Like I said, if you have any sensible open source operating system, you should be able to repeat what I'll show you. Well, unfortunately, this isn't a sensible open source operating system. This happens to be Windows. No, don't worry, I'm not at the wrong conference, but let's not allow this inconvenience to get to me. So what I have here is I have a directory which has a file system image. At least file claims that it's a Unix fast file system image. Now let's assume I'd want to read this image. Well, Windows doesn't at least natively support this file system, so am I at a luck? Do I need to install a virtual machine with NetBSD on it to be able to read this? Well, let's see. So, yeah, I have a file system image, then I just have a funny test file which just contains the text on file. Then I have a short bit of application code which does a few very simple things. It bootstraps a virtual kernel on this line here. It does a bunch of other things, creates a directory within the Rump kernel namespace, mounts a file system within the Rump kernel, changes the current process in the Rump kernel context into this directory, and then it calls this routine myread file twice. Once it passes function pointers to host system calls open and read, once it passes them to Rump kernel function pointers, and finally it reboots. Yeah. And what myread does, it opens a file here, it reads it, and then it prints the contents. Okay, so let's compile it. And then let's run it. Now, what this program will do is it will bootstrap a virtual kernel, mount a file system in it, read a file using the virtual kernel, and reboot. So how long do you think this process will take? Ten seconds, five seconds, a second? Well, it actually depends on if SIGWIN is merciful. So that's how long. And most of the time we spent loading the thing into cache. So if I run it a number of times, sometimes SIGWIN makes it run faster, sometimes it doesn't. But anyway, it's a pretty instant operation for... Let's try to adjust this a bit. Okay, that's better. So anyway, bootstrapping the virtual Rump kernel, mounting a file system, reading a file, rebooting the Rump kernel, that's a pretty instantaneous operation. And what happens here is I'm actually using unmodified NetBSD kernel code in this virtualized kernel. So what else can we do with this? Well, this is something which is running in user space, so can we run GDB on it? Sure, why not? I mean, let's put a breakpoint somewhere like into the kernel's FFS file system read method. Let's run this thing. And we hit the breakpoint, we can examine the stack trace just as we could in a normal application. We can single step it. We can print the values of arguments and set the right, set the right, set the right. So that works. Well, okay, so now we have a virtualized kernel running in user space with unmodified kernel code working very fast. So of course, now immediately already if you're a kernel developer, you'll maybe start thinking, hey, could I actually use this stuff for my advantage? Can I actually develop and debug code with this? And yes, you can. And iteration is really, really fast. Since running it takes a fraction of a second. And since it's virtualized, let's just do a small experiment. So what would happen if this is the same read method we put the breakpoint into previously? So let's just put panic there, because panics are always fun. Everyone likes kernel panics. Let's recompile the file system component, install it, then let's rebuild this and run it. And now we got a kernel panic. But of course, our host system wasn't affected since everything is running in a process. Yeah. So what happened here? So this is kind of now, you know, I showed a demo in the beginning to give you some concrete idea of what the talk is about. And now I'm actually going to explain what the hell happened. So this is kind of some sort of fancy picture of what was going on. So this entire screen represents one process. So we have the application layer on the top, which was the source code I showed you. That's the one which decided to where we bootstrap the Rumpkernel. We mounted the file system, we read the file system contents and so forth and so forth. And you can see there's the blue arrow which goes directly from the application to the host, and there's the green arrow which goes through the Rumpkernel. Now, we called the MyRead file routine twice. Once it read the host, read the file using host calls, and once it read it using Rumpkernel system calls. And most of the stuff in here is unmodified netBSD kernel code. Actually, it can even be unmodified netBSD kernel binaries on some systems. Like if you host this, I'm hosting this on Siegwin so it won't work because the object formats are different. But if you're, for example, hosting a Rumpkernel on a Linux system, you can load netBSD kernel code into it just fine. And the Rumpkernel runs on top of a very lightweight hypervisor, so you could say that all netBSD kernel drivers have been virtualized now. And the hypervisor is the guy who makes the calls into the host. So this Rumpkernel bit is completely self-contained. It can only access the world through the hypervisor. And the hypervisor, it's not a hypervisor in the normal virtualization sense, like where you virtualize an operating system where it's a, the hypervisor takes care of low-level fiddly bits like maintaining page tables and so forth. This guy can pretty much just do create a thread, allocate a page of memory, and then access some IO routines. And yeah, this is a single, or in this case it was a single process, runs on a host. It doesn't have to be a comical host, but this host happened to be comical, so that's why I depicted it as comical. Okay, so that's what happened now. I'll backtrack even more and I'll explain the actual concept. So someone asked me before to talk what is an any kernel. So now I'll actually tell you what's an any kernel. And any kernel is just a kind of tongue-in-cheek term I invented. You know how operating systems, you usually wonder, is this a microkernel operating system? Is this a monolithic kernel operating system? Is this an exo kernel operating system? Does this use partitioned kernels or whatever? And my view on this problem is that it's irrelevant because you can write your operating system so that you can choose at runtime which model you want to use. And one very important thing to understand is I'm talking about running the same code, the same drivers, line by line. You don't need to rewrite drivers. I'm not talking about the actual possibility to interconnect some components. Like can you attach or can you mount a file system using a user space server or a internal file system? That's not what I'm talking about. That's the kind of request transport method. I'm talking about the actual driver implementations. So simply put, the any kernel is a way to write kernel drivers where you're not restricting yourself to a monolithic kernel, a microkernel or whatever. They all just work. And well, if this sounds like it wouldn't be possible, well, you saw it running. You're welcome to look at the code and tell me that it doesn't work. So that's the concept of the any kernel. Well, moving on from any kernels, the other part of the topic of the talk was Rump kernels. Now, what is a Rump kernel? A Rump kernel is simply a virtualized instance of a driver when it's not running inside the monolithic kernel. So why don't I just call it a driver then? Well, a driver requires some amount of supporting functionality to work. So someone needs to schedule the driver. The driver needs to allocate memory. The driver needs to do locking. The driver needs to do this and that or the driver needs to be able to do this and that to actually function properly. And that's where the Rump kernel comes in. The Rump kernel provides the necessary set of support bits for running the driver on top of the lightweight hypervisor we saw or which was depicted in the previous picture. A Rump kernel is component oriented in the sense that as you saw, if you were paying attention when I linked the example, there was just one C file and then there was a huge amount of libraries. And the libraries were the components that the Rump kernel was built from. If you looked very carefully, you saw that there was a VFS component, there was the driver component, there was a disk driver component and a few bits like that. So the idea is pretty much the same as with kernel modules. You compile a bunch of code into libraries or modules or whatever you want to call them. When you want to instantiate a Rump kernel, you just link stuff together. Do you need to link them together statically? No, you can do it dynamically too. There's actually a utility in NETBSD called Rump Server which does all of this for you. So to recap, the any kernel is the possibility of running kernel drivers in any configuration feasible, the same code. That's the whole point. You don't need to go rewriting drivers. And the Rump kernel is the lightweight virtualized instance. Okay. So let's do another demonstration because demonstrations are fun. They give an opportunity for things to go wrong. So we did file systems. Let's do TCP IP networking this time. So what I'll do here is I'll run two NETBSD kernel TCP IP stacks and have them talk to each other. This whole thing runs completely unprivileged and how it works is pretty much as the first example. You have a Rump application layer. You have a Rump kernel which makes, sorry, you have an application layer which makes some sort of calls into the Rump kernel. Then drivers do stuff. In this case, the driver is the TCP IP stack. And the bottom layer of the TCP IP stack is the interface driver here. And here I've implemented the interface driver which allows multiple different Rump kernels running on the same host to communicate with each other using shared memory. So this is something that would be and could be pretty neat for TCP IP implementation testing. Let me just adjust this. Yesterday I wanted to check that all of these terminals are the right size. Then I checked it. Then they told me that they'll change some things. Now they're all the wrong size. Okay. So in this directory we have, well, again, a make file. Then we have a source file called net worker which just has two routines. Both bootstrap, a Rump kernel, open a socket. One starts listening on it and, sorry, well, this one obviously doesn't start listening on it. This one does a connect to the server. This happens to be the address I configure for the server. Does a connect and then it just reads whatever the server sends and exits. And then the server conversely opens a socket, starts listening on it and accepts a connection and then it writes a test string into the socket and then exits. Of course the text string is conveniently non-terminated so the client doesn't have to bother with non-termination. Very good networking programming practice. But this is demonstration code. So, okay, let's compile it. So now you see a different set of components linked together. For example, you see you have the network driver component and you have the TCP IP stack. You'll notice that there's a very distinct lack of anything related to file systems in this example because this Rump kernel has absolutely zero file system support. Where's my other terminal? It's here in a very weird size. Now it's even a weirder size. That's why I love doing demos on Windows because I can blame everything on Windows. Okay, so let's run the server. Then we run the client and then it worked. So what actually happened here? So we have this, I told you it uses shared memory. The shared memory handle is a file name which is memory mapped into both Rump kernels and if you examine it, for example, dump the hex, you can clearly see the network traffic which was going on. Well, even I would admit that that's a bit annoying to read. Now conveniently, I wrote a tool which can convert this into the pick up format so you can actually open it in Wireshark. And there you can see the network or the exchange between the two stacks. So there's some initial bootstrapping spam which TCP IP stacks usually do when you configure an address and then you actually see the TCP exchange here. The SIN was sent and the SIN act and the ACK and here the server wrote the data to the client and we can see that we have this. Strange sensation that we saw BSD networking stacks running on Windows earlier too. And well, these two processes are completely independent. So I can again do something like run GDB on the server. Let's put a breakpoint into TCP input. Let's run the server. Then let's run the client. Then of course we get nothing back from the server because the server is now stuck in the breakpoint. We can trace it. We see that we came somewhere through an IP soft interrupt. You know, we can single step this server. The client is waiting here or maybe timing out. We can again print values like we can see that the source address of the IP packet that the server is processing is what we expect and so is the destination address. So this example was about showing how you can run the unmodified NetBSD kernel TCP IP stack without any privileges in user space. Now how far does this scale? I showed you two. Well, I've personally done up to, was it 255? So I bootstrapped 255 TCP IP stacks in Rump kernels and I configured their routing tables so that they were just, they had a linear topology. So you know, this guy could talk to this guy, who could talk to this guy, who could talk to this guy, who could talk to this guy. So why 255? Well, that's the maximum value you can give the IP time to live. So you know, I didn't have an imagination to come up with a more and more difficult test. So my test was start this number of TCP IP stacks, configure networking for them, configure routing and send a packet from the other end to the other end and wait for it to come back. So how long do you think this whole process took? Starting the clock from when I started running the TCP IP stacks and stopping the clock when I received the response. Well, I actually did two versions. One version I did with a shell script using tools which you can find from a standard NetBSD installation. The script is actually documented on some page in the dissertation. So if you're interested what went on there, you can find out that took somewhere around eight seconds. Then I did another version. I optimized the process a bit and I got this kind of, I got this amount of TCP IP stacks bootstrapped and a thing sent from one end to the other end in about a second. So if you're doing massive network testing and whenever you want to restart your cluster, it takes a while, then you're maybe doing something wrong. Okay. Would anyone have any questions at any point or everyone's busy trying out the stuff I showed for themselves? No, I can put the source code somewhere on the demo source code. All of the other, the driver source code is obviously out there but I can put the demo source code on the net if someone kind of wants to start experimenting and use this as a basis for their experiments. Okay. So I've shown you file systems. I've shown you TCP IP networking. What about hardware? Is hardware supported? I don't like hardware. It's very difficult because you can't virtualize it. You actually have to own the hardware. My hardware budget for this project was five euros I used it to buy a Bluetooth USB thing because I wanted to support USB hardware drivers because that allowed me to test a lot of other things like it allowed to test me Bluetooth, it allowed me to test the scuzzy mid layer, it allowed me to test parts of net 802.11 support, et cetera, et cetera. Now, would hardware be impossible to support? No. I think if you really wanted to and you started hacking now, you could have something limping along by the end of the conference or maybe you can start after my talk, I don't know. So things you just probably need to do is somehow map the device registers into the Rumpkernel, figure out how you deliver interrupts there and figure out what you want to do with the DMA. It shouldn't be difficult but I just haven't done it. Okay. So both examples I've shown you so far have had a one process model. So the application was in the same package as the Rumpkernel. Now, this is very convenient for things like let's say you want to use unmodified net BSD kernel drivers on an embedded system because you can't be or because it's just a lot of effort to implement them yourself. So hey, we have these drivers which can which are ported on top of this convenient hypervisor. Let's just slurp them into our embedded system. Our embedded system doesn't even support virtual memory, doesn't support the multiprocess model. Well, that doesn't matter. Rumpkernels will run just fine. But for other purposes, let's say this little thing called network configuration is typically implemented in the sense that you have a server which in the normal cases is your kernel. You run one command. It does something to the network configuration. You run another command. It maybe modifies the routing tables a bit, et cetera, et cetera. So it's quite inconvenient if the command and the Rumpkernel run in single process because when the command exits, the Rumpkernel goes away too. So you won't get very far with this approach. But does this model actually work where you have a separate server and a separate client and they somehow communicate with each other? The short answer to that is yes. The long answer is a bit longer. That's why they call it the long answer. Okay. So third demo. Everyone's still awake. So now this time we have two processes. We have a server process and we have a client process and they're going to talk to each other. I'll just show you the code for the server because it's called simple server and it's really simple. So what it does, it bootstraps a Rumpkernel. It initializes some listening address where the kernel is waiting for clients to send communications and then it just pauses. So if we run that, then if we run the client, actually, we need to tell the client that the server is listening here. This could also be a... It doesn't need to be TCP. It can use local domain transport and actually this thing can run over anything. I do have factual knowledge about that. But let's see, there's like getPID which is very interesting. So every time we run it, we are in a different process. If we kill the kernel, restart the kernel, the numbers start from... This is a very interesting example I can see. Let's do something to modify the state a bit. Let's create a directory called mount. If we try to create it again, it fails because it exists already. We can... There's something called readFile which can read a file. So let's try to read something from the mount point. Well, opening the file fails because it doesn't exist anymore. This demo client, we can mount the kernel file system into there and after that we can actually do readFile. And as usual, in my demos, GDB usage is mandatory. So this time let's put a break point into copy out. If you know a bit about how things work between an application and the kernel, the kernel always copies everything in and out from application, the application address space. Now in this case, the application is in different process. So we again run it and actually now we need to execute... Oh, now we rerun it. So let's just remount the file system and try to read. And now this obviously hangs because the server is hanging in a break point and you can actually look at this on your own time. So that's remote system calls. This is kind of another way to put it. You have a kernel server somewhere and you have applications which communicate with the server using some method of transport. For example, local domain sockets are extremely convenient for kernel testing because you have unlimited namespace between... Or for binding the socket so you're not limited to the amount of TCP IP ports. You don't have to do any negotiation if you run... I don't know. If you want to run 10,000 tests in parallel, you can run 10,000 tests in parallel. Well, whatever. So usually when I present this, a lot of people tell me, aha, but you're reinventing plan nine. So then I ask them, how can I reinvent plan nine using unmodified unique system calls? So that's again something to think about. Now let's talk about something else. The any kernel is interesting in the sense that it allows the user or the operating system administrator to decide if now is a good time to trade performance for stability or so forth. So one argument you hear often is performance doesn't matter anymore. But for some reason you never hear anyone say it doesn't matter if we need to buy 10% more hardware into this server form because maybe performance does matter after all. There are some cases where stability is more important than performance. There are cases where performance is more important than stability. And this just allows you to choose yourself. The evil operating system implementers don't have that power over you anymore. So you can choose whatever. Well, then let's talk quickly about the performance of Rump kernels. How fast do they run? Well, all the code runs as native code. There's no kind of the any kernel guarantees us that there's no privileged instructions in a driver code that's been all abstracted away. So it runs the same speed as native code. So it boils down to how fast you can do IO. If, for example, you run a file system driver, how efficient is your communication with the disk driver? And in some tests I've run the performance was worse. In some cases it was better because in the disk driver I was using in user space, the caching behavior was slightly different from the one in the kernel. So you actually got better performance. Well, another way to look at this problem is kind of not from a micro optimization perspective, but from an algorithmic perspective. Developing kernel code with literally zero iteration time. First of all, it allows you to try out wild ideas. So you know, what if we do this crazy thing or what if we do this crazy thing and very quickly see how it affects things. Second of all, since you don't run anything, everything in a monolithic kernel anymore, you don't have to worry necessarily about like, hey, let's do this hack into the TCP IP stack. So what kind of effect does it have on every possible user of the TCP IP stack? You can just have your own virtual TCP IP stack dedicated to a certain application and hack the hell out of that. So that's, you know, the bigger picture on performance. Now, just to give an example of things you can do, these partitioned operating systems, which are kind of a, well, kind of a recent trend, tend to run operating system components one per CPU and do message passing between them. And the part of the idea for this is that you don't need any atomic bus locks because you just have one non-conflicting thread running on one CPU. Then when it wants to do something else, it sends a message to another CPU and then something else runs there. So I kind of read about this, then I got interested, could you actually do this with existing drivers, kind of optimize a way locking completely if you limited yourself to one processor? And turns out you can. It's implemented in a source file called locksup, which I at some point realized maybe wasn't the best name, but it's not. Well, then something interesting, or I don't know interesting, but something I did just for fun recently. So, you know, we have these drivers, you can compile them into virtualized environments, you can run them, they're pretty flexible. So I thought, hmm, could we compile these so that they run natively on a web browser? So then I went out in search for a C2JavaScript compiler and some amount of hacking later I had the answer for that tool. Yes, you can. So I don't know if you've seen this, but this is available on the Internet and it's just a RAM kernel compiled into JavaScript. Now, this is pretty much the same set of drivers we saw in the first demo. You can execute things, well, there's a very small command interface. You can execute various commands. And if you actually a, no, wrong button, I forget how this web stuff works. So if you can run it, for example, run the fire bug and then you can take a while. So let's, well, I had a break point in there already. Actually, let's put a break point here into FFS read because that's what I showed. Showed in the, which one is the play button? This one. So this is what I showed in the, this is good. So now we hit the break point and if you memorize the stack trace from the first example, which of course everyone did, you will no doubt notice that the stack trace is very similar because this is exactly the same unmodified NetBSD kernel driver, which instead of being compiled into assembly, it has been compiled into JavaScript and is now being run in a browser and it's very useless otherwise, but it's fun. Okay. So I promise to tell you how you can run this stuff on, if you have a sensible, sensible open source operating system in just in user space. You don't need to install any virtual machines. You don't need to do anything like that. You just get a script called buildrump.sh, which is hosted on GitHub. You get the NetBSD source tree. You run the script and some minutes later you will be rewarded with a lot of libraries. I'll give you the URL at the end of the talk, although I'm fairly certain that you are imaginative enough to be able to find it yourself without my help. So then there's this kind of if our kernel or our Rump kernel is from NetBSD and we try to call it from, let's say Linux or I don't know, Sigwin or whatever. There's this huge potential for type conflicts. You know, some constants might have different values when you pass them to the system calls. They just don't agree. And this is kind of something I've not entirely solved yet. You can actually fairly easily brute force your way around this. Like depending on what you want to do, you just, you apply brute force, which means that you create copies of types. You create copies of constants. You know, like it's half a day of hacking to or maybe a whole day. I can't quite remember. Or maybe it was a night of hacking if you want to reimplement IF config as a kernel component. That kind of solves the type problem in a brute force way, but it's not very nice. Well, anyway, moving on a bit. What this slide is trying to say that if your operating system isn't an any kernel, it sucks. Well, I believe there are so many both internal and external benefits that every operating system, at least any every journal purpose operating system should try their hands at being an any kernel. It's the question, well, the interesting question is how can this possibly work? What proof do I have that it works? Well, that's the proof. Is there any grand theoretical model? Is there any great wisdom behind it? Well, like I like to say, the main reason operating systems work anyway is luck. So this kind of taps into that same principle. It just happens to work. And before you try it, you can't know for sure, but at least at least this thing has been implemented and working in net BST since 2007. So maybe that proves something. I actually tried to come up with an analogy for this. I came with an analogy that trying this is kind of like throwing spaghetti on the wall. And if it sticks, you can take a photograph and call it art. So then I of course wanted to try this. And that night I had a very long discussion with my girlfriend. See, she told me that you're not supposed to throw it on the ceiling. Sorry, the wall, you're supposed to throw it on the ceiling. And that's not for creating art. That's how you test if spaghetti is done. And then she added, everyone knows that. So, well, anyway, so if you don't remember anything from this talk except you test spaghetti is done by throwing it on the ceiling, then I've done something useful. So in addition to converting all of the other operating systems into any kernels, there's a bunch of things you could do with the net BST implementation should you be so inclined. First of all, you can test the build drum script on a platform of your choosing. And if you do so, please report the result. If it works, I'm still collecting a list of hosts where it's been known to work. So if it works, you know, send an addition to the list of hosts where it's known to work. If it doesn't work, please, please at least file a bug report on GitHub or if you can manage a patch, it's even better. Something else you can do, you can package RAMP kernels for your or the net BST kernel drivers for your favorite operating systems, packaging system so that, for example, if you happen to need to read file systems, which there are only net BST kernel drivers available for, you can actually do it afterwards. Well, one of the funny things you can do with this is you can very easily run while grind on kernel code. There's a few rough edges still like examining the string length, which leads into interesting results because string length is a routine which actually the optimized version looks beyond the end of the string for each word. So there's a few recipes which you can maybe use to kind of improve the while grind, report it, and then maybe document them somewhere. Well, one very useful thing would be to use autocon for the hypervisor. So now the hypervisor runs on net BST, runs on JavaScript, runs on Linux, runs on Solaris, runs on FreeBSD, et cetera, et cetera, but I've been very lazy so I haven't done anything fancy. I've just put some if-defs in there and with different versions of different operating systems that approach is kind of starting to come to an end. So if you like autocon, you could maybe apply autocons to the hypervisor and then you can tell all your friends that you used autocon to make a kernel more portable. Then they will maybe look at you in a slightly funny way. But then there's things like authentication support for remote clients. Shortly put, currently every remotely connecting process gets root privileges, which is good for testing, but if you want to use this for something else, it might not be what you want. And then there's the whole issue with type incompatibilities, which if you run this on Linux, I'm sure you'll understand sooner or later. I have a theory that it could be solved automatically with a toolchain driven approach, but that requires someone with maybe more toolchain knowledge and I don't know, maybe it could be if you know someone who's looking for a good master's thesis project, that could be something to look at. Well, that's a pretty accurate conclusion. And you? Yes, question? For Walgreen, you can just use dash F no built in. Yes, but see, okay, so what he said is then it uses the C library string length function, but see the rom kernel is self-contained, so it doesn't use the C libraries string length function. Yes, but you might run into some issues if you try to use the C library past the hypervisor. So everything into the C library should go through the hypervisor. And actual question is what why does it? Why does it use threads? So the actual question was why does it use threads? Well, because the drivers require threads. Actually, you don't need threads in all circumstances, like for example, file systems run just fine without threads. And you can kind of probably fake threads. Let's assume you probably need some sort of concept of multiple stacks, but you don't maybe necessarily need to use threads as we understand them. But yeah, the underlying reason why does it use threads is that because the kernel drivers depend on threads. For example, to deliver network packets. So there's no model for interrupts. Everything always runs completely to finish, and that's for simplicity reasons. Yes? Yes? Sorry? Yes? What new kinds errors? Okay. Okay. So two questions. First question. So if you use remote system calls and it could theoretically introduce some new kinds of failures. For example, well, the thing you said TCP packets could be lost. But it uses TCP stream so that the stream isn't lost. I mean, the stream itself will eventually always, unless the network breaks, unless the kernel crashes. Especially when I was testing this, I, for weeks and weeks, I ran an unmodified Firefox against Rumpkernel TCP IP stack because I wanted to know if it works. And then I thought, hey, it would be cool if I could actually kill the TCP IP stack and restart it and have my web browser notice nothing. So I also added reconnection support to the remote protocol. So if Firefox is loading a page, you see kill the TCP IP stack. Well, then obviously packets don't flow anymore. But if you restart the TCP IP stack, reconfigure it and hit control R, then things work again. Yes. So he said that system calls are always expected to return and the transport code must guarantee that. Well, that's what the regular application expects anyway. Now, the system call might not return, for example, if the system crashes. I mean, that's, you know, but yeah, adding distribution adds a whole other array of problems. So, you know, it works. But if you want to take the problems of distribution into account, then maybe you need to do a bit more work. But that's kind of a use case specific. I mean, you can run an application here and a Rumpkernel in Japan. But I haven't found very many good uses for that because the latency is quite huge. So things tend to go slow. Well, does anyone else have questions? Okay. So you are allowed to ask a question. The second question was how do I deal with signals? So signals are always something that the host provides where you run the application. So, okay, so there's a kind of, well, I don't know how to answer this in a time which will not make the moderator more upset. So maybe we can discuss it afterwards. Okay.
like that. This will be a very high-level, fashion, so I discuss more the implementation and in production quality code, it's part of the main idea of this talk is to introduce the should be able to repeat what I'll show you. application code which does a few very simple what this program will do is it will bootstrap a using unmodified NetBSD kernel code in this like into the kernel's FFS file system read as we could in a normal application. We can let's just do a small experiment. So what would of a very lightweight hypervisor, so you could And the hypervisor, it's not a hypervisor in the normal virtualization sense, like where you system? Is this a monolithic kernel operating system so that you can choose at runtime which kernel is a way to write kernel drivers where be possible, well, you saw it running. You're kernels. Now, what is a Rump kernel? A Rump driver on top of the lightweight hypervisor we running kernel drivers in any configuration Rump kernel is the lightweight virtualized do stuff. In this case, the driver is the TCP IP implemented the interface driver which allows shared memory. So this is something that would be imagination to come up with a more and more from a standard NetBSD installation. The script experimenting and use this as a basis for their have something limping along by the end of the model actually work where you have a separate start from... This is a very interesting example these partitioned operating systems, which are C2JavaScript compiler and some amount of hacking this is available on the Internet and it's just a being run in a browser and it's very useless fairly certain that you are imaginative enough to should try their hands at being an any kernel. done something useful. So in addition to cool if I could actually kill the TCP IP stack