CheriBSD: A research fork of FreeBSD
Davis, Brooks
CheriBSD is a fork of FreeBSD to support the CHERI research CPU. We have extended the kernel to provide support for CHERI memory capabilities as well as modifying applications and libraries including tcpdump, libmagic, and libz to take advantage of these capabilities for improved memory safety and compartmentalization. We have also developed custom demo applications and deployment infrastructure for our table demo platform. In this talk I will discuss the challenges facing a long running, public fork of FreeBSD. The challenges I discuss will include keeping up with current, our migration from Perforce to Git and the difficulty-and value-of upstreaming improvements. I will also cover our internal and external release process and the products we produce. CheriBSD targets a research environment, but lessons learned will apply to many environments building products or services on customized versions of FreeBSD.
 I'm Brooks Davis. I'm with SRI International, and I'm part of the team of researchers working on the Cherry project, which I'll get into in a moment. I'm here to talk to you today about CherryBSD, which is our fork of FreeBSD to support the Cherry processor. In many ways, similar to, for instance, the TrustedBSD project or the HardenedBSD project, and that we're going off in another direction, we're going off in a direction that FreeBSD isn't ready for, for one reason or another. In the case of TrustedBSD, it was that there was large swaths of fairly disruptive technology that had to be proved out at scale before they could be merged into the tree, before you're going to make changes to thousands of places in the code. CherryBSD is a little different in that CherryBSD is about porting FreeBSD to a new CPU with new instructions, a new C compiler, some significant changes to the C language. So a wide variety of things that obviously we can't just dump in the FreeBSD tree, nothing else you can't buy the CPU. So people might object to large-scale changes for something you can't even use. But first, a little background. As hardly a week goes by where you don't hear about some new malware, some new breach or malware problem or whatever, Anthem losing 80 million customer records, the banks losing hundreds of millions of dollars in things like the target breach. Or I think this one actually, if I remember right when I created this slide, this is a reference to the banks simply failing to notice hundreds of millions of dollars in fraudulent transactions that were made to move money to another bank, and then they took the money out and poof. Or one of the most recent ones, the Office of Personnel Management in the US, basically HR for the civilian part of the US government, they lost 3.2 million HR records. So basically everything you need to know to steal someone's identity. So this is the daily reality of computing and the internet. So we decided it was time to do something about it a bit. The approach we're taking is one that's been taken for quite a while, which is application compartmentalization. Compartmentalization decomposes software and isolated components. Each sandbox runs with only the rights needed. So you can allow a least privilege approach. One common example of this that you probably as Unix users use every day is SSH has privilege separation. So that the most risky bits that must run as root, the bits that must run as root are separate from the most risky bits that do all the crypto handling, or as much as possible. And the goal here is that you can take an application, you can start with an application like in this example, GZIP, and you can cut it up into multiple pieces. So the compression logic, which is what people screw up because they write it in tight C code, designed to be fast, designed to trick the compiler into generating the best code they can, at least 20 years ago whenever they made those decisions. You can put that off in a process which has limited rights. In this example, this example maps pretty well to Capsicum, which we already have in FreeBSD. It is a process-based framework where we have capabilities, which are unforgible tokens of authority, are file handles, very fundamental thing in Unix. And you can enter a capability mode where you cannot open any new file handles through arbitrary namespaces. You must obtain them via other capabilities. And you can restrict the rights on those capabilities. So for instance, in the GZIP example, what GZIP does in its most basic mode is it opens one file, and it opens another file handle, creates a new file, and it does a bunch of work and then copy moves data from one side to the other. You obviously don't want the input file to be writable. You don't want the output file to be readable, or that's probably fairly harmless. But in principle, you don't really want that. Capsicum works great for something like GZIP. It works great for OpenSSH, where the existing framework was already, the existing privilege separation was already based around Unix principles. But it doesn't scale to the kind of things you might want to do. For example, in a web browser, right now, a web browser like Chromium has a separate process for each tab, at least until you open too many of them. The problem is that if you have too many processes, you start to run out of TLB entries on your CPU. I know of one instance where, in addition to running out of TLB entries, some systems are running out of address-based identifiers. So even though their processes are often quite small, if you have too many of them, you still get TLB misses simply because you've had to reuse an address-based identifier. So there's some serious scaling problems. And that's even just with tabs. What you'd actually like is to render every image in your browser in a separate sandbox so that when something does go wrong, the email that somebody sent you that had a bad image embedded in it can't read your bank statements, can't hijack your password reset URLs, all of which are in your email and all of which are in that tab and in that process. So we're trying to make this, we want to make this scale. So we're doing that in hardware. So as I was saying, with process separation, you can avoid this problem where you have one process here and as a pointer to a buffer in some other part of the program or in some other part of the application. If it's in another process, this arrow doesn't work. And we'd like to do that in the single application and have that mechanism for every pointer. So we can have every C object be a separate thing that you cannot manufacture a pointer to. So with Cherry, we are doing that with capabilities. So we've created fat pointers. These are 256 bit pointers. Yes, that's big. So each pointer has an offset which is in practice the pointer. It is where you're pointing in memory. And you have a base and a length, which are relative to your virtual address space. And there's guarded manipulation of this, which means that you can only create a capability by deriving it from another capability. And the only derivations that are allowed are to shrink it. So you can increment the base and move up in address space or you can shrink the length. You can also reduce the permissions of the pointer. One important thing here, though, is that, and the reason we have an offset, in our initial design, we did not have an offset. It turned out that in real world C code, that's not OK. Because you can't just keep shrinking your data. Because what you do is if you see an application program like FFMPEG, inside LiveAV codec, there is code where they pass a pointer to the middle of a piece of data to be decoded. Because the compiler generates better code that way, because it can use small, positive, and negative intermediates. So we have this offset, which allows our capabilities to be near perfect replacements of C pointers. And the reason this works is that you only check that the offset is inside the base and length at dereference time. So as I alluded, we want these to be C pointers. So we've done that. And we have two modes where you can use them. One is a hybrid mode. This is where you have conventional, in our case, MIPS64 code. And you annotate some of the pointers in your code as being capabilities. That means the compiler saves more space for them, uses the correct manipulation instructions to access them. And it works. So you can use this in code if you want to just protect a few buffers that are very, very important. But it is a lot of work. We made some changes to TCP. And I'll talk a little more about them later, where I added bounds to the packet buffer that was being dissected. And that worked, but that was many days of unpleasant changes to the code and thousands of changes. So we ended up adding a pure capability mode, where every pointer is a capability. So in translation units or files that are compiled in this mode, all pointers are 256 bits. They're all bounds checked. Almost all bounds are correctly inferred. So if you malloc something, you get a pointer that's exactly the size of the object that you allocated. If you try to go outside the bounds, things go boom in a nice predictable way, rather than randomly corrupting your memory. So as I say, we have a range of ABIs here. So we have the Pyramid 64. This is conventional FreeBSD runs on this processor without modification. And if you use it in this mode, you get no benefits, but it also doesn't require any work. The important part is to work your way through. And so there's a bunch of different approaches we can take. But we have a lot of hybrid code today and some pure capability code. We're increasingly moving towards pure capability code. And here's an example of the sort of things that we can support with the processor. So this is our current situation for the most part. We have a conventional kernel. It's running standard MIPS 64 code. We have a conventional application, also running standard MIPS 64 code. And we have Zlib in a compartment. Zlib's chosen because it has a small interface and it was easy and it compiled. And it has pure capability code inside and then a little hybrid wrapper around the outside that lets us call in. The application, in one example that we've talked about in one of our papers, is a GIFT to PNG. Doesn't know that the library is protecting it. And in fact, there's essentially no performance impact in this case. But if there's a bug in Zlib, it will fail stop. And even if it doesn't manage to fail stop, it's very difficult to gain control in the application. If it's a pure, within the bounds of C, control flow bug. You can also see a world where we're working right now actively to add pure capability application, which means yet another syscall interface. And pure libraries, one interesting thing though, and this example actually isn't ideal. But one of the interesting things is we can put conventional MIPS 64 code, say, to a binary library that we don't have any source code to for either because we bought it from someone who doesn't want to give a source code or because we lost the source code, which I've heard of some large internet companies doing. And we can put that in a little sandbox. And we can have a wrapper interface. So our library can be pure even though we have this bit that we don't have any control over, it can't get out of its sandbox. So it's less likely to fail stop if something goes horribly wrong. But what it can do is very little. It has access to a few, it might have, if it's a proprietary video codec, all it can do is write the wrong video frames and read some video frames, which it was supposed to be doing anyway. Because also, you can go farther along and of course start writing a microkernel, having a single address space application, lots of neat things. But we're quite a ways from that. If nothing else, LLVM doesn't yet compile MIPS 64 in a way that's usable for a kernel. So just a little bit more on the CPU. We have a prototype CPU. It's a 64-bit MIPS CPU, sort of R4K, so nicely out of patent. It has the Cherry ISA extensions that I've been talking about. It runs at 100 megahertz, which is a bit slow. But we lived with it in the 90s. And we actually have quite a bit more RAM, which helps. Although we have a gig of RAM on the boards and could have four gigs of RAM, which would have been a little unimaginable when I was in college, which is mostly good, except occasionally it gets a little exciting. One of the early bugs we had was a bug in the SD card controller that we got from the FPGA manufacturer that if you wrote a byte to the buffer two cycles in a row, it threw the second one away. There literally was code in there that said, we got two writes in a row, throw it away. We commented that out, and it works better. But one interesting thing is we only found this problem after reboot, because we had so much RAM that the entire file system fit in the buffer cache. So it was only when we flushed the buffer cache by rebooting the machine that we discovered that we'd corrupted the on-disk data structures. And Fisk would say, oh, what have you done to this poor file system? So it's an interesting one of the interesting challenges here. But the neat thing that we have is that we have this CPU. We have this operating system that I'm going to talk more about, and we have a modified LLVM. So we can run real software. And that's where things get exciting. We can really test things out. So CherryBSD is, of course, the BSD to support Cherry. It's a mix of platform support, which is to say drivers for things on the FPGA and peripherals, which is the very CPU. You can compile our CPU without the Cherry bits if you want. It's open source. And if you want to do things like run an hardware class, you can use that fairly simple CPU and add things like a better branch predictor as an exercise. And people at Cambridge have done that. So there's also support for the new ISA features. I'll give you a little rundown of how much changes that was in a bit. There's infrastructure to support the compartmentalization of libraries or whatever it is you want to stick in a compartment. There are some custom applications. I'll talk more about TCP dump and a few others. And then there are a bunch of build system improvements, because we're doing lots of slightly weird things. So we've done a lot of work like we have the ability to build free BSD and install it in a directory without any privilege, which is now being used in parts of the release build infrastructure. We did it because we wanted to have grad students building Cherry BSD. And I didn't want to give them a root and mess up our nice fancy machines. Here's a snapshot of our page on GitHub. So you notice almost 6,000 changes here relative to free BSD. And we're actually farther. And there's quite a few more than 6,000 now. That was a week ago. And we're a bit behind. We merge periodically. Talk quite a bit more about merging. But the main thing is, and what I think makes our project interesting, is that we both want this to be publicly accessible, because we have collaborators, which means we can't do things like rebase. But we've merged probably 200 changes or 300 changes out of this tree. So we have this huge set of changes that we have to maintain over time, which leads to some challenges in version control. So next up, here's a breakdown of the kernel changes. This was in our paper at IEEE Security and Privacy. So we added a bunch of headers, lots of various things, lots of access to assembly functions. There's some setup of Cherry and the kernel basically saying, turn it on. Give me the default capability, which at boot, there's a default capability, which is the address space. And you can start chopping that up later. There's contact switch code, because our new capabilities have to go into special registers. So we need to save and restore those registers. There's exception handling, because we've created all sorts of new ways to cause exceptions, since now an out of bounds pointer to your reference is a trap. There's some memory management, memory copying, swap. There's some support for the actual departmentalization bit, which is somewhat separate from the memory safety. Bit for system calls, bit for signal delivery, et cetera. Few thousand lines of code, which actually is not too bad, because 3DSD is several million lines of kernel code. And we've actually written a very tiny little microcernel before that's not very big. I don't even think it works anymore. So in addition to kernel changes, of course, had to make some tweaks to the runtime. The first, the biggest thing is mem copy and mem move need to be capability aware, even in capability oblivious code. One of the interesting things here, and one of the reasons why it's really important to have a team as large as we do, is actually our first version of the ISA didn't allow you to implement mem copy efficiently. You actually would have had to check every 256 bit chunk of memory to see whether or not it was a capability, and then copy it with capability instructions or copy it with regular instructions. That would be insane. So you can now use capability instructions to copy non-capability memory. And it actually turns out that 3DSD's generic mem copy and mem move implementations in C required almost no changes. We just had to tell it, oh, just use the right size as the basic word size when doing copies. And it generates the right code. I was actually pleasantly surprised. We, of course, have assembly versions as well that are a little better, but that's the basic. The nice thing is with the C integration, we get a lot of that. We also had to add explicit versions for hybrid code. We had to have explicit versions of things like string and memory manipulation functions so that we could have a mix of arguments. That, I think, I'm not sure that's going to stay. But in our current infrastructure, that's how it has to be. And then there's a bunch of interesting cases. In current C code, like in the sterln function, it assumes that once it's aligned to the front to the size of a word, then it can always read whole words. But we have byte granularity restrictions. So if you can definitely read the next byte, that doesn't mean you can definitely read the rest of the word. So that had to be fixed. It was easy to do, but required a bit of change. And one thing that we're working on right now is taking the syscall implementation in libc and moving it into a lib syscalls. So yes, I'm going to be asking for X runs on the ports at some point here, because that's going to be exciting. But it'll both help us, because inside a sandbox, we may want to attempt to make syscalls and have the kernel mediate based on some new configuration that's capsicum-like. Or we may want to have something that just simply proxies the syscalls back to the privileged part of the application. It can make the decisions. It's also be useful for people like Google, who are building Android with FreeBSD. And their syscall layers, obviously, totally differ. So the separation, I think, will be generally useful. Next bit, bits for compartmentalization. So libcherry is the library for creating sandbox libraries. Or for creating sandboxes, you can instantiate objects. You can give them types. And then you can allocate copies of the objects, do resets, that sort of thing. So that's the core functionality for implementing compartmentalized libraries or compartmentalized TCP dump, for instance. It also has a loader and a runtime linker, since objects have a new calling convention. And we don't want to have to write horrible little RPC-like things, where you say, well, I'm going to pass eight integer arguments and eight capability arguments, and try to remember which ones go where and whatnot. We did that for a while. It sucked. We're also working on a syscall, it also includes some syscall implementation bits. So compartmentalized can call out to an object in the privileged process. Something with a confusingly similar name, we have the libcherry directory. This is where peer capability versions of libraries live. It's also where the objects live. It's a lot like the 32-bit support for 64-bit machines. In fact, I copied and pasted this stuff in makefile.inc1. One reason why merges suck is lots of turn in that file. So eventually, right now, we're only using those libraries to build little compartmentalized libraries that have a hybrid outside interface. Longer term, the plan is to have pure capability libraries. So as part of your transition from a conventional ISA to cherry, you'll be able to pick points along the way, not only at the library level, but at the application level. So you can have your most important or, well, either most important or easiest to modify applications be pure capability. And other applications can remain as they are. You can still have your binary only whatever program it is that you have. So we also have demo applications. This is a screenshot here or a picture here of the very weak PowerPoint-like program that I wrote as a demo. So we actually gave our talk to DARPA at the principal investigators meeting on our tablet, which you can see Robert holding here. And when we got to the end, we said one more thing. We're running on our, not only are we running on our hard drive, but the slide deck hasn't exploited it and triggered a Trojan. And Cherry successfully defeated the Trojan. I don't know if it was very hard since I wrote all of it. But nonetheless, the technology did work and does do this. So we've got a bunch of little custom applications as part of CherryBSD. Those don't present any particular challenges for us in terms of CherryBSD maintenance. The thing that does present bigger challenges is we're also modifying existing applications. So at one point, we took a look at what's compartmentalized Wireshark. Wireshark's full of vulnerabilities. It's a giant program. We spent quite a bit of time on it and said, well, that was insane. It's 3 million lines of code and uses Glib. And it's all very complicated. We decided to do TCP dump instead, which has the advantage of being in the base, which is kind of good and bad. So our first version, we had a version that was very simple. We just compartmentalized a section, and it was standard MIPS code on both sides. It was just a little bit of hybrid code to get in and out of the sandbox. We later added memory safety, which I alluded to before. And that's 6,000 lines of changes, a whole lot of not fun. And part of that, though, was actually one of the things that we learned an interesting lesson, which is that we had no ability in TCP dump. There's a modest amount of code that advances the buffer, advances the buffer, advances the buffer, and that's, oh, whoops, I forgot. I want something a little back behind my pointer. That's fine in C. That's a perfectly legitimate and reasonable thing to do. However, before we had the offsets, that didn't work, because we'd incremented the base, we incremented the base, we're going to base, oops, can't go back. That's not allowed. So we added offsets. That helped quite a bit. I added per-particle disector. So this is the most protection you could get in TCP dump, which is to say every protocol lives in its own sandbox. So as your dissecting IP goes fine, TCP goes fine, you call into HTTP, or maybe probably more realistically, you call into something like SNMP, the ASN1 parser is broken as it always is, and gets exploited. You can still trust the TCP and IP dissection, because you've failed in a deeper sandbox. So that required a modest amount of code change. You had to change all the call sites. That's pretty reasonable. We went to pure capability mode. That got rid of tons of annotations, which was nice. And then the pure capability mode, though, removing annotations, was actually driven by the fact that FreeBSD got a new version of TCP dump. So this is one of the things that's really interesting about working on a real world project and keeping your tree up to date, is that we learned some lessons about maintainability the hard way. We tried to do this import here. There were 3,600 conflicts in Git. Seemed like a bit much. So we made a bunch of changes, got back to here. When we added the linker support, that reduced the amount of junk in the code base. And actually, this is dropped dramatically, because somewhat surprisingly, the TCP dump maintainers accepted my change to shuffle 1,000 lines of code around within the tree and put it in different places. So the interface between the deceptors and the front end is both fairly narrow in that it's only five or six functions and fairly simple, which should help build a better capsicumized version of TCP dump, but also simplifies my code. We'll see how my next merge goes, because the approach I took when I upstreamed it is a bit different. There's also a bit of infrastructure work we've done. I've alluded to force of the unprivileged builds. We've added some hacks to let us do change the compiler on a per program and per file basis. That's because early on, our custom LLVM was not robust enough to actually compile all the code. So we wanted to focus on the code where we could do something interesting. And then over time, expanded out, it's getting close to be unable to compile everything. I spent quite a bit of time the last few weeks trying to compile things and then sending David Bug reports when the compiler crashes. And some other hacks, which will help some upcoming changes, like we strip binaries during the build rather than using the install program, things like that. For more information on the general what's in CherryBSD, particularly Barry's stuff, I suggest reading the journal article that I wrote a few months ago. Pretty good. Now on a bit to revision control. This is one of the places where we've had a lot of challenges. We started off in Perforce, which was the conventional way that people did forks of FreeBSD in the past. Supported on FreeBSD project infrastructure, which is good. Merging is very good. It really is a good way to maintain something that's a fork of FreeBSD in the long term. I've done it at previous jobs as well. And it's easy to maintain what I'm calling here stacked branches. So for a while we had a BarryBSD, which was the platform bits. And it sat in between CherryBSD and FreeBSD so that we could try and keep, maintain some separation there. We merged everything in the BarryBSD branch. That wasn't too weird and then got rid of it at some point. But that was nonetheless quite helpful. And our team already knew it. So that was a good reason to stick with Perforce. Downside is Perforce sucks at public access. You have to give people an account to give them access to the system. Every checkout involves adding server state. So even if we were willing to give out a lot of accounts, eventually the project would run out of resources. It's very easy to get the situation where your Perforce server needs to have a half terabyte of RAM. We probably would have had to buy it for the project in that case. Adding users is a bit annoying. And the offline support isn't very good. That's not too big a deal, but it's not good. So in October 2013, we decided we needed to have public access. There were people at MIT Lincoln Labs and some other places who wanted to start using CherryBSD. So we wanted to give them direct access to the repository rather than having to take the time to package up dumps and package up snapshots and push them out and QA them and all that. So we moved over to GitHub. We lost a little history granularity in the process because many of the commits couldn't just be applied one at a time because things had moved on and they'd been merging. But really not too bad. However, it was a bit of a trial by fire for using Git sort of at scale. FreeBSD is a bit on the big side for Git. And our export has some weird features that it'll get to. Also, I and Robert, who are the main CherryBSD developers, were not experienced Git users at the time. So lots of excitement. It's not clear that our model was the right model. But it's the one we've got. So we're kind of stuck with it. What we ended up doing is we forked the FreeBSD repo on GitHub. One thing that I found is kind of weird about GitHub's forking model is if you want to fork CherryBSD, it seems that you get a copy of the FreeBSD repo. That's what happened when I tried to do it recently. That might simply be because I already have a copy of the FreeBSD repo. But so if someone wants to try it, it's only a gig or so. GitHub will never notice. But that was a bit odd. And we did all the commits to the master branch. I'm not sure that's the right solution, but it's what we've done. And at this point, we're stuck. We're not going to do a forced rebase and mess everything up. And then we merged changes from the FreeBSD upstream periodically. Typically, the typical working model is we merge changes when we need something. Or after a big deadline's passed and we realize we're behind, we'll do a merge just to sort of catch up while we have some breathing room. So our first attempt is sort of the basic obvious thing that you might think to do is we fetch upstream. We merge master into our current tree in a branch. But nonetheless, we just merge it. It mostly works. The first few times, it went pretty smoothly. There were some sort of strange looking conflicts that I didn't understand at the time. But it worked. We got past them. Although one annoying thing that we still not haven't resolved is that rebases go horribly wrong. So if somebody else pushes, well, you're doing a merge, you just have to throw the merge away. Rebases never work in this case. So however, after a few times, we came along. We started doing work integrating the VT console stuff into our tablet platform. And I did the merge. Everything compiled. Everything seemed to work. Pushed it, and something was broken in VT. Who knows where? It wasn't due to a merge conflict. It was actually due to an API change, it turned out. The problem is, so this is sort of a notional model of how it works. You're going along, you're going along, you're going along. You merge upstream. It's all good. Problem is, this is more like it. And this is actually much simpler than reality. Reality is you're going along, you're going along, you're going along. Thousands of changes occur. And then you merge from upstream, and you pull in another three months of development, which is typically several thousands of changes. And if you try to bisect, well, in our case, it was an API had changed. So all these were fine, but they don't include any of our code. And all these are fine, because the problem was here. Well, here, I guess, technically. And so there's nothing to look for. So that was really annoying. Ed found it eventually. So I ignored it for a week. I didn't need IOS, so it was OK. So I wrote a tool that I whimsically named Mergeify. It merges one commit at a time. Because from the perspective of a consumer of FreeBSD, every change is a feature. That's not perfectly accurate, because sometimes it's a commit that's broken, and then another commit that fixes it, or a commit that's broken, and then a few commits, and then a commit that fixes the previous one. There's no way to deal with that case in a sensible way, so I just punted. But merging one commit at a time does help in that the overall system is much, it does help in that you now have commits you can bisect. Each of those merge commits is useful. The feature I haven't added to the tool yet is something to knock out all of the child commits from the bisect. It would be pretty easy to do. I just haven't written it yet. So that you only consider the commits that actually change your branch. It's one of the key things that I figured out over time is that from our perspective, it's only those merges that matter. Everything else, it's got some history who cares. So the first attempt, we just merged every commit. And then TCP came along, and there was an update in contribute, and things went really strange. We got these merge results that were completely nuts. Mostly, it was the top level make file would get something from contrib squished into it. Turns out that what's happening in the FreeBSD export is things in the vendor branch have a common parent with the FreeBSD tree, the empty repository. So in fact, there's a slash make file in both of them. And Git says, they've got common blank lines, squishes them together, and really goes badly. The problem is actually, you don't care about that commit. It's not important. What you care about is the commit that merged that into the FreeBSD tree. So I changed the code to only pick the direct commits to the branch and merge each one of those one at a time. I was going to do a demo here, but I'm getting a little short on time, so I think I will skip the demo and come back to it at the end, particularly since there's a bit of fussing with the projector to make it work. So I alluded to be for rebases broken. And I think it's, again, because rebase is applying all the commits individually rather than those individual merge commits. So it's on my list to attempt the change to try applying those commits one at a time, basically re-implement rebase. Should be doable, but I haven't done it yet. I have an upcoming merge that looks like it's going to be exciting, so maybe I'll fix it then. So I've got some to-dos for mergeify. I need to add this rebase mode, this bisect mode I've talked about before, where you can skip all the commits that don't make any sense to look at. And one thing I would like to do eventually is periodically, say, may every 10 commits or every 100 commits, check that things build. Right now, I don't get to do that, which is sometimes a little frustrating. I get to the end and I discover get botched a merge somewhere, or I botched a merge. My current workflow is anytime I do anything by hand, I assume I screwed it up and do a full build just to make sure that it's right, that I'd really like to do it every commit. That would be really slow. Right now, it takes several seconds to merge each change because get is fast, but even if it's all in RAM on a fast machine, previous these repo is big. So this would take quite a bit longer, but hopefully, Simon Gerrity's meta mode will make it fast enough that I could try every 10 commits or so, and I could bisect within that once I knew what was going on. So on to another topic, upstreaming. The best way to remove merge conflicts, of course, is to upstream your changes. That way, they become everyone else's merge conflicts, and not mine. And if that section of the make file is the same, then things are good. Particularly, the top level make files in FreeBSD, I have made so many changes to them. I get conflicts all the time. Just a bit. Anyway, so there's some questions of what to upstream. I think there's some philosophical questions here. The answers will vary depending on your project, the nature of your work. Obviously, drivers for things that people can use should be upstreamed. And drivers are one of the better things to upstream as the owner of the driver, because it means when somebody changes the infrastructure, they have to update your driver. And you don't have to come along three months later and say, what the heck happened? Why does my driver not work anymore? Why does it not compile? That sort of thing. Also, general infrastructure. We've built quite a lot of infrastructure along the way. So we've been upstreaming that as we can, or as we have time. So things that are shared by multiple external consumers. I like to try to upstream things that are useful to multiple people, even if they're not useful to most consumers of the project, are not entirely useful in the base system. If you do a cleanup that is critical to you, but doesn't matter to the base system, but other people are using it, I think that's a good thing to upstream. But that's a little tricky in that those are the sort of things that get broken. And I've had some issues with that. And also, things that are just low impact and are likely to generate conflicts. So we actually have a new signal that we've been meaning to upstream. Because Capsicum might use it eventually. It won't immediately. But every time someone adds a new signal, it creates a ton of conflicts. Because in every single case, it's adding something in exactly all the spots we changed. So I think we'll upstream SIGProt soon. So things we have upstreamed. FDT support for MIPS, so a flattened device tree. It was there. It didn't really work. We fixed that. Bunch of drivers, bunch of driver improvements. We added a way to turn on the floating point support that was in the kernel. And then we made it actually work for MIPS. We added bootloaders for MIPS. We've added the unprivileged build stuff. It's a bunch of other stuff. And I've had it for four years now, so lots of things. We also have some sort of related upstreaming we've been doing, not to FreeBSD, but to other projects. So if you saw Stacy and Sean's talk earlier today, the QMU user mode work that's letting us build ARM in MIPS 64 packages, which will soon be official, that's work we did because we needed packages. Turns out we don't actually use them very much right now. But notionally, we knew we were going to want to do demos without a tree code eventually. So we needed some way to build them other than trying to build 2T or something on 100 megahertz MIPS. Didn't seem like a good idea. Meantime, to reboot good power failure would be an issue. So we've done a lot of improvements to Clang and LLVM on MIPS 64. The imagination technology people's focus is definitely on little Indian 32-bit MIPS, and we're big Indian 64-bit MIPS. So we've had to fix a lot of things. And I've been upstreaming stuff to TCP dump, mostly to make my life easier merge-wise. But they seem to like the direction, the compartmentalization direction. So that's nice. Internally, we've been doing some releases. We've done them internally for a long time, mostly snapshots. Initially, it would be periodically I would do a release build using our little build system and push something out and put it on the wiki. We've done a couple of restricted releases to partners. One of the problems we have is with the FPGA, the license agreements mean you can't share compiled bit files. So we give them out to a few people who are in the program and we know how it licenses. But it's sort of the AT&T Unix problem that Kirk's alluded to. And we started doing public releases. So we just did our first public release three weeks ago, I think. So cherrycpu.org, you can download our CPU. You can compile it. Don't run the bit file that comes out without some careful testing, because at least the ones that our Jenkins build is producing, the fan doesn't work. Seems to run OK as long as you don't do too much. Then it gets hot and things misbehave. So another small change gears. Since my focus on this is a free BSD developer, some tips for developers, many of which go around the idea of as you develop new ways to build the kernel or new ways to build the OS, you spend an awful lot of time compiling. So we've got some suggestions for ways to deal with this. The first and probably most important one, use a big enough machine. Seriously, if somebody is paying you to wait for compiles, they can buy quite a lot of hardware fairly quickly for what they're paying you. My view is you want enough RAM to hold all the source and all the output. We know 128 gigs is enough. John Anderson has one that's working well for him. The machine we use is a 256 gig machine. We have fast ZFS, a Z mirror with a half terabyte of SSD. Works pretty well. I would say that 128 gigs should be enough. 256 gigs definitely should be enough for anyone, except last week when there was a compiler bug. And we had a dozen clang processes using 70 gigabytes a piece. We ran out of swap. It was exciting. And it turns out, well, they ran out of swap, and then they started dumping core. And you can't kill processes that are dumping core, so that was unfortunate. But usually it's big enough, and we literally could not do the work we're doing without this machine. It's only about $5,000. It's definitely worth it. Another thing that I've found really helpful is I use a little push notification service that has a rest interface. So the one I use is pushover.net. There's a web browser-based client that's $10. And there's Android and iOS, which also are $10 a piece to activate. You couldn't write the service for that, so it's totally worth it. And I have a little command-blend wrapper. So I run this command-notice, run a command, and then I get something like this on my screen and a bing and my phone buzzes. And it's really great at reducing the latency between when my compile finished and I noticed that it finished. So that's been really helpful. Another thing, I build everything in Tmux sessions. And actually one of the things that I discovered part way through is really important to switch away from the build. Do not send it over the network. Even over gigabit locally, enough stuff buffers that it significantly delays your ability to get back to the prompt. So better to not render it at all. That was slightly surprising. I found that at Cambridge. I knew it was an issue over the Atlantic. I would switch away when I was tethering. So I didn't just waste all that bandwidth. But it turns out, yeah, always switch away. And I guess one final little tip, continuous integration is really great. We do full OS builds after every compiler or OS change. They take about 20 minutes. We also do full releases at least a couple times a day. Keeps everything working. And one of the key things for us is because our architecture is weird and we are making strange changes, we build both Cherry, MIPS 64, and AMD 64 all the time. AMD 64 is actually a really good canary because it also builds I3D6 libraries. So that means we build a whole lot of stuff and just keeps us honest. We are a research project, but we really want this stuff to be used for real. So we don't want to get to veer down some blind alley in code changes and then come back and say, oh, I have three months of work to get this thing functional again. We're into the shape where I could merge it. So we do that all the time. And also, when we're working on a release now, we create a separate set of Jenkins jobs to build the release daily in the release plan, just to make sure everything is stable. I'll mention some of our papers that we've got published. We have three top two papers in the last year. We had a Cherry hardware paper. So here's the hardware paper. The Cherry capability model, we're visiting risk in an age of risk. That's kind of mostly what I talked about earlier. We have beyond the PDP11 processor support for a memory safe C abstract machine. That was an ass-bloss. If you're a C-geek, it's a great paper. I definitely recommend it. I learned all sorts of weird things about C writing the paper. So that was fun. And then we had a compartmentalization paper at IEEE Security and Privacy a few weeks ago. There's also an ISA document and whatnot on the Cambridge website. We have a future work here. So we're working on a pure capability for VSD. So probably it'd be a very long time, even once hardware exists, before you'll ship a version of VSD that only uses pure capability code. But it's pretty likely that you'll want to run a fair bit of pure capability code. And we also realized recently that the best way to get a ton of code running is to be able to have a pure capability build within free VSD. So we can just compile everything and try to use it, run it through the test suite and see what happens. It'll no doubt be exciting. We're also working on, we'd like to add Cherry to the kernel. The code we have in there is all assembly or macros around inline assembly. It works for what we're doing, but we'd like to be able to do things like protect mBuffs, protect storage buffers, compartmentalize the kernel, make it into a microkernel. And we're hoping, well, so one of the other things, as I said, early on, 256 bits is a pretty big pointer. There'll be lots of overhead in terms of cache footprint and whatnot. In extraordinarily pointer-heavy benchmarks, it's about 20% performance overhead, which is not acceptable. So we were working on 128 bit compressed capabilities. Some interesting trade-offs we're exploring. We're pretty confident it's going to work, but there are a lot of details. And in our simulations, that should get the overhead down to about 3% in a benchmark that's basically data structures that are pointers. And then we're also looking at non-nibs architectures. So happy to answer any questions. I just threw up on here our timelines. We've been at this about five years now. I am guessing we've got over 50 years of work into it. And we've got quite a bit to go. But it's a fun project. And I think we've done a lot of interesting development and stuff that's generally useful for free BSD along the way. Any questions? Yes? So your communication stuff, are you synthesizing varialog stuff and then running your compiler software on that full stack integration? Yes. So we're actually using BlueSpec System Varialog, which is a Haskell-derived HDL. We both compile it. So it has a mode to compile to a cycle accurate C simulator. So we've been doing it for a long time. More recently, we've started synthesizing bit files, loading them on FPGAs, and then running the operating system on them. And that's hide in Jenkins? Yeah. That's Jenkins. The Jenkins cluster has grown quite a lot over the last year. On the hardware side, one of the RA's is amazingly patient and willing to deal with broken junk. Is that part of your structure open for? No. I don't think so. Parts of it might be in the release. I'd have to go look. If you want to ask me later, I can take a poke at the GitHub repo of what we've actually put out and what's buried in the Jenkins config. You said you're writing out another curriculum you just call in your face. Is that the Cloud ABI work? No, this is not the Cloud ABI work. I probably should have gone to the Cloud ABI talk before, but I talked to it a bit. So what we're doing is adding a new syscall ABI. So like the lib32 and the free BST32 emulation, we're adding a cherry ABI. It's sort of the first cut. Not clear what the long-term right answer is, but that's one we know how to do. So I did a first pass at syscalls.master. The great thing is you take all the lines that say compact, clear them out. We've never shipped with those versions, and we never will. Well, thank you all for coming. If you have any questions later, we'll bring them back to you.
 I'm Brooks Davis.  I'm with SRI International,  and I'm part of the team of researchers  working on the Cherry project,  which I'll get into in a moment.  I'm here to talk to you today about CherryBSD,  which is our fork of FreeBSD to support the Cherry processor.  In many ways, similar to, for instance,  the TrustedBSD project or the HardenedBSD project,  and that we're going off in another direction,  we're going off in a direction that FreeBSD isn't ready for,  for one reason or another.  In the case of TrustedBSD,  it was that there was large swaths of fairly disruptive technology  that had to be proved out at scale  before they could be merged into the tree,  before you're going to make changes to thousands of places in the code.  CherryBSD is a little different in that CherryBSD  is about porting FreeBSD to a new CPU with new instructions,  a new C compiler,  some significant changes to the C language.  So a wide variety of things that obviously we can't just dump in the FreeBSD tree,  nothing else you can't buy the CPU.  So people might object to large-scale changes for something you can't even use.  But first, a little background.  As hardly a week goes by where you don't hear about some new malware,  some new breach or malware problem or whatever,  Anthem losing 80 million customer records,  the banks losing hundreds of millions of dollars in things like the target breach.  Or I think this one actually,  if I remember right when I created this slide,  this is a reference to the banks simply failing to notice  hundreds of millions of dollars in fraudulent transactions  that were made to move money to another bank,  and then they took the money out and poof.  Or one of the most recent ones,  the Office of Personnel Management in the US,  basically HR for the civilian part of the US government,  they lost 3.2 million HR records.  So basically everything you need to know to steal someone's identity.  So this is the daily reality of computing and the internet.  So we decided it was time to do something about it a bit.  The approach we're taking is one that's been taken for quite a while,  which is application compartmentalization.  Compartmentalization decomposes software and isolated components.  Each sandbox runs with only the rights needed.  So you can allow a least privilege approach.  One common example of this that you probably as Unix users use every day is SSH has privilege separation.  So that the most risky bits that must run as root,  the bits that must run as root are separate from the most risky bits that do all the crypto handling,  or as much as possible.  And the goal here is that you can take an application,  you can start with an application like in this example,  GZIP, and you can cut it up into multiple pieces.  So the compression logic,  which is what people screw up because they write it in tight C code,  designed to be fast, designed to trick the compiler into generating the best code they can,  at least 20 years ago whenever they made those decisions.  You can put that off in a process which has limited rights.  In this example, this example maps pretty well to Capsicum,  which we already have in FreeBSD.  It is a process-based framework where we have capabilities,  which are unforgible tokens of authority,  are file handles, very fundamental thing in Unix.  And you can enter a capability mode where you cannot  open any new file handles through arbitrary namespaces.  You must obtain them via other capabilities.  And you can restrict the rights on those capabilities.  So for instance, in the GZIP example,  what GZIP does in its most basic mode is it opens one file,  and it opens another file handle, creates a new file,  and it does a bunch of work and then copy moves data from one side to the other.  You obviously don't want the input file to be writable.  You don't want the output file to be readable,  or that's probably fairly harmless.  But in principle, you don't really want that.  Capsicum works great for something like GZIP.  It works great for OpenSSH, where the existing framework was already,  the existing privilege separation was already based around Unix principles.  But it doesn't scale to the kind of things you might want to do.  For example, in a web browser, right now,  a web browser like Chromium has a separate process for each tab,  at least until you open too many of them.  The problem is that if you have too many processes,  you start to run out of TLB entries on your CPU.  I know of one instance where, in addition to running out of TLB entries,  some systems are running out of address-based identifiers.  So even though their processes are often quite small,  if you have too many of them, you still get TLB misses  simply because you've had to reuse an address-based identifier.  So there's some serious scaling problems.  And that's even just with tabs.  What you'd actually like is to render every image in your browser  in a separate sandbox so that when something does go wrong,  the email that somebody sent you that had a bad image embedded in it  can't read your bank statements,  can't hijack your password reset URLs,  all of which are in your email and all of which are in that tab  and in that process.  So we're trying to make this, we want to make this scale.  So we're doing that in hardware.  So as I was saying, with process separation,  you can avoid this problem where you have one process here  and as a pointer to a buffer in some other part of the program  or in some other part of the application.  If it's in another process, this arrow doesn't work.  And we'd like to do that in the single application  and have that mechanism for every pointer.  So we can have every C object be a separate thing  that you cannot manufacture a pointer to.  So with Cherry, we are doing that with capabilities.  So we've created fat pointers.  These are 256 bit pointers.  Yes, that's big.  So each pointer has an offset which is in practice the pointer.  It is where you're pointing in memory.  And you have a base and a length, which  are relative to your virtual address space.  And there's guarded manipulation of this,  which means that you can only create a capability  by deriving it from another capability.  And the only derivations that are allowed are to shrink it.  So you can increment the base and move up in address space  or you can shrink the length.  You can also reduce the permissions of the pointer.  One important thing here, though, is that,  and the reason we have an offset, in our initial design,  we did not have an offset.  It turned out that in real world C code, that's not OK.  Because you can't just keep shrinking your data.  Because what you do is if you see an application program  like FFMPEG, inside LiveAV codec,  there is code where they pass a pointer to the middle of a piece  of data to be decoded.  Because the compiler generates better code that way,  because it can use small, positive, and negative  intermediates.  So we have this offset, which allows our capabilities  to be near perfect replacements of C pointers.  And the reason this works is that you only  check that the offset is inside the base and length  at dereference time.  So as I alluded, we want these to be C pointers.  So we've done that.  And we have two modes where you can use them.  One is a hybrid mode.  This is where you have conventional, in our case, MIPS64  code.  And you annotate some of the pointers in your code  as being capabilities.  That means the compiler saves more space for them,  uses the correct manipulation instructions to access them.  And it works.  So you can use this in code if you  want to just protect a few buffers that are very, very  important.  But it is a lot of work.  We made some changes to TCP.  And I'll talk a little more about them later,  where I added bounds to the packet  buffer that was being dissected.  And that worked, but that was many days of unpleasant changes  to the code and thousands of changes.  So we ended up adding a pure capability mode,  where every pointer is a capability.  So in translation units or files that  are compiled in this mode, all pointers are 256 bits.  They're all bounds checked.  Almost all bounds are correctly inferred.  So if you malloc something, you get a pointer  that's exactly the size of the object that you allocated.  If you try to go outside the bounds,  things go boom in a nice predictable way,  rather than randomly corrupting your memory.  So as I say, we have a range of ABIs here.  So we have the Pyramid 64.  This is conventional FreeBSD runs on this processor  without modification.  And if you use it in this mode, you get no benefits,  but it also doesn't require any work.  The important part is to work your way through.  And so there's a bunch of different approaches we can take.  But we have a lot of hybrid code today  and some pure capability code.  We're increasingly moving towards pure capability code.  And here's an example of the sort of things  that we can support with the processor.  So this is our current situation for the most part.  We have a conventional kernel.  It's running standard MIPS 64 code.  We have a conventional application, also running standard MIPS  64 code.  And we have Zlib in a compartment.  Zlib's chosen because it has a small interface  and it was easy and it compiled.  And it has pure capability code inside and then  a little hybrid wrapper around the outside that  lets us call in.  The application, in one example that we've talked about  in one of our papers, is a GIFT to PNG.  Doesn't know that the library is protecting it.  And in fact, there's essentially no performance impact  in this case.  But if there's a bug in Zlib, it will fail stop.  And even if it doesn't manage to fail stop,  it's very difficult to gain control in the application.  If it's a pure, within the bounds of C, control flow bug.  You can also see a world where we're working right now  actively to add pure capability application, which  means yet another syscall interface.  And pure libraries, one interesting thing though,  and this example actually isn't ideal.  But one of the interesting things is we can put conventional  MIPS 64 code, say, to a binary library that we don't have  any source code to for either because we bought it from  someone who doesn't want to give a source code or because we  lost the source code, which I've heard of some large  internet companies doing.  And we can put that in a little sandbox.  And we can have a wrapper interface.  So our library can be pure even though we have this bit that  we don't have any control over, it can't get out of its  sandbox.  So it's less likely to fail stop if something goes  horribly wrong.  But what it can do is very little.  It has access to a few, it might have, if it's a  proprietary video codec, all it can do is write the wrong  video frames and read some video frames, which it was  supposed to be doing anyway.  Because also, you can go farther along and of course  start writing a microkernel, having a single address space  application, lots of neat things.  But we're quite a ways from that.  If nothing else, LLVM doesn't yet compile MIPS 64 in a way  that's usable for a kernel.  So just a little bit more on the CPU.  We have a prototype CPU.  It's a 64-bit MIPS CPU, sort of R4K, so  nicely out of patent.  It has the Cherry ISA extensions that I've been  talking about.  It runs at 100 megahertz, which is a bit slow.  But we lived with it in the 90s.  And we actually have quite a bit more RAM, which helps.  Although we have a gig of RAM on the boards and could have  four gigs of RAM, which would have been a little unimaginable  when I was in college, which is mostly good, except  occasionally it gets a little exciting.  One of the early bugs we had was a bug in the SD card  controller that we got from the FPGA manufacturer that if you  wrote a byte to the buffer two cycles in a row, it threw the  second one away.  There literally was code in there that said, we got two  writes in a row, throw it away.  We commented that out, and it works better.  But one interesting thing is we only found this problem  after reboot, because we had so much RAM that the entire file  system fit in the buffer cache.  So it was only when we flushed the buffer cache by  rebooting the machine that we discovered that we'd corrupted  the on-disk data structures.  And Fisk would say, oh, what have you done to this poor file  system?  So it's an interesting one of the interesting challenges  here.  But the neat thing that we have is that we have this CPU.  We have this operating system that I'm going to talk more  about, and we have a modified LLVM.  So we can run real software.  And that's where things get exciting.  We can really test things out.  So CherryBSD is, of course, the BSD to support Cherry.  It's a mix of platform support, which is to say drivers  for things on the FPGA and peripherals, which is the  very CPU.  You can compile our CPU without the Cherry bits if you  want.  It's open source.  And if you want to do things like run an hardware class, you  can use that fairly simple CPU and add things like a better  branch predictor as an exercise.  And people at Cambridge have done that.  So there's also support for the new ISA features.  I'll give you a little rundown of how much changes that was  in a bit.  There's infrastructure to support the compartmentalization  of libraries or whatever it is you want to stick in a  compartment.  There are some custom applications.  I'll talk more about TCP dump and a few others.  And then there are a bunch of build system improvements,  because we're doing lots of slightly weird things.  So we've done a lot of work like we have the ability to  build free BSD and install it in a directory without any  privilege, which is now being used in parts of the release  build infrastructure.  We did it because we wanted to have grad students building  Cherry BSD.  And I didn't want to give them a root and mess up our nice  fancy machines.  Here's a snapshot of our page on GitHub.  So you notice almost 6,000 changes here relative to  free BSD.  And we're actually farther.  And there's quite a few more than 6,000 now.  That was a week ago.  And we're a bit behind.  We merge periodically.  Talk quite a bit more about merging.  But the main thing is, and what I think makes our project  interesting, is that we both want this to be publicly  accessible, because we have collaborators, which means we  can't do things like rebase.  But we've merged probably 200 changes or 300 changes out of  this tree.  So we have this huge set of changes that we have to  maintain over time, which leads to some challenges in  version control.  So next up, here's a breakdown of the kernel changes.  This was in our paper at IEEE Security and Privacy.  So we added a bunch of headers, lots of various things,  lots of access to assembly functions.  There's some setup of Cherry and the kernel basically  saying, turn it on.  Give me the default capability, which at boot, there's a  default capability, which is the address space.  And you can start chopping that up later.  There's contact switch code, because our new capabilities  have to go into special registers.  So we need to save and restore those registers.  There's exception handling, because we've created all  sorts of new ways to cause exceptions, since now an out  of bounds pointer to your reference is a trap.  There's some memory management, memory copying, swap.  There's some support for the actual  departmentalization bit, which is somewhat separate from the  memory safety.  Bit for system calls, bit for signal delivery, et cetera.  Few thousand lines of code, which actually is not too bad,  because 3DSD is several million lines of kernel code.  And we've actually written a very tiny little microcernel  before that's not very big.  I don't even think it works anymore.  So in addition to kernel changes, of course, had to  make some tweaks to the runtime.  The first, the biggest thing is mem copy and mem move need  to be capability aware, even in capability oblivious code.  One of the interesting things here, and one of the reasons  why it's really important to have a team as large as we do,  is actually our first version of the ISA didn't allow you to  implement mem copy efficiently.  You actually would have had to check every 256 bit chunk of  memory to see whether or not it was a capability, and then  copy it with capability instructions or copy it with  regular instructions.  That would be insane.  So you can now use capability instructions to copy non-capability  memory.  And it actually turns out that 3DSD's generic mem copy and  mem move implementations in C required almost no changes.  We just had to tell it, oh, just use the right size as the  basic word size when doing copies.  And it generates the right code.  I was actually pleasantly surprised.  We, of course, have assembly versions as well that are a  little better, but that's the basic.  The nice thing is with the C integration, we get a lot of  that.  We also had to add explicit versions for hybrid code.  We had to have explicit versions of things like string and  memory manipulation functions so that we could have a mix of  arguments.  That, I think, I'm not sure that's going to stay.  But in our current infrastructure, that's how it  has to be.  And then there's a bunch of interesting cases.  In current C code, like in the sterln function, it assumes  that once it's aligned to the front to the size of a word,  then it can always read whole words.  But we have byte granularity restrictions.  So if you can definitely read the next byte, that doesn't  mean you can definitely read the rest of the word.  So that had to be fixed.  It was easy to do, but required a bit of change.  And one thing that we're working on right now is taking  the syscall implementation in libc and moving it into a lib  syscalls.  So yes, I'm going to be asking for X runs on the ports at  some point here, because that's going to be exciting.  But it'll both help us, because inside a sandbox, we  may want to attempt to make syscalls and have the  kernel mediate based on some new configuration that's  capsicum-like.  Or we may want to have something that just simply  proxies the syscalls back to the privileged part of the  application.  It can make the decisions.  It's also be useful for people like Google, who are building  Android with FreeBSD.  And their syscall layers, obviously, totally differ.  So the separation, I think, will be generally useful.  Next bit, bits for compartmentalization.  So libcherry is the library for creating sandbox libraries.  Or for creating sandboxes, you can instantiate objects.  You can give them types.  And then you can allocate copies of the objects, do  resets, that sort of thing.  So that's the core functionality for implementing  compartmentalized libraries or compartmentalized  TCP dump, for instance.  It also has a loader and a runtime linker, since  objects have a new calling convention.  And we don't want to have to write horrible little RPC-like  things, where you say, well, I'm going to pass eight  integer arguments and eight capability arguments, and try  to remember which ones go where and whatnot.  We did that for a while.  It sucked.  We're also working on a syscall, it also includes some  syscall implementation bits.  So compartmentalized can call out to an object in the  privileged process.  Something with a confusingly similar name, we have the  libcherry directory.  This is where peer capability versions of libraries live.  It's also where the objects live.  It's a lot like the 32-bit support for 64-bit machines.  In fact, I copied and pasted this stuff in  makefile.inc1.  One reason why merges suck is lots of turn in that file.  So eventually, right now, we're only using those  libraries to build little compartmentalized libraries  that have a hybrid outside interface.  Longer term, the plan is to have pure capability libraries.  So as part of your transition from a conventional ISA to  cherry, you'll be able to pick points along the way, not  only at the library level, but at the application level.  So you can have your most important or, well, either  most important or easiest to modify applications be  pure capability.  And other applications can remain as they are.  You can still have your binary only whatever program it is  that you have.  So we also have demo applications.  This is a screenshot here or a picture here of the very weak  PowerPoint-like program that I wrote as a demo.  So we actually gave our talk to DARPA at the principal  investigators meeting on our tablet, which you can see  Robert holding here.  And when we got to the end, we said one more thing.  We're running on our, not only are we running on our  hard drive, but the slide deck hasn't exploited it and  triggered a Trojan.  And Cherry successfully defeated the Trojan.  I don't know if it was very hard since I wrote all of it.  But nonetheless, the technology did work and does do this.  So we've got a bunch of little custom applications as part  of CherryBSD.  Those don't present any particular challenges for us in  terms of CherryBSD maintenance.  The thing that does present bigger challenges is we're  also modifying existing applications.  So at one point, we took a look at what's compartmentalized  Wireshark.  Wireshark's full of vulnerabilities.  It's a giant program.  We spent quite a bit of time on it and said, well, that was  insane.  It's 3 million lines of code and uses Glib.  And it's all very complicated.  We decided to do TCP dump instead, which has the  advantage of being in the base, which is kind of good and bad.  So our first version, we had a version that was very simple.  We just compartmentalized a section, and it was standard  MIPS code on both sides.  It was just a little bit of hybrid code to get in and out  of the sandbox.  We later added memory safety, which I alluded to before.  And that's 6,000 lines of changes, a whole lot of not  fun.  And part of that, though, was actually one of the things  that we learned an interesting lesson, which is that we had  no ability in TCP dump.  There's a modest amount of code that advances the  buffer, advances the buffer, advances the buffer, and  that's, oh, whoops, I forgot.  I want something a little back behind my pointer.  That's fine in C. That's a perfectly legitimate and  reasonable thing to do.  However, before we had the offsets, that didn't work,  because we'd incremented the base, we incremented the  base, we're going to base, oops, can't go back.  That's not allowed.  So we added offsets.  That helped quite a bit.  I added per-particle disector.  So this is the most protection you could get in TCP dump,  which is to say every protocol lives in its own sandbox.  So as your dissecting IP goes fine, TCP goes fine, you  call into HTTP, or maybe probably more realistically,  you call into something like SNMP, the ASN1 parser is  broken as it always is, and gets exploited.  You can still trust the TCP and IP dissection, because you've  failed in a deeper sandbox.  So that required a modest amount of code change.  You had to change all the call sites.  That's pretty reasonable.  We went to pure capability mode.  That got rid of tons of annotations, which was nice.  And then the pure capability mode, though, removing  annotations, was actually driven by the fact that FreeBSD  got a new version of TCP dump.  So this is one of the things that's really interesting  about working on a real world project and keeping your tree  up to date, is that we learned some lessons about  maintainability the hard way.  We tried to do this import here.  There were 3,600 conflicts in Git.  Seemed like a bit much.  So we made a bunch of changes, got back to here.  When we added the linker support, that reduced the amount  of junk in the code base.  And actually, this is dropped dramatically, because  somewhat surprisingly, the TCP dump maintainers accepted my  change to shuffle 1,000 lines of code around within the tree  and put it in different places.  So the interface between the deceptors and the front end is  both fairly narrow in that it's only five or six functions  and fairly simple, which should help build a better  capsicumized version of TCP dump, but also simplifies my  code.  We'll see how my next merge goes, because the approach I  took when I upstreamed it is a bit different.  There's also a bit of infrastructure work we've done.  I've alluded to force of the unprivileged builds.  We've added some hacks to let us do change the compiler on a  per program and per file basis.  That's because early on, our custom LLVM was not  robust enough to actually compile all the code.  So we wanted to focus on the code where we could do  something interesting.  And then over time, expanded out, it's getting close to be  unable to compile everything.  I spent quite a bit of time the last few weeks trying to  compile things and then sending David Bug reports when the  compiler crashes.  And some other hacks, which will help some upcoming  changes, like we strip binaries during the build rather  than using the install program, things like that.  For more information on the general what's in CherryBSD,  particularly Barry's stuff, I suggest reading the journal  article that I wrote a few months ago.  Pretty good.  Now on a bit to revision control.  This is one of the places where we've had a lot of  challenges.  We started off in Perforce, which was the conventional way  that people did forks of FreeBSD in the past.  Supported on FreeBSD project infrastructure, which is good.  Merging is very good.  It really is a good way to maintain something that's a  fork of FreeBSD in the long term.  I've done it at previous jobs as well.  And it's easy to maintain what I'm calling here  stacked branches.  So for a while we had a BarryBSD, which was the  platform bits.  And it sat in between CherryBSD and FreeBSD so that we  could try and keep, maintain some separation there.  We merged everything in the BarryBSD branch.  That wasn't too weird and then got rid of it at some point.  But that was nonetheless quite helpful.  And our team already knew it.  So that was a good reason to stick with Perforce.  Downside is Perforce sucks at public access.  You have to give people an account to give them access  to the system.  Every checkout involves adding server state.  So even if we were willing to give out a lot of accounts,  eventually the project would run out of resources.  It's very easy to get the situation where your Perforce  server needs to have a half terabyte of RAM.  We probably would have had to buy it for the  project in that case.  Adding users is a bit annoying.  And the offline support isn't very good.  That's not too big a deal, but it's not good.  So in October 2013, we decided we needed to have public  access.  There were people at MIT Lincoln Labs and some other  places who wanted to start using CherryBSD.  So we wanted to give them direct access to the  repository rather than having to take the time to package up  dumps and package up snapshots and push them out and  QA them and all that.  So we moved over to GitHub.  We lost a little history granularity in the process  because many of the commits couldn't just be applied one  at a time because things had moved on and  they'd been merging.  But really not too bad.  However, it was a bit of a trial by fire for using  Git sort of at scale.  FreeBSD is a bit on the big side for Git.  And our export has some weird features that it'll get to.  Also, I and Robert, who are the main CherryBSD developers,  were not experienced Git users at the time.  So lots of excitement.  It's not clear that our model was the right model.  But it's the one we've got.  So we're kind of stuck with it.  What we ended up doing is we forked the FreeBSD repo on  GitHub.  One thing that I found is kind of weird about GitHub's  forking model is if you want to fork CherryBSD, it seems  that you get a copy of the FreeBSD repo.  That's what happened when I tried to do it recently.  That might simply be because I already have a copy of the  FreeBSD repo.  But so if someone wants to try it, it's only a gig or so.  GitHub will never notice.  But that was a bit odd.  And we did all the commits to the master branch.  I'm not sure that's the right solution, but it's what we've  done.  And at this point, we're stuck.  We're not going to do a forced rebase and mess  everything up.  And then we merged changes from the FreeBSD upstream  periodically.  Typically, the typical working model is we merge changes  when we need something.  Or after a big deadline's passed and we realize we're  behind, we'll do a merge just to sort of catch up while we  have some breathing room.  So our first attempt is sort of the basic obvious thing  that you might think to do is we fetch upstream.  We merge master into our current tree in a branch.  But nonetheless, we just merge it.  It mostly works.  The first few times, it went pretty smoothly.  There were some sort of strange looking conflicts that I  didn't understand at the time.  But it worked.  We got past them.  Although one annoying thing that we still not haven't  resolved is that rebases go horribly wrong.  So if somebody else pushes, well, you're doing a merge,  you just have to throw the merge away.  Rebases never work in this case.  So however, after a few times, we came along.  We started doing work integrating the VT console  stuff into our tablet platform.  And I did the merge.  Everything compiled.  Everything seemed to work.  Pushed it, and something was broken in VT.  Who knows where?  It wasn't due to a merge conflict.  It was actually due to an API change, it turned out.  The problem is, so this is sort of a notional model of how  it works.  You're going along, you're going along, you're going along.  You merge upstream.  It's all good.  Problem is, this is more like it.  And this is actually much simpler than reality.  Reality is you're going along, you're going along,  you're going along.  Thousands of changes occur.  And then you merge from upstream, and you pull in another  three months of development, which is typically  several thousands of changes.  And if you try to bisect, well, in our case, it was an API  had changed.  So all these were fine, but they don't  include any of our code.  And all these are fine, because the problem was here.  Well, here, I guess, technically.  And so there's nothing to look for.  So that was really annoying.  Ed found it eventually.  So I ignored it for a week.  I didn't need IOS, so it was OK.  So I wrote a tool that I whimsically named Mergeify.  It merges one commit at a time.  Because from the perspective of a consumer of FreeBSD, every  change is a feature.  That's not perfectly accurate, because sometimes it's a  commit that's broken, and then another commit that fixes it,  or a commit that's broken, and then a few commits, and then  a commit that fixes the previous one.  There's no way to deal with that case in a sensible way, so  I just punted.  But merging one commit at a time does help in that the  overall system is much, it does help in that you now have  commits you can bisect.  Each of those merge commits is useful.  The feature I haven't added to the tool yet is something to  knock out all of the child commits from the bisect.  It would be pretty easy to do.  I just haven't written it yet.  So that you only consider the commits that actually change  your branch.  It's one of the key things that I figured out over time is  that from our perspective, it's only those  merges that matter.  Everything else, it's got some history who cares.  So the first attempt, we just merged every commit.  And then TCP came along, and there was an update in  contribute, and things went really strange.  We got these merge results that were completely nuts.  Mostly, it was the top level make file would get something  from contrib squished into it.  Turns out that what's happening in the FreeBSD export is  things in the vendor branch have a common parent with the  FreeBSD tree, the empty repository.  So in fact, there's a slash make file in both of them.  And Git says, they've got common blank lines, squishes  them together, and really goes badly.  The problem is actually, you don't care about that commit.  It's not important.  What you care about is the commit that merged that into  the FreeBSD tree.  So I changed the code to only pick the direct commits to  the branch and merge each one of those one at a time.  I was going to do a demo here, but I'm getting a little  short on time, so I think I will skip the demo and come  back to it at the end, particularly since there's a  bit of fussing with the projector to make it work.  So I alluded to be for rebases broken.  And I think it's, again, because rebase is applying all  the commits individually rather than those individual  merge commits.  So it's on my list to attempt the change to try applying  those commits one at a time, basically  re-implement rebase.  Should be doable, but I haven't done it yet.  I have an upcoming merge that looks like it's going to be  exciting, so maybe I'll fix it then.  So I've got some to-dos for mergeify.  I need to add this rebase mode, this bisect mode I've  talked about before, where you can skip all the commits that  don't make any sense to look at.  And one thing I would like to do eventually is periodically,  say, may every 10 commits or every 100 commits, check that  things build.  Right now, I don't get to do that, which is sometimes a  little frustrating.  I get to the end and I discover get botched a merge  somewhere, or I botched a merge.  My current workflow is anytime I do anything by hand, I  assume I screwed it up and do a full build just to make  sure that it's right, that I'd really like to do it every  commit.  That would be really slow.  Right now, it takes several seconds to merge each change  because get is fast, but even if it's all in RAM on a fast  machine, previous these repo is big.  So this would take quite a bit longer, but hopefully,  Simon Gerrity's meta mode will make it fast enough that I  could try every 10 commits or so, and I could bisect within  that once I knew what was going on.  So on to another topic, upstreaming.  The best way to remove merge conflicts, of course, is to  upstream your changes.  That way, they become everyone else's merge conflicts, and  not mine.  And if that section of the make file is the same, then  things are good.  Particularly, the top level make files in FreeBSD, I have  made so many changes to them.  I get conflicts all the time.  Just a bit.  Anyway, so there's some questions of what to upstream.  I think there's some philosophical questions here.  The answers will vary depending on your project, the  nature of your work.  Obviously, drivers for things that people can use should be  upstreamed.  And drivers are one of the better things to upstream as  the owner of the driver, because it means when somebody  changes the infrastructure, they have to update your driver.  And you don't have to come along three months later and  say, what the heck happened?  Why does my driver not work anymore?  Why does it not compile?  That sort of thing.  Also, general infrastructure.  We've built quite a lot of infrastructure along the way.  So we've been upstreaming that as we can, or as we have time.  So things that are shared by multiple external consumers.  I like to try to upstream things that are useful to  multiple people, even if they're not useful to most  consumers of the project, are not entirely useful in the  base system.  If you do a cleanup that is critical to you, but doesn't  matter to the base system, but other people are using it, I  think that's a good thing to upstream.  But that's a little tricky in that those are the sort of  things that get broken.  And I've had some issues with that.  And also, things that are just low impact and are likely to  generate conflicts.  So we actually have a new signal that we've been  meaning to upstream.  Because Capsicum might use it eventually.  It won't immediately.  But every time someone adds a new signal, it creates a ton  of conflicts.  Because in every single case, it's adding something in  exactly all the spots we changed.  So I think we'll upstream SIGProt soon.  So things we have upstreamed.  FDT support for MIPS, so a flattened device tree.  It was there.  It didn't really work.  We fixed that.  Bunch of drivers, bunch of driver improvements.  We added a way to turn on the floating point support that  was in the kernel.  And then we made it actually work for MIPS.  We added bootloaders for MIPS.  We've added the unprivileged build stuff.  It's a bunch of other stuff.  And I've had it for four years now, so lots of things.  We also have some sort of related upstreaming we've  been doing, not to FreeBSD, but to other projects.  So if you saw Stacy and Sean's talk earlier today, the  QMU user mode work that's letting us build ARM in MIPS  64 packages, which will soon be official, that's work we did  because we needed packages.  Turns out we don't actually use them very much right now.  But notionally, we knew we were going to want to do demos  without a tree code eventually.  So we needed some way to build them other than trying to  build 2T or something on 100 megahertz MIPS.  Didn't seem like a good idea.  Meantime, to reboot good power failure would be an issue.  So we've done a lot of improvements to  Clang and LLVM on MIPS 64.  The imagination technology people's focus is definitely  on little Indian 32-bit MIPS, and we're big Indian 64-bit  MIPS.  So we've had to fix a lot of things.  And I've been upstreaming stuff to TCP dump, mostly to make  my life easier merge-wise.  But they seem to like the direction, the  compartmentalization direction.  So that's nice.  Internally, we've been doing some releases.  We've done them internally for a long time, mostly  snapshots.  Initially, it would be periodically I would do a  release build using our little build system and push  something out and put it on the wiki.  We've done a couple of restricted releases to  partners.  One of the problems we have is with the FPGA, the license  agreements mean you can't share compiled bit files.  So we give them out to a few people who are in the program  and we know how it licenses.  But it's sort of the AT&T Unix problem that Kirk's  alluded to.  And we started doing public releases.  So we just did our first public release three weeks ago,  I think.  So cherrycpu.org, you can download our CPU.  You can compile it.  Don't run the bit file that comes out without some careful  testing, because at least the ones that our Jenkins build  is producing, the fan doesn't work.  Seems to run OK as long as you don't do too much.  Then it gets hot and things misbehave.  So another small change gears.  Since my focus on this is a free BSD developer, some tips  for developers, many of which go around the idea of as you  develop new ways to build the kernel or new ways to build  the OS, you spend an awful lot of time compiling.  So we've got some suggestions for ways to deal with this.  The first and probably most important one, use a big  enough machine.  Seriously, if somebody is paying you to wait for  compiles, they can buy quite a lot of hardware fairly  quickly for what they're paying you.  My view is you want enough RAM to hold all the source and  all the output.  We know 128 gigs is enough.  John Anderson has one that's working well for him.  The machine we use is a 256 gig machine.  We have fast ZFS, a Z mirror with a half terabyte of SSD.  Works pretty well.  I would say that 128 gigs should be enough.  256 gigs definitely should be enough for anyone, except  last week when there was a compiler bug.  And we had a dozen clang processes using 70 gigabytes  a piece.  We ran out of swap.  It was exciting.  And it turns out, well, they ran out of swap, and then  they started dumping core.  And you can't kill processes that are dumping core, so  that was unfortunate.  But usually it's big enough, and we literally could not do  the work we're doing without this machine.  It's only about $5,000.  It's definitely worth it.  Another thing that I've found really helpful is I use a  little push notification service that has a rest  interface.  So the one I use is pushover.net.  There's a web browser-based client that's $10.  And there's Android and iOS, which also are $10 a piece  to activate.  You couldn't write the service for that, so it's  totally worth it.  And I have a little command-blend wrapper.  So I run this command-notice, run a command, and then I get  something like this on my screen and a bing and my  phone buzzes.  And it's really great at reducing the latency between  when my compile finished and I noticed that it finished.  So that's been really helpful.  Another thing, I build everything in Tmux sessions.  And actually one of the things that I discovered part way  through is really important to switch away from the build.  Do not send it over the network.  Even over gigabit locally, enough stuff buffers that  it significantly delays your ability to get back to the  prompt.  So better to not render it at all.  That was slightly surprising.  I found that at Cambridge.  I knew it was an issue over the Atlantic.  I would switch away when I was tethering.  So I didn't just waste all that bandwidth.  But it turns out, yeah, always switch away.  And I guess one final little tip, continuous integration is  really great.  We do full OS builds after every compiler or OS change.  They take about 20 minutes.  We also do full releases at least a couple times a day.  Keeps everything working.  And one of the key things for us is because our architecture  is weird and we are making strange changes, we build  both Cherry, MIPS 64, and AMD 64 all the time.  AMD 64 is actually a really good canary because it also  builds I3D6 libraries.  So that means we build a whole lot of stuff and just  keeps us honest.  We are a research project, but we really want this stuff to  be used for real.  So we don't want to get to veer down some blind alley in  code changes and then come back and say, oh, I have three  months of work to get this thing functional again.  We're into the shape where I could merge it.  So we do that all the time.  And also, when we're working on a release now, we create a  separate set of Jenkins jobs to build the release daily in  the release plan, just to make sure everything is stable.  I'll mention some of our papers that we've got published.  We have three top two papers in the last year.  We had a Cherry hardware paper.  So here's the hardware paper.  The Cherry capability model, we're visiting risk in an  age of risk.  That's kind of mostly what I talked about earlier.  We have beyond the PDP11 processor support for a  memory safe C abstract machine.  That was an ass-bloss.  If you're a C-geek, it's a great paper.  I definitely recommend it.  I learned all sorts of weird things about C writing the  paper.  So that was fun.  And then we had a compartmentalization paper at  IEEE Security and Privacy a few weeks ago.  There's also an ISA document and whatnot on the Cambridge  website.  We have a future work here.  So we're working on a pure capability for VSD.  So probably it'd be a very long time, even once hardware  exists, before you'll ship a version of VSD that only  uses pure capability code.  But it's pretty likely that you'll want to run a fair bit  of pure capability code.  And we also realized recently that the best way to get a  ton of code running is to be able to have a pure capability  build within free VSD.  So we can just compile everything and try to use it,  run it through the test suite and see what happens.  It'll no doubt be exciting.  We're also working on, we'd like to add Cherry to the kernel.  The code we have in there is all assembly or macros  around inline assembly.  It works for what we're doing, but we'd like to be able to  do things like protect mBuffs, protect storage buffers,  compartmentalize the kernel, make it into a microkernel.  And we're hoping, well, so one of the other things, as I  said, early on, 256 bits is a pretty big pointer.  There'll be lots of overhead in terms of  cache footprint and whatnot.  In extraordinarily pointer-heavy benchmarks, it's  about 20% performance overhead, which is not acceptable.  So we were working on 128 bit compressed capabilities.  Some interesting trade-offs we're exploring.  We're pretty confident it's going to work, but there are a  lot of details.  And in our simulations, that should get the overhead down  to about 3% in a benchmark that's basically data  structures that are pointers.  And then we're also looking at non-nibs architectures.  So happy to answer any questions.  I just threw up on here our timelines.  We've been at this about five years now.  I am guessing we've got over 50 years of work into it.  And we've got quite a bit to go.  But it's a fun project.  And I think we've done a lot of interesting development and  stuff that's generally useful for free BSD along the way.  Any questions?  Yes?  So your communication stuff, are you synthesizing  varialog stuff and then running your compiler software on  that full stack integration?  Yes.  So we're actually using BlueSpec System Varialog, which  is a Haskell-derived HDL.  We both compile it.  So it has a mode to compile to a cycle accurate C simulator.  So we've been doing it for a long time.  More recently, we've started synthesizing bit files,  loading them on FPGAs, and then running the  operating system on them.  And that's hide in Jenkins?  Yeah.  That's Jenkins.  The Jenkins cluster has grown quite a lot over the last  year.  On the hardware side, one of the RA's is amazingly patient  and willing to deal with broken junk.  Is that part of your structure open for?  No.  I don't think so.  Parts of it might be in the release.  I'd have to go look.  If you want to ask me later, I can take a poke at the GitHub  repo of what we've actually put out and what's buried in  the Jenkins config.  You said you're writing out another  curriculum you just call in your face.  Is that the Cloud ABI work?  No, this is not the Cloud ABI work.  I probably should have gone to the Cloud ABI talk before,  but I talked to it a bit.  So what we're doing is adding a new syscall ABI.  So like the lib32 and the free BST32 emulation, we're  adding a cherry ABI.  It's sort of the first cut.  Not clear what the long-term right answer is, but that's  one we know how to do.  So I did a first pass at syscalls.master.  The great thing is you take all the lines that say compact,  clear them out.  We've never shipped with those versions, and we never will.  Well, thank you all for coming.  If you have any questions later, we'll  bring them back to you.
CheriBSD__A_research_fork_of_FreeBSD_0_1859.png CheriBSD__A_research_fork_of_FreeBSD_1860_2111.png CheriBSD__A_research_fork_of_FreeBSD_2112_2243.png CheriBSD__A_research_fork_of_FreeBSD_2244_2879.png CheriBSD__A_research_fork_of_FreeBSD_2880_3503.png CheriBSD__A_research_fork_of_FreeBSD_3504_5207.png CheriBSD__A_research_fork_of_FreeBSD_5208_8843.png CheriBSD__A_research_fork_of_FreeBSD_8844_9647.png CheriBSD__A_research_fork_of_FreeBSD_9648_12191.png CheriBSD__A_research_fork_of_FreeBSD_12192_14651.png CheriBSD__A_research_fork_of_FreeBSD_14652_15467.png CheriBSD__A_research_fork_of_FreeBSD_15468_19439.png CheriBSD__A_research_fork_of_FreeBSD_19440_22187.png CheriBSD__A_research_fork_of_FreeBSD_22188_24527.png CheriBSD__A_research_fork_of_FreeBSD_24528_25931.png CheriBSD__A_research_fork_of_FreeBSD_25932_27959.png CheriBSD__A_research_fork_of_FreeBSD_27960_32459.png CheriBSD__A_research_fork_of_FreeBSD_32460_34091.png CheriBSD__A_research_fork_of_FreeBSD_34092_36083.png CheriBSD__A_research_fork_of_FreeBSD_36084_37559.png CheriBSD__A_research_fork_of_FreeBSD_37560_42779.png CheriBSD__A_research_fork_of_FreeBSD_42780_44147.png CheriBSD__A_research_fork_of_FreeBSD_44148_44447.png CheriBSD__A_research_fork_of_FreeBSD_44448_46751.png CheriBSD__A_research_fork_of_FreeBSD_46752_48383.png CheriBSD__A_research_fork_of_FreeBSD_48384_50075.png CheriBSD__A_research_fork_of_FreeBSD_50076_50363.png CheriBSD__A_research_fork_of_FreeBSD_50364_50507.png CheriBSD__A_research_fork_of_FreeBSD_50508_50579.png CheriBSD__A_research_fork_of_FreeBSD_50580_50867.png CheriBSD__A_research_fork_of_FreeBSD_50868_51011.png CheriBSD__A_research_fork_of_FreeBSD_51012_51131.png CheriBSD__A_research_fork_of_FreeBSD_51132_52067.png CheriBSD__A_research_fork_of_FreeBSD_52068_53411.png CheriBSD__A_research_fork_of_FreeBSD_53412_55247.png CheriBSD__A_research_fork_of_FreeBSD_55248_56747.png CheriBSD__A_research_fork_of_FreeBSD_56748_56795.png CheriBSD__A_research_fork_of_FreeBSD_56796_56843.png CheriBSD__A_research_fork_of_FreeBSD_56844_56891.png CheriBSD__A_research_fork_of_FreeBSD_56892_56939.png CheriBSD__A_research_fork_of_FreeBSD_56940_56987.png CheriBSD__A_research_fork_of_FreeBSD_56988_58031.png CheriBSD__A_research_fork_of_FreeBSD_58032_58091.png CheriBSD__A_research_fork_of_FreeBSD_58092_58139.png CheriBSD__A_research_fork_of_FreeBSD_58140_58211.png CheriBSD__A_research_fork_of_FreeBSD_58212_58331.png CheriBSD__A_research_fork_of_FreeBSD_58332_58391.png CheriBSD__A_research_fork_of_FreeBSD_58392_58475.png CheriBSD__A_research_fork_of_FreeBSD_58476_58523.png CheriBSD__A_research_fork_of_FreeBSD_58524_58571.png CheriBSD__A_research_fork_of_FreeBSD_58572_58619.png CheriBSD__A_research_fork_of_FreeBSD_58620_58667.png CheriBSD__A_research_fork_of_FreeBSD_58668_58715.png CheriBSD__A_research_fork_of_FreeBSD_58716_58787.png CheriBSD__A_research_fork_of_FreeBSD_58788_58871.png CheriBSD__A_research_fork_of_FreeBSD_58872_58943.png CheriBSD__A_research_fork_of_FreeBSD_58944_59087.png CheriBSD__A_research_fork_of_FreeBSD_59088_59135.png CheriBSD__A_research_fork_of_FreeBSD_59136_59195.png CheriBSD__A_research_fork_of_FreeBSD_59196_59243.png CheriBSD__A_research_fork_of_FreeBSD_59244_59363.png CheriBSD__A_research_fork_of_FreeBSD_59364_59411.png CheriBSD__A_research_fork_of_FreeBSD_59412_59459.png CheriBSD__A_research_fork_of_FreeBSD_59460_59519.png CheriBSD__A_research_fork_of_FreeBSD_59520_59675.png CheriBSD__A_research_fork_of_FreeBSD_59676_63011.png CheriBSD__A_research_fork_of_FreeBSD_63012_63887.png CheriBSD__A_research_fork_of_FreeBSD_63888_65699.png CheriBSD__A_research_fork_of_FreeBSD_65700_67523.png CheriBSD__A_research_fork_of_FreeBSD_67524_68087.png CheriBSD__A_research_fork_of_FreeBSD_68088_70103.png CheriBSD__A_research_fork_of_FreeBSD_70104_70895.png CheriBSD__A_research_fork_of_FreeBSD_70896_71327.png CheriBSD__A_research_fork_of_FreeBSD_71328_72287.png CheriBSD__A_research_fork_of_FreeBSD_72288_74159.png CheriBSD__A_research_fork_of_FreeBSD_74160_75467.png CheriBSD__A_research_fork_of_FreeBSD_75468_78215.png CheriBSD__A_research_fork_of_FreeBSD_78216_82170.png
I'm Brooks Davis. I'm with SRI International, and I'm part of the team of researchers working on the Cherry project, which I'll get into in a moment. I'm here to talk to you today about CherryBSD, which is our fork of FreeBSD to support the Cherry processor. In many ways, similar to, for instance, the TrustedBSD project or the HardenedBSD project, and that we're going off in another direction, we're going off in a direction that FreeBSD isn't ready for, for one reason or another. In the case of TrustedBSD, it was that there was large swaths of fairly disruptive technology that had to be proved out at scale before they could be merged into the tree, before you're going to make changes to thousands of places in the code. CherryBSD is a little different in that CherryBSD is about porting FreeBSD to a new CPU with new instructions, a new C compiler, some significant changes to the C language. So a wide variety of things that obviously we can't just dump in the FreeBSD tree, nothing else you can't buy the CPU. So people might object to large-scale changes for something you can't even use. But first, a little background. As hardly a week goes by where you don't hear about some new malware, some new breach or malware problem or whatever, Anthem losing 80 million customer records, the banks losing hundreds of millions of dollars in things like the target breach. Or I think this one actually, if I remember right when I created this slide, this is a reference to the banks simply failing to notice hundreds of millions of dollars in fraudulent transactions that were made to move money to another bank, and then they took the money out and poof. Or one of the most recent ones, the Office of Personnel Management in the US, basically HR for the civilian part of the US government, they lost 3.2 million HR records. So basically everything you need to know to steal someone's identity. So this is the daily reality of computing and the internet. So we decided it was time to do something about it a bit. The approach we're taking is one that's been taken for quite a while, which is application compartmentalization. Compartmentalization decomposes software and isolated components. Each sandbox runs with only the rights needed. So you can allow a least privilege approach. One common example of this that you probably as Unix users use every day is SSH has privilege separation. So that the most risky bits that must run as root, the bits that must run as root are separate from the most risky bits that do all the crypto handling, or as much as possible. And the goal here is that you can take an application, you can start with an application like in this example, GZIP, and you can cut it up into multiple pieces. So the compression logic, which is what people screw up because they write it in tight C code, designed to be fast, designed to trick the compiler into generating the best code they can, at least 20 years ago whenever they made those decisions. You can put that off in a process which has limited rights. In this example, this example maps pretty well to Capsicum, which we already have in FreeBSD. It is a process-based framework where we have capabilities, which are unforgible tokens of authority, are file handles, very fundamental thing in Unix. And you can enter a capability mode where you cannot open any new file handles through arbitrary namespaces. You must obtain them via other capabilities. And you can restrict the rights on those capabilities. So for instance, in the GZIP example, what GZIP does in its most basic mode is it opens one file, and it opens another file handle, creates a new file, and it does a bunch of work and then copy moves data from one side to the other. You obviously don't want the input file to be writable. You don't want the output file to be readable, or that's probably fairly harmless. But in principle, you don't really want that. Capsicum works great for something like GZIP. It works great for OpenSSH, where the existing framework was already, the existing privilege separation was already based around Unix principles. But it doesn't scale to the kind of things you might want to do. For example, in a web browser, right now, a web browser like Chromium has a separate process for each tab, at least until you open too many of them. The problem is that if you have too many processes, you start to run out of TLB entries on your CPU. I know of one instance where, in addition to running out of TLB entries, some systems are running out of address-based identifiers. So even though their processes are often quite small, if you have too many of them, you still get TLB misses simply because you've had to reuse an address-based identifier. So there's some serious scaling problems. And that's even just with tabs. What you'd actually like is to render every image in your browser in a separate sandbox so that when something does go wrong, the email that somebody sent you that had a bad image embedded in it can't read your bank statements, can't hijack your password reset URLs, all of which are in your email and all of which are in that tab and in that process. So we're trying to make this, we want to make this scale. So we're doing that in hardware. So as I was saying, with process separation, you can avoid this problem where you have one process here and as a pointer to a buffer in some other part of the program or in some other part of the application. If it's in another process, this arrow doesn't work. And we'd like to do that in the single application and have that mechanism for every pointer. So we can have every C object be a separate thing that you cannot manufacture a pointer to. So with Cherry, we are doing that with capabilities. So we've created fat pointers. These are 256 bit pointers. Yes, that's big. So each pointer has an offset which is in practice the pointer. It is where you're pointing in memory. And you have a base and a length, which are relative to your virtual address space. And there's guarded manipulation of this, which means that you can only create a capability by deriving it from another capability. And the only derivations that are allowed are to shrink it. So you can increment the base and move up in address space or you can shrink the length. You can also reduce the permissions of the pointer. One important thing here, though, is that, and the reason we have an offset, in our initial design, we did not have an offset. It turned out that in real world C code, that's not OK. Because you can't just keep shrinking your data. Because what you do is if you see an application program like FFMPEG, inside LiveAV codec, there is code where they pass a pointer to the middle of a piece of data to be decoded. Because the compiler generates better code that way, because it can use small, positive, and negative intermediates. So we have this offset, which allows our capabilities to be near perfect replacements of C pointers. And the reason this works is that you only check that the offset is inside the base and length at dereference time. So as I alluded, we want these to be C pointers. So we've done that. And we have two modes where you can use them. One is a hybrid mode. This is where you have conventional, in our case, MIPS64 code. And you annotate some of the pointers in your code as being capabilities. That means the compiler saves more space for them, uses the correct manipulation instructions to access them. And it works. So you can use this in code if you want to just protect a few buffers that are very, very important. But it is a lot of work. We made some changes to TCP. And I'll talk a little more about them later, where I added bounds to the packet buffer that was being dissected. And that worked, but that was many days of unpleasant changes to the code and thousands of changes. So we ended up adding a pure capability mode, where every pointer is a capability. So in translation units or files that are compiled in this mode, all pointers are 256 bits. They're all bounds checked. Almost all bounds are correctly inferred. So if you malloc something, you get a pointer that's exactly the size of the object that you allocated. If you try to go outside the bounds, things go boom in a nice predictable way, rather than randomly corrupting your memory. So as I say, we have a range of ABIs here. So we have the Pyramid 64. This is conventional FreeBSD runs on this processor without modification. And if you use it in this mode, you get no benefits, but it also doesn't require any work. The important part is to work your way through. And so there's a bunch of different approaches we can take. But we have a lot of hybrid code today and some pure capability code. We're increasingly moving towards pure capability code. And here's an example of the sort of things that we can support with the processor. So this is our current situation for the most part. We have a conventional kernel. It's running standard MIPS 64 code. We have a conventional application, also running standard MIPS 64 code. And we have Zlib in a compartment. Zlib's chosen because it has a small interface and it was easy and it compiled. And it has pure capability code inside and then a little hybrid wrapper around the outside that lets us call in. The application, in one example that we've talked about in one of our papers, is a GIFT to PNG. Doesn't know that the library is protecting it. And in fact, there's essentially no performance impact in this case. But if there's a bug in Zlib, it will fail stop. And even if it doesn't manage to fail stop, it's very difficult to gain control in the application. If it's a pure, within the bounds of C, control flow bug. You can also see a world where we're working right now actively to add pure capability application, which means yet another syscall interface. And pure libraries, one interesting thing though, and this example actually isn't ideal. But one of the interesting things is we can put conventional MIPS 64 code, say, to a binary library that we don't have any source code to for either because we bought it from someone who doesn't want to give a source code or because we lost the source code, which I've heard of some large internet companies doing. And we can put that in a little sandbox. And we can have a wrapper interface. So our library can be pure even though we have this bit that we don't have any control over, it can't get out of its sandbox. So it's less likely to fail stop if something goes horribly wrong. But what it can do is very little. It has access to a few, it might have, if it's a proprietary video codec, all it can do is write the wrong video frames and read some video frames, which it was supposed to be doing anyway. Because also, you can go farther along and of course start writing a microkernel, having a single address space application, lots of neat things. But we're quite a ways from that. If nothing else, LLVM doesn't yet compile MIPS 64 in a way that's usable for a kernel. So just a little bit more on the CPU. We have a prototype CPU. It's a 64-bit MIPS CPU, sort of R4K, so nicely out of patent. It has the Cherry ISA extensions that I've been talking about. It runs at 100 megahertz, which is a bit slow. But we lived with it in the 90s. And we actually have quite a bit more RAM, which helps. Although we have a gig of RAM on the boards and could have four gigs of RAM, which would have been a little unimaginable when I was in college, which is mostly good, except occasionally it gets a little exciting. One of the early bugs we had was a bug in the SD card controller that we got from the FPGA manufacturer that if you wrote a byte to the buffer two cycles in a row, it threw the second one away. There literally was code in there that said, we got two writes in a row, throw it away. We commented that out, and it works better. But one interesting thing is we only found this problem after reboot, because we had so much RAM that the entire file system fit in the buffer cache. So it was only when we flushed the buffer cache by rebooting the machine that we discovered that we'd corrupted the on-disk data structures. And Fisk would say, oh, what have you done to this poor file system? So it's an interesting one of the interesting challenges here. But the neat thing that we have is that we have this CPU. We have this operating system that I'm going to talk more about, and we have a modified LLVM. So we can run real software. And that's where things get exciting. We can really test things out. So CherryBSD is, of course, the BSD to support Cherry. It's a mix of platform support, which is to say drivers for things on the FPGA and peripherals, which is the very CPU. You can compile our CPU without the Cherry bits if you want. It's open source. And if you want to do things like run an hardware class, you can use that fairly simple CPU and add things like a better branch predictor as an exercise. And people at Cambridge have done that. So there's also support for the new ISA features. I'll give you a little rundown of how much changes that was in a bit. There's infrastructure to support the compartmentalization of libraries or whatever it is you want to stick in a compartment. There are some custom applications. I'll talk more about TCP dump and a few others. And then there are a bunch of build system improvements, because we're doing lots of slightly weird things. So we've done a lot of work like we have the ability to build free BSD and install it in a directory without any privilege, which is now being used in parts of the release build infrastructure. We did it because we wanted to have grad students building Cherry BSD. And I didn't want to give them a root and mess up our nice fancy machines. Here's a snapshot of our page on GitHub. So you notice almost 6,000 changes here relative to free BSD. And we're actually farther. And there's quite a few more than 6,000 now. That was a week ago. And we're a bit behind. We merge periodically. Talk quite a bit more about merging. But the main thing is, and what I think makes our project interesting, is that we both want this to be publicly accessible, because we have collaborators, which means we can't do things like rebase. But we've merged probably 200 changes or 300 changes out of this tree. So we have this huge set of changes that we have to maintain over time, which leads to some challenges in version control. So next up, here's a breakdown of the kernel changes. This was in our paper at IEEE Security and Privacy. So we added a bunch of headers, lots of various things, lots of access to assembly functions. There's some setup of Cherry and the kernel basically saying, turn it on. Give me the default capability, which at boot, there's a default capability, which is the address space. And you can start chopping that up later. There's contact switch code, because our new capabilities have to go into special registers. So we need to save and restore those registers. There's exception handling, because we've created all sorts of new ways to cause exceptions, since now an out of bounds pointer to your reference is a trap. There's some memory management, memory copying, swap. There's some support for the actual departmentalization bit, which is somewhat separate from the memory safety. Bit for system calls, bit for signal delivery, et cetera. Few thousand lines of code, which actually is not too bad, because 3DSD is several million lines of kernel code. And we've actually written a very tiny little microcernel before that's not very big. I don't even think it works anymore. So in addition to kernel changes, of course, had to make some tweaks to the runtime. The first, the biggest thing is mem copy and mem move need to be capability aware, even in capability oblivious code. One of the interesting things here, and one of the reasons why it's really important to have a team as large as we do, is actually our first version of the ISA didn't allow you to implement mem copy efficiently. You actually would have had to check every 256 bit chunk of memory to see whether or not it was a capability, and then copy it with capability instructions or copy it with regular instructions. That would be insane. So you can now use capability instructions to copy non-capability memory. And it actually turns out that 3DSD's generic mem copy and mem move implementations in C required almost no changes. We just had to tell it, oh, just use the right size as the basic word size when doing copies. And it generates the right code. I was actually pleasantly surprised. We, of course, have assembly versions as well that are a little better, but that's the basic. The nice thing is with the C integration, we get a lot of that. We also had to add explicit versions for hybrid code. We had to have explicit versions of things like string and memory manipulation functions so that we could have a mix of arguments. That, I think, I'm not sure that's going to stay. But in our current infrastructure, that's how it has to be. And then there's a bunch of interesting cases. In current C code, like in the sterln function, it assumes that once it's aligned to the front to the size of a word, then it can always read whole words. But we have byte granularity restrictions. So if you can definitely read the next byte, that doesn't mean you can definitely read the rest of the word. So that had to be fixed. It was easy to do, but required a bit of change. And one thing that we're working on right now is taking the syscall implementation in libc and moving it into a lib syscalls. So yes, I'm going to be asking for X runs on the ports at some point here, because that's going to be exciting. But it'll both help us, because inside a sandbox, we may want to attempt to make syscalls and have the kernel mediate based on some new configuration that's capsicum-like. Or we may want to have something that just simply proxies the syscalls back to the privileged part of the application. It can make the decisions. It's also be useful for people like Google, who are building Android with FreeBSD. And their syscall layers, obviously, totally differ. So the separation, I think, will be generally useful. Next bit, bits for compartmentalization. So libcherry is the library for creating sandbox libraries. Or for creating sandboxes, you can instantiate objects. You can give them types. And then you can allocate copies of the objects, do resets, that sort of thing. So that's the core functionality for implementing compartmentalized libraries or compartmentalized TCP dump, for instance. It also has a loader and a runtime linker, since objects have a new calling convention. And we don't want to have to write horrible little RPC-like things, where you say, well, I'm going to pass eight integer arguments and eight capability arguments, and try to remember which ones go where and whatnot. We did that for a while. It sucked. We're also working on a syscall, it also includes some syscall implementation bits. So compartmentalized can call out to an object in the privileged process. Something with a confusingly similar name, we have the libcherry directory. This is where peer capability versions of libraries live. It's also where the objects live. It's a lot like the 32-bit support for 64-bit machines. In fact, I copied and pasted this stuff in makefile.inc1. One reason why merges suck is lots of turn in that file. So eventually, right now, we're only using those libraries to build little compartmentalized libraries that have a hybrid outside interface. Longer term, the plan is to have pure capability libraries. So as part of your transition from a conventional ISA to cherry, you'll be able to pick points along the way, not only at the library level, but at the application level. So you can have your most important or, well, either most important or easiest to modify applications be pure capability. And other applications can remain as they are. You can still have your binary only whatever program it is that you have. So we also have demo applications. This is a screenshot here or a picture here of the very weak PowerPoint-like program that I wrote as a demo. So we actually gave our talk to DARPA at the principal investigators meeting on our tablet, which you can see Robert holding here. And when we got to the end, we said one more thing. We're running on our, not only are we running on our hard drive, but the slide deck hasn't exploited it and triggered a Trojan. And Cherry successfully defeated the Trojan. I don't know if it was very hard since I wrote all of it. But nonetheless, the technology did work and does do this. So we've got a bunch of little custom applications as part of CherryBSD. Those don't present any particular challenges for us in terms of CherryBSD maintenance. The thing that does present bigger challenges is we're also modifying existing applications. So at one point, we took a look at what's compartmentalized Wireshark. Wireshark's full of vulnerabilities. It's a giant program. We spent quite a bit of time on it and said, well, that was insane. It's 3 million lines of code and uses Glib. And it's all very complicated. We decided to do TCP dump instead, which has the advantage of being in the base, which is kind of good and bad. So our first version, we had a version that was very simple. We just compartmentalized a section, and it was standard MIPS code on both sides. It was just a little bit of hybrid code to get in and out of the sandbox. We later added memory safety, which I alluded to before. And that's 6,000 lines of changes, a whole lot of not fun. And part of that, though, was actually one of the things that we learned an interesting lesson, which is that we had no ability in TCP dump. There's a modest amount of code that advances the buffer, advances the buffer, advances the buffer, and that's, oh, whoops, I forgot. I want something a little back behind my pointer. That's fine in C. That's a perfectly legitimate and reasonable thing to do. However, before we had the offsets, that didn't work, because we'd incremented the base, we incremented the base, we're going to base, oops, can't go back. That's not allowed. So we added offsets. That helped quite a bit. I added per-particle disector. So this is the most protection you could get in TCP dump, which is to say every protocol lives in its own sandbox. So as your dissecting IP goes fine, TCP goes fine, you call into HTTP, or maybe probably more realistically, you call into something like SNMP, the ASN1 parser is broken as it always is, and gets exploited. You can still trust the TCP and IP dissection, because you've failed in a deeper sandbox. So that required a modest amount of code change. You had to change all the call sites. That's pretty reasonable. We went to pure capability mode. That got rid of tons of annotations, which was nice. And then the pure capability mode, though, removing annotations, was actually driven by the fact that FreeBSD got a new version of TCP dump. So this is one of the things that's really interesting about working on a real world project and keeping your tree up to date, is that we learned some lessons about maintainability the hard way. We tried to do this import here. There were 3,600 conflicts in Git. Seemed like a bit much. So we made a bunch of changes, got back to here. When we added the linker support, that reduced the amount of junk in the code base. And actually, this is dropped dramatically, because somewhat surprisingly, the TCP dump maintainers accepted my change to shuffle 1,000 lines of code around within the tree and put it in different places. So the interface between the deceptors and the front end is both fairly narrow in that it's only five or six functions and fairly simple, which should help build a better capsicumized version of TCP dump, but also simplifies my code. We'll see how my next merge goes, because the approach I took when I upstreamed it is a bit different. There's also a bit of infrastructure work we've done. I've alluded to force of the unprivileged builds. We've added some hacks to let us do change the compiler on a per program and per file basis. That's because early on, our custom LLVM was not robust enough to actually compile all the code. So we wanted to focus on the code where we could do something interesting. And then over time, expanded out, it's getting close to be unable to compile everything. I spent quite a bit of time the last few weeks trying to compile things and then sending David Bug reports when the compiler crashes. And some other hacks, which will help some upcoming changes, like we strip binaries during the build rather than using the install program, things like that. For more information on the general what's in CherryBSD, particularly Barry's stuff, I suggest reading the journal article that I wrote a few months ago. Pretty good. Now on a bit to revision control. This is one of the places where we've had a lot of challenges. We started off in Perforce, which was the conventional way that people did forks of FreeBSD in the past. Supported on FreeBSD project infrastructure, which is good. Merging is very good. It really is a good way to maintain something that's a fork of FreeBSD in the long term. I've done it at previous jobs as well. And it's easy to maintain what I'm calling here stacked branches. So for a while we had a BarryBSD, which was the platform bits. And it sat in between CherryBSD and FreeBSD so that we could try and keep, maintain some separation there. We merged everything in the BarryBSD branch. That wasn't too weird and then got rid of it at some point. But that was nonetheless quite helpful. And our team already knew it. So that was a good reason to stick with Perforce. Downside is Perforce sucks at public access. You have to give people an account to give them access to the system. Every checkout involves adding server state. So even if we were willing to give out a lot of accounts, eventually the project would run out of resources. It's very easy to get the situation where your Perforce server needs to have a half terabyte of RAM. We probably would have had to buy it for the project in that case. Adding users is a bit annoying. And the offline support isn't very good. That's not too big a deal, but it's not good. So in October 2013, we decided we needed to have public access. There were people at MIT Lincoln Labs and some other places who wanted to start using CherryBSD. So we wanted to give them direct access to the repository rather than having to take the time to package up dumps and package up snapshots and push them out and QA them and all that. So we moved over to GitHub. We lost a little history granularity in the process because many of the commits couldn't just be applied one at a time because things had moved on and they'd been merging. But really not too bad. However, it was a bit of a trial by fire for using Git sort of at scale. FreeBSD is a bit on the big side for Git. And our export has some weird features that it'll get to. Also, I and Robert, who are the main CherryBSD developers, were not experienced Git users at the time. So lots of excitement. It's not clear that our model was the right model. But it's the one we've got. So we're kind of stuck with it. What we ended up doing is we forked the FreeBSD repo on GitHub. One thing that I found is kind of weird about GitHub's forking model is if you want to fork CherryBSD, it seems that you get a copy of the FreeBSD repo. That's what happened when I tried to do it recently. That might simply be because I already have a copy of the FreeBSD repo. But so if someone wants to try it, it's only a gig or so. GitHub will never notice. But that was a bit odd. And we did all the commits to the master branch. I'm not sure that's the right solution, but it's what we've done. And at this point, we're stuck. We're not going to do a forced rebase and mess everything up. And then we merged changes from the FreeBSD upstream periodically. Typically, the typical working model is we merge changes when we need something. Or after a big deadline's passed and we realize we're behind, we'll do a merge just to sort of catch up while we have some breathing room. So our first attempt is sort of the basic obvious thing that you might think to do is we fetch upstream. We merge master into our current tree in a branch. But nonetheless, we just merge it. It mostly works. The first few times, it went pretty smoothly. There were some sort of strange looking conflicts that I didn't understand at the time. But it worked. We got past them. Although one annoying thing that we still not haven't resolved is that rebases go horribly wrong. So if somebody else pushes, well, you're doing a merge, you just have to throw the merge away. Rebases never work in this case. So however, after a few times, we came along. We started doing work integrating the VT console stuff into our tablet platform. And I did the merge. Everything compiled. Everything seemed to work. Pushed it, and something was broken in VT. Who knows where? It wasn't due to a merge conflict. It was actually due to an API change, it turned out. The problem is, so this is sort of a notional model of how it works. You're going along, you're going along, you're going along. You merge upstream. It's all good. Problem is, this is more like it. And this is actually much simpler than reality. Reality is you're going along, you're going along, you're going along. Thousands of changes occur. And then you merge from upstream, and you pull in another three months of development, which is typically several thousands of changes. And if you try to bisect, well, in our case, it was an API had changed. So all these were fine, but they don't include any of our code. And all these are fine, because the problem was here. Well, here, I guess, technically. And so there's nothing to look for. So that was really annoying. Ed found it eventually. So I ignored it for a week. I didn't need IOS, so it was OK. So I wrote a tool that I whimsically named Mergeify. It merges one commit at a time. Because from the perspective of a consumer of FreeBSD, every change is a feature. That's not perfectly accurate, because sometimes it's a commit that's broken, and then another commit that fixes it, or a commit that's broken, and then a few commits, and then a commit that fixes the previous one. There's no way to deal with that case in a sensible way, so I just punted. But merging one commit at a time does help in that the overall system is much, it does help in that you now have commits you can bisect. Each of those merge commits is useful. The feature I haven't added to the tool yet is something to knock out all of the child commits from the bisect. It would be pretty easy to do. I just haven't written it yet. So that you only consider the commits that actually change your branch. It's one of the key things that I figured out over time is that from our perspective, it's only those merges that matter. Everything else, it's got some history who cares. So the first attempt, we just merged every commit. And then TCP came along, and there was an update in contribute, and things went really strange. We got these merge results that were completely nuts. Mostly, it was the top level make file would get something from contrib squished into it. Turns out that what's happening in the FreeBSD export is things in the vendor branch have a common parent with the FreeBSD tree, the empty repository. So in fact, there's a slash make file in both of them. And Git says, they've got common blank lines, squishes them together, and really goes badly. The problem is actually, you don't care about that commit. It's not important. What you care about is the commit that merged that into the FreeBSD tree. So I changed the code to only pick the direct commits to the branch and merge each one of those one at a time. I was going to do a demo here, but I'm getting a little short on time, so I think I will skip the demo and come back to it at the end, particularly since there's a bit of fussing with the projector to make it work. So I alluded to be for rebases broken. And I think it's, again, because rebase is applying all the commits individually rather than those individual merge commits. So it's on my list to attempt the change to try applying those commits one at a time, basically re-implement rebase. Should be doable, but I haven't done it yet. I have an upcoming merge that looks like it's going to be exciting, so maybe I'll fix it then. So I've got some to-dos for mergeify. I need to add this rebase mode, this bisect mode I've talked about before, where you can skip all the commits that don't make any sense to look at. And one thing I would like to do eventually is periodically, say, may every 10 commits or every 100 commits, check that things build. Right now, I don't get to do that, which is sometimes a little frustrating. I get to the end and I discover get botched a merge somewhere, or I botched a merge. My current workflow is anytime I do anything by hand, I assume I screwed it up and do a full build just to make sure that it's right, that I'd really like to do it every commit. That would be really slow. Right now, it takes several seconds to merge each change because get is fast, but even if it's all in RAM on a fast machine, previous these repo is big. So this would take quite a bit longer, but hopefully, Simon Gerrity's meta mode will make it fast enough that I could try every 10 commits or so, and I could bisect within that once I knew what was going on. So on to another topic, upstreaming. The best way to remove merge conflicts, of course, is to upstream your changes. That way, they become everyone else's merge conflicts, and not mine. And if that section of the make file is the same, then things are good. Particularly, the top level make files in FreeBSD, I have made so many changes to them. I get conflicts all the time. Just a bit. Anyway, so there's some questions of what to upstream. I think there's some philosophical questions here. The answers will vary depending on your project, the nature of your work. Obviously, drivers for things that people can use should be upstreamed. And drivers are one of the better things to upstream as the owner of the driver, because it means when somebody changes the infrastructure, they have to update your driver. And you don't have to come along three months later and say, what the heck happened? Why does my driver not work anymore? Why does it not compile? That sort of thing. Also, general infrastructure. We've built quite a lot of infrastructure along the way. So we've been upstreaming that as we can, or as we have time. So things that are shared by multiple external consumers. I like to try to upstream things that are useful to multiple people, even if they're not useful to most consumers of the project, are not entirely useful in the base system. If you do a cleanup that is critical to you, but doesn't matter to the base system, but other people are using it, I think that's a good thing to upstream. But that's a little tricky in that those are the sort of things that get broken. And I've had some issues with that. And also, things that are just low impact and are likely to generate conflicts. So we actually have a new signal that we've been meaning to upstream. Because Capsicum might use it eventually. It won't immediately. But every time someone adds a new signal, it creates a ton of conflicts. Because in every single case, it's adding something in exactly all the spots we changed. So I think we'll upstream SIGProt soon. So things we have upstreamed. FDT support for MIPS, so a flattened device tree. It was there. It didn't really work. We fixed that. Bunch of drivers, bunch of driver improvements. We added a way to turn on the floating point support that was in the kernel. And then we made it actually work for MIPS. We added bootloaders for MIPS. We've added the unprivileged build stuff. It's a bunch of other stuff. And I've had it for four years now, so lots of things. We also have some sort of related upstreaming we've been doing, not to FreeBSD, but to other projects. So if you saw Stacy and Sean's talk earlier today, the QMU user mode work that's letting us build ARM in MIPS 64 packages, which will soon be official, that's work we did because we needed packages. Turns out we don't actually use them very much right now. But notionally, we knew we were going to want to do demos without a tree code eventually. So we needed some way to build them other than trying to build 2T or something on 100 megahertz MIPS. Didn't seem like a good idea. Meantime, to reboot good power failure would be an issue. So we've done a lot of improvements to Clang and LLVM on MIPS 64. The imagination technology people's focus is definitely on little Indian 32-bit MIPS, and we're big Indian 64-bit MIPS. So we've had to fix a lot of things. And I've been upstreaming stuff to TCP dump, mostly to make my life easier merge-wise. But they seem to like the direction, the compartmentalization direction. So that's nice. Internally, we've been doing some releases. We've done them internally for a long time, mostly snapshots. Initially, it would be periodically I would do a release build using our little build system and push something out and put it on the wiki. We've done a couple of restricted releases to partners. One of the problems we have is with the FPGA, the license agreements mean you can't share compiled bit files. So we give them out to a few people who are in the program and we know how it licenses. But it's sort of the AT&T Unix problem that Kirk's alluded to. And we started doing public releases. So we just did our first public release three weeks ago, I think. So cherrycpu.org, you can download our CPU. You can compile it. Don't run the bit file that comes out without some careful testing, because at least the ones that our Jenkins build is producing, the fan doesn't work. Seems to run OK as long as you don't do too much. Then it gets hot and things misbehave. So another small change gears. Since my focus on this is a free BSD developer, some tips for developers, many of which go around the idea of as you develop new ways to build the kernel or new ways to build the OS, you spend an awful lot of time compiling. So we've got some suggestions for ways to deal with this. The first and probably most important one, use a big enough machine. Seriously, if somebody is paying you to wait for compiles, they can buy quite a lot of hardware fairly quickly for what they're paying you. My view is you want enough RAM to hold all the source and all the output. We know 128 gigs is enough. John Anderson has one that's working well for him. The machine we use is a 256 gig machine. We have fast ZFS, a Z mirror with a half terabyte of SSD. Works pretty well. I would say that 128 gigs should be enough. 256 gigs definitely should be enough for anyone, except last week when there was a compiler bug. And we had a dozen clang processes using 70 gigabytes a piece. We ran out of swap. It was exciting. And it turns out, well, they ran out of swap, and then they started dumping core. And you can't kill processes that are dumping core, so that was unfortunate. But usually it's big enough, and we literally could not do the work we're doing without this machine. It's only about $5,000. It's definitely worth it. Another thing that I've found really helpful is I use a little push notification service that has a rest interface. So the one I use is pushover.net. There's a web browser-based client that's $10. And there's Android and iOS, which also are $10 a piece to activate. You couldn't write the service for that, so it's totally worth it. And I have a little command-blend wrapper. So I run this command-notice, run a command, and then I get something like this on my screen and a bing and my phone buzzes. And it's really great at reducing the latency between when my compile finished and I noticed that it finished. So that's been really helpful. Another thing, I build everything in Tmux sessions. And actually one of the things that I discovered part way through is really important to switch away from the build. Do not send it over the network. Even over gigabit locally, enough stuff buffers that it significantly delays your ability to get back to the prompt. So better to not render it at all. That was slightly surprising. I found that at Cambridge. I knew it was an issue over the Atlantic. I would switch away when I was tethering. So I didn't just waste all that bandwidth. But it turns out, yeah, always switch away. And I guess one final little tip, continuous integration is really great. We do full OS builds after every compiler or OS change. They take about 20 minutes. We also do full releases at least a couple times a day. Keeps everything working. And one of the key things for us is because our architecture is weird and we are making strange changes, we build both Cherry, MIPS 64, and AMD 64 all the time. AMD 64 is actually a really good canary because it also builds I3D6 libraries. So that means we build a whole lot of stuff and just keeps us honest. We are a research project, but we really want this stuff to be used for real. So we don't want to get to veer down some blind alley in code changes and then come back and say, oh, I have three months of work to get this thing functional again. We're into the shape where I could merge it. So we do that all the time. And also, when we're working on a release now, we create a separate set of Jenkins jobs to build the release daily in the release plan, just to make sure everything is stable. I'll mention some of our papers that we've got published. We have three top two papers in the last year. We had a Cherry hardware paper. So here's the hardware paper. The Cherry capability model, we're visiting risk in an age of risk. That's kind of mostly what I talked about earlier. We have beyond the PDP11 processor support for a memory safe C abstract machine. That was an ass-bloss. If you're a C-geek, it's a great paper. I definitely recommend it. I learned all sorts of weird things about C writing the paper. So that was fun. And then we had a compartmentalization paper at IEEE Security and Privacy a few weeks ago. There's also an ISA document and whatnot on the Cambridge website. We have a future work here. So we're working on a pure capability for VSD. So probably it'd be a very long time, even once hardware exists, before you'll ship a version of VSD that only uses pure capability code. But it's pretty likely that you'll want to run a fair bit of pure capability code. And we also realized recently that the best way to get a ton of code running is to be able to have a pure capability build within free VSD. So we can just compile everything and try to use it, run it through the test suite and see what happens. It'll no doubt be exciting. We're also working on, we'd like to add Cherry to the kernel. The code we have in there is all assembly or macros around inline assembly. It works for what we're doing, but we'd like to be able to do things like protect mBuffs, protect storage buffers, compartmentalize the kernel, make it into a microkernel. And we're hoping, well, so one of the other things, as I said, early on, 256 bits is a pretty big pointer. There'll be lots of overhead in terms of cache footprint and whatnot. In extraordinarily pointer-heavy benchmarks, it's about 20% performance overhead, which is not acceptable. So we were working on 128 bit compressed capabilities. Some interesting trade-offs we're exploring. We're pretty confident it's going to work, but there are a lot of details. And in our simulations, that should get the overhead down to about 3% in a benchmark that's basically data structures that are pointers. And then we're also looking at non-nibs architectures. So happy to answer any questions. I just threw up on here our timelines. We've been at this about five years now. I am guessing we've got over 50 years of work into it. And we've got quite a bit to go. But it's a fun project. And I think we've done a lot of interesting development and stuff that's generally useful for free BSD along the way. Any questions? Yes? So your communication stuff, are you synthesizing varialog stuff and then running your compiler software on that full stack integration? Yes. So we're actually using BlueSpec System Varialog, which is a Haskell-derived HDL. We both compile it. So it has a mode to compile to a cycle accurate C simulator. So we've been doing it for a long time. More recently, we've started synthesizing bit files, loading them on FPGAs, and then running the operating system on them. And that's hide in Jenkins? Yeah. That's Jenkins. The Jenkins cluster has grown quite a lot over the last year. On the hardware side, one of the RA's is amazingly patient and willing to deal with broken junk. Is that part of your structure open for? No. I don't think so. Parts of it might be in the release. I'd have to go look. If you want to ask me later, I can take a poke at the GitHub repo of what we've actually put out and what's buried in the Jenkins config. You said you're writing out another curriculum you just call in your face. Is that the Cloud ABI work? No, this is not the Cloud ABI work. I probably should have gone to the Cloud ABI talk before, but I talked to it a bit. So what we're doing is adding a new syscall ABI. So like the lib32 and the free BST32 emulation, we're adding a cherry ABI. It's sort of the first cut. Not clear what the long-term right answer is, but that's one we know how to do. So I did a first pass at syscalls.master. The great thing is you take all the lines that say compact, clear them out. We've never shipped with those versions, and we never will. Well, thank you all for coming. If you have any questions later, we'll bring them back to you.
CherryBSD, which is our fork of FreeBSD to which has limited rights. In this example, this little bit more on the CPU. We have a prototype in a bit. There's infrastructure to support the compartmentalization of libraries or whatever it some custom applications. I'll talk more about some setup of Cherry and the kernel basically separate from the memory safety. Bit for system assembly versions as well that are a little our current infrastructure, that's how it has to have demo applications. This is a screenshot here present bigger challenges is we're also modifying which has the advantage of being in the base, date, is that we learned some lessons about something that's a fork of FreeBSD in the long reason to stick with Perforce. Downside is repository rather than having to take the time to granularity in the process because many of the getting a little short on time, so I think I will are a research project, but we really want this