The Evolution of FreeBSD Governance
McKusick, Kirk
This talk describes the evolution of the governance of the FreeBSD project from the time of its conception until the present day. All open sources projects develop a culture. Often the culture builds organically, but as a project becomes larger and more successful it is necessary to put more formal governance in place to ensure that the culture is robust and welcoming. This talk studies the evolution of the culture and governance of the FreeBSD project. It begins by looking at the formation of the FreeBSD project and the organizing principles of its founders. As the project became more successful, it began being used and depended on by companies which in turn required that the adhoc governance become more structured and documented through the adoption of a set of bylaws. The talk will cover the governance issues that the project has had to handle in recent years including: FreeBSD Source-Code Control FreeBSD Workflow Guidelines on How to Work and Play Together FreeBSD Recruitment FreeBSD Development Model FreeBSD Core Team Interaction with the FreeBSD Committers FreeBSD Security Team The talk concludes by prognosticating on future governance issues that may have to be handled.
 Okay, I think I'm going to try and get started here, because if I get started on time, I can end on time. And I know many of you are here because you actually want to be here for the auction and have a good seat for the auction. But hopefully I can give you a little entertainment in the meantime. So the evolution of free BSD governance, if you've ever wondered what happens when you get too old to code, then you get to talk about things like governance instead. And because I wanted to have this talk cover at least something a little beyond free BSD, I decided to start it out by talking about this booklet that I received. It's the promise of Berkeley for alumni, parents, and friends who shape Cal's future, which is to say they contribute money to the university. At any rate, this year is the 150th year of the University of California, which I realized for those of you from Europe is very young. But by the standards of the United States, it's getting up there. I mean, it's not Harvard or one of those places. But anyway, there's a section called Game Changer Discoveries. And they list six discoveries that have been made at Berkeley that had worldwide impact. So it includes things like in 1980 when they discovered that it was an asteroid that struck the Earth 65 million years ago that triggered the mass extinction of the dinosaurs. Or in 2012, when Jennifer D'Anna came up with the CRISPR, the thing for splicing and cutting up DNA. And in 1942, when Glenn Seaborg discovered radioactive plutonium, one of 16 chemical elements including beryllium and californium that added to the periodic table. In 1998, Sal Herpmuter with the acceleration expansion of the universe in 1939, when Ernest Lawrence became Berkeley's first Nobelist with things like the cyclotron. And number six, in 1977, Bob Fabry and his student Bill Joy created Berkeley's software distribution blah, blah, blah. So anyway, I'll pass a copy of it around in case you want to get all the details. Pass it around. Okay, so BSD is a big whoop, at least as far as the University of California is concerned. Never mind the fact that they run Linux from top to bottom and you try and find a student that knows anything about BSD and they look at you rather strangely. Is that like LSD? No, no, slightly different. All right, so for those from open BSD and that BSD, that's all you're going to hear about your two operating systems for the rest of this lecture. Sorry about that. Okay, so let me just start off with a background and introduction. Where did all this come from? The Berkeley software distribution, BSD, started in 1977 as a project of Bill Joy at the University of California. And I got involved in 1977 because I was one of Bill Joy's office mates and it was impossible to be around Bill Joy and not get sucked in. It's like, oh, could you just do this one little thing? You know, it won't take but a few hours and two weeks later, blah, blah, blah. Anyway, that originally was just a distribution of utilities, VI, and the Pascal compiler, C-shell and other things like that. But it became a full distribution with the release of three BSD for the VAX in 1979. So that was the first release of BSD that included the operating system and the utilities around it. Initially that was a port of the 32V, which was the version 7 UNIX ported to the VAX but with virtual memory added to it. That's the main addition that was in BSD. It was just a swap-based system before that, 32V. I had the utilities that came with 32V, so all the usual CAT and LS and so on and so forth, plus all of the utilities that Bill had been hacking on or pulling in. People like to think of Bill as a great software developer, but in fact, he was even better at marketing, so he would go out and talk up all this stuff and other people would say, oh, I've got this or that, and he'd bring it in and pee on it to make it smell like Berkeley and add it to the distribution. At any rate, we moved to using source code control in 1980, a very early source code control system called SCCS came out of the labs. Initially, it just had the kernel on it under it, but fairly quickly, we got the rest of the system under there. In fact, the SCCS itself got converted over to an SVN repository by Peter Wem, I believe it was, and no, it was John Baldwin did that. He has put it up. If you actually want to go back and find out who to blame for various and sundry things, you can do that. Anyway, about a decade of releases were managed by the computer systems research group, a four-person development team that, of course, started out as Bill, and then Mike Carls, who's here today, and myself, and Keith Bostick and Keith Sclauer, then continued pushing that out the door with the help of various and sundry other people that were peripherally involved with us there. And finally, at the end of the 1980s, actually, in 1991, we did a release, which we called Networking Release 2, which was more or less all of the system. We did a Net1, which was just the TCP IP stack in the associated networking utilities, and we decided we wanted to expand that a bit, and we didn't really want to have to go through all the rigmarole of our release with the lawyers. So we said, well, this is just a revision of Net1. Never mind that it contains 10 times as much software. At any rate, that landed a lawsuit against the university, which finally got resolved in the university's favor in 1994 when we did a release of 4.4 BSD Lite, which was Net2 minus about six files to make the lawyers happy, and, of course, the addition of everything else that had happened in the meantime. So that 4.4 BSD Lite, well, Net2 was what started out being the first release from, open source release from Berkeley, and Bill Joelitz took that, added a few missing files to get it as a bootable system for the 386 architecture, and was releasing that. And then, because he was sort of slow moving two groups, NetBSD Group and the FreeBSD Group, both formed patch kits that came out of that, and so those two projects then evolved out of there. With the settling of the lawsuit, one of the requirements was that you had to switch over to using 4.4 BSD Lite as your base. I think that perhaps the people that pushed that requirement thought that that somehow was going to stop FreeBSD and NetBSD in their tracks, but it did not have that effect, thankfully, and so away they went. Okay, but that was the last release from the university. They switched, since SCCS was proprietary, they switched to the CVS source code control system, which NetBSD continues to use to this day. I'll talk more about that later in terms of FreeBSD. They also created a core team, core team that had lifetime terms created to decide who should be allowed to commit, and the initial distribution actually got done by Walnut Creek CD-ROM. Jordan Harbert, who was one of that original core team, ended up being hired by Walnut Creek CD-ROM along with Rodney Grimes, who's around here somewhere as well. Walnut Creek basically provided the place to do the builds and to get the distribution going and so on. So that was sort of the first home base, if you will, for FreeBSD. One of the problems was that stuff just kept getting accreted and accreted, and the distribution kept getting larger and larger. And so fairly early on, they realized, all right, we're going to switch to having a base system and then ports. And if we sort of keep the base system as a manageable thing, we can then let ports sort of grow and not have to worry about the whole distribution getting too big. And then NETS also got brought up to manage the inevitable bug reports. They were just as well managed then as they are today. Well, it's just to say, it needs work. FreeBSD continued to move forward and started being picked up by commercial organizations. In particular, Yahoo was one of the very early, very big users of FreeBSD, which ran pretty much their entire organization on it. And as the demands for what we needed for machines to maintain all of the state of the project grew, it moved from the couple of machines at Walnut Creek CD-ROM to Yahoo, who had not only the machine resources and the 24-hour management of the machine rooms and particularly network capacity, which was needed to do the ever-growing distributions. And this was great. Having a company that's willing to basically run your infrastructure at no cost to the project won't cost the project had no money, so there wasn't a lot of choice there. But Justin Gibbs decides that this is not a good long-term strategy because companies might change directions or fade from the scene, and then we would be up the creek. And so he started the FreeBSD Foundation with the concept that it could bring in enough money to provide for the FreeBSD infrastructure. Well, that took about 10 years before it actually had enough income, reliable income, that it could start to do that. And luckily, Yahoo was perfectly happy to continue supporting the project over that period of time. So it's only been in about the last five to seven years that the FreeBSD Foundation has slowly taken over, the infrastructure management moved it out of Yahoo and a couple other places where it was scattered around and is now able to fully handle that. All right, so meanwhile, how about the project itself? Well, as you'll see as a sort of recurring theme, one of the problems you have with open-source projects is that people get all excited and they do a few things, and then they kind of get less excited and other things start picking up in their life. And so you end up with Deadwood. And if you get too much Deadwood, then the whole project sinks just by virtue of there being so much inertia and inability to get things done. And the core team had grown to about 20 people because once you were on, you were on for life. You didn't want to leave because then you'd lose the prestige of being on core. So this core had gotten up to 20 people and only about, let's conservatively say maybe half of which were doing anything and about a quarter of whom were really actively doing things. And so it became evident that we needed to do something about this. And so a group of previous D folks got together and decided that they would create a set of bylaws. And part of setting up the bylaws would be that we would have elected core. So we'd take the rules that you know today pretty much. You have the anyone that's a committer of any sort is allowed to run and all the committer get to vote and the top nine vote debtors become core for the next two years. Well, of course, in order to get the bylaws put in place, the existing core team had to vote for it. And if you're about to put bylaws in which say you need to be elected and there's only going to be nine of you, then that means by definition half of you are not going to be on core. And so they're not incented to put these bylaws in place. And so instead what we had to do in order to make this work was to grandfather the existing core and just hope they didn't actually read the bylaws which most of them didn't and put them in place. And then and only then two years later when they were asked if they wanted to run and said, oh, but I'm grandfathered, you said, didn't you read them? You were grandfathered for the first term. There's a couple of them that still will not speak to me, but. All right, so now we have core and they're elected and this is, you know, it's not a great system, but it's the best system that seems to be generally available. And it's, I mean, we went from, you know, Grand Unnipotent High Stomper, the Linus Torvold's model, which was Bill Joy and Mike Carlson myself. Then we went to the sort of semi-Grand Unnipotent High Stamper, which was the core for life. And now we've moved into where we can routinely change out the core team. And you obviously don't want everyone to change all in one election because you would lose too much institutional memory. But in fact, it seems to sort of move forward, you know, about half of it turns over with each election or slightly less than that, which seems to be good. We've had sort of, I would sort of call it three leadership groups if you count the original core and then we've had at least two leadership groups in the elected core since. So we actually are doing a good job of moving forward and letting people that have gotten burned out move along. All right. So what do the core teams do? I assume most of you know this, but just for reference, we have, they oversee the port manager team who oversees the ports committers. We have the documentation team, which oversees, I was impressed to say, 126 dock committers. If you, you know, look at almost any other project, if they even have dock committers, I bet that it's, they're lucky to break a two digit number. We have the security officer who handles security issues, alerts and updates. The administration, system administration team that oversees the infrastructure, the release engineering team that oversees releases and the QA team, which we're just now really getting going that deals with continuous integration builds and expands the regression tests and so on. And you know, core nominally is responsible for all of these things, but they break it out and there's really teams that are doing these things. And so core mostly gets involved when things go off the rails. Okay. So the FreeBSD project today, initially, believe it or not, there was the FreeBSD mailing list. You can imagine how well that worked. It, of course, got broken up into a lot of area specific lists like architecture and networking and file systems and blah, blah, blah, blah. But the problem is there got to be so many lists that if you really wanted to know what's going on, you had to watch too many lists. And so it became necessary to have some way of getting cross area collaboration. And initially it was done sort of through the bug tracking system. Nats later got upgraded to Bugzilla. The project moved to Fabricator in 2014, primarily so that we could expand the discussion to non-committers. It was very difficult for people that were doing development but not committers to really interact. And so Fabricator made that possible. And you know, there's been a long list of complaints about Fabricator, but it's a step forward from where we were before. So we'll leave it at that. Chorus code control. Well, the project started on CVS. And there was a huge amount of debate about moving to something else. And NetBST has been struggling with this of late as well. And so eventually in 2000, and I'll talk a little bit more about that later, we did switch to subversion. And now there's been ongoing debate about switching to Git with all of the same arguments that we had before. The arguments in favor, there's a lot of programmers that no one used Git. There's for several years, there's been a everything that gets checked in immediately gets pushed down to from subversion to GitHub. But the problem is upstreaming from GitHub requires going through Fabricator. So you've got to pull a change request up to Fabricator and then go through that and then get that pulled into the main system. And that there's too much friction there. And so being able to just do pull requests directly would be helpful. On the other hand, it wouldn't be on GitHub because the project just sort of in its DNA wants to control that repository. And so there's a lot that would have to happen to be able to get to Git. We'll see where we go. OK. So the workflow, committers are up to three groups. We have documentation, which is all the previous documentation, ports, which are all the ports, and source, which is the base system. And the key thing is the changes that can affect a kernel interface or operational semantics and various other things get put up in Fabricator for review. And all non-privilege changes require a review by somebody else. So I may want to change the fast file system. And I can't just say, well, I've done it for 30 years. Just trust me on this one. I actually have to get someone else to look at it. And it turns out to be a good idea because sometimes even if you think you know what you're doing, you don't, in fact. And having someone point this out to you before you've broken things is generally better than afterwards. So the, and again, a lot of this depends on people working together. I mean, you have to essentially, you know, at least put feelers out and know some other people that can do reviews of things you're working on. You have to actively go out and review things for other people. And so meetings like this are ideal because you actually get everybody in a room and you actually meet a bunch of new people and you find out about things that they're doing. And that will then go on and help integrate making some of these other things that need to happen, happen. Okay. So FreeBSD has commit tags. And the commit tags include a bunch of information, which other project member has reviewed the changes. Now, this is an important one because it forces you to put the name of somebody on which you can add at FreeBSD.org. And if something really stupid goes in there, you can say to that other person, you know, how did you possibly pass that in review? And if they say, they just made that up, you're in deep yogurt. So you really do have to get the reviews and you really do have to listen to it because that other person, you know, if you've been asked to review, you can't just sort of say, oh, yeah, they know what they're doing and say yes, because if it blows up, you're going to be almost as much to blame as the person who put it in there in the first place. And this is especially true if you are mentoring somebody else because you're supposed to make sure that they don't goof up. And so if they goof up, that makes you look like a really bad mentor. Many things do come in from bug reports. So if there's a related bug report, you tag that. And that will also put a message back to whoever submitted the bug report and anyone that's following that bug report so that they know that things have happened. Things that get discussed on Fabricator, again, are noted so that the linkage goes back to Fabricator. And then if the report, if the bug is something that should be pushed down into the stable levels, you note when that's supposed to happen. And one of the more interesting and more recent things that's gotten added is a sponsoring organization. So if you work for a company and you're, you know, let's say, pick a random example, Netflix, and so there's something that you've done there but you're pushing up to the master tree. You put that in so that your organization gets some credit for the fact that they're paying for someone who's actually doing useful stuff in free BSD. And my favorite one is at the end, sometimes what you're doing is you're checking in because you did some particularly stupid thing earlier. And then you get, it's the pointy hat tag. And you put that in if it's basically a maya culpa fixed to an earlier thing that you checked in. Maybe with a note of which one that was. All right. So now here, now we start to the hard stuff. How do you work and play well together? All right. So the project started out with behavior guidelines based around commit behavior and rules for suspending commit privileges if you misbehave. So, you know, don't get in commit wars. Like, one person put in a change and then the other person go, no, that's not right and put in their change. And then the first person switch it back to what they had before. And this hasn't actually happened too horribly in recent years but it used to be a not infrequent enough occurrence that there actually got to be some fairly detailed rules. Inevitably, you know, Cora has to step in at some point and say, you know, knock a couple heads together, you know, get the two to agree on what it ought to be and put that in. And if they're too busy counter committing against each other, then Cora can pull the plug and just say, okay, the two of you are, you know, in the corner for a week. Neither of you can commit for a week until, you know, everything calms down. And then after a week, if that's not enough, then it can go to two weeks, four weeks, six weeks, et cetera. And sometimes it just gets to the point where a person has to leave the project. That's only happened two or three times in the 25 years that the project has been around. But sometimes there's just such fundamental differences of opinion that that's the only solution. And obviously you don't want to do that but if that's all you can do, that's what happens. Okay, so having Cora kind of act as the adult in the room worked very well for a long time until Gamergate blew up. And when Gamergate blew up, that just, you know, it was no longer in the FreeBSD project, it was all over the web. And it became clear that what we had as rules which had worked very well for a long time were just not up to the task of that situation. So it was decided that we needed a more formal code of conduct which was drawn up by a well-meaning but inexperienced set of folk. That is people that hadn't really done a code of conduct before, they just kind of looked around the web and pulled some things together and put it here. Here it is. And the feedback was this is insufficient. It just, it looks like it was written by amateurs that don't understand how to do this kind of thing, which it was. So okay, fine. We're going to do it right this time. So it was replaced by a code of conduct drawn up by folks with more experience. But the problem here was that it was overbearing. And it was a code of conduct that might work well in certain situations, particularly corporate situations, but it really wasn't tailored to what free bestie needed or wants. So a third revision of the code of conduct I am told is in the works. Jeff Roberson has collected a bunch of information and it's pretty much geared up to wait for the next core to come in rather than the existing core which has been involved in the first two of these. So that we can get some new eyeballs on it and hopefully get some more feedback from all of everybody in this room and elsewhere in the project. There are people that would just like to go back to the old days and we already have proof that that doesn't work. So we need to do something here with a code of conduct and hopefully the third time will be the charm and we will have something that people are both happy with and will bring us into the modern world of what we need. So we'll see where that goes. Free BSD Committer Turnover. I've already talked about how we had to get rid of dead wood in core. Well that's not just core, that's the issue. We have the same issue with committers as well. People get excited, they work for a while and then they sort of fall off the, you know, and move on to other things and you have to have some way of eliminating them because otherwise you just end up with too much dead wood. And you have to have a good way of doing that in a way in which it's not perceived as the, you know, people in charge picking on individuals. And so there, it has to be essentially an algorithmic way of deciding that you are dead wood and it's time for you to move along. And it has to, everyone has to understand what the algorithm is. The algorithm is pretty easy. It's you drop commit privilege after one year of inactivity. And so if you just don't do anything and this is like a microscopic, all you got to do is go once a year and change your personal information in your, you know, committer description and that does it. Okay. And it's not like a lot of effort to remain a committer. And I did notice a large number of those commits happening recently so that people would be eligible to vote for core. Good on you. Okay. And in fact, if you haven't made an entry in the calendar file yet, please do that. That'll count. All right. So for up to about two years and this is not quite as hard and fast because core sign gets to decide but after the one year where it's dropped off, if you won't come back, you know, after 18, 20 months or so, core will generally say, okay, fine, you know, just turn it back on again. But after it's been more than about two years, you actually have to go through the whole getting on board process again. And a lot of that is because the way the project does things just changes over time. You know, we change from NETs to Bugzilla, we've added fabricator, you know, there's just all kinds of things that have just been done in different ways. And so you can't just come back and just jump into it doing things the way they did were done five years ago because people will get upset reasonably. So what ends up happening is that you still get a mentor just as you would if you were a brand new committer but typically you don't have to be, you know, a mentee for a year. You can often just in three to six months be up to speed again and, you know, your mentor will say, okay, you got it, go. Which means they're off the hook for you doing stupid things. All right, so the FreeBSD project has done a very good job of, you know, getting rid of the deadwood but of course the other half of that is if you just get rid of people then pretty soon you don't have anybody left. And so recruitment turns out to be just as important if not more important than getting rid of the deadwood. Okay, so you've got to bring in new developers and there's quite a few places that this can be done. A lot is contact via university courses. There's this whole website that several of us have been working on called TeachBSD which is setting up curricula that can be taught in universities and going out and actively doing stuff in universities. Robert Watson and George Neville-Neal in particular have been doing a lot of stuff at Cambridge. Another key place to get them is people working in a company that's using FreeBSD. And I mean, if the company is using FreeBSD they are almost certainly doing some level of development and almost certainly has things that need to be upstreamed and so it's both to the company's advantage and to the project's advantage to get those people brought on board as committers so that instead of having to nag somebody to get things put in they can just do it themselves. And this is of late proven to be a very good channel for bringing people in. Now you say why would someone want to become a committer and just have extra work to do after you've done your 50 hours of work at the company. Now you're going to do another 10 hours for FreeBSD. But it turns out that if you're a committer to FreeBSD it actually raises your value. So if you change jobs it tends to give you a salary bump when you go to another company that's using FreeBSD because you're already a committer and this is viewed as a very useful asset to have. So in fact we've had very good uptake on bringing in people that are working for companies and become FreeBSD committers. Another key place that has brought in a lot of new committers over the years has been the Google Summer of Code project. FreeBSD has typically been given 10 to 15 slots in Google Summer of Code and for those of you that have pay any attention to Google Summer of Code most projects are thrilled to get two or three spaces and we are routinely getting 10 to 15 and we could have more if we could get more mentors and more projects in line. So you know there's a lot of there's a small set of people that work very hard on Google Summer of Code and it would be great if we could get some more people involved in that. And then finally there's just discovering the project at a conference or through social media and particularly EuroBSD which moves around to different countries every year. We get a large influx of people that have never heard of BSD and all the projects and not just FreeBSD but NetBSD and OpenBSD tend to get a lot of excitement and new people that come on and start working with the project. It's key that your project be welcoming and easily entered. You come here and it's like oh yeah I know everybody blah blah blah. Well if you're new to the project you don't know anybody here hardly other than maybe their name and so especially those of you that are older timers try and make a point of talking to the new people and welcoming here and making it sound like it's a fun place to be and not a terrorizing place to be. So provide an easy path to connect with the project and provide mentors to help get people up to speed. Again mentoring is not for everybody. There's some people that come up and say we want to be a mentor and it's like we have some other things that you could help us with. But most of the people here could be good mentors and you know well how do I become a mentor. Well talk to us. Anyone from CORE say you want to be a mentor you'll be their best friend for a day. They might even buy you a beer. So what is the net effect of all this. Well I've been collecting statistics over the years. In fact this is something I cranked this script actually an hour ago and updated my slide because one of our new committers is 21 and previously it was 23 I guess was the youngest committer that we had. So I was very happy to be able to pull this slide out. I didn't knock the median or average down but nevertheless it helped push it that way. At the other end here I'm unfortunately three from the top here. Diane Bruce has got me beat she's here and Greg Lee he was not here but he's out here at the end. If you just take off the over 60 crowd here the median and average both dropped down to around 38 39. So you know just get rid of the 10 old geezers and your graph will look better. At any rate the key thing to note here is first of all this big huge spike which is late 30s early 40s. And the project has stayed right in that number range for more or less a decade. When it was very when it first started up it was down in the low 30s and it has slowly risen it slowly rose into the high 30s and it's pretty much been there for at least the last eight to ten years. So we are doing a good job of bringing people in and that's good. If a project is going to have a long lifetime you've got to keep this age demographic you don't want it to age out. And in particular if you look at the Linux project Linus Torvalds and his lieutenants are pushing out on that curve and they are not doing a good job of bringing in younger people. They are bringing in a lot of people in companies so it's not like that project is going to go away but the sort of image of what Linux ought to be is going to become a corporate one if they don't somehow work to replace that top set of people. I'd iterate some of the other numbers are kind of interesting here. The calendar.previousd is where I get the ages so there's 46 of you who won't ante up what your age is. I actually sent out an email middle of last year to everyone that I didn't have an age for and said either put yourself in the calendar or tell me what it is and I'll put it in my secret one that I won't tell anybody else what it is. And it's about time to do that again because I had it down to around 25 at one point. And someone was saying oh yeah I saw a big flurry of changes to calendar.previousd. What was that all about like me? An email. So anyway if you're not in the calendar or you haven't already sent the information to me I'd appreciate it especially if you're younger. Bring that age demographic down. Great. So the previous development model some of it works and some of it not so much. What we do great at is small changes especially ones that are easy to identify, reach some consensus on and implement. And when I say small changes I mean changes that can be done by one or two or three you know individuals working together. And we have numerous examples of things and you know it's not like that small of a change and we got SCTP brought into the system but that was basically Randall and a couple of his cronies that did that and they didn't need like you know everybody in the project to be involved in order to make that work. It's the larger wide ranging changes that are hard. So some examples switching from CVS to Subversion took nine years. And the problem was that we would start with the discussion and everyone would say oh well there's this and there's this and there's this and have you considered this and have you considered that. And blah blah blah blah blah blah blah blah blah and there would be no consolidation on one particular thing and it would die out without resolution. And 18 months to two years later it'd be like oh we need to do this and we'd forget the entire discussion we'd previously had and we'd have the same discussion all over again and the mailing list would be flooded with stuff and blah blah blah it would die out. And it was finally accomplished by Peter Wemm who just said I'm sick and tired of this I'm just going to do it. And he just did it and one day it was click and now we're under Subversion. This is not an ideal way of handling this. Okay. The other huge change came when we went from the single threaded to the symmetric multi-processing which was between FreeBus D4 and 5. And this required changes obviously throughout the entire system. And you sort of have this theory that well we'll just put a giant lock in and then we will just slowly unlock things bit by bit. And you know forget PID that's pretty easy to do. But when it gets to things like the networking stack where you've got locking well let's see we've got packets going down so we're going to lock from top to bottom. Oh and we have packets coming up so we're going to lock from bottom to top and blah blah deadlock. So it's not that easy. So we started just to make it interesting with a big debate as to what kind of locking we were going to do. The Solaris style locking or the Amigo style lockless stuff. And that debate raged for a while. And eventually it got decided that we were going to use the Solaris style of locking. And the lockless people weren't willing to give up and kept trying to put that in and that's when there was an eviction of a member of the team. But you know once we got that sorted out then it still required this just massive coordination and for any of you that remember it was a long gap between four and five coming out. And five came out kind of because we had to it had been so long and five left a bit to be desired in terms of stability and so on. And it was really not until six came out that we really had sort of gotten over the hump. But you know that both of these are examples where the model of just sort of making changes and everyone sort of coordinating on mailing lists just doesn't cut it. So how do we do better on those things? So we recently as in about a year ago, CORE created the FreeBisD community process which is modeled on the Python enhancement process. And the idea is we want to get away from this, have a huge argument and then do nothing and forget about it and have the same huge argument again and do nothing about it and so on. And just for an example of something that's on the horizon that we're going to have to deal with is deprecating I386. Okay. We've already had, you know, one round of discussion on this. It's clearly not time for it yet but the time is going to come. And we really don't need to keep every, you know, 18 months having a big discussion with the same arguments over and over again. So this is a classic case where this will hopefully help us do this in a much more rational way. So the major changes are described in the document. Well, the major change is pretty easy. We drop I386 support. But then there's, you know, here's the reasons why, here's the reasons why not, here's, you know, reasons, the why not reasons may die down over time, here's some of the ways that we can determine that they're dying down. And so we have that whole, you know, distillation, let's say, of the discussion that we've just recently had on the mailing list and put that into one of these documents. And this gets reviewed by an editor whose job it is to, you know, edit, make sure all the opinions are there and, you know, so on. And then it gets published for discussion. And as the discussion continues on, those comments get put in there. And when the core team feels that the discussion has kind of died down, then they make a vote on deciding, you know, whether we're going to move forward with that particular thing or not. Now, in the case of I386, they will almost certainly decide, no, we're not going to do it at this time. But this document is there. We've had all these discussions, so 18 months from now, instead of starting from ground zero, we can restart this document, which we already have a bunch of things in. We say, all right, what's changed? All right, well, some of the reasons for keeping it now have gone away. So we'll move them from this pile to that pile. But then there's some other reasons that have come back in. And so, again, we'll have, you know, sort of, the discussion will move forward. We will probably still decide two years from now that we're not ready to get rid of I386. But, you know, eventually, we will come around and the document will get to the point where the reasons to keep it will be dominated by the reasons to get rid of it and it will happen. And it will happen with, let's say, one-tenth of the amount of mail on the mailing list as we would have if we just keep with the current scheme of arguing about it every 18 months. So this hasn't really gotten a good test yet, but I think we've got some ideal candidates that should go through this process. Okay, another issue that has arisen of late is the core team interaction with the free BSD committers. In particular, there were some that felt there wasn't enough communication about the code of conduct, can you imagine? And so there's been a push to try and get core to be a little less opaque. Historically, all the core communications were private among core and a monthly report would be thrown over the transom once a month saying, oh, this is what we did. These people got commit bits and we had discussions about these topics and we made these conclusions, these decisions. Period. That was it. So there was a feeling that there wasn't a good way of giving feedback to core. All you got to do is say, well, I don't like that decision. It's like, well, tough, it's been made. Wait for next month. So in recent time, core has started releasing an agenda of the discussion topics so that you can see what they're going to be talking about. And if there's something that is of interest to you, you can provide some feedback before they've had the discussion. Now this is a nice idea, but the problem is that they have, I mean, when you look at the thing, it's this laundry list of stuff of which they don't actually end up doing all of those things in that meeting. They just want to pick through a few of them. And so it would be nice to know, well, what are the things you're going to actually talk about in this upcoming meeting as opposed to, well, someday you're going to talk about that maybe. So a little bit of work needs to be done there. There's also contemplating allowing committers to join the video conference, not being able to talk, but being able to at least listen and perhaps poke in questions through IRC or something. Now there's certain things where they have to deal with personnel issues. And obviously that is just going to have to be a closed session. But just like when you have a board of directors, most of the meeting is open and then there's a small section that you have where it's just the board. But again, that's a relatively tiny piece of what they have to deal with most of the time. So there's a hope, and this is sort of my throwing it out to whoever the incoming core cuts to be, that this is a nice start, but keep pushing this forward. OK, finally, the FreeBSD security team. Security has been a lot in the news lately. And on average, the project has done well in this domain, but unfortunately it's a job that's really hard for volunteers in particular because it's mostly nothing to do until there's a fire drill in which they need 110% of your time. And you might have another job, and your employer doesn't want to just give you a month off with 30 seconds of notice. And so the people that have done it have done an excellent job, but they get burned out. And so we'd had a series of very good security officers, but we'd gotten to a point where we needed someone who could do it, especially with a lot of things that have been coming down more recently that was becoming much more of a job than it had been. And so it seemed like a place where the FreeBSD Foundation could come in and at least provide the person who's going to do the sort of management piece of it. So not necessarily to figure out the technical part because we have a lot of members in our community that can help with the technical issues, but can at least be there and deal with the immediate responses that need to go out and coordinate, find the correct people to deal with the problem and get them pulled in and working on it, et cetera. So in fact, as far back as 2002, we expanded it to a security team where we had the security officer who was doing the kind of management bit, and then the security officer, as I say, oversees the pool, people getting that done. And it used to be a part-time person, and now it's become almost a full-time job, at least the last couple years. And so that's where the FreeBSD Foundation came in to help provide that person. It doesn't mean that everybody's off the hook, no means, but it does mean that there's going to be somebody that when the reporter calls up and says, what is FreeBSD doing, they at least get someone to answer the phone. All right. So the grand summary, the evolution of governance model for leadership, this has been something that has been of particular interest to me because having been one of the benevolent dictators for life, that was clearly not a model for something where it was going to outlast me. And so moving into having a core team for life was sort of the next logical step. Moving to an elected core team seems to have worked well, and the project continues to move forward and grow, which is great. If I get hit by a bus, the FreeBSD project isn't going to notice the difference. Four major changes in leadership, each allowed the project to move forward and tackle new problems. You get people that come in and the early getting the BSD out in the first place, getting it open sourced, took a few years off of several of our lives. It has certainly helped the project avoid the aging out. The median age has remained in the mid to high 30s, at least over the last decade. And the elected core, I mean, there's all kinds of things where that can go wrong. You can end up electing people that just aren't, you know, core is much more of a management position than a technical position. I mean, they obviously have to have technical chops, but a lot of what they do day to day management is interacting with people. And so the project as a whole seems to be pretty good at bringing up people that have those skill sets. And you want some turnover, you don't want it to just be the same group of people forever and ever, but you also don't want to turn everybody all at once unless they've gone really off the rails. And so, you know, again, bad things could happen, but so far they haven't, and hopefully it will stay that way. The FreeBSD Foundation has provided not needed resources and stability. They are somewhat the long-term memory of the organization. They deal with the long-term issues like making sure that the copyrights don't get abused and, you know, all those kinds of things that, you know, just you need somebody that's going to just do that more or less continuously. So they have the solitude of longevity that's needed for that. Okay, so what's my conclusion? My conclusion is the governance is mundane. I'm amazed to have this many people sitting here listening to me talk about it for an entire hour. But nevertheless, too much of it is stifling. You can certainly overdo it. I have been in organizations that all you want to do is run out the door to get a breath of air. But you can also go to the other extreme and say, oh, you know, it's all laissez-faire, blah, blah, blah. Too little in the project goes off the rails, collapses from lack of infrastructure. There's all kinds of bad things that can happen if you're not paying some attention to this stuff. So it's a fine line. People say, oh, you can't have enough. Oh, yes, you can. Oh, you can't have too little. Oh, yes, you can. So we seem to be doing pretty well here, and hopefully we will stay on that track. The thing to really note is that governments requires constant tuning to keep that rate balance. You know, you ebb and flow. You tend to bounce back and forth between too much and too little. And you do have to pay attention to it. But when it's working, it should just be in the background, and you shouldn't really be all that aware of it. OK, so with that, I will take any questions. Yes? What is core doing about it? That's an excellent question, and that's why I'm not running for core. You couldn't pay me. What was it? If nominated, I will not run, and if elected, I will not serve. I did my 10 years. Someone else has a problem. OK, anything else? Yeah? In 2018, do you see core as more of a technical body or a board of directors? Do I see core more as a technical body or a board of directors? Yes, is the answer to that. They are, in fact, doing both of those things. I mean, ultimately, there are some technical decisions that need to be made where there's disagreement, and hopefully they can help those people come to a resolution, but they have to be technical enough to be able to do that. They also are a board of directors. I mean, they are overseeing all those different groups of people. They need to make sure that those groups are working properly, and if they're broken to figure out how to fix them. That's management through and through. That's why I'm very pleased when I looked at the list of people running for core that we have people that are across that entire spectrum. I think I'm convinced that the core will have the various assets that they need. People, oh, I can't run for core. I'm not a kernel hacker. That is malarkey. Yeah? How long have you been using video chat for months to meet? How long have I been using? Video chat for months to meetings of what tools? You would have to ask the core people that. I haven't actually been paying attention to what they're using. Can you repeat that? Three. They use Google Hangouts? Three. What? We don't say hangout for three years. Okay, thank you, Ben-O. See, you have to ask someone on core. They'll have the answer. All right, anything? Yeah? Generally speaking, we provide some things to core, the pay for when they were using conference phoning. In the case of code of conduct, they came and said they wanted some money to get people to help them with it when they did Rev2. Generally speaking, now with my foundation hat on, we're very sensitive to not somehow being viewed as taking over the project. Now, I will point out that some of core is actually also foundation board members. So it's a fine line that we're walking there, but our goal is that the project runs and we are there as a support service. And when they want some help, they'll come and ask, and we will try and provide it. But a lot of we're doing things that are sort of on the side. So trying to go out and raise awareness and provide travel grants. A bunch of you here are being at least partially paid for your travel to come here. And Google Summer of Code related things and so on. But the goal of the foundation is to just be sort of quietly on the side and that the project and core should be driving what's going on in the project itself. Okay? All right. Thank you.
 Okay, I think I'm going to try and get started here, because if I get started on time, I  can end on time.  And I know many of you are here because you actually want to be here for the auction  and have a good seat for the auction.  But hopefully I can give you a little entertainment in the meantime.  So the evolution of free BSD governance, if you've ever wondered what happens when you  get too old to code, then you get to talk about things like governance instead.  And because I wanted to have this talk cover at least something a little beyond free BSD,  I decided to start it out by talking about this booklet that I received.  It's the promise of Berkeley for alumni, parents, and friends who shape Cal's future, which  is to say they contribute money to the university.  At any rate, this year is the 150th year of the University of California, which I realized  for those of you from Europe is very young.  But by the standards of the United States, it's getting up there.  I mean, it's not Harvard or one of those places.  But anyway, there's a section called Game Changer Discoveries.  And they list six discoveries that have been made at Berkeley that had worldwide impact.  So it includes things like in 1980 when they discovered that it was an asteroid that struck  the Earth 65 million years ago that triggered the mass extinction of the dinosaurs.  Or in 2012, when Jennifer D'Anna came up with the CRISPR, the thing for splicing and  cutting up DNA.  And in 1942, when Glenn Seaborg discovered radioactive plutonium, one of 16 chemical  elements including beryllium and californium that added to the periodic table.  In 1998, Sal Herpmuter with the acceleration expansion of the universe in 1939, when Ernest  Lawrence became Berkeley's first Nobelist with things like the cyclotron.  And number six, in 1977, Bob Fabry and his student Bill Joy created Berkeley's software  distribution blah, blah, blah.  So anyway, I'll pass a copy of it around in case you want to get all the details.  Pass it around.  Okay, so BSD is a big whoop, at least as far as the University of California is concerned.  Never mind the fact that they run Linux from top to bottom and you try and find a student  that knows anything about BSD and they look at you rather strangely.  Is that like LSD?  No, no, slightly different.  All right, so for those from open BSD and that BSD, that's all you're going to hear  about your two operating systems for the rest of this lecture.  Sorry about that.  Okay, so let me just start off with a background and introduction.  Where did all this come from?  The Berkeley software distribution, BSD, started in 1977 as a project of Bill Joy at  the University of California.  And I got involved in 1977 because I was one of Bill Joy's office mates and it was impossible  to be around Bill Joy and not get sucked in.  It's like, oh, could you just do this one little thing?  You know, it won't take but a few hours and two weeks later, blah, blah, blah.  Anyway, that originally was just a distribution of utilities, VI, and the Pascal compiler,  C-shell and other things like that.  But it became a full distribution with the release of three BSD for the VAX in 1979.  So that was the first release of BSD that included the operating system and the utilities  around it.  Initially that was a port of the 32V, which was the version 7 UNIX ported to the VAX but  with virtual memory added to it.  That's the main addition that was in BSD.  It was just a swap-based system before that, 32V.  I had the utilities that came with 32V, so all the usual CAT and LS and so on and so  forth, plus all of the utilities that Bill had been hacking on or pulling in.  People like to think of Bill as a great software developer, but in fact, he was even better  at marketing, so he would go out and talk up all this stuff and other people would say,  oh, I've got this or that, and he'd bring it in and pee on it to make it smell like  Berkeley and add it to the distribution.  At any rate, we moved to using source code control in 1980, a very early source code  control system called SCCS came out of the labs.  Initially, it just had the kernel on it under it, but fairly quickly, we got the rest of  the system under there.  In fact, the SCCS itself got converted over to an SVN repository by Peter Wem, I believe  it was, and no, it was John Baldwin did that.  He has put it up.  If you actually want to go back and find out who to blame for various and sundry things,  you can do that.  Anyway, about a decade of releases were managed by the computer systems research group, a  four-person development team that, of course, started out as Bill, and then Mike Carls,  who's here today, and myself, and Keith Bostick and Keith Sclauer, then continued pushing  that out the door with the help of various and sundry other people that were peripherally  involved with us there.  And finally, at the end of the 1980s, actually, in 1991, we did a release, which we called  Networking Release 2, which was more or less all of the system.  We did a Net1, which was just the TCP IP stack in the associated networking utilities, and  we decided we wanted to expand that a bit, and we didn't really want to have to go through  all the rigmarole of our release with the lawyers.  So we said, well, this is just a revision of Net1.  Never mind that it contains 10 times as much software.  At any rate, that landed a lawsuit against the university, which finally got resolved  in the university's favor in 1994 when we did a release of 4.4 BSD Lite, which was  Net2 minus about six files to make the lawyers happy, and, of course, the addition of everything  else that had happened in the meantime.  So that 4.4 BSD Lite, well, Net2 was what started out being the first release from, open  source release from Berkeley, and Bill Joelitz took that, added a few missing files to get  it as a bootable system for the 386 architecture, and was releasing that.  And then, because he was sort of slow moving two groups, NetBSD Group and the FreeBSD Group,  both formed patch kits that came out of that, and so those two projects then evolved out  of there.  With the settling of the lawsuit, one of the requirements was that you had to switch over  to using 4.4 BSD Lite as your base.  I think that perhaps the people that pushed that requirement thought that that somehow  was going to stop FreeBSD and NetBSD in their tracks, but it did not have that effect, thankfully,  and so away they went.  Okay, but that was the last release from the university.  They switched, since SCCS was proprietary, they switched to the CVS source code control  system, which NetBSD continues to use to this day.  I'll talk more about that later in terms of FreeBSD.  They also created a core team, core team that had lifetime terms created to decide who should  be allowed to commit, and the initial distribution actually got done by Walnut Creek CD-ROM.  Jordan Harbert, who was one of that original core team, ended up being hired by Walnut  Creek CD-ROM along with Rodney Grimes, who's around here somewhere as well.  Walnut Creek basically provided the place to do the builds and to get the distribution  going and so on.  So that was sort of the first home base, if you will, for FreeBSD.  One of the problems was that stuff just kept getting accreted and accreted, and the distribution  kept getting larger and larger.  And so fairly early on, they realized, all right, we're going to switch to having a base  system and then ports.  And if we sort of keep the base system as a manageable thing, we can then let ports sort  of grow and not have to worry about the whole distribution getting too big.  And then NETS also got brought up to manage the inevitable bug reports.  They were just as well managed then as they are today.  Well, it's just to say, it needs work.  FreeBSD continued to move forward and started being picked up by commercial organizations.  In particular, Yahoo was one of the very early, very big users of FreeBSD, which ran pretty  much their entire organization on it.  And as the demands for what we needed for machines to maintain all of the state of the  project grew, it moved from the couple of machines at Walnut Creek CD-ROM to Yahoo, who  had not only the machine resources and the 24-hour management of the machine rooms and  particularly network capacity, which was needed to do the ever-growing distributions.  And this was great.  Having a company that's willing to basically run your infrastructure at no cost to the project  won't cost the project had no money, so there wasn't a lot of choice there.  But Justin Gibbs decides that this is not a good long-term strategy because companies  might change directions or fade from the scene, and then we would be up the creek.  And so he started the FreeBSD Foundation with the concept that it could bring in enough  money to provide for the FreeBSD infrastructure.  Well, that took about 10 years before it actually had enough income, reliable income, that it  could start to do that.  And luckily, Yahoo was perfectly happy to continue supporting the project over that  period of time.  So it's only been in about the last five to seven years that the FreeBSD Foundation has  slowly taken over, the infrastructure management moved it out of Yahoo and a couple other places  where it was scattered around and is now able to fully handle that.  All right, so meanwhile, how about the project itself?  Well, as you'll see as a sort of recurring theme, one of the problems you have with open-source  projects is that people get all excited and they do a few things, and then they kind of  get less excited and other things start picking up in their life.  And so you end up with Deadwood.  And if you get too much Deadwood, then the whole project sinks just by virtue of there  being so much inertia and inability to get things done.  And the core team had grown to about 20 people because once you were on, you were on for  life.  You didn't want to leave because then you'd lose the prestige of being on core.  So this core had gotten up to 20 people and only about, let's conservatively say maybe  half of which were doing anything and about a quarter of whom were really actively doing  things.  And so it became evident that we needed to do something about this.  And so a group of previous D folks got together and decided that they would create a set of  bylaws.  And part of setting up the bylaws would be that we would have elected core.  So we'd take the rules that you know today pretty much.  You have the anyone that's a committer of any sort is allowed to run and all the committer  get to vote and the top nine vote debtors become core for the next two years.  Well, of course, in order to get the bylaws put in place, the existing core team had to  vote for it.  And if you're about to put bylaws in which say you need to be elected and there's only  going to be nine of you, then that means by definition half of you are not going to be  on core.  And so they're not incented to put these bylaws in place.  And so instead what we had to do in order to make this work was to grandfather the existing  core and just hope they didn't actually read the bylaws which most of them didn't and put  them in place.  And then and only then two years later when they were asked if they wanted to run and said,  oh, but I'm grandfathered, you said, didn't you read them?  You were grandfathered for the first term.  There's a couple of them that still will not speak to me, but.  All right, so now we have core and they're elected and this is, you know, it's not a  great system, but it's the best system that seems to be generally available.  And it's, I mean, we went from, you know, Grand Unnipotent High Stomper, the Linus  Torvold's model, which was Bill Joy and Mike Carlson myself.  Then we went to the sort of semi-Grand Unnipotent High Stamper, which was the core for life.  And now we've moved into where we can routinely change out the core team.  And you obviously don't want everyone to change all in one election because you would lose  too much institutional memory.  But in fact, it seems to sort of move forward, you know, about half of it turns over with  each election or slightly less than that, which seems to be good.  We've had sort of, I would sort of call it three leadership groups if you count the original  core and then we've had at least two leadership groups in the elected core since.  So we actually are doing a good job of moving forward and letting people that have gotten  burned out move along.  All right.  So what do the core teams do?  I assume most of you know this, but just for reference, we have, they oversee the port  manager team who oversees the ports committers.  We have the documentation team, which oversees, I was impressed to say, 126 dock committers.  If you, you know, look at almost any other project, if they even have dock committers,  I bet that it's, they're lucky to break a two digit number.  We have the security officer who handles security issues, alerts and updates.  The administration, system administration team that oversees the infrastructure, the  release engineering team that oversees releases and the QA team, which we're just now really  getting going that deals with continuous integration builds and expands the regression tests and  so on.  And you know, core nominally is responsible for all of these things, but they break it  out and there's really teams that are doing these things.  And so core mostly gets involved when things go off the rails.  Okay.  So the FreeBSD project today, initially, believe it or not, there was the FreeBSD mailing list.  You can imagine how well that worked.  It, of course, got broken up into a lot of area specific lists like architecture and  networking and file systems and blah, blah, blah, blah.  But the problem is there got to be so many lists that if you really wanted to know what's  going on, you had to watch too many lists.  And so it became necessary to have some way of getting cross area collaboration.  And initially it was done sort of through the bug tracking system.  Nats later got upgraded to Bugzilla.  The project moved to Fabricator in 2014, primarily so that we could expand the discussion to  non-committers.  It was very difficult for people that were doing development but not committers to really  interact.  And so Fabricator made that possible.  And you know, there's been a long list of complaints about Fabricator, but it's a step  forward from where we were before.  So we'll leave it at that.  Chorus code control.  Well, the project started on CVS.  And there was a huge amount of debate about moving to something else.  And NetBST has been struggling with this of late as well.  And so eventually in 2000, and I'll talk a little bit more about that later, we did switch  to subversion.  And now there's been ongoing debate about switching to Git with all of the same arguments  that we had before.  The arguments in favor, there's a lot of programmers that no one used Git.  There's for several years, there's been a everything that gets checked in immediately  gets pushed down to from subversion to GitHub.  But the problem is upstreaming from GitHub requires going through Fabricator.  So you've got to pull a change request up to Fabricator and then go through that and  then get that pulled into the main system.  And that there's too much friction there.  And so being able to just do pull requests directly would be helpful.  On the other hand, it wouldn't be on GitHub because the project just sort of in its DNA  wants to control that repository.  And so there's a lot that would have to happen to be able to get to Git.  We'll see where we go.  OK.  So the workflow, committers are up to three groups.  We have documentation, which is all the previous documentation, ports, which are all the ports,  and source, which is the base system.  And the key thing is the changes that can affect a kernel interface or operational semantics  and various other things get put up in Fabricator for review.  And all non-privilege changes require a review by somebody else.  So I may want to change the fast file system.  And I can't just say, well, I've done it for 30 years.  Just trust me on this one.  I actually have to get someone else to look at it.  And it turns out to be a good idea because sometimes even if you think you know what  you're doing, you don't, in fact.  And having someone point this out to you before you've broken things is generally better than  afterwards.  So the, and again, a lot of this depends on people working together.  I mean, you have to essentially, you know, at least put feelers out and know some other  people that can do reviews of things you're working on.  You have to actively go out and review things for other people.  And so meetings like this are ideal because you actually get everybody in a room and you  actually meet a bunch of new people and you find out about things that they're doing.  And that will then go on and help integrate making some of these other things that need  to happen, happen.  Okay.  So FreeBSD has commit tags.  And the commit tags include a bunch of information, which other project member has reviewed the  changes.  Now, this is an important one because it forces you to put the name of somebody on which you  can add at FreeBSD.org.  And if something really stupid goes in there, you can say to that other person, you know,  how did you possibly pass that in review?  And if they say, they just made that up, you're in deep yogurt.  So you really do have to get the reviews and you really do have to listen to it because  that other person, you know, if you've been asked to review, you can't just sort of say,  oh, yeah, they know what they're doing and say yes, because if it blows up, you're going  to be almost as much to blame as the person who put it in there in the first place.  And this is especially true if you are mentoring somebody else because you're supposed to make  sure that they don't goof up.  And so if they goof up, that makes you look like a really bad mentor.  Many things do come in from bug reports.  So if there's a related bug report, you tag that.  And that will also put a message back to whoever submitted the bug report and anyone that's  following that bug report so that they know that things have happened.  Things that get discussed on Fabricator, again, are noted so that the linkage goes back to  Fabricator.  And then if the report, if the bug is something that should be pushed down into the stable  levels, you note when that's supposed to happen.  And one of the more interesting and more recent things that's gotten added is a sponsoring  organization.  So if you work for a company and you're, you know, let's say, pick a random example, Netflix,  and so there's something that you've done there but you're pushing up to the master  tree.  You put that in so that your organization gets some credit for the fact that they're  paying for someone who's actually doing useful stuff in free BSD.  And my favorite one is at the end, sometimes what you're doing is you're checking in because  you did some particularly stupid thing earlier.  And then you get, it's the pointy hat tag.  And you put that in if it's basically a maya culpa fixed to an earlier thing that you  checked in.  Maybe with a note of which one that was.  All right.  So now here, now we start to the hard stuff.  How do you work and play well together?  All right.  So the project started out with behavior guidelines based around commit behavior and rules for  suspending commit privileges if you misbehave.  So, you know, don't get in commit wars.  Like, one person put in a change and then the other person go, no, that's not right  and put in their change.  And then the first person switch it back to what they had before.  And this hasn't actually happened too horribly in recent years but it used to be a not infrequent  enough occurrence that there actually got to be some fairly detailed rules.  Inevitably, you know, Cora has to step in at some point and say, you know, knock a couple  heads together, you know, get the two to agree on what it ought to be and put that in.  And if they're too busy counter committing against each other, then Cora can pull the  plug and just say, okay, the two of you are, you know, in the corner for a week.  Neither of you can commit for a week until, you know, everything calms down.  And then after a week, if that's not enough, then it can go to two weeks, four weeks, six  weeks, et cetera.  And sometimes it just gets to the point where a person has to leave the project.  That's only happened two or three times in the 25 years that the project has been around.  But sometimes there's just such fundamental differences of opinion that that's the only  solution.  And obviously you don't want to do that but if that's all you can do, that's what happens.  Okay, so having Cora kind of act as the adult in the room worked very well for a long time  until Gamergate blew up.  And when Gamergate blew up, that just, you know, it was no longer in the FreeBSD project,  it was all over the web.  And it became clear that what we had as rules which had worked very well for a long time  were just not up to the task of that situation.  So it was decided that we needed a more formal code of conduct which was drawn up by a well-meaning  but inexperienced set of folk.  That is people that hadn't really done a code of conduct before, they just kind of looked  around the web and pulled some things together and put it here.  Here it is.  And the feedback was this is insufficient.  It just, it looks like it was written by amateurs that don't understand how to do this kind  of thing, which it was.  So okay, fine.  We're going to do it right this time.  So it was replaced by a code of conduct drawn up by folks with more experience.  But the problem here was that it was overbearing.  And it was a code of conduct that might work well in certain situations, particularly corporate  situations, but it really wasn't tailored to what free bestie needed or wants.  So a third revision of the code of conduct I am told is in the works.  Jeff Roberson has collected a bunch of information and it's pretty much geared up to wait for  the next core to come in rather than the existing core which has been involved in the first  two of these.  So that we can get some new eyeballs on it and hopefully get some more feedback from  all of everybody in this room and elsewhere in the project.  There are people that would just like to go back to the old days and we already have proof  that that doesn't work.  So we need to do something here with a code of conduct and hopefully the third time will  be the charm and we will have something that people are both happy with and will bring  us into the modern world of what we need.  So we'll see where that goes.  Free BSD Committer Turnover.  I've already talked about how we had to get rid of dead wood in core.  Well that's not just core, that's the issue.  We have the same issue with committers as well.  People get excited, they work for a while and then they sort of fall off the, you know,  and move on to other things and you have to have some way of eliminating them because  otherwise you just end up with too much dead wood.  And you have to have a good way of doing that in a way in which it's not perceived as the,  you know, people in charge picking on individuals.  And so there, it has to be essentially an algorithmic way of deciding that you are dead  wood and it's time for you to move along.  And it has to, everyone has to understand what the algorithm is.  The algorithm is pretty easy.  It's you drop commit privilege after one year of inactivity.  And so if you just don't do anything and this is like a microscopic, all you got to do is  go once a year and change your personal information in your, you know, committer description and  that does it.  Okay.  And it's not like a lot of effort to remain a committer.  And I did notice a large number of those commits happening recently so that people would be  eligible to vote for core.  Good on you.  Okay.  And in fact, if you haven't made an entry in the calendar file yet, please do that.  That'll count.  All right.  So for up to about two years and this is not quite as hard and fast because core sign  gets to decide but after the one year where it's dropped off, if you won't come back,  you know, after 18, 20 months or so, core will generally say, okay, fine, you know,  just turn it back on again.  But after it's been more than about two years, you actually have to go through the whole  getting on board process again.  And a lot of that is because the way the project does things just changes over time.  You know, we change from NETs to Bugzilla, we've added fabricator, you know, there's  just all kinds of things that have just been done in different ways.  And so you can't just come back and just jump into it doing things the way they did were  done five years ago because people will get upset reasonably.  So what ends up happening is that you still get a mentor just as you would if you were  a brand new committer but typically you don't have to be, you know, a mentee for a year.  You can often just in three to six months be up to speed again and, you know, your mentor  will say, okay, you got it, go.  Which means they're off the hook for you doing stupid things.  All right, so the FreeBSD project has done a very good job of, you know, getting rid  of the deadwood but of course the other half of that is if you just get rid of people then  pretty soon you don't have anybody left.  And so recruitment turns out to be just as important if not more important than getting  rid of the deadwood.  Okay, so you've got to bring in new developers and there's quite a few places that this can  be done.  A lot is contact via university courses.  There's this whole website that several of us have been working on called TeachBSD which  is setting up curricula that can be taught in universities and going out and actively  doing stuff in universities.  Robert Watson and George Neville-Neal in particular have been doing a lot of stuff at Cambridge.  Another key place to get them is people working in a company that's using FreeBSD.  And I mean, if the company is using FreeBSD they are almost certainly doing some level  of development and almost certainly has things that need to be upstreamed and so it's both  to the company's advantage and to the project's advantage to get those people brought on board  as committers so that instead of having to nag somebody to get things put in they can  just do it themselves.  And this is of late proven to be a very good channel for bringing people in.  Now you say why would someone want to become a committer and just have extra work to do  after you've done your 50 hours of work at the company.  Now you're going to do another 10 hours for FreeBSD.  But it turns out that if you're a committer to FreeBSD it actually raises your value.  So if you change jobs it tends to give you a salary bump when you go to another company  that's using FreeBSD because you're already a committer and this is viewed as a very useful  asset to have.  So in fact we've had very good uptake on bringing in people that are working for companies  and become FreeBSD committers.  Another key place that has brought in a lot of new committers over the years has been the  Google Summer of Code project.  FreeBSD has typically been given 10 to 15 slots in Google Summer of Code and for those  of you that have pay any attention to Google Summer of Code most projects are thrilled to  get two or three spaces and we are routinely getting 10 to 15 and we could have more if  we could get more mentors and more projects in line.  So you know there's a lot of there's a small set of people that work very hard on Google  Summer of Code and it would be great if we could get some more people involved in that.  And then finally there's just discovering the project at a conference or through social  media and particularly EuroBSD which moves around to different countries every year.  We get a large influx of people that have never heard of BSD and all the projects and  not just FreeBSD but NetBSD and OpenBSD tend to get a lot of excitement and new people  that come on and start working with the project.  It's key that your project be welcoming and easily entered.  You come here and it's like oh yeah I know everybody blah blah blah.  Well if you're new to the project you don't know anybody here hardly other than maybe  their name and so especially those of you that are older timers try and make a point  of talking to the new people and welcoming here and making it sound like it's a fun place  to be and not a terrorizing place to be.  So provide an easy path to connect with the project and provide mentors to help get people  up to speed.  Again mentoring is not for everybody.  There's some people that come up and say we want to be a mentor and it's like we have  some other things that you could help us with.  But most of the people here could be good mentors and you know well how do I become  a mentor.  Well talk to us.  Anyone from CORE say you want to be a mentor you'll be their best friend for a day.  They might even buy you a beer.  So what is the net effect of all this.  Well I've been collecting statistics over the years.  In fact this is something I cranked this script actually an hour ago and updated my slide  because one of our new committers is 21 and previously it was 23 I guess was the youngest  committer that we had.  So I was very happy to be able to pull this slide out.  I didn't knock the median or average down but nevertheless it helped push it that way.  At the other end here I'm unfortunately three from the top here.  Diane Bruce has got me beat she's here and Greg Lee he was not here but he's out here  at the end.  If you just take off the over 60 crowd here the median and average both dropped down to  around 38 39.  So you know just get rid of the 10 old geezers and your graph will look better.  At any rate the key thing to note here is first of all this big huge spike which is late 30s  early 40s.  And the project has stayed right in that number range for more or less a decade.  When it was very when it first started up it was down in the low 30s and it has slowly  risen it slowly rose into the high 30s and it's pretty much been there for at least the  last eight to ten years.  So we are doing a good job of bringing people in and that's good.  If a project is going to have a long lifetime you've got to keep this age demographic you  don't want it to age out.  And in particular if you look at the Linux project Linus Torvalds and his lieutenants  are pushing out on that curve and they are not doing a good job of bringing in younger  people.  They are bringing in a lot of people in companies so it's not like that project is going to  go away but the sort of image of what Linux ought to be is going to become a corporate  one if they don't somehow work to replace that top set of people.  I'd iterate some of the other numbers are kind of interesting here.  The calendar.previousd is where I get the ages so there's 46 of you who won't ante  up what your age is.  I actually sent out an email middle of last year to everyone that I didn't have an age  for and said either put yourself in the calendar or tell me what it is and I'll put it in my  secret one that I won't tell anybody else what it is.  And it's about time to do that again because I had it down to around 25 at one point.  And someone was saying oh yeah I saw a big flurry of changes to calendar.previousd.  What was that all about like me?  An email.  So anyway if you're not in the calendar or you haven't already sent the information  to me I'd appreciate it especially if you're younger.  Bring that age demographic down.  Great.  So the previous development model some of it works and some of it not so much.  What we do great at is small changes especially ones that are easy to identify, reach some  consensus on and implement.  And when I say small changes I mean changes that can be done by one or two or three you  know individuals working together.  And we have numerous examples of things and you know it's not like that small of a change  and we got SCTP brought into the system but that was basically Randall and a couple of  his cronies that did that and they didn't need like you know everybody in the project  to be involved in order to make that work.  It's the larger wide ranging changes that are hard.  So some examples switching from CVS to Subversion took nine years.  And the problem was that we would start with the discussion and everyone would say oh well  there's this and there's this and there's this and have you considered this and have  you considered that.  And blah blah blah blah blah blah blah blah blah and there would be no consolidation on  one particular thing and it would die out without resolution.  And 18 months to two years later it'd be like oh we need to do this and we'd forget the  entire discussion we'd previously had and we'd have the same discussion all over again  and the mailing list would be flooded with stuff and blah blah blah it would die out.  And it was finally accomplished by Peter Wemm who just said I'm sick and tired of this I'm  just going to do it.  And he just did it and one day it was click and now we're under Subversion.  This is not an ideal way of handling this.  Okay.  The other huge change came when we went from the single threaded to the symmetric multi-processing  which was between FreeBus D4 and 5.  And this required changes obviously throughout the entire system.  And you sort of have this theory that well we'll just put a giant lock in and then we  will just slowly unlock things bit by bit.  And you know forget PID that's pretty easy to do.  But when it gets to things like the networking stack where you've got locking well let's  see we've got packets going down so we're going to lock from top to bottom.  Oh and we have packets coming up so we're going to lock from bottom to top and blah blah  deadlock.  So it's not that easy.  So we started just to make it interesting with a big debate as to what kind of locking  we were going to do.  The Solaris style locking or the Amigo style lockless stuff.  And that debate raged for a while.  And eventually it got decided that we were going to use the Solaris style of locking.  And the lockless people weren't willing to give up and kept trying to put that in and  that's when there was an eviction of a member of the team.  But you know once we got that sorted out then it still required this just massive coordination  and for any of you that remember it was a long gap between four and five coming out.  And five came out kind of because we had to it had been so long and five left a bit to  be desired in terms of stability and so on.  And it was really not until six came out that we really had sort of gotten over the hump.  But you know that both of these are examples where the model of just sort of making changes  and everyone sort of coordinating on mailing lists just doesn't cut it.  So how do we do better on those things?  So we recently as in about a year ago, CORE created the FreeBisD community process which  is modeled on the Python enhancement process.  And the idea is we want to get away from this, have a huge argument and then do nothing and  forget about it and have the same huge argument again and do nothing about it and so on.  And just for an example of something that's on the horizon that we're going to have to  deal with is deprecating I386.  Okay.  We've already had, you know, one round of discussion on this.  It's clearly not time for it yet but the time is going to come.  And we really don't need to keep every, you know, 18 months having a big discussion with  the same arguments over and over again.  So this is a classic case where this will hopefully help us do this in a much more rational  way.  So the major changes are described in the document.  Well, the major change is pretty easy.  We drop I386 support.  But then there's, you know, here's the reasons why, here's the reasons why not, here's,  you know, reasons, the why not reasons may die down over time, here's some of the ways  that we can determine that they're dying down.  And so we have that whole, you know, distillation, let's say, of the discussion that we've just  recently had on the mailing list and put that into one of these documents.  And this gets reviewed by an editor whose job it is to, you know, edit, make sure all  the opinions are there and, you know, so on.  And then it gets published for discussion.  And as the discussion continues on, those comments get put in there.  And when the core team feels that the discussion has kind of died down, then they make a vote  on deciding, you know, whether we're going to move forward with that particular thing  or not.  Now, in the case of I386, they will almost certainly decide, no, we're not going to do  it at this time.  But this document is there.  We've had all these discussions, so 18 months from now, instead of starting from ground  zero, we can restart this document, which we already have a bunch of things in.  We say, all right, what's changed?  All right, well, some of the reasons for keeping it now have gone away.  So we'll move them from this pile to that pile.  But then there's some other reasons that have come back in.  And so, again, we'll have, you know, sort of, the discussion will move forward.  We will probably still decide two years from now that we're not ready to get rid of I386.  But, you know, eventually, we will come around and the document will get to the point where  the reasons to keep it will be dominated by the reasons to get rid of it and it will  happen.  And it will happen with, let's say, one-tenth of the amount of mail on the mailing list  as we would have if we just keep with the current scheme of arguing about it every 18  months.  So this hasn't really gotten a good test yet, but I think we've got some ideal candidates  that should go through this process.  Okay, another issue that has arisen of late is the core team interaction with the free  BSD committers.  In particular, there were some that felt there wasn't enough communication about the code  of conduct, can you imagine?  And so there's been a push to try and get core to be a little less opaque.  Historically, all the core communications were private among core and a monthly report  would be thrown over the transom once a month saying, oh, this is what we did.  These people got commit bits and we had discussions about these topics and we made these conclusions,  these decisions.  Period.  That was it.  So there was a feeling that there wasn't a good way of giving feedback to core.  All you got to do is say, well, I don't like that decision.  It's like, well, tough, it's been made.  Wait for next month.  So in recent time, core has started releasing an agenda of the discussion topics so that  you can see what they're going to be talking about.  And if there's something that is of interest to you, you can provide some feedback before  they've had the discussion.  Now this is a nice idea, but the problem is that they have, I mean, when you look at the  thing, it's this laundry list of stuff of which they don't actually end up doing all  of those things in that meeting.  They just want to pick through a few of them.  And so it would be nice to know, well, what are the things you're going to actually talk  about in this upcoming meeting as opposed to, well, someday you're going to talk about  that maybe.  So a little bit of work needs to be done there.  There's also contemplating allowing committers to join the video conference, not being able  to talk, but being able to at least listen and perhaps poke in questions through IRC  or something.  Now there's certain things where they have to deal with personnel issues.  And obviously that is just going to have to be a closed session.  But just like when you have a board of directors, most of the meeting is open and then there's  a small section that you have where it's just the board.  But again, that's a relatively tiny piece of what they have to deal with most of the  time.  So there's a hope, and this is sort of my throwing it out to whoever the incoming core  cuts to be, that this is a nice start, but keep pushing this forward.  OK, finally, the FreeBSD security team.  Security has been a lot in the news lately.  And on average, the project has done well in this domain, but unfortunately it's a job  that's really hard for volunteers in particular because it's mostly nothing to do until there's  a fire drill in which they need 110% of your time.  And you might have another job, and your employer doesn't want to just give you a month off  with 30 seconds of notice.  And so the people that have done it have done an excellent job, but they get burned out.  And so we'd had a series of very good security officers, but we'd gotten to a point where  we needed someone who could do it, especially with a lot of things that have been coming  down more recently that was becoming much more of a job than it had been.  And so it seemed like a place where the FreeBSD Foundation could come in and at least provide  the person who's going to do the sort of management piece of it.  So not necessarily to figure out the technical part because we have a lot of members in our  community that can help with the technical issues, but can at least be there and deal  with the immediate responses that need to go out and coordinate, find the correct people  to deal with the problem and get them pulled in and working on it, et cetera.  So in fact, as far back as 2002, we expanded it to a security team where we had the security  officer who was doing the kind of management bit, and then the security officer, as I say,  oversees the pool, people getting that done.  And it used to be a part-time person, and now it's become almost a full-time job, at  least the last couple years.  And so that's where the FreeBSD Foundation came in to help provide that person.  It doesn't mean that everybody's off the hook, no means, but it does mean that there's going  to be somebody that when the reporter calls up and says, what is FreeBSD doing, they at  least get someone to answer the phone.  All right.  So the grand summary, the evolution of governance model for leadership, this has been something  that has been of particular interest to me because having been one of the benevolent  dictators for life, that was clearly not a model for something where it was going to  outlast me.  And so moving into having a core team for life was sort of the next logical step.  Moving to an elected core team seems to have worked well, and the project continues to  move forward and grow, which is great.  If I get hit by a bus, the FreeBSD project isn't going to notice the difference.  Four major changes in leadership, each allowed the project to move forward and tackle new  problems.  You get people that come in and the early getting the BSD out in the first place, getting it  open sourced, took a few years off of several of our lives.  It has certainly helped the project avoid the aging out.  The median age has remained in the mid to high 30s, at least over the last decade.  And the elected core, I mean, there's all kinds of things where that can go wrong.  You can end up electing people that just aren't, you know, core is much more of a management  position than a technical position.  I mean, they obviously have to have technical chops, but a lot of what they do day to day  management is interacting with people.  And so the project as a whole seems to be pretty good at bringing up people that have  those skill sets.  And you want some turnover, you don't want it to just be the same group of people forever  and ever, but you also don't want to turn everybody all at once unless they've gone  really off the rails.  And so, you know, again, bad things could happen, but so far they haven't, and hopefully  it will stay that way.  The FreeBSD Foundation has provided not needed resources and stability.  They are somewhat the long-term memory of the organization.  They deal with the long-term issues like making sure that the copyrights don't get abused  and, you know, all those kinds of things that, you know, just you need somebody that's going  to just do that more or less continuously.  So they have the solitude of longevity that's needed for that.  Okay, so what's my conclusion?  My conclusion is the governance is mundane.  I'm amazed to have this many people sitting here listening to me talk about it for an  entire hour.  But nevertheless, too much of it is stifling.  You can certainly overdo it.  I have been in organizations that all you want to do is run out the door to get a breath  of air.  But you can also go to the other extreme and say, oh, you know, it's all laissez-faire,  blah, blah, blah.  Too little in the project goes off the rails, collapses from lack of infrastructure.  There's all kinds of bad things that can happen if you're not paying some attention to this  stuff.  So it's a fine line.  People say, oh, you can't have enough.  Oh, yes, you can.  Oh, you can't have too little.  Oh, yes, you can.  So we seem to be doing pretty well here, and hopefully we will stay on that track.  The thing to really note is that governments requires constant tuning to keep that rate  balance.  You know, you ebb and flow.  You tend to bounce back and forth between too much and too little.  And you do have to pay attention to it.  But when it's working, it should just be in the background, and you shouldn't really be  all that aware of it.  OK, so with that, I will take any questions.  Yes?  What is core doing about it?  That's an excellent question, and that's why I'm not running for core.  You couldn't pay me.  What was it?  If nominated, I will not run, and if elected, I will not serve.  I did my 10 years.  Someone else has a problem.  OK, anything else?  Yeah?  In 2018, do you see core as more of a technical body or a board of directors?  Do I see core more as a technical body or a board of directors?  Yes, is the answer to that.  They are, in fact, doing both of those things.  I mean, ultimately, there are some technical decisions that need to be made where there's  disagreement, and hopefully they can help those people come to a resolution, but they  have to be technical enough to be able to do that.  They also are a board of directors.  I mean, they are overseeing all those different groups of people.  They need to make sure that those groups are working properly, and if they're broken to  figure out how to fix them.  That's management through and through.  That's why I'm very pleased when I looked at the list of people running for core that  we have people that are across that entire spectrum.  I think I'm convinced that the core will have the various assets that they need.  People, oh, I can't run for core.  I'm not a kernel hacker.  That is malarkey.  Yeah?  How long have you been using video chat for months to meet?  How long have I been using?  Video chat for months to meetings of what tools?  You would have to ask the core people that.  I haven't actually been paying attention to what they're using.  Can you repeat that?  Three.  They use Google Hangouts?  Three.  What?  We don't say hangout for three years.  Okay, thank you, Ben-O.  See, you have to ask someone on core.  They'll have the answer.  All right, anything?  Yeah?  Generally speaking, we provide some things to core, the pay for when they were using  conference phoning.  In the case of code of conduct, they came and said they wanted some money to get people  to help them with it when they did Rev2.  Generally speaking, now with my foundation hat on, we're very sensitive to not somehow  being viewed as taking over the project.  Now, I will point out that some of core is actually also foundation board members.  So it's a fine line that we're walking there, but our goal is that the project runs and  we are there as a support service.  And when they want some help, they'll come and ask, and we will try and provide it.  But a lot of we're doing things that are sort of on the side.  So trying to go out and raise awareness and provide travel grants.  A bunch of you here are being at least partially paid for your travel to come here.  And Google Summer of Code related things and so on.  But the goal of the foundation is to just be sort of quietly on the side and that the  project and core should be driving what's going on in the project itself.  Okay?  All right.  Thank you.
Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_0_107.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_108_251.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_252_311.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_312_1223.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_1224_3707.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_3708_4595.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_4596_5051.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_5052_5171.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_5172_5303.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_5304_5447.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_5448_5567.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_5568_6167.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_6168_6635.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_6636_7571.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_7572_10823.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_10824_11183.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_11184_11315.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_11316_14615.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_14616_18383.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_18384_18515.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_18516_20123.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_20124_20255.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_20256_20963.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_20964_21323.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_21324_21419.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_21420_21611.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_21612_25163.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_25164_26807.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_26808_26927.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_26928_28919.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_28920_31415.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_31416_35363.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_35364_41675.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_41676_42947.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_42948_43079.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_43080_45743.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_45744_45959.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_45960_46007.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_46008_46343.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_46344_52847.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_52848_57215.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_57216_62867.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_62868_67595.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_67596_70703.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_70704_71339.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_71340_74843.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_74844_78743.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_78744_80711.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_80712_87287.png Kirk_McKusick__The_Evolution_of_FreeBSD_Governance_--_BSDCan_2018_87288_87475.png
Okay, I think I'm going to try and get started here, because if I get started on time, I can end on time. And I know many of you are here because you actually want to be here for the auction and have a good seat for the auction. But hopefully I can give you a little entertainment in the meantime. So the evolution of free BSD governance, if you've ever wondered what happens when you get too old to code, then you get to talk about things like governance instead. And because I wanted to have this talk cover at least something a little beyond free BSD, I decided to start it out by talking about this booklet that I received. It's the promise of Berkeley for alumni, parents, and friends who shape Cal's future, which is to say they contribute money to the university. At any rate, this year is the 150th year of the University of California, which I realized for those of you from Europe is very young. But by the standards of the United States, it's getting up there. I mean, it's not Harvard or one of those places. But anyway, there's a section called Game Changer Discoveries. And they list six discoveries that have been made at Berkeley that had worldwide impact. So it includes things like in 1980 when they discovered that it was an asteroid that struck the Earth 65 million years ago that triggered the mass extinction of the dinosaurs. Or in 2012, when Jennifer D'Anna came up with the CRISPR, the thing for splicing and cutting up DNA. And in 1942, when Glenn Seaborg discovered radioactive plutonium, one of 16 chemical elements including beryllium and californium that added to the periodic table. In 1998, Sal Herpmuter with the acceleration expansion of the universe in 1939, when Ernest Lawrence became Berkeley's first Nobelist with things like the cyclotron. And number six, in 1977, Bob Fabry and his student Bill Joy created Berkeley's software distribution blah, blah, blah. So anyway, I'll pass a copy of it around in case you want to get all the details. Pass it around. Okay, so BSD is a big whoop, at least as far as the University of California is concerned. Never mind the fact that they run Linux from top to bottom and you try and find a student that knows anything about BSD and they look at you rather strangely. Is that like LSD? No, no, slightly different. All right, so for those from open BSD and that BSD, that's all you're going to hear about your two operating systems for the rest of this lecture. Sorry about that. Okay, so let me just start off with a background and introduction. Where did all this come from? The Berkeley software distribution, BSD, started in 1977 as a project of Bill Joy at the University of California. And I got involved in 1977 because I was one of Bill Joy's office mates and it was impossible to be around Bill Joy and not get sucked in. It's like, oh, could you just do this one little thing? You know, it won't take but a few hours and two weeks later, blah, blah, blah. Anyway, that originally was just a distribution of utilities, VI, and the Pascal compiler, C-shell and other things like that. But it became a full distribution with the release of three BSD for the VAX in 1979. So that was the first release of BSD that included the operating system and the utilities around it. Initially that was a port of the 32V, which was the version 7 UNIX ported to the VAX but with virtual memory added to it. That's the main addition that was in BSD. It was just a swap-based system before that, 32V. I had the utilities that came with 32V, so all the usual CAT and LS and so on and so forth, plus all of the utilities that Bill had been hacking on or pulling in. People like to think of Bill as a great software developer, but in fact, he was even better at marketing, so he would go out and talk up all this stuff and other people would say, oh, I've got this or that, and he'd bring it in and pee on it to make it smell like Berkeley and add it to the distribution. At any rate, we moved to using source code control in 1980, a very early source code control system called SCCS came out of the labs. Initially, it just had the kernel on it under it, but fairly quickly, we got the rest of the system under there. In fact, the SCCS itself got converted over to an SVN repository by Peter Wem, I believe it was, and no, it was John Baldwin did that. He has put it up. If you actually want to go back and find out who to blame for various and sundry things, you can do that. Anyway, about a decade of releases were managed by the computer systems research group, a four-person development team that, of course, started out as Bill, and then Mike Carls, who's here today, and myself, and Keith Bostick and Keith Sclauer, then continued pushing that out the door with the help of various and sundry other people that were peripherally involved with us there. And finally, at the end of the 1980s, actually, in 1991, we did a release, which we called Networking Release 2, which was more or less all of the system. We did a Net1, which was just the TCP IP stack in the associated networking utilities, and we decided we wanted to expand that a bit, and we didn't really want to have to go through all the rigmarole of our release with the lawyers. So we said, well, this is just a revision of Net1. Never mind that it contains 10 times as much software. At any rate, that landed a lawsuit against the university, which finally got resolved in the university's favor in 1994 when we did a release of 4.4 BSD Lite, which was Net2 minus about six files to make the lawyers happy, and, of course, the addition of everything else that had happened in the meantime. So that 4.4 BSD Lite, well, Net2 was what started out being the first release from, open source release from Berkeley, and Bill Joelitz took that, added a few missing files to get it as a bootable system for the 386 architecture, and was releasing that. And then, because he was sort of slow moving two groups, NetBSD Group and the FreeBSD Group, both formed patch kits that came out of that, and so those two projects then evolved out of there. With the settling of the lawsuit, one of the requirements was that you had to switch over to using 4.4 BSD Lite as your base. I think that perhaps the people that pushed that requirement thought that that somehow was going to stop FreeBSD and NetBSD in their tracks, but it did not have that effect, thankfully, and so away they went. Okay, but that was the last release from the university. They switched, since SCCS was proprietary, they switched to the CVS source code control system, which NetBSD continues to use to this day. I'll talk more about that later in terms of FreeBSD. They also created a core team, core team that had lifetime terms created to decide who should be allowed to commit, and the initial distribution actually got done by Walnut Creek CD-ROM. Jordan Harbert, who was one of that original core team, ended up being hired by Walnut Creek CD-ROM along with Rodney Grimes, who's around here somewhere as well. Walnut Creek basically provided the place to do the builds and to get the distribution going and so on. So that was sort of the first home base, if you will, for FreeBSD. One of the problems was that stuff just kept getting accreted and accreted, and the distribution kept getting larger and larger. And so fairly early on, they realized, all right, we're going to switch to having a base system and then ports. And if we sort of keep the base system as a manageable thing, we can then let ports sort of grow and not have to worry about the whole distribution getting too big. And then NETS also got brought up to manage the inevitable bug reports. They were just as well managed then as they are today. Well, it's just to say, it needs work. FreeBSD continued to move forward and started being picked up by commercial organizations. In particular, Yahoo was one of the very early, very big users of FreeBSD, which ran pretty much their entire organization on it. And as the demands for what we needed for machines to maintain all of the state of the project grew, it moved from the couple of machines at Walnut Creek CD-ROM to Yahoo, who had not only the machine resources and the 24-hour management of the machine rooms and particularly network capacity, which was needed to do the ever-growing distributions. And this was great. Having a company that's willing to basically run your infrastructure at no cost to the project won't cost the project had no money, so there wasn't a lot of choice there. But Justin Gibbs decides that this is not a good long-term strategy because companies might change directions or fade from the scene, and then we would be up the creek. And so he started the FreeBSD Foundation with the concept that it could bring in enough money to provide for the FreeBSD infrastructure. Well, that took about 10 years before it actually had enough income, reliable income, that it could start to do that. And luckily, Yahoo was perfectly happy to continue supporting the project over that period of time. So it's only been in about the last five to seven years that the FreeBSD Foundation has slowly taken over, the infrastructure management moved it out of Yahoo and a couple other places where it was scattered around and is now able to fully handle that. All right, so meanwhile, how about the project itself? Well, as you'll see as a sort of recurring theme, one of the problems you have with open-source projects is that people get all excited and they do a few things, and then they kind of get less excited and other things start picking up in their life. And so you end up with Deadwood. And if you get too much Deadwood, then the whole project sinks just by virtue of there being so much inertia and inability to get things done. And the core team had grown to about 20 people because once you were on, you were on for life. You didn't want to leave because then you'd lose the prestige of being on core. So this core had gotten up to 20 people and only about, let's conservatively say maybe half of which were doing anything and about a quarter of whom were really actively doing things. And so it became evident that we needed to do something about this. And so a group of previous D folks got together and decided that they would create a set of bylaws. And part of setting up the bylaws would be that we would have elected core. So we'd take the rules that you know today pretty much. You have the anyone that's a committer of any sort is allowed to run and all the committer get to vote and the top nine vote debtors become core for the next two years. Well, of course, in order to get the bylaws put in place, the existing core team had to vote for it. And if you're about to put bylaws in which say you need to be elected and there's only going to be nine of you, then that means by definition half of you are not going to be on core. And so they're not incented to put these bylaws in place. And so instead what we had to do in order to make this work was to grandfather the existing core and just hope they didn't actually read the bylaws which most of them didn't and put them in place. And then and only then two years later when they were asked if they wanted to run and said, oh, but I'm grandfathered, you said, didn't you read them? You were grandfathered for the first term. There's a couple of them that still will not speak to me, but. All right, so now we have core and they're elected and this is, you know, it's not a great system, but it's the best system that seems to be generally available. And it's, I mean, we went from, you know, Grand Unnipotent High Stomper, the Linus Torvold's model, which was Bill Joy and Mike Carlson myself. Then we went to the sort of semi-Grand Unnipotent High Stamper, which was the core for life. And now we've moved into where we can routinely change out the core team. And you obviously don't want everyone to change all in one election because you would lose too much institutional memory. But in fact, it seems to sort of move forward, you know, about half of it turns over with each election or slightly less than that, which seems to be good. We've had sort of, I would sort of call it three leadership groups if you count the original core and then we've had at least two leadership groups in the elected core since. So we actually are doing a good job of moving forward and letting people that have gotten burned out move along. All right. So what do the core teams do? I assume most of you know this, but just for reference, we have, they oversee the port manager team who oversees the ports committers. We have the documentation team, which oversees, I was impressed to say, 126 dock committers. If you, you know, look at almost any other project, if they even have dock committers, I bet that it's, they're lucky to break a two digit number. We have the security officer who handles security issues, alerts and updates. The administration, system administration team that oversees the infrastructure, the release engineering team that oversees releases and the QA team, which we're just now really getting going that deals with continuous integration builds and expands the regression tests and so on. And you know, core nominally is responsible for all of these things, but they break it out and there's really teams that are doing these things. And so core mostly gets involved when things go off the rails. Okay. So the FreeBSD project today, initially, believe it or not, there was the FreeBSD mailing list. You can imagine how well that worked. It, of course, got broken up into a lot of area specific lists like architecture and networking and file systems and blah, blah, blah, blah. But the problem is there got to be so many lists that if you really wanted to know what's going on, you had to watch too many lists. And so it became necessary to have some way of getting cross area collaboration. And initially it was done sort of through the bug tracking system. Nats later got upgraded to Bugzilla. The project moved to Fabricator in 2014, primarily so that we could expand the discussion to non-committers. It was very difficult for people that were doing development but not committers to really interact. And so Fabricator made that possible. And you know, there's been a long list of complaints about Fabricator, but it's a step forward from where we were before. So we'll leave it at that. Chorus code control. Well, the project started on CVS. And there was a huge amount of debate about moving to something else. And NetBST has been struggling with this of late as well. And so eventually in 2000, and I'll talk a little bit more about that later, we did switch to subversion. And now there's been ongoing debate about switching to Git with all of the same arguments that we had before. The arguments in favor, there's a lot of programmers that no one used Git. There's for several years, there's been a everything that gets checked in immediately gets pushed down to from subversion to GitHub. But the problem is upstreaming from GitHub requires going through Fabricator. So you've got to pull a change request up to Fabricator and then go through that and then get that pulled into the main system. And that there's too much friction there. And so being able to just do pull requests directly would be helpful. On the other hand, it wouldn't be on GitHub because the project just sort of in its DNA wants to control that repository. And so there's a lot that would have to happen to be able to get to Git. We'll see where we go. OK. So the workflow, committers are up to three groups. We have documentation, which is all the previous documentation, ports, which are all the ports, and source, which is the base system. And the key thing is the changes that can affect a kernel interface or operational semantics and various other things get put up in Fabricator for review. And all non-privilege changes require a review by somebody else. So I may want to change the fast file system. And I can't just say, well, I've done it for 30 years. Just trust me on this one. I actually have to get someone else to look at it. And it turns out to be a good idea because sometimes even if you think you know what you're doing, you don't, in fact. And having someone point this out to you before you've broken things is generally better than afterwards. So the, and again, a lot of this depends on people working together. I mean, you have to essentially, you know, at least put feelers out and know some other people that can do reviews of things you're working on. You have to actively go out and review things for other people. And so meetings like this are ideal because you actually get everybody in a room and you actually meet a bunch of new people and you find out about things that they're doing. And that will then go on and help integrate making some of these other things that need to happen, happen. Okay. So FreeBSD has commit tags. And the commit tags include a bunch of information, which other project member has reviewed the changes. Now, this is an important one because it forces you to put the name of somebody on which you can add at FreeBSD.org. And if something really stupid goes in there, you can say to that other person, you know, how did you possibly pass that in review? And if they say, they just made that up, you're in deep yogurt. So you really do have to get the reviews and you really do have to listen to it because that other person, you know, if you've been asked to review, you can't just sort of say, oh, yeah, they know what they're doing and say yes, because if it blows up, you're going to be almost as much to blame as the person who put it in there in the first place. And this is especially true if you are mentoring somebody else because you're supposed to make sure that they don't goof up. And so if they goof up, that makes you look like a really bad mentor. Many things do come in from bug reports. So if there's a related bug report, you tag that. And that will also put a message back to whoever submitted the bug report and anyone that's following that bug report so that they know that things have happened. Things that get discussed on Fabricator, again, are noted so that the linkage goes back to Fabricator. And then if the report, if the bug is something that should be pushed down into the stable levels, you note when that's supposed to happen. And one of the more interesting and more recent things that's gotten added is a sponsoring organization. So if you work for a company and you're, you know, let's say, pick a random example, Netflix, and so there's something that you've done there but you're pushing up to the master tree. You put that in so that your organization gets some credit for the fact that they're paying for someone who's actually doing useful stuff in free BSD. And my favorite one is at the end, sometimes what you're doing is you're checking in because you did some particularly stupid thing earlier. And then you get, it's the pointy hat tag. And you put that in if it's basically a maya culpa fixed to an earlier thing that you checked in. Maybe with a note of which one that was. All right. So now here, now we start to the hard stuff. How do you work and play well together? All right. So the project started out with behavior guidelines based around commit behavior and rules for suspending commit privileges if you misbehave. So, you know, don't get in commit wars. Like, one person put in a change and then the other person go, no, that's not right and put in their change. And then the first person switch it back to what they had before. And this hasn't actually happened too horribly in recent years but it used to be a not infrequent enough occurrence that there actually got to be some fairly detailed rules. Inevitably, you know, Cora has to step in at some point and say, you know, knock a couple heads together, you know, get the two to agree on what it ought to be and put that in. And if they're too busy counter committing against each other, then Cora can pull the plug and just say, okay, the two of you are, you know, in the corner for a week. Neither of you can commit for a week until, you know, everything calms down. And then after a week, if that's not enough, then it can go to two weeks, four weeks, six weeks, et cetera. And sometimes it just gets to the point where a person has to leave the project. That's only happened two or three times in the 25 years that the project has been around. But sometimes there's just such fundamental differences of opinion that that's the only solution. And obviously you don't want to do that but if that's all you can do, that's what happens. Okay, so having Cora kind of act as the adult in the room worked very well for a long time until Gamergate blew up. And when Gamergate blew up, that just, you know, it was no longer in the FreeBSD project, it was all over the web. And it became clear that what we had as rules which had worked very well for a long time were just not up to the task of that situation. So it was decided that we needed a more formal code of conduct which was drawn up by a well-meaning but inexperienced set of folk. That is people that hadn't really done a code of conduct before, they just kind of looked around the web and pulled some things together and put it here. Here it is. And the feedback was this is insufficient. It just, it looks like it was written by amateurs that don't understand how to do this kind of thing, which it was. So okay, fine. We're going to do it right this time. So it was replaced by a code of conduct drawn up by folks with more experience. But the problem here was that it was overbearing. And it was a code of conduct that might work well in certain situations, particularly corporate situations, but it really wasn't tailored to what free bestie needed or wants. So a third revision of the code of conduct I am told is in the works. Jeff Roberson has collected a bunch of information and it's pretty much geared up to wait for the next core to come in rather than the existing core which has been involved in the first two of these. So that we can get some new eyeballs on it and hopefully get some more feedback from all of everybody in this room and elsewhere in the project. There are people that would just like to go back to the old days and we already have proof that that doesn't work. So we need to do something here with a code of conduct and hopefully the third time will be the charm and we will have something that people are both happy with and will bring us into the modern world of what we need. So we'll see where that goes. Free BSD Committer Turnover. I've already talked about how we had to get rid of dead wood in core. Well that's not just core, that's the issue. We have the same issue with committers as well. People get excited, they work for a while and then they sort of fall off the, you know, and move on to other things and you have to have some way of eliminating them because otherwise you just end up with too much dead wood. And you have to have a good way of doing that in a way in which it's not perceived as the, you know, people in charge picking on individuals. And so there, it has to be essentially an algorithmic way of deciding that you are dead wood and it's time for you to move along. And it has to, everyone has to understand what the algorithm is. The algorithm is pretty easy. It's you drop commit privilege after one year of inactivity. And so if you just don't do anything and this is like a microscopic, all you got to do is go once a year and change your personal information in your, you know, committer description and that does it. Okay. And it's not like a lot of effort to remain a committer. And I did notice a large number of those commits happening recently so that people would be eligible to vote for core. Good on you. Okay. And in fact, if you haven't made an entry in the calendar file yet, please do that. That'll count. All right. So for up to about two years and this is not quite as hard and fast because core sign gets to decide but after the one year where it's dropped off, if you won't come back, you know, after 18, 20 months or so, core will generally say, okay, fine, you know, just turn it back on again. But after it's been more than about two years, you actually have to go through the whole getting on board process again. And a lot of that is because the way the project does things just changes over time. You know, we change from NETs to Bugzilla, we've added fabricator, you know, there's just all kinds of things that have just been done in different ways. And so you can't just come back and just jump into it doing things the way they did were done five years ago because people will get upset reasonably. So what ends up happening is that you still get a mentor just as you would if you were a brand new committer but typically you don't have to be, you know, a mentee for a year. You can often just in three to six months be up to speed again and, you know, your mentor will say, okay, you got it, go. Which means they're off the hook for you doing stupid things. All right, so the FreeBSD project has done a very good job of, you know, getting rid of the deadwood but of course the other half of that is if you just get rid of people then pretty soon you don't have anybody left. And so recruitment turns out to be just as important if not more important than getting rid of the deadwood. Okay, so you've got to bring in new developers and there's quite a few places that this can be done. A lot is contact via university courses. There's this whole website that several of us have been working on called TeachBSD which is setting up curricula that can be taught in universities and going out and actively doing stuff in universities. Robert Watson and George Neville-Neal in particular have been doing a lot of stuff at Cambridge. Another key place to get them is people working in a company that's using FreeBSD. And I mean, if the company is using FreeBSD they are almost certainly doing some level of development and almost certainly has things that need to be upstreamed and so it's both to the company's advantage and to the project's advantage to get those people brought on board as committers so that instead of having to nag somebody to get things put in they can just do it themselves. And this is of late proven to be a very good channel for bringing people in. Now you say why would someone want to become a committer and just have extra work to do after you've done your 50 hours of work at the company. Now you're going to do another 10 hours for FreeBSD. But it turns out that if you're a committer to FreeBSD it actually raises your value. So if you change jobs it tends to give you a salary bump when you go to another company that's using FreeBSD because you're already a committer and this is viewed as a very useful asset to have. So in fact we've had very good uptake on bringing in people that are working for companies and become FreeBSD committers. Another key place that has brought in a lot of new committers over the years has been the Google Summer of Code project. FreeBSD has typically been given 10 to 15 slots in Google Summer of Code and for those of you that have pay any attention to Google Summer of Code most projects are thrilled to get two or three spaces and we are routinely getting 10 to 15 and we could have more if we could get more mentors and more projects in line. So you know there's a lot of there's a small set of people that work very hard on Google Summer of Code and it would be great if we could get some more people involved in that. And then finally there's just discovering the project at a conference or through social media and particularly EuroBSD which moves around to different countries every year. We get a large influx of people that have never heard of BSD and all the projects and not just FreeBSD but NetBSD and OpenBSD tend to get a lot of excitement and new people that come on and start working with the project. It's key that your project be welcoming and easily entered. You come here and it's like oh yeah I know everybody blah blah blah. Well if you're new to the project you don't know anybody here hardly other than maybe their name and so especially those of you that are older timers try and make a point of talking to the new people and welcoming here and making it sound like it's a fun place to be and not a terrorizing place to be. So provide an easy path to connect with the project and provide mentors to help get people up to speed. Again mentoring is not for everybody. There's some people that come up and say we want to be a mentor and it's like we have some other things that you could help us with. But most of the people here could be good mentors and you know well how do I become a mentor. Well talk to us. Anyone from CORE say you want to be a mentor you'll be their best friend for a day. They might even buy you a beer. So what is the net effect of all this. Well I've been collecting statistics over the years. In fact this is something I cranked this script actually an hour ago and updated my slide because one of our new committers is 21 and previously it was 23 I guess was the youngest committer that we had. So I was very happy to be able to pull this slide out. I didn't knock the median or average down but nevertheless it helped push it that way. At the other end here I'm unfortunately three from the top here. Diane Bruce has got me beat she's here and Greg Lee he was not here but he's out here at the end. If you just take off the over 60 crowd here the median and average both dropped down to around 38 39. So you know just get rid of the 10 old geezers and your graph will look better. At any rate the key thing to note here is first of all this big huge spike which is late 30s early 40s. And the project has stayed right in that number range for more or less a decade. When it was very when it first started up it was down in the low 30s and it has slowly risen it slowly rose into the high 30s and it's pretty much been there for at least the last eight to ten years. So we are doing a good job of bringing people in and that's good. If a project is going to have a long lifetime you've got to keep this age demographic you don't want it to age out. And in particular if you look at the Linux project Linus Torvalds and his lieutenants are pushing out on that curve and they are not doing a good job of bringing in younger people. They are bringing in a lot of people in companies so it's not like that project is going to go away but the sort of image of what Linux ought to be is going to become a corporate one if they don't somehow work to replace that top set of people. I'd iterate some of the other numbers are kind of interesting here. The calendar.previousd is where I get the ages so there's 46 of you who won't ante up what your age is. I actually sent out an email middle of last year to everyone that I didn't have an age for and said either put yourself in the calendar or tell me what it is and I'll put it in my secret one that I won't tell anybody else what it is. And it's about time to do that again because I had it down to around 25 at one point. And someone was saying oh yeah I saw a big flurry of changes to calendar.previousd. What was that all about like me? An email. So anyway if you're not in the calendar or you haven't already sent the information to me I'd appreciate it especially if you're younger. Bring that age demographic down. Great. So the previous development model some of it works and some of it not so much. What we do great at is small changes especially ones that are easy to identify, reach some consensus on and implement. And when I say small changes I mean changes that can be done by one or two or three you know individuals working together. And we have numerous examples of things and you know it's not like that small of a change and we got SCTP brought into the system but that was basically Randall and a couple of his cronies that did that and they didn't need like you know everybody in the project to be involved in order to make that work. It's the larger wide ranging changes that are hard. So some examples switching from CVS to Subversion took nine years. And the problem was that we would start with the discussion and everyone would say oh well there's this and there's this and there's this and have you considered this and have you considered that. And blah blah blah blah blah blah blah blah blah and there would be no consolidation on one particular thing and it would die out without resolution. And 18 months to two years later it'd be like oh we need to do this and we'd forget the entire discussion we'd previously had and we'd have the same discussion all over again and the mailing list would be flooded with stuff and blah blah blah it would die out. And it was finally accomplished by Peter Wemm who just said I'm sick and tired of this I'm just going to do it. And he just did it and one day it was click and now we're under Subversion. This is not an ideal way of handling this. Okay. The other huge change came when we went from the single threaded to the symmetric multi-processing which was between FreeBus D4 and 5. And this required changes obviously throughout the entire system. And you sort of have this theory that well we'll just put a giant lock in and then we will just slowly unlock things bit by bit. And you know forget PID that's pretty easy to do. But when it gets to things like the networking stack where you've got locking well let's see we've got packets going down so we're going to lock from top to bottom. Oh and we have packets coming up so we're going to lock from bottom to top and blah blah deadlock. So it's not that easy. So we started just to make it interesting with a big debate as to what kind of locking we were going to do. The Solaris style locking or the Amigo style lockless stuff. And that debate raged for a while. And eventually it got decided that we were going to use the Solaris style of locking. And the lockless people weren't willing to give up and kept trying to put that in and that's when there was an eviction of a member of the team. But you know once we got that sorted out then it still required this just massive coordination and for any of you that remember it was a long gap between four and five coming out. And five came out kind of because we had to it had been so long and five left a bit to be desired in terms of stability and so on. And it was really not until six came out that we really had sort of gotten over the hump. But you know that both of these are examples where the model of just sort of making changes and everyone sort of coordinating on mailing lists just doesn't cut it. So how do we do better on those things? So we recently as in about a year ago, CORE created the FreeBisD community process which is modeled on the Python enhancement process. And the idea is we want to get away from this, have a huge argument and then do nothing and forget about it and have the same huge argument again and do nothing about it and so on. And just for an example of something that's on the horizon that we're going to have to deal with is deprecating I386. Okay. We've already had, you know, one round of discussion on this. It's clearly not time for it yet but the time is going to come. And we really don't need to keep every, you know, 18 months having a big discussion with the same arguments over and over again. So this is a classic case where this will hopefully help us do this in a much more rational way. So the major changes are described in the document. Well, the major change is pretty easy. We drop I386 support. But then there's, you know, here's the reasons why, here's the reasons why not, here's, you know, reasons, the why not reasons may die down over time, here's some of the ways that we can determine that they're dying down. And so we have that whole, you know, distillation, let's say, of the discussion that we've just recently had on the mailing list and put that into one of these documents. And this gets reviewed by an editor whose job it is to, you know, edit, make sure all the opinions are there and, you know, so on. And then it gets published for discussion. And as the discussion continues on, those comments get put in there. And when the core team feels that the discussion has kind of died down, then they make a vote on deciding, you know, whether we're going to move forward with that particular thing or not. Now, in the case of I386, they will almost certainly decide, no, we're not going to do it at this time. But this document is there. We've had all these discussions, so 18 months from now, instead of starting from ground zero, we can restart this document, which we already have a bunch of things in. We say, all right, what's changed? All right, well, some of the reasons for keeping it now have gone away. So we'll move them from this pile to that pile. But then there's some other reasons that have come back in. And so, again, we'll have, you know, sort of, the discussion will move forward. We will probably still decide two years from now that we're not ready to get rid of I386. But, you know, eventually, we will come around and the document will get to the point where the reasons to keep it will be dominated by the reasons to get rid of it and it will happen. And it will happen with, let's say, one-tenth of the amount of mail on the mailing list as we would have if we just keep with the current scheme of arguing about it every 18 months. So this hasn't really gotten a good test yet, but I think we've got some ideal candidates that should go through this process. Okay, another issue that has arisen of late is the core team interaction with the free BSD committers. In particular, there were some that felt there wasn't enough communication about the code of conduct, can you imagine? And so there's been a push to try and get core to be a little less opaque. Historically, all the core communications were private among core and a monthly report would be thrown over the transom once a month saying, oh, this is what we did. These people got commit bits and we had discussions about these topics and we made these conclusions, these decisions. Period. That was it. So there was a feeling that there wasn't a good way of giving feedback to core. All you got to do is say, well, I don't like that decision. It's like, well, tough, it's been made. Wait for next month. So in recent time, core has started releasing an agenda of the discussion topics so that you can see what they're going to be talking about. And if there's something that is of interest to you, you can provide some feedback before they've had the discussion. Now this is a nice idea, but the problem is that they have, I mean, when you look at the thing, it's this laundry list of stuff of which they don't actually end up doing all of those things in that meeting. They just want to pick through a few of them. And so it would be nice to know, well, what are the things you're going to actually talk about in this upcoming meeting as opposed to, well, someday you're going to talk about that maybe. So a little bit of work needs to be done there. There's also contemplating allowing committers to join the video conference, not being able to talk, but being able to at least listen and perhaps poke in questions through IRC or something. Now there's certain things where they have to deal with personnel issues. And obviously that is just going to have to be a closed session. But just like when you have a board of directors, most of the meeting is open and then there's a small section that you have where it's just the board. But again, that's a relatively tiny piece of what they have to deal with most of the time. So there's a hope, and this is sort of my throwing it out to whoever the incoming core cuts to be, that this is a nice start, but keep pushing this forward. OK, finally, the FreeBSD security team. Security has been a lot in the news lately. And on average, the project has done well in this domain, but unfortunately it's a job that's really hard for volunteers in particular because it's mostly nothing to do until there's a fire drill in which they need 110% of your time. And you might have another job, and your employer doesn't want to just give you a month off with 30 seconds of notice. And so the people that have done it have done an excellent job, but they get burned out. And so we'd had a series of very good security officers, but we'd gotten to a point where we needed someone who could do it, especially with a lot of things that have been coming down more recently that was becoming much more of a job than it had been. And so it seemed like a place where the FreeBSD Foundation could come in and at least provide the person who's going to do the sort of management piece of it. So not necessarily to figure out the technical part because we have a lot of members in our community that can help with the technical issues, but can at least be there and deal with the immediate responses that need to go out and coordinate, find the correct people to deal with the problem and get them pulled in and working on it, et cetera. So in fact, as far back as 2002, we expanded it to a security team where we had the security officer who was doing the kind of management bit, and then the security officer, as I say, oversees the pool, people getting that done. And it used to be a part-time person, and now it's become almost a full-time job, at least the last couple years. And so that's where the FreeBSD Foundation came in to help provide that person. It doesn't mean that everybody's off the hook, no means, but it does mean that there's going to be somebody that when the reporter calls up and says, what is FreeBSD doing, they at least get someone to answer the phone. All right. So the grand summary, the evolution of governance model for leadership, this has been something that has been of particular interest to me because having been one of the benevolent dictators for life, that was clearly not a model for something where it was going to outlast me. And so moving into having a core team for life was sort of the next logical step. Moving to an elected core team seems to have worked well, and the project continues to move forward and grow, which is great. If I get hit by a bus, the FreeBSD project isn't going to notice the difference. Four major changes in leadership, each allowed the project to move forward and tackle new problems. You get people that come in and the early getting the BSD out in the first place, getting it open sourced, took a few years off of several of our lives. It has certainly helped the project avoid the aging out. The median age has remained in the mid to high 30s, at least over the last decade. And the elected core, I mean, there's all kinds of things where that can go wrong. You can end up electing people that just aren't, you know, core is much more of a management position than a technical position. I mean, they obviously have to have technical chops, but a lot of what they do day to day management is interacting with people. And so the project as a whole seems to be pretty good at bringing up people that have those skill sets. And you want some turnover, you don't want it to just be the same group of people forever and ever, but you also don't want to turn everybody all at once unless they've gone really off the rails. And so, you know, again, bad things could happen, but so far they haven't, and hopefully it will stay that way. The FreeBSD Foundation has provided not needed resources and stability. They are somewhat the long-term memory of the organization. They deal with the long-term issues like making sure that the copyrights don't get abused and, you know, all those kinds of things that, you know, just you need somebody that's going to just do that more or less continuously. So they have the solitude of longevity that's needed for that. Okay, so what's my conclusion? My conclusion is the governance is mundane. I'm amazed to have this many people sitting here listening to me talk about it for an entire hour. But nevertheless, too much of it is stifling. You can certainly overdo it. I have been in organizations that all you want to do is run out the door to get a breath of air. But you can also go to the other extreme and say, oh, you know, it's all laissez-faire, blah, blah, blah. Too little in the project goes off the rails, collapses from lack of infrastructure. There's all kinds of bad things that can happen if you're not paying some attention to this stuff. So it's a fine line. People say, oh, you can't have enough. Oh, yes, you can. Oh, you can't have too little. Oh, yes, you can. So we seem to be doing pretty well here, and hopefully we will stay on that track. The thing to really note is that governments requires constant tuning to keep that rate balance. You know, you ebb and flow. You tend to bounce back and forth between too much and too little. And you do have to pay attention to it. But when it's working, it should just be in the background, and you shouldn't really be all that aware of it. OK, so with that, I will take any questions. Yes? What is core doing about it? That's an excellent question, and that's why I'm not running for core. You couldn't pay me. What was it? If nominated, I will not run, and if elected, I will not serve. I did my 10 years. Someone else has a problem. OK, anything else? Yeah? In 2018, do you see core as more of a technical body or a board of directors? Do I see core more as a technical body or a board of directors? Yes, is the answer to that. They are, in fact, doing both of those things. I mean, ultimately, there are some technical decisions that need to be made where there's disagreement, and hopefully they can help those people come to a resolution, but they have to be technical enough to be able to do that. They also are a board of directors. I mean, they are overseeing all those different groups of people. They need to make sure that those groups are working properly, and if they're broken to figure out how to fix them. That's management through and through. That's why I'm very pleased when I looked at the list of people running for core that we have people that are across that entire spectrum. I think I'm convinced that the core will have the various assets that they need. People, oh, I can't run for core. I'm not a kernel hacker. That is malarkey. Yeah? How long have you been using video chat for months to meet? How long have I been using? Video chat for months to meetings of what tools? You would have to ask the core people that. I haven't actually been paying attention to what they're using. Can you repeat that? Three. They use Google Hangouts? Three. What? We don't say hangout for three years. Okay, thank you, Ben-O. See, you have to ask someone on core. They'll have the answer. All right, anything? Yeah? Generally speaking, we provide some things to core, the pay for when they were using conference phoning. In the case of code of conduct, they came and said they wanted some money to get people to help them with it when they did Rev2. Generally speaking, now with my foundation hat on, we're very sensitive to not somehow being viewed as taking over the project. Now, I will point out that some of core is actually also foundation board members. So it's a fine line that we're walking there, but our goal is that the project runs and we are there as a support service. And when they want some help, they'll come and ask, and we will try and provide it. But a lot of we're doing things that are sort of on the side. So trying to go out and raise awareness and provide travel grants. A bunch of you here are being at least partially paid for your travel to come here. And Google Summer of Code related things and so on. But the goal of the foundation is to just be sort of quietly on the side and that the project and core should be driving what's going on in the project itself. Okay? All right. Thank you.
future, which is to say they contribute money to BSD, started in 1977 as a project of Bill Joy at it became a full distribution with the release of the first release from, open source release from kept getting accreted and accreted, and the distribution kept getting larger and larger. And handle that. All right, so meanwhile, how about decided that they would create a set of bylaws. had to vote for it. And if you're about to put handles security issues, alerts and updates. The many lists. And so it became necessary to have actually happened too horribly in recent years decided that we needed a more formal code of years, you actually have to go through the whole things. All right, so the FreeBSD project has companies and become FreeBSD committers. Another the new people and welcoming here and making it sound like it's a fun place to be and not a reasons to keep it will be dominated by the being able to talk, but being able to at least with most of the time. So there's a hope, and finally, the FreeBSD security team. Security has the grand summary, the evolution of governance quietly on the side and that the project and core