FreeBSD bmake and meta mode
Gerraty, Simon J.
The Junos "meta mode" build is the coolest thing since sliced bread. The FreeBSD projects/bmake branch provides a proof of concept for the more general use of this technology. The latest release of FreeBSD uses bmake by default, a prerequisite for the "meta mode" build. This talk will cover the adoption of bmake and how the projects/bmake branch differs from traditional FreeBSD build.
 Let's see if we can make this do something. Good. We're here to talk about building FreeBSD with Beemake. I'm going to do a quick intro. The question came up recently on the developers list, why Beemake. I'll briefly talk about that. I'll talk about the transition of FreeBSD to using Beemake. And one of the reasons for giving this talk is to provide a venue for people who have questions to ask them. So if you have any questions, feel free. I'll then talk a bit about the metamode and DERDEPs builds and projects Beemake where you can find them and play with them, which was, to a large extent, the reason for this whole exercise. So I started Beemake back in the early 90s up until then. I'd been mostly a sun-o-wes guy. Because sun in their wisdom didn't provide a tool chain. Everybody used Gmake and GCC and so on. And in 93, my sun workstation got hit by lightning. And I decided that I had to replace it. And I didn't really fancy spending another $20,000 on another sun workstation when I could spend $3,000 on a PC and run $386. BSD. And so that's when I discovered the BSD make. And it didn't take very long to decide that that was a winner. So Beemake is basically net BSDs make with autoconf added. Much as I liked building on BSD, net BSD in particular at the time, most of my clients used crazy stuff like HPUX and UTS and AIX and all sorts of nonsense. And so I wanted to be able to have a nice portable build that was as nice as the BSD build, but portable to everything under the sun. And that's what I got with Beemake. I've personally run it on everything from AIX to UTS and pretty much in between. Back in 2000, when I started at Juniper, their build was in a fairly sorry state. And my first project was to fix it. And so it's been building with Beemake since 2000. And to a large extent, Juniper pretty much funded me to develop make for net BSD and anybody else that wants it for the last 14 years. I put a lot of features into net BSDs make in that interim. And one of the things that always surprised me was, although I didn't do a lot of evangelizing within the net BSD community of these features, almost without exception, within a week of me putting some new cool feature into make, it was being absorbed into the net BSD build without me even mentioning it. And I always thought that was pretty cool. So up until fairly recently, the Juniper build had diverged considerably from free BSD. Juniper, we run Junos, which is basically free BSD. And we modified the build considerably. And we never really bothered tracking the free BSD build until a few years ago when we decided that we wanted to move to an environment where we had free BSD building as close to stock free BSD as we could and then adding our Juniper bits on top of it. And then all of a sudden, we had the situation of, well, we don't want to have to continually retrofit our build to free BSD. And we don't think free BSD are quite ready for swallowing all of our build. So let's see if we can meet somewhere in the middle. And so 2011, I gave a talk here and we had a chat to some of the free BSD folk about whether there was any interest in picking up some of our build technology for free BSD. And there was actually considerable interest. So we started the process. And so let's see. The first commit of being made into free BSD head, October 12, 2012, we had a bit of a headshot. We had a bit of a hiccup with the ports. We needed to get a ports run to confirm that we weren't going to destroy ports. And they had to rebuild all their infrastructure, which delayed that for a while. But nonetheless, free BSD 10 shipped with building with be made by default. It's been back ported to 9.3, so the port folk can use that. And I have to say none of this would have been successful without the goodwill and cooperation of both projects. There's a number of changes that have gone into NetBSD's make to make it easier to do this project. The NetBSD folk were more than willing to absorb some of those changes. And so thanks to all of them on both sides. So why Bmake? Somebody asked this quite recently. And actually Warner answered it for me. And he put it very well. Basically, Bmake is little more than NetBSD's make. NetBSD have a very active team of folk who are contributing regularly to Bmake. So it's not a situation if I get hit by a bus that everybody's through. There's plenty of people around who understand it and can support it and maintain it. It has a plethora of cool features. Not least is an abundance of modifiers. And quite importantly, in that regard, I forget some years ago, I reworked the modifier handling so that you could use it recursively. And what that lets you do is stick complex constructions of modifiers into variables which you can then reuse without having to cut and paste them everywhere and getting them wrong. Things like multiple iterator values for for loops is quite nice. There's at least one for loop in the port stuff that would have been much nicer if it had been able to use that. Things like being able to auto-ignore stale dependencies found via dot-depend is actually very helpful. I did a clean on a FreeBSD tree that I've been doing for production builds for a year now. And I had to clean it for the first time a couple of weeks ago, which was really annoying. Because we'd switched branches. I had to switch to a new branch. And there was something I had to clean. It was annoying. I'm going to talk a bit about dirt-eps.mk. It uses pretty much every single feature of Bemake. And so it alone is a cool reason to do this. So why meta-mode and or dirt-eps? It's worth noting. I usually talk about them as one and the same thing. They're not. You can actually use each of them independently. You could use dirt-eps with a completely manually maintained build system. It would work very nicely. But it's certainly nicer and easier if you have the meta-mode stuff as well. Together, they give you a very simple and reliable and maintainable solution to building complex chunks of software. The FreeBSD build is a decent size. The Junos build is a lot bigger. And building large sets of software in parallel is actually a somewhat complex task. But we like to make it as simple as possible. One of the nice things about the dirt-eps build in particular is the build works exactly the same way regardless of where you start. So if you like to edit bin cat and run meta-ex compile with any Macs such that you're effectively doing a CD into bin cat and running make from there, it will do all the right stuff even if it's a clean tree. Similarly, if you're at the top level and you want to do a quote production build or you just want to build bin cat and so on, it works exactly the same way. I'll probably cover it again. If you look at the top level make files in, say, FreeBSD or NetBSD or Junos, if you could, in the pre-meta-mode days, there's some pretty large make files. I think FreeBSD is currently up to about 1,800 lines. The pre-meta-mode Junos build was 5,000 lines. And I could guarantee that there would be very few people in the company that could actually read it all and follow what was going on. When we cut over to the meta-mode stuff, that was reduced to less than 200 lines. And 200 lines that virtually anybody could read. So it's affordable. I don't know about you guys, but at Juniper I have close to 2,000 developers. And believe it or not, a lot of them don't log their builds. And most of them are not build geeks. And so when something goes wrong, they send an email to BuildTrolls. I updated my tree and it doesn't work. Yeah, a bit more info would help. We actually, actually, you'll see, I've got an example. We actually trained Bmake years ago to spew mountains of useful data that is exactly what we need to see when you say, my build broke. And we document all this stuff. And he says, if you want to report a build problem, Bmake, when it blew up, it spewed out. All this info, please include all that in your problem report, and you'll get much faster resolution. Well, of course, you never get any of that. And when you ask them, well, where's your log? Oh, I don't have one. And so you're very often stuck with saying, well, go do it again, log it, and come back to me. With the metamode stuff, we actually log the critical info whether they want us to or not. And so very often, all the important data you need to analyze a failure is there. And everything works better. All right, here's a quick teaser. It's very hard to make a build log look interesting. I'm sorry. This is building BNSH in a clean tree. I usually use bin cat, but some people think bin cat's a bit simple. So BNSH is a lot more complex. So here we are building BNSH in a clean tree. We just seeded into his direct, well, actually we seeded into BNSH, and we said make minus j8. And he just went crazy and did it all. And you can see, and how long did it take? It took 58 seconds. It's not a particularly fast machine. So that was building libc and all that sort of stuff. The things to note are the object directories were created automatically. There was no make, depend, done anywhere. Everything ran in parallel, but in the correct order. The log was easy to read. There was no noise about compilers and all that sort of stuff. We only built that which was necessary. And we visited the leaf directories directly. There's no tree walks involved. And at the end there, in this slide, you can see the last thing he did was update make, file, depend, or at least check whether it needed updating. All right, try it again. So this is a quick look at one of those make, file, depend. This is actually, I think, the one for BNSH. In the Junos build, because we're always cross-compiling and we're always building for multiple architectures at the same time. We actually, by default, use a make, file, dot, depend, dot, dollar machine so that we can have them updating in parallel without any concerns for contention. When we looked at doing this for FreeBSD, we figured that was going to be a little bit too much. And so one of the exercises we went through was to look at making it all work with a simpler model. And it's actually worked out very nicely, I think. So what we have here, you can see from the client, it's an auto-generated make, file. That means you don't touch it. The first thing it does there is it sets a variable that basically represents the relative location of this directory within the source tree. And that's key to how the whole thing works. Pazder is one of those magic variables that B makes sets for you. It represents the directory where it found the file that it's currently reading. And one of the modifiers that B make has, it says you can take the value that you've got here and you can pass it to a real path and turn it into an absolute path, which is extremely useful for this process. The key data in this file, apart from that, is DERDEPs, which is a list of the directories. And again, these are those directories relative to the top of the source tree that need to be built before we can build BNSH. And I mean, it's pretty simple to read. The only complexity here, if you want to call it that, is we substitute the variable CSU DER for what it actually was. This allows us to use the exact same list of dependencies regardless of the architecture. I386 is a bit of a wart in that for most architectures, the CSU DER is easily derivable from the machine value, not quite so for I386. And then the last thing it does is include DERDEPs.mk. And also at the end there, it will capture, if necessary, any local dependencies. These are what allow us to do a parallel build in a clean tree without doing make-depend. This is effectively capturing just the stuff within each directory, which are the sort of things that would normally break a parallel build in a clean tree. We'll look at this in more detail later. So building previous, the projects we make, as I briefly mentioned, is a test vehicle. It's an exercise in putting all of this building into a tree which is very close to head. It was synced last week from head. And it uses all the metamode and DERDEPs and all that sort of thing. The goal is to be able to easily cross-build FreeBSD as close to stock as possible, while minimizing changes to FreeBSD itself. Within Juniper, we have a team of FreeBSD folk. We've been building head and stable 10 now from back around 9, using almost exactly the same setup. There's a bit more stuff there. For instance, we use external tool chains. And we generate packages which are basically ISO FS's and all that sort of stuff. And disk images for feeding into VMs. But from the pure build point of view, there's very little difference. So just quickly, on the transition for FreeBSD, we first talked about it with the FreeBSD folk in 2011. Converting the base to use Bmake, which is the first step to be able to do any of this, was actually very simple. It took a patch of less than 300 lines to allow the FreeBSD head to build using Bmake instead of Fmake. And we're going to use Fmake to represent the old FreeBSD make, just to avoid confusion. Ports was a little bit more complex, because they have to support building on older versions of FreeBSD. At the time, it was 8.3, I think. And they don't branch. So that makes for a bit of a challenge. More importantly, it's reasonably simple to fix the base. It's a bit more complex to fix ports. The most difficult problem is all of the thousands of people out in the world who are using FreeBSD and things derived from FreeBSD who we have no visibility into. And for those people, it's a bit rude to go and pull a rug out from under them. And so as we'll see, we made a number of changes to Bmake to minimize the impact on those people as we go. And so again, just call if anybody does have horror stories or something like that, feel free to let us know so that we can maybe do something about them. So there were changes made to Bmake to support FreeBSD. Almost without exception, those changes go into NetBSD first, and then get imported back in. One of the goals of the exercise is to minimize divergence between FreeBSD and NetBSD with respect to make so that we all benefit. So both Bmake and Fmake are descended from Adam DeBoer's Pmake from Sprite. They've diverged quite a bit. One of the most obvious differences is the colon U and L modifiers, which I think FreeBSD got from OpenBSD, around about the same time that I was sticking all the OSF modifiers into Bmake, which, oddly enough, had colon U and colon L as well. Fortunately, these aren't used by the base system at all. And as of last week or this week, Ports doesn't use them either. But for the purpose of our story, it's still an issue. Or it was an issue. The other, I guess, key difference is Bmake is, aggressive is putting it nicely with respect to $.path. It will find anything by a.path that it possibly can. And that isn't always what you want. So you sometimes have to curb his enthusiasm. And there's a.noPath thing to do that. Fortunately, just simply saying.noPath, $clean files, covers 99% of the problem. NetBSD's bestyone.make marks all of the standard targets as both phony and not main. So that was another change. One of the real fun ones is handling job tokens. All modern makes, when you do a make minus j8, you don't want every sub-make to start another 8. And so on, you get what is it? Geometric increase in load. So they use the means of having a token pool of some sort to constrain that. Both Fmake and Bmake do that, but they do it differently. Fmake uses a FIFO, which the first instance, when Fmake starts up, he has a look to see if the FIFO exists. And if it doesn't, he assumes he's the master, creates the FIFO, exports its name, and everybody else then uses it. Bmake, on the other hand, uses a pipe. And he passes a magic argument to his children with the descriptors for the pipe. And so if Bmake doesn't get that argument, he assumes that he's the master, and so he goes and creates it and passes it on to his children. Both systems work nicely, but they don't work well together. In particular, if you had Fmake as the initial instance, he would export the name of his FIFO to the environment, go and start a bunch of sub-makes. And if they happen to be Bmake, each of them would think that he was the master, because he'd had no clue that he wasn't. And so he would start another N. It almost works the same going the other way, except that Fmake would blow up if Bmake passed him his minus capital J argument. So you're sort of saved there by that incompatibility. But if you were careful to cleanse the environment that you passed to Fmake to not have the minus J, then each Fmake would think that he was the master, and you'd have the same problem. There's other useful stuff in there. So a quick talk about the changes to the base. Any of the Bmake-specific syntax we bounded with ifdef.pasta, that's one of those magic variables that it defines for you. We added.nopath to all the generated files and phony and not main for the standard targets. We created a.weight as a no-op target for Fmake so that you can stick.weight into subdur lists and other things like that. And we ended up adding an option without Bmake so that people who, for whatever reason, didn't want to venture into the sharp pool could avoid it for a while. Another change, Fmake had a preference for BSD makefile before it looked for the traditional makefile and makefile. Bmake only looked for the traditional stuff, but it's configurable. So by simply sticking a list into sys.mp you can make it look for anything you want to. And in all of these cases, if we could address an incompatibility by just looking at the magic variable in sys.mp, that was the preferred solution. For the job token one, that's pretty much what we had to do. Bmake, in particular, only passes his token pool descriptors to submakes if the target is tagged with.make, the special source. In Fremisty Fmake, the man page says exactly the same things that it is supposed to do the same thing. But it doesn't seem to work, which is evidenced by the fact that somebody went and added this $ underscore plus thing to basically do what.make is supposed to do. And so what that means is that the Fremisty makefiles and presumably the thousands of makefiles written by people who use Fremisty but aren't in source have not been necessarily sticking.make on all the targets where they would want.make if it had worked. And so the only reasonable solution we could come up with for that one was to set a little knob in sys.mp to tell Bmake to say, you know what, just pass your descriptors to all the children. And that sort of makes the problem go away. Oh, the error token is a nice one. So normally when Bmake fails, this is in jobs mode. He sticks an error token back into the token pool. And the next time all the other submakes go to take a token out of the pool, they get the error token. They say, oh my god. And they bail. Which is wonderful because your build stops immediately on error, which 99.999% of the time is exactly what you want. Apparently it's not what you want for make universe. In that case, people would prefer that if some variable of MIPS is broken, that 99.36 and all the rest could just chug along and get as far as they could. And so we added another knob to say, you know what, if this make job error token is set to false, then don't stick an error token into the pool when you die. And so in the top level make file, if it sees that you're doing make universe, it'll set that knob to false. And so you can get the make universe behavior that you want and still get the nice behavior for all the normal cases. Oh, another. This is another fun one. So in the long ago, make use set minus e to fail things on statements. I don't recall exactly why. But I think it was mainly to get more consistency in target handling between jobs mode and compact mode. In NetBSD, they did away with all that. And they made it work very reliably such that the unit of failure was the command line rather than statements within the command line. So with fmake, you could do CD, to some bogus directory, semicolon, rm minus rf star. And it would be reasonably safe because unless you had a directory called that, it would fail at the CD and never get to the rm minus star. If you fed that same command to bmake, though, he would happily remove everything in the current directory, which may or may not be good. Well, the object directory anyway, if you had one. The format at the bottom is, of course, safe with any version of make and is therefore preferable. And while it would have been a quite simple exercise to go through all the make files in previous d and fix them so that they did the latter form instead of the former, again, that doesn't do anything for the thousands of people out in the world who may not get the memo. And so we had to do something about that. So again, this is one of those things that you can configure with make. So we just defined a shell for previous d to use by default, which effectively makes bmake behave the way fmake does and use set minus e by default. The option, the bmake option. So I think when we originally proposed adding bmake to make to previous d to start this transition process, we had in mind that we would just add bmake. It would get installed as bmake and people could choose whether they used it or not and so on. That wasn't considered ideal by the project. They wanted us to do the cut over as soon as possible and effectively, one day you'd go to bed and it was fmake and you'd wake up the next day and it's bmake. And that would have been feasible for the very first part of the project itself, but it's a bit more of a challenge again for the third parties. And so we ended up having to add that option so that people who didn't want to face the transition right now could avoid it. And so we added this option source makefile had to do the right thing, which typically involved it has a dance for you and you're upgrading to recognize that the make that you have on the system is not quite adequate for building the new version of software and it would go and bootstrap a new version of make and so on. And once it existed, it would use it. But it always called it make. And so he would say, oh, I have this temporary version of make, I'll go use it, even though it might have been six years old. So we fixed that so that if you wanted to be building with fmake, it made sure that temporary make was called fmake. And if you wanted to bmake, it would make sure that temporary make was called bmake. And if you change your mind, it would do the right thing. That's all being ripped out now. Thank you, Wana. The apparently was broken for a while. Anyway, nobody noticed. So obviously. That's OK. It's OK. It's gone. So we've burned the bridge. So that's progress. I mentioned the colon U and L for ports. I'd had a cunning plan for how to deal with bootstrapping ports, which worked brilliantly unless you were the portsfolk, because of the way that they want to be able to build ports. And so that wasn't going to work. So yet again, we added a local hack for ports to be able to set a knob saying, I really want the old colon U and colon L. And even with this set, actually, in many circumstances, bmake can do the right thing, because it can tell whether it's looking at a colon U with a value after it, or whether it's looking at a colon U with nothing in which case it might be the old modifier. But again, that's due to be removed after I get back from this conference. Another thing that fmake had, which netpstmake didn't have, was the tolerance for quoted strings as iterator variables. And so that was added to netpst. And then I think one of the final ones was the minus v behavior. If you do fmake minus v foo, it spits out the fully resolved value for you. If you do the same thing for bmake, it will give you the literal value, which may or may not include other variables. This is brilliant if you want to do debugging of what's going on. I use it a lot. And with bmake, you can always put dollar foo in there to expand it fully. So this is great for command line type stuff, but it's not necessarily ideal for the build itself. For the build itself, the fmake behavior is actually better. And so I proposed changing the default behavior and adding a debug flag so that you could give me the literal value. We're compromised by adding yet another knob to say which behavior you like. So revistd sets make expand variables to true, in which so you get the default revistd behavior. But regardless of the setting, you always have the debug flag to give you the literal value. And so both projects are able to have the semantics that they know and like. Oh yes, this was the last one. So bmake, some years ago, they switched the way iterator variables within for loops are handled to avoid namespace pollution. And so they use this colon u value. So what the colon u does is say, if the variable doesn't have a value, use what follows as the value. And so we have effectively a null variable. And so we're going to use value as the value. And this avoids, as I say, namespace pollution. Trouble is it makes it very difficult to try and play games like escaping iterator variables within for loops. And especially when you have nested for loops and you've got to do multiple escapes to try and escape the escapes within the for loops and so on. And I looked at this one for a while and I thought, you know what, let's forget it. Let's just use the inline loops, which are much easier to follow. So I basically replaced a 20 line block of nested for loops with a single line, which produces exactly the same output, of course. Oh, OK. All right. Did we have a pointer? There was a pointer at some stage. Here is one. Doesn't matter. I can't reach. All right, so this thing is the inline loop modifier. The first thing that follows it is going to be the iterator variable. And then everything between the next at and a closing one. And this isn't the closing one because this is introducing a nested loop onto the end. So the closing one is like this followed by another colon, for example. Oh, brilliant. Thank you. Hope I don't kill anybody. Ah, excellent. All right. This is so cool. All right, so in the man page we mentioned that in the OSF development environment where this came from, they actually use a convention of uppercase variable name with dot c the end for the iterator variable. That's really hard to say. But that's really hard to type. I'm lazy, so I prefer single character iterator variables because then you can just use dollar p unless you need to do modifiers on it. So anyway, this is two nested loops. So the first thing we do is for the Mlang stuff, we're going to stick man slash at the start of it. And if we find that we've got an empty one, we get rid of it. And then for those, so each value that we've got at that is going to be an M valuable. And then we're going to also iterate through the M links. Hang on. We're going to come up. What are we doing? Yeah, so each of those is going to be p. So we've got a p, so there's another loop. So we've got p colon e is the extension of p. And so this is actually referencing a man dollar p blah prefix. And then we've got m, which is our, from Mlang. And then again, we stick the p with his extension at the end. And unfortunately, when I did a full screen on my terminal with the normal font, you could actually see the end of the line. And it finishes with a couple of colon closed brackets. This is fire. The colon at colon u colon l colon d and colon p all came from the OSF version of P make. And the inline loop operator is particularly wonderful. Because unlike a for loop where everything is getting expanded as you read the for loop, in this case, the loop is expanded at the time of reference. And so you can do some very cool stuff with it that you can't possibly do with normal for loops. So this is saying for each n in what precedes it rather than for each n in the demo, and then for each p within the loop? Yes. Yeah, I figured you guys would be computer science, but you probably were able to get the hang of it. So do you make after the inline loopers, if they don't have a lot of loops sometimes like this? Yeah, both of them have their place. Don't get me wrong. Dot for loops are definitely useful at times. But there are things that to do it with a dot for loop, you have to do obscene things, which are just quite natural with the inline loops. All right, so why is b make so cool? Well, it can do dirt apps. That's reason enough. You can see dirt apps in the tree. It's in contrary of b make mk, dirt apps mk. I don't recommend reading it on an empty stomach. I've been doing this sort of stuff for 20 odd years, and I consider dirt apps one to read on a full stomach. But anyway, as I mentioned, we have a plethora of modifiers. One of the ones that somebody added not that long ago is this one. So I saw in previous D they have a fixed value that used for the random seed for C++ compiler. This allows you to have a repeatable build, but have a different anonymous initializer seed for every library that you build, which is actually quite a nice thing. This is an example of using the parse dir and parse file. This just ends up being a token that you can use. If you want to create a little marker target, makefile.inc, for example, to test whether that makefile.inc has been included already, you can make a target that includes this or this, or you can spell this all out, but it's more readable to just say this. This will ensure that you'll get a target that is the same value regardless of whether you read it via an absolute path, a relative path, or any combination thereof, because the column TA turns it into an absolute path using real path. So you canonicalize it. This one's an example of, as I mentioned, you can stick complex lists of modifiers into a variable and then reuse it. And there's an example down here. So here's a list with a number of variables, and we basically just apply this set of modifiers to it, and we get a list of exclusions. And that sort of thing gets used quite heavily by DERDeps and lots of other cool stuff. It can do the equivalent of strf time using either local time or GM time. This in the Junos build, we like to have timestamps on the build logs so that we can see where all the time's going. This allows us to have those timestamps in the output logs for essentially free, and so we do it. And we use this format typically so that you get a human readable thing. You get something you can actually do math with and a handy little token to be able to grep them out. And one of the tools I have in my kit bag is a little Python script, which will read one of these logs, and it can basically give you a progress bar so you can see how long the build's got to go until it's done. By the way, I'm not going to get through all these slides. I've stuck a number of slides in towards the end to just cover random stuff that people may or may not be interested in, so don't be concerned that we're going to run out of time, because we will. So it was last sync. The Projects Be Make branch is the one that hopefully most of you are interested in. This was last synced about a week ago, and as of this morning, it completely builds again. The transition to clang through some spanners into the work, but I got some help from David last night to sort that out and finally finished getting it all building again. As I mentioned, unlike the Junos build where we use this format of MakeFile.depend pretty much universally, in FreeBSD we use this format. There are, I think, about five places where we do this, and that's invariably because there are places where the auto-generated, the locally-generated files that you need to capture the dependencies for have names that vary with the architecture. And so the way the build-in for here works is if you had previously built one of these directories for, say, I386, and therefore there was a MakeFile.depend.I386, and then you then went and built it for MIPS when he gets to the end and says, oh, I need to update MakeFile.depend. He says, oh, I see there isn't one, but I see there's a MakeFile.depend.some other machine. I better do the same thing, and he'll just automatically create a MakeFile.depend.MIPS for you. And so all the right stuff happens. The packages MakeFile acts as a top-level MakeFile. In the Junos build, the source MakeFile just includes packages MakeFile. It just makes it simpler from transition point of view. And in the FreeBSD build, we do the same thing. We leave the FreeBSD top-level MakeFiles alone so that we can still do build world so we can make sure we're not breaking FreeBSD when we want to commit stuff back up from that environment. But we don't want to use it for our stuff. So this does the job. Under packages, in the normal world, you would have a whole bunch of packages, foo, to build a package called foo. We use the pseudo directory as a place to put these all represent targets because they happen to have a MakeFile.depend file under them. And so when you go into packages and you say Make Bootstrap Tools, for example, it knows that that's something it can build. And it also knows because it's got.host that it's something that it has to build for the magic machine called host, which happens to be whatever host you're building on as opposed to the host, maybe i3-at-6 or AMD64. But it's not necessarily the same thing as an i3-at-6 or AMD64 target because you may be building FreeBSD 10, but you might be building on FreeBSD 7, which is unfortunately what we're doing at the moment. But it doesn't matter because it works. So for the environment, I think in most of these examples, I use a little tool called MK. Its join life is from wherever it is to have a look upwards until it finds a file called sandbox-env. And wherever it finds that file defines the top of the quote sandbox, and he sets a variable SB to that location. And then he reads that file up to condition his environment. This is great if you're an Emacs user and you're editing branch, I have trees from NetBSD, Junos, FreeBSD, all over the shop, and you can be editing them all within the same Emacs and just do metarex compile in any of them. And it'll just do the right thing. But if you don't like using wrappers and so on, all you really need to do is make sure that you have makesyspaths set, and everything else that needs to be done can be done by, say, local sys.mk or anything else. Projects.bmk has a local sys.mk, which I've checked into the tree. The main reason is it contains a lot of stuff which I haven't taken care of yet. But generally speaking, you can build all of use-land, kernel, tool chains for both the host and target. I added a boot. Clang 3.3 is unable to build Clang 3.4, it seems, or at least the table gen can't handle the headers and so on. So you very likely need to bootstrap the tool chain in order to be able to get building with the new Clang. So I added a target to do that. And since bootstrapping tool chains is not a wheel worth reinventing, I just made it leverage the existing targets in the source makefiling one. We're now using sysroot, which works much better with Clang than pretty much any alternative. You can do build world. You can have build world work, like build world, but produce metafiles along the way. And that's actually very useful both for comparing against when you're scratching your head as to why Clang built when you did build world versus not building when you didn't. You can actually go look at all the syscalls involved and see what was going wrong. It also gives you a means for bootstrapping makefoldup depend if you were wanting to start this exercise all over again. So here's a quick thing about getting started. So minimal environment setup. Once you've got that, you should be able to just do make bootstrap tools. You'd want to do at least J8, because it's going to take a little while. If you're not using my little MK wrapper and you want to then build the tool chain again, and you remember the days of building GCC where you used the host compiler to build stage one and then used stage one to build stage two and eventually. Same sort of thing. You could skip this step, but you would have to add this with tools to these ones to basically use the set of tools that you've built in this stage. It's probably neater to go through this step once and then put it all aside and be done with it. And then you can have fun cd-ing to random places and building them. Or the lot is a pseudo target that just basically builds everything. And again, it should all just build in clean tree. Debugging failure. I was talking to Kasey or somebody last night. And so I gave him this example as to how it's cool. So I mentioned before, nobody logs their builds and they get build failures and they give you absolutely useless information. These days, though, as long as they can tell you where their tree is, we can go find the clues. And as an example, we'll edit bincat and sticker a bogus include in there. And then we go and compile it. And lo and behold, it explodes. Oops, yeah, couldn't find that. And b makes spews out. This is some of the noise I mentioned that b makes spews out on failure. This is all configurable, of course. By default, it doesn't do any of this. You have to set some variables. But you can set it to spit out any variables you like. So we have a number that we like to see. The level is useful, the current directory, the object directory. So it spits out the target that it was building, the metafile that it was creating. And it will actually copy this metafile into an error directory at the top of the tree so they're easy to find. And if you're doing things like I have a continuous build machine, and so he knows to go, build failed. The first thing he does is look in the error directory. If there's anything in there, that's why the build failed. That makes the analysis very simple. And so here's the metafile that got copied into source top dot dot error with a name that we can easily find. And yes, there we go. Even if the guy hadn't logged his build in, this error had actually happened 5,000 lines earlier and is not possibly in his scroll back anymore, we can still go find what the exact error was, why it was happening. We can go and see which version of the compiler he was running and all of the files that were being read and so on. And it may be that, oh yes, you've got the wrong version of the compiler, or no, you tweet your spell. Oops, wrong. But one way or another, you have the information that you need to be able to work out what was wrong and do something about it. For next steps, we think it would be useful to actually add a bit more to it so that you could, for instance, have a target in the Projects Be Make branch that would let you go and effectively just build a little bootable VM image that you could then throw into Be Hive, which would be quite useful. Another thing that may or may not be interesting is distributed building. This is actually very easy to do with DERDeps. But that's probably the subject for a whole new talk. But if anybody's interested, I can bend your ear on that. Are we going? OK, so now we're into just the almost random stuff. But one of the things I was talking about with Warner recently, and he's thankfully been doing, is separating the BSD.star stuff from, so you can think of these things as classes of make files. The BSD.star.mk files are things for building stuff in the BSD way. But you don't necessarily want to have them only able to build user source. And you can have source.ops, for example, as options which only apply to building user source that you don't necessarily install, but are still useful. And similarly, you can have local.star.mk, local.com for anything else you like, as things which are either site local or tree local. They're not part of the repository, per se. But they allow you to do things like customization without hacking the original source. If you look at DERDeps.mk, most of the meta and similar make files, I include a.local, whatever the name of the thing is, to allow customization without having to touch the original make file, which in the case of something like DERDeps.mk is very important. Because, like I said, I only touch it when I'm sober. And I would recommend others do the same. Actually, I would recommend not touching it. So source.ops, this is something that has been very handy. It's now something that we can include within those make files that depends that we use in the pseudo targets. So for instance, this one is for the tool chain. So we can now say, well, is make clang define? No, it's not. Oh gosh, let's go include source.ops. Now we know it will be defined. And we say, is clang yes? OK, we'll add clang to the set of stuff we're going to build. Is GCC yes? We'll add GCC, and so on and so forth. And that way, you can easily handle building all the optional stuff. One of the other things that we saw earlier was auto creation of object directories. A nice feature of Bmake is any source that you give to the special target object sets, the object. It has to exist, of course. So this little dance up here makes sure that it exists. And this is how we do auto creation of object directories. If you are going to do this sort of dance, though, you have to do it very early in the game. Because this has to be done before you start reading make files that are going to potentially influence.path. Because when you do.path some directory, if it doesn't exist, it doesn't get added. And there are cases where it will exist. But if you do it too late in the game or too early in the game, you'll get a path that's in the source tree, rather than the object directory, which was what was intended. So again, if you want to play with auto object directories, you really need to do it during the sys.mk phase. But this works really well. I was talking to, again, Chat last night, 10 something years ago within Junos. We went through the whole phase of the camp of people who like to dribble object directories in the source tree. You had the camp who liked to have just a single directory called obj. And then you had the other guys who were building multiple archers. So they wanted obj.dollarmachine and so on. So we got to a point of having five different ways to support building the tree for all these various combinations. And it just got ridiculous. So we just said, forget it. We're going to create object directories automatically. And no questions asked. And it allows you to simplify things enormously. And simple is good. All right, so since nobody's asking any questions, I'll waffle a bit about meta mode. Basically, what we do is when we build a target, we create, amazingly enough, a file called target.meta. And it collects stuff like the expanded command line, which, if all you did, would be useful. Because that allows you to say, all right, last time I built this target, I used that command line. Now I'm building it, and I'm going to use this command line, and they're different. The target is out of date. And this allows you to do things like skipping. You've all seen make files where you have comments like, oh, if any of the make files change, we better regenerate this in case. And that can lead to all sorts of unnecessary compilation. By being able to be sure that the target is out of date, because its command line actually was impacted, you have a much better situation. We capture any command output, which, nine times out of 10, is just the error. In most cases, it's empty. And we capture all the interesting system calls. By interesting, I mean the ones that are interesting to make. And most typically, he's interested in read, execute. He has to track changes of directory so that he can work out path names and stuff like that. But mostly read and execute is what he cares about. And why do we do this? Well, the automated capture of all this information helps us optimize the build and improve reliability. And by optimizing the build, I mean doing as little as possible, doing it in parallel, but doing it correctly. And capturing the command output, again, makes failure analysis much easier, even when people refuse to log their builds. Metamode and Dirt Eps help with all of the above. As I mentioned, we don't do make to pen. We still do make to pen within our build for the kernel, because at least last time I looked, there were certain files that only get generated via the make to pen phase of the kernel. But I'm told by O'Brien that maybe even that's not necessary anymore, so we may be ditching that soon. But for the rest of the tree, we never do make to pen. And it saves you a lot of time. And while, even before we had all this, we were using GCC to dribble out info so that you could auto-gather dependency information, using the filemon kernel module lets you gather all that information for everything, not just the compiler. Let's see. And so you automatically catch tool change changes, all that sort of stuff. Also, here's the example of an unnecessary recompile that you can avoid. You can use things like DP add to bootstrap your dependencies if you're just getting started. And interestingly enough, you can get into problems where, if you have actually spurious entries in DP add, you've said, oh, DP add, I need lib foo-mumble, but you don't actually use it. It won't be captured in the auto-captured dependencies because you didn't use it. And then you get to the ultimate directory, and it says, well, I can't do anything because foo-mumble's not there. It says, well, that's because you didn't need it. So once we've finished processing DERDAT DP add to add C flags and stuff that we may or may not want to as a result, we actually throw it away to avoid those sort of errors. Some targets will need work. For instance, it's quite common to have a target that is going to include some kind of timestamp. You may or may not want that target to be regenerated every time you build. If you don't, you can actually tell make, for this particular target, I don't want you to do command line comparison because that would make you do the wrong thing as far as I'm concerned. You can actually be clever about it. Make knows that if your target actually involves the.o-o date, does anybody use that these days? It's basically expands to the list of sources that are at a date with respect to the target. And therefore, the expansion of it is going to be different almost every time you do it. And so there is no way that you can reasonably compare a command line that involves any use of this variable. So make knows this. And so if he spots this variable in the script, he'll say, well, I can't compare that command line. I'll skip it, which means that you can cheat because you can deliberately stick that variable in there, match something that will never be in there. And so you actually have a token here which will never expand to anything in the command line. But it'll be seen by Bmake, and he'll say, oh, he doesn't want to compare this command line. But you can still come. So in this target here, we won't regenerate just because the timestamp changed. But if you change anything in DP add, that changes the command line. And so we will regenerate the target. You enable it just by setting the word meta in a variable called make.mode. It gets read by all the make files. It gets looked at after you've read all the make files. So your make files can control this variable pretty much anytime. And effectively, meta.sys.mk does a bit more. But the key bit it does is it says, well, the target all depends on dirt-eps. And then we're going to wait for that to finish. As in, we're not going to do anything else until dirt-eps is finished. And that line there, if you like, is the magic that makes the whole thing work. Well, there's a bit more to it, but you get the idea. So when we write them for $target, we create $target.meta. Excuse me. We skip it normally for any quote special target, like begin and error. Unless you also sometimes you do want a metaphor for these, in which case you can always stick the special source dot meta on there to say, I really want a metaphor for this. It's never created if the target is flagged. No meta. I don't care what you think. I don't want a metaphor for this place. And normally, metafiles are skipped if the object directory and the current directory are the same. But in the case of, for instance, building the kernel, where we try to leverage the existing kernel build to the extent possible, so we actually do the CD into the object directory and run make in there. And in that case, we do want metafiles. And so we stick a little knob in make mode to say, making metafiles in the current directory is OK. And lastly, if the target that you're building is outside of your object directory, he will create a metafile with all the slashes replaced with underscores, just so that you end up with a file that you can reference within that directory. I think we've covered the content. Important to note, there are two variables that are used to track metafiles. MakeMetaFiles contains a list of all the metafiles that we read or examined or in any way knew anything about while we're building this particular directory. And makeMetaCreated is all the ones that we updated. And so this allows us, when we get to the end of having built this particular directory, if this variable is empty, we did nothing. We know that we don't need to go and consider whether we should update the makeFile.depends, which can be a reasonably expensive process, so we skip it when we can. Oh, if the meta verb, if the meta, we're being metavobose, then we'll expand the makeMetaPrefix, which defaults to just building a title name, but you can make it do anything you like. And in fact, I have a fairly cunning makeFile that sets this to a variable that actually is used to track all of the directories that you visited during the build, so that you can then go back and reexam them to see if there's any files that you should have SVN added but hadn't, so you can throw an error. That's another very cool trick. A little bit about filemon. It's a reasonably simple kernel module. The original prototype that John Birrell did of this idea used detrace. Detrace mostly works. It doesn't seem to let you reliably get the absolute path of the program that you're invoking. There's also a little bit of overhead, but more importantly, it needed special privileges to be able to do this. And we didn't necessarily like any of those. So we did this kernel module. It's available for FreeBSD, NetBSD, and there is a Linux version, which they promise me they will make public. And I will nag them continually until they do, because it's actually quite useful. Each record in the metafile that we get from filemon is the whole thing is designed to be easily parsed by a shell script. And the original thing that did the post-pressing was a shell script. So we have a single character to denote the system call that we did. And then typically you get the tag, the process ID, and the data. And the data is 9 times out of 10, just a path name. In the case of linking and moving, it's two path names with quotes, just in case there were spaces in the names. And then in the NetBSD version, I don't bother with stat calls, because they fill the thing tremendously, but they're not particularly useful to make. But they are interesting from a human point of view, I guess. When we read them, OK. So when we read them, and we only read them if Bmake thinks that the target is possibly up to date. So if you're reading a make file, and it's a clean tree, for example, all the targets are going to be out of date, he's never going to look for any metafiles, because he's just going to go and do it. But if the normal rules for make look at all the directory and so on, and thinks that the target is up to date, then and only then will he go look at a metafile, and he'll go and read it. And one of the first things he does is compare the command line. If the command line's different, and you haven't told him not to, he'll say, it's out of date. Go do it. He skips past the command output, and he gets to the system calls, and he needs to be able to pass the system call stuff, and for each file that you read or executed, if it's newer than the target, then the target's out of date. If one of the interesting ones is, yeah, so we have this concept of a bailiwick, anybody like the history, that defines to Bmake the sphere of his influence. So if he's responsible for populating that directory over there on the table, and he's reading a metafile that put a file over there, and it's gone, that means he needs to redo it. And so the target will be out of date as a result. Yep, that covers that. So one of the nice things that we can do with this, as I mentioned, you could write all these makefile depends manually if you wanted to. But it's nicer to do it automatically. So when we finish building, we have a tool that can go and read these things. And he can look at the make.meta created to know that he has to do something. But he actually looks at the full list to go have a look at what he needs to read to get all of the dependency stuff. And this whole process is greatly simplified by having your object directories and your source trees not overlapping. So object directories within the source tree complicate the thing. So it's much nicer to use something like the model we use where your object directories are either completely aside using makeobjder prefix or the fancy makeobjder that we use where you have essentially parallel trees. So here's an example of one. I forget what this was. Oh, this was building something in BNSH. So you can see the things that he read. Anything that he reads out of an object directory that isn't the current object directory defines something that needs to have happened before this directory can be built. And that's the essential idea. And we extract those and we stick them into DERDeps. You can also have me extract a variable called sourceDERDeps if you're doing things like originally within the genus world, we support the idea of subset trees where you can say, I just want to check out enough of the tree to be able to build bin cat. Everybody needs to be able to build bin cat. And so you can look at the source DERDeps for bin cat and say, oh, well, that means I need libc and I need level r and you can follow exactly the same logic for checking out the tree as for building it. But we haven't done that for a while and we decided, because these actually result in far more churn than these do, we actually took them out. And we don't bother collecting them in the Projects Beemake branch. Mapping object to sourceDERD. If you're reading things out of the object directory where they were created, it's a very simple exercise. You typically just substitute object for source top and you've got the directory. Because again, we keep everything nice and neat. If you're pulling things out of a stage tree, for example, which is you can just think of it as a dester that you've done auto install into, you need a little bit of help. Because there isn't necessarily a directory in the source tree that corresponds to user slash include. So what we do when we stage files is we stick a file called.derdep. And when the tool that processes these things, when he goes and reads and he says, oh, the guy used a file from here called uni standard, he says, is there a uni standard.h.derdep? Oh my gosh, there is. Let's have a look. And that gives him the value that he needs to stick into.derdep. And so he can proceed without any further ado. Logs are still useful, though. In particular, we can do things like capturing the number of metafiles that we examined for building libc, how many we actually created. So you can tell from the discrepancy in the numbers that this was an update build. It's nice to see lots of directories where you spent zero seconds doing absolutely nothing. Because that means that you don't have bugs in your make files that are causing things to be built unnecessarily. Over. Oh my gosh. Could it say something? All right. I'm curious. Anybody interested or want questions or anything like that? Because I can skip straight to the end. Where are we? Yep. So you can handle the writing itself as one of the dependencies. Yes. So you can do that distribution, it's really good to know from doing anything about the build, so you could like, you're at the same height of the build. Everything. If you want to. If you do distributed compilation like using, say, something like DCC, you can do that. It's a. See, you have to question the other side of the compiler that you do. Yes. Absolutely. If you do distributed, it's just a little more solution. Yep. Then you can send the compiler over. Yeah. So the way we do that is, if I'm the master of the build and I'm going to ask you to build something, I actually make you look at my tree and you will read the list of tool chains that you're supposed to use from that and you will use the same tool chains by default. Otherwise you'll explode and we won't do anything. You can do the same thing with DCC as in you send the job to a machine that you know has the tool chain that you want to have used. There's any number of ways to skin that cat. Any other questions? All right. Hang on. Woop, woop, woop. There you go. All right. So it may be the greatest thing since sliced bread, but it's maybe not for everybody. But it does provide a simple solution to some rather complex problems. And if anybody has any questions, there's some further reading and you can always catch me at s.o.g. at free sd, netbisd, juniper, and so on and so forth. Thank you. Thank you.
 Let's see if we can make this do something.  Good.  We're here to talk about building FreeBSD with Beemake.  I'm going to do a quick intro.  The question came up recently on the developers list, why  Beemake.  I'll briefly talk about that.  I'll talk about the transition of FreeBSD to using Beemake.  And one of the reasons for giving this talk is to  provide a venue for people who have questions to ask them.  So if you have any questions, feel free.  I'll then talk a bit about the metamode and DERDEPs builds  and projects Beemake where you can find them and play with  them, which was, to a large extent, the reason for this  whole exercise.  So I started Beemake back in the early 90s up until then.  I'd been mostly a sun-o-wes guy.  Because sun in their wisdom didn't provide a tool chain.  Everybody used Gmake and GCC and so on.  And in 93, my sun workstation got hit by lightning.  And I decided that I had to replace it.  And I didn't really fancy spending another $20,000 on  another sun workstation when I could spend $3,000 on a PC  and run $386.  BSD.  And so that's when I discovered the BSD make.  And it didn't take very long to decide that that was a winner.  So Beemake is basically net BSDs make with autoconf added.  Much as I liked building on BSD, net BSD in particular at  the time, most of my clients used crazy stuff like HPUX  and UTS and AIX and all sorts of nonsense.  And so I wanted to be able to have a nice portable build  that was as nice as the BSD build, but portable to  everything under the sun.  And that's what I got with Beemake.  I've personally run it on everything from AIX to UTS  and pretty much in between.  Back in 2000, when I started at Juniper, their build was in a  fairly sorry state.  And my first project was to fix it.  And so it's been building with Beemake since 2000.  And to a large extent, Juniper pretty much funded me to  develop make for net BSD and anybody else that wants it  for the last 14 years.  I put a lot of features into net BSDs make in that interim.  And one of the things that always surprised me was,  although I didn't do a lot of evangelizing within the net  BSD community of these features, almost without  exception, within a week of me putting some new cool feature  into make, it was being absorbed into the net BSD build  without me even mentioning it.  And I always thought that was pretty cool.  So up until fairly recently, the Juniper build had diverged  considerably from free BSD.  Juniper, we run Junos, which is basically free BSD.  And we modified the build considerably.  And we never really bothered tracking the free BSD build  until a few years ago when we decided that we wanted to move  to an environment where we had free BSD building as close to  stock free BSD as we could and then adding our Juniper bits  on top of it.  And then all of a sudden, we had the situation of, well,  we don't want to have to continually retrofit our build  to free BSD.  And we don't think free BSD are quite ready for swallowing all  of our build.  So let's see if we can meet somewhere in the middle.  And so 2011, I gave a talk here and we had a chat to  some of the free BSD folk about whether there was any  interest in picking up some of our build technology for free  BSD.  And there was actually considerable interest.  So we started the process.  And so let's see.  The first commit of being made into free BSD head, October  12, 2012, we had a bit of a headshot.  We had a bit of a hiccup with the ports.  We needed to get a ports run to confirm that we weren't going  to destroy ports.  And they had to rebuild all their infrastructure, which  delayed that for a while.  But nonetheless, free BSD 10 shipped with building with  be made by default.  It's been back ported to 9.3, so the port  folk can use that.  And I have to say none of this would have been successful  without the goodwill and cooperation of both projects.  There's a number of changes that have gone into NetBSD's  make to make it easier to do this project.  The NetBSD folk were more than willing to absorb some of  those changes.  And so thanks to all of them on both sides.  So why Bmake?  Somebody asked this quite recently.  And actually Warner answered it for me.  And he put it very well.  Basically, Bmake is little more than NetBSD's make.  NetBSD have a very active team of folk who are contributing  regularly to Bmake.  So it's not a situation if I get hit by a bus that  everybody's through.  There's plenty of people around who understand it and can  support it and maintain it.  It has a plethora of cool features.  Not least is an abundance of modifiers.  And quite importantly, in that regard, I forget some years  ago, I reworked the modifier handling so that you could  use it recursively.  And what that lets you do is stick complex constructions of  modifiers into variables which you can then reuse without  having to cut and paste them everywhere and getting them  wrong.  Things like multiple iterator values for for loops is  quite nice.  There's at least one for loop in the port stuff that would  have been much nicer if it had been able to use that.  Things like being able to auto-ignore stale dependencies  found via dot-depend is actually very helpful.  I did a clean on a FreeBSD tree that I've been doing for  production builds for a year now.  And I had to clean it for the first time a couple of weeks  ago, which was really annoying.  Because we'd switched branches.  I had to switch to a new branch.  And there was something I had to clean.  It was annoying.  I'm going to talk a bit about dirt-eps.mk.  It uses pretty much every single feature of Bemake.  And so it alone is a cool reason to do this.  So why meta-mode and or dirt-eps?  It's worth noting.  I usually talk about them as one and the same thing.  They're not.  You can actually use each of them independently.  You could use dirt-eps with a completely manually maintained  build system.  It would work very nicely.  But it's certainly nicer and easier if you have the  meta-mode stuff as well.  Together, they give you a very simple and reliable and  maintainable solution to building complex chunks of  software.  The FreeBSD build is a decent size.  The Junos build is a lot bigger.  And building large sets of software in parallel is  actually a somewhat complex task.  But we like to make it as simple as possible.  One of the nice things about the dirt-eps build in  particular is the build works exactly the same way regardless  of where you start.  So if you like to edit bin cat and run meta-ex compile  with any Macs such that you're effectively doing a CD into  bin cat and running make from there, it will do all the  right stuff even if it's a clean tree.  Similarly, if you're at the top level and you want to do a  quote production build or you just want to build bin cat and  so on, it works exactly the same way.  I'll probably cover it again.  If you look at the top level make files in, say, FreeBSD or  NetBSD or Junos, if you could, in the pre-meta-mode days,  there's some pretty large make files.  I think FreeBSD is currently up to about 1,800 lines.  The pre-meta-mode Junos build was 5,000 lines.  And I could guarantee that there would be very few people  in the company that could actually read it all and  follow what was going on.  When we cut over to the meta-mode stuff, that was  reduced to less than 200 lines.  And 200 lines that virtually anybody could read.  So it's affordable.  I don't know about you guys, but at Juniper I have close to  2,000 developers.  And believe it or not, a lot of them don't log their builds.  And most of them are not build geeks.  And so when something goes wrong, they send an email to  BuildTrolls.  I updated my tree and it doesn't work.  Yeah, a bit more info would help.  We actually, actually, you'll see, I've got an example.  We actually trained Bmake years ago to spew mountains of  useful data that is exactly what we need to see when you  say, my build broke.  And we document all this stuff.  And he says, if you want to report a build problem, Bmake,  when it blew up, it spewed out.  All this info, please include all that in your problem  report, and you'll get much faster resolution.  Well, of course, you never get any of that.  And when you ask them, well, where's your log?  Oh, I don't have one.  And so you're very often stuck with saying, well, go do it  again, log it, and come back to me.  With the metamode stuff, we actually log the critical  info whether they want us to or not.  And so very often, all the important data you need to  analyze a failure is there.  And everything works better.  All right, here's a quick teaser.  It's very hard to make a build log look interesting.  I'm sorry.  This is building BNSH in a clean tree.  I usually use bin cat, but some people think bin cat's a  bit simple.  So BNSH is a lot more complex.  So here we are building BNSH in a clean tree.  We just seeded into his direct, well, actually we seeded  into BNSH, and we said make minus j8.  And he just went crazy and did it all.  And you can see, and how long did it take?  It took 58 seconds.  It's not a particularly fast machine.  So that was building libc and all that sort of stuff.  The things to note are the object directories were  created automatically.  There was no make, depend, done anywhere.  Everything ran in parallel, but in the correct order.  The log was easy to read.  There was no noise about compilers and all that sort of  stuff.  We only built that which was necessary.  And we visited the leaf directories directly.  There's no tree walks involved.  And at the end there, in this slide, you can see the last  thing he did was update make, file, depend, or at least  check whether it needed updating.  All right, try it again.  So this is a quick look at one of those make, file, depend.  This is actually, I think, the one for BNSH.  In the Junos build, because we're always cross-compiling  and we're always building for multiple architectures at  the same time.  We actually, by default, use a make, file, dot, depend, dot,  dollar machine so that we can have them updating in  parallel without any concerns for contention.  When we looked at doing this for FreeBSD, we figured that  was going to be a little bit too much.  And so one of the exercises we went through was to look at  making it all work with a simpler model.  And it's actually worked out very nicely, I think.  So what we have here, you can see from the client, it's an  auto-generated make, file.  That means you don't touch it.  The first thing it does there is it sets a variable that  basically represents the relative location of this  directory within the source tree.  And that's key to how the whole thing works.  Pazder is one of those magic variables that  B makes sets for you.  It represents the directory where it found the file that  it's currently reading.  And one of the modifiers that B make has, it says you can take  the value that you've got here and you can pass it to a real  path and turn it into an absolute path, which is  extremely useful for this process.  The key data in this file, apart from that, is DERDEPs,  which is a list of the directories.  And again, these are those directories relative to the  top of the source tree that need to be built before we  can build BNSH.  And I mean, it's pretty simple to read.  The only complexity here, if you want to call it that, is  we substitute the variable CSU DER for what it actually was.  This allows us to use the exact same list of dependencies  regardless of the architecture.  I386 is a bit of a wart in that for most architectures, the  CSU DER is easily derivable from the machine value, not  quite so for I386.  And then the last thing it does is include DERDEPs.mk.  And also at the end there, it will capture, if necessary,  any local dependencies.  These are what allow us to do a parallel build in a clean  tree without doing make-depend.  This is effectively capturing just the stuff within each  directory, which are the sort of things that would normally  break a parallel build in a clean tree.  We'll look at this in more detail later.  So building previous, the projects we make, as I briefly  mentioned, is a test vehicle.  It's an exercise in putting all of this building into a tree  which is very close to head.  It was synced last week from head.  And it uses all the metamode and DERDEPs and all that sort  of thing.  The goal is to be able to easily cross-build FreeBSD as  close to stock as possible, while minimizing changes to  FreeBSD itself.  Within Juniper, we have a team of FreeBSD folk.  We've been building head and stable 10 now from back around  9, using almost exactly the same setup.  There's a bit more stuff there.  For instance, we use external tool chains.  And we generate packages which are basically ISO FS's and  all that sort of stuff.  And disk images for feeding into VMs.  But from the pure build point of view, there's very little  difference.  So just quickly, on the transition for FreeBSD, we  first talked about it with the FreeBSD folk in 2011.  Converting the base to use Bmake, which is the first step  to be able to do any of this, was actually very simple.  It took a patch of less than 300 lines to allow the FreeBSD  head to build using Bmake instead of Fmake.  And we're going to use Fmake to represent the old FreeBSD  make, just to avoid confusion.  Ports was a little bit more complex, because they have to  support building on older versions of FreeBSD.  At the time, it was 8.3, I think.  And they don't branch.  So that makes for a bit of a challenge.  More importantly, it's reasonably simple to fix the  base.  It's a bit more complex to fix ports.  The most difficult problem is all of the thousands of people  out in the world who are using FreeBSD and things derived  from FreeBSD who we have no visibility into.  And for those people, it's a bit rude to go and pull a rug  out from under them.  And so as we'll see, we made a number of changes to Bmake to  minimize the impact on those people as we go.  And so again, just call if anybody does have horror  stories or something like that, feel free to let us know so  that we can maybe do something about them.  So there were changes made to Bmake to support FreeBSD.  Almost without exception, those changes go into NetBSD  first, and then get imported back in.  One of the goals of the exercise is to minimize  divergence between FreeBSD and NetBSD with respect to make  so that we all benefit.  So both Bmake and Fmake are descended from Adam DeBoer's  Pmake from Sprite.  They've diverged quite a bit.  One of the most obvious differences is the colon U and  L modifiers, which I think FreeBSD got from OpenBSD,  around about the same time that I was sticking all the OSF  modifiers into Bmake, which, oddly enough, had colon U  and colon L as well.  Fortunately, these aren't used by the base system at all.  And as of last week or this week,  Ports doesn't use them either.  But for the purpose of our story, it's still an issue.  Or it was an issue.  The other, I guess, key difference is Bmake is,  aggressive is putting it nicely with respect to  $.path.  It will find anything by a.path that it possibly can.  And that isn't always what you want.  So you sometimes have to curb his enthusiasm.  And there's a.noPath thing to do that.  Fortunately, just simply saying.noPath, $clean files,  covers 99% of the problem.  NetBSD's bestyone.make marks all of the standard targets  as both phony and not main.  So that was another change.  One of the real fun ones is handling job tokens.  All modern makes, when you do a make minus j8,  you don't want every sub-make to start another 8.  And so on, you get what is it?  Geometric increase in load.  So they use the means of having a token pool of some sort  to constrain that.  Both Fmake and Bmake do that, but they do it differently.  Fmake uses a FIFO, which the first instance, when Fmake  starts up, he has a look to see if the FIFO exists.  And if it doesn't, he assumes he's the master, creates the  FIFO, exports its name, and everybody else then uses it.  Bmake, on the other hand, uses a pipe.  And he passes a magic argument to his children with the  descriptors for the pipe.  And so if Bmake doesn't get that argument, he assumes that  he's the master, and so he goes and creates it and passes  it on to his children.  Both systems work nicely, but they  don't work well together.  In particular, if you had Fmake as the initial instance, he  would export the name of his FIFO to the environment, go and  start a bunch of sub-makes.  And if they happen to be Bmake, each of them would think that  he was the master, because he'd had no clue that he wasn't.  And so he would start another N. It almost works the same  going the other way, except that Fmake would blow up if  Bmake passed him his minus capital J argument.  So you're sort of saved there by that incompatibility.  But if you were careful to cleanse the environment that  you passed to Fmake to not have the minus J, then each Fmake  would think that he was the master, and you'd have the  same problem.  There's other useful stuff in there.  So a quick talk about the changes to the base.  Any of the Bmake-specific syntax we bounded with ifdef.pasta,  that's one of those magic variables that it defines for  you.  We added.nopath to all the generated files and phony and  not main for the standard targets.  We created a.weight as a no-op target for Fmake so that you  can stick.weight into subdur lists and other things like  that.  And we ended up adding an option without Bmake so that  people who, for whatever reason, didn't want to venture  into the sharp pool could avoid it for a while.  Another change, Fmake had a preference for BSD makefile  before it looked for the traditional makefile and  makefile.  Bmake only looked for the traditional stuff, but it's  configurable.  So by simply sticking a list into sys.mp you can make it  look for anything you want to.  And in all of these cases, if we could address an  incompatibility by just looking at the  magic variable in sys.mp, that was the preferred solution.  For the job token one, that's pretty much what we had to  do.  Bmake, in particular, only passes his token pool descriptors  to submakes if the target is tagged with.make, the special  source.  In Fremisty Fmake, the man page says exactly the same  things that it is supposed to do the same thing.  But it doesn't seem to work, which is evidenced by the  fact that somebody went and added this $  underscore plus thing to basically do what.make is  supposed to do.  And so what that means is that the Fremisty makefiles and  presumably the thousands of makefiles written by people  who use Fremisty but aren't in source have not been  necessarily sticking.make on all the  targets where they would want.make if it had worked.  And so the only reasonable solution we could come up  with for that one was to set a little knob in sys.mp to  tell Bmake to say, you know what, just pass your  descriptors to all the children.  And that sort of makes the problem go away.  Oh, the error token is a nice one.  So normally when Bmake fails, this is in jobs mode.  He sticks an error token back into the token pool.  And the next time all the other submakes go to take a token  out of the pool, they get the error token.  They say, oh my god.  And they bail.  Which is wonderful because your build stops immediately on  error, which 99.999% of the time is exactly what you want.  Apparently it's not what you want for make universe.  In that case, people would prefer that if some variable of  MIPS is broken, that 99.36 and all the rest could just  chug along and get as far as they could.  And so we added another knob to say, you know what, if this  make job error token is set to false, then don't stick an  error token into the pool when you die.  And so in the top level make file, if it sees that you're  doing make universe, it'll set that knob to false.  And so you can get the make universe behavior that you  want and still get the nice behavior for all the normal  cases.  Oh, another.  This is another fun one.  So in the long ago, make use set minus e to fail things on  statements.  I don't recall exactly why.  But I think it was mainly to get more consistency in target  handling between jobs mode and compact mode.  In NetBSD, they did away with all that.  And they made it work very reliably such that the unit  of failure was the command line rather than statements  within the command line.  So with fmake, you could do CD,  to some bogus directory, semicolon, rm minus rf star.  And it would be reasonably safe because unless you had a  directory called that, it would fail at the CD and never get  to the rm minus star.  If you fed that same command to bmake, though, he would  happily remove everything in the current directory, which  may or may not be good.  Well, the object directory anyway, if you had one.  The format at the bottom is, of course, safe with any version  of make and is therefore preferable.  And while it would have been a quite simple exercise to go  through all the make files in previous d and fix them so that  they did the latter form instead of the former, again,  that doesn't do anything for the thousands of people out in  the world who may not get the memo.  And so we had to do something about that.  So again, this is one of those things that you can configure  with make.  So we just defined a shell for previous d to use by default,  which effectively makes bmake behave the way fmake does and  use set minus e by default.  The option, the bmake option.  So I think when we originally proposed adding bmake to make  to previous d to start this transition process, we had in  mind that we would just add bmake.  It would get installed as bmake and people could choose  whether they used it or not and so on.  That wasn't considered ideal by the project.  They wanted us to do the cut over as soon as possible and  effectively, one day you'd go to bed and it was fmake and you'd  wake up the next day and it's bmake.  And that would have been feasible for the very first  part of the project itself, but it's a bit more of a challenge  again for the third parties.  And so we ended up having to add that option so that people  who didn't want to face the transition right now could  avoid it.  And so we added this option source makefile had to do the  right thing, which typically involved it has a dance for  you and you're upgrading to recognize that the make that  you have on the system is not quite adequate for building  the new version of software and it would go and bootstrap a  new version of make and so on.  And once it existed, it would use it.  But it always called it make.  And so he would say, oh, I have this temporary version of  make, I'll go use it, even though it might have been  six years old.  So we fixed that so that if you wanted to be building with  fmake, it made sure that temporary make was called fmake.  And if you wanted to bmake, it would make sure that  temporary make was called bmake.  And if you change your mind, it would do the right thing.  That's all being ripped out now.  Thank you, Wana.  The apparently was broken for a while.  Anyway, nobody noticed.  So obviously.  That's OK.  It's OK.  It's gone.  So we've burned the bridge.  So that's progress.  I mentioned the colon U and L for ports.  I'd had a cunning plan for how to deal with bootstrapping  ports, which worked brilliantly unless you were the  portsfolk, because of the way that they want to be able to  build ports.  And so that wasn't going to work.  So yet again, we added a local hack for ports to be able to  set a knob saying, I really want the old colon U and  colon L.  And even with this set, actually, in many circumstances,  bmake can do the right thing, because it can tell whether  it's looking at a colon U with a value after it, or  whether it's looking at a colon U with nothing in which  case it might be the old modifier.  But again, that's due to be removed after I get back  from this conference.  Another thing that fmake had, which netpstmake didn't have,  was the tolerance for quoted strings as iterator variables.  And so that was added to netpst.  And then I think one of the final ones was the  minus v behavior.  If you do fmake minus v foo, it spits out the fully  resolved value for you.  If you do the same thing for bmake, it will give you the  literal value, which may or may not include other variables.  This is brilliant if you want to do debugging of what's  going on.  I use it a lot.  And with bmake, you can always put dollar foo in there to  expand it fully.  So this is great for command line type stuff, but it's not  necessarily ideal for the build itself.  For the build itself, the fmake behavior is actually better.  And so I proposed changing the default behavior and adding a  debug flag so that you could give me the literal value.  We're compromised by adding yet another knob to say which  behavior you like.  So revistd sets make expand variables to true, in which so  you get the default revistd behavior.  But regardless of the setting, you always have the debug  flag to give you the literal value.  And so both projects are able to have the semantics that they  know and like.  Oh yes, this was the last one.  So bmake, some years ago, they switched the way iterator  variables within for loops are handled to avoid namespace  pollution.  And so they use this colon u value.  So what the colon u does is say, if the variable doesn't  have a value, use what follows as the value.  And so we have effectively a null variable.  And so we're going to use value as the value.  And this avoids, as I say, namespace pollution.  Trouble is it makes it very difficult to try and play  games like escaping iterator variables within for loops.  And especially when you have nested for loops and you've  got to do multiple escapes to try and escape the escapes  within the for loops and so on.  And I looked at this one for a while and I thought, you know  what, let's forget it.  Let's just use the inline loops, which are much easier to  follow.  So I basically replaced a 20 line block of nested for  loops with a single line, which produces exactly the same  output, of course.  Oh, OK.  All right.  Did we have a pointer?  There was a pointer at some stage.  Here is one.  Doesn't matter.  I can't reach.  All right, so this thing is the inline loop modifier.  The first thing that follows it is going to be the  iterator variable.  And then everything between the next at and a closing one.  And this isn't the closing one because this is introducing a  nested loop onto the end.  So the closing one is like this followed by another colon,  for example.  Oh, brilliant.  Thank you.  Hope I don't kill anybody.  Ah, excellent.  All right.  This is so cool.  All right, so in the man page we mentioned that in the OSF  development environment where this came from, they actually  use a convention of uppercase variable name with dot  c the end for the iterator variable.  That's really hard to say.  But that's really hard to type.  I'm lazy, so I prefer single character iterator variables  because then you can just use dollar p unless you need to  do modifiers on it.  So anyway, this is two nested loops.  So the first thing we do is for the Mlang stuff, we're going  to stick man slash at the start of it.  And if we find that we've got an empty one, we get rid of it.  And then for those, so each value that we've got at that is  going to be an M valuable.  And then we're going to also iterate through the M links.  Hang on.  We're going to come up.  What are we doing?  Yeah, so each of those is going to be p.  So we've got a p, so there's another loop.  So we've got p colon e is the extension of p.  And so this is actually referencing a man dollar p  blah prefix.  And then we've got m, which is our, from Mlang.  And then again, we stick the p with his extension at the end.  And unfortunately, when I did a full screen on my terminal  with the normal font, you could actually see the end of the  line.  And it finishes with a couple of colon closed brackets.  This is fire.  The colon at colon u colon l colon d and colon p all came  from the OSF version of P make.  And the inline loop operator is particularly wonderful.  Because unlike a for loop where everything is getting  expanded as you read the for loop, in this case, the loop  is expanded at the time of reference.  And so you can do some very cool stuff with it that you  can't possibly do with normal for loops.  So this is saying for each n in what precedes it rather than  for each n in the demo, and then for each p within the  loop?  Yes.  Yeah, I figured you guys would be computer science, but  you probably were able to get the hang of it.  So do you make after the inline loopers, if they  don't have a lot of loops sometimes like this?  Yeah, both of them have their place.  Don't get me wrong.  Dot for loops are definitely useful at times.  But there are things that to do it with a dot for loop, you  have to do obscene things, which are just quite natural  with the inline loops.  All right, so why is b make so cool?  Well, it can do dirt apps.  That's reason enough.  You can see dirt apps in the tree.  It's in contrary of b make mk, dirt apps mk.  I don't recommend reading it on an empty stomach.  I've been doing this sort of stuff for 20 odd years, and I  consider dirt apps one to read on a full stomach.  But anyway, as I mentioned, we have a plethora of modifiers.  One of the ones that somebody added not that long ago is  this one.  So I saw in previous D they have a fixed value that used  for the random seed for C++ compiler.  This allows you to have a repeatable build, but have a  different anonymous initializer seed for every library  that you build, which is actually quite a nice thing.  This is an example of using the parse dir and parse file.  This just ends up being a token that you can use.  If you want to create a little marker target, makefile.inc,  for example, to test whether that makefile.inc has been  included already, you can make a target that includes this  or this, or you can spell this all out, but it's more  readable to just say this.  This will ensure that you'll get a target that is the same  value regardless of whether you read it via an absolute path,  a relative path, or any combination thereof, because  the column TA turns it into an absolute path using real path.  So you canonicalize it.  This one's an example of, as I mentioned, you can stick  complex lists of modifiers into a variable and then reuse it.  And there's an example down here.  So here's a list with a number of variables, and we  basically just apply this set of modifiers to it, and we get  a list of exclusions.  And that sort of thing gets used quite heavily by DERDeps  and lots of other cool stuff.  It can do the equivalent of strf time using either local  time or GM time.  This in the Junos build, we like to have timestamps on the  build logs so that we can see where all the time's going.  This allows us to have those timestamps in the output logs  for essentially free, and so we do it.  And we use this format typically so that you get a  human readable thing.  You get something you can actually do math with and a  handy little token to be able to grep them out.  And one of the tools I have in my kit bag is a little Python  script, which will read one of these logs, and it can  basically give you a progress bar so you can see how long the  build's got to go until it's done.  By the way, I'm not going to get through all these slides.  I've stuck a number of slides in towards the end to just  cover random stuff that people may or may not be interested  in, so don't be concerned that we're going to run out of  time, because we will.  So it was last sync.  The Projects Be Make branch is the one that hopefully most of  you are interested in.  This was last synced about a week ago, and as of this  morning, it completely builds again.  The transition to clang through some spanners into the  work, but I got some help from David last night to sort that  out and finally finished getting it all building again.  As I mentioned, unlike the Junos build where we use this  format of MakeFile.depend pretty much universally, in  FreeBSD we use this format.  There are, I think, about five places where we do this, and  that's invariably because there are places where the  auto-generated, the locally-generated files that you  need to capture the dependencies for have names that  vary with the architecture.  And so the way the build-in for here works is if you had  previously built one of these directories for, say, I386,  and therefore there was a MakeFile.depend.I386, and  then you then went and built it for MIPS when he gets to the  end and says, oh, I need to update MakeFile.depend.  He says, oh, I see there isn't one, but I see there's a  MakeFile.depend.some other machine.  I better do the same thing, and he'll just automatically  create a MakeFile.depend.MIPS for you.  And so all the right stuff happens.  The packages MakeFile acts as a top-level MakeFile.  In the Junos build, the source MakeFile just includes  packages MakeFile.  It just makes it simpler from transition point of view.  And in the FreeBSD build, we do the same thing.  We leave the FreeBSD top-level MakeFiles alone so that we  can still do build world so we can make sure we're not  breaking FreeBSD when we want to commit stuff back up from  that environment.  But we don't want to use it for our stuff.  So this does the job.  Under packages, in the normal world, you would have a whole  bunch of packages, foo, to build a package called foo.  We use the pseudo directory as a place to put these all  represent targets because they happen to have a MakeFile.depend  file under them.  And so when you go into packages and you say Make  Bootstrap Tools, for example, it knows that that's  something it can build.  And it also knows because it's got.host that it's  something that it has to build for the magic machine called  host, which happens to be whatever host you're building  on as opposed to the host, maybe i3-at-6 or AMD64.  But it's not necessarily the same thing as an i3-at-6 or  AMD64 target because you may be building FreeBSD 10, but you  might be building on FreeBSD 7, which is unfortunately what  we're doing at the moment.  But it doesn't matter because it works.  So for the environment, I think in most of these examples, I  use a little tool called MK.  Its join life is from wherever it is to have a look  upwards until it finds a file called sandbox-env.  And wherever it finds that file defines the top of the  quote sandbox, and he sets a variable SB to that location.  And then he reads that file up to condition his  environment.  This is great if you're an Emacs user and you're editing  branch, I have trees from NetBSD, Junos, FreeBSD, all over  the shop, and you can be editing them all within the  same Emacs and just do metarex compile in any of them.  And it'll just do the right thing.  But if you don't like using wrappers and so on, all you  really need to do is make sure that you have makesyspaths  set, and everything else that needs to be done can be  done by, say, local sys.mk or anything else.  Projects.bmk has a local sys.mk, which I've checked  into the tree.  The main reason is it contains a lot of stuff which I  haven't taken care of yet.  But generally speaking, you can build all of use-land,  kernel, tool chains for both the host and target.  I added a boot.  Clang 3.3 is unable to build Clang 3.4, it seems, or at  least the table gen can't handle the headers and so on.  So you very likely need to bootstrap the tool chain in  order to be able to get building with the new Clang.  So I added a target to do that.  And since bootstrapping tool chains is not a  wheel worth reinventing, I just made it leverage the  existing targets in the source makefiling one.  We're now using sysroot, which works much better with  Clang than pretty much any alternative.  You can do build world.  You can have build world work, like build world, but  produce metafiles along the way.  And that's actually very useful both for comparing  against when you're scratching your head as to why Clang  built when you did build world versus not building when  you didn't.  You can actually go look at all the syscalls involved and  see what was going wrong.  It also gives you a means for bootstrapping makefoldup  depend if you were wanting to start this  exercise all over again.  So here's a quick thing about getting started.  So minimal environment setup.  Once you've got that, you should be able to just do make  bootstrap tools.  You'd want to do at least J8, because it's going to take a  little while.  If you're not using my little MK wrapper and you want to  then build the tool chain again, and you remember the  days of building GCC where you used the host compiler to  build stage one and then used stage one to build stage two  and eventually.  Same sort of thing.  You could skip this step, but you would have to add this  with tools to these ones to basically use the set of tools  that you've built in this stage.  It's probably neater to go through this step once and  then put it all aside and be done with it.  And then you can have fun cd-ing to random places and  building them.  Or the lot is a pseudo target that just basically builds  everything.  And again, it should all just build in clean tree.  Debugging failure.  I was talking to Kasey or somebody last night.  And so I gave him this example as to how it's cool.  So I mentioned before, nobody logs their builds and they  get build failures and they give you absolutely useless  information.  These days, though, as long as they can tell you where their  tree is, we can go find the clues.  And as an example, we'll edit bincat and sticker a bogus  include in there.  And then we go and compile it.  And lo and behold, it explodes.  Oops, yeah, couldn't find that.  And b makes spews out.  This is some of the noise I mentioned that b makes spews  out on failure.  This is all configurable, of course.  By default, it doesn't do any of this.  You have to set some variables.  But you can set it to spit out any variables you like.  So we have a number that we like to see.  The level is useful, the current directory, the object  directory.  So it spits out the target that it was building, the  metafile that it was creating.  And it will actually copy this metafile into an error  directory at the top of the tree so they're easy to find.  And if you're doing things like I have a continuous build  machine, and so he knows to go, build failed.  The first thing he does is look in the error directory.  If there's anything in there, that's why the build failed.  That makes the analysis very simple.  And so here's the metafile that got copied into  source top dot dot error with a name that we can easily find.  And yes, there we go.  Even if the guy hadn't logged his build in, this error  had actually happened 5,000 lines earlier and is not  possibly in his scroll back anymore, we can still go  find what the exact error was, why it was happening.  We can go and see which version of the compiler he was  running and all of the files that were being read and so on.  And it may be that, oh yes, you've got the wrong version  of the compiler, or no, you tweet your spell.  Oops, wrong.  But one way or another, you have the information that you  need to be able to work out what was wrong and do  something about it.  For next steps, we think it would be useful to actually  add a bit more to it so that you could, for instance, have a  target in the Projects Be Make branch that would let you go  and effectively just build a little bootable VM image  that you could then throw into Be Hive, which would be  quite useful.  Another thing that may or may not be interesting is  distributed building.  This is actually very easy to do with DERDeps.  But that's probably the subject for a whole new talk.  But if anybody's interested, I can bend your ear on that.  Are we going?  OK, so now we're into just the almost random stuff.  But one of the things I was talking about with Warner  recently, and he's thankfully been doing, is separating the  BSD.star stuff from, so you can think of these things as  classes of make files.  The BSD.star.mk files are things for building stuff in  the BSD way.  But you don't necessarily want to have them only able to  build user source.  And you can have source.ops, for example, as options which  only apply to building user source that you don't  necessarily install, but are still useful.  And similarly, you can have local.star.mk, local.com for  anything else you like, as things which are either site  local or tree local.  They're not part of the repository, per se.  But they allow you to do things like customization  without hacking the original source.  If you look at DERDeps.mk, most of the meta and similar  make files, I include a.local, whatever the name of the  thing is, to allow customization without having to touch  the original make file, which in the case of something like  DERDeps.mk is very important.  Because, like I said, I only touch it when I'm sober.  And I would recommend others do the same.  Actually, I would recommend not touching it.  So source.ops, this is something that has been very handy.  It's now something that we can include within those make  files that depends that we use in the pseudo targets.  So for instance, this one is for the tool chain.  So we can now say, well, is make clang define?  No, it's not.  Oh gosh, let's go include source.ops.  Now we know it will be defined.  And we say, is clang yes?  OK, we'll add clang to the set of stuff we're going to build.  Is GCC yes?  We'll add GCC, and so on and so forth.  And that way, you can easily handle building all the  optional stuff.  One of the other things that we saw earlier was auto  creation of object directories.  A nice feature of Bmake is any source that you give to the  special target object sets, the object.  It has to exist, of course.  So this little dance up here makes sure that it exists.  And this is how we do auto creation of object directories.  If you are going to do this sort of dance, though, you have  to do it very early in the game.  Because this has to be done before you start reading  make files that are going to potentially  influence.path.  Because when you do.path some directory, if it doesn't  exist, it doesn't get added.  And there are cases where it will exist.  But if you do it too late in the game or too early in the  game, you'll get a path that's in the source tree, rather than  the object directory, which was what was intended.  So again, if you want to play with auto object directories,  you really need to do it during the sys.mk phase.  But this works really well.  I was talking to, again, Chat last night, 10 something years  ago within Junos.  We went through the whole phase of the camp of people who  like to dribble object directories in the source tree.  You had the camp who liked to have just a single directory  called obj.  And then you had the other guys who were building  multiple archers.  So they wanted obj.dollarmachine and so on.  So we got to a point of having five different ways to  support building the tree for all these various  combinations.  And it just got ridiculous.  So we just said, forget it.  We're going to create object directories automatically.  And no questions asked.  And it allows you to simplify things enormously.  And simple is good.  All right, so since nobody's asking any questions, I'll  waffle a bit about meta mode.  Basically, what we do is when we build a target, we create,  amazingly enough, a file called target.meta.  And it collects stuff like the expanded command line, which,  if all you did, would be useful.  Because that allows you to say, all right, last time I  built this target, I used that command line.  Now I'm building it, and I'm going to use this command line,  and they're different.  The target is out of date.  And this allows you to do things like skipping.  You've all seen make files where you have comments like,  oh, if any of the make files change, we better regenerate  this in case.  And that can lead to all sorts of unnecessary compilation.  By being able to be sure that the target is out of date,  because its command line actually was impacted, you have  a much better situation.  We capture any command output, which, nine times out of 10,  is just the error.  In most cases, it's empty.  And we capture all the interesting system calls.  By interesting, I mean the ones that are interesting to make.  And most typically, he's interested in read, execute.  He has to track changes of directory so that he can  work out path names and stuff like that.  But mostly read and execute is what he cares about.  And why do we do this?  Well, the automated capture of all this information  helps us optimize the build and improve reliability.  And by optimizing the build, I mean doing as little as possible,  doing it in parallel, but doing it correctly.  And capturing the command output, again,  makes failure analysis much easier, even when people  refuse to log their builds.  Metamode and Dirt Eps help with all of the above.  As I mentioned, we don't do make to pen.  We still do make to pen within our build for the kernel,  because at least last time I looked,  there were certain files that only get generated via the make  to pen phase of the kernel.  But I'm told by O'Brien that maybe even that's not necessary  anymore, so we may be ditching that soon.  But for the rest of the tree, we never do make to pen.  And it saves you a lot of time.  And while, even before we had all this,  we were using GCC to dribble out info  so that you could auto-gather dependency information,  using the filemon kernel module lets you gather all that  information for everything, not just the compiler.  Let's see.  And so you automatically catch tool change changes,  all that sort of stuff.  Also, here's the example of an unnecessary recompile  that you can avoid.  You can use things like DP add to bootstrap your dependencies  if you're just getting started.  And interestingly enough, you can get into problems where,  if you have actually spurious entries in DP add,  you've said, oh, DP add, I need lib foo-mumble,  but you don't actually use it.  It won't be captured in the auto-captured dependencies  because you didn't use it.  And then you get to the ultimate directory,  and it says, well, I can't do anything  because foo-mumble's not there.  It says, well, that's because you didn't need it.  So once we've finished processing  DERDAT DP add to add C flags and stuff  that we may or may not want to as a result,  we actually throw it away to avoid those sort of errors.  Some targets will need work.  For instance, it's quite common to have a target that  is going to include some kind of timestamp.  You may or may not want that target  to be regenerated every time you build.  If you don't, you can actually tell make,  for this particular target, I don't  want you to do command line comparison  because that would make you do the wrong thing as far  as I'm concerned.  You can actually be clever about it.  Make knows that if your target actually involves the.o-o  date, does anybody use that these days?  It's basically expands to the list of sources  that are at a date with respect to the target.  And therefore, the expansion of it  is going to be different almost every time you do it.  And so there is no way that you can reasonably  compare a command line that involves any use of this variable.  So make knows this.  And so if he spots this variable in the script,  he'll say, well, I can't compare that command line.  I'll skip it, which means that you can cheat  because you can deliberately stick that variable in there,  match something that will never be in there.  And so you actually have a token here  which will never expand to anything in the command line.  But it'll be seen by Bmake, and he'll say, oh,  he doesn't want to compare this command line.  But you can still come.  So in this target here, we won't regenerate just  because the timestamp changed.  But if you change anything in DP add,  that changes the command line.  And so we will regenerate the target.  You enable it just by setting the word meta in a variable  called make.mode.  It gets read by all the make files.  It gets looked at after you've read all the make files.  So your make files can control this variable pretty much  anytime.  And effectively, meta.sys.mk does a bit more.  But the key bit it does is it says, well, the target all  depends on dirt-eps.  And then we're going to wait for that to finish.  As in, we're not going to do anything else until dirt-eps  is finished.  And that line there, if you like, is the magic that makes  the whole thing work.  Well, there's a bit more to it, but you get the idea.  So when we write them for $target, we create $target.meta.  Excuse me.  We skip it normally for any quote special target, like  begin and error.  Unless you also sometimes you do want a  metaphor for these, in which case you can always stick the  special source dot meta on there to say, I really want a  metaphor for this.  It's never created if the target is flagged.  No meta.  I don't care what you think.  I don't want a metaphor for this place.  And normally, metafiles are skipped if the object  directory and the current directory are the same.  But in the case of, for instance, building the kernel,  where we try to leverage the existing kernel build to the  extent possible, so we actually do the CD into the object  directory and run make in there.  And in that case, we do want metafiles.  And so we stick a little knob in make mode to say, making  metafiles in the current directory is OK.  And lastly, if the target that you're building is outside of  your object directory, he will create a metafile with all  the slashes replaced with underscores, just so that you  end up with a file that you can reference  within that directory.  I think we've covered the content.  Important to note, there are two variables that are used to  track metafiles.  MakeMetaFiles contains a list of all the metafiles that we  read or examined or in any way knew anything about while we're  building this particular directory.  And makeMetaCreated is all the ones that we updated.  And so this allows us, when we get to the end of having built  this particular directory, if this variable is empty, we  did nothing.  We know that we don't need to go and consider whether we  should update the makeFile.depends, which can  be a reasonably expensive process, so we skip it when we can.  Oh, if the meta verb, if the meta, we're being metavobose,  then we'll expand the makeMetaPrefix, which defaults to  just building a title name, but you can make it do anything  you like.  And in fact, I have a fairly cunning makeFile that sets this  to a variable that actually is used to track all of the  directories that you visited during the build, so that you  can then go back and reexam them to see if there's any  files that you should have SVN added but hadn't, so you can  throw an error.  That's another very cool trick.  A little bit about filemon.  It's a reasonably simple kernel module.  The original prototype that John Birrell did of this idea  used detrace.  Detrace mostly works.  It doesn't seem to let you reliably get the absolute  path of the program that you're invoking.  There's also a little bit of overhead, but more importantly,  it needed special privileges to be able to do this.  And we didn't necessarily like any of those.  So we did this kernel module.  It's available for FreeBSD, NetBSD, and there is a Linux  version, which they promise me they will make public.  And I will nag them continually until they do,  because it's actually quite useful.  Each record in the metafile that we get from filemon is  the whole thing is designed to be easily parsed by a shell  script.  And the original thing that did the post-pressing was a  shell script.  So we have a single character to denote the system call that  we did.  And then typically you get the tag, the process ID, and the  data.  And the data is 9 times out of 10, just a path name.  In the case of linking and moving, it's two path names  with quotes, just in case there were spaces in the names.  And then in the NetBSD version, I don't bother with  stat calls, because they fill the thing tremendously, but  they're not particularly useful to make.  But they are interesting from a human point of view, I  guess.  When we read them, OK.  So when we read them, and we only read them if Bmake thinks  that the target is possibly up to date.  So if you're reading a make file, and it's a clean tree,  for example, all the targets are going to be out of date,  he's never going to look for any metafiles, because he's  just going to go and do it.  But if the normal rules for make look at all the directory  and so on, and thinks that the target is up to date, then and  only then will he go look at a metafile, and he'll go and  read it.  And one of the first things he does is compare the command  line.  If the command line's different, and you haven't told  him not to, he'll say, it's out of date.  Go do it.  He skips past the command output, and he gets to the  system calls, and he needs to be able to pass the system call  stuff, and for each file that you read or executed, if it's  newer than the target, then the target's out of date.  If one of the interesting ones is, yeah, so we have this  concept of a bailiwick, anybody like the history, that  defines to Bmake the sphere of his influence.  So if he's responsible for populating that directory over  there on the table, and he's reading a metafile that put a  file over there, and it's gone, that means he needs to  redo it.  And so the target will be out of date as a result.  Yep, that covers that.  So one of the nice things that we can do with this, as I  mentioned, you could write all these makefile depends  manually if you wanted to.  But it's nicer to do it automatically.  So when we finish building, we have a tool that can go and  read these things.  And he can look at the make.meta created to know  that he has to do something.  But he actually looks at the full list to go have a look at  what he needs to read to get all of the dependency stuff.  And this whole process is greatly simplified by having  your object directories and your source trees not  overlapping.  So object directories within the source tree  complicate the thing.  So it's much nicer to use something like the model we  use where your object directories are either  completely aside using makeobjder prefix or the  fancy makeobjder that we use where you have  essentially parallel trees.  So here's an example of one.  I forget what this was.  Oh, this was building something in BNSH.  So you can see the things that he read.  Anything that he reads out of an object directory that  isn't the current object directory defines something  that needs to have happened before this  directory can be built.  And that's the essential idea.  And we extract those and we stick them into DERDeps.  You can also have me extract a variable called sourceDERDeps  if you're doing things like originally within the  genus world, we support the idea of subset trees where  you can say, I just want to check out enough of the  tree to be able to build bin cat.  Everybody needs to be able to build bin cat.  And so you can look at the source DERDeps for bin cat  and say, oh, well, that means I need libc and I need  level r and you can follow exactly the same logic for  checking out the tree as for building it.  But we haven't done that for a while and we decided,  because these actually result in far more churn than  these do, we actually took them out.  And we don't bother collecting them in the Projects  Beemake branch.  Mapping object to sourceDERD.  If you're reading things out of the object directory where  they were created, it's a very simple exercise.  You typically just substitute object for source top and  you've got the directory.  Because again, we keep everything nice and neat.  If you're pulling things out of a stage tree, for example,  which is you can just think of it as a dester that you've  done auto install into, you need a little bit of help.  Because there isn't necessarily a directory in the source  tree that corresponds to user slash include.  So what we do when we stage files is we stick a file  called.derdep.  And when the tool that processes these things, when he  goes and reads and he says, oh, the guy used a file from  here called uni standard, he says, is there a uni  standard.h.derdep?  Oh my gosh, there is.  Let's have a look.  And that gives him the value that he needs to stick into .derdep.  And so he can proceed without any further ado.  Logs are still useful, though.  In particular, we can do things like capturing the  number of metafiles that we examined for building libc,  how many we actually created.  So you can tell from the discrepancy in the numbers  that this was an update build.  It's nice to see lots of directories where you spent  zero seconds doing absolutely nothing.  Because that means that you don't have bugs in your make  files that are causing things to be built unnecessarily.  Over.  Oh my gosh.  Could it say something?  All right.  I'm curious.  Anybody interested or want questions or anything like  that?  Because I can skip straight to the end.  Where are we?  Yep.  So you can handle the writing itself as one of the  dependencies.  Yes.  So you can do that distribution,  it's really good to know from doing anything about the  build, so you could like, you're at the same height of  the build.  Everything.  If you want to.  If you do distributed compilation like using, say,  something like DCC, you can do that.  It's a.  See, you have to question the other side of the compiler  that you do.  Yes.  Absolutely.  If you do distributed, it's just a little more  solution.  Yep.  Then you can send the compiler over.  Yeah.  So the way we do that is, if I'm the master of the build  and I'm going to ask you to build something, I actually  make you look at my tree and you will read the list of  tool chains that you're supposed to use from that and  you will use the same tool chains by default.  Otherwise you'll explode and we won't do anything.  You can do the same thing with DCC as in you send the job  to a machine that you know has the tool chain that you  want to have used.  There's any number of ways to skin that cat.  Any other questions?  All right.  Hang on.  Woop, woop, woop.  There you go.  All right.  So it may be the greatest thing since sliced bread, but  it's maybe not for everybody.  But it does provide a simple solution to some rather  complex problems.  And if anybody has any questions, there's some further  reading and you can always catch me at s.o.g. at free  sd, netbisd, juniper, and so on and so forth.  Thank you.  Thank you.
FreeBSD_bmake_and_meta_mode__480p__0_47.png FreeBSD_bmake_and_meta_mode__480p__48_1343.png FreeBSD_bmake_and_meta_mode__480p__1344_8003.png FreeBSD_bmake_and_meta_mode__480p__8004_11063.png FreeBSD_bmake_and_meta_mode__480p__11064_16283.png FreeBSD_bmake_and_meta_mode__480p__16284_16331.png FreeBSD_bmake_and_meta_mode__480p__16332_17135.png FreeBSD_bmake_and_meta_mode__480p__17136_18335.png FreeBSD_bmake_and_meta_mode__480p__18336_22487.png FreeBSD_bmake_and_meta_mode__480p__22488_24347.png FreeBSD_bmake_and_meta_mode__480p__24348_27131.png FreeBSD_bmake_and_meta_mode__480p__27132_29171.png FreeBSD_bmake_and_meta_mode__480p__29172_32279.png FreeBSD_bmake_and_meta_mode__480p__32280_33395.png FreeBSD_bmake_and_meta_mode__480p__33396_33971.png FreeBSD_bmake_and_meta_mode__480p__33972_36047.png FreeBSD_bmake_and_meta_mode__480p__36048_37751.png FreeBSD_bmake_and_meta_mode__480p__37752_40151.png FreeBSD_bmake_and_meta_mode__480p__40152_40667.png FreeBSD_bmake_and_meta_mode__480p__40668_43175.png FreeBSD_bmake_and_meta_mode__480p__43176_45335.png FreeBSD_bmake_and_meta_mode__480p__45336_47231.png FreeBSD_bmake_and_meta_mode__480p__47232_54731.png FreeBSD_bmake_and_meta_mode__480p__54732_59591.png FreeBSD_bmake_and_meta_mode__480p__59592_64319.png FreeBSD_bmake_and_meta_mode__480p__64320_66131.png FreeBSD_bmake_and_meta_mode__480p__66132_68183.png FreeBSD_bmake_and_meta_mode__480p__68184_69863.png FreeBSD_bmake_and_meta_mode__480p__69864_72059.png FreeBSD_bmake_and_meta_mode__480p__72060_73187.png FreeBSD_bmake_and_meta_mode__480p__73188_74183.png FreeBSD_bmake_and_meta_mode__480p__74184_76655.png FreeBSD_bmake_and_meta_mode__480p__76656_77603.png FreeBSD_bmake_and_meta_mode__480p__77604_80471.png FreeBSD_bmake_and_meta_mode__480p__80472_82667.png FreeBSD_bmake_and_meta_mode__480p__82668_83411.png FreeBSD_bmake_and_meta_mode__480p__83412_84767.png FreeBSD_bmake_and_meta_mode__480p__84768_86135.png FreeBSD_bmake_and_meta_mode__480p__86136_88619.png FreeBSD_bmake_and_meta_mode__480p__88620_89759.png FreeBSD_bmake_and_meta_mode__480p__89760_91787.png FreeBSD_bmake_and_meta_mode__480p__91788_93743.png FreeBSD_bmake_and_meta_mode__480p__93744_96215.png FreeBSD_bmake_and_meta_mode__480p__96216_96299.png FreeBSD_bmake_and_meta_mode__480p__96300_98495.png FreeBSD_bmake_and_meta_mode__480p__98496_99923.png FreeBSD_bmake_and_meta_mode__480p__99924_101735.png FreeBSD_bmake_and_meta_mode__480p__101736_103247.png FreeBSD_bmake_and_meta_mode__480p__103248_104027.png FreeBSD_bmake_and_meta_mode__480p__104028_106595.png FreeBSD_bmake_and_meta_mode__480p__106596_106643.png FreeBSD_bmake_and_meta_mode__480p__106644_106691.png FreeBSD_bmake_and_meta_mode__480p__106692_107195.png FreeBSD_bmake_and_meta_mode__480p__107196_107243.png FreeBSD_bmake_and_meta_mode__480p__107244_107291.png FreeBSD_bmake_and_meta_mode__480p__107292_107339.png FreeBSD_bmake_and_meta_mode__480p__107340_107387.png FreeBSD_bmake_and_meta_mode__480p__107388_107435.png FreeBSD_bmake_and_meta_mode__480p__107436_107475.png
Let's see if we can make this do something. Good. We're here to talk about building FreeBSD with Beemake. I'm going to do a quick intro. The question came up recently on the developers list, why Beemake. I'll briefly talk about that. I'll talk about the transition of FreeBSD to using Beemake. And one of the reasons for giving this talk is to provide a venue for people who have questions to ask them. So if you have any questions, feel free. I'll then talk a bit about the metamode and DERDEPs builds and projects Beemake where you can find them and play with them, which was, to a large extent, the reason for this whole exercise. So I started Beemake back in the early 90s up until then. I'd been mostly a sun-o-wes guy. Because sun in their wisdom didn't provide a tool chain. Everybody used Gmake and GCC and so on. And in 93, my sun workstation got hit by lightning. And I decided that I had to replace it. And I didn't really fancy spending another $20,000 on another sun workstation when I could spend $3,000 on a PC and run $386. BSD. And so that's when I discovered the BSD make. And it didn't take very long to decide that that was a winner. So Beemake is basically net BSDs make with autoconf added. Much as I liked building on BSD, net BSD in particular at the time, most of my clients used crazy stuff like HPUX and UTS and AIX and all sorts of nonsense. And so I wanted to be able to have a nice portable build that was as nice as the BSD build, but portable to everything under the sun. And that's what I got with Beemake. I've personally run it on everything from AIX to UTS and pretty much in between. Back in 2000, when I started at Juniper, their build was in a fairly sorry state. And my first project was to fix it. And so it's been building with Beemake since 2000. And to a large extent, Juniper pretty much funded me to develop make for net BSD and anybody else that wants it for the last 14 years. I put a lot of features into net BSDs make in that interim. And one of the things that always surprised me was, although I didn't do a lot of evangelizing within the net BSD community of these features, almost without exception, within a week of me putting some new cool feature into make, it was being absorbed into the net BSD build without me even mentioning it. And I always thought that was pretty cool. So up until fairly recently, the Juniper build had diverged considerably from free BSD. Juniper, we run Junos, which is basically free BSD. And we modified the build considerably. And we never really bothered tracking the free BSD build until a few years ago when we decided that we wanted to move to an environment where we had free BSD building as close to stock free BSD as we could and then adding our Juniper bits on top of it. And then all of a sudden, we had the situation of, well, we don't want to have to continually retrofit our build to free BSD. And we don't think free BSD are quite ready for swallowing all of our build. So let's see if we can meet somewhere in the middle. And so 2011, I gave a talk here and we had a chat to some of the free BSD folk about whether there was any interest in picking up some of our build technology for free BSD. And there was actually considerable interest. So we started the process. And so let's see. The first commit of being made into free BSD head, October 12, 2012, we had a bit of a headshot. We had a bit of a hiccup with the ports. We needed to get a ports run to confirm that we weren't going to destroy ports. And they had to rebuild all their infrastructure, which delayed that for a while. But nonetheless, free BSD 10 shipped with building with be made by default. It's been back ported to 9.3, so the port folk can use that. And I have to say none of this would have been successful without the goodwill and cooperation of both projects. There's a number of changes that have gone into NetBSD's make to make it easier to do this project. The NetBSD folk were more than willing to absorb some of those changes. And so thanks to all of them on both sides. So why Bmake? Somebody asked this quite recently. And actually Warner answered it for me. And he put it very well. Basically, Bmake is little more than NetBSD's make. NetBSD have a very active team of folk who are contributing regularly to Bmake. So it's not a situation if I get hit by a bus that everybody's through. There's plenty of people around who understand it and can support it and maintain it. It has a plethora of cool features. Not least is an abundance of modifiers. And quite importantly, in that regard, I forget some years ago, I reworked the modifier handling so that you could use it recursively. And what that lets you do is stick complex constructions of modifiers into variables which you can then reuse without having to cut and paste them everywhere and getting them wrong. Things like multiple iterator values for for loops is quite nice. There's at least one for loop in the port stuff that would have been much nicer if it had been able to use that. Things like being able to auto-ignore stale dependencies found via dot-depend is actually very helpful. I did a clean on a FreeBSD tree that I've been doing for production builds for a year now. And I had to clean it for the first time a couple of weeks ago, which was really annoying. Because we'd switched branches. I had to switch to a new branch. And there was something I had to clean. It was annoying. I'm going to talk a bit about dirt-eps.mk. It uses pretty much every single feature of Bemake. And so it alone is a cool reason to do this. So why meta-mode and or dirt-eps? It's worth noting. I usually talk about them as one and the same thing. They're not. You can actually use each of them independently. You could use dirt-eps with a completely manually maintained build system. It would work very nicely. But it's certainly nicer and easier if you have the meta-mode stuff as well. Together, they give you a very simple and reliable and maintainable solution to building complex chunks of software. The FreeBSD build is a decent size. The Junos build is a lot bigger. And building large sets of software in parallel is actually a somewhat complex task. But we like to make it as simple as possible. One of the nice things about the dirt-eps build in particular is the build works exactly the same way regardless of where you start. So if you like to edit bin cat and run meta-ex compile with any Macs such that you're effectively doing a CD into bin cat and running make from there, it will do all the right stuff even if it's a clean tree. Similarly, if you're at the top level and you want to do a quote production build or you just want to build bin cat and so on, it works exactly the same way. I'll probably cover it again. If you look at the top level make files in, say, FreeBSD or NetBSD or Junos, if you could, in the pre-meta-mode days, there's some pretty large make files. I think FreeBSD is currently up to about 1,800 lines. The pre-meta-mode Junos build was 5,000 lines. And I could guarantee that there would be very few people in the company that could actually read it all and follow what was going on. When we cut over to the meta-mode stuff, that was reduced to less than 200 lines. And 200 lines that virtually anybody could read. So it's affordable. I don't know about you guys, but at Juniper I have close to 2,000 developers. And believe it or not, a lot of them don't log their builds. And most of them are not build geeks. And so when something goes wrong, they send an email to BuildTrolls. I updated my tree and it doesn't work. Yeah, a bit more info would help. We actually, actually, you'll see, I've got an example. We actually trained Bmake years ago to spew mountains of useful data that is exactly what we need to see when you say, my build broke. And we document all this stuff. And he says, if you want to report a build problem, Bmake, when it blew up, it spewed out. All this info, please include all that in your problem report, and you'll get much faster resolution. Well, of course, you never get any of that. And when you ask them, well, where's your log? Oh, I don't have one. And so you're very often stuck with saying, well, go do it again, log it, and come back to me. With the metamode stuff, we actually log the critical info whether they want us to or not. And so very often, all the important data you need to analyze a failure is there. And everything works better. All right, here's a quick teaser. It's very hard to make a build log look interesting. I'm sorry. This is building BNSH in a clean tree. I usually use bin cat, but some people think bin cat's a bit simple. So BNSH is a lot more complex. So here we are building BNSH in a clean tree. We just seeded into his direct, well, actually we seeded into BNSH, and we said make minus j8. And he just went crazy and did it all. And you can see, and how long did it take? It took 58 seconds. It's not a particularly fast machine. So that was building libc and all that sort of stuff. The things to note are the object directories were created automatically. There was no make, depend, done anywhere. Everything ran in parallel, but in the correct order. The log was easy to read. There was no noise about compilers and all that sort of stuff. We only built that which was necessary. And we visited the leaf directories directly. There's no tree walks involved. And at the end there, in this slide, you can see the last thing he did was update make, file, depend, or at least check whether it needed updating. All right, try it again. So this is a quick look at one of those make, file, depend. This is actually, I think, the one for BNSH. In the Junos build, because we're always cross-compiling and we're always building for multiple architectures at the same time. We actually, by default, use a make, file, dot, depend, dot, dollar machine so that we can have them updating in parallel without any concerns for contention. When we looked at doing this for FreeBSD, we figured that was going to be a little bit too much. And so one of the exercises we went through was to look at making it all work with a simpler model. And it's actually worked out very nicely, I think. So what we have here, you can see from the client, it's an auto-generated make, file. That means you don't touch it. The first thing it does there is it sets a variable that basically represents the relative location of this directory within the source tree. And that's key to how the whole thing works. Pazder is one of those magic variables that B makes sets for you. It represents the directory where it found the file that it's currently reading. And one of the modifiers that B make has, it says you can take the value that you've got here and you can pass it to a real path and turn it into an absolute path, which is extremely useful for this process. The key data in this file, apart from that, is DERDEPs, which is a list of the directories. And again, these are those directories relative to the top of the source tree that need to be built before we can build BNSH. And I mean, it's pretty simple to read. The only complexity here, if you want to call it that, is we substitute the variable CSU DER for what it actually was. This allows us to use the exact same list of dependencies regardless of the architecture. I386 is a bit of a wart in that for most architectures, the CSU DER is easily derivable from the machine value, not quite so for I386. And then the last thing it does is include DERDEPs.mk. And also at the end there, it will capture, if necessary, any local dependencies. These are what allow us to do a parallel build in a clean tree without doing make-depend. This is effectively capturing just the stuff within each directory, which are the sort of things that would normally break a parallel build in a clean tree. We'll look at this in more detail later. So building previous, the projects we make, as I briefly mentioned, is a test vehicle. It's an exercise in putting all of this building into a tree which is very close to head. It was synced last week from head. And it uses all the metamode and DERDEPs and all that sort of thing. The goal is to be able to easily cross-build FreeBSD as close to stock as possible, while minimizing changes to FreeBSD itself. Within Juniper, we have a team of FreeBSD folk. We've been building head and stable 10 now from back around 9, using almost exactly the same setup. There's a bit more stuff there. For instance, we use external tool chains. And we generate packages which are basically ISO FS's and all that sort of stuff. And disk images for feeding into VMs. But from the pure build point of view, there's very little difference. So just quickly, on the transition for FreeBSD, we first talked about it with the FreeBSD folk in 2011. Converting the base to use Bmake, which is the first step to be able to do any of this, was actually very simple. It took a patch of less than 300 lines to allow the FreeBSD head to build using Bmake instead of Fmake. And we're going to use Fmake to represent the old FreeBSD make, just to avoid confusion. Ports was a little bit more complex, because they have to support building on older versions of FreeBSD. At the time, it was 8.3, I think. And they don't branch. So that makes for a bit of a challenge. More importantly, it's reasonably simple to fix the base. It's a bit more complex to fix ports. The most difficult problem is all of the thousands of people out in the world who are using FreeBSD and things derived from FreeBSD who we have no visibility into. And for those people, it's a bit rude to go and pull a rug out from under them. And so as we'll see, we made a number of changes to Bmake to minimize the impact on those people as we go. And so again, just call if anybody does have horror stories or something like that, feel free to let us know so that we can maybe do something about them. So there were changes made to Bmake to support FreeBSD. Almost without exception, those changes go into NetBSD first, and then get imported back in. One of the goals of the exercise is to minimize divergence between FreeBSD and NetBSD with respect to make so that we all benefit. So both Bmake and Fmake are descended from Adam DeBoer's Pmake from Sprite. They've diverged quite a bit. One of the most obvious differences is the colon U and L modifiers, which I think FreeBSD got from OpenBSD, around about the same time that I was sticking all the OSF modifiers into Bmake, which, oddly enough, had colon U and colon L as well. Fortunately, these aren't used by the base system at all. And as of last week or this week, Ports doesn't use them either. But for the purpose of our story, it's still an issue. Or it was an issue. The other, I guess, key difference is Bmake is, aggressive is putting it nicely with respect to $.path. It will find anything by a.path that it possibly can. And that isn't always what you want. So you sometimes have to curb his enthusiasm. And there's a.noPath thing to do that. Fortunately, just simply saying.noPath, $clean files, covers 99% of the problem. NetBSD's bestyone.make marks all of the standard targets as both phony and not main. So that was another change. One of the real fun ones is handling job tokens. All modern makes, when you do a make minus j8, you don't want every sub-make to start another 8. And so on, you get what is it? Geometric increase in load. So they use the means of having a token pool of some sort to constrain that. Both Fmake and Bmake do that, but they do it differently. Fmake uses a FIFO, which the first instance, when Fmake starts up, he has a look to see if the FIFO exists. And if it doesn't, he assumes he's the master, creates the FIFO, exports its name, and everybody else then uses it. Bmake, on the other hand, uses a pipe. And he passes a magic argument to his children with the descriptors for the pipe. And so if Bmake doesn't get that argument, he assumes that he's the master, and so he goes and creates it and passes it on to his children. Both systems work nicely, but they don't work well together. In particular, if you had Fmake as the initial instance, he would export the name of his FIFO to the environment, go and start a bunch of sub-makes. And if they happen to be Bmake, each of them would think that he was the master, because he'd had no clue that he wasn't. And so he would start another N. It almost works the same going the other way, except that Fmake would blow up if Bmake passed him his minus capital J argument. So you're sort of saved there by that incompatibility. But if you were careful to cleanse the environment that you passed to Fmake to not have the minus J, then each Fmake would think that he was the master, and you'd have the same problem. There's other useful stuff in there. So a quick talk about the changes to the base. Any of the Bmake-specific syntax we bounded with ifdef.pasta, that's one of those magic variables that it defines for you. We added.nopath to all the generated files and phony and not main for the standard targets. We created a.weight as a no-op target for Fmake so that you can stick.weight into subdur lists and other things like that. And we ended up adding an option without Bmake so that people who, for whatever reason, didn't want to venture into the sharp pool could avoid it for a while. Another change, Fmake had a preference for BSD makefile before it looked for the traditional makefile and makefile. Bmake only looked for the traditional stuff, but it's configurable. So by simply sticking a list into sys.mp you can make it look for anything you want to. And in all of these cases, if we could address an incompatibility by just looking at the magic variable in sys.mp, that was the preferred solution. For the job token one, that's pretty much what we had to do. Bmake, in particular, only passes his token pool descriptors to submakes if the target is tagged with.make, the special source. In Fremisty Fmake, the man page says exactly the same things that it is supposed to do the same thing. But it doesn't seem to work, which is evidenced by the fact that somebody went and added this $ underscore plus thing to basically do what.make is supposed to do. And so what that means is that the Fremisty makefiles and presumably the thousands of makefiles written by people who use Fremisty but aren't in source have not been necessarily sticking.make on all the targets where they would want.make if it had worked. And so the only reasonable solution we could come up with for that one was to set a little knob in sys.mp to tell Bmake to say, you know what, just pass your descriptors to all the children. And that sort of makes the problem go away. Oh, the error token is a nice one. So normally when Bmake fails, this is in jobs mode. He sticks an error token back into the token pool. And the next time all the other submakes go to take a token out of the pool, they get the error token. They say, oh my god. And they bail. Which is wonderful because your build stops immediately on error, which 99.999% of the time is exactly what you want. Apparently it's not what you want for make universe. In that case, people would prefer that if some variable of MIPS is broken, that 99.36 and all the rest could just chug along and get as far as they could. And so we added another knob to say, you know what, if this make job error token is set to false, then don't stick an error token into the pool when you die. And so in the top level make file, if it sees that you're doing make universe, it'll set that knob to false. And so you can get the make universe behavior that you want and still get the nice behavior for all the normal cases. Oh, another. This is another fun one. So in the long ago, make use set minus e to fail things on statements. I don't recall exactly why. But I think it was mainly to get more consistency in target handling between jobs mode and compact mode. In NetBSD, they did away with all that. And they made it work very reliably such that the unit of failure was the command line rather than statements within the command line. So with fmake, you could do CD, to some bogus directory, semicolon, rm minus rf star. And it would be reasonably safe because unless you had a directory called that, it would fail at the CD and never get to the rm minus star. If you fed that same command to bmake, though, he would happily remove everything in the current directory, which may or may not be good. Well, the object directory anyway, if you had one. The format at the bottom is, of course, safe with any version of make and is therefore preferable. And while it would have been a quite simple exercise to go through all the make files in previous d and fix them so that they did the latter form instead of the former, again, that doesn't do anything for the thousands of people out in the world who may not get the memo. And so we had to do something about that. So again, this is one of those things that you can configure with make. So we just defined a shell for previous d to use by default, which effectively makes bmake behave the way fmake does and use set minus e by default. The option, the bmake option. So I think when we originally proposed adding bmake to make to previous d to start this transition process, we had in mind that we would just add bmake. It would get installed as bmake and people could choose whether they used it or not and so on. That wasn't considered ideal by the project. They wanted us to do the cut over as soon as possible and effectively, one day you'd go to bed and it was fmake and you'd wake up the next day and it's bmake. And that would have been feasible for the very first part of the project itself, but it's a bit more of a challenge again for the third parties. And so we ended up having to add that option so that people who didn't want to face the transition right now could avoid it. And so we added this option source makefile had to do the right thing, which typically involved it has a dance for you and you're upgrading to recognize that the make that you have on the system is not quite adequate for building the new version of software and it would go and bootstrap a new version of make and so on. And once it existed, it would use it. But it always called it make. And so he would say, oh, I have this temporary version of make, I'll go use it, even though it might have been six years old. So we fixed that so that if you wanted to be building with fmake, it made sure that temporary make was called fmake. And if you wanted to bmake, it would make sure that temporary make was called bmake. And if you change your mind, it would do the right thing. That's all being ripped out now. Thank you, Wana. The apparently was broken for a while. Anyway, nobody noticed. So obviously. That's OK. It's OK. It's gone. So we've burned the bridge. So that's progress. I mentioned the colon U and L for ports. I'd had a cunning plan for how to deal with bootstrapping ports, which worked brilliantly unless you were the portsfolk, because of the way that they want to be able to build ports. And so that wasn't going to work. So yet again, we added a local hack for ports to be able to set a knob saying, I really want the old colon U and colon L. And even with this set, actually, in many circumstances, bmake can do the right thing, because it can tell whether it's looking at a colon U with a value after it, or whether it's looking at a colon U with nothing in which case it might be the old modifier. But again, that's due to be removed after I get back from this conference. Another thing that fmake had, which netpstmake didn't have, was the tolerance for quoted strings as iterator variables. And so that was added to netpst. And then I think one of the final ones was the minus v behavior. If you do fmake minus v foo, it spits out the fully resolved value for you. If you do the same thing for bmake, it will give you the literal value, which may or may not include other variables. This is brilliant if you want to do debugging of what's going on. I use it a lot. And with bmake, you can always put dollar foo in there to expand it fully. So this is great for command line type stuff, but it's not necessarily ideal for the build itself. For the build itself, the fmake behavior is actually better. And so I proposed changing the default behavior and adding a debug flag so that you could give me the literal value. We're compromised by adding yet another knob to say which behavior you like. So revistd sets make expand variables to true, in which so you get the default revistd behavior. But regardless of the setting, you always have the debug flag to give you the literal value. And so both projects are able to have the semantics that they know and like. Oh yes, this was the last one. So bmake, some years ago, they switched the way iterator variables within for loops are handled to avoid namespace pollution. And so they use this colon u value. So what the colon u does is say, if the variable doesn't have a value, use what follows as the value. And so we have effectively a null variable. And so we're going to use value as the value. And this avoids, as I say, namespace pollution. Trouble is it makes it very difficult to try and play games like escaping iterator variables within for loops. And especially when you have nested for loops and you've got to do multiple escapes to try and escape the escapes within the for loops and so on. And I looked at this one for a while and I thought, you know what, let's forget it. Let's just use the inline loops, which are much easier to follow. So I basically replaced a 20 line block of nested for loops with a single line, which produces exactly the same output, of course. Oh, OK. All right. Did we have a pointer? There was a pointer at some stage. Here is one. Doesn't matter. I can't reach. All right, so this thing is the inline loop modifier. The first thing that follows it is going to be the iterator variable. And then everything between the next at and a closing one. And this isn't the closing one because this is introducing a nested loop onto the end. So the closing one is like this followed by another colon, for example. Oh, brilliant. Thank you. Hope I don't kill anybody. Ah, excellent. All right. This is so cool. All right, so in the man page we mentioned that in the OSF development environment where this came from, they actually use a convention of uppercase variable name with dot c the end for the iterator variable. That's really hard to say. But that's really hard to type. I'm lazy, so I prefer single character iterator variables because then you can just use dollar p unless you need to do modifiers on it. So anyway, this is two nested loops. So the first thing we do is for the Mlang stuff, we're going to stick man slash at the start of it. And if we find that we've got an empty one, we get rid of it. And then for those, so each value that we've got at that is going to be an M valuable. And then we're going to also iterate through the M links. Hang on. We're going to come up. What are we doing? Yeah, so each of those is going to be p. So we've got a p, so there's another loop. So we've got p colon e is the extension of p. And so this is actually referencing a man dollar p blah prefix. And then we've got m, which is our, from Mlang. And then again, we stick the p with his extension at the end. And unfortunately, when I did a full screen on my terminal with the normal font, you could actually see the end of the line. And it finishes with a couple of colon closed brackets. This is fire. The colon at colon u colon l colon d and colon p all came from the OSF version of P make. And the inline loop operator is particularly wonderful. Because unlike a for loop where everything is getting expanded as you read the for loop, in this case, the loop is expanded at the time of reference. And so you can do some very cool stuff with it that you can't possibly do with normal for loops. So this is saying for each n in what precedes it rather than for each n in the demo, and then for each p within the loop? Yes. Yeah, I figured you guys would be computer science, but you probably were able to get the hang of it. So do you make after the inline loopers, if they don't have a lot of loops sometimes like this? Yeah, both of them have their place. Don't get me wrong. Dot for loops are definitely useful at times. But there are things that to do it with a dot for loop, you have to do obscene things, which are just quite natural with the inline loops. All right, so why is b make so cool? Well, it can do dirt apps. That's reason enough. You can see dirt apps in the tree. It's in contrary of b make mk, dirt apps mk. I don't recommend reading it on an empty stomach. I've been doing this sort of stuff for 20 odd years, and I consider dirt apps one to read on a full stomach. But anyway, as I mentioned, we have a plethora of modifiers. One of the ones that somebody added not that long ago is this one. So I saw in previous D they have a fixed value that used for the random seed for C++ compiler. This allows you to have a repeatable build, but have a different anonymous initializer seed for every library that you build, which is actually quite a nice thing. This is an example of using the parse dir and parse file. This just ends up being a token that you can use. If you want to create a little marker target, makefile.inc, for example, to test whether that makefile.inc has been included already, you can make a target that includes this or this, or you can spell this all out, but it's more readable to just say this. This will ensure that you'll get a target that is the same value regardless of whether you read it via an absolute path, a relative path, or any combination thereof, because the column TA turns it into an absolute path using real path. So you canonicalize it. This one's an example of, as I mentioned, you can stick complex lists of modifiers into a variable and then reuse it. And there's an example down here. So here's a list with a number of variables, and we basically just apply this set of modifiers to it, and we get a list of exclusions. And that sort of thing gets used quite heavily by DERDeps and lots of other cool stuff. It can do the equivalent of strf time using either local time or GM time. This in the Junos build, we like to have timestamps on the build logs so that we can see where all the time's going. This allows us to have those timestamps in the output logs for essentially free, and so we do it. And we use this format typically so that you get a human readable thing. You get something you can actually do math with and a handy little token to be able to grep them out. And one of the tools I have in my kit bag is a little Python script, which will read one of these logs, and it can basically give you a progress bar so you can see how long the build's got to go until it's done. By the way, I'm not going to get through all these slides. I've stuck a number of slides in towards the end to just cover random stuff that people may or may not be interested in, so don't be concerned that we're going to run out of time, because we will. So it was last sync. The Projects Be Make branch is the one that hopefully most of you are interested in. This was last synced about a week ago, and as of this morning, it completely builds again. The transition to clang through some spanners into the work, but I got some help from David last night to sort that out and finally finished getting it all building again. As I mentioned, unlike the Junos build where we use this format of MakeFile.depend pretty much universally, in FreeBSD we use this format. There are, I think, about five places where we do this, and that's invariably because there are places where the auto-generated, the locally-generated files that you need to capture the dependencies for have names that vary with the architecture. And so the way the build-in for here works is if you had previously built one of these directories for, say, I386, and therefore there was a MakeFile.depend.I386, and then you then went and built it for MIPS when he gets to the end and says, oh, I need to update MakeFile.depend. He says, oh, I see there isn't one, but I see there's a MakeFile.depend.some other machine. I better do the same thing, and he'll just automatically create a MakeFile.depend.MIPS for you. And so all the right stuff happens. The packages MakeFile acts as a top-level MakeFile. In the Junos build, the source MakeFile just includes packages MakeFile. It just makes it simpler from transition point of view. And in the FreeBSD build, we do the same thing. We leave the FreeBSD top-level MakeFiles alone so that we can still do build world so we can make sure we're not breaking FreeBSD when we want to commit stuff back up from that environment. But we don't want to use it for our stuff. So this does the job. Under packages, in the normal world, you would have a whole bunch of packages, foo, to build a package called foo. We use the pseudo directory as a place to put these all represent targets because they happen to have a MakeFile.depend file under them. And so when you go into packages and you say Make Bootstrap Tools, for example, it knows that that's something it can build. And it also knows because it's got.host that it's something that it has to build for the magic machine called host, which happens to be whatever host you're building on as opposed to the host, maybe i3-at-6 or AMD64. But it's not necessarily the same thing as an i3-at-6 or AMD64 target because you may be building FreeBSD 10, but you might be building on FreeBSD 7, which is unfortunately what we're doing at the moment. But it doesn't matter because it works. So for the environment, I think in most of these examples, I use a little tool called MK. Its join life is from wherever it is to have a look upwards until it finds a file called sandbox-env. And wherever it finds that file defines the top of the quote sandbox, and he sets a variable SB to that location. And then he reads that file up to condition his environment. This is great if you're an Emacs user and you're editing branch, I have trees from NetBSD, Junos, FreeBSD, all over the shop, and you can be editing them all within the same Emacs and just do metarex compile in any of them. And it'll just do the right thing. But if you don't like using wrappers and so on, all you really need to do is make sure that you have makesyspaths set, and everything else that needs to be done can be done by, say, local sys.mk or anything else. Projects.bmk has a local sys.mk, which I've checked into the tree. The main reason is it contains a lot of stuff which I haven't taken care of yet. But generally speaking, you can build all of use-land, kernel, tool chains for both the host and target. I added a boot. Clang 3.3 is unable to build Clang 3.4, it seems, or at least the table gen can't handle the headers and so on. So you very likely need to bootstrap the tool chain in order to be able to get building with the new Clang. So I added a target to do that. And since bootstrapping tool chains is not a wheel worth reinventing, I just made it leverage the existing targets in the source makefiling one. We're now using sysroot, which works much better with Clang than pretty much any alternative. You can do build world. You can have build world work, like build world, but produce metafiles along the way. And that's actually very useful both for comparing against when you're scratching your head as to why Clang built when you did build world versus not building when you didn't. You can actually go look at all the syscalls involved and see what was going wrong. It also gives you a means for bootstrapping makefoldup depend if you were wanting to start this exercise all over again. So here's a quick thing about getting started. So minimal environment setup. Once you've got that, you should be able to just do make bootstrap tools. You'd want to do at least J8, because it's going to take a little while. If you're not using my little MK wrapper and you want to then build the tool chain again, and you remember the days of building GCC where you used the host compiler to build stage one and then used stage one to build stage two and eventually. Same sort of thing. You could skip this step, but you would have to add this with tools to these ones to basically use the set of tools that you've built in this stage. It's probably neater to go through this step once and then put it all aside and be done with it. And then you can have fun cd-ing to random places and building them. Or the lot is a pseudo target that just basically builds everything. And again, it should all just build in clean tree. Debugging failure. I was talking to Kasey or somebody last night. And so I gave him this example as to how it's cool. So I mentioned before, nobody logs their builds and they get build failures and they give you absolutely useless information. These days, though, as long as they can tell you where their tree is, we can go find the clues. And as an example, we'll edit bincat and sticker a bogus include in there. And then we go and compile it. And lo and behold, it explodes. Oops, yeah, couldn't find that. And b makes spews out. This is some of the noise I mentioned that b makes spews out on failure. This is all configurable, of course. By default, it doesn't do any of this. You have to set some variables. But you can set it to spit out any variables you like. So we have a number that we like to see. The level is useful, the current directory, the object directory. So it spits out the target that it was building, the metafile that it was creating. And it will actually copy this metafile into an error directory at the top of the tree so they're easy to find. And if you're doing things like I have a continuous build machine, and so he knows to go, build failed. The first thing he does is look in the error directory. If there's anything in there, that's why the build failed. That makes the analysis very simple. And so here's the metafile that got copied into source top dot dot error with a name that we can easily find. And yes, there we go. Even if the guy hadn't logged his build in, this error had actually happened 5,000 lines earlier and is not possibly in his scroll back anymore, we can still go find what the exact error was, why it was happening. We can go and see which version of the compiler he was running and all of the files that were being read and so on. And it may be that, oh yes, you've got the wrong version of the compiler, or no, you tweet your spell. Oops, wrong. But one way or another, you have the information that you need to be able to work out what was wrong and do something about it. For next steps, we think it would be useful to actually add a bit more to it so that you could, for instance, have a target in the Projects Be Make branch that would let you go and effectively just build a little bootable VM image that you could then throw into Be Hive, which would be quite useful. Another thing that may or may not be interesting is distributed building. This is actually very easy to do with DERDeps. But that's probably the subject for a whole new talk. But if anybody's interested, I can bend your ear on that. Are we going? OK, so now we're into just the almost random stuff. But one of the things I was talking about with Warner recently, and he's thankfully been doing, is separating the BSD.star stuff from, so you can think of these things as classes of make files. The BSD.star.mk files are things for building stuff in the BSD way. But you don't necessarily want to have them only able to build user source. And you can have source.ops, for example, as options which only apply to building user source that you don't necessarily install, but are still useful. And similarly, you can have local.star.mk, local.com for anything else you like, as things which are either site local or tree local. They're not part of the repository, per se. But they allow you to do things like customization without hacking the original source. If you look at DERDeps.mk, most of the meta and similar make files, I include a.local, whatever the name of the thing is, to allow customization without having to touch the original make file, which in the case of something like DERDeps.mk is very important. Because, like I said, I only touch it when I'm sober. And I would recommend others do the same. Actually, I would recommend not touching it. So source.ops, this is something that has been very handy. It's now something that we can include within those make files that depends that we use in the pseudo targets. So for instance, this one is for the tool chain. So we can now say, well, is make clang define? No, it's not. Oh gosh, let's go include source.ops. Now we know it will be defined. And we say, is clang yes? OK, we'll add clang to the set of stuff we're going to build. Is GCC yes? We'll add GCC, and so on and so forth. And that way, you can easily handle building all the optional stuff. One of the other things that we saw earlier was auto creation of object directories. A nice feature of Bmake is any source that you give to the special target object sets, the object. It has to exist, of course. So this little dance up here makes sure that it exists. And this is how we do auto creation of object directories. If you are going to do this sort of dance, though, you have to do it very early in the game. Because this has to be done before you start reading make files that are going to potentially influence.path. Because when you do.path some directory, if it doesn't exist, it doesn't get added. And there are cases where it will exist. But if you do it too late in the game or too early in the game, you'll get a path that's in the source tree, rather than the object directory, which was what was intended. So again, if you want to play with auto object directories, you really need to do it during the sys.mk phase. But this works really well. I was talking to, again, Chat last night, 10 something years ago within Junos. We went through the whole phase of the camp of people who like to dribble object directories in the source tree. You had the camp who liked to have just a single directory called obj. And then you had the other guys who were building multiple archers. So they wanted obj.dollarmachine and so on. So we got to a point of having five different ways to support building the tree for all these various combinations. And it just got ridiculous. So we just said, forget it. We're going to create object directories automatically. And no questions asked. And it allows you to simplify things enormously. And simple is good. All right, so since nobody's asking any questions, I'll waffle a bit about meta mode. Basically, what we do is when we build a target, we create, amazingly enough, a file called target.meta. And it collects stuff like the expanded command line, which, if all you did, would be useful. Because that allows you to say, all right, last time I built this target, I used that command line. Now I'm building it, and I'm going to use this command line, and they're different. The target is out of date. And this allows you to do things like skipping. You've all seen make files where you have comments like, oh, if any of the make files change, we better regenerate this in case. And that can lead to all sorts of unnecessary compilation. By being able to be sure that the target is out of date, because its command line actually was impacted, you have a much better situation. We capture any command output, which, nine times out of 10, is just the error. In most cases, it's empty. And we capture all the interesting system calls. By interesting, I mean the ones that are interesting to make. And most typically, he's interested in read, execute. He has to track changes of directory so that he can work out path names and stuff like that. But mostly read and execute is what he cares about. And why do we do this? Well, the automated capture of all this information helps us optimize the build and improve reliability. And by optimizing the build, I mean doing as little as possible, doing it in parallel, but doing it correctly. And capturing the command output, again, makes failure analysis much easier, even when people refuse to log their builds. Metamode and Dirt Eps help with all of the above. As I mentioned, we don't do make to pen. We still do make to pen within our build for the kernel, because at least last time I looked, there were certain files that only get generated via the make to pen phase of the kernel. But I'm told by O'Brien that maybe even that's not necessary anymore, so we may be ditching that soon. But for the rest of the tree, we never do make to pen. And it saves you a lot of time. And while, even before we had all this, we were using GCC to dribble out info so that you could auto-gather dependency information, using the filemon kernel module lets you gather all that information for everything, not just the compiler. Let's see. And so you automatically catch tool change changes, all that sort of stuff. Also, here's the example of an unnecessary recompile that you can avoid. You can use things like DP add to bootstrap your dependencies if you're just getting started. And interestingly enough, you can get into problems where, if you have actually spurious entries in DP add, you've said, oh, DP add, I need lib foo-mumble, but you don't actually use it. It won't be captured in the auto-captured dependencies because you didn't use it. And then you get to the ultimate directory, and it says, well, I can't do anything because foo-mumble's not there. It says, well, that's because you didn't need it. So once we've finished processing DERDAT DP add to add C flags and stuff that we may or may not want to as a result, we actually throw it away to avoid those sort of errors. Some targets will need work. For instance, it's quite common to have a target that is going to include some kind of timestamp. You may or may not want that target to be regenerated every time you build. If you don't, you can actually tell make, for this particular target, I don't want you to do command line comparison because that would make you do the wrong thing as far as I'm concerned. You can actually be clever about it. Make knows that if your target actually involves the.o-o date, does anybody use that these days? It's basically expands to the list of sources that are at a date with respect to the target. And therefore, the expansion of it is going to be different almost every time you do it. And so there is no way that you can reasonably compare a command line that involves any use of this variable. So make knows this. And so if he spots this variable in the script, he'll say, well, I can't compare that command line. I'll skip it, which means that you can cheat because you can deliberately stick that variable in there, match something that will never be in there. And so you actually have a token here which will never expand to anything in the command line. But it'll be seen by Bmake, and he'll say, oh, he doesn't want to compare this command line. But you can still come. So in this target here, we won't regenerate just because the timestamp changed. But if you change anything in DP add, that changes the command line. And so we will regenerate the target. You enable it just by setting the word meta in a variable called make.mode. It gets read by all the make files. It gets looked at after you've read all the make files. So your make files can control this variable pretty much anytime. And effectively, meta.sys.mk does a bit more. But the key bit it does is it says, well, the target all depends on dirt-eps. And then we're going to wait for that to finish. As in, we're not going to do anything else until dirt-eps is finished. And that line there, if you like, is the magic that makes the whole thing work. Well, there's a bit more to it, but you get the idea. So when we write them for $target, we create $target.meta. Excuse me. We skip it normally for any quote special target, like begin and error. Unless you also sometimes you do want a metaphor for these, in which case you can always stick the special source dot meta on there to say, I really want a metaphor for this. It's never created if the target is flagged. No meta. I don't care what you think. I don't want a metaphor for this place. And normally, metafiles are skipped if the object directory and the current directory are the same. But in the case of, for instance, building the kernel, where we try to leverage the existing kernel build to the extent possible, so we actually do the CD into the object directory and run make in there. And in that case, we do want metafiles. And so we stick a little knob in make mode to say, making metafiles in the current directory is OK. And lastly, if the target that you're building is outside of your object directory, he will create a metafile with all the slashes replaced with underscores, just so that you end up with a file that you can reference within that directory. I think we've covered the content. Important to note, there are two variables that are used to track metafiles. MakeMetaFiles contains a list of all the metafiles that we read or examined or in any way knew anything about while we're building this particular directory. And makeMetaCreated is all the ones that we updated. And so this allows us, when we get to the end of having built this particular directory, if this variable is empty, we did nothing. We know that we don't need to go and consider whether we should update the makeFile.depends, which can be a reasonably expensive process, so we skip it when we can. Oh, if the meta verb, if the meta, we're being metavobose, then we'll expand the makeMetaPrefix, which defaults to just building a title name, but you can make it do anything you like. And in fact, I have a fairly cunning makeFile that sets this to a variable that actually is used to track all of the directories that you visited during the build, so that you can then go back and reexam them to see if there's any files that you should have SVN added but hadn't, so you can throw an error. That's another very cool trick. A little bit about filemon. It's a reasonably simple kernel module. The original prototype that John Birrell did of this idea used detrace. Detrace mostly works. It doesn't seem to let you reliably get the absolute path of the program that you're invoking. There's also a little bit of overhead, but more importantly, it needed special privileges to be able to do this. And we didn't necessarily like any of those. So we did this kernel module. It's available for FreeBSD, NetBSD, and there is a Linux version, which they promise me they will make public. And I will nag them continually until they do, because it's actually quite useful. Each record in the metafile that we get from filemon is the whole thing is designed to be easily parsed by a shell script. And the original thing that did the post-pressing was a shell script. So we have a single character to denote the system call that we did. And then typically you get the tag, the process ID, and the data. And the data is 9 times out of 10, just a path name. In the case of linking and moving, it's two path names with quotes, just in case there were spaces in the names. And then in the NetBSD version, I don't bother with stat calls, because they fill the thing tremendously, but they're not particularly useful to make. But they are interesting from a human point of view, I guess. When we read them, OK. So when we read them, and we only read them if Bmake thinks that the target is possibly up to date. So if you're reading a make file, and it's a clean tree, for example, all the targets are going to be out of date, he's never going to look for any metafiles, because he's just going to go and do it. But if the normal rules for make look at all the directory and so on, and thinks that the target is up to date, then and only then will he go look at a metafile, and he'll go and read it. And one of the first things he does is compare the command line. If the command line's different, and you haven't told him not to, he'll say, it's out of date. Go do it. He skips past the command output, and he gets to the system calls, and he needs to be able to pass the system call stuff, and for each file that you read or executed, if it's newer than the target, then the target's out of date. If one of the interesting ones is, yeah, so we have this concept of a bailiwick, anybody like the history, that defines to Bmake the sphere of his influence. So if he's responsible for populating that directory over there on the table, and he's reading a metafile that put a file over there, and it's gone, that means he needs to redo it. And so the target will be out of date as a result. Yep, that covers that. So one of the nice things that we can do with this, as I mentioned, you could write all these makefile depends manually if you wanted to. But it's nicer to do it automatically. So when we finish building, we have a tool that can go and read these things. And he can look at the make.meta created to know that he has to do something. But he actually looks at the full list to go have a look at what he needs to read to get all of the dependency stuff. And this whole process is greatly simplified by having your object directories and your source trees not overlapping. So object directories within the source tree complicate the thing. So it's much nicer to use something like the model we use where your object directories are either completely aside using makeobjder prefix or the fancy makeobjder that we use where you have essentially parallel trees. So here's an example of one. I forget what this was. Oh, this was building something in BNSH. So you can see the things that he read. Anything that he reads out of an object directory that isn't the current object directory defines something that needs to have happened before this directory can be built. And that's the essential idea. And we extract those and we stick them into DERDeps. You can also have me extract a variable called sourceDERDeps if you're doing things like originally within the genus world, we support the idea of subset trees where you can say, I just want to check out enough of the tree to be able to build bin cat. Everybody needs to be able to build bin cat. And so you can look at the source DERDeps for bin cat and say, oh, well, that means I need libc and I need level r and you can follow exactly the same logic for checking out the tree as for building it. But we haven't done that for a while and we decided, because these actually result in far more churn than these do, we actually took them out. And we don't bother collecting them in the Projects Beemake branch. Mapping object to sourceDERD. If you're reading things out of the object directory where they were created, it's a very simple exercise. You typically just substitute object for source top and you've got the directory. Because again, we keep everything nice and neat. If you're pulling things out of a stage tree, for example, which is you can just think of it as a dester that you've done auto install into, you need a little bit of help. Because there isn't necessarily a directory in the source tree that corresponds to user slash include. So what we do when we stage files is we stick a file called.derdep. And when the tool that processes these things, when he goes and reads and he says, oh, the guy used a file from here called uni standard, he says, is there a uni standard.h.derdep? Oh my gosh, there is. Let's have a look. And that gives him the value that he needs to stick into.derdep. And so he can proceed without any further ado. Logs are still useful, though. In particular, we can do things like capturing the number of metafiles that we examined for building libc, how many we actually created. So you can tell from the discrepancy in the numbers that this was an update build. It's nice to see lots of directories where you spent zero seconds doing absolutely nothing. Because that means that you don't have bugs in your make files that are causing things to be built unnecessarily. Over. Oh my gosh. Could it say something? All right. I'm curious. Anybody interested or want questions or anything like that? Because I can skip straight to the end. Where are we? Yep. So you can handle the writing itself as one of the dependencies. Yes. So you can do that distribution, it's really good to know from doing anything about the build, so you could like, you're at the same height of the build. Everything. If you want to. If you do distributed compilation like using, say, something like DCC, you can do that. It's a. See, you have to question the other side of the compiler that you do. Yes. Absolutely. If you do distributed, it's just a little more solution. Yep. Then you can send the compiler over. Yeah. So the way we do that is, if I'm the master of the build and I'm going to ask you to build something, I actually make you look at my tree and you will read the list of tool chains that you're supposed to use from that and you will use the same tool chains by default. Otherwise you'll explode and we won't do anything. You can do the same thing with DCC as in you send the job to a machine that you know has the tool chain that you want to have used. There's any number of ways to skin that cat. Any other questions? All right. Hang on. Woop, woop, woop. There you go. All right. So it may be the greatest thing since sliced bread, but it's maybe not for everybody. But it does provide a simple solution to some rather complex problems. And if anybody has any questions, there's some further reading and you can always catch me at s.o.g. at free sd, netbisd, juniper, and so on and so forth. Thank you. Thank you.
talk about the transition of FreeBSD to using complex chunks of software. The FreeBSD build is a decent size. The Junos build is a lot bigger. time. We actually, by default, use a make, file, before it looked for the traditional makefile and the same thing for bmake, it will give you the meta mode. Basically, what we do is when we build any use of this variable. So make knows this. And sliced bread, but it's maybe not for everybody.