Package management and creation in Gentoo Linux
Berkholz, Donnie
This talk is designed for developers and users of other Linux distributions to gain an understanding of the Gentoo approach and philosophy to package management. The goal isn't to get people using Gentoo but rather for attendees to walk away thinking about how they might be able to apply that to their own distributions or other projects. We'll cover how the package manager (Portage) works, what packages (ebuilds) look like, our focus on ease of use for package authors, how we've iterated on our package format over time, etc.
 I'm Donnie, I've done Gen2 for a pretty long time now and I'll tell you a little bit more about that in a minute. But basically this talk is about what kind of value we can provide to the other distros in terms of what kind of things you could learn and take away and apply to your own. So what we're going to do is talk through some of the philosophies, think about how you can take a philosophy from a distribution into how you turn that into package management and that sort of thing, and then run through some of the individual features we've got and the way we've iterated on those features in our package format over the past 10, 15 years to continue thinking about how can we keep improving the packaging format and not just how we handle things on top of that format like the last talk, which is mostly iterating on top of RPM rather than iterating RPM itself, although RPM does very slowly move forward. So I apologize for the terrible quality of that picture, it's still off somebody else's site. But so this is basically what I used to do in Gen2 is I was somewhat overweight and I also wore a lot of hats. I did things including maintaining most of our X packages, which is about, well first it was like five and then the modularization happened and then it was 400. And then there was a lot of other science packages and miscellaneous stuff. And I think still the desktop group manager on the council, which is the group of seven people that decides all the technical stuff about Gen2 and so on. But really these days, first of all I lost some weight, but second of all I pretty much just talk because I had a few kids, I got married, I graduated from college and suddenly there's not so much free time as they used to be. So the philosophy of Gen2 really derives from two things. One is the guy who started it named Daniel Robbins previously worked on a district called Stampede and that's purposely pixelated because that's how old the distro is and how long since it went anywhere. Stampede was a very old school distro back in the late 90s, early 2000s. And then Daniel Robbins went to FreeBSD because he got pissed off at the developer community at the guys just being total dicks about everything. So he left, decided, you know, turns out he thinks the developer community is a really important factor, not just the technical code itself but actually the quality of the interactions within the community. So he left, went to FreeBSD and he said, well FreeBSD has a really nice way to handle the packages, right? Because they've got, you know, they've built everything from source, they've got these make files but on the other hand, all the make files, everything is controlled by an individual variable, right? Everything is a unique snowflake. So the same feature in one package versus another package might be controlled by a variable with a totally separate name and it makes it really hard to deal with it. And so he said, you know, I wish I had this FreeBSD thing but in a more cohesive way and not Linux with a good developer community and that's why he started Gentoo. This is a picture of a Gentoo penguin. It's supposedly the fastest penguin which is terrible for Gentoo as a reputation because really it's not about speed at all. It's about flexibility. But we've got all these ricers who think, oh, I use Gentoo because I get faster code. No, that's not at all the case. Well, you might get 5% if you're lucky. But that's after quite a bit of work. I don't know what's going on with all these weird little points. So in terms of philosophies, it's really, five more bullets or not. Oh, yeah, I do. That was weird. Okay. So basically we were thinking, how can we have something that's user-friendly despite being something where you spend a lot of time on a command line? And so we spend a lot of time working on our package manager portage, which the main command for that is you merge, and working on the CLI actually being something that's very colorful and easy to understand because we don't really support any GUIs for package management. Everything is done through the command line on a terminal. And so the title is Gentoo package management goals up there. So thinking about if we've got these goals, what can we do to make that a reality? And really the goals were, one, it's great ease of use despite a very low-level interface because we're trying to understand what's inside the black box of Linux distros. The second goal is really, how can you minimize the developer effort for people who are writing the packages? And that pervades through a lot of the changes we've made to the formats over the years. And even more so, a lot of the changes we've refused to make over the years because they make the format too complex. And one of the benefits we've had is that it's very easy to start writing packages, and this has helped us get a lot more people involved because packages look like bash scripts. And everybody who runs Gentoo, you have to run a lot of bash stuff to just install the thing, a lot of it. And you probably screw up four or five times, and then you figure out it's important to read the docs. And so we've got people who by the time they even managed to install Gentoo, they're already at a point where they could build a very basic package because it's just a bash script and wrapped around with some fancy features to build software more conveniently. And so simplicity is a very, very important thing when you're thinking about what should your packaging format look like. And that's the easy maintenance part. And in addition to that, it's thinking about what does a package format have to look like if you want to support tons of architectures and tons of operating systems? What kind of things do you have to add to deal with that? Wow. Those transitions are wacky, aren't they? And then the other major features, these aren't as interesting, but really the ones that are cool are thinking about how do you support the possibility of installing, say, you know, five versions of GCC on the same system? Or dealing with GTK1 and GTK2 and GTK3 all installed at the same time? And then thinking, you know, Gentoo is a source-based distro, so we have to handle source-based packages, but we also want to enable people to do what they want to do. So you know, how do you support binary packages for the people who want to do binary packages? And all of these kinds of thoughts go into, you know, how do you create a package format that makes sense? I don't even understand what is happening with these transitions. It's so weird. I'm just going to skip that stuff. That's not very interesting. But sort of, you know, it's always about flexibility from Gentoo's point of view and thinking, how can you enable, you know, whether it's users or developers to do whatever they want to do? All right, so how do you enable them to use all their processors? You know, we added a bunch of parallelization stuff. How do you make it less of a pain in the ass to upgrade things when you're sitting around, if you're compiling for like 12 hours or something crazy like that, you don't want to find out that after three hours when you went to sleep after too many beers that it failed. So we said, okay, let's resume that thing. Or dealing with, you know, it turns out it's amazing how many packages out there install binaries with the same names. And it's really a pain in the ass for just to deal with this when you've got some weird, obscure thing that installs, you know, binary with a certain name. And then you've got something that's super common that installs the same thing. It's like, ah, just makes you want to kill somebody. But then also things like, you know, making it super easy for developers building on Gentoo to do what they want to do by saying, okay, it's really easy to build packages, keep them integrated with the system, and yet have them very easy to, you know, work with and code on by adding, you know, debugging support, installing the sources without, you know, flipping the binaries and that sort of thing. Is there anything that's worth saying there? I have no idea what's going on with those transitions. I apologize about that. And then the other things that are really interesting to think about are how do you deal with, you know, multiple different repositories and thinking about, you know, do you want to add dependencies across those repositories so you can actually depend on a certain repository being installed? But that really creates some issues when you're thinking about, okay, but somebody might derive from that repository and then do something incompatible and change the name. And so how do you, how can you deal with these kinds of things inside of a package format? Wow, that's so weird. I just can't get over how weird that is. So the ebuild file itself is the package format and basically it's a big bash script that's split into a series of functions that map logically to what a build and installation process looks like. So you've got, you know, some main functions that are like, okay, let's unpack the source, let's screw around with it a little bit, patch it up, let's configure it, let's compile it, let's install it and then let's do whatever weird screen around the system you have to do to actually make the thing work. And how many of you are actually developers for a distro or packages? Awesome. So all of you know that build and install process is when you're actually writing a package or nowhere near as simple as you wish they were. There's always some stupid fiddly thing you have to do on any package that's anything beyond the trivial. And you know, our goal with a package format is to make it really easy to do that kind of thing, right? So you do, okay, here's the default and then let me tack some things on to that to do whatever screwy fiddly things I have to do, whether it's, you know, doing a set on a configuration file or, you know, fiddling with a couple files on the live system, you know, renaming some things so they don't overlap and so on. And then the last thing that's really interesting about the package format is this idea of we have some level of sort of object oriented style inheritance so that we have this thing called e-classes where they're essentially like a library of functions and the individual e-builds can actually inherit from that library, use all the defaults from that library in terms of, you know, the unpacking, the compilation, the installation and then vary from those a little bit as they have to. And these other things are just some basic metadata about the package, right? What architectures does it work on? The slot is the thing that enables you to install multiple versions of the same thing at the same time, right? So, GCC 3.5, 3.6, 4.0, 4.7 and so on. And then we've got, you know, a few different types of dependencies and we're actually thinking about adding more because we haven't made it complex enough yet. So it turns out when you're making a source-based distribution, you have to think about a lot more types of dependencies that have to be on every user system, right? It's not just about the runtime dependencies, it's also about the build time dependencies, it's about the dependencies that have to be installed after the package is installed because they depend on it but it won't run without them. And you've got some other styles of dependencies like, oh, I need the headers to be available while I'm building it, but I'm building it for an embedded system on this box and so things get really weird and complex. Yeah. I'm sorry? Oh, yeah. Yeah, we have conflicts between packages. Yep. Yeah. Thank you. Yeah, so we deal with conflicts and we've got to mention that in a little bit because one of the things we did as we were iterating on the package format was think about what are the different types of conflicts and how can we enable the package manager portage to deal with those types of conflicts more effectively, right? So there's one possibility which is the two packages. One can overwrite the other before the other is uninstalled but then the other type is, you know, sort of mutually exclusive thing where they can't be touching the live file system at the same time at all. So the first one has to be uninstalled completely before the second one can be put into place which may happen if you're replacing a sim link with a file, for example, or a directory with a file. So then we've got this thing. Since we're source based, we have to think, okay, how do we enable people to build only the futures they want to use and that's what we created with use flags. They sort of map onto configure options. So you might have a, let's say you've got a package that supports a GTK interface or a Qt interface and you have a use flag for GTK, you'll have a use flag for Qt. It might add some extra functionality if you actually build for GNOME specifically beyond GTK so there might be another use flag for that that will pull in another 15 libraries. And we have lots of different profiles so that people can basically pick a reasonable starting point and then iterate from there. And so it's, you know, both on the user and the developer level, it's thinking, how can you set up people as close to where they need to be as possible and then just let them define a couple of things to vary from that. So, you know, the value of actually providing a lot of different types of user personas or user profiles is actually really high. So the basic story behind how we've iterated on our eBuild API or the EAPI over the years, every year to year and a half for the past six years we've actually come out with a new EAPI, some of them adding more features than others. So you can see, you know, this is, EAPI one was the first time we changed things. Before that it was EAPI zero. And so zero was basically like, what is this thing we have now? It's been around for like 10 years and it's just a gigantic mess. And it was a huge effort to just document the thing. And in fact, it actually has turned out to be pretty valuable to have multiple different package managers implementing the same spec. So we've got Portage, which is our default package manager, but then there's also a couple of other ones. There's a Pollutus and package core that implement the same spec and all of them deal with eBuild files. And this has been pretty nice and actually forcing us to really think through, you know, what does a package format look like? How do you describe it in such a way that it's useful to lots of different people? And so then we came out with one, a year later we came out with two, a little bit over a year later we came out with three, another year later four, another about a year and a half later five. But the funny thing that you don't appreciate looking at this, it looks like this is a nice regular March. And in fact, it turns out that three happened because we were so sick of the stuff we were working on for three, we decided to push most of it to four and just get something out the door. So in reality, it's like big features, big features, almost nothing, huge pile of features, bunch of features. And we're just going to kind of run through some of those so you can get a feel for what those features looks like. So actually why don't we just pop out and look at some actual code instead of just sitting here looking at slides all day. What did I say? That was MPC? Yeah, okay. Devlib's MPC. So this is what it looks like when you're in a package tree. Yeah, it looks beautiful over here, guys. Yeah, I'll slide it over there. See if I can just maximize the thing. And maybe make it a bit bigger. Can you read this in the back yet? Cool. Let me see. All right. So this is basically what an e-build looks like. And it's just a simple text file. I used nano for e-builds because I'm terribly pathetic. So at the top you've got some basic header info, copyrights, and so on. Then here, one of the first things we do is define the EAPI so that the package manager understands how do you parse the rest of this thing. We inherit some e-classes to give us some default ways to handle various things and a few sets of utility functions. So eutils is sort of a basic subset of functions. We've got a set of functions to deal with libtool. We've got a set of functions to try and handle multi-lib systems like 32-bit and 64-bit on the same system simultaneously. And then you've got your basic sort of metadata, right? What is this package? Where do you find it? How do you download the thing? What's the license? Because we actually do support things like only installing packages with a certain set of licenses. So you can have, for example, a completely free or a Libre software system. You've got a slot. And the slot is what you change if you want to install multiple versions at the same time. You've got what architectures you work on. This I use thing defines the set of possible features you can vary on this package. So in this case, there's only one, you can build the library statically or not. And that means you'll always get it dynamically, but then you can also tack on the static ones too. And then you've got the dependencies. And these are just very basic sort of standard notations for indicating the numbers and things. So you've got a greater than or equal to. Here's a category. Here's a package name and then the version number. And the same thing in the second place. And we've actually got a fairly flexible syntax for dealing with that. So you can have greater than or equal or less than or equal or just greater or less. And you can also have a star and asterisk if you don't care about the version or you don't care about some set of the version. And then it's just regular old bash, right? So the runtime dependencies, you just use the depend variable. And then you can go down and look at what the functions look like when you're doing this thing. And you've got an unpack function which just calls out to unpack the tarball, go into the source directory, and run some lib-tools stuff. And you compile. So you have a configure function that has some convenience features like a bunch of default arguments. And then you've got here's handling the use flags. So use enable. And then when this use flag is specified, here's the configure flag you passed. So you pass enable static. And then if it doesn't work, you fail. Die. Yeah, right. So I'm purposely showing these in order of how we came up with them. So this is the old school EAPI zero. We actually decided, you know, in the case of following our philosophy, trying to make it simpler for developers to write these things that, in fact, it would be nice if you didn't ever have to write this die thing, right? And just that was how it worked by default. Because normally, if something wrong happens when you're building or installing a package, you don't want to install the thing. It's going to be broken. So you want it to die by default and instead specify something if you don't want it to die. So one of the things we ended up adding in a later EAPI was this idea of a sort of non-fatal prefix where it doesn't die. And normally it just happens that it always dies or something goes wrong with nearly every one of these commands. So you unpack, you compile, you install, right, which is just basically a make install with a destination directory and die if it fails. And then there's the little bit of fiddling that I'm talking about. Every time you're writing a package, there's always some fiddling that has to happen. So in this case, it's getting rid of lib tool archives, installing a couple of docs. Then something I should say is every one of these functions that's prefixed with source, those ones happen when you're doing a source build, right, only when you're doing a source build. But every time you're installing a package onto the live file system from the source directory where it's compiled, then you've got package functions that run package prefixes. So you've got a pre-installation function, a post-installation function that you can do run before and after all the files from the build are merged onto the live file system. Which enables you to get things ready, make sure nothing is in the way. And then it seems like there's always packages out there with terrible build systems. And so you have to tweak things in the file to get installed because they don't realize they were going to get moved after they were built, right? It's surprising how many people write just awful build systems that expect you to download the thing on packet, build it, and then run it in place and never install it onto a real system as part of a distribution. Okay, so that's what a very basic EAPI0 eBuild looks like. Now let's see if I can manage to get back into the presentation. Okay, so then we came out with the EAPI1 about five, six years ago. We added a few really interesting features. Thank you, and I'm only showing the ones that are kind of interesting versus the ones that are minor tweaks. So we added this idea that you could actually have a use flag default in a single package that wasn't the same as the default for the whole system. Because sometimes with a package it makes more sense to enable a feature than not. And so we said, okay, in this case, you could turn off GTK support but it would be kind of stupid because then you wouldn't have a GUI at all. So if they haven't manually specified switching away from their global default in their profile, then it will, for example, add GTK instead of not adding GTK, even if the global default is to not add GTK. But if they say, I don't want GTK, it will still override that. So there's a series of overrides. We also had this idea of being able to actually properly depend on things in a certain way. So we added a certain slot. This was very useful, for example, with GTK1 and GTK2, where it became increasingly difficult to specify all of the things that matched a slot using a series of version numbers. And so it ended up working out pretty well to add a way to depend on a certain slot. And a lot of distros, instead of doing the slot thing, they decided, oh, we're just going to put the version number into the package name. So there was a package that was GTK1-1.2.10, for example. And we thought that that wasn't really an elegant way to handle that. So we added a new concept to deal with the parallel installations instead. And it actually turned out to be really useful on development systems, because you could have multiple versions of GCC. You could have multiple versions of GCC that cross-compile due to different architectures on the same system. And it actually had a lot more benefits than anybody ever expected, I think. So we'll just take a quick look at one example of an e-build that does this, WCS.CVS, 0.4.1. So it's all the same stuff from last time. You've got your basic header. This time, the difference is EAPI1. You scan down. Most of the stuff looks about the same. But you start looking into dependencies, and now we've got this colon and a version number after, indicating that we're depending on a certain slot. So it's only the JCOM and one slot, and presumably JCOM and also has two series. This is actually really common with Java packages, is that they'll have simultaneous sort of independent series, all of which are maintained ongoing, because Java is a really nice, really popular architecture for doing enterprise software development, and those things have to be supported for a decade. So this is just an example of what the slot dependencies look like. But other than that, all of this should look a little bit familiar, because you saw it a few minutes ago. You've still got your basic build dependencies, runtime dependencies, unpack function. You've got, we're doing some utility functions that will unpack the source code, apply some patches, and some convenience variables to say, OK, the patches are in that files directory for that package. And then this one expands to actually the package name and the package version. And let's see. Trying to remember, does that one actually have any, use defaults in it? OK. Well, maybe not. So that was a couple of minor features. But then we came out with EAPI 2, and we added a couple of really cool things to it. One of them was this idea of arrows for where to download it. And this is just another, yeah. OK, sure. So the notion of slot, it's primarily intended for packages that were intended to be installed in parallel at the same time. So like GTK 1, GTK 2 are a really nice example of this. There's GCC, for example, so that you could have GCC 3.6, 4.0, 4.5, all at the same time on the same system. The slot is just a way you tell the package manager that these are all the same package and we should enable them to be installed at the same time instead of forcing the lower version to one to be upgraded and removed. So that means that the package maintainer is the one who decides what would be the slot number? So the slot number is specified in the eBuild itself. So the package writing the eBuild specifies the slot number. And normally the slot number would be zero. But in a case where you're actually dealing with multiple slots, it normally corresponds to something like the major version. Or in the case of Python, you might want 2.5, 2.6, 2.7, 3.1, 3.2, all installed at the same time and the slots would map exactly to those major dot miners. So it's just a string. It's not numeric necessarily or anything. And it's just a way of saying, install all of these things at the same time rather than removing the lower versions when you upgrade to a higher version. So in EAPI 2, we said, all right, let's deal with some more of just the weird build stuff that people have to deal with. And so we added these things called source-era arrows. The source-era is the download location for files. And so normally it looks something like this first piece before the arrow. But it turns out there's a lot of developers who don't really understand how to make packaging friendly who don't put a version on their tar balls, for example. Or who name their tar balls in such a way that the suffix doesn't map to what the file format actually is. Like this dot tar might actually be a dot tar dot gz. And so how do you deal with that in a way that doesn't force people to download them manually so that everything else works nicely after the fact? And so you're not having hash collisions because everything is hashed. And so we say, OK, foo.tar.gz is got an md5 and a SHA-1 and a SHA-256-X. And so you download that thing. And if it's not versioned, then you can't do that. Because all the different versions are going to have the same file name. And there's going to be five different hashes for the same file name, which doesn't work out too well. And so we added this thing that said, OK, download this file with this name, but actually rename it to this other name so that your hashes work out so that you're not overwriting files with each other and so on. And then we also added this idea of use dependencies, which is not just depending on a certain package. Because it used to be before this. So we would depend on a package. And then in the functions within the eBuild, check and see how that package was built. And so you would depend on it. And then you would start building your package. And you would say, let me go look at that library, see if it was built the right way. If it wasn't, I fail. And I'd die. But that sucked. Because what would happen is you would die three hours into a 12-hour upgrade cycle. And then people come back the next day and they'd be like, what happened here? Why did I die? I wanted that thing to keep going. I don't care whether it doesn't have the right feature. You should have just skipped the package and kept going. And so we said, yeah, you're right. You're right. Let's add a way to depend on that properly instead of hacking around later on after we've actually started the build. So let's depend on the library built with the right feature in the first place. And so it's about simplifying the developer's job, but also making things easier for the user and pissing them off less, basically. And so we've got a lot of different styles of this. And they all basically map to build with this flag, build without this flag. If I'm building with this feature enabled, make sure my libraries underlying me are built with this feature enabled or vice versa. If I'm building without this feature, maybe my libraries also have to be built without this feature because they may be incompatible features. And let's pop up an example of this one and check it out quick. Or 5.1. All right. So in this case, you can see, again, EAPI2, right? Everything else looks very similar. You've got your basic metadata. You've got your inheriting, your E classes for convenience functions and so on. But right here, you see they name their tarball, rbsource.tgz. No version numbers or anything. Cool. Thanks. Which is a serious pain. And so we just say, OK, let's have an arrow and let's name it rb-5.1. Because that's the name of this package which this thing expands to. Let's see if we actually have any. Oh, cool. And in this case, we've also got use dependencies as well. So depends on a bunch of stuff. You've got some logic in there. If you want OpenGL, you have to get this, this, and one of these things. One of them is the Mesa library built with motif support. And the other one is this libglw thing. If you avoid that, then you get this and another thing. So does that make sense? Cool. All right. So that's two. I guess we're going to have to move a little faster because I have 10 minutes left and three more to get through. So oh yeah, here's some more stuff in two. So here's the blocker syntax that somebody was asking about earlier. We've got blockers that can be installed on the file system at the same time or ones where the first package has to be completely gone before the second one can go over. And that's like, OK, let's block this really hard. Let's double block this thing. And we also added more granular builds. And this turned out to be really nice in terms of letting the packages be shorter because we started out having just a big compile function, do all of the build things. And then we said, actually, you know what? It would be nice if configure and make were actually separate functions so that if the make part is normal, we can just let that happen and not have to write that code. But if we have to tweak around with configure, add some more arguments and so on, so every time you have use flags, you always have to deal with those for configure. So we have the separate configure function where we just basically call configure with the right arguments for the features we want. But then make just happens and it all works. Well, the same thing, we split out the unpacking from the preparing. And preparing is basically, for the most part, patching. So you want to just, don't worry about the unpacking. That just happens by itself. But we're going to have to apply some patches, maybe tweak some auto tools, files around, do an auto recap, and so on. And the other thing that really made developers lives easier was adding defaults for every one of these functions. So all of these steps are called phases. So these are phase functions, unpack, configure, compile, so on. And so we made it really easy to do the default behavior and then tweak on top by having a function within each phase that's just called default. So if you're starting the configure function, you can just say, okay, default. That's it, and then it runs configure and everything. And then you can play around a little bit afterwards instead of having to manually specify the configure part. And we'll pop up a quick example of this. Cairo, whatever version that is, 1.8.10. Hopefully we've got some default functions in here. So, yeah, here's an example of prepare. We don't specify unpack at all. It just happens because it knows what kind of format the file is. It downloads that file and it says, oh, that file's got the right suffix. I'm just going to unpack it and put you in the right place. And then some patching happens. Let's see if we've got it. And the same thing with configure, right? So we've got source configure. You don't see source compile anywhere. It goes straight to the install. So the compilation just happens. It uses the default information. But the configure part, we have to tweak around, you know, skewer around based on what the host is, append to some C flags, pass all our configure arguments based on the use flags, that sort of thing. Now three, I'm not going to bother showing you this one because we're running out of time here basically. But what we did with three is we were going to add a ton of stuff. We worked on it for like a year and a half. And then we're like, this sucks. We're never going to get this out the door. So we'll just do something now to help people who really need something and then keep working on these other features. So the main thing we did with three was prefix support, which means, so this is a Mac here. It's actually running OSX right now because I haven't installed Gentoo on it yet. And it's got Gentoo installed on it in a prefix. So in my home directory on this thing, I've got a Gentoo subset on its own little piece of the file system. And it doesn't install on the root. It just sits there in my home directory. So that's what prefix support is. And you have to do a lot of stuff to deal with that based on packages that assume they're going to be living in the root and look for other packages from the root on down instead of from the top of the Gentoo installation on down. So we had to do some tweaking around for that. Now four is a big one. We added a ton of stuff, again, trying to make developers' lives easier. Thinking about, OK, some of these things are just ludicrous where it was just like a historical accident that we didn't have it. So we've got this install function. But the default behavior for the install function was do nothing. So you would build the thing. It would be sitting there in the build directory. And then if you didn't specify an install function, it would just sit there in the build directory, get deleted, nothing would happen, which is ludicrous. So finally we said, OK, we're going to install, do something that makes sense. And it's going to install stuff by default instead of doing nothing by default and just leaving things there. And we added a bunch more capabilities to basically fail fast and fail early so that users, again, aren't getting screwed like two or three hours into this four or five-hour-long thing so that you can actually run some code from inside the packages before the full installation starts happening while it's still at the dependency calculation and asking whether you want to proceed phase. Some more things for use dependencies. Here's one of the questions we had earlier on. We added this non-fatal prefix to say, OK, actually the normal behavior we want is to die by default if anything goes wrong. So we shouldn't have to specify that. But instead, we're going to make that the normal behavior. And if we don't want to die for some reason for something that truly doesn't matter, then you have to say non-fatal before that. And then we added a new variable for basically dealing with installations that have to be interactive for some reason or another. There always seem to be a few of them out there for some weird packages. But we're always adding things to deal with all the weirdness things that obscure packages do. So in some cases, they have to be interactive because you have to accept some license before you can proceed, for example. And you have to actually visually see that license. You can't just accept it implicitly. So let's pop up, get quick, and take a look at that. Whatever version that was. Get 181. All right. So you've got EAPI 4 up top. All this other stuff is pretty normal. So you've got some bashiness. You've got inheriting some functionality from E-classes. Your default metadata, your dependencies. Let's see here. Pretty sure there's a non-fatal here. OK. So here's an example, right? If we're running the test suite and it doesn't successfully clean out beforehand, let's not die and just keep going. Because nobody wants to fail out because the test suite directory didn't clean up first. Everybody's going to want to install and have something that works, even if that doesn't work properly. And then a bunch more stuff. One really cool thing was, so we've got package dependencies that can go in these hierarchies and do ors and mutually exclusives and so on. But we couldn't do the same thing for features, for use flags. So we added the same capability to specify feature dependencies in the same way we could specify package dependencies. Because sometimes features do depend on one another. And I've only got one minute left, so I'm not going to show you that. But I hope you get the idea because it looks exactly like package dependencies. Sometimes you have a stack of features, none of which work without the next one down. So if you've got GNOME support, you might require GTK support first. It's a great example of this. EAPI 5, which is the last one we came out with, adds capabilities to do rebuilding better when library versions change, which is the main idea here. So you've got these ideas of sub-slots and slot dependencies that are smarter so that, for example, if the slot version stays the same, then I keep working. But if the slot version changes, then it will forcibly rebuild all the packages that correctly specify that they require to that slot. And that's really all I'm going to say about that so we can just wrap up. So EAPI 6 is something we're now thinking about, thinking about how can we keep making developers' lives easier and users' lives easier. And I'd like to thank you for your time. Do we have time for questions? Great. Anybody got questions? Yeah. It seems like a lot of this stuff that you have built in is set up for Auto Tools, but what about other stuff like SCONs and stuff like that? That's a great question. So we do, I was talking about E-classes, which are basically our sort of libraries of functions, and we've got E-classes for dealing with different build systems. Auto Tools is by far the most common, so our default functionality is up to deal with that. But we've got a SCONs E-class, for example, where you are a CMake E-class, where you inherit that E-class and that replaces all the default functions. So now, source configure and source compile are set up to deal with SCONs or CMake instead of being the default Auto Tools behavior. So we try really hard to make it easy to deal with that. And it's the same thing for version control systems. We've got ways to very easily build from live version control systems using E-classes. So it's sort of CVS subversion, and now we've got Git, we've got BZR, Mercurial, and so on, so that you just inherit something, say, okay, the repository's over there, and it knows how to get the code, how to download it, how to build it, and everything. Yeah. Yeah. Then DOS upstream. Well, GitHub doesn't DOS though, right? Given how complex it's got now to write an e-build file, is there anything that, looking back, any features you wouldn't add if you were doing all again? The features that I kind of regret, like we needed them to have a complete system, but are the things with really complex sets of syntax, like all the different ways to do use dependencies, for example, make it a little bit complex to remember, like you've got, like, an exclamation point in front and a question mark in back, and there's an equals sign there somewhere where most people actually have to look at, like, a reference card when they want to figure out how to get it right, and that's exactly what you don't want. But, you know, that said, so we try to keep things as simple as possible, but no simpler, right? And sometimes there's just not a way to do it simply. So we try and just do the minimum we have to do to get the job done, but not adding extra syntax for its own sake. So, it's easy to write e-bills as the upstream behaves, so if you get upstream to change, then it's easy to write e-bills, so it's just kind of hand in hand with the distro and the upstream. Yeah, yeah. Upstream cooperation definitely makes doing packaging a lot easier. No question about it.
 I'm Donnie, I've done Gen2 for a pretty long time now and I'll tell you a little bit more  about that in a minute.  But basically this talk is about what kind of value we can provide to the other distros  in terms of what kind of things you could learn and take away and apply to your own.  So what we're going to do is talk through some of the philosophies, think about how  you can take a philosophy from a distribution into how you turn that into package management  and that sort of thing, and then run through some of the individual features we've got  and the way we've iterated on those features in our package format over the past 10, 15  years to continue thinking about how can we keep improving the packaging format and not  just how we handle things on top of that format like the last talk, which is mostly iterating  on top of RPM rather than iterating RPM itself, although RPM does very slowly move forward.  So I apologize for the terrible quality of that picture, it's still off somebody else's  site.  But so this is basically what I used to do in Gen2 is I was somewhat overweight and I  also wore a lot of hats.  I did things including maintaining most of our X packages, which is about, well first  it was like five and then the modularization happened and then it was 400.  And then there was a lot of other science packages and miscellaneous stuff.  And I think still the desktop group manager on the council, which is the group of seven  people that decides all the technical stuff about Gen2 and so on.  But really these days, first of all I lost some weight, but second of all I pretty much  just talk because I had a few kids, I got married, I graduated from college and suddenly  there's not so much free time as they used to be.  So the philosophy of Gen2 really derives from two things.  One is the guy who started it named Daniel Robbins previously worked on a district called  Stampede and that's purposely pixelated because that's how old the distro is and how long  since it went anywhere.  Stampede was a very old school distro back in the late 90s, early 2000s.  And then Daniel Robbins went to FreeBSD because he got pissed off at the developer community  at the guys just being total dicks about everything.  So he left, decided, you know, turns out he thinks the developer community is a really  important factor, not just the technical code itself but actually the quality of the interactions  within the community.  So he left, went to FreeBSD and he said, well FreeBSD has a really nice way to handle the  packages, right?  Because they've got, you know, they've built everything from source, they've got these  make files but on the other hand, all the make files, everything is controlled by an  individual variable, right?  Everything is a unique snowflake.  So the same feature in one package versus another package might be controlled by a variable  with a totally separate name and it makes it really hard to deal with it.  And so he said, you know, I wish I had this FreeBSD thing but in a more cohesive way and  not Linux with a good developer community and that's why he started Gentoo.  This is a picture of a Gentoo penguin.  It's supposedly the fastest penguin which is terrible for Gentoo as a reputation because  really it's not about speed at all.  It's about flexibility.  But we've got all these ricers who think, oh, I use Gentoo because I get faster code.  No, that's not at all the case.  Well, you might get 5% if you're lucky.  But that's after quite a bit of work.  I don't know what's going on with all these weird little points.  So in terms of philosophies, it's really, five more bullets or not.  Oh, yeah, I do.  That was weird.  Okay.  So basically we were thinking, how can we have something that's user-friendly despite  being something where you spend a lot of time on a command line?  And so we spend a lot of time working on our package manager portage, which the main command  for that is you merge, and working on the CLI actually being something that's very colorful  and easy to understand because we don't really support any GUIs for package management.  Everything is done through the command line on a terminal.  And so the title is Gentoo package management goals up there.  So thinking about if we've got these goals, what can we do to make that a reality?  And really the goals were, one, it's great ease of use despite a very low-level interface  because we're trying to understand what's inside the black box of Linux distros.  The second goal is really, how can you minimize the developer effort for people who are writing  the packages?  And that pervades through a lot of the changes we've made to the formats over the years.  And even more so, a lot of the changes we've refused to make over the years because they  make the format too complex.  And one of the benefits we've had is that it's very easy to start writing packages,  and this has helped us get a lot more people involved because packages look like bash scripts.  And everybody who runs Gentoo, you have to run a lot of bash stuff to just install the  thing, a lot of it.  And you probably screw up four or five times, and then you figure out it's important to  read the docs.  And so we've got people who by the time they even managed to install Gentoo, they're already  at a point where they could build a very basic package because it's just a bash script and  wrapped around with some fancy features to build software more conveniently.  And so simplicity is a very, very important thing when you're thinking about what should  your packaging format look like.  And that's the easy maintenance part.  And in addition to that, it's thinking about what does a package format have to look like  if you want to support tons of architectures and tons of operating systems?  What kind of things do you have to add to deal with that?  Wow.  Those transitions are wacky, aren't they?  And then the other major features, these aren't as interesting, but really the ones that are  cool are thinking about how do you support the possibility of installing, say, you know,  five versions of GCC on the same system?  Or dealing with GTK1 and GTK2 and GTK3 all installed at the same time?  And then thinking, you know, Gentoo is a source-based distro, so we have to handle source-based  packages, but we also want to enable people to do what they want to do.  So you know, how do you support binary packages for the people who want to do binary packages?  And all of these kinds of thoughts go into, you know, how do you create a package format  that makes sense?  I don't even understand what is happening with these transitions.  It's so weird.  I'm just going to skip that stuff.  That's not very interesting.  But sort of, you know, it's always about flexibility from Gentoo's point of view and thinking,  how can you enable, you know, whether it's users or developers to do whatever they want  to do?  All right, so how do you enable them to use all their processors?  You know, we added a bunch of parallelization stuff.  How do you make it less of a pain in the ass to upgrade things when you're sitting around,  if you're compiling for like 12 hours or something crazy like that, you don't want to find out  that after three hours when you went to sleep after too many beers that it failed.  So we said, okay, let's resume that thing.  Or dealing with, you know, it turns out it's amazing how many packages out there install  binaries with the same names.  And it's really a pain in the ass for just to deal with this when you've got some weird,  obscure thing that installs, you know, binary with a certain name.  And then you've got something that's super common that installs the same thing.  It's like, ah, just makes you want to kill somebody.  But then also things like, you know, making it super easy for developers building on Gentoo  to do what they want to do by saying, okay, it's really easy to build packages, keep them  integrated with the system, and yet have them very easy to, you know, work with and code  on by adding, you know, debugging support, installing the sources without, you know,  flipping the binaries and that sort of thing.  Is there anything that's worth saying there?  I have no idea what's going on with those transitions.  I apologize about that.  And then the other things that are really interesting to think about are how do you deal  with, you know, multiple different repositories and thinking about, you know, do you want  to add dependencies across those repositories so you can actually depend on a certain repository  being installed?  But that really creates some issues when you're thinking about, okay, but somebody might derive  from that repository and then do something incompatible and change the name.  And so how do you, how can you deal with these kinds of things inside of a package format?  Wow, that's so weird.  I just can't get over how weird that is.  So the ebuild file itself is the package format and basically it's a big bash script that's  split into a series of functions that map logically to what a build and installation  process looks like.  So you've got, you know, some main functions that are like, okay, let's unpack the source,  let's screw around with it a little bit, patch it up, let's configure it, let's compile it,  let's install it and then let's do whatever weird screen around the system you have to  do to actually make the thing work.  And how many of you are actually developers for a distro or packages?  Awesome.  So all of you know that build and install process is when you're actually writing a package  or nowhere near as simple as you wish they were.  There's always some stupid fiddly thing you have to do on any package that's anything  beyond the trivial.  And you know, our goal with a package format is to make it really easy to do that kind  of thing, right?  So you do, okay, here's the default and then let me tack some things on to that to do whatever  screwy fiddly things I have to do, whether it's, you know, doing a set on a configuration  file or, you know, fiddling with a couple files on the live system, you know, renaming  some things so they don't overlap and so on.  And then the last thing that's really interesting about the package format is this idea of we  have some level of sort of object oriented style inheritance so that we have this thing  called e-classes where they're essentially like a library of functions and the individual  e-builds can actually inherit from that library, use all the defaults from that library in  terms of, you know, the unpacking, the compilation, the installation and then vary from those  a little bit as they have to.  And these other things are just some basic metadata about the package, right?  What architectures does it work on?  The slot is the thing that enables you to install multiple versions of the same thing  at the same time, right?  So, GCC 3.5, 3.6, 4.0, 4.7 and so on.  And then we've got, you know, a few different types of dependencies and we're actually  thinking about adding more because we haven't made it complex enough yet.  So it turns out when you're making a source-based distribution, you have to think about a lot  more types of dependencies that have to be on every user system, right?  It's not just about the runtime dependencies, it's also about the build time dependencies,  it's about the dependencies that have to be installed after the package is installed because  they depend on it but it won't run without them.  And you've got some other styles of dependencies like, oh, I need the headers to be available  while I'm building it, but I'm building it for an embedded system on this box and so  things get really weird and complex.  Yeah.  I'm sorry?  Oh, yeah.  Yeah, we have conflicts between packages.  Yep.  Yeah.  Thank you.  Yeah, so we deal with conflicts and we've got to mention that in a little bit because  one of the things we did as we were iterating on the package format was think about what  are the different types of conflicts and how can we enable the package manager portage  to deal with those types of conflicts more effectively, right?  So there's one possibility which is the two packages.  One can overwrite the other before the other is uninstalled but then the other type is,  you know, sort of mutually exclusive thing where they can't be touching the live file  system at the same time at all.  So the first one has to be uninstalled completely before the second one can be put into place  which may happen if you're replacing a sim link with a file, for example, or a directory  with a file.  So then we've got this thing.  Since we're source based, we have to think, okay, how do we enable people to build only  the futures they want to use and that's what we created with use flags.  They sort of map onto configure options.  So you might have a, let's say you've got a package that supports a GTK interface or  a Qt interface and you have a use flag for GTK, you'll have a use flag for Qt.  It might add some extra functionality if you actually build for GNOME specifically beyond  GTK so there might be another use flag for that that will pull in another 15 libraries.  And we have lots of different profiles so that people can basically pick a reasonable  starting point and then iterate from there.  And so it's, you know, both on the user and the developer level, it's thinking, how can  you set up people as close to where they need to be as possible and then just let them define  a couple of things to vary from that.  So, you know, the value of actually providing a lot of different types of user personas  or user profiles is actually really high.  So the basic story behind how we've iterated on our eBuild API or the EAPI over the years,  every year to year and a half for the past six years we've actually come out with a new  EAPI, some of them adding more features than others.  So you can see, you know, this is, EAPI one was the first time we changed things.  Before that it was EAPI zero.  And so zero was basically like, what is this thing we have now?  It's been around for like 10 years and it's just a gigantic mess.  And it was a huge effort to just document the thing.  And in fact, it actually has turned out to be pretty valuable to have multiple different  package managers implementing the same spec.  So we've got Portage, which is our default package manager, but then there's also a couple  of other ones.  There's a Pollutus and package core that implement the same spec and all of them deal with eBuild  files.  And this has been pretty nice and actually forcing us to really think through, you know,  what does a package format look like?  How do you describe it in such a way that it's useful to lots of different people?  And so then we came out with one, a year later we came out with two, a little bit over a  year later we came out with three, another year later four, another about a year and  a half later five.  But the funny thing that you don't appreciate looking at this, it looks like this is a nice  regular March.  And in fact, it turns out that three happened because we were so sick of the stuff we were  working on for three, we decided to push most of it to four and just get something out the  door.  So in reality, it's like big features, big features, almost nothing, huge pile of features,  bunch of features.  And we're just going to kind of run through some of those so you can get a feel for what  those features looks like.  So actually why don't we just pop out and look at some actual code instead of just sitting  here looking at slides all day.  What did I say?  That was MPC?  Yeah, okay.  Devlib's MPC.  So this is what it looks like when you're in a package tree.  Yeah, it looks beautiful over here, guys.  Yeah, I'll slide it over there.  See if I can just maximize the thing.  And maybe make it a bit bigger.  Can you read this in the back yet?  Cool.  Let me see.  All right.  So this is basically what an e-build looks like.  And it's just a simple text file.  I used nano for e-builds because I'm terribly pathetic.  So at the top you've got some basic header info, copyrights, and so on.  Then here, one of the first things we do is define the EAPI so that the package manager  understands how do you parse the rest of this thing.  We inherit some e-classes to give us some default ways to handle various things and  a few sets of utility functions.  So eutils is sort of a basic subset of functions.  We've got a set of functions to deal with libtool.  We've got a set of functions to try and handle multi-lib systems like 32-bit and 64-bit on  the same system simultaneously.  And then you've got your basic sort of metadata, right?  What is this package?  Where do you find it?  How do you download the thing?  What's the license?  Because we actually do support things like only installing packages with a certain set  of licenses.  So you can have, for example, a completely free or a Libre software system.  You've got a slot.  And the slot is what you change if you want to install multiple versions at the same time.  You've got what architectures you work on.  This I use thing defines the set of possible features you can vary on this package.  So in this case, there's only one, you can build the library statically or not.  And that means you'll always get it dynamically, but then you can also tack on the static ones  too.  And then you've got the dependencies.  And these are just very basic sort of standard notations for indicating the numbers and things.  So you've got a greater than or equal to.  Here's a category.  Here's a package name and then the version number.  And the same thing in the second place.  And we've actually got a fairly flexible syntax for dealing with that.  So you can have greater than or equal or less than or equal or just greater or less.  And you can also have a star and asterisk if you don't care about the version or you  don't care about some set of the version.  And then it's just regular old bash, right?  So the runtime dependencies, you just use the depend variable.  And then you can go down and look at what the functions look like when you're doing  this thing.  And you've got an unpack function which just calls out to unpack the tarball, go into the  source directory, and run some lib-tools stuff.  And you compile.  So you have a configure function that has some convenience features like a bunch of default  arguments.  And then you've got here's handling the use flags.  So use enable.  And then when this use flag is specified, here's the configure flag you passed.  So you pass enable static.  And then if it doesn't work, you fail.  Die.  Yeah, right.  So I'm purposely showing these in order of how we came up with them.  So this is the old school EAPI zero.  We actually decided, you know, in the case of following our philosophy, trying to make  it simpler for developers to write these things that, in fact, it would be nice if you didn't  ever have to write this die thing, right?  And just that was how it worked by default.  Because normally, if something wrong happens when you're building or installing a package,  you don't want to install the thing.  It's going to be broken.  So you want it to die by default and instead specify something if you don't want it to  die.  So one of the things we ended up adding in a later EAPI was this idea of a sort of non-fatal  prefix where it doesn't die.  And normally it just happens that it always dies or something goes wrong with nearly every  one of these commands.  So you unpack, you compile, you install, right, which is just basically a make install with  a destination directory and die if it fails.  And then there's the little bit of fiddling that I'm talking about.  Every time you're writing a package, there's always some fiddling that has to happen.  So in this case, it's getting rid of lib tool archives, installing a couple of docs.  Then something I should say is every one of these functions that's prefixed with source,  those ones happen when you're doing a source build, right, only when you're doing a source  build.  But every time you're installing a package onto the live file system from the source  directory where it's compiled, then you've got package functions that run package prefixes.  So you've got a pre-installation function, a post-installation function that you can  do run before and after all the files from the build are merged onto the live file system.  Which enables you to get things ready, make sure nothing is in the way.  And then it seems like there's always packages out there with terrible build systems.  And so you have to tweak things in the file to get installed because they don't realize  they were going to get moved after they were built, right?  It's surprising how many people write just awful build systems that expect you to download  the thing on packet, build it, and then run it in place and never install it onto a real  system as part of a distribution.  Okay, so that's what a very basic EAPI0 eBuild looks like.  Now let's see if I can manage to get back into the presentation.  Okay, so then we came out with the EAPI1 about five, six years ago.  We added a few really interesting features.  Thank you, and I'm only showing the ones that are kind of interesting versus the ones that  are minor tweaks.  So we added this idea that you could actually have a use flag default in a single package  that wasn't the same as the default for the whole system.  Because sometimes with a package it makes more sense to enable a feature than not.  And so we said, okay, in this case, you could turn off GTK support but it would be kind  of stupid because then you wouldn't have a GUI at all.  So if they haven't manually specified switching away from their global default in their profile,  then it will, for example, add GTK instead of not adding GTK, even if the global default  is to not add GTK.  But if they say, I don't want GTK, it will still override that.  So there's a series of overrides.  We also had this idea of being able to actually properly depend on things in a certain way.  So we added a certain slot.  This was very useful, for example, with GTK1 and GTK2, where it became increasingly difficult  to specify all of the things that matched a slot using a series of version numbers.  And so it ended up working out pretty well to add a way to depend on a certain slot.  And a lot of distros, instead of doing the slot thing, they decided, oh, we're just going  to put the version number into the package name.  So there was a package that was GTK1-1.2.10, for example.  And we thought that that wasn't really an elegant way to handle that.  So we added a new concept to deal with the parallel installations instead.  And it actually turned out to be really useful on development systems, because you could  have multiple versions of GCC.  You could have multiple versions of GCC that cross-compile due to different architectures  on the same system.  And it actually had a lot more benefits than anybody ever expected, I think.  So we'll just take a quick look at one example of an e-build that does this, WCS.CVS, 0.4.1.  So it's all the same stuff from last time.  You've got your basic header.  This time, the difference is EAPI1.  You scan down.  Most of the stuff looks about the same.  But you start looking into dependencies, and now we've got this colon and a version  number after, indicating that we're depending on a certain slot.  So it's only the JCOM and one slot, and presumably JCOM and also has two series.  This is actually really common with Java packages, is that they'll have simultaneous sort of independent  series, all of which are maintained ongoing, because Java is a really nice, really popular  architecture for doing enterprise software development, and those things have to be supported  for a decade.  So this is just an example of what the slot dependencies look like.  But other than that, all of this should look a little bit familiar, because you saw it  a few minutes ago.  You've still got your basic build dependencies, runtime dependencies, unpack function.  You've got, we're doing some utility functions that will unpack the source code, apply some  patches, and some convenience variables to say, OK, the patches are in that files directory  for that package.  And then this one expands to actually the package name and the package version.  And let's see.  Trying to remember, does that one actually have any, use defaults in it?  OK.  Well, maybe not.  So that was a couple of minor features.  But then we came out with EAPI 2, and we added a couple of really cool things to it.  One of them was this idea of arrows for where to download it.  And this is just another, yeah.  OK, sure.  So the notion of slot, it's primarily intended for packages that were intended to be installed  in parallel at the same time.  So like GTK 1, GTK 2 are a really nice example of this.  There's GCC, for example, so that you could have GCC 3.6, 4.0, 4.5, all at the same time  on the same system.  The slot is just a way you tell the package manager that these are all the same package  and we should enable them to be installed at the same time instead of forcing the lower  version to one to be upgraded and removed.  So that means that the package maintainer is the one who decides what would be the slot  number?  So the slot number is specified in the eBuild itself.  So the package writing the eBuild specifies the slot number.  And normally the slot number would be zero.  But in a case where you're actually dealing with multiple slots, it normally corresponds  to something like the major version.  Or in the case of Python, you might want 2.5, 2.6, 2.7, 3.1, 3.2, all installed at the same  time and the slots would map exactly to those major dot miners.  So it's just a string.  It's not numeric necessarily or anything.  And it's just a way of saying, install all of these things at the same time rather than  removing the lower versions when you upgrade to a higher version.  So in EAPI 2, we said, all right, let's deal with some more of just the weird build stuff  that people have to deal with.  And so we added these things called source-era arrows.  The source-era is the download location for files.  And so normally it looks something like this first piece before the arrow.  But it turns out there's a lot of developers who don't really understand how to make packaging  friendly who don't put a version on their tar balls, for example.  Or who name their tar balls in such a way that the suffix doesn't map to what the file  format actually is.  Like this dot tar might actually be a dot tar dot gz.  And so how do you deal with that in a way that doesn't force people to download them  manually so that everything else works nicely after the fact?  And so you're not having hash collisions because everything is hashed.  And so we say, OK, foo.tar.gz is got an md5 and a SHA-1 and a SHA-256-X.  And so you download that thing.  And if it's not versioned, then you can't do that.  Because all the different versions are going to have the same file name.  And there's going to be five different hashes for the same file name, which doesn't work  out too well.  And so we added this thing that said, OK, download this file with this name, but actually  rename it to this other name so that your hashes work out so that you're not overwriting  files with each other and so on.  And then we also added this idea of use dependencies, which is not just depending on a certain  package.  Because it used to be before this.  So we would depend on a package.  And then in the functions within the eBuild, check and see how that package was built.  And so you would depend on it.  And then you would start building your package.  And you would say, let me go look at that library, see if it was built the right way.  If it wasn't, I fail.  And I'd die.  But that sucked.  Because what would happen is you would die three hours into a 12-hour upgrade cycle.  And then people come back the next day and they'd be like, what happened here?  Why did I die?  I wanted that thing to keep going.  I don't care whether it doesn't have the right feature.  You should have just skipped the package and kept going.  And so we said, yeah, you're right.  You're right.  Let's add a way to depend on that properly instead of hacking around later on after we've  actually started the build.  So let's depend on the library built with the right feature in the first place.  And so it's about simplifying the developer's job, but also making things easier for the  user and pissing them off less, basically.  And so we've got a lot of different styles of this.  And they all basically map to build with this flag, build without this flag.  If I'm building with this feature enabled, make sure my libraries underlying me are built  with this feature enabled or vice versa.  If I'm building without this feature, maybe my libraries also have to be built without  this feature because they may be incompatible features.  And let's pop up an example of this one and check it out quick.  Or 5.1.  All right.  So in this case, you can see, again, EAPI2, right?  Everything else looks very similar.  You've got your basic metadata.  You've got your inheriting, your E classes for convenience functions and so on.  But right here, you see they name their tarball, rbsource.tgz.  No version numbers or anything.  Cool.  Thanks.  Which is a serious pain.  And so we just say, OK, let's have an arrow and let's name it rb-5.1.  Because that's the name of this package which this thing expands to.  Let's see if we actually have any.  Oh, cool.  And in this case, we've also got use dependencies as well.  So depends on a bunch of stuff.  You've got some logic in there.  If you want OpenGL, you have to get this, this, and one of these things.  One of them is the Mesa library built with motif support.  And the other one is this libglw thing.  If you avoid that, then you get this and another thing.  So does that make sense?  Cool.  All right.  So that's two.  I guess we're going to have to move a little faster because I have 10 minutes left and  three more to get through.  So oh yeah, here's some more stuff in two.  So here's the blocker syntax that somebody was asking about earlier.  We've got blockers that can be installed on the file system at the same time or ones where  the first package has to be completely gone before the second one can go over.  And that's like, OK, let's block this really hard.  Let's double block this thing.  And we also added more granular builds.  And this turned out to be really nice in terms of letting the packages be shorter because  we started out having just a big compile function, do all of the build things.  And then we said, actually, you know what?  It would be nice if configure and make were actually separate functions so that if the  make part is normal, we can just let that happen and not have to write that code.  But if we have to tweak around with configure, add some more arguments and so on, so every  time you have use flags, you always have to deal with those for configure.  So we have the separate configure function where we just basically call configure with  the right arguments for the features we want.  But then make just happens and it all works.  Well, the same thing, we split out the unpacking from the preparing.  And preparing is basically, for the most part, patching.  So you want to just, don't worry about the unpacking.  That just happens by itself.  But we're going to have to apply some patches, maybe tweak some auto tools, files around,  do an auto recap, and so on.  And the other thing that really made developers lives easier was adding defaults for every  one of these functions.  So all of these steps are called phases.  So these are phase functions, unpack, configure, compile, so on.  And so we made it really easy to do the default behavior and then tweak on top by having a  function within each phase that's just called default.  So if you're starting the configure function, you can just say, okay, default.  That's it, and then it runs configure and everything.  And then you can play around a little bit afterwards instead of having to manually specify  the configure part.  And we'll pop up a quick example of this.  Cairo, whatever version that is, 1.8.10.  Hopefully we've got some default functions in here.  So, yeah, here's an example of prepare.  We don't specify unpack at all.  It just happens because it knows what kind of format the file is.  It downloads that file and it says, oh, that file's got the right suffix.  I'm just going to unpack it and put you in the right place.  And then some patching happens.  Let's see if we've got it.  And the same thing with configure, right?  So we've got source configure.  You don't see source compile anywhere.  It goes straight to the install.  So the compilation just happens.  It uses the default information.  But the configure part, we have to tweak around, you know, skewer around based on what the  host is, append to some C flags, pass all our configure arguments based on the use flags,  that sort of thing.  Now three, I'm not going to bother showing you this one because we're running out of  time here basically.  But what we did with three is we were going to add a ton of stuff.  We worked on it for like a year and a half.  And then we're like, this sucks.  We're never going to get this out the door.  So we'll just do something now to help people who really need something and then keep working  on these other features.  So the main thing we did with three was prefix support, which means, so this is a Mac here.  It's actually running OSX right now because I haven't installed Gentoo on it yet.  And it's got Gentoo installed on it in a prefix.  So in my home directory on this thing, I've got a Gentoo subset on its own little piece  of the file system.  And it doesn't install on the root.  It just sits there in my home directory.  So that's what prefix support is.  And you have to do a lot of stuff to deal with that based on packages that assume they're  going to be living in the root and look for other packages from the root on down instead  of from the top of the Gentoo installation on down.  So we had to do some tweaking around for that.  Now four is a big one.  We added a ton of stuff, again, trying to make developers' lives easier.  Thinking about, OK, some of these things are just ludicrous where it was just like a historical  accident that we didn't have it.  So we've got this install function.  But the default behavior for the install function was do nothing.  So you would build the thing.  It would be sitting there in the build directory.  And then if you didn't specify an install function, it would just sit there in the build  directory, get deleted, nothing would happen, which is ludicrous.  So finally we said, OK, we're going to install, do something that makes sense.  And it's going to install stuff by default instead of doing nothing by default and just  leaving things there.  And we added a bunch more capabilities to basically fail fast and fail early so that  users, again, aren't getting screwed like two or three hours into this four or five-hour-long  thing so that you can actually run some code from inside the packages before the full installation  starts happening while it's still at the dependency calculation and asking whether you want to  proceed phase.  Some more things for use dependencies.  Here's one of the questions we had earlier on.  We added this non-fatal prefix to say, OK, actually the normal behavior we want is to  die by default if anything goes wrong.  So we shouldn't have to specify that.  But instead, we're going to make that the normal behavior.  And if we don't want to die for some reason for something that truly doesn't matter, then  you have to say non-fatal before that.  And then we added a new variable for basically dealing with installations that have to be  interactive for some reason or another.  There always seem to be a few of them out there for some weird packages.  But we're always adding things to deal with all the weirdness things that obscure packages  do.  So in some cases, they have to be interactive because you have to accept some license before  you can proceed, for example.  And you have to actually visually see that license.  You can't just accept it implicitly.  So let's pop up, get quick, and take a look at that.  Whatever version that was.  Get 181.  All right.  So you've got EAPI 4 up top.  All this other stuff is pretty normal.  So you've got some bashiness.  You've got inheriting some functionality from E-classes.  Your default metadata, your dependencies.  Let's see here.  Pretty sure there's a non-fatal here.  OK.  So here's an example, right?  If we're running the test suite and it doesn't successfully clean out beforehand, let's not  die and just keep going.  Because nobody wants to fail out because the test suite directory didn't clean up first.  Everybody's going to want to install and have something that works, even if that doesn't  work properly.  And then a bunch more stuff.  One really cool thing was, so we've got package dependencies that can go in these hierarchies  and do ors and mutually exclusives and so on.  But we couldn't do the same thing for features, for use flags.  So we added the same capability to specify feature dependencies in the same way we could  specify package dependencies.  Because sometimes features do depend on one another.  And I've only got one minute left, so I'm not going to show you that.  But I hope you get the idea because it looks exactly like package dependencies.  Sometimes you have a stack of features, none of which work without the next one down.  So if you've got GNOME support, you might require GTK support first.  It's a great example of this.  EAPI 5, which is the last one we came out with, adds capabilities to do rebuilding better  when library versions change, which is the main idea here.  So you've got these ideas of sub-slots and slot dependencies that are smarter so that,  for example, if the slot version stays the same, then I keep working.  But if the slot version changes, then it will forcibly rebuild all the packages that correctly  specify that they require to that slot.  And that's really all I'm going to say about that so we can just wrap up.  So EAPI 6 is something we're now thinking about, thinking about how can we keep making  developers' lives easier and users' lives easier.  And I'd like to thank you for your time.  Do we have time for questions?  Great.  Anybody got questions?  Yeah.  It seems like a lot of this stuff that you have built in is set up for Auto Tools, but  what about other stuff like SCONs and stuff like that?  That's a great question.  So we do, I was talking about E-classes, which are basically our sort of libraries of functions,  and we've got E-classes for dealing with different build systems.  Auto Tools is by far the most common, so our default functionality is up to deal with that.  But we've got a SCONs E-class, for example, where you are a CMake E-class, where you inherit  that E-class and that replaces all the default functions.  So now, source configure and source compile are set up to deal with SCONs or CMake instead  of being the default Auto Tools behavior.  So we try really hard to make it easy to deal with that.  And it's the same thing for version control systems.  We've got ways to very easily build from live version control systems using E-classes.  So it's sort of CVS subversion, and now we've got Git, we've got BZR, Mercurial, and so  on, so that you just inherit something, say, okay, the repository's over there, and it  knows how to get the code, how to download it, how to build it, and everything.  Yeah.  Yeah.  Then DOS upstream.  Well, GitHub doesn't DOS though, right?  Given how complex it's got now to write an e-build file, is there anything that, looking  back, any features you wouldn't add if you were doing all again?  The features that I kind of regret, like we needed them to have a complete system, but  are the things with really complex sets of syntax, like all the different ways to do  use dependencies, for example, make it a little bit complex to remember, like you've got,  like, an exclamation point in front and a question mark in back, and there's an equals  sign there somewhere where most people actually have to look at, like, a reference card when  they want to figure out how to get it right, and that's exactly what you don't want.  But, you know, that said, so we try to keep things as simple as possible, but no simpler,  right?  And sometimes there's just not a way to do it simply.  So we try and just do the minimum we have to do to get the job done, but not adding extra  syntax for its own sake.  So, it's easy to write e-bills as the upstream behaves, so if you get upstream to change,  then it's easy to write e-bills, so it's just kind of hand in hand with the distro and the  upstream.  Yeah, yeah.  Upstream cooperation definitely makes doing packaging a lot easier.  No question about it.
Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_0_179.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_180_1943.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_1944_2087.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_2088_2219.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_2220_3659.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_3660_3719.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_3720_4043.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_4044_4991.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_4992_8435.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_8436_8627.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_8628_8951.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_8952_8999.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_9000_10415.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_10416_12911.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_12912_13067.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_13068_13115.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_13116_14771.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_14772_14891.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_14892_15563.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_15564_21755.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_21756_22211.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_22212_23363.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_23364_23447.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_23448_24251.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_24252_24491.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_24492_28295.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_28296_28979.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_28980_33059.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_33060_35135.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_35136_36287.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_36288_39071.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_39072_52439.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_52440_52943.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_52944_53519.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_53520_54323.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_54324_54443.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_54444_58223.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_58224_59459.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_59460_59723.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_59724_59843.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_59844_60803.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_60804_63059.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_63060_63623.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_63624_63863.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_63864_64967.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_64968_65435.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_65436_65591.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_65592_65675.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_65676_66863.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_66864_67331.png Package_management_and_creation_in_Gentoo_Linux_0P983kGXdKo_2_67332_67377.png
I'm Donnie, I've done Gen2 for a pretty long time now and I'll tell you a little bit more about that in a minute. But basically this talk is about what kind of value we can provide to the other distros in terms of what kind of things you could learn and take away and apply to your own. So what we're going to do is talk through some of the philosophies, think about how you can take a philosophy from a distribution into how you turn that into package management and that sort of thing, and then run through some of the individual features we've got and the way we've iterated on those features in our package format over the past 10, 15 years to continue thinking about how can we keep improving the packaging format and not just how we handle things on top of that format like the last talk, which is mostly iterating on top of RPM rather than iterating RPM itself, although RPM does very slowly move forward. So I apologize for the terrible quality of that picture, it's still off somebody else's site. But so this is basically what I used to do in Gen2 is I was somewhat overweight and I also wore a lot of hats. I did things including maintaining most of our X packages, which is about, well first it was like five and then the modularization happened and then it was 400. And then there was a lot of other science packages and miscellaneous stuff. And I think still the desktop group manager on the council, which is the group of seven people that decides all the technical stuff about Gen2 and so on. But really these days, first of all I lost some weight, but second of all I pretty much just talk because I had a few kids, I got married, I graduated from college and suddenly there's not so much free time as they used to be. So the philosophy of Gen2 really derives from two things. One is the guy who started it named Daniel Robbins previously worked on a district called Stampede and that's purposely pixelated because that's how old the distro is and how long since it went anywhere. Stampede was a very old school distro back in the late 90s, early 2000s. And then Daniel Robbins went to FreeBSD because he got pissed off at the developer community at the guys just being total dicks about everything. So he left, decided, you know, turns out he thinks the developer community is a really important factor, not just the technical code itself but actually the quality of the interactions within the community. So he left, went to FreeBSD and he said, well FreeBSD has a really nice way to handle the packages, right? Because they've got, you know, they've built everything from source, they've got these make files but on the other hand, all the make files, everything is controlled by an individual variable, right? Everything is a unique snowflake. So the same feature in one package versus another package might be controlled by a variable with a totally separate name and it makes it really hard to deal with it. And so he said, you know, I wish I had this FreeBSD thing but in a more cohesive way and not Linux with a good developer community and that's why he started Gentoo. This is a picture of a Gentoo penguin. It's supposedly the fastest penguin which is terrible for Gentoo as a reputation because really it's not about speed at all. It's about flexibility. But we've got all these ricers who think, oh, I use Gentoo because I get faster code. No, that's not at all the case. Well, you might get 5% if you're lucky. But that's after quite a bit of work. I don't know what's going on with all these weird little points. So in terms of philosophies, it's really, five more bullets or not. Oh, yeah, I do. That was weird. Okay. So basically we were thinking, how can we have something that's user-friendly despite being something where you spend a lot of time on a command line? And so we spend a lot of time working on our package manager portage, which the main command for that is you merge, and working on the CLI actually being something that's very colorful and easy to understand because we don't really support any GUIs for package management. Everything is done through the command line on a terminal. And so the title is Gentoo package management goals up there. So thinking about if we've got these goals, what can we do to make that a reality? And really the goals were, one, it's great ease of use despite a very low-level interface because we're trying to understand what's inside the black box of Linux distros. The second goal is really, how can you minimize the developer effort for people who are writing the packages? And that pervades through a lot of the changes we've made to the formats over the years. And even more so, a lot of the changes we've refused to make over the years because they make the format too complex. And one of the benefits we've had is that it's very easy to start writing packages, and this has helped us get a lot more people involved because packages look like bash scripts. And everybody who runs Gentoo, you have to run a lot of bash stuff to just install the thing, a lot of it. And you probably screw up four or five times, and then you figure out it's important to read the docs. And so we've got people who by the time they even managed to install Gentoo, they're already at a point where they could build a very basic package because it's just a bash script and wrapped around with some fancy features to build software more conveniently. And so simplicity is a very, very important thing when you're thinking about what should your packaging format look like. And that's the easy maintenance part. And in addition to that, it's thinking about what does a package format have to look like if you want to support tons of architectures and tons of operating systems? What kind of things do you have to add to deal with that? Wow. Those transitions are wacky, aren't they? And then the other major features, these aren't as interesting, but really the ones that are cool are thinking about how do you support the possibility of installing, say, you know, five versions of GCC on the same system? Or dealing with GTK1 and GTK2 and GTK3 all installed at the same time? And then thinking, you know, Gentoo is a source-based distro, so we have to handle source-based packages, but we also want to enable people to do what they want to do. So you know, how do you support binary packages for the people who want to do binary packages? And all of these kinds of thoughts go into, you know, how do you create a package format that makes sense? I don't even understand what is happening with these transitions. It's so weird. I'm just going to skip that stuff. That's not very interesting. But sort of, you know, it's always about flexibility from Gentoo's point of view and thinking, how can you enable, you know, whether it's users or developers to do whatever they want to do? All right, so how do you enable them to use all their processors? You know, we added a bunch of parallelization stuff. How do you make it less of a pain in the ass to upgrade things when you're sitting around, if you're compiling for like 12 hours or something crazy like that, you don't want to find out that after three hours when you went to sleep after too many beers that it failed. So we said, okay, let's resume that thing. Or dealing with, you know, it turns out it's amazing how many packages out there install binaries with the same names. And it's really a pain in the ass for just to deal with this when you've got some weird, obscure thing that installs, you know, binary with a certain name. And then you've got something that's super common that installs the same thing. It's like, ah, just makes you want to kill somebody. But then also things like, you know, making it super easy for developers building on Gentoo to do what they want to do by saying, okay, it's really easy to build packages, keep them integrated with the system, and yet have them very easy to, you know, work with and code on by adding, you know, debugging support, installing the sources without, you know, flipping the binaries and that sort of thing. Is there anything that's worth saying there? I have no idea what's going on with those transitions. I apologize about that. And then the other things that are really interesting to think about are how do you deal with, you know, multiple different repositories and thinking about, you know, do you want to add dependencies across those repositories so you can actually depend on a certain repository being installed? But that really creates some issues when you're thinking about, okay, but somebody might derive from that repository and then do something incompatible and change the name. And so how do you, how can you deal with these kinds of things inside of a package format? Wow, that's so weird. I just can't get over how weird that is. So the ebuild file itself is the package format and basically it's a big bash script that's split into a series of functions that map logically to what a build and installation process looks like. So you've got, you know, some main functions that are like, okay, let's unpack the source, let's screw around with it a little bit, patch it up, let's configure it, let's compile it, let's install it and then let's do whatever weird screen around the system you have to do to actually make the thing work. And how many of you are actually developers for a distro or packages? Awesome. So all of you know that build and install process is when you're actually writing a package or nowhere near as simple as you wish they were. There's always some stupid fiddly thing you have to do on any package that's anything beyond the trivial. And you know, our goal with a package format is to make it really easy to do that kind of thing, right? So you do, okay, here's the default and then let me tack some things on to that to do whatever screwy fiddly things I have to do, whether it's, you know, doing a set on a configuration file or, you know, fiddling with a couple files on the live system, you know, renaming some things so they don't overlap and so on. And then the last thing that's really interesting about the package format is this idea of we have some level of sort of object oriented style inheritance so that we have this thing called e-classes where they're essentially like a library of functions and the individual e-builds can actually inherit from that library, use all the defaults from that library in terms of, you know, the unpacking, the compilation, the installation and then vary from those a little bit as they have to. And these other things are just some basic metadata about the package, right? What architectures does it work on? The slot is the thing that enables you to install multiple versions of the same thing at the same time, right? So, GCC 3.5, 3.6, 4.0, 4.7 and so on. And then we've got, you know, a few different types of dependencies and we're actually thinking about adding more because we haven't made it complex enough yet. So it turns out when you're making a source-based distribution, you have to think about a lot more types of dependencies that have to be on every user system, right? It's not just about the runtime dependencies, it's also about the build time dependencies, it's about the dependencies that have to be installed after the package is installed because they depend on it but it won't run without them. And you've got some other styles of dependencies like, oh, I need the headers to be available while I'm building it, but I'm building it for an embedded system on this box and so things get really weird and complex. Yeah. I'm sorry? Oh, yeah. Yeah, we have conflicts between packages. Yep. Yeah. Thank you. Yeah, so we deal with conflicts and we've got to mention that in a little bit because one of the things we did as we were iterating on the package format was think about what are the different types of conflicts and how can we enable the package manager portage to deal with those types of conflicts more effectively, right? So there's one possibility which is the two packages. One can overwrite the other before the other is uninstalled but then the other type is, you know, sort of mutually exclusive thing where they can't be touching the live file system at the same time at all. So the first one has to be uninstalled completely before the second one can be put into place which may happen if you're replacing a sim link with a file, for example, or a directory with a file. So then we've got this thing. Since we're source based, we have to think, okay, how do we enable people to build only the futures they want to use and that's what we created with use flags. They sort of map onto configure options. So you might have a, let's say you've got a package that supports a GTK interface or a Qt interface and you have a use flag for GTK, you'll have a use flag for Qt. It might add some extra functionality if you actually build for GNOME specifically beyond GTK so there might be another use flag for that that will pull in another 15 libraries. And we have lots of different profiles so that people can basically pick a reasonable starting point and then iterate from there. And so it's, you know, both on the user and the developer level, it's thinking, how can you set up people as close to where they need to be as possible and then just let them define a couple of things to vary from that. So, you know, the value of actually providing a lot of different types of user personas or user profiles is actually really high. So the basic story behind how we've iterated on our eBuild API or the EAPI over the years, every year to year and a half for the past six years we've actually come out with a new EAPI, some of them adding more features than others. So you can see, you know, this is, EAPI one was the first time we changed things. Before that it was EAPI zero. And so zero was basically like, what is this thing we have now? It's been around for like 10 years and it's just a gigantic mess. And it was a huge effort to just document the thing. And in fact, it actually has turned out to be pretty valuable to have multiple different package managers implementing the same spec. So we've got Portage, which is our default package manager, but then there's also a couple of other ones. There's a Pollutus and package core that implement the same spec and all of them deal with eBuild files. And this has been pretty nice and actually forcing us to really think through, you know, what does a package format look like? How do you describe it in such a way that it's useful to lots of different people? And so then we came out with one, a year later we came out with two, a little bit over a year later we came out with three, another year later four, another about a year and a half later five. But the funny thing that you don't appreciate looking at this, it looks like this is a nice regular March. And in fact, it turns out that three happened because we were so sick of the stuff we were working on for three, we decided to push most of it to four and just get something out the door. So in reality, it's like big features, big features, almost nothing, huge pile of features, bunch of features. And we're just going to kind of run through some of those so you can get a feel for what those features looks like. So actually why don't we just pop out and look at some actual code instead of just sitting here looking at slides all day. What did I say? That was MPC? Yeah, okay. Devlib's MPC. So this is what it looks like when you're in a package tree. Yeah, it looks beautiful over here, guys. Yeah, I'll slide it over there. See if I can just maximize the thing. And maybe make it a bit bigger. Can you read this in the back yet? Cool. Let me see. All right. So this is basically what an e-build looks like. And it's just a simple text file. I used nano for e-builds because I'm terribly pathetic. So at the top you've got some basic header info, copyrights, and so on. Then here, one of the first things we do is define the EAPI so that the package manager understands how do you parse the rest of this thing. We inherit some e-classes to give us some default ways to handle various things and a few sets of utility functions. So eutils is sort of a basic subset of functions. We've got a set of functions to deal with libtool. We've got a set of functions to try and handle multi-lib systems like 32-bit and 64-bit on the same system simultaneously. And then you've got your basic sort of metadata, right? What is this package? Where do you find it? How do you download the thing? What's the license? Because we actually do support things like only installing packages with a certain set of licenses. So you can have, for example, a completely free or a Libre software system. You've got a slot. And the slot is what you change if you want to install multiple versions at the same time. You've got what architectures you work on. This I use thing defines the set of possible features you can vary on this package. So in this case, there's only one, you can build the library statically or not. And that means you'll always get it dynamically, but then you can also tack on the static ones too. And then you've got the dependencies. And these are just very basic sort of standard notations for indicating the numbers and things. So you've got a greater than or equal to. Here's a category. Here's a package name and then the version number. And the same thing in the second place. And we've actually got a fairly flexible syntax for dealing with that. So you can have greater than or equal or less than or equal or just greater or less. And you can also have a star and asterisk if you don't care about the version or you don't care about some set of the version. And then it's just regular old bash, right? So the runtime dependencies, you just use the depend variable. And then you can go down and look at what the functions look like when you're doing this thing. And you've got an unpack function which just calls out to unpack the tarball, go into the source directory, and run some lib-tools stuff. And you compile. So you have a configure function that has some convenience features like a bunch of default arguments. And then you've got here's handling the use flags. So use enable. And then when this use flag is specified, here's the configure flag you passed. So you pass enable static. And then if it doesn't work, you fail. Die. Yeah, right. So I'm purposely showing these in order of how we came up with them. So this is the old school EAPI zero. We actually decided, you know, in the case of following our philosophy, trying to make it simpler for developers to write these things that, in fact, it would be nice if you didn't ever have to write this die thing, right? And just that was how it worked by default. Because normally, if something wrong happens when you're building or installing a package, you don't want to install the thing. It's going to be broken. So you want it to die by default and instead specify something if you don't want it to die. So one of the things we ended up adding in a later EAPI was this idea of a sort of non-fatal prefix where it doesn't die. And normally it just happens that it always dies or something goes wrong with nearly every one of these commands. So you unpack, you compile, you install, right, which is just basically a make install with a destination directory and die if it fails. And then there's the little bit of fiddling that I'm talking about. Every time you're writing a package, there's always some fiddling that has to happen. So in this case, it's getting rid of lib tool archives, installing a couple of docs. Then something I should say is every one of these functions that's prefixed with source, those ones happen when you're doing a source build, right, only when you're doing a source build. But every time you're installing a package onto the live file system from the source directory where it's compiled, then you've got package functions that run package prefixes. So you've got a pre-installation function, a post-installation function that you can do run before and after all the files from the build are merged onto the live file system. Which enables you to get things ready, make sure nothing is in the way. And then it seems like there's always packages out there with terrible build systems. And so you have to tweak things in the file to get installed because they don't realize they were going to get moved after they were built, right? It's surprising how many people write just awful build systems that expect you to download the thing on packet, build it, and then run it in place and never install it onto a real system as part of a distribution. Okay, so that's what a very basic EAPI0 eBuild looks like. Now let's see if I can manage to get back into the presentation. Okay, so then we came out with the EAPI1 about five, six years ago. We added a few really interesting features. Thank you, and I'm only showing the ones that are kind of interesting versus the ones that are minor tweaks. So we added this idea that you could actually have a use flag default in a single package that wasn't the same as the default for the whole system. Because sometimes with a package it makes more sense to enable a feature than not. And so we said, okay, in this case, you could turn off GTK support but it would be kind of stupid because then you wouldn't have a GUI at all. So if they haven't manually specified switching away from their global default in their profile, then it will, for example, add GTK instead of not adding GTK, even if the global default is to not add GTK. But if they say, I don't want GTK, it will still override that. So there's a series of overrides. We also had this idea of being able to actually properly depend on things in a certain way. So we added a certain slot. This was very useful, for example, with GTK1 and GTK2, where it became increasingly difficult to specify all of the things that matched a slot using a series of version numbers. And so it ended up working out pretty well to add a way to depend on a certain slot. And a lot of distros, instead of doing the slot thing, they decided, oh, we're just going to put the version number into the package name. So there was a package that was GTK1-1.2.10, for example. And we thought that that wasn't really an elegant way to handle that. So we added a new concept to deal with the parallel installations instead. And it actually turned out to be really useful on development systems, because you could have multiple versions of GCC. You could have multiple versions of GCC that cross-compile due to different architectures on the same system. And it actually had a lot more benefits than anybody ever expected, I think. So we'll just take a quick look at one example of an e-build that does this, WCS.CVS, 0.4.1. So it's all the same stuff from last time. You've got your basic header. This time, the difference is EAPI1. You scan down. Most of the stuff looks about the same. But you start looking into dependencies, and now we've got this colon and a version number after, indicating that we're depending on a certain slot. So it's only the JCOM and one slot, and presumably JCOM and also has two series. This is actually really common with Java packages, is that they'll have simultaneous sort of independent series, all of which are maintained ongoing, because Java is a really nice, really popular architecture for doing enterprise software development, and those things have to be supported for a decade. So this is just an example of what the slot dependencies look like. But other than that, all of this should look a little bit familiar, because you saw it a few minutes ago. You've still got your basic build dependencies, runtime dependencies, unpack function. You've got, we're doing some utility functions that will unpack the source code, apply some patches, and some convenience variables to say, OK, the patches are in that files directory for that package. And then this one expands to actually the package name and the package version. And let's see. Trying to remember, does that one actually have any, use defaults in it? OK. Well, maybe not. So that was a couple of minor features. But then we came out with EAPI 2, and we added a couple of really cool things to it. One of them was this idea of arrows for where to download it. And this is just another, yeah. OK, sure. So the notion of slot, it's primarily intended for packages that were intended to be installed in parallel at the same time. So like GTK 1, GTK 2 are a really nice example of this. There's GCC, for example, so that you could have GCC 3.6, 4.0, 4.5, all at the same time on the same system. The slot is just a way you tell the package manager that these are all the same package and we should enable them to be installed at the same time instead of forcing the lower version to one to be upgraded and removed. So that means that the package maintainer is the one who decides what would be the slot number? So the slot number is specified in the eBuild itself. So the package writing the eBuild specifies the slot number. And normally the slot number would be zero. But in a case where you're actually dealing with multiple slots, it normally corresponds to something like the major version. Or in the case of Python, you might want 2.5, 2.6, 2.7, 3.1, 3.2, all installed at the same time and the slots would map exactly to those major dot miners. So it's just a string. It's not numeric necessarily or anything. And it's just a way of saying, install all of these things at the same time rather than removing the lower versions when you upgrade to a higher version. So in EAPI 2, we said, all right, let's deal with some more of just the weird build stuff that people have to deal with. And so we added these things called source-era arrows. The source-era is the download location for files. And so normally it looks something like this first piece before the arrow. But it turns out there's a lot of developers who don't really understand how to make packaging friendly who don't put a version on their tar balls, for example. Or who name their tar balls in such a way that the suffix doesn't map to what the file format actually is. Like this dot tar might actually be a dot tar dot gz. And so how do you deal with that in a way that doesn't force people to download them manually so that everything else works nicely after the fact? And so you're not having hash collisions because everything is hashed. And so we say, OK, foo.tar.gz is got an md5 and a SHA-1 and a SHA-256-X. And so you download that thing. And if it's not versioned, then you can't do that. Because all the different versions are going to have the same file name. And there's going to be five different hashes for the same file name, which doesn't work out too well. And so we added this thing that said, OK, download this file with this name, but actually rename it to this other name so that your hashes work out so that you're not overwriting files with each other and so on. And then we also added this idea of use dependencies, which is not just depending on a certain package. Because it used to be before this. So we would depend on a package. And then in the functions within the eBuild, check and see how that package was built. And so you would depend on it. And then you would start building your package. And you would say, let me go look at that library, see if it was built the right way. If it wasn't, I fail. And I'd die. But that sucked. Because what would happen is you would die three hours into a 12-hour upgrade cycle. And then people come back the next day and they'd be like, what happened here? Why did I die? I wanted that thing to keep going. I don't care whether it doesn't have the right feature. You should have just skipped the package and kept going. And so we said, yeah, you're right. You're right. Let's add a way to depend on that properly instead of hacking around later on after we've actually started the build. So let's depend on the library built with the right feature in the first place. And so it's about simplifying the developer's job, but also making things easier for the user and pissing them off less, basically. And so we've got a lot of different styles of this. And they all basically map to build with this flag, build without this flag. If I'm building with this feature enabled, make sure my libraries underlying me are built with this feature enabled or vice versa. If I'm building without this feature, maybe my libraries also have to be built without this feature because they may be incompatible features. And let's pop up an example of this one and check it out quick. Or 5.1. All right. So in this case, you can see, again, EAPI2, right? Everything else looks very similar. You've got your basic metadata. You've got your inheriting, your E classes for convenience functions and so on. But right here, you see they name their tarball, rbsource.tgz. No version numbers or anything. Cool. Thanks. Which is a serious pain. And so we just say, OK, let's have an arrow and let's name it rb-5.1. Because that's the name of this package which this thing expands to. Let's see if we actually have any. Oh, cool. And in this case, we've also got use dependencies as well. So depends on a bunch of stuff. You've got some logic in there. If you want OpenGL, you have to get this, this, and one of these things. One of them is the Mesa library built with motif support. And the other one is this libglw thing. If you avoid that, then you get this and another thing. So does that make sense? Cool. All right. So that's two. I guess we're going to have to move a little faster because I have 10 minutes left and three more to get through. So oh yeah, here's some more stuff in two. So here's the blocker syntax that somebody was asking about earlier. We've got blockers that can be installed on the file system at the same time or ones where the first package has to be completely gone before the second one can go over. And that's like, OK, let's block this really hard. Let's double block this thing. And we also added more granular builds. And this turned out to be really nice in terms of letting the packages be shorter because we started out having just a big compile function, do all of the build things. And then we said, actually, you know what? It would be nice if configure and make were actually separate functions so that if the make part is normal, we can just let that happen and not have to write that code. But if we have to tweak around with configure, add some more arguments and so on, so every time you have use flags, you always have to deal with those for configure. So we have the separate configure function where we just basically call configure with the right arguments for the features we want. But then make just happens and it all works. Well, the same thing, we split out the unpacking from the preparing. And preparing is basically, for the most part, patching. So you want to just, don't worry about the unpacking. That just happens by itself. But we're going to have to apply some patches, maybe tweak some auto tools, files around, do an auto recap, and so on. And the other thing that really made developers lives easier was adding defaults for every one of these functions. So all of these steps are called phases. So these are phase functions, unpack, configure, compile, so on. And so we made it really easy to do the default behavior and then tweak on top by having a function within each phase that's just called default. So if you're starting the configure function, you can just say, okay, default. That's it, and then it runs configure and everything. And then you can play around a little bit afterwards instead of having to manually specify the configure part. And we'll pop up a quick example of this. Cairo, whatever version that is, 1.8.10. Hopefully we've got some default functions in here. So, yeah, here's an example of prepare. We don't specify unpack at all. It just happens because it knows what kind of format the file is. It downloads that file and it says, oh, that file's got the right suffix. I'm just going to unpack it and put you in the right place. And then some patching happens. Let's see if we've got it. And the same thing with configure, right? So we've got source configure. You don't see source compile anywhere. It goes straight to the install. So the compilation just happens. It uses the default information. But the configure part, we have to tweak around, you know, skewer around based on what the host is, append to some C flags, pass all our configure arguments based on the use flags, that sort of thing. Now three, I'm not going to bother showing you this one because we're running out of time here basically. But what we did with three is we were going to add a ton of stuff. We worked on it for like a year and a half. And then we're like, this sucks. We're never going to get this out the door. So we'll just do something now to help people who really need something and then keep working on these other features. So the main thing we did with three was prefix support, which means, so this is a Mac here. It's actually running OSX right now because I haven't installed Gentoo on it yet. And it's got Gentoo installed on it in a prefix. So in my home directory on this thing, I've got a Gentoo subset on its own little piece of the file system. And it doesn't install on the root. It just sits there in my home directory. So that's what prefix support is. And you have to do a lot of stuff to deal with that based on packages that assume they're going to be living in the root and look for other packages from the root on down instead of from the top of the Gentoo installation on down. So we had to do some tweaking around for that. Now four is a big one. We added a ton of stuff, again, trying to make developers' lives easier. Thinking about, OK, some of these things are just ludicrous where it was just like a historical accident that we didn't have it. So we've got this install function. But the default behavior for the install function was do nothing. So you would build the thing. It would be sitting there in the build directory. And then if you didn't specify an install function, it would just sit there in the build directory, get deleted, nothing would happen, which is ludicrous. So finally we said, OK, we're going to install, do something that makes sense. And it's going to install stuff by default instead of doing nothing by default and just leaving things there. And we added a bunch more capabilities to basically fail fast and fail early so that users, again, aren't getting screwed like two or three hours into this four or five-hour-long thing so that you can actually run some code from inside the packages before the full installation starts happening while it's still at the dependency calculation and asking whether you want to proceed phase. Some more things for use dependencies. Here's one of the questions we had earlier on. We added this non-fatal prefix to say, OK, actually the normal behavior we want is to die by default if anything goes wrong. So we shouldn't have to specify that. But instead, we're going to make that the normal behavior. And if we don't want to die for some reason for something that truly doesn't matter, then you have to say non-fatal before that. And then we added a new variable for basically dealing with installations that have to be interactive for some reason or another. There always seem to be a few of them out there for some weird packages. But we're always adding things to deal with all the weirdness things that obscure packages do. So in some cases, they have to be interactive because you have to accept some license before you can proceed, for example. And you have to actually visually see that license. You can't just accept it implicitly. So let's pop up, get quick, and take a look at that. Whatever version that was. Get 181. All right. So you've got EAPI 4 up top. All this other stuff is pretty normal. So you've got some bashiness. You've got inheriting some functionality from E-classes. Your default metadata, your dependencies. Let's see here. Pretty sure there's a non-fatal here. OK. So here's an example, right? If we're running the test suite and it doesn't successfully clean out beforehand, let's not die and just keep going. Because nobody wants to fail out because the test suite directory didn't clean up first. Everybody's going to want to install and have something that works, even if that doesn't work properly. And then a bunch more stuff. One really cool thing was, so we've got package dependencies that can go in these hierarchies and do ors and mutually exclusives and so on. But we couldn't do the same thing for features, for use flags. So we added the same capability to specify feature dependencies in the same way we could specify package dependencies. Because sometimes features do depend on one another. And I've only got one minute left, so I'm not going to show you that. But I hope you get the idea because it looks exactly like package dependencies. Sometimes you have a stack of features, none of which work without the next one down. So if you've got GNOME support, you might require GTK support first. It's a great example of this. EAPI 5, which is the last one we came out with, adds capabilities to do rebuilding better when library versions change, which is the main idea here. So you've got these ideas of sub-slots and slot dependencies that are smarter so that, for example, if the slot version stays the same, then I keep working. But if the slot version changes, then it will forcibly rebuild all the packages that correctly specify that they require to that slot. And that's really all I'm going to say about that so we can just wrap up. So EAPI 6 is something we're now thinking about, thinking about how can we keep making developers' lives easier and users' lives easier. And I'd like to thank you for your time. Do we have time for questions? Great. Anybody got questions? Yeah. It seems like a lot of this stuff that you have built in is set up for Auto Tools, but what about other stuff like SCONs and stuff like that? That's a great question. So we do, I was talking about E-classes, which are basically our sort of libraries of functions, and we've got E-classes for dealing with different build systems. Auto Tools is by far the most common, so our default functionality is up to deal with that. But we've got a SCONs E-class, for example, where you are a CMake E-class, where you inherit that E-class and that replaces all the default functions. So now, source configure and source compile are set up to deal with SCONs or CMake instead of being the default Auto Tools behavior. So we try really hard to make it easy to deal with that. And it's the same thing for version control systems. We've got ways to very easily build from live version control systems using E-classes. So it's sort of CVS subversion, and now we've got Git, we've got BZR, Mercurial, and so on, so that you just inherit something, say, okay, the repository's over there, and it knows how to get the code, how to download it, how to build it, and everything. Yeah. Yeah. Then DOS upstream. Well, GitHub doesn't DOS though, right? Given how complex it's got now to write an e-build file, is there anything that, looking back, any features you wouldn't add if you were doing all again? The features that I kind of regret, like we needed them to have a complete system, but are the things with really complex sets of syntax, like all the different ways to do use dependencies, for example, make it a little bit complex to remember, like you've got, like, an exclamation point in front and a question mark in back, and there's an equals sign there somewhere where most people actually have to look at, like, a reference card when they want to figure out how to get it right, and that's exactly what you don't want. But, you know, that said, so we try to keep things as simple as possible, but no simpler, right? And sometimes there's just not a way to do it simply. So we try and just do the minimum we have to do to get the job done, but not adding extra syntax for its own sake. So, it's easy to write e-bills as the upstream behaves, so if you get upstream to change, then it's easy to write e-bills, so it's just kind of hand in hand with the distro and the upstream. Yeah, yeah. Upstream cooperation definitely makes doing packaging a lot easier. No question about it.
could learn and take away and apply to your own. itself but actually the quality of the versus another package might be controlled by a working on our package manager portage, which the really support any GUIs for package management. it's great ease of use despite a very low-level people involved because packages look like bash to that, it's thinking about what does a package developers building on Gentoo to do what they goal with a package format is to make it really behind how we've iterated on our eBuild API or over there, and it knows how to get the code, how