A Method for Distributing Applications Independent from the Distro
White, Langdon
For many years the Linux distro concept has been about "inclusion of applications" sometimes at the detriment to co-habitating applications and the stability of the core OS. Much discussion has been made over the years about JEOS, embedded Linux, custom distros, applicance building, etc, but not a lot of discussion about how applications could be delivered such that they were more readily able to co-habitate. Open source applications (because distros are so "inclusive") are put through significant scrutiny around their design and deployment related to their integration with the core OS that may or may not make sense. The scrutiny is certainly more intense than proprietary software is required to undergo. This talk proposes a new'ish technology, Software Collections, as a method for resolution.
 smarter than me. There we go. I wonder why I didn't echo. So as I said, I was a camp counselor for a long time, so you know, big parade fields, etc. And what I do is I work for Red Hat. I'm what, you know, this kind of new thing called a developer advocate or developer evangelist. And so I've spent, I don't know, about 15 years as a, mostly as a software consultant. So a lot of different projects, mostly kind of big enterprise sort of things. And now what I do is I talk to kind of developers who use kind of Red Hat products, in particular Enterprise Linux, talk to them about what they hate about it, and then I go and talk to engineering about how we can fix that stuff. So that's mostly what I do. And that's a little bit about me. So this talk is actually, it's kind of essentially about this concept called software collections. But it also, I come definitely from the developer background rather than coming from like a packages background. So I think I have a lot of biases towards being a developer and, you know, not needing things like QA or, you know, any of that, any release stuff, you know, and it will just send it out there and everything will be fine. No security updates, et cetera. But so just a little bit of, you know, on me. So here's a little bit of my argument. Over the last, I don't know, whatever, 15 years, Linux distributions, you know, they kind of started in this kind of world where, you know, the open source community was relatively small. The even proprietary software and open source software, neither one could you trust basically at all. They, you know, it was all run by developers who didn't really know or care what they were doing. So things like security updates would never happen. QA, if you were lucky, you know, et cetera. So early on, I argue that the distributions kind of formed to try to basically solve some of that ease of use problem and kind of redistribution problem around security patches and software and kind of things like that. So that you kind of had, you knew where you could get stuff from and you could also redistribute things so that they would stay secure. It was also kind of over the, arguably earlier than that, but, you know, over the same 15 years or so is where, you know, kind of the dynamic linking concept really took off and caught on. You know, there were definitely pockets before that, but it wasn't until, I don't know, like 96, 97 where it was like, it started to become like the way to do things. So, and then the last thing was actually license assurance. This is, was much less of a problem with proprietary software, obviously, although in fact they were hugely violating licenses anyway, but you couldn't tell. But with open source, it was a big concern, right? So you wanted to make sure, particularly as open source started to be adopted by the enterprise, that you knew what kind of licenses you were getting into. Also, feel free to interrupt me. So you can tell me I'm crazy. If it's just blanket crazy though, well, let's wait till the end. So particularly now in the last maybe five years, there's been, and I couldn't find a good graph for this. I've actually seen one before. I was looking at, I was talking about somebody earlier that, you know, kind of just the package count in Fedora, each version. It's just scaling like crazy. Obviously in something like enterprise Linux, we have significantly less packages, but even that is still, it's still a nice, you know, strong curve. And so between that, now we also have a massive plethora of actual Linux distributions that are reasonably popular. So it's not, you know, three big ones, right? And then everybody else, right? There's, you know, there's a good chunk at the top, right? And then Linux in general is also becoming hugely adopted, particularly in the enterprise, you know, kind of in server rooms in particular. But then even, even to some extent in the enterprise on desktops, you know, I don't think it's ever, you know, I don't think we're ever going to see the year of the desktop, but, you know, it is in some places, you know, I think it's largely going to be the only people who use kind of desktops and by desktop, I mean laptop half the time, are going to be people like you guys, right? It's going to be just developers, you know, just people who actually make software who use these things at all, right? It's everything else is going to be tablets and, you know, in that kind of tool chain. So we really need to kind of focus on making sure that we can distribute things well. So here's my, here's the extent of the math I can do anymore. You know, so we have a whole lot of projects, and then we have versions of each project, then we have it across distributions, and then we have versions of the distributions, right? So now we just made the thing massively wide, right? So kind of typically, you know, we, we just do this kind of, you know, take a tar ball, right? And then kind of have to figure out the distros, packaging rules, and then kind of figure out how to build the thing itself, and then start to, you know, kind of put that into a package. We got to follow all these rules and make sure it goes in the right places, etc. And then on top of that, inside an individual organization, there may be kind of a second set or third or fifth or 47th set of packaging rules within that organization. So you have, you kind of have to solve the problem over and over again. So what would be a lot nicer is if we could kind of say, okay, I have this package, right? And if you saw, I don't know how much of Donnie's talk covered this earlier, I know he's covered the past, but, you know, whatever that means, right? So a package. But essentially, if we had a blob of thing that we could run, say, on any distro, right, that was OS independent version wise as well, so you can kind of go this way, right, for the package could drop on each one, or you could go this way, right? So, you know, version two, version three, version four of the OS. And then the other thing is, can you also are there, can you start to trust essentially the developers to have packaging rules that make sense, you know, for their application, rather than it being a distribution problem? So lots and lots and lots of people have tried to solve this problem. Here are whatever six, five that I thought of off the top of my head, and ones that I could find random, easy links for. Actually, I had Linux apps in there too, but I couldn't find a good, like, place to point people off the top of my head. So they, some of them solve the exact same problem, some of them kind of take it from a slightly different perspective. So that's what I think is kind of interesting is to, it's kind of, you know, every piece of software anybody writes has trade-offs. And these people clearly were taking different trade-offs, right? So kind of the environment modules, SEP, and to some extent Gen 2 slots kind of all take roughly the same set of trade-offs, very similar actually to software collections, where it's kind of like, I want to have multiple versions of the same thing installed on the OS at a time, and I want to rely on the user to choose between them when they're running applications. OS Tree, relatively new thing that is really actually about, it's really about supporting, arguably supporting debugging. So you can take an entire set of binaries and kind of swap them out in and out, and it's for, you know, doing like debugging of GNOME, right? So it's not just one application. Yeah. We're going to look at doing that for off-rope data. Using OS Tree? Yeah. So mind you, GNOME I think has three or four different ones that all try to accomplish this goal. There's Glick 2, Glick, it's another one too. And then you have update alternatives, which is also trying to solve the same problem, but in a very different way, which is I want to be able to switch for the entire OS what the version of something is. So instead of per application, which is what these guys do, it's actually for the whole OS, right? And that can kind of cause some problems, particularly something like Python, right? So you switch the version of Python and your OS on Linux, right? And bad things are going to happen. And like I said, there's lots and lots more. So taking some of that thought, right? And taking some of the... So Red Hat has a little bit of a problem where, particularly for Enterprise Linux, a lot of our customers want to use whatever version of Enterprise Linux that they're using forever. And I mean forever, right? Because, well, one of the reasons is because the application that they're running on it is working just fine, right? So why would they want to invest normally? What you have to do is you have to invest development in effort, QA effort, release engineering effort into modifying the application to be able to run on some future version of the operating system. If it's a junk application that just does what it does, why would you make that investment? So as a result, they instead yell at companies like ours, right? And lots of other ones. Microsoft has this problem a huge way as well, as you all have, I'm sure, seen with their big EOL announcements every time they announce an EOL. And then it'll... It's like five times they EOL it, and then it'll actually die. So they have this huge problem as well. They do some other interesting things. Look up the civilization fix, the video game civilization, the fix they did in Windows sometime. So what a software collection tries to do is let's instead of having the application kind of natively touch the OS itself, let's put a layer in between it, put a layer of indirection in there so that we can now take this blob that's in between. It might be Ruby, right? Or it might be kind of a full stack. So it might be a web server plus Rails and Ruby and everything else. The idea is there is a blob that you depend on that's kind of above generally, above the actual OS and below your application. So that's kind of what a software collection is trying to do is saying, okay, let's identify that blob and let's try to make that blob portable. So now what you can do is you can say, okay, I have this janky old Ruby app, let's say Puppet, that is running on Ruby 18, right? Why is it running on 18? The way I hear it is because Enterprise Linux 5 runs Ruby 18 and why is Enterprise Linux 5 running 18? Because Puppet runs so. But any which way, the point is, you know, you take any old application or older application and what you can do is you can kind of say, here's this blob, I'm going to guarantee the API on the top of it and then I'll change how it delivers that API underneath so that it can be portable across multiple versions of the OS. You also have the problem the other way, right? So there's a lot of push for people to move to Python 3. However, not all the distributions are ready to make all the changes to be able to move to Python 3. So how do you start writing applications in Python 3 right now without the, you know, kind of working your Linux? Well, one of the ways you can do is with a software collection. So it's kind of, you know, by the two examples I'm kind of giving, right? You can kind of see it's OS independent, right? The OS can continue on its merry way and your application can decide to upgrade when it's ready. Or you can be aggressive and say, hey, I'm writing something bleeding edge brand new and I know I want to use the most cutting edge version of whatever because I'm not going to change it again for 10 years. So let me get that right now and use it right now to write my application. So it kind of going into it a little bit more. So it really is, as it says, it's kind of defining a process a little bit more than a product, you know, so it's kind of a conceptual thing. You know, there is an implementation, but that's kind of what it does. The other big thing that it does is it packages outside of the normal kind of Linux file system. And that's sometimes a problem for some of the people I think in the room. And it allows multiple versions to be co-installed, right? And then per application you can essentially start, or you can say, this application gets this blob rather than that blob. So the way we kind of defined it is it installs under opt with, you know, the L acronym that is the Linux naming that I can't actually say. You go register for a Linux name so that you can get kind of a short code there and then you put your application name there. And that's kind of where it installs. And then you have kind of some scriptlets that know where that is and can provide that information to the application. And the way it basically does it is by fancy-path-ing, right? So the application doesn't have to know, this is the other, sorry I didn't mention this part. One of the big differences between the ones that I was talking about earlier and this one is that the application doesn't have to know that it's not the base OS version, right? Which is, I think, hugely important because you don't want to have to rely on your average developer to figure out where it's going to be. And again, to allow for that OS independence, you know, someday, right, all the Linux distributions will be on Python 3 and that will be what's at user bin Python. So what you want to do is be able to write the application into, say, user bin Python at the top and then it just goes to the right place rather than having to, you know, do fancy sim links or, you know, or make a mistake and say, or have user bin Python 3 and then find out now you have to do engineering effort to port it. So in a little more detail, here is kind of the way the one for Ruby works that we have built. So here's kind of regular Ruby, right? You know, assuming that you installed Ruby, the other nice thing about this is that you don't actually have to have, I can't remember if Ruby is installed by default, but you don't have to have Ruby installed except for this particular application. So in other words, it doesn't have to be anything in user bin Ruby. It can just be down here. So I think it's pretty obvious, but basically here's the normal path, right? And here is our special path. You know, we ship a Ruby 193 variant, you know, one of these blobs. It sits under our age because that's our name. And that's pretty much all there is to it. The disk little script here is what does all of the magic such as it is, which it basically kind of provides those paths so that the command to run these, which I think is down here. So SEL enabled Ruby 193 bash. So what you do is you kind of say, here's the command, right, SEL? I want to turn it on. All right. And then which one do I want Ruby 193? And then what do I want to execute with it? So if you just want to get a new shell back where Ruby 193 is available, you just run bash. But it could just as easily be, you know, some app, right? So from a kind of, if you're a sysadmin, it would be like an app that, and then you put this in, you know, a shell script or a start script or something script, you know, for a developer, he's going to drop in a bash most of the time. You can add this to your.profile. And so it's just always there if you want as well. Then we talk about, you know, there's basically some tools out there that kind of help make this stuff work. And so there's some links. I think all these slides are available somewhere. Joe would know better than me. I will email them to Joe. He will make them go. This is the ones that we've done already. Actually, it's some of the ones we've done already. We actually have some that are out there for like hdbd 2.4. I can't remember what else is up there. That's just kind of public. There are some other distributions that are related to ours that also have all of these. There's a guy, I think, who's talking next to who had a hand in those. So I think they are well represented now actually in CentOS, Scientific Linux, and Fedora. So Fedora just approved them whatever it was last week or was it the week before? Okay. I can always pray. All right. So it's getting there. There is some of this stuff built for Fedora, but you can't get it in kind of the mainline distro repos yet. So that's basically most of my slides. I was expecting more interruptions. I invited hecklers and they're not delivering. Exactly. That's true. That's true. So here's kind of my argument. I'm probably crazy, but I think vendors in general are a lot more trustworthy. When I say vendor, proprietary software, open source projects, everything, any kind of application, I think the distribution doesn't have to take on as much of a responsibility for making sure that all the stuff in the world runs. Yeah. Yeah. Yeah. Which is not a high bar, but let's say are reasonably trustworthy. So they're even more than that. I think a lot of this stuff, most people understand, most people who do this kind of work, most of them they understand what's important, how to break their things up. Yeah, they're still false, but I think a lot of them are starting to get it. There's well-known libraries for particularly security-related things. So people mostly use those libraries. I think Bruce Schneier is finally getting through to people and saying, stop writing your own encryption libraries. So same kind of thing. I think in general they're considerably more trustworthy so that I don't necessarily think that every distribution needs to package everything under the sun. I also think another big reason is redistribution. One of the things that software collections has that kind of fails and goes backwards in time a little bit is a lot of things are considered to be statically linked because the only way you can be portable across OS is if you carry the bits you need. So that means static linking, sort of, or at least it means that it's got to be carried in the bundle. So it can still be dynamically linked, however it's got to be dynamically linked from within the software collection. So one of the things that I was certainly confused about when I first looked at this stuff, so if you think about an RPM, one software collection does not equal one RPM. A software collection is more like 70, 1,000 RPMs. So it's broken up just like you would do a normal package. It's just that it's all kind of in this one context to say, hey, I'm part of this one software collection. So you do, it is better, quote, unquote, than straight out statically linking, but you still have a problem of redistribution. So one of the reasons that dynamic libraries are nice is that you only have to copy that one file onto every machine. With a software collection, let's say it's open SSL and you have 37 software collections on every machine all using open SSL. That means you have at least 37 different copies of that SL RPM. So you need to distribute that all the way through. However, and judging by the overflowingness of the config management talk, config management has really finally taken. I remember the fits and starts. I mean, there was a huge push for it, I think, around 2001 and just never quite took, but it's really kind of taken now. So I think redistribution, particularly in big data centers, has gotten a lot easier. So I think that's why we can kind of take this risk. The other thing is the other risk you have is that now you have a copy, you have 37 copies of open SSL. Disface is not that big a deal anymore. So does it really matter that you have lots of different copies of the same stuff on one machine? And deduping file systems someday, I pray, will be default and so it won't matter anyway. The other thing is that I think users are more sophisticated. So I think that they can, they understand the context that they're, and when I say user here, I mean I develop them, right? So somebody who's going to consume one of these software questions, they're way more sophisticated about what is going on underneath and can usually figure this stuff out. Yeah. I was going to say that the nature of the software collection does not mandate that you have 37 copies of open SSL on the system. It may be that you require three specific compatible versions. And given that it's built on.cl, you can build a, an SCL top and other SCLs. So you can narrow that down manually as well. Yeah. So I think dependent collections, right, are a very real and very important aspect of this that we're still working through some of the, like there's a, I don't know where it's published right now, but we've been working on trying to get a good guideline for how to do dependent collections, right? So do you really want to do a rail software collection? Well, not really, right? You want to do a Ruby software collection and then have a rail software collection that depends on it, right? So, and what, what's the right size of a software collection, right? What is, you know, should it be a full on, you know, kind of web server with, you know, like say Ruby and Rails, or should it be, you know, piecemeal? And I think a lot of those questions I think are yet to be known. The other thing I think that is going to be interesting about this is kind of the, the move towards containerization, right? So containerization, and I'm using that term to mean, to be as far away from possible from any individual implementation, right? So you have things like Docker, you have LXC, you have, you know, this combination of open, we use an open shift which is like C groups and SC Linux. You have, you have actually much older and much more sophisticated versions in other operating systems that are not Linux. But the kind of this containerization concept where now we kind of say, hey, this whole application is now a blob and it's portable and we can run it into certain kinds of conditions and it can be updated on its own. I think that's going to be the next step for some of this stuff. And I think a lot of that will rely on something either exactly like software collections or something very similar because it, it allows you to kind of bring that bundle inside, you know, your container. Are you ready for questions? Yeah. All right. I've got a mic. So if you have questions, raise your hand, we'll bring the mic around so it turns up on the video later. One thing I didn't follow. If you've got two software collections installed for two different versions of something, so Ruby, what's the mechanism by which the application gets to use the appropriate version? So this versus like update alternatives. So the question was basically like, how do you say I want this application to use Ruby 1.9 and that application to use, you know, Ruby 2, right? So this, of course, is only at the very bottom of the screen here. So you probably can't see it. I should have probably put it at the top somewhere too. But this unlike, say, like update alternatives, you, you tell the application which one it's using later. You know, the application doesn't actually know, but you say SEL enable whatever collection you want to use, and then the application you want to run so that it's basically running it inside a shell of its own that has the kind of correct paths. So it finds it as if it was at its normal locations. Right. So this is changing the environment. Essentially. Yeah. Try to organize your questions by row. I don't even know why I got up at 6.30 this morning to go to the fitness center at the hotel. I'd like to ask how security updates are propagated to containerized applications in software collections. So say I have a Ruby software collection and Ruby depends on some library that got the security update upstream. Does it mean that the maintainer of the collection will have to update it manually? And if I have like five collections which depend on this library, does it mean that five different people will have to update their collections to include the update? Exactly. Yeah. That's the problem. Not if you do dependencies. So like I said, if you use kind of, let's say you had a Rails 3 and a Rails 4 both dependent on the Ruby 1.9 software collection and the security update happened in the Ruby 1, you wouldn't have to update the Rails 1s in theory. I mean one of the things that I like about software collections is it kind of also lets you, well, like dislike I don't know, but it also lets you defend kind of your QA problem, right? So one of the reasons that the enterprise doesn't want to upgrade, often it may be that they expect the code to completely work, but they don't want to test it, right? QA resources are often the ones that you have the least of in any organization. It's often the constraint at any engineering organization I've ever worked with that keeps features from happening. And so when you're doing, again, kind of talking about that janky old app that is just running and doing its thing, why would you want to waste that very limited resource of your QA people to test the upgrade? So I also would hope that the, you know, if, I guess I kind of expect that in a particular enterprise, much like I would kind of expect with OpenShift is like in a particular enterprise, I would kind of expect them to build their own software collections for use in house, probably depend, hopefully, right? Dependent on, you know, for a large percentage, 90, 95% of the functionality on somebody else, right? So like on Red Hat or on the community in general or on something, but that they would have their own one so that they could be confident that the API layer at the top is what they expect completely for the applications that they're building. You know, it does two things. One, it gives them more, you know, kind of tolerance for moving across versions, but then it also does that kind of opposite effect, which then you can still control your developers by saying, here is the approved Ruby software collection, right? Here is the approved Python software collection. Don't go installing random other stuff unless it's made it into the software collection. One back there. Yes. I have a question on how this SCL enable works. So can I use multiple of these environments at the same time? And so it would sort of only map this for this one process with something like P-Root or Mountbind or how does it get into the environment? Yeah. So you separate it by a space, right? But you can just kind of list out multiple ones that you want. And it's not that cool. It's just paths, right? So this is not, you know, teruding some brand new file system or something and running in that context. There's no security here, right? None of that stuff. This is just to provide the environment that is kind of so when you make a function call, you get the right binary. That's it. Yeah. So what is like the lowest level of libraries that you include? For example, these packages, would they include Lipsy or do you still expect that from the distribution itself? The package maintainer needs to decide that. And everything that you don't include makes you less portable, right? But makes it so that you have less of the problem that he was talking about back there of massive redistribution updates. But I think it's hard to know like all the pieces that you need. So I wouldn't even know like if I need Apache, I wouldn't even know all the libraries that it uses. So well, I mean, there's the one aspect of you need to know what libraries you care about, right? But then there's the other aspect of where's this fine line of how much stuff to include. I think we're still struggling with that too. I think, you know, I don't think there is, well, I don't think there will be a sort of bullet kind of answer. What we are trying to do is do a better job documenting how to think about it. And there are starting to be a lot of examples, right? So now we have whatever, you know, 10 or 12 that we're shipping. You know, there's another problem. There's at least another five or 10 that I know about and kind of in the software world. So there's starting to be examples that are making decisions. I think we're also going to run into problems. You know, it's really nice actually that REL7 is kind of on its way out the door, because that'll be the first test of some of this stuff. So, you know, I'm running the beta here. And so what I'm, but it's only been like a couple weeks. So what I want to do is start to try to run, take an application that I wrote on the software collection for Ruby on REL6, run that hopefully unchanged on REL7. And then I actually want to be able to run the same thing unchanged on OpenShift. Because OpenShift actually uses software collections to, that's how it delivers those blobs as well. Do I see a question over here? Yeah. No, no, he's first, then you, because that way he can keep walking in the right direction. Thank you. So I'm no advanced coder. I'm not a coder at all, but I'm aware of distributions and dependencies and versions and stuff. And I just want to understand correctly what this means. So you say I can make a package of a software and include my own dependencies, what I need, and do it all into the opt system. And all I need to do to make this work and any distribution is to insert this SCL trick. Is that about correctly? So it's a little more complicated than that. So the first part is you have to create the thing, right? So you have to have your application and all its dependencies. And then you have to shove that all into some sort of package. In the case that I'm talking about, that's all RPM. Then on deployment, so on install of the application is where the opt stuff comes in. So it's where it's going to show up. And then this SCL stuff works because basically now you have all of the pieces that you need. Some of it might be coming from the operating system itself. Some of it might be coming from this set of packages called the software collection to actually deliver the API for, say, Ruby to another application. So we expect, and I know of a couple of companies that are doing this, but we expect that some companies will kind of, whether it's open source or proprietary, they're going to actually create their own application as a software collection, which will then have a dependent collection that they also carry or set of dependent collections. So that then the whole thing will kind of get installed there. But then you'll have that, so when you install X, Y, and Z software, you actually get that whole collection from them and including any dependent collections that they may get from elsewhere. So every package brings its own environment completely? Right. So it may rely on some stuff that's in the OS too, or it may not. It's up to the package or the collection. Okay. My question is, are you aware of Nix and Guix package managers? Because they are doing a similar job, but I believe slightly better and without RPM. So as I said kind of on that earlier slides, like everybody has had this problem a ton all the time, right? Microsoft has a solution for this problem as well. This, I don't know if you guys know anything about.NET, but one of the major changes you may have remembered the advertising was.NET solve DLL hell, right? And the way it did that was having this thing called the global cache and then a per application cache. And what ended up happening, they had this magical mystical idea that in the global cache would be things like G-Lib C. What in fact happened is that every single application that got installed picked up their own, right? Mac has always, and to the best of my knowledge, and I don't know a whole lot about Mac packaging, but Mac has pretty much always worked this way in that every application basically carries its entire dependency tree every time you install it. So like it's not limited to Linux, number one, and it's certainly not limited to the examples I gave. I don't actually know the one you mentioned, so I'll go ask Google. Okay. And another thing, I didn't understand that what are the new demands for software vendors to be able to use this? Well, so it's really just a style. It's a way of saying, hey, this is a way you could deliver your application that, because this happens to the RPM, right? But the same concept could work in anything. It doesn't really matter that much. You just need kind of something like YUM to manage where they're coming from and getting updates. But I think a software vendor of some kind could say, hey, here's a way to distribute it, and it might be easier for you to build the packages for, let's say, Enterprise Linux 4, 5, and 6, they might be able to find a way to create a software collection that will install on all of them, right? So that they don't have to recertify for every minor version, right? That they don't have to, you know, they can also deploy in Fedora, even though Fedora is way ahead of RAL, right? Does that make sense? Arguably? So I've got a question on, I suppose this goes back to how big the software collections are, but say you have a software collection for different versions of Rails and a software collection for different versions of Ruby, is this just to suggest that for every permutation you want, you would then need to build another independent software collection on top of that? Theoretically, right. So theoretically, and that's the struggle, right, is where do you want to draw that line? So right now, for example, the one we built, right, this Ruby 193 actually includes Rails 3 as well. So even though it's called Ruby 193, there's actually, I don't know, 40, 30, some number of gems in there, I can't remember how many. But a whole bunch, right, which if you, if I read Ruby 193, I don't hear gems, right? So I'm not sure that we're entirely happy with that choice, that maybe Rails should have been separate, because then it would have been easier for us to say, okay, now we have a Rails 3 software collection and a Rails 4 software collection, both dependent on the Ruby 193 software collection. Maybe I missed it when I was just looking at my phone during your talk or whatever, sorry about that. Thanks, Don. I'm curious about some of the details. How do you, how do people actually create these collections? How do they deliver them to users? How do users install them? How do they manage them? What do the tools look like in practice? So I was going to include some of the macros, but I decided not to. But basically, when you're writing an RPM, right, you have this big thing called the spec file, which kind of describes what's in the RPM, right? In that spec file, we just add some new macros that indicate that this is a software collection and that it's going to go kind of in a special place. And that's about it as far as the difference. One thing that was kind of a design goal was that the same spec file, if it knew about software collections, could also be built to not be a software collection. So hopefully what will happen is that all spec files everywhere will have kind of the support for a software collection so you could build it that way or decide to not build it that way, depending on your use case. And then as far as the tools are concerned, it's just yum install, you know, X, Y, and Z, you know, software collection dot RPM, you know, whatever happens to be called. Though, you know, in Enterprise Linux, you have to add what's called a channel, which will give me started on that. But basically, you have to add another repo coming from Red Hat and it's got a bunch of them in there. You just say yum install, blah, blah, blah. I think CentOS, I think they're main lines. Did they make it to main line or are they still in test? They are. Okay. So in CentOS, you can actually just log in yum install, Ruby 193. So it's all the tools you're already familiar with for the most part. You know, they'll work just like the, you know, RPM does with puppet. So it has also all the failings of RPM, but, you know, it's easy. Hi. Do you have an example yet of an application that's packaged in the normal places, like where a user would expect them, but it depends on an SCL. So for example, you know, user would run the application where it normally is, doesn't, has never seen the SCL enable utility, but the application then enables the SCL it requires itself. Sort of, you know, basically, so I've done that with like ACPD, right? So, you know, to get the web server to use the Ruby 193 stuff, I need to kind of modify its start script. And I was, for that to go example, I was using the native Apache web server rather than a software collection. I think we want to be mildly careful of that because that's one of my fears is that if you have something that is distributed by the normal OS that's dependent on a software collection, that might be weird. You know, it kind of goes out of the user's, or when I say user here, right? The system administrator or developers' expectation of where stuff is going to show up, you know, where you go find stuff. So my assumption is it's distributed separately in a separate Jom repo, even maybe with an SCL, but when a user installs it, the application appears in its normal places, but it's using an SCL behind the scenes. Exactly. So that's exactly what I did for, like, ACPD. You know, if you want to set up a bash profile that it just runs, right? So, however, in general, if I had a, like, kind of a real top-level application, you know, so, you know, my cool website that runs in your office, I would rather that people package that as a software collection. And then it's dependent on software collection. I think user expectation. So, you know, I think having a shell script or something that's kind of off in a more normal place that points at it is fine, but I don't know, I guess I don't like if an application installs under opt, then it shouldn't touch the rest of the OS, right? If I choose to make it touch the rest of the OS, so, an example, a Calibre installs, if you install it from the Python script, installs under opt, it doesn't touch the normal operating system basically at all, but I can put a sim link to it from user bin. Right? Sorry, I just, one technical add, there's no reason you can't. It's just, that's my preference. Well, I don't know if you have already answered my question, but it's another twist on the situation. Let's say we have a Ruby collection or a Python collection, and I want to add a particular gem or Python module. I know how to create a packet for a native environment. Can it be easily added to the existing collection? Do I have to create a new collection and overwrite or remove the previous one? Right, so that's actually exactly the problem in some ways, we didn't really talk about it here, but the other thing that a dependent collection could or should be able to do, right, is we're talking about, you know, we have the Ruby collection, which this one happens to include Rails, right? So let's say I want to write a dependent collection that swaps Ruby, I'm sorry, swaps Rails 3 and puts in Rails 4. Well, we don't want you to have to rebuild this one, right? So one of the things that we're working on from a, I think it's more of a documentation problem, hopefully it's not a bug problem, but there's a documentation problem to say, okay, how do you write a dependent collection that swaps part of a collection, right? And does that make sense or is that completely insane? I've got a question, does the software collecting packages run independently from each other, means if I run two at the same time, which have got a dependency inside which breaks if they run at the same time? Do they run independent of each other? So I was kind of saying that earlier. You can kind of, so I had an example where I was doing, I think it was Postgres. So we have a Postgres software collection, so I had an app that used Postgres, whatever, 9, something that we're shipping, and Ruby 1.9. So what I do here is I kind of say, SEL enabled Ruby 1.9, space Postgres bash, okay? So that gives them both there. If there's a dependency between the two, then I think you should have a software collection that is dependent on the two, and then you use that software collection if you follow me. So in the Rails, Ruby and Rails example, I wouldn't kind of run them both. I would have built the Rails one such that it's dependent, and then I would SEL enable Rails 4 bash. Yes, and it's just that one depends on the other, rather than using both. The only time I would use both is because for my application, I need two, three, four different pieces of software, Postgres, Memcache D, and Ruby. Okay. We have time for this one and one more. So the next question goes to whoever's going to go get me a beer after the question. In the Retard developer blog, there was this topic one year ago, and compared to this, what's the news? Not a lot. Like I said, actually what I do think we've found, like kind of over the course of, I mean, software collections have been now around what, almost three years. So what I think we're finding, particularly since we started shipping some stuff, whatever year and change ago, is the problems people run into. So like this dependent collection problem. And so getting real world use starts to poke some of the holes in it. I think the other big stuff that's changing is, like I said, containerization is already hugely important and becoming way more important. And how that integrates with packaging, I think is also going to be a really interesting ride. Is there any project page or something where the community can... Coming soon. Yeah. Let's just say, I would really, really like there to be kind of a proper upstream for kind of software collections as a concept. It is all open source, but it's kind of like all over the place. So while it's open source, there's no practical means of finding it all, right? Unless you work with the CentOS guys and have this magic ability to find things. But so one of the things that we have in the works is to try to get kind of a landing place for software collections as a whole. And we've heard from people about the software collections that they've built that they would like other people to be able to use. And so kind of allow for a community to form around it. Everybody's got lots of things to do. Any other questions? Apparently no one wants to give me a beer. All right. Thanks, Langdon. You're welcome. Thank you. Thank you.
 smarter than me. There we go. I wonder why I didn't echo. So as I said, I was a camp  counselor for a long time, so you know, big parade fields, etc. And what I do is I work  for Red Hat. I'm what, you know, this kind of new thing called a developer advocate or  developer evangelist. And so I've spent, I don't know, about 15 years as a, mostly as  a software consultant. So a lot of different projects, mostly kind of big enterprise sort  of things. And now what I do is I talk to kind of developers who use kind of Red Hat  products, in particular Enterprise Linux, talk to them about what they hate about it,  and then I go and talk to engineering about how we can fix that stuff. So that's mostly  what I do. And that's a little bit about me. So this talk is actually, it's kind of essentially  about this concept called software collections. But it also, I come definitely from the developer  background rather than coming from like a packages background. So I think I have a lot  of biases towards being a developer and, you know, not needing things like QA or, you know,  any of that, any release stuff, you know, and it will just send it out there and everything  will be fine. No security updates, et cetera. But so just a little bit of, you know, on  me. So here's a little bit of my argument. Over the last, I don't know, whatever, 15  years, Linux distributions, you know, they kind of started in this kind of world where,  you know, the open source community was relatively small. The even proprietary software and open  source software, neither one could you trust basically at all. They, you know, it was all  run by developers who didn't really know or care what they were doing. So things like  security updates would never happen. QA, if you were lucky, you know, et cetera. So early  on, I argue that the distributions kind of formed to try to basically solve some of that  ease of use problem and kind of redistribution problem around security patches and software  and kind of things like that. So that you kind of had, you knew where you could get  stuff from and you could also redistribute things so that they would stay secure. It  was also kind of over the, arguably earlier than that, but, you know, over the same 15  years or so is where, you know, kind of the dynamic linking concept really took off and  caught on. You know, there were definitely pockets before that, but it wasn't until,  I don't know, like 96, 97 where it was like, it started to become like the way to do things.  So, and then the last thing was actually license assurance. This is, was much less of a problem  with proprietary software, obviously, although in fact they were hugely violating licenses  anyway, but you couldn't tell. But with open source, it was a big concern, right? So you  wanted to make sure, particularly as open source started to be adopted by the enterprise,  that you knew what kind of licenses you were getting into. Also, feel free to interrupt  me. So you can tell me I'm crazy. If it's just blanket crazy though, well, let's wait  till the end. So particularly now in the last maybe five years, there's been, and I couldn't  find a good graph for this. I've actually seen one before. I was looking at, I was talking  about somebody earlier that, you know, kind of just the package count in Fedora, each  version. It's just scaling like crazy. Obviously in something like enterprise Linux, we have  significantly less packages, but even that is still, it's still a nice, you know, strong  curve. And so between that, now we also have a massive plethora of actual Linux distributions  that are reasonably popular. So it's not, you know, three big ones, right? And then everybody  else, right? There's, you know, there's a good chunk at the top, right? And then Linux  in general is also becoming hugely adopted, particularly in the enterprise, you know,  kind of in server rooms in particular. But then even, even to some extent in the enterprise  on desktops, you know, I don't think it's ever, you know, I don't think we're ever  going to see the year of the desktop, but, you know, it is in some places, you know,  I think it's largely going to be the only people who use kind of desktops and by desktop,  I mean laptop half the time, are going to be people like you guys, right? It's going  to be just developers, you know, just people who actually make software who use these things  at all, right? It's everything else is going to be tablets and, you know, in that kind  of tool chain. So we really need to kind of focus on making sure that we can distribute  things well. So here's my, here's the extent of the math I can do anymore. You know, so  we have a whole lot of projects, and then we have versions of each project, then we  have it across distributions, and then we have versions of the distributions, right?  So now we just made the thing massively wide, right? So kind of typically, you know, we,  we just do this kind of, you know, take a tar ball, right? And then kind of have to  figure out the distros, packaging rules, and then kind of figure out how to build the thing  itself, and then start to, you know, kind of put that into a package. We got to follow  all these rules and make sure it goes in the right places, etc. And then on top of that,  inside an individual organization, there may be kind of a second set or third or fifth  or 47th set of packaging rules within that organization. So you have, you kind of have  to solve the problem over and over again. So what would be a lot nicer is if we could  kind of say, okay, I have this package, right? And if you saw, I don't know how much of Donnie's  talk covered this earlier, I know he's covered the past, but, you know, whatever that means,  right? So a package. But essentially, if we had a blob of thing that we could run, say,  on any distro, right, that was OS independent version wise as well, so you can kind of go  this way, right, for the package could drop on each one, or you could go this way, right?  So, you know, version two, version three, version four of the OS. And then the other  thing is, can you also are there, can you start to trust essentially the developers to  have packaging rules that make sense, you know, for their application, rather than it  being a distribution problem? So lots and lots and lots of people have tried to solve  this problem. Here are whatever six, five that I thought of off the top of my head,  and ones that I could find random, easy links for. Actually, I had Linux apps in there too,  but I couldn't find a good, like, place to point people off the top of my head. So they,  some of them solve the exact same problem, some of them kind of take it from a slightly  different perspective. So that's what I think is kind of interesting is to, it's kind of,  you know, every piece of software anybody writes has trade-offs. And these people clearly  were taking different trade-offs, right? So kind of the environment modules, SEP, and  to some extent Gen 2 slots kind of all take roughly the same set of trade-offs, very similar  actually to software collections, where it's kind of like, I want to have multiple versions  of the same thing installed on the OS at a time, and I want to rely on the user to choose  between them when they're running applications. OS Tree, relatively new thing that is really  actually about, it's really about supporting, arguably supporting debugging. So you can  take an entire set of binaries and kind of swap them out in and out, and it's for, you  know, doing like debugging of GNOME, right? So it's not just one application. Yeah.  We're going to look at doing that for off-rope data.  Using OS Tree? Yeah. So mind you, GNOME I think has three or four different ones that  all try to accomplish this goal. There's Glick 2, Glick, it's another one too. And  then you have update alternatives, which is also trying to solve the same problem, but  in a very different way, which is I want to be able to switch for the entire OS what the  version of something is. So instead of per application, which is what these guys do,  it's actually for the whole OS, right? And that can kind of cause some problems, particularly  something like Python, right? So you switch the version of Python and your OS on Linux,  right? And bad things are going to happen. And like I said, there's lots and lots more.  So taking some of that thought, right? And taking some of the... So Red Hat has a little  bit of a problem where, particularly for Enterprise Linux, a lot of our customers want to use  whatever version of Enterprise Linux that they're using forever. And I mean forever,  right? Because, well, one of the reasons is because the application that they're running  on it is working just fine, right? So why would they want to invest normally? What you have  to do is you have to invest development in effort, QA effort, release engineering effort  into modifying the application to be able to run on some future version of the operating  system. If it's a junk application that just does what it does, why would you make that  investment? So as a result, they instead yell at companies like ours, right? And lots of  other ones. Microsoft has this problem a huge way as well, as you all have, I'm sure, seen  with their big EOL announcements every time they announce an EOL. And then it'll... It's  like five times they EOL it, and then it'll actually die. So they have this huge problem  as well. They do some other interesting things. Look up the civilization fix, the video game  civilization, the fix they did in Windows sometime.  So what a software collection tries to do is let's instead of having the application  kind of natively touch the OS itself, let's put a layer in between it, put a layer of  indirection in there so that we can now take this blob that's in between. It might be Ruby,  right? Or it might be kind of a full stack. So it might be a web server plus Rails and  Ruby and everything else. The idea is there is a blob that you depend on that's kind of  above generally, above the actual OS and below your application. So that's kind of what a  software collection is trying to do is saying, okay, let's identify that blob and let's try  to make that blob portable. So now what you can do is you can say, okay, I have this janky  old Ruby app, let's say Puppet, that is running on Ruby 18, right? Why is it running on 18?  The way I hear it is because Enterprise Linux 5 runs Ruby 18 and why is Enterprise Linux  5 running 18? Because Puppet runs so. But any which way, the point is, you know, you  take any old application or older application and what you can do is you can kind of say,  here's this blob, I'm going to guarantee the API on the top of it and then I'll change  how it delivers that API underneath so that it can be portable across multiple versions  of the OS. You also have the problem the other way, right? So there's a lot of push for people  to move to Python 3. However, not all the distributions are ready to make all the changes  to be able to move to Python 3. So how do you start writing applications in Python 3  right now without the, you know, kind of working your Linux? Well, one of the ways you can  do is with a software collection. So it's kind of, you know, by the two examples I'm  kind of giving, right? You can kind of see it's OS independent, right? The OS can continue  on its merry way and your application can decide to upgrade when it's ready. Or you  can be aggressive and say, hey, I'm writing something bleeding edge brand new and I know  I want to use the most cutting edge version of whatever because I'm not going to change  it again for 10 years. So let me get that right now and use it right now to write my  application. So it kind of going into it a little bit more. So it really is, as it says,  it's kind of defining a process a little bit more than a product, you know, so it's kind  of a conceptual thing. You know, there is an implementation, but that's kind of what  it does. The other big thing that it does is it packages outside of the normal kind  of Linux file system. And that's sometimes a problem for some of the people I think in  the room. And it allows multiple versions to be co-installed, right? And then per application  you can essentially start, or you can say, this application gets this blob rather than  that blob. So the way we kind of defined it is it installs under opt with, you know, the  L acronym that is the Linux naming that I can't actually say. You go register for a  Linux name so that you can get kind of a short code there and then you put your application  name there. And that's kind of where it installs. And then you have kind of some scriptlets  that know where that is and can provide that information to the application. And the way  it basically does it is by fancy-path-ing, right? So the application doesn't have to  know, this is the other, sorry I didn't mention this part. One of the big differences between  the ones that I was talking about earlier and this one is that the application doesn't  have to know that it's not the base OS version, right? Which is, I think, hugely important  because you don't want to have to rely on your average developer to figure out where  it's going to be. And again, to allow for that OS independence, you know, someday, right,  all the Linux distributions will be on Python 3 and that will be what's at user bin Python.  So what you want to do is be able to write the application into, say, user bin Python  at the top and then it just goes to the right place rather than having to, you know, do fancy  sim links or, you know, or make a mistake and say, or have user bin Python 3 and then  find out now you have to do engineering effort to port it.  So in a little more detail, here is kind of the way the one for Ruby works that we have  built. So here's kind of regular Ruby, right? You know, assuming that you installed Ruby,  the other nice thing about this is that you don't actually have to have, I can't remember  if Ruby is installed by default, but you don't have to have Ruby installed except for this  particular application. So in other words, it doesn't have to be anything in user bin  Ruby. It can just be down here. So I think it's pretty obvious, but basically here's  the normal path, right? And here is our special path. You know, we ship a Ruby 193 variant,  you know, one of these blobs. It sits under our age because that's our name. And that's  pretty much all there is to it. The disk little script here is what does all of the magic  such as it is, which it basically kind of provides those paths so that the command to  run these, which I think is down here. So SEL enabled Ruby 193 bash. So what you do  is you kind of say, here's the command, right, SEL? I want to turn it on. All right. And  then which one do I want Ruby 193? And then what do I want to execute with it? So if you  just want to get a new shell back where Ruby 193 is available, you just run bash. But it  could just as easily be, you know, some app, right? So from a kind of, if you're a sysadmin,  it would be like an app that, and then you put this in, you know, a shell script or a  start script or something script, you know, for a developer, he's going to drop in a bash  most of the time. You can add this to your.profile. And so it's just always there if  you want as well. Then we talk about, you know, there's basically some tools out there  that kind of help make this stuff work. And so there's some links. I think all these slides  are available somewhere. Joe would know better than me. I will email them to Joe. He will  make them go. This is the ones that we've done already. Actually, it's some of the  ones we've done already. We actually have some that are out there for like hdbd 2.4.  I can't remember what else is up there. That's just kind of public. There are some other  distributions that are related to ours that also have all of these. There's a guy, I think,  who's talking next to who had a hand in those. So I think they are well represented now actually  in CentOS, Scientific Linux, and Fedora. So Fedora just approved them whatever it was  last week or was it the week before? Okay. I can always pray. All right. So it's getting  there. There is some of this stuff built for Fedora, but you can't get it in kind of the  mainline distro repos yet. So that's basically most of my slides. I was expecting more interruptions.  I invited hecklers and they're not delivering. Exactly. That's true. That's true. So here's  kind of my argument. I'm probably crazy, but I think vendors in general are a lot more trustworthy.  When I say vendor, proprietary software, open source projects, everything, any kind of application,  I think the distribution doesn't have to take on as much of a responsibility for making  sure that all the stuff in the world runs. Yeah. Yeah. Yeah. Which is not a high bar,  but let's say are reasonably trustworthy. So they're even more than that. I think a  lot of this stuff, most people understand, most people who do this kind of work, most  of them they understand what's important, how to break their things up. Yeah, they're  still false, but I think a lot of them are starting to get it. There's well-known libraries  for particularly security-related things. So people mostly use those libraries. I think  Bruce Schneier is finally getting through to people and saying, stop writing your own  encryption libraries. So same kind of thing. I think in general they're considerably more  trustworthy so that I don't necessarily think that every distribution needs to package  everything under the sun. I also think another big reason is redistribution. One of the  things that software collections has that kind of fails and goes backwards in time a  little bit is a lot of things are considered to be statically linked because the only way  you can be portable across OS is if you carry the bits you need. So that means static linking,  sort of, or at least it means that it's got to be carried in the bundle. So it can still  be dynamically linked, however it's got to be dynamically linked from within the software  collection. So one of the things that I was certainly confused about when I first looked  at this stuff, so if you think about an RPM, one software collection does not equal one  RPM. A software collection is more like 70, 1,000 RPMs. So it's broken up just like you  would do a normal package. It's just that it's all kind of in this one context to say,  hey, I'm part of this one software collection. So you do, it is better, quote, unquote, than  straight out statically linking, but you still have a problem of redistribution. So one of  the reasons that dynamic libraries are nice is that you only have to copy that one file  onto every machine. With a software collection, let's say it's open SSL and you have 37 software  collections on every machine all using open SSL. That means you have at least 37 different  copies of that SL RPM. So you need to distribute that all the way through. However, and judging  by the overflowingness of the config management talk, config management has really finally  taken. I remember the fits and starts. I mean, there was a huge push for it, I think, around  2001 and just never quite took, but it's really kind of taken now. So I think redistribution,  particularly in big data centers, has gotten a lot easier. So I think that's why we can  kind of take this risk. The other thing is the other risk you have is that now you have  a copy, you have 37 copies of open SSL. Disface is not that big a deal anymore. So does it  really matter that you have lots of different copies of the same stuff on one machine? And  deduping file systems someday, I pray, will be default and so it won't matter anyway.  The other thing is that I think users are more sophisticated. So I think that they can,  they understand the context that they're, and when I say user here, I mean I develop  them, right? So somebody who's going to consume one of these software questions, they're way  more sophisticated about what is going on underneath and can usually figure this stuff  out. Yeah.  I was going to say that the nature of the software collection does not mandate that  you have 37 copies of open SSL on the system. It may be that you require three specific  compatible versions. And given that it's built on.cl, you can build a, an SCL top and other  SCLs. So you can narrow that down manually as well.  Yeah. So I think dependent collections, right, are a very real and very important aspect  of this that we're still working through some of the, like there's a, I don't know where  it's published right now, but we've been working on trying to get a good guideline for how  to do dependent collections, right? So do you really want to do a rail software collection?  Well, not really, right? You want to do a Ruby software collection and then have a rail  software collection that depends on it, right? So, and what, what's the right size of a software  collection, right? What is, you know, should it be a full on, you know, kind of web server  with, you know, like say Ruby and Rails, or should it be, you know, piecemeal? And I think  a lot of those questions I think are yet to be known.  The other thing I think that is going to be interesting about this is kind of the, the  move towards containerization, right? So containerization, and I'm using that term to mean, to be as far  away from possible from any individual implementation, right? So you have things like Docker, you  have LXC, you have, you know, this combination of open, we use an open shift which is like  C groups and SC Linux. You have, you have actually much older and much more sophisticated  versions in other operating systems that are not Linux. But the kind of this containerization  concept where now we kind of say, hey, this whole application is now a blob and it's portable  and we can run it into certain kinds of conditions and it can be updated on its own. I think  that's going to be the next step for some of this stuff. And I think a lot of that will  rely on something either exactly like software collections or something very similar because  it, it allows you to kind of bring that bundle inside, you know, your container.  Are you ready for questions?  Yeah.  All right. I've got a mic. So if you have questions, raise your hand, we'll bring the  mic around so it turns up on the video later.  One thing I didn't follow. If you've got two software collections installed for two different  versions of something, so Ruby, what's the mechanism by which the application gets to  use the appropriate version?  So this versus like update alternatives. So the question was basically like, how do you  say I want this application to use Ruby 1.9 and that application to use, you know, Ruby  2, right? So this, of course, is only at the very bottom of the screen here. So you probably  can't see it. I should have probably put it at the top somewhere too. But this unlike,  say, like update alternatives, you, you tell the application which one it's using later.  You know, the application doesn't actually know, but you say SEL enable whatever collection  you want to use, and then the application you want to run so that it's basically running  it inside a shell of its own that has the kind of correct paths. So it finds it as if  it was at its normal locations.  Right. So this is changing the environment.  Essentially. Yeah.  Try to organize your questions by row.  I don't even know why I got up at 6.30 this morning to go to the fitness center at the  hotel.  I'd like to ask how security updates are propagated to containerized applications in software  collections. So say I have a Ruby software collection and Ruby depends on some library  that got the security update upstream. Does it mean that the maintainer of the collection  will have to update it manually? And if I have like five collections which depend on  this library, does it mean that five different people will have to update their collections  to include the update?  Exactly. Yeah. That's the problem.  Not if you do dependencies. So like I said, if you use kind of, let's say you had a Rails  3 and a Rails 4 both dependent on the Ruby 1.9 software collection and the security update  happened in the Ruby 1, you wouldn't have to update the Rails 1s in theory. I mean one  of the things that I like about software collections is it kind of also lets you, well, like dislike  I don't know, but it also lets you defend kind of your QA problem, right? So one of  the reasons that the enterprise doesn't want to upgrade, often it may be that they expect  the code to completely work, but they don't want to test it, right? QA resources are often  the ones that you have the least of in any organization. It's often the constraint at  any engineering organization I've ever worked with that keeps features from happening. And  so when you're doing, again, kind of talking about that janky old app that is just running  and doing its thing, why would you want to waste that very limited resource of your QA  people to test the upgrade?  So I also would hope that the, you know, if, I guess I kind of expect that in a particular  enterprise, much like I would kind of expect with OpenShift is like in a particular enterprise,  I would kind of expect them to build their own software collections for use in house,  probably depend, hopefully, right? Dependent on, you know, for a large percentage, 90,  95% of the functionality on somebody else, right? So like on Red Hat or on the community  in general or on something, but that they would have their own one so that they could  be confident that the API layer at the top is what they expect completely for the applications  that they're building. You know, it does two things. One, it gives them more, you know,  kind of tolerance for moving across versions, but then it also does that kind of opposite  effect, which then you can still control your developers by saying, here is the approved  Ruby software collection, right? Here is the approved Python software collection. Don't  go installing random other stuff unless it's made it into the software collection.  One back there.  Yes. I have a question on how this SCL enable works. So can I use multiple of these environments  at the same time? And so it would sort of only map this for this one process with something  like P-Root or Mountbind or how does it get into the environment?  Yeah. So you separate it by a space, right? But you can just kind of list out multiple  ones that you want. And it's not that cool. It's just paths, right? So this is not, you  know, teruding some brand new file system or something and running in that context. There's  no security here, right? None of that stuff. This is just to provide the environment that  is kind of so when you make a function call, you get the right binary. That's it.  Yeah. So what is like the lowest level of libraries that you include? For example, these  packages, would they include Lipsy or do you still expect that from the distribution itself?  The package maintainer needs to decide that. And everything that you don't include makes  you less portable, right? But makes it so that you have less of the problem that he  was talking about back there of massive redistribution updates.  But I think it's hard to know like all the pieces that you need. So I wouldn't even know  like if I need Apache, I wouldn't even know all the libraries that it uses. So well, I  mean, there's the one aspect of you need to know what libraries you care about, right?  But then there's the other aspect of where's this fine line of how much stuff to include.  I think we're still struggling with that too. I think, you know, I don't think there is,  well, I don't think there will be a sort of bullet kind of answer. What we are trying  to do is do a better job documenting how to think about it. And there are starting to  be a lot of examples, right? So now we have whatever, you know, 10 or 12 that we're shipping.  You know, there's another problem. There's at least another five or 10 that I know about  and kind of in the software world. So there's starting to be examples that are making decisions.  I think we're also going to run into problems. You know, it's really nice actually that  REL7 is kind of on its way out the door, because that'll be the first test of some of this  stuff. So, you know, I'm running the beta here. And so what I'm, but it's only been  like a couple weeks. So what I want to do is start to try to run, take an application  that I wrote on the software collection for Ruby on REL6, run that hopefully unchanged  on REL7. And then I actually want to be able to run the same thing unchanged on OpenShift.  Because OpenShift actually uses software collections to, that's how it delivers those blobs as well.  Do I see a question over here? Yeah. No, no, he's first, then you, because that way he  can keep walking in the right direction.  Thank you. So I'm no advanced coder. I'm not a coder at all, but I'm aware of distributions  and dependencies and versions and stuff. And I just want to understand correctly what this  means. So you say I can make a package of a software and include my own dependencies,  what I need, and do it all into the opt system. And all I need to do to make this work and  any distribution is to insert this SCL trick. Is that about correctly?  So it's a little more complicated than that. So the first part is you have to create the  thing, right? So you have to have your application and all its dependencies. And then you have  to shove that all into some sort of package. In the case that I'm talking about, that's  all RPM. Then on deployment, so on install of the application is where the opt stuff  comes in. So it's where it's going to show up. And then this SCL stuff works because  basically now you have all of the pieces that you need. Some of it might be coming from  the operating system itself. Some of it might be coming from this set of packages called  the software collection to actually deliver the API for, say, Ruby to another application.  So we expect, and I know of a couple of companies that are doing this, but we expect that some  companies will kind of, whether it's open source or proprietary, they're going to actually  create their own application as a software collection, which will then have a dependent  collection that they also carry or set of dependent collections. So that then the whole  thing will kind of get installed there. But then you'll have that, so when you install  X, Y, and Z software, you actually get that whole collection from them and including any  dependent collections that they may get from elsewhere.  So every package brings its own environment completely?  Right. So it may rely on some stuff that's in the OS too, or it may not. It's up to the  package or the collection.  Okay. My question is, are you aware of Nix and Guix package managers? Because they are  doing a similar job, but I believe slightly better and without RPM.  So as I said kind of on that earlier slides, like everybody has had this problem a ton  all the time, right? Microsoft has a solution for this problem as well. This, I don't know  if you guys know anything about.NET, but one of the major changes you may have remembered  the advertising was.NET solve DLL hell, right? And the way it did that was having this thing  called the global cache and then a per application cache. And what ended up happening, they had  this magical mystical idea that in the global cache would be things like G-Lib C. What in  fact happened is that every single application that got installed picked up their own, right?  Mac has always, and to the best of my knowledge, and I don't know a whole lot about Mac packaging,  but Mac has pretty much always worked this way in that every application basically carries  its entire dependency tree every time you install it. So like it's not limited to Linux,  number one, and it's certainly not limited to the examples I gave. I don't actually know  the one you mentioned, so I'll go ask Google.  Okay. And another thing, I didn't understand that what are the new demands for software  vendors to be able to use this?  Well, so it's really just a style. It's a way of saying, hey, this is a way you could  deliver your application that, because this happens to the RPM, right? But the same concept  could work in anything. It doesn't really matter that much. You just need kind of something  like YUM to manage where they're coming from and getting updates. But I think a software  vendor of some kind could say, hey, here's a way to distribute it, and it might be easier  for you to build the packages for, let's say, Enterprise Linux 4, 5, and 6, they might be  able to find a way to create a software collection that will install on all of them, right? So  that they don't have to recertify for every minor version, right? That they don't have  to, you know, they can also deploy in Fedora, even though Fedora is way ahead of RAL, right?  Does that make sense? Arguably?  So I've got a question on, I suppose this goes back to how big the software collections  are, but say you have a software collection for different versions of Rails and a software  collection for different versions of Ruby, is this just to suggest that for every permutation  you want, you would then need to build another independent software collection on top of  that?  Theoretically, right. So theoretically, and that's the struggle, right, is where do  you want to draw that line? So right now, for example, the one we built, right, this  Ruby 193 actually includes Rails 3 as well. So even though it's called Ruby 193, there's  actually, I don't know, 40, 30, some number of gems in there, I can't remember how many.  But a whole bunch, right, which if you, if I read Ruby 193, I don't hear gems, right?  So I'm not sure that we're entirely happy with that choice, that maybe Rails should  have been separate, because then it would have been easier for us to say, okay, now  we have a Rails 3 software collection and a Rails 4 software collection, both dependent  on the Ruby 193 software collection.  Maybe I missed it when I was just looking at my phone during your talk or whatever, sorry  about that.  Thanks, Don.  I'm curious about some of the details. How do you, how do people actually create these  collections? How do they deliver them to users? How do users install them? How do they  manage them? What do the tools look like in practice?  So I was going to include some of the macros, but I decided not to. But basically, when  you're writing an RPM, right, you have this big thing called the spec file, which kind  of describes what's in the RPM, right?  In that spec file, we just add some new macros that indicate that this is a software collection  and that it's going to go kind of in a special place. And that's about it as far as the difference.  One thing that was kind of a design goal was that the same spec file, if it knew about  software collections, could also be built to not be a software collection.  So hopefully what will happen is that all spec files everywhere will have kind of the  support for a software collection so you could build it that way or decide to not build it  that way, depending on your use case.  And then as far as the tools are concerned, it's just yum install, you know, X, Y, and  Z, you know, software collection dot RPM, you know, whatever happens to be called.  Though, you know, in Enterprise Linux, you have to add what's called a channel, which  will give me started on that. But basically, you have to add another repo coming from Red  Hat and it's got a bunch of them in there. You just say yum install, blah, blah, blah.  I think CentOS, I think they're main lines. Did they make it to main line or are they  still in test?  They are. Okay. So in CentOS, you can actually just log in yum install, Ruby 193.  So it's all the tools you're already familiar with for the most part. You know, they'll  work just like the, you know, RPM does with puppet. So it has also all the failings of  RPM, but, you know, it's easy.  Hi. Do you have an example yet of an application that's packaged in the normal places, like  where a user would expect them, but it depends on an SCL. So for example, you know, user  would run the application where it normally is, doesn't, has never seen the SCL enable  utility, but the application then enables the SCL it requires itself.  Sort of, you know, basically, so I've done that with like ACPD, right? So, you know,  to get the web server to use the Ruby 193 stuff, I need to kind of modify its start  script. And I was, for that to go example, I was using the native Apache web server  rather than a software collection. I think we want to be mildly careful of that because  that's one of my fears is that if you have something that is distributed by the normal  OS that's dependent on a software collection, that might be weird. You know, it kind of  goes out of the user's, or when I say user here, right? The system administrator or developers'  expectation of where stuff is going to show up, you know, where you go find stuff.  So my assumption is it's distributed separately in a separate Jom repo, even maybe with an  SCL, but when a user installs it, the application appears in its normal places, but it's using  an SCL behind the scenes.  Exactly. So that's exactly what I did for, like, ACPD. You know, if you want to set up  a bash profile that it just runs, right? So, however, in general, if I had a, like, kind  of a real top-level application, you know, so, you know, my cool website that runs in  your office, I would rather that people package that as a software collection. And then it's  dependent on software collection. I think user expectation. So, you know, I think having  a shell script or something that's kind of off in a more normal place that points at  it is fine, but I don't know, I guess I don't like if an application installs under opt,  then it shouldn't touch the rest of the OS, right? If I choose to make it touch the rest  of the OS, so, an example, a Calibre installs, if you install it from the Python script,  installs under opt, it doesn't touch the normal operating system basically at all, but I can  put a sim link to it from user bin. Right?  Sorry, I just, one technical add, there's no reason you can't. It's just, that's my  preference. Well, I don't know if you have already answered my question, but it's another  twist on the situation. Let's say we have a Ruby collection or a Python collection, and  I want to add a particular gem or Python module. I know how to create a packet for a native  environment. Can it be easily added to the existing collection? Do I have to create a  new collection and overwrite or remove the previous one?  Right, so that's actually exactly the problem in some ways, we didn't really talk about  it here, but the other thing that a dependent collection could or should be able to do,  right, is we're talking about, you know, we have the Ruby collection, which this one happens  to include Rails, right? So let's say I want to write a dependent collection that swaps  Ruby, I'm sorry, swaps Rails 3 and puts in Rails 4. Well, we don't want you to have to  rebuild this one, right? So one of the things that we're working on from a, I think it's  more of a documentation problem, hopefully it's not a bug problem, but there's a documentation  problem to say, okay, how do you write a dependent collection that swaps part of a collection,  right? And does that make sense or is that completely insane?  I've got a question, does the software collecting packages run independently from each other,  means if I run two at the same time, which have got a dependency inside which breaks  if they run at the same time? Do they run independent of each other?  So I was kind of saying that earlier. You can kind of, so I had an example where I was  doing, I think it was Postgres. So we have a Postgres software collection, so I had an  app that used Postgres, whatever, 9, something that we're shipping, and Ruby 1.9. So what  I do here is I kind of say, SEL enabled Ruby 1.9, space Postgres bash, okay? So that gives  them both there. If there's a dependency between the two, then I think you should have a software  collection that is dependent on the two, and then you use that software collection if you  follow me. So in the Rails, Ruby and Rails example, I wouldn't kind of run them both.  I would have built the Rails one such that it's dependent, and then I would SEL enable  Rails 4 bash.  Yes, and it's just that one depends on the other, rather than using both. The only time  I would use both is because for my application, I need two, three, four different pieces of  software, Postgres, Memcache D, and Ruby.  Okay. We have time for this one and one more. So the next question goes to whoever's going  to go get me a beer after the question.  In the Retard developer blog, there was this topic one year ago, and compared to this,  what's the news?  Not a lot. Like I said, actually what I do think we've found, like kind of over the course  of, I mean, software collections have been now around what, almost three years. So what  I think we're finding, particularly since we started shipping some stuff, whatever year  and change ago, is the problems people run into. So like this dependent collection problem.  And so getting real world use starts to poke some of the holes in it. I think the other  big stuff that's changing is, like I said, containerization is already hugely important  and becoming way more important.  And how that integrates with packaging, I think is also going to be a really interesting  ride.  Is there any project page or something where the community can...  Coming soon.  Yeah. Let's just say, I would really, really like there to be kind of a proper upstream  for kind of software collections as a concept. It is all open source, but it's kind of like  all over the place. So while it's open source, there's no practical means of finding it all,  right? Unless you work with the CentOS guys and have this magic ability to find things.  But so one of the things that we have in the works is to try to get kind of a landing place  for software collections as a whole. And we've heard from people about the software collections  that they've built that they would like other people to be able to use. And so kind of allow  for a community to form around it. Everybody's got lots of things to do.  Any other questions? Apparently no one wants to give me a beer.  All right. Thanks, Langdon.  You're welcome.  Thank you.  Thank you.
_FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_0_1043.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_1044_1643.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_1644_3491.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_3492_4259.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_4260_4667.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_4668_5231.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_5232_5555.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_5556_6143.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_6144_6323.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_6324_7895.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_7896_7967.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_7968_8315.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_8316_8411.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_8412_10007.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_10008_10115.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_10116_13571.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_13572_13763.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_13764_17267.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_17268_17483.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_17484_18359.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_18360_18791.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_18792_19103.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_19104_19247.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_19248_19367.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_19368_20519.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_20520_20723.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_20724_22751.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_22752_22967.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_22968_23075.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_23076_23363.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_23364_23567.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_23568_23687.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_23688_23867.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_23868_23951.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_23952_24191.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_24192_24239.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_24240_24347.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_24348_24395.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_24396_24611.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_24612_30563.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_30564_32615.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_32616_32663.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_32664_34031.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_34032_35195.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_35196_35603.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_35604_38015.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_38016_38663.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_38664_44699.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_44700_44759.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_44760_46967.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_46968_48167.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_48168_48899.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_48900_50111.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_50112_50243.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_50244_65183.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_65184_65279.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_65280_65771.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_65772_67571.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_67572_67739.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_67740_67871.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_67872_68231.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_68232_69059.png _FOSDEM_2014__A_Method_for_Distributing_Applications_Independent_from_the_Distro_69060_72333.png
smarter than me. There we go. I wonder why I didn't echo. So as I said, I was a camp counselor for a long time, so you know, big parade fields, etc. And what I do is I work for Red Hat. I'm what, you know, this kind of new thing called a developer advocate or developer evangelist. And so I've spent, I don't know, about 15 years as a, mostly as a software consultant. So a lot of different projects, mostly kind of big enterprise sort of things. And now what I do is I talk to kind of developers who use kind of Red Hat products, in particular Enterprise Linux, talk to them about what they hate about it, and then I go and talk to engineering about how we can fix that stuff. So that's mostly what I do. And that's a little bit about me. So this talk is actually, it's kind of essentially about this concept called software collections. But it also, I come definitely from the developer background rather than coming from like a packages background. So I think I have a lot of biases towards being a developer and, you know, not needing things like QA or, you know, any of that, any release stuff, you know, and it will just send it out there and everything will be fine. No security updates, et cetera. But so just a little bit of, you know, on me. So here's a little bit of my argument. Over the last, I don't know, whatever, 15 years, Linux distributions, you know, they kind of started in this kind of world where, you know, the open source community was relatively small. The even proprietary software and open source software, neither one could you trust basically at all. They, you know, it was all run by developers who didn't really know or care what they were doing. So things like security updates would never happen. QA, if you were lucky, you know, et cetera. So early on, I argue that the distributions kind of formed to try to basically solve some of that ease of use problem and kind of redistribution problem around security patches and software and kind of things like that. So that you kind of had, you knew where you could get stuff from and you could also redistribute things so that they would stay secure. It was also kind of over the, arguably earlier than that, but, you know, over the same 15 years or so is where, you know, kind of the dynamic linking concept really took off and caught on. You know, there were definitely pockets before that, but it wasn't until, I don't know, like 96, 97 where it was like, it started to become like the way to do things. So, and then the last thing was actually license assurance. This is, was much less of a problem with proprietary software, obviously, although in fact they were hugely violating licenses anyway, but you couldn't tell. But with open source, it was a big concern, right? So you wanted to make sure, particularly as open source started to be adopted by the enterprise, that you knew what kind of licenses you were getting into. Also, feel free to interrupt me. So you can tell me I'm crazy. If it's just blanket crazy though, well, let's wait till the end. So particularly now in the last maybe five years, there's been, and I couldn't find a good graph for this. I've actually seen one before. I was looking at, I was talking about somebody earlier that, you know, kind of just the package count in Fedora, each version. It's just scaling like crazy. Obviously in something like enterprise Linux, we have significantly less packages, but even that is still, it's still a nice, you know, strong curve. And so between that, now we also have a massive plethora of actual Linux distributions that are reasonably popular. So it's not, you know, three big ones, right? And then everybody else, right? There's, you know, there's a good chunk at the top, right? And then Linux in general is also becoming hugely adopted, particularly in the enterprise, you know, kind of in server rooms in particular. But then even, even to some extent in the enterprise on desktops, you know, I don't think it's ever, you know, I don't think we're ever going to see the year of the desktop, but, you know, it is in some places, you know, I think it's largely going to be the only people who use kind of desktops and by desktop, I mean laptop half the time, are going to be people like you guys, right? It's going to be just developers, you know, just people who actually make software who use these things at all, right? It's everything else is going to be tablets and, you know, in that kind of tool chain. So we really need to kind of focus on making sure that we can distribute things well. So here's my, here's the extent of the math I can do anymore. You know, so we have a whole lot of projects, and then we have versions of each project, then we have it across distributions, and then we have versions of the distributions, right? So now we just made the thing massively wide, right? So kind of typically, you know, we, we just do this kind of, you know, take a tar ball, right? And then kind of have to figure out the distros, packaging rules, and then kind of figure out how to build the thing itself, and then start to, you know, kind of put that into a package. We got to follow all these rules and make sure it goes in the right places, etc. And then on top of that, inside an individual organization, there may be kind of a second set or third or fifth or 47th set of packaging rules within that organization. So you have, you kind of have to solve the problem over and over again. So what would be a lot nicer is if we could kind of say, okay, I have this package, right? And if you saw, I don't know how much of Donnie's talk covered this earlier, I know he's covered the past, but, you know, whatever that means, right? So a package. But essentially, if we had a blob of thing that we could run, say, on any distro, right, that was OS independent version wise as well, so you can kind of go this way, right, for the package could drop on each one, or you could go this way, right? So, you know, version two, version three, version four of the OS. And then the other thing is, can you also are there, can you start to trust essentially the developers to have packaging rules that make sense, you know, for their application, rather than it being a distribution problem? So lots and lots and lots of people have tried to solve this problem. Here are whatever six, five that I thought of off the top of my head, and ones that I could find random, easy links for. Actually, I had Linux apps in there too, but I couldn't find a good, like, place to point people off the top of my head. So they, some of them solve the exact same problem, some of them kind of take it from a slightly different perspective. So that's what I think is kind of interesting is to, it's kind of, you know, every piece of software anybody writes has trade-offs. And these people clearly were taking different trade-offs, right? So kind of the environment modules, SEP, and to some extent Gen 2 slots kind of all take roughly the same set of trade-offs, very similar actually to software collections, where it's kind of like, I want to have multiple versions of the same thing installed on the OS at a time, and I want to rely on the user to choose between them when they're running applications. OS Tree, relatively new thing that is really actually about, it's really about supporting, arguably supporting debugging. So you can take an entire set of binaries and kind of swap them out in and out, and it's for, you know, doing like debugging of GNOME, right? So it's not just one application. Yeah. We're going to look at doing that for off-rope data. Using OS Tree? Yeah. So mind you, GNOME I think has three or four different ones that all try to accomplish this goal. There's Glick 2, Glick, it's another one too. And then you have update alternatives, which is also trying to solve the same problem, but in a very different way, which is I want to be able to switch for the entire OS what the version of something is. So instead of per application, which is what these guys do, it's actually for the whole OS, right? And that can kind of cause some problems, particularly something like Python, right? So you switch the version of Python and your OS on Linux, right? And bad things are going to happen. And like I said, there's lots and lots more. So taking some of that thought, right? And taking some of the... So Red Hat has a little bit of a problem where, particularly for Enterprise Linux, a lot of our customers want to use whatever version of Enterprise Linux that they're using forever. And I mean forever, right? Because, well, one of the reasons is because the application that they're running on it is working just fine, right? So why would they want to invest normally? What you have to do is you have to invest development in effort, QA effort, release engineering effort into modifying the application to be able to run on some future version of the operating system. If it's a junk application that just does what it does, why would you make that investment? So as a result, they instead yell at companies like ours, right? And lots of other ones. Microsoft has this problem a huge way as well, as you all have, I'm sure, seen with their big EOL announcements every time they announce an EOL. And then it'll... It's like five times they EOL it, and then it'll actually die. So they have this huge problem as well. They do some other interesting things. Look up the civilization fix, the video game civilization, the fix they did in Windows sometime. So what a software collection tries to do is let's instead of having the application kind of natively touch the OS itself, let's put a layer in between it, put a layer of indirection in there so that we can now take this blob that's in between. It might be Ruby, right? Or it might be kind of a full stack. So it might be a web server plus Rails and Ruby and everything else. The idea is there is a blob that you depend on that's kind of above generally, above the actual OS and below your application. So that's kind of what a software collection is trying to do is saying, okay, let's identify that blob and let's try to make that blob portable. So now what you can do is you can say, okay, I have this janky old Ruby app, let's say Puppet, that is running on Ruby 18, right? Why is it running on 18? The way I hear it is because Enterprise Linux 5 runs Ruby 18 and why is Enterprise Linux 5 running 18? Because Puppet runs so. But any which way, the point is, you know, you take any old application or older application and what you can do is you can kind of say, here's this blob, I'm going to guarantee the API on the top of it and then I'll change how it delivers that API underneath so that it can be portable across multiple versions of the OS. You also have the problem the other way, right? So there's a lot of push for people to move to Python 3. However, not all the distributions are ready to make all the changes to be able to move to Python 3. So how do you start writing applications in Python 3 right now without the, you know, kind of working your Linux? Well, one of the ways you can do is with a software collection. So it's kind of, you know, by the two examples I'm kind of giving, right? You can kind of see it's OS independent, right? The OS can continue on its merry way and your application can decide to upgrade when it's ready. Or you can be aggressive and say, hey, I'm writing something bleeding edge brand new and I know I want to use the most cutting edge version of whatever because I'm not going to change it again for 10 years. So let me get that right now and use it right now to write my application. So it kind of going into it a little bit more. So it really is, as it says, it's kind of defining a process a little bit more than a product, you know, so it's kind of a conceptual thing. You know, there is an implementation, but that's kind of what it does. The other big thing that it does is it packages outside of the normal kind of Linux file system. And that's sometimes a problem for some of the people I think in the room. And it allows multiple versions to be co-installed, right? And then per application you can essentially start, or you can say, this application gets this blob rather than that blob. So the way we kind of defined it is it installs under opt with, you know, the L acronym that is the Linux naming that I can't actually say. You go register for a Linux name so that you can get kind of a short code there and then you put your application name there. And that's kind of where it installs. And then you have kind of some scriptlets that know where that is and can provide that information to the application. And the way it basically does it is by fancy-path-ing, right? So the application doesn't have to know, this is the other, sorry I didn't mention this part. One of the big differences between the ones that I was talking about earlier and this one is that the application doesn't have to know that it's not the base OS version, right? Which is, I think, hugely important because you don't want to have to rely on your average developer to figure out where it's going to be. And again, to allow for that OS independence, you know, someday, right, all the Linux distributions will be on Python 3 and that will be what's at user bin Python. So what you want to do is be able to write the application into, say, user bin Python at the top and then it just goes to the right place rather than having to, you know, do fancy sim links or, you know, or make a mistake and say, or have user bin Python 3 and then find out now you have to do engineering effort to port it. So in a little more detail, here is kind of the way the one for Ruby works that we have built. So here's kind of regular Ruby, right? You know, assuming that you installed Ruby, the other nice thing about this is that you don't actually have to have, I can't remember if Ruby is installed by default, but you don't have to have Ruby installed except for this particular application. So in other words, it doesn't have to be anything in user bin Ruby. It can just be down here. So I think it's pretty obvious, but basically here's the normal path, right? And here is our special path. You know, we ship a Ruby 193 variant, you know, one of these blobs. It sits under our age because that's our name. And that's pretty much all there is to it. The disk little script here is what does all of the magic such as it is, which it basically kind of provides those paths so that the command to run these, which I think is down here. So SEL enabled Ruby 193 bash. So what you do is you kind of say, here's the command, right, SEL? I want to turn it on. All right. And then which one do I want Ruby 193? And then what do I want to execute with it? So if you just want to get a new shell back where Ruby 193 is available, you just run bash. But it could just as easily be, you know, some app, right? So from a kind of, if you're a sysadmin, it would be like an app that, and then you put this in, you know, a shell script or a start script or something script, you know, for a developer, he's going to drop in a bash most of the time. You can add this to your.profile. And so it's just always there if you want as well. Then we talk about, you know, there's basically some tools out there that kind of help make this stuff work. And so there's some links. I think all these slides are available somewhere. Joe would know better than me. I will email them to Joe. He will make them go. This is the ones that we've done already. Actually, it's some of the ones we've done already. We actually have some that are out there for like hdbd 2.4. I can't remember what else is up there. That's just kind of public. There are some other distributions that are related to ours that also have all of these. There's a guy, I think, who's talking next to who had a hand in those. So I think they are well represented now actually in CentOS, Scientific Linux, and Fedora. So Fedora just approved them whatever it was last week or was it the week before? Okay. I can always pray. All right. So it's getting there. There is some of this stuff built for Fedora, but you can't get it in kind of the mainline distro repos yet. So that's basically most of my slides. I was expecting more interruptions. I invited hecklers and they're not delivering. Exactly. That's true. That's true. So here's kind of my argument. I'm probably crazy, but I think vendors in general are a lot more trustworthy. When I say vendor, proprietary software, open source projects, everything, any kind of application, I think the distribution doesn't have to take on as much of a responsibility for making sure that all the stuff in the world runs. Yeah. Yeah. Yeah. Which is not a high bar, but let's say are reasonably trustworthy. So they're even more than that. I think a lot of this stuff, most people understand, most people who do this kind of work, most of them they understand what's important, how to break their things up. Yeah, they're still false, but I think a lot of them are starting to get it. There's well-known libraries for particularly security-related things. So people mostly use those libraries. I think Bruce Schneier is finally getting through to people and saying, stop writing your own encryption libraries. So same kind of thing. I think in general they're considerably more trustworthy so that I don't necessarily think that every distribution needs to package everything under the sun. I also think another big reason is redistribution. One of the things that software collections has that kind of fails and goes backwards in time a little bit is a lot of things are considered to be statically linked because the only way you can be portable across OS is if you carry the bits you need. So that means static linking, sort of, or at least it means that it's got to be carried in the bundle. So it can still be dynamically linked, however it's got to be dynamically linked from within the software collection. So one of the things that I was certainly confused about when I first looked at this stuff, so if you think about an RPM, one software collection does not equal one RPM. A software collection is more like 70, 1,000 RPMs. So it's broken up just like you would do a normal package. It's just that it's all kind of in this one context to say, hey, I'm part of this one software collection. So you do, it is better, quote, unquote, than straight out statically linking, but you still have a problem of redistribution. So one of the reasons that dynamic libraries are nice is that you only have to copy that one file onto every machine. With a software collection, let's say it's open SSL and you have 37 software collections on every machine all using open SSL. That means you have at least 37 different copies of that SL RPM. So you need to distribute that all the way through. However, and judging by the overflowingness of the config management talk, config management has really finally taken. I remember the fits and starts. I mean, there was a huge push for it, I think, around 2001 and just never quite took, but it's really kind of taken now. So I think redistribution, particularly in big data centers, has gotten a lot easier. So I think that's why we can kind of take this risk. The other thing is the other risk you have is that now you have a copy, you have 37 copies of open SSL. Disface is not that big a deal anymore. So does it really matter that you have lots of different copies of the same stuff on one machine? And deduping file systems someday, I pray, will be default and so it won't matter anyway. The other thing is that I think users are more sophisticated. So I think that they can, they understand the context that they're, and when I say user here, I mean I develop them, right? So somebody who's going to consume one of these software questions, they're way more sophisticated about what is going on underneath and can usually figure this stuff out. Yeah. I was going to say that the nature of the software collection does not mandate that you have 37 copies of open SSL on the system. It may be that you require three specific compatible versions. And given that it's built on.cl, you can build a, an SCL top and other SCLs. So you can narrow that down manually as well. Yeah. So I think dependent collections, right, are a very real and very important aspect of this that we're still working through some of the, like there's a, I don't know where it's published right now, but we've been working on trying to get a good guideline for how to do dependent collections, right? So do you really want to do a rail software collection? Well, not really, right? You want to do a Ruby software collection and then have a rail software collection that depends on it, right? So, and what, what's the right size of a software collection, right? What is, you know, should it be a full on, you know, kind of web server with, you know, like say Ruby and Rails, or should it be, you know, piecemeal? And I think a lot of those questions I think are yet to be known. The other thing I think that is going to be interesting about this is kind of the, the move towards containerization, right? So containerization, and I'm using that term to mean, to be as far away from possible from any individual implementation, right? So you have things like Docker, you have LXC, you have, you know, this combination of open, we use an open shift which is like C groups and SC Linux. You have, you have actually much older and much more sophisticated versions in other operating systems that are not Linux. But the kind of this containerization concept where now we kind of say, hey, this whole application is now a blob and it's portable and we can run it into certain kinds of conditions and it can be updated on its own. I think that's going to be the next step for some of this stuff. And I think a lot of that will rely on something either exactly like software collections or something very similar because it, it allows you to kind of bring that bundle inside, you know, your container. Are you ready for questions? Yeah. All right. I've got a mic. So if you have questions, raise your hand, we'll bring the mic around so it turns up on the video later. One thing I didn't follow. If you've got two software collections installed for two different versions of something, so Ruby, what's the mechanism by which the application gets to use the appropriate version? So this versus like update alternatives. So the question was basically like, how do you say I want this application to use Ruby 1.9 and that application to use, you know, Ruby 2, right? So this, of course, is only at the very bottom of the screen here. So you probably can't see it. I should have probably put it at the top somewhere too. But this unlike, say, like update alternatives, you, you tell the application which one it's using later. You know, the application doesn't actually know, but you say SEL enable whatever collection you want to use, and then the application you want to run so that it's basically running it inside a shell of its own that has the kind of correct paths. So it finds it as if it was at its normal locations. Right. So this is changing the environment. Essentially. Yeah. Try to organize your questions by row. I don't even know why I got up at 6.30 this morning to go to the fitness center at the hotel. I'd like to ask how security updates are propagated to containerized applications in software collections. So say I have a Ruby software collection and Ruby depends on some library that got the security update upstream. Does it mean that the maintainer of the collection will have to update it manually? And if I have like five collections which depend on this library, does it mean that five different people will have to update their collections to include the update? Exactly. Yeah. That's the problem. Not if you do dependencies. So like I said, if you use kind of, let's say you had a Rails 3 and a Rails 4 both dependent on the Ruby 1.9 software collection and the security update happened in the Ruby 1, you wouldn't have to update the Rails 1s in theory. I mean one of the things that I like about software collections is it kind of also lets you, well, like dislike I don't know, but it also lets you defend kind of your QA problem, right? So one of the reasons that the enterprise doesn't want to upgrade, often it may be that they expect the code to completely work, but they don't want to test it, right? QA resources are often the ones that you have the least of in any organization. It's often the constraint at any engineering organization I've ever worked with that keeps features from happening. And so when you're doing, again, kind of talking about that janky old app that is just running and doing its thing, why would you want to waste that very limited resource of your QA people to test the upgrade? So I also would hope that the, you know, if, I guess I kind of expect that in a particular enterprise, much like I would kind of expect with OpenShift is like in a particular enterprise, I would kind of expect them to build their own software collections for use in house, probably depend, hopefully, right? Dependent on, you know, for a large percentage, 90, 95% of the functionality on somebody else, right? So like on Red Hat or on the community in general or on something, but that they would have their own one so that they could be confident that the API layer at the top is what they expect completely for the applications that they're building. You know, it does two things. One, it gives them more, you know, kind of tolerance for moving across versions, but then it also does that kind of opposite effect, which then you can still control your developers by saying, here is the approved Ruby software collection, right? Here is the approved Python software collection. Don't go installing random other stuff unless it's made it into the software collection. One back there. Yes. I have a question on how this SCL enable works. So can I use multiple of these environments at the same time? And so it would sort of only map this for this one process with something like P-Root or Mountbind or how does it get into the environment? Yeah. So you separate it by a space, right? But you can just kind of list out multiple ones that you want. And it's not that cool. It's just paths, right? So this is not, you know, teruding some brand new file system or something and running in that context. There's no security here, right? None of that stuff. This is just to provide the environment that is kind of so when you make a function call, you get the right binary. That's it. Yeah. So what is like the lowest level of libraries that you include? For example, these packages, would they include Lipsy or do you still expect that from the distribution itself? The package maintainer needs to decide that. And everything that you don't include makes you less portable, right? But makes it so that you have less of the problem that he was talking about back there of massive redistribution updates. But I think it's hard to know like all the pieces that you need. So I wouldn't even know like if I need Apache, I wouldn't even know all the libraries that it uses. So well, I mean, there's the one aspect of you need to know what libraries you care about, right? But then there's the other aspect of where's this fine line of how much stuff to include. I think we're still struggling with that too. I think, you know, I don't think there is, well, I don't think there will be a sort of bullet kind of answer. What we are trying to do is do a better job documenting how to think about it. And there are starting to be a lot of examples, right? So now we have whatever, you know, 10 or 12 that we're shipping. You know, there's another problem. There's at least another five or 10 that I know about and kind of in the software world. So there's starting to be examples that are making decisions. I think we're also going to run into problems. You know, it's really nice actually that REL7 is kind of on its way out the door, because that'll be the first test of some of this stuff. So, you know, I'm running the beta here. And so what I'm, but it's only been like a couple weeks. So what I want to do is start to try to run, take an application that I wrote on the software collection for Ruby on REL6, run that hopefully unchanged on REL7. And then I actually want to be able to run the same thing unchanged on OpenShift. Because OpenShift actually uses software collections to, that's how it delivers those blobs as well. Do I see a question over here? Yeah. No, no, he's first, then you, because that way he can keep walking in the right direction. Thank you. So I'm no advanced coder. I'm not a coder at all, but I'm aware of distributions and dependencies and versions and stuff. And I just want to understand correctly what this means. So you say I can make a package of a software and include my own dependencies, what I need, and do it all into the opt system. And all I need to do to make this work and any distribution is to insert this SCL trick. Is that about correctly? So it's a little more complicated than that. So the first part is you have to create the thing, right? So you have to have your application and all its dependencies. And then you have to shove that all into some sort of package. In the case that I'm talking about, that's all RPM. Then on deployment, so on install of the application is where the opt stuff comes in. So it's where it's going to show up. And then this SCL stuff works because basically now you have all of the pieces that you need. Some of it might be coming from the operating system itself. Some of it might be coming from this set of packages called the software collection to actually deliver the API for, say, Ruby to another application. So we expect, and I know of a couple of companies that are doing this, but we expect that some companies will kind of, whether it's open source or proprietary, they're going to actually create their own application as a software collection, which will then have a dependent collection that they also carry or set of dependent collections. So that then the whole thing will kind of get installed there. But then you'll have that, so when you install X, Y, and Z software, you actually get that whole collection from them and including any dependent collections that they may get from elsewhere. So every package brings its own environment completely? Right. So it may rely on some stuff that's in the OS too, or it may not. It's up to the package or the collection. Okay. My question is, are you aware of Nix and Guix package managers? Because they are doing a similar job, but I believe slightly better and without RPM. So as I said kind of on that earlier slides, like everybody has had this problem a ton all the time, right? Microsoft has a solution for this problem as well. This, I don't know if you guys know anything about.NET, but one of the major changes you may have remembered the advertising was.NET solve DLL hell, right? And the way it did that was having this thing called the global cache and then a per application cache. And what ended up happening, they had this magical mystical idea that in the global cache would be things like G-Lib C. What in fact happened is that every single application that got installed picked up their own, right? Mac has always, and to the best of my knowledge, and I don't know a whole lot about Mac packaging, but Mac has pretty much always worked this way in that every application basically carries its entire dependency tree every time you install it. So like it's not limited to Linux, number one, and it's certainly not limited to the examples I gave. I don't actually know the one you mentioned, so I'll go ask Google. Okay. And another thing, I didn't understand that what are the new demands for software vendors to be able to use this? Well, so it's really just a style. It's a way of saying, hey, this is a way you could deliver your application that, because this happens to the RPM, right? But the same concept could work in anything. It doesn't really matter that much. You just need kind of something like YUM to manage where they're coming from and getting updates. But I think a software vendor of some kind could say, hey, here's a way to distribute it, and it might be easier for you to build the packages for, let's say, Enterprise Linux 4, 5, and 6, they might be able to find a way to create a software collection that will install on all of them, right? So that they don't have to recertify for every minor version, right? That they don't have to, you know, they can also deploy in Fedora, even though Fedora is way ahead of RAL, right? Does that make sense? Arguably? So I've got a question on, I suppose this goes back to how big the software collections are, but say you have a software collection for different versions of Rails and a software collection for different versions of Ruby, is this just to suggest that for every permutation you want, you would then need to build another independent software collection on top of that? Theoretically, right. So theoretically, and that's the struggle, right, is where do you want to draw that line? So right now, for example, the one we built, right, this Ruby 193 actually includes Rails 3 as well. So even though it's called Ruby 193, there's actually, I don't know, 40, 30, some number of gems in there, I can't remember how many. But a whole bunch, right, which if you, if I read Ruby 193, I don't hear gems, right? So I'm not sure that we're entirely happy with that choice, that maybe Rails should have been separate, because then it would have been easier for us to say, okay, now we have a Rails 3 software collection and a Rails 4 software collection, both dependent on the Ruby 193 software collection. Maybe I missed it when I was just looking at my phone during your talk or whatever, sorry about that. Thanks, Don. I'm curious about some of the details. How do you, how do people actually create these collections? How do they deliver them to users? How do users install them? How do they manage them? What do the tools look like in practice? So I was going to include some of the macros, but I decided not to. But basically, when you're writing an RPM, right, you have this big thing called the spec file, which kind of describes what's in the RPM, right? In that spec file, we just add some new macros that indicate that this is a software collection and that it's going to go kind of in a special place. And that's about it as far as the difference. One thing that was kind of a design goal was that the same spec file, if it knew about software collections, could also be built to not be a software collection. So hopefully what will happen is that all spec files everywhere will have kind of the support for a software collection so you could build it that way or decide to not build it that way, depending on your use case. And then as far as the tools are concerned, it's just yum install, you know, X, Y, and Z, you know, software collection dot RPM, you know, whatever happens to be called. Though, you know, in Enterprise Linux, you have to add what's called a channel, which will give me started on that. But basically, you have to add another repo coming from Red Hat and it's got a bunch of them in there. You just say yum install, blah, blah, blah. I think CentOS, I think they're main lines. Did they make it to main line or are they still in test? They are. Okay. So in CentOS, you can actually just log in yum install, Ruby 193. So it's all the tools you're already familiar with for the most part. You know, they'll work just like the, you know, RPM does with puppet. So it has also all the failings of RPM, but, you know, it's easy. Hi. Do you have an example yet of an application that's packaged in the normal places, like where a user would expect them, but it depends on an SCL. So for example, you know, user would run the application where it normally is, doesn't, has never seen the SCL enable utility, but the application then enables the SCL it requires itself. Sort of, you know, basically, so I've done that with like ACPD, right? So, you know, to get the web server to use the Ruby 193 stuff, I need to kind of modify its start script. And I was, for that to go example, I was using the native Apache web server rather than a software collection. I think we want to be mildly careful of that because that's one of my fears is that if you have something that is distributed by the normal OS that's dependent on a software collection, that might be weird. You know, it kind of goes out of the user's, or when I say user here, right? The system administrator or developers' expectation of where stuff is going to show up, you know, where you go find stuff. So my assumption is it's distributed separately in a separate Jom repo, even maybe with an SCL, but when a user installs it, the application appears in its normal places, but it's using an SCL behind the scenes. Exactly. So that's exactly what I did for, like, ACPD. You know, if you want to set up a bash profile that it just runs, right? So, however, in general, if I had a, like, kind of a real top-level application, you know, so, you know, my cool website that runs in your office, I would rather that people package that as a software collection. And then it's dependent on software collection. I think user expectation. So, you know, I think having a shell script or something that's kind of off in a more normal place that points at it is fine, but I don't know, I guess I don't like if an application installs under opt, then it shouldn't touch the rest of the OS, right? If I choose to make it touch the rest of the OS, so, an example, a Calibre installs, if you install it from the Python script, installs under opt, it doesn't touch the normal operating system basically at all, but I can put a sim link to it from user bin. Right? Sorry, I just, one technical add, there's no reason you can't. It's just, that's my preference. Well, I don't know if you have already answered my question, but it's another twist on the situation. Let's say we have a Ruby collection or a Python collection, and I want to add a particular gem or Python module. I know how to create a packet for a native environment. Can it be easily added to the existing collection? Do I have to create a new collection and overwrite or remove the previous one? Right, so that's actually exactly the problem in some ways, we didn't really talk about it here, but the other thing that a dependent collection could or should be able to do, right, is we're talking about, you know, we have the Ruby collection, which this one happens to include Rails, right? So let's say I want to write a dependent collection that swaps Ruby, I'm sorry, swaps Rails 3 and puts in Rails 4. Well, we don't want you to have to rebuild this one, right? So one of the things that we're working on from a, I think it's more of a documentation problem, hopefully it's not a bug problem, but there's a documentation problem to say, okay, how do you write a dependent collection that swaps part of a collection, right? And does that make sense or is that completely insane? I've got a question, does the software collecting packages run independently from each other, means if I run two at the same time, which have got a dependency inside which breaks if they run at the same time? Do they run independent of each other? So I was kind of saying that earlier. You can kind of, so I had an example where I was doing, I think it was Postgres. So we have a Postgres software collection, so I had an app that used Postgres, whatever, 9, something that we're shipping, and Ruby 1.9. So what I do here is I kind of say, SEL enabled Ruby 1.9, space Postgres bash, okay? So that gives them both there. If there's a dependency between the two, then I think you should have a software collection that is dependent on the two, and then you use that software collection if you follow me. So in the Rails, Ruby and Rails example, I wouldn't kind of run them both. I would have built the Rails one such that it's dependent, and then I would SEL enable Rails 4 bash. Yes, and it's just that one depends on the other, rather than using both. The only time I would use both is because for my application, I need two, three, four different pieces of software, Postgres, Memcache D, and Ruby. Okay. We have time for this one and one more. So the next question goes to whoever's going to go get me a beer after the question. In the Retard developer blog, there was this topic one year ago, and compared to this, what's the news? Not a lot. Like I said, actually what I do think we've found, like kind of over the course of, I mean, software collections have been now around what, almost three years. So what I think we're finding, particularly since we started shipping some stuff, whatever year and change ago, is the problems people run into. So like this dependent collection problem. And so getting real world use starts to poke some of the holes in it. I think the other big stuff that's changing is, like I said, containerization is already hugely important and becoming way more important. And how that integrates with packaging, I think is also going to be a really interesting ride. Is there any project page or something where the community can... Coming soon. Yeah. Let's just say, I would really, really like there to be kind of a proper upstream for kind of software collections as a concept. It is all open source, but it's kind of like all over the place. So while it's open source, there's no practical means of finding it all, right? Unless you work with the CentOS guys and have this magic ability to find things. But so one of the things that we have in the works is to try to get kind of a landing place for software collections as a whole. And we've heard from people about the software collections that they've built that they would like other people to be able to use. And so kind of allow for a community to form around it. Everybody's got lots of things to do. Any other questions? Apparently no one wants to give me a beer. All right. Thanks, Langdon. You're welcome. Thank you. Thank you.
mostly as a software consultant. So a lot of and talk to engineering about how we can fix that proprietary software and open source software, that, but, you know, over the same 15 years or so packaging rules that make sense, you know, for running applications. OS Tree, relatively new Linux? Well, one of the ways you can do is with a know, someday, right, all the Linux distributions some other distributions that are related to ours probably put it at the top somewhere too. But their own one so that they could be confident more complicated than that. So the first part is and it's certainly not limited to the examples I happens to the RPM, right? But the same concept have been separate, because then it would have be able to do, right, is we're talking about, you such that it's dependent, and then I would SEL