Dinosaurs are running wild!!
Brown, Richard
Containerised Application technologies like AppImage, Snappy and Flatpak promise a brave new world for Linux applications, free from the worries of shared libraries and dependency issues. Just one problem, this is a road long travelled before, such as in the application dark ages of Win32 applications and DLLs. And it worked out so wonderfully there... Do we risk a future where, like the resurrected dinosaurs of Jurassic Park, this family of applications will break their containment and start eating our users? This session will try to present a fair argument about the situation, frankly discussing the benefits promised by these technologies, but highlighting the very real issues and risks their widespread adoption could, and in some cases are, already bringing to the table. The talk with cover the promised benefits of application containers, such as AppImage, Snappy and Flatpak. It will detail the empowerment of developers who use the technologies, the ability for upstream projects to have a much closer role in delivering their software, and the benefits that brings to both the upstream projects and their users. But as a counter to those benefits, the session will detail some of the risks and responsibilities that come with that technology. The complexities of library integration, the risk of introducing new forms of dependency issues, and the transference of responsibility for those issues, plus security, away from the current Distributions delivering upstream projects towards those upstream projects directly. As a conclusion, the session will start to ask the question, what the hell should openSUSE do about this mess? How much can we help fix it or mitigate the problems? How much do we want to be involved in that new world?
 Wow. Thanks everybody for coming. It's like the last, well, second to last talk for me. After this, we've got the board meeting in the main hall and then we're done. So I'm Richard. I'm here to talk about, well, dinosaurs or as most other people know them, containerized applications and how we need to deal with them. Now they're out there. Now users are really using them. Now we're starting to see the problems with them in the real world. And this is a variation on talk I did at FOSSTEM earlier in the year. So I'm going to start out rehashing a lot of that. So if you've seen my talk, I'll try and run through it a little bit quicker. And if you haven't and you're more interested, you can go back and watch that. And then I'll talk about some of the new and exciting stuff and why I had to rewrite my slides twice during this conference. But really, when I started looking at these technologies, looking at Snappy, looking at Flatback and looking at App Image, it struck me that I'd seen all of this before. And in fact, where I first sort of saw similarities was actually back in sort of the Windows architecture, the original Windows application architecture and how Windows deals with loading up libraries and dependencies in the Windows world. And I mean, to start off with, look at it from another perspective, Linux has a lot of traditional Linux, traditional Linux packaging has an awful lot in common with Windows 3.1. It's a similar world. There is no ABI backwards compatibility. Things are constantly changing. Things are constantly evolved. There's one great big file system where everything is dumped into Windows, Windows system or C Windows, global identifiers for all the symbols that everything starts clashing all over the place. It's an absolute maintenance nightmare. And it's where the term DLL comes from. Because ultimately, all developers want to do is have a nice, well, simple environment to work with. And in the Windows 3.1 world, they had to development and test every single possible DLL combination that might be seen in the wild possibly anywhere. And then every time they had a patch, test that app patch in every single combination everywhere. And then when there was a dependency or library patch, test everything in every single combination everywhere. And they would do that and they would try and Windows was being used everywhere and then they'd cry because it would all break anyway and it would all go horribly, horribly wrong. And Microsoft thought they could fix this. And they tried very, very well to do that. And they were somewhat successful. Now, Windows 2000 introduced this concept of side-by-side assembly, which is basically containerization or application isolation for the Windows world. Having a separate memory space for every single application and all of its DLLs, loading up those DLLs privately from a folder in the file system, having Windows files protection, doing disk isolation of system DLLs, and having these fancy tools to audit all of that and migrate those legacy applications and deal with those problems. And you ended up with this wonderful situation, if you're a Windows user, where your Windows 2000 or later could run an application for Win32 or even for POSIX or even OS2, or using these fancy little runtimes that were packaged up in that Windows environment. So, problem solved, right? Well, no, of course, it all went horribly wrong. And not just because it was Microsoft doing it. There were very, very real sort of social and practical problems that evolved over time that we all saw. It was a security nightmare. All of these libraries, all of these dependencies end up lurking in countless folders, all being maintained to various degrees by the developers that put them there, all then becoming lovely security vulnerability, security gaps, attack services for things like WannaCry and other malware to go and abuse and misuse when certain applications are loaded in memory and loading up the bad DLLs in question. It's also maintenance nightmare as a user. How do you then update that application on the user's machine? Anyone using Windows, how many application uploaders do you have in your system tray? I mean, everyone builds another system update. And it just doesn't scale. It ends up, especially in the open source world, being a bit of a legal nightmare, one of the biggest issues of getting open source software in Windows is actually this problem. It's actually figuring out, okay, can we put this open source DLL in this container with everything else we need to put in there to get the thing working? And we, in the case of the Windows world, with these DLL-encompassed applications, the developer is the distributor. They have to worry about those legal issues and those legal concerns. But there is one bunch of people happy, hard disk vendors, because everybody's using up more disk space. People need more bytes. It's not terrible. And this was, yeah, like I said, since 2000. Meanwhile in Linux land, we were looking on smugly because we'd already solved all these problems. Sort of. And the way we solved these problems was with the traditional Linux distribution. And primarily, the things that distributions brought to the table and still bring to the table today, isn't the technical stuff per say. We all solve the technical problems in our own way. We all have our different package managers and our different philosophies on how you should do this engineering and packaging stuff. But the universal thing we all really bring to the table is we care about the security of the operating system and its applications in the context of a user looking after it. We're maintaining this stuff. We're auditing this stuff. We're constantly monitoring CVEs, pushing out these updates. And especially with the open source side of things, a major security vulnerability needs to be handled in a very particular way. You're going to have embargoed security mailing lists. You need to have trusted people on there. So you need to have the right people there. You need to have the right relationships there to get on those embargoed lists. And distributions play that key role of being there and able to get those fixes out before issues start hitting your code tree. And like I said, maintaining it, packaging those updates, keeping them updated, dealing with upstreams, helping work with upstreams with that. And lawyers auditing all of this stuff, checking it's compatible, making sure that the license is being chosen are sane and consistent with each other. So when I talk about this distribution stuff, you know, lots of people and it's kind of spawned by these new technologies or this kind of resurgence of the bundled application side of things of, you know, shared libraries are a problem. We're trying to solve the shared library problem. Dependencies are a pain in the ass. You know, I don't want to worry about the dependencies. I just want to worry about my app. Shared libraries do solve real problems. It's not just a case of being more efficient on disk, although that is a benefit as well. But having fewer libraries to worry about, having fewer dependencies to worry about, or having fewer repeating copies of the same dependency to worry about is a very beneficial thing. You know, when something goes wrong, when something is insecure, you have fewer copies of that thing to worry about. You have fewer places to patch. You have less manpower required to patch it. Less double work sending multiple copies of your vulnerable lib SSL out there or your vulnerable SAMBA libraries out there. And that makes it easier to then review it both as a user and as a distributor or as a developer who is distributing and ensuring that legal and security compliance, you know, this is something you can trust and rely on today, tomorrow. And also when it stops being maintained, it's still doing it in a way that's, you know, at least was saying at the time of stopping it. So from the open source sort of distribution side of things, mission accomplished, right? Well, no, like I already said, the open source distribution way of doing things was very similar to actually how Windows did it. You know, we still had these problems. We still had the issues of compatibility of making sure all these bits and pieces work together, of, you know, portability. How does an application built in one context work elsewhere? And how do you handle this issue of keeping something, keeping this software being delivered and just working and handling the fact that the open source world and therefore everything you're distributing is moving at a constantly changing pace of change. But we're not Windows. And when you're distributing in the open source world, there's different factors at play. So thinking about the compatibility issue, we all end up doing different distributions. We all end up having different libraries and different applications. Different applications require different libraries so the problem becomes exponentially complicated. And application developers don't want to worry about all that mess. They just want to deliver their software in the hands of users. I get it. They don't want to have to worry about which choice of dependencies did certain distributions pick. But most of the time they don't have to. In reality, distributions have their own maintainers. We have our own communities all here who care about this stuff and this part of it. And are the sort of the second tier making sure that the application gets in the hands of users. And you care about it in the same way as the upstream maintainers care about it. So very rarely does it really become a problem because most times it is really being done by the distro maintainers who care about this. And that's what you all do and that's what we've been doing for years and we're bloody good at it. As you can just see in Tumbleweed where we repeatedly ship stuff as fast as the upstream shipping it. But it is importable. It's open SUSE, bi open SUSE, for open SUSE. And an application developer wants to make sure that their software runs in a many different context as possible and as many different distributions as possible. And they don't want to learn a whole bunch of different build tools and they don't want to learn 20 different ways of doing things and they don't want to retest it 20 different places. But again, distribution communities often take care of that problem for the application developers anyway. And then pace of change. Every distribution does everything at a different pace. Heck, if you're open SUSE we do it at two different places. We'd leap, we'd do it regularly every year with major versions every few years and we do Tumbleweed where we just go nuts and as fast as every upstream wants us to go. But in the traditional old fashioned way of doing things, the regular release process gets in the way of that application delivery desire. In the traditional model you can't necessarily run the latest version of software on your stable open SUSE or your Debian system. Debian in particular is the perfect example because they freeze so well and so hard and so solidly that that becomes even harder and they're proud of that. Leap we purposefully design it in a way to try and bend the issue around the edges and deliver faster stuff when we can, how we can. We have the build servers, we have OpenCure to help with that. But it's still a very real problem. Sometimes we just have to say no because the technology can't do it. But that problem, that balance, how do we deliver this software? That is again something that distributions do currently take care of. So how much of a problem is it really? Doesn't matter. App image, flat packs, and App here to solve all the problems left anyway. And they exist to solve that issue. They exist to try and move these problems out of the hand of the distributions or reduce the need for the distributions to do this. So application developers can get that software in the hands of users at the pace the application developers want to be. And they do so by providing a bundle containing the app and the libraries, all the dependencies they need in then some kind of container or context or bubble or whatever. I'm going to keep on saying container although technically speaking, you know, that's open for interpretation. And the big promises of all these technologies, despite details around the edges of how they do things, is to solve all these compatibility and portability issues. It's only going to have the compatible libraries in the bundle. So you don't have to worry about anything from the distribution. You just put your application there, everything you need, you know. It will be portable. It will work everywhere because all the dependencies will be solved in there. You'll never have to worry about what does a distribution ship. And of course that means you can ship it at the pace you want whenever, however, don't have to worry about what the distribution is doing. And it's just going to work. And it's going to be wonderful. That's the promise. And then you have nice architecture diagrams like the snappy one here, or you have really stupidly complicated architecture diagrams like the flatback one, where you have this kind of model of, you know, just ignoring the operating system down the bottom pretty much. It's just there. And then there is some layer on top of that, be it the frameworks or the run times, which, you know, provide this sort of layer of dependencies, which are an awful lot like the dependencies the distributions are currently doing anyway. And then the bundle itself contains the library, the code, the application, everything's fine. But it doesn't work. In practice, it doesn't work. Most of the time it does, but there are still some very, very real issues there. And in fact, the biggest problem that comes around in reality when you start using these in production is this issue of, you know, compatibility and portability. The myth is not true. Because at some point, be it app image, flatback or snappy, there are some assumptions made about the stuff below the system. And we talked about this. Like, you know, in the case of snaps, you know, that is still the kernel. You know, everything above the kernel is assumed to be delivered by some snap somehow. But no kernel is equal. You know, every distribution has a different kernel with different conflicts. There's still problems there that get introduced by different, you know, by different kernel arrangements. We see this in OpenSUSE most actually with Steam, which isn't using one of these technologies, but basically uses the same approach of having a containerized Steam, you know, runtime that gets put in your user area and run that way. And everything was fine on OpenSUSE with Steam for the longest time. It just worked. It did its job. We could move everything in OpenSUSE and everything was fine. And then we changed our Glib C and it exploded spectacularly because their Steam runtime was built with an older Glib C and therefore nothing would run anymore. All those libraries would go horribly, horribly wrong until we started scripting around injecting our Glib C into there and rebuilding everything. And it was a complete mess. And we've seen this issue with flatpacks. We've seen this issue with some of the experience with snaps. We've seen the issue sometimes with app images. But one of the nice things and one of the reasons why I've always come into this liking app image is at least app image documents this problem. You know, it's stated there that it isn't trying to be a universal portable application solution. You're going to have to gather the binaries for the dependencies for the distributions you're targeting. You're not going to magically solve this problem everywhere. But that means that if you're using these technologies, you still have to worry about all of the compatible dependencies which might not be provided by any distribution you might want to run it on. That's a lot of stuff to worry about. Heck, that's the stuff that we all do at OpenSUSE all the damn time and it takes all of us to do it. If you don't get your head around that, your users need to expect crashes. So is it hopeful? Is it hopeless? You know, well, you know, you talked to flat pack people like this, oh, no, we've solved it. You know, we've got these run times or on the snap side of things, you know, we've got these base snaps. Well, those base snaps, those run times just end up being some second curated middle distro. It's middle where for the containerized world. Cool, fine. But you haven't solved the problem. You've just moved it into a different context. It's still another distribution. You're still having to have distribution engineers worry about this stuff and curate it and secure it and patch it and maintain it. Maybe it isn't a real solution. Maybe a real solution is actually figuring out a way of commonly agreeing between us, the distributions as a platform, the application run times as a delivery mechanism and hopefully even the developers on common, I use the word standards here, but let's say common agreements of, you know, what can you expect from your base system? What can application developers expect from their run times? So people can go into this and we won't just have random crashes when you install your app image or your snap on an open Susan machine and it doesn't deliver the kernel the way it's expected or the library is the way it's expected. And until we do that, the compatibility problem isn't really going to be solved. The portability problem isn't really going to be solved. But what about pace of changes? And, you know, well, yeah, what about it just working? Well, back to the window side of things. This is what Windows did. This is very, very similar to what these technologies are doing as well. Is history just repeating itself? Because when you're delivering these libraries in there, it's going to be a security nightmare. Maybe not in a practical sense, you know, because we are talking about putting these stuff in jails and, you know, some kind of isolation. But to be honest, when it comes to these, you know, these bits of isolation, it's a firewall. And I don't like the idea of trusting a firewall with my system security. I like a firewall being there when everything goes wrong and it's my last line of defense, but it's not my first line of defense. I want sensible engineering as my first line of defense. And therefore, I kind of forget about the isolation entirely and want to make sure that someone is taking care of the security of the libraries in my bundle and assume that at some point someone's going to escape the jail. There's no answer for that right now. There's no clean answer for that right now. Or there wasn't a week ago. Same with the maintenance side of things. Who's going to be patching these libraries in there? Who's going to be making sure that those libraries are moving forward? Who's going to be making sure that what I've installed is actually legally allowed to be on my machine? Who's going to make sure that the GPO is correctly being used and cited in there with my LGP stuff, for example? But it's okay. Store adventurers are still going to be happy because all these bundles are going to be using up more disk space. And then this is where my slide decor starts going out of Kilda because I was going to then talk about how we need to start conveying these responsibilities to the various maintainers of App Image and Flapback and Snappy and start talking about how are we going to get this message out there. And I was going to be talking about considering ABI changes and how do you rebuild bundles when ABI changes happen. And I was going to be talking about testing all of that. And I was going to be talking about the security maintenance issues. And I was going to ask the question about, you know, what are we going to do? And I was going to suggest a few things as well. But it all went horribly, horribly wrong because of the Open Suicide Conference. It's all changed. My question no longer is what are we going to do? It's actually what has been done already. And, well, I was going to change the title of this talk because, you know, from being a real bad skeptic of all these technologies, from what's been done, I now love App Image. I really love App Image. Because, well, OBS now builds App Images. Our build service now can take our packages that we have in there for Tomberwee to leap in for everything else, even our develop projects, and build App Images from that. So all those problems about the sort of security compliance, the security auditing, license tracking, dependency tracking, figuring out how to rebuild stuff, when to rebuild stuff, all these problems that we'd already solved in the distribution space, the App Image guys by working with us have now solved it in the App Image space as well. And, you know, we can host them on the build service too. So we have even, you know, changed the context of how you can deliver the software to the users. And we've managed, you know, the OBS team, we've managed to do all of this without impeding App Images' strengths and flexibilities of just being a nice, easy, lightweight of doing this thing in their hands of users. It just gets there. It's easy to deploy, you know, on single click run, it unpacks, it runs. This is really exciting. This totally changes my outlook on all these technologies because suddenly I don't have to be a skeptic anymore. I trust the build service, I trust the tools we have there, I trust the processes we have there. And it makes both sides of the equation more interesting to me. Just a few ideas that are kind of bouncing around my head since I heard about this two days ago. I want to see if we can do something like open SUSE a leap with App Images with user space applications being built from the tumbleweed sources. Because the build service can do that. We can build tumbleweed sources for leap. We can wrap that up in an App image. Last year I had this long one hour ranty talk about how I hated developer projects. Now we can kill them in the sense of killing the publishing of the, we still need them for building stuff for tumbleweed. But if we do this, users will be able to hopefully get the latest version of LibreOffice on their leap machine without having to change everything on their leap machine. That's really cool. We announced earlier this week open SUSE cubic, which is currently very much targeted for the Docker and Kubernetes world of a very, very stable atomic file system, atomic distribution with transactional updates on the base system. Well, now we have this. If we shove a graphical environment on there, maybe a nice tightly polished one, something like GNOME, and do all of the user applications with this, suddenly there's an option of an open SUSE Chrome OS style thing, a nice simple appliance for your grandma, which is something, well, when someone talked to me about that just last week, I said, yeah, good luck, have fun. It's crazy. It would take tons of people, tons of hours, and it's never going to happen. Now I can see one or two maintainers taking what we do in the build service and taking what the app image guys have done with us, and being able to knock that out in a couple of weeks. That is awesome. Admittedly, I'm not going to use it, but it's awesome if someone wants to take and use it. There was a talk today about the package hub and all the stuff we're going in package hub. Hopefully no one from SUSE sees this part. I have a bit of a mixed opinion of package hub because it's really exciting me that we're delivering open SUSE packages to enterprise customers, but at the same time, the way we're doing it is really similar to how tumbleweed back three years ago used to be, where SLEE is a really nice stable base and we keep on putting new versions of everything into package hub, rolling along the top of that. With old tumbleweed, we learned eventually that gets too big and too unwieldy and it starts getting a little bit breaky. It hasn't had that problem with package hub yet, but that's a risk if package hub just keeps on ballooning. For the user space applications, at least, the desktop applications, this app image stuff gives us an easy way of insulating that problem. We can actually define the scope a little bit better, start using the app images there, maybe start delivering app images to SLEE via package hub. Solve the same problem, does it in a slightly more sane way and uses these technologies to solve real problems that we would otherwise run headlong into. But I'm not finished because this is just app image. I was thinking, what's left? Well, snappy, flatback, sorry. With this now, you're not just part of the equation, you're behind. App image now has a better build story than you do. They've got a stronger compliance story than you do and they've got a more straightforward user experience in different distributions than you do because you still haven't got snappy in tumbleweed. Even if you ignore the technical stuff and you want to argue the details with me, they're kicking your ass when it comes to working with others because it's not just the fact that you're both here, it's the tone, it's the style, it's the way they've really got their hands dirty and messed around with the build service. Please be more like app image. It's been so fun working with them and seeing them change my mind and seeing us change their mind a little bit about a few things. We've got the tools, we've got the talent. Please work with us because I think we can do really exciting things in this space. But it's going to need to work in the kind of way that these guys already are working with us because it's just really exciting doing it that way. And it's not all good news or it's not all bad news for you. There are still problems across the entire thing. Dependency hell is still on the horizon. All of these tools still have very limited or no way of really solving this issue of what's coming from the base system. There's still assumptions being made there. We need to get together. These tools need to get together, the distribution needs to get together. And we need to discuss common standards and design common standards so everybody can go into this equation with simple ideas of what's going on there. Without that common understanding, application developers will just find frustration and users will just find crash stuff and distributions will just keep on doing what they're doing at the context they're doing. And then we'll actually miss out on cool opportunities to use this stuff in the way like I was talking about. And security, sandboxing, the app isolation side of things is a complete mess right now. Everybody has cool ideas and no one's finished implementing anything. The app image side of things, well, I kind of understand that because they've gone into this with the approach of use fire jail or whatever the hell you want. That's cool. But whatever the hell you want to be a little more defined than that. SnapD obviously has the app armor side of things. We love app armor, but your patches aren't upstream yet. And I know yet that's cool. Fixing that. But let's get that done. Let's get that in. Because if I had my way of doing my way, I'd like to see app armor kind of become the single way of doing this. I think it makes more sense. I understand that armor or more than I can understand bubble wrap and what they're trying to do there. I think that bubble wrap stuff is a little bit too desktop application oriented, which is cool for desktop applications, but there's scope here for using it in other wonderful ways and IOT and stuff. So let's see what we can do about getting app armor in there, polished up, do it all the way. And with that, I just want to kind of say thank you. I mean, this is becoming a really good lesson for me of two months ago, I was screaming that I thought the world was ending and this would never work out all right. And I've become a convert. I want to help make this better now. So let's just get onto it and does anybody have any questions? Anybody? Go on then. Obviously. I was hoping you did. So I'm curious about security and licensing compliance you mentioned. So you say you have an app image built on OBS. What does actually check that maybe that app image is built from a Git tree somewhere else that you're still compliant that the Git tree does not contain stuff in a different and compatible license and it doesn't contain a bundled copy of a library that has security vulnerabilities? You're technically right. I mean, yeah, if someone's taking an OBS project, a random home OBS project and building app images from that, there's no magic license solution there. But like we were talking about earlier this week, like my other talk from Friday, if you look at what we're doing with tumbleweed, you've got a pool of software and tumbleweed where we are keeping up with upstreams. Heck, we were just about to publish a new GCC7 version that we did while we were here. The pace of changing tumbleweed is fine. So you've got this huge pool that is audited, that is there, that is done, that's keeping up with upstreams. So if you're building your app image based on tumbleweed packages, you've just got to worry then about your tiny little diff, your little part there, which means all of the other dependencies you're feeding into that, they've been audited, they've been checked, they work in the sense of tumbleweed done. Completely agree and snaps have exactly the same thing with the build snap. I talked about it. Yeah, but you're doing exactly the same thing with Ubuntu and only Ubuntu. Well, it's still we're doing it and exactly the same as you do. Yeah, but with the build service way, you can do it with everything I just said with tumbleweed and you can do it with Ubuntu and you can do it with that. And we're going to get there. Yeah, but you can do that your way or you can just copy what they didn't do with the build service. No, because the problems are not like that. The problems are technical below the stack. And it's not a problem of building, you can build it, you just can't run it yet. Once we get to the running, you can build it on top of anything and run it. And that's fine. So we have the same goal here. There's no disagreement. Okay, I'll believe it when I see it. Any more comments? Questions? Cool. So there's two issues actually don't really see solved yet. So first of all, you still end up with a certain amount of size, right? Yeah, because you need redundancy. And of course, one thing that always concerns me is the laziness of developers because they start to rely on compatibility or outdated libraries. So especially if you have something large like say, framework X, which depends on a lot of libraries, then you have a large footprint, which becomes outdated now. I don't want that to happen with stuff like an open SSL, for example. What do you think about that? It's a perfectly fair point. It's a perfectly fair problem. And the answers are exactly... I don't see that problem any different in the containerized application context than I do in the distribution one actually. We suffer that same pain. I'm kind of hoping that getting distributions around the tape, distributions and projects around the table and the kind of common framework idea that we're sort of coalescing might give a little bit of a push to help drive that problem away a little bit. Just like it did with like KDE and Leapneeding and LTS release and KDE, making a commitment to that. That's how they're going to do that and solve that problem for us. Yeah, it's a very real problem. We never really solved it on the dependence, on the distribution side of things on our own. This actually makes it potentially a little bit more... It gives us a second chance to do it right this time, hopefully. There's a question there. Yeah, if I will be developer of... I still feel that Android kick our ass because it's so much easier to develop for Android. And they have the same problem. They have, let's call it distributors, which is actually hardware vendors that provide their phones. And they have some stable base, which is versioned Android. And yeah, of course they have problems with security and such stuff, but I still feel that from developer point of view, it's much easier to develop for Android than any app images. Why? What's the difference there? Because ultimately, an Android app is just a bundle of a whole bunch of dependencies. It might be easy... How would that be easier, both... The question I've got to get to you then is, how is that easier, both getting the application app in the first place and then how is it easier maintaining it? Yeah, one part that's easier, they have a common base. They can easily... It's back... Somehow back what compatible. And you can say, I support this version of Android or newer. And it keeps working. But yeah, I think finding such base in Linux is, as you mentioned, in the standard base is basically missing. And there's a lot... Much more stuff in Android that's common than in Linux, which is... Yeah, that's because you have to bundle much more in Linux than on Android. Yeah, I can't argue with that part. That's why I think we need this. Because I think the problem with the Linux standard base is the scope was too broad. It always was, trying to define everything at every level of every bit of the stack. The nice thing with these technologies is they push that problem down to a certain amount. I mean, Snap tries to push it down to the kernel. App image tries to push it down quite low as well. Flatpack keeps on changing its mind because then the runtimes move the line all over the place. But at least the line tries to get defined further down the stack. So if we just figure out where that line is and define a common base below there, common... Let's say if it line gets drawn near the kernel, basically common standard configs of the kernel. What config is likely to be there? What is an LTS... Are we going to follow the upstream LTS kernel and move along at that kind of pace? Just so you can tag that with a version and say, okay, I'm supporting container-based version blah, and then you get that solution. You get that situation you have with Android. I think we need that in the Linux containerized app side of things as well, totally. Yeah, and another part that sucks for me as developer is that we don't have a common place where to distribute such stuff. They have the Android Store, how they call it, and you just upload it there and you are fine. Every Android user, even if it's from different distribution already having it. And we still miss it in Linux for many years. Yeah, I... Given the nature of the open-source word, I don't think you will have a single place. I mean, you might have a dominant place because someone's going to win the popularity war, but whether it will be a single one, I mean, even Snap has multiple store options already. OBS is now another one for App Image. There will be fragmentation there. I'd love to see a way of pulling it all together somehow because I think that will help in the long run. But that's a problem for the future, I think. Yeah, something we have to worry about, definitely. Yeah, so what I want to say is that we are still behind Android. We're still behind Android. No doubt about that. From developer point of view. And I think user point of view is also behind Android. And that's something that I... That's a drum I think I'll keep on beating because I think if we can get a bit more commonality between these different tool sets, it makes that a little bit easier to catch up with where Android are, makes it a little bit easier also to be honest, the kind of motivation behind this. I can see how this makes distribution's lives easier, less stuff, less to maintain. And we're all lazy. So I can see how we can actually use this to change everything in a rather nice way. But it's only going to work if we kind of find ways of applying focus to that and coming up with some common standards and then seeing how the technologies actually shake out in the long run. Cool. Any more questions? No? Okay then. Thank you very much.
 Wow. Thanks everybody for coming. It's like the last, well, second to last talk for me.  After this, we've got the board meeting in the main hall and then we're done. So I'm  Richard. I'm here to talk about, well, dinosaurs or as most other people know them, containerized  applications and how we need to deal with them. Now they're out there. Now users are  really using them. Now we're starting to see the problems with them in the real world.  And this is a variation on talk I did at FOSSTEM earlier in the year. So I'm going to start  out rehashing a lot of that. So if you've seen my talk, I'll try and run through it  a little bit quicker. And if you haven't and you're more interested, you can go back and  watch that. And then I'll talk about some of the new and exciting stuff and why I had  to rewrite my slides twice during this conference. But really, when I started looking at these  technologies, looking at Snappy, looking at Flatback and looking at App Image, it struck  me that I'd seen all of this before. And in fact, where I first sort of saw similarities  was actually back in sort of the Windows architecture, the original Windows application  architecture and how Windows deals with loading up libraries and dependencies in the Windows  world. And I mean, to start off with, look at it from another perspective, Linux has  a lot of traditional Linux, traditional Linux packaging has an awful lot in common with  Windows 3.1. It's a similar world. There is no ABI backwards compatibility. Things are  constantly changing. Things are constantly evolved. There's one great big file system  where everything is dumped into Windows, Windows system or C Windows, global identifiers  for all the symbols that everything starts clashing all over the place. It's an absolute  maintenance nightmare. And it's where the term DLL comes from. Because ultimately, all  developers want to do is have a nice, well, simple environment to work with. And in the  Windows 3.1 world, they had to development and test every single possible DLL combination  that might be seen in the wild possibly anywhere. And then every time they had a patch, test  that app patch in every single combination everywhere. And then when there was a dependency  or library patch, test everything in every single combination everywhere. And they would  do that and they would try and Windows was being used everywhere and then they'd cry  because it would all break anyway and it would all go horribly, horribly wrong. And Microsoft  thought they could fix this. And they tried very, very well to do that. And they were  somewhat successful. Now, Windows 2000 introduced this concept of side-by-side assembly, which  is basically containerization or application isolation for the Windows world. Having a  separate memory space for every single application and all of its DLLs, loading up those DLLs  privately from a folder in the file system, having Windows files protection, doing disk  isolation of system DLLs, and having these fancy tools to audit all of that and migrate  those legacy applications and deal with those problems. And you ended up with this wonderful  situation, if you're a Windows user, where your Windows 2000 or later could run an application  for Win32 or even for POSIX or even OS2, or using these fancy little runtimes that were  packaged up in that Windows environment. So, problem solved, right? Well, no, of course,  it all went horribly wrong. And not just because it was Microsoft doing it. There were very,  very real sort of social and practical problems that evolved over time that we all saw. It  was a security nightmare. All of these libraries, all of these dependencies end up lurking  in countless folders, all being maintained to various degrees by the developers that  put them there, all then becoming lovely security vulnerability, security gaps, attack services  for things like WannaCry and other malware to go and abuse and misuse when certain applications  are loaded in memory and loading up the bad DLLs in question. It's also maintenance nightmare  as a user. How do you then update that application on the user's machine? Anyone using Windows,  how many application uploaders do you have in your system tray? I mean, everyone builds  another system update. And it just doesn't scale. It ends up, especially in the open  source world, being a bit of a legal nightmare, one of the biggest issues of getting open  source software in Windows is actually this problem. It's actually figuring out, okay,  can we put this open source DLL in this container with everything else we need to put in there  to get the thing working? And we, in the case of the Windows world, with these DLL-encompassed  applications, the developer is the distributor. They have to worry about those legal issues  and those legal concerns. But there is one bunch of people happy, hard disk vendors,  because everybody's using up more disk space. People need more bytes. It's not terrible.  And this was, yeah, like I said, since 2000. Meanwhile in Linux land, we were looking on  smugly because we'd already solved all these problems. Sort of. And the way we solved these  problems was with the traditional Linux distribution. And primarily, the things that distributions  brought to the table and still bring to the table today, isn't the technical stuff per  say. We all solve the technical problems in our own way. We all have our different package  managers and our different philosophies on how you should do this engineering and packaging  stuff. But the universal thing we all really bring to the table is we care about the security  of the operating system and its applications in the context of a user looking after it.  We're maintaining this stuff. We're auditing this stuff. We're constantly monitoring CVEs,  pushing out these updates. And especially with the open source side of things, a major  security vulnerability needs to be handled in a very particular way. You're going to  have embargoed security mailing lists. You need to have trusted people on there. So you  need to have the right people there. You need to have the right relationships there to get  on those embargoed lists. And distributions play that key role of being there and able  to get those fixes out before issues start hitting your code tree. And like I said, maintaining  it, packaging those updates, keeping them updated, dealing with upstreams, helping work  with upstreams with that. And lawyers auditing all of this stuff, checking it's compatible,  making sure that the license is being chosen are sane and consistent with each other. So  when I talk about this distribution stuff, you know, lots of people and it's kind of  spawned by these new technologies or this kind of resurgence of the bundled application  side of things of, you know, shared libraries are a problem. We're trying to solve the shared  library problem. Dependencies are a pain in the ass. You know, I don't want to worry about  the dependencies. I just want to worry about my app. Shared libraries do solve real problems.  It's not just a case of being more efficient on disk, although that is a benefit as well.  But having fewer libraries to worry about, having fewer dependencies to worry about,  or having fewer repeating copies of the same dependency to worry about is a very beneficial  thing. You know, when something goes wrong, when something is insecure, you have fewer  copies of that thing to worry about. You have fewer places to patch. You have less manpower  required to patch it. Less double work sending multiple copies of your vulnerable lib SSL  out there or your vulnerable SAMBA libraries out there. And that makes it easier to then  review it both as a user and as a distributor or as a developer who is distributing and  ensuring that legal and security compliance, you know, this is something you can trust  and rely on today, tomorrow. And also when it stops being maintained, it's still doing  it in a way that's, you know, at least was saying at the time of stopping it. So from  the open source sort of distribution side of things, mission accomplished, right? Well,  no, like I already said, the open source distribution way of doing things was very similar to actually  how Windows did it. You know, we still had these problems. We still had the issues of  compatibility of making sure all these bits and pieces work together, of, you know, portability.  How does an application built in one context work elsewhere? And how do you handle this  issue of keeping something, keeping this software being delivered and just working and handling  the fact that the open source world and therefore everything you're distributing is moving at  a constantly changing pace of change. But we're not Windows. And when you're distributing  in the open source world, there's different factors at play. So thinking about the compatibility  issue, we all end up doing different distributions. We all end up having different libraries and  different applications. Different applications require different libraries so the problem  becomes exponentially complicated. And application developers don't want to worry about all that  mess. They just want to deliver their software in the hands of users. I get it. They don't  want to have to worry about which choice of dependencies did certain distributions pick.  But most of the time they don't have to. In reality, distributions have their own maintainers.  We have our own communities all here who care about this stuff and this part of it. And  are the sort of the second tier making sure that the application gets in the hands of  users. And you care about it in the same way as the upstream maintainers care about it.  So very rarely does it really become a problem because most times it is really being done  by the distro maintainers who care about this. And that's what you all do and that's what  we've been doing for years and we're bloody good at it. As you can just see in Tumbleweed  where we repeatedly ship stuff as fast as the upstream shipping it. But it is importable.  It's open SUSE, bi open SUSE, for open SUSE. And an application developer wants to make  sure that their software runs in a many different context as possible and as many different  distributions as possible. And they don't want to learn a whole bunch of different build  tools and they don't want to learn 20 different ways of doing things and they don't want  to retest it 20 different places. But again, distribution communities often take care of  that problem for the application developers anyway. And then pace of change. Every distribution  does everything at a different pace. Heck, if you're open SUSE we do it at two different  places. We'd leap, we'd do it regularly every year with major versions every few years and  we do Tumbleweed where we just go nuts and as fast as every upstream wants us to go.  But in the traditional old fashioned way of doing things, the regular release process  gets in the way of that application delivery desire. In the traditional model you can't  necessarily run the latest version of software on your stable open SUSE or your Debian system.  Debian in particular is the perfect example because they freeze so well and so hard and  so solidly that that becomes even harder and they're proud of that. Leap we purposefully  design it in a way to try and bend the issue around the edges and deliver faster stuff  when we can, how we can. We have the build servers, we have OpenCure to help with that.  But it's still a very real problem. Sometimes we just have to say no because the technology  can't do it. But that problem, that balance, how do we deliver this software? That is again  something that distributions do currently take care of. So how much of a problem is  it really? Doesn't matter. App image, flat packs, and App here to solve all the problems  left anyway. And they exist to solve that issue. They exist to try and move these problems  out of the hand of the distributions or reduce the need for the distributions to do this.  So application developers can get that software in the hands of users at the pace the application  developers want to be. And they do so by providing a bundle containing the app and the libraries,  all the dependencies they need in then some kind of container or context or bubble or  whatever. I'm going to keep on saying container although technically speaking, you know, that's  open for interpretation. And the big promises of all these technologies, despite details  around the edges of how they do things, is to solve all these compatibility and portability  issues. It's only going to have the compatible libraries in the bundle. So you don't have  to worry about anything from the distribution. You just put your application there, everything  you need, you know. It will be portable. It will work everywhere because all the dependencies  will be solved in there. You'll never have to worry about what does a distribution ship.  And of course that means you can ship it at the pace you want whenever, however, don't  have to worry about what the distribution is doing. And it's just going to work. And  it's going to be wonderful. That's the promise. And then you have nice architecture diagrams  like the snappy one here, or you have really stupidly complicated architecture diagrams  like the flatback one, where you have this kind of model of, you know, just ignoring  the operating system down the bottom pretty much. It's just there. And then there is some  layer on top of that, be it the frameworks or the run times, which, you know, provide  this sort of layer of dependencies, which are an awful lot like the dependencies the  distributions are currently doing anyway. And then the bundle itself contains the library,  the code, the application, everything's fine. But it doesn't work. In practice, it doesn't  work. Most of the time it does, but there are still some very, very real issues there.  And in fact, the biggest problem that comes around in reality when you start using these  in production is this issue of, you know, compatibility and portability. The myth is  not true. Because at some point, be it app image, flatback or snappy, there are some  assumptions made about the stuff below the system. And we talked about this. Like, you  know, in the case of snaps, you know, that is still the kernel. You know, everything  above the kernel is assumed to be delivered by some snap somehow. But no kernel is equal.  You know, every distribution has a different kernel with different conflicts. There's  still problems there that get introduced by different, you know, by different kernel  arrangements. We see this in OpenSUSE most actually with Steam, which isn't using one  of these technologies, but basically uses the same approach of having a containerized  Steam, you know, runtime that gets put in your user area and run that way. And everything  was fine on OpenSUSE with Steam for the longest time. It just worked. It did its job. We could  move everything in OpenSUSE and everything was fine. And then we changed our Glib C and  it exploded spectacularly because their Steam runtime was built with an older Glib C and  therefore nothing would run anymore. All those libraries would go horribly, horribly  wrong until we started scripting around injecting our Glib C into there and rebuilding everything.  And it was a complete mess. And we've seen this issue with flatpacks. We've seen this  issue with some of the experience with snaps. We've seen the issue sometimes with app images.  But one of the nice things and one of the reasons why I've always come into this liking  app image is at least app image documents this problem. You know, it's stated there that  it isn't trying to be a universal portable application solution. You're going to have  to gather the binaries for the dependencies for the distributions you're targeting. You're  not going to magically solve this problem everywhere. But that means that if you're  using these technologies, you still have to worry about all of the compatible dependencies  which might not be provided by any distribution you might want to run it on. That's a lot  of stuff to worry about. Heck, that's the stuff that we all do at OpenSUSE all the damn  time and it takes all of us to do it. If you don't get your head around that, your users  need to expect crashes. So is it hopeful? Is it hopeless? You know, well, you know, you  talked to flat pack people like this, oh, no, we've solved it. You know, we've got these  run times or on the snap side of things, you know, we've got these base snaps. Well, those  base snaps, those run times just end up being some second curated middle distro. It's middle  where for the containerized world. Cool, fine. But you haven't solved the problem. You've  just moved it into a different context. It's still another distribution. You're still having  to have distribution engineers worry about this stuff and curate it and secure it and  patch it and maintain it. Maybe it isn't a real solution. Maybe a real solution is actually  figuring out a way of commonly agreeing between us, the distributions as a platform, the application  run times as a delivery mechanism and hopefully even the developers on common, I use the word  standards here, but let's say common agreements of, you know, what can you expect from your  base system? What can application developers expect from their run times? So people can  go into this and we won't just have random crashes when you install your app image or  your snap on an open Susan machine and it doesn't deliver the kernel the way it's expected  or the library is the way it's expected. And until we do that, the compatibility problem  isn't really going to be solved. The portability problem isn't really going to be solved. But  what about pace of changes? And, you know, well, yeah, what about it just working? Well,  back to the window side of things. This is what Windows did. This is very, very similar  to what these technologies are doing as well. Is history just repeating itself? Because  when you're delivering these libraries in there, it's going to be a security nightmare. Maybe  not in a practical sense, you know, because we are talking about putting these stuff in  jails and, you know, some kind of isolation. But to be honest, when it comes to these, you  know, these bits of isolation, it's a firewall. And I don't like the idea of trusting a firewall  with my system security. I like a firewall being there when everything goes wrong and  it's my last line of defense, but it's not my first line of defense. I want sensible  engineering as my first line of defense. And therefore, I kind of forget about the isolation  entirely and want to make sure that someone is taking care of the security of the libraries  in my bundle and assume that at some point someone's going to escape the jail. There's  no answer for that right now. There's no clean answer for that right now. Or there wasn't  a week ago. Same with the maintenance side of things. Who's going to be patching these  libraries in there? Who's going to be making sure that those libraries are moving forward?  Who's going to be making sure that what I've installed is actually legally allowed to be  on my machine? Who's going to make sure that the GPO is correctly being used and cited  in there with my LGP stuff, for example? But it's okay. Store adventurers are still  going to be happy because all these bundles are going to be using up more disk space.  And then this is where my slide decor starts going out of Kilda because I was going to  then talk about how we need to start conveying these responsibilities to the various maintainers  of App Image and Flapback and Snappy and start talking about how are we going to get this  message out there. And I was going to be talking about considering ABI changes and how do you  rebuild bundles when ABI changes happen. And I was going to be talking about testing all  of that. And I was going to be talking about the security maintenance issues. And I was  going to ask the question about, you know, what are we going to do? And I was going to  suggest a few things as well. But it all went horribly, horribly wrong because of the Open  Suicide Conference. It's all changed. My question no longer is what are we going to  do? It's actually what has been done already. And, well, I was going to change the title  of this talk because, you know, from being a real bad skeptic of all these technologies,  from what's been done, I now love App Image. I really love App Image. Because, well, OBS  now builds App Images. Our build service now can take our packages that we have in there  for Tomberwee to leap in for everything else, even our develop projects, and build App Images  from that. So all those problems about the sort of security compliance, the security  auditing, license tracking, dependency tracking, figuring out how to rebuild stuff, when to  rebuild stuff, all these problems that we'd already solved in the distribution space,  the App Image guys by working with us have now solved it in the App Image space as well.  And, you know, we can host them on the build service too. So we have even, you know, changed  the context of how you can deliver the software to the users. And we've managed, you know,  the OBS team, we've managed to do all of this without impeding App Images' strengths and  flexibilities of just being a nice, easy, lightweight of doing this thing in their hands  of users. It just gets there. It's easy to deploy, you know, on single click run, it  unpacks, it runs. This is really exciting. This totally changes my outlook on all these  technologies because suddenly I don't have to be a skeptic anymore. I trust the build  service, I trust the tools we have there, I trust the processes we have there. And it  makes both sides of the equation more interesting to me. Just a few ideas that are kind of bouncing  around my head since I heard about this two days ago. I want to see if we can do something  like open SUSE a leap with App Images with user space applications being built from the  tumbleweed sources. Because the build service can do that. We can build tumbleweed sources  for leap. We can wrap that up in an App image. Last year I had this long one hour ranty talk  about how I hated developer projects. Now we can kill them in the sense of killing the  publishing of the, we still need them for building stuff for tumbleweed. But if we do  this, users will be able to hopefully get the latest version of LibreOffice on their  leap machine without having to change everything on their leap machine. That's really cool.  We announced earlier this week open SUSE cubic, which is currently very much targeted for  the Docker and Kubernetes world of a very, very stable atomic file system, atomic distribution  with transactional updates on the base system. Well, now we have this. If we shove a graphical  environment on there, maybe a nice tightly polished one, something like GNOME, and do  all of the user applications with this, suddenly there's an option of an open SUSE Chrome OS  style thing, a nice simple appliance for your grandma, which is something, well, when someone  talked to me about that just last week, I said, yeah, good luck, have fun. It's crazy.  It would take tons of people, tons of hours, and it's never going to happen. Now I can  see one or two maintainers taking what we do in the build service and taking what the  app image guys have done with us, and being able to knock that out in a couple of weeks.  That is awesome. Admittedly, I'm not going to use it, but it's awesome if someone wants  to take and use it. There was a talk today about the package hub and all the stuff we're  going in package hub. Hopefully no one from SUSE sees this part. I have a bit of a mixed  opinion of package hub because it's really exciting me that we're delivering open SUSE  packages to enterprise customers, but at the same time, the way we're doing it is really  similar to how tumbleweed back three years ago used to be, where SLEE is a really nice  stable base and we keep on putting new versions of everything into package hub, rolling along  the top of that. With old tumbleweed, we learned eventually that gets too big and too unwieldy  and it starts getting a little bit breaky. It hasn't had that problem with package hub  yet, but that's a risk if package hub just keeps on ballooning. For the user space applications,  at least, the desktop applications, this app image stuff gives us an easy way of insulating  that problem. We can actually define the scope a little bit better, start using the app images  there, maybe start delivering app images to SLEE via package hub. Solve the same problem,  does it in a slightly more sane way and uses these technologies to solve real problems  that we would otherwise run headlong into. But I'm not finished because this is just  app image. I was thinking, what's left? Well, snappy, flatback, sorry. With this now, you're  not just part of the equation, you're behind. App image now has a better build story than  you do. They've got a stronger compliance story than you do and they've got a more straightforward  user experience in different distributions than you do because you still haven't got  snappy in tumbleweed. Even if you ignore the technical stuff and you want to argue the  details with me, they're kicking your ass when it comes to working with others because  it's not just the fact that you're both here, it's the tone, it's the style, it's the way  they've really got their hands dirty and messed around with the build service. Please be more  like app image. It's been so fun working with them and seeing them change my mind and seeing  us change their mind a little bit about a few things. We've got the tools, we've got  the talent. Please work with us because I think we can do really exciting things in this  space. But it's going to need to work in the kind of way that these guys already are working  with us because it's just really exciting doing it that way. And it's not all good news  or it's not all bad news for you. There are still problems across the entire thing. Dependency  hell is still on the horizon. All of these tools still have very limited or no way of  really solving this issue of what's coming from the base system. There's still assumptions  being made there. We need to get together. These tools need to get together, the distribution  needs to get together. And we need to discuss common standards and design common standards  so everybody can go into this equation with simple ideas of what's going on there. Without  that common understanding, application developers will just find frustration and users will  just find crash stuff and distributions will just keep on doing what they're doing at  the context they're doing. And then we'll actually miss out on cool opportunities to  use this stuff in the way like I was talking about. And security, sandboxing, the app isolation  side of things is a complete mess right now. Everybody has cool ideas and no one's finished  implementing anything. The app image side of things, well, I kind of understand that  because they've gone into this with the approach of use fire jail or whatever the hell you  want. That's cool. But whatever the hell you want to be a little more defined than  that. SnapD obviously has the app armor side of things. We love app armor, but your patches  aren't upstream yet. And I know yet that's cool. Fixing that. But let's get that done.  Let's get that in. Because if I had my way of doing my way, I'd like to see app armor  kind of become the single way of doing this. I think it makes more sense. I understand  that armor or more than I can understand bubble wrap and what they're trying to do there.  I think that bubble wrap stuff is a little bit too desktop application oriented, which  is cool for desktop applications, but there's scope here for using it in other wonderful  ways and IOT and stuff. So let's see what we can do about getting app armor in there,  polished up, do it all the way. And with that, I just want to kind of say thank you. I mean,  this is becoming a really good lesson for me of two months ago, I was screaming that  I thought the world was ending and this would never work out all right. And I've become  a convert. I want to help make this better now. So let's just get onto it and does anybody  have any questions? Anybody? Go on then. Obviously. I was hoping you did.  So I'm curious about security and licensing compliance you mentioned. So you say you have  an app image built on OBS. What does actually check that maybe that app image is built from  a Git tree somewhere else that you're still compliant that the Git tree does not contain  stuff in a different and compatible license and it doesn't contain a bundled copy of a  library that has security vulnerabilities? You're technically right. I mean, yeah, if  someone's taking an OBS project, a random home OBS project and building app images from  that, there's no magic license solution there. But like we were talking about earlier this  week, like my other talk from Friday, if you look at what we're doing with tumbleweed,  you've got a pool of software and tumbleweed where we are keeping up with upstreams. Heck,  we were just about to publish a new GCC7 version that we did while we were here. The pace of  changing tumbleweed is fine. So you've got this huge pool that is audited, that is there,  that is done, that's keeping up with upstreams. So if you're building your app image based  on tumbleweed packages, you've just got to worry then about your tiny little diff, your  little part there, which means all of the other dependencies you're feeding into that,  they've been audited, they've been checked, they work in the sense of tumbleweed done.  Completely agree and snaps have exactly the same thing with the build snap. I talked about  it. Yeah, but you're doing exactly the same thing with Ubuntu and only Ubuntu. Well,  it's still we're doing it and exactly the same as you do. Yeah, but with the build service  way, you can do it with everything I just said with tumbleweed and you can do it with  Ubuntu and you can do it with that. And we're going to get there. Yeah, but you can do that  your way or you can just copy what they didn't do with the build service. No, because the  problems are not like that. The problems are technical below the stack. And it's not a  problem of building, you can build it, you just can't run it yet. Once we get to the  running, you can build it on top of anything and run it. And that's fine. So we have the  same goal here. There's no disagreement. Okay, I'll believe it when I see it.  Any more comments? Questions? Cool. So there's two issues actually don't really see solved  yet. So first of all, you still end up with a certain amount of size, right? Yeah, because  you need redundancy. And of course, one thing that always concerns me is the laziness of  developers because they start to rely on compatibility or outdated libraries. So especially if you  have something large like say, framework X, which depends on a lot of libraries, then  you have a large footprint, which becomes outdated now. I don't want that to happen with  stuff like an open SSL, for example. What do you think about that?  It's a perfectly fair point. It's a perfectly fair problem. And the answers are exactly ... I don't see that problem any different in the containerized application context than  I do in the distribution one actually. We suffer that same pain. I'm kind of hoping  that getting distributions around the tape, distributions and projects around the table  and the kind of common framework idea that we're sort of coalescing might give a little  bit of a push to help drive that problem away a little bit. Just like it did with like  KDE and Leapneeding and LTS release and KDE, making a commitment to that. That's how they're  going to do that and solve that problem for us. Yeah, it's a very real problem. We never  really solved it on the dependence, on the distribution side of things on our own. This  actually makes it potentially a little bit more... It gives us a second chance to do  it right this time, hopefully. There's a question there.  Yeah, if I will be developer of... I still feel that Android kick our ass because it's  so much easier to develop for Android. And they have the same problem. They have, let's  call it distributors, which is actually hardware vendors that provide their phones. And they  have some stable base, which is versioned Android. And yeah, of course they have problems  with security and such stuff, but I still feel that from developer point of view, it's  much easier to develop for Android than any app images.  Why? What's the difference there? Because ultimately, an Android app is just a bundle  of a whole bunch of dependencies. It might be easy... How would that be easier, both ... The question I've got to get to you then is, how is that easier, both getting the application  app in the first place and then how is it easier maintaining it?  Yeah, one part that's easier, they have a common base. They can easily... It's back ... Somehow back what compatible. And you can say, I support this version of Android  or newer. And it keeps working. But yeah, I think finding such base in Linux is, as you  mentioned, in the standard base is basically missing. And there's a lot... Much more stuff  in Android that's common than in Linux, which is... Yeah, that's because you have to bundle  much more in Linux than on Android.  Yeah, I can't argue with that part. That's why I think we need this. Because I think  the problem with the Linux standard base is the scope was too broad. It always was, trying  to define everything at every level of every bit of the stack. The nice thing with these  technologies is they push that problem down to a certain amount. I mean, Snap tries to  push it down to the kernel. App image tries to push it down quite low as well. Flatpack  keeps on changing its mind because then the runtimes move the line all over the place.  But at least the line tries to get defined further down the stack. So if we just figure  out where that line is and define a common base below there, common... Let's say if  it line gets drawn near the kernel, basically common standard configs of the kernel. What  config is likely to be there? What is an LTS... Are we going to follow the upstream  LTS kernel and move along at that kind of pace? Just so you can tag that with a version  and say, okay, I'm supporting container-based version blah, and then you get that solution.  You get that situation you have with Android. I think we need that in the Linux containerized  app side of things as well, totally.  Yeah, and another part that sucks for me as developer is that we don't have a common  place where to distribute such stuff. They have the Android Store, how they call it,  and you just upload it there and you are fine. Every Android user, even if it's from different  distribution already having it. And we still miss it in Linux for many years.  Yeah, I... Given the nature of the open-source word, I don't think you will have a single  place. I mean, you might have a dominant place because someone's going to win the popularity  war, but whether it will be a single one, I mean, even Snap has multiple store options  already. OBS is now another one for App Image. There will be fragmentation there. I'd love  to see a way of pulling it all together somehow because I think that will help in the long  run. But that's a problem for the future, I think. Yeah, something we have to worry about,  definitely.  Yeah, so what I want to say is that we are still behind Android.  We're still behind Android. No doubt about that.  From developer point of view. And I think user point of view is also behind Android.  And that's something that I... That's a drum I think I'll keep on beating because I think  if we can get a bit more commonality between these different tool sets, it makes that a  little bit easier to catch up with where Android are, makes it a little bit easier also to  be honest, the kind of motivation behind this. I can see how this makes distribution's  lives easier, less stuff, less to maintain. And we're all lazy.  So I can see how we can actually use this to change everything in a rather nice way.  But it's only going to work if we kind of find ways of applying focus to that and coming  up with some common standards and then seeing how the technologies actually shake out in  the long run.  Cool.  Any more questions? No? Okay then. Thank you very much.
openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_0_71.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_72_515.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_516_1487.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_1488_1703.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_1704_1775.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_1776_1907.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_1908_2459.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_2460_5207.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_5208_8195.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_8196_8447.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_8448_11219.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_11220_13235.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_13236_16463.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_16464_18287.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_18288_18383.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_18384_21683.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_21684_30167.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_30168_30311.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_30312_30995.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_30996_31127.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_31128_31175.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_31176_31247.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_31248_31511.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_31512_32039.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_32040_38615.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_38616_38855.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_38856_40667.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_40668_43583.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_43584_49127.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_49128_49199.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_49200_49271.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_49272_50735.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_50736_51707.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_51708_51791.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_51792_51839.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_51840_52403.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_52404_52991.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_52992_53099.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_53100_57467.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_57468_57647.png openSUSE_Conference_2017_openSUSE___Jurrassic_Park_BrWB2OZ9h2Y_57648_57769.png
Wow. Thanks everybody for coming. It's like the last, well, second to last talk for me. After this, we've got the board meeting in the main hall and then we're done. So I'm Richard. I'm here to talk about, well, dinosaurs or as most other people know them, containerized applications and how we need to deal with them. Now they're out there. Now users are really using them. Now we're starting to see the problems with them in the real world. And this is a variation on talk I did at FOSSTEM earlier in the year. So I'm going to start out rehashing a lot of that. So if you've seen my talk, I'll try and run through it a little bit quicker. And if you haven't and you're more interested, you can go back and watch that. And then I'll talk about some of the new and exciting stuff and why I had to rewrite my slides twice during this conference. But really, when I started looking at these technologies, looking at Snappy, looking at Flatback and looking at App Image, it struck me that I'd seen all of this before. And in fact, where I first sort of saw similarities was actually back in sort of the Windows architecture, the original Windows application architecture and how Windows deals with loading up libraries and dependencies in the Windows world. And I mean, to start off with, look at it from another perspective, Linux has a lot of traditional Linux, traditional Linux packaging has an awful lot in common with Windows 3.1. It's a similar world. There is no ABI backwards compatibility. Things are constantly changing. Things are constantly evolved. There's one great big file system where everything is dumped into Windows, Windows system or C Windows, global identifiers for all the symbols that everything starts clashing all over the place. It's an absolute maintenance nightmare. And it's where the term DLL comes from. Because ultimately, all developers want to do is have a nice, well, simple environment to work with. And in the Windows 3.1 world, they had to development and test every single possible DLL combination that might be seen in the wild possibly anywhere. And then every time they had a patch, test that app patch in every single combination everywhere. And then when there was a dependency or library patch, test everything in every single combination everywhere. And they would do that and they would try and Windows was being used everywhere and then they'd cry because it would all break anyway and it would all go horribly, horribly wrong. And Microsoft thought they could fix this. And they tried very, very well to do that. And they were somewhat successful. Now, Windows 2000 introduced this concept of side-by-side assembly, which is basically containerization or application isolation for the Windows world. Having a separate memory space for every single application and all of its DLLs, loading up those DLLs privately from a folder in the file system, having Windows files protection, doing disk isolation of system DLLs, and having these fancy tools to audit all of that and migrate those legacy applications and deal with those problems. And you ended up with this wonderful situation, if you're a Windows user, where your Windows 2000 or later could run an application for Win32 or even for POSIX or even OS2, or using these fancy little runtimes that were packaged up in that Windows environment. So, problem solved, right? Well, no, of course, it all went horribly wrong. And not just because it was Microsoft doing it. There were very, very real sort of social and practical problems that evolved over time that we all saw. It was a security nightmare. All of these libraries, all of these dependencies end up lurking in countless folders, all being maintained to various degrees by the developers that put them there, all then becoming lovely security vulnerability, security gaps, attack services for things like WannaCry and other malware to go and abuse and misuse when certain applications are loaded in memory and loading up the bad DLLs in question. It's also maintenance nightmare as a user. How do you then update that application on the user's machine? Anyone using Windows, how many application uploaders do you have in your system tray? I mean, everyone builds another system update. And it just doesn't scale. It ends up, especially in the open source world, being a bit of a legal nightmare, one of the biggest issues of getting open source software in Windows is actually this problem. It's actually figuring out, okay, can we put this open source DLL in this container with everything else we need to put in there to get the thing working? And we, in the case of the Windows world, with these DLL-encompassed applications, the developer is the distributor. They have to worry about those legal issues and those legal concerns. But there is one bunch of people happy, hard disk vendors, because everybody's using up more disk space. People need more bytes. It's not terrible. And this was, yeah, like I said, since 2000. Meanwhile in Linux land, we were looking on smugly because we'd already solved all these problems. Sort of. And the way we solved these problems was with the traditional Linux distribution. And primarily, the things that distributions brought to the table and still bring to the table today, isn't the technical stuff per say. We all solve the technical problems in our own way. We all have our different package managers and our different philosophies on how you should do this engineering and packaging stuff. But the universal thing we all really bring to the table is we care about the security of the operating system and its applications in the context of a user looking after it. We're maintaining this stuff. We're auditing this stuff. We're constantly monitoring CVEs, pushing out these updates. And especially with the open source side of things, a major security vulnerability needs to be handled in a very particular way. You're going to have embargoed security mailing lists. You need to have trusted people on there. So you need to have the right people there. You need to have the right relationships there to get on those embargoed lists. And distributions play that key role of being there and able to get those fixes out before issues start hitting your code tree. And like I said, maintaining it, packaging those updates, keeping them updated, dealing with upstreams, helping work with upstreams with that. And lawyers auditing all of this stuff, checking it's compatible, making sure that the license is being chosen are sane and consistent with each other. So when I talk about this distribution stuff, you know, lots of people and it's kind of spawned by these new technologies or this kind of resurgence of the bundled application side of things of, you know, shared libraries are a problem. We're trying to solve the shared library problem. Dependencies are a pain in the ass. You know, I don't want to worry about the dependencies. I just want to worry about my app. Shared libraries do solve real problems. It's not just a case of being more efficient on disk, although that is a benefit as well. But having fewer libraries to worry about, having fewer dependencies to worry about, or having fewer repeating copies of the same dependency to worry about is a very beneficial thing. You know, when something goes wrong, when something is insecure, you have fewer copies of that thing to worry about. You have fewer places to patch. You have less manpower required to patch it. Less double work sending multiple copies of your vulnerable lib SSL out there or your vulnerable SAMBA libraries out there. And that makes it easier to then review it both as a user and as a distributor or as a developer who is distributing and ensuring that legal and security compliance, you know, this is something you can trust and rely on today, tomorrow. And also when it stops being maintained, it's still doing it in a way that's, you know, at least was saying at the time of stopping it. So from the open source sort of distribution side of things, mission accomplished, right? Well, no, like I already said, the open source distribution way of doing things was very similar to actually how Windows did it. You know, we still had these problems. We still had the issues of compatibility of making sure all these bits and pieces work together, of, you know, portability. How does an application built in one context work elsewhere? And how do you handle this issue of keeping something, keeping this software being delivered and just working and handling the fact that the open source world and therefore everything you're distributing is moving at a constantly changing pace of change. But we're not Windows. And when you're distributing in the open source world, there's different factors at play. So thinking about the compatibility issue, we all end up doing different distributions. We all end up having different libraries and different applications. Different applications require different libraries so the problem becomes exponentially complicated. And application developers don't want to worry about all that mess. They just want to deliver their software in the hands of users. I get it. They don't want to have to worry about which choice of dependencies did certain distributions pick. But most of the time they don't have to. In reality, distributions have their own maintainers. We have our own communities all here who care about this stuff and this part of it. And are the sort of the second tier making sure that the application gets in the hands of users. And you care about it in the same way as the upstream maintainers care about it. So very rarely does it really become a problem because most times it is really being done by the distro maintainers who care about this. And that's what you all do and that's what we've been doing for years and we're bloody good at it. As you can just see in Tumbleweed where we repeatedly ship stuff as fast as the upstream shipping it. But it is importable. It's open SUSE, bi open SUSE, for open SUSE. And an application developer wants to make sure that their software runs in a many different context as possible and as many different distributions as possible. And they don't want to learn a whole bunch of different build tools and they don't want to learn 20 different ways of doing things and they don't want to retest it 20 different places. But again, distribution communities often take care of that problem for the application developers anyway. And then pace of change. Every distribution does everything at a different pace. Heck, if you're open SUSE we do it at two different places. We'd leap, we'd do it regularly every year with major versions every few years and we do Tumbleweed where we just go nuts and as fast as every upstream wants us to go. But in the traditional old fashioned way of doing things, the regular release process gets in the way of that application delivery desire. In the traditional model you can't necessarily run the latest version of software on your stable open SUSE or your Debian system. Debian in particular is the perfect example because they freeze so well and so hard and so solidly that that becomes even harder and they're proud of that. Leap we purposefully design it in a way to try and bend the issue around the edges and deliver faster stuff when we can, how we can. We have the build servers, we have OpenCure to help with that. But it's still a very real problem. Sometimes we just have to say no because the technology can't do it. But that problem, that balance, how do we deliver this software? That is again something that distributions do currently take care of. So how much of a problem is it really? Doesn't matter. App image, flat packs, and App here to solve all the problems left anyway. And they exist to solve that issue. They exist to try and move these problems out of the hand of the distributions or reduce the need for the distributions to do this. So application developers can get that software in the hands of users at the pace the application developers want to be. And they do so by providing a bundle containing the app and the libraries, all the dependencies they need in then some kind of container or context or bubble or whatever. I'm going to keep on saying container although technically speaking, you know, that's open for interpretation. And the big promises of all these technologies, despite details around the edges of how they do things, is to solve all these compatibility and portability issues. It's only going to have the compatible libraries in the bundle. So you don't have to worry about anything from the distribution. You just put your application there, everything you need, you know. It will be portable. It will work everywhere because all the dependencies will be solved in there. You'll never have to worry about what does a distribution ship. And of course that means you can ship it at the pace you want whenever, however, don't have to worry about what the distribution is doing. And it's just going to work. And it's going to be wonderful. That's the promise. And then you have nice architecture diagrams like the snappy one here, or you have really stupidly complicated architecture diagrams like the flatback one, where you have this kind of model of, you know, just ignoring the operating system down the bottom pretty much. It's just there. And then there is some layer on top of that, be it the frameworks or the run times, which, you know, provide this sort of layer of dependencies, which are an awful lot like the dependencies the distributions are currently doing anyway. And then the bundle itself contains the library, the code, the application, everything's fine. But it doesn't work. In practice, it doesn't work. Most of the time it does, but there are still some very, very real issues there. And in fact, the biggest problem that comes around in reality when you start using these in production is this issue of, you know, compatibility and portability. The myth is not true. Because at some point, be it app image, flatback or snappy, there are some assumptions made about the stuff below the system. And we talked about this. Like, you know, in the case of snaps, you know, that is still the kernel. You know, everything above the kernel is assumed to be delivered by some snap somehow. But no kernel is equal. You know, every distribution has a different kernel with different conflicts. There's still problems there that get introduced by different, you know, by different kernel arrangements. We see this in OpenSUSE most actually with Steam, which isn't using one of these technologies, but basically uses the same approach of having a containerized Steam, you know, runtime that gets put in your user area and run that way. And everything was fine on OpenSUSE with Steam for the longest time. It just worked. It did its job. We could move everything in OpenSUSE and everything was fine. And then we changed our Glib C and it exploded spectacularly because their Steam runtime was built with an older Glib C and therefore nothing would run anymore. All those libraries would go horribly, horribly wrong until we started scripting around injecting our Glib C into there and rebuilding everything. And it was a complete mess. And we've seen this issue with flatpacks. We've seen this issue with some of the experience with snaps. We've seen the issue sometimes with app images. But one of the nice things and one of the reasons why I've always come into this liking app image is at least app image documents this problem. You know, it's stated there that it isn't trying to be a universal portable application solution. You're going to have to gather the binaries for the dependencies for the distributions you're targeting. You're not going to magically solve this problem everywhere. But that means that if you're using these technologies, you still have to worry about all of the compatible dependencies which might not be provided by any distribution you might want to run it on. That's a lot of stuff to worry about. Heck, that's the stuff that we all do at OpenSUSE all the damn time and it takes all of us to do it. If you don't get your head around that, your users need to expect crashes. So is it hopeful? Is it hopeless? You know, well, you know, you talked to flat pack people like this, oh, no, we've solved it. You know, we've got these run times or on the snap side of things, you know, we've got these base snaps. Well, those base snaps, those run times just end up being some second curated middle distro. It's middle where for the containerized world. Cool, fine. But you haven't solved the problem. You've just moved it into a different context. It's still another distribution. You're still having to have distribution engineers worry about this stuff and curate it and secure it and patch it and maintain it. Maybe it isn't a real solution. Maybe a real solution is actually figuring out a way of commonly agreeing between us, the distributions as a platform, the application run times as a delivery mechanism and hopefully even the developers on common, I use the word standards here, but let's say common agreements of, you know, what can you expect from your base system? What can application developers expect from their run times? So people can go into this and we won't just have random crashes when you install your app image or your snap on an open Susan machine and it doesn't deliver the kernel the way it's expected or the library is the way it's expected. And until we do that, the compatibility problem isn't really going to be solved. The portability problem isn't really going to be solved. But what about pace of changes? And, you know, well, yeah, what about it just working? Well, back to the window side of things. This is what Windows did. This is very, very similar to what these technologies are doing as well. Is history just repeating itself? Because when you're delivering these libraries in there, it's going to be a security nightmare. Maybe not in a practical sense, you know, because we are talking about putting these stuff in jails and, you know, some kind of isolation. But to be honest, when it comes to these, you know, these bits of isolation, it's a firewall. And I don't like the idea of trusting a firewall with my system security. I like a firewall being there when everything goes wrong and it's my last line of defense, but it's not my first line of defense. I want sensible engineering as my first line of defense. And therefore, I kind of forget about the isolation entirely and want to make sure that someone is taking care of the security of the libraries in my bundle and assume that at some point someone's going to escape the jail. There's no answer for that right now. There's no clean answer for that right now. Or there wasn't a week ago. Same with the maintenance side of things. Who's going to be patching these libraries in there? Who's going to be making sure that those libraries are moving forward? Who's going to be making sure that what I've installed is actually legally allowed to be on my machine? Who's going to make sure that the GPO is correctly being used and cited in there with my LGP stuff, for example? But it's okay. Store adventurers are still going to be happy because all these bundles are going to be using up more disk space. And then this is where my slide decor starts going out of Kilda because I was going to then talk about how we need to start conveying these responsibilities to the various maintainers of App Image and Flapback and Snappy and start talking about how are we going to get this message out there. And I was going to be talking about considering ABI changes and how do you rebuild bundles when ABI changes happen. And I was going to be talking about testing all of that. And I was going to be talking about the security maintenance issues. And I was going to ask the question about, you know, what are we going to do? And I was going to suggest a few things as well. But it all went horribly, horribly wrong because of the Open Suicide Conference. It's all changed. My question no longer is what are we going to do? It's actually what has been done already. And, well, I was going to change the title of this talk because, you know, from being a real bad skeptic of all these technologies, from what's been done, I now love App Image. I really love App Image. Because, well, OBS now builds App Images. Our build service now can take our packages that we have in there for Tomberwee to leap in for everything else, even our develop projects, and build App Images from that. So all those problems about the sort of security compliance, the security auditing, license tracking, dependency tracking, figuring out how to rebuild stuff, when to rebuild stuff, all these problems that we'd already solved in the distribution space, the App Image guys by working with us have now solved it in the App Image space as well. And, you know, we can host them on the build service too. So we have even, you know, changed the context of how you can deliver the software to the users. And we've managed, you know, the OBS team, we've managed to do all of this without impeding App Images' strengths and flexibilities of just being a nice, easy, lightweight of doing this thing in their hands of users. It just gets there. It's easy to deploy, you know, on single click run, it unpacks, it runs. This is really exciting. This totally changes my outlook on all these technologies because suddenly I don't have to be a skeptic anymore. I trust the build service, I trust the tools we have there, I trust the processes we have there. And it makes both sides of the equation more interesting to me. Just a few ideas that are kind of bouncing around my head since I heard about this two days ago. I want to see if we can do something like open SUSE a leap with App Images with user space applications being built from the tumbleweed sources. Because the build service can do that. We can build tumbleweed sources for leap. We can wrap that up in an App image. Last year I had this long one hour ranty talk about how I hated developer projects. Now we can kill them in the sense of killing the publishing of the, we still need them for building stuff for tumbleweed. But if we do this, users will be able to hopefully get the latest version of LibreOffice on their leap machine without having to change everything on their leap machine. That's really cool. We announced earlier this week open SUSE cubic, which is currently very much targeted for the Docker and Kubernetes world of a very, very stable atomic file system, atomic distribution with transactional updates on the base system. Well, now we have this. If we shove a graphical environment on there, maybe a nice tightly polished one, something like GNOME, and do all of the user applications with this, suddenly there's an option of an open SUSE Chrome OS style thing, a nice simple appliance for your grandma, which is something, well, when someone talked to me about that just last week, I said, yeah, good luck, have fun. It's crazy. It would take tons of people, tons of hours, and it's never going to happen. Now I can see one or two maintainers taking what we do in the build service and taking what the app image guys have done with us, and being able to knock that out in a couple of weeks. That is awesome. Admittedly, I'm not going to use it, but it's awesome if someone wants to take and use it. There was a talk today about the package hub and all the stuff we're going in package hub. Hopefully no one from SUSE sees this part. I have a bit of a mixed opinion of package hub because it's really exciting me that we're delivering open SUSE packages to enterprise customers, but at the same time, the way we're doing it is really similar to how tumbleweed back three years ago used to be, where SLEE is a really nice stable base and we keep on putting new versions of everything into package hub, rolling along the top of that. With old tumbleweed, we learned eventually that gets too big and too unwieldy and it starts getting a little bit breaky. It hasn't had that problem with package hub yet, but that's a risk if package hub just keeps on ballooning. For the user space applications, at least, the desktop applications, this app image stuff gives us an easy way of insulating that problem. We can actually define the scope a little bit better, start using the app images there, maybe start delivering app images to SLEE via package hub. Solve the same problem, does it in a slightly more sane way and uses these technologies to solve real problems that we would otherwise run headlong into. But I'm not finished because this is just app image. I was thinking, what's left? Well, snappy, flatback, sorry. With this now, you're not just part of the equation, you're behind. App image now has a better build story than you do. They've got a stronger compliance story than you do and they've got a more straightforward user experience in different distributions than you do because you still haven't got snappy in tumbleweed. Even if you ignore the technical stuff and you want to argue the details with me, they're kicking your ass when it comes to working with others because it's not just the fact that you're both here, it's the tone, it's the style, it's the way they've really got their hands dirty and messed around with the build service. Please be more like app image. It's been so fun working with them and seeing them change my mind and seeing us change their mind a little bit about a few things. We've got the tools, we've got the talent. Please work with us because I think we can do really exciting things in this space. But it's going to need to work in the kind of way that these guys already are working with us because it's just really exciting doing it that way. And it's not all good news or it's not all bad news for you. There are still problems across the entire thing. Dependency hell is still on the horizon. All of these tools still have very limited or no way of really solving this issue of what's coming from the base system. There's still assumptions being made there. We need to get together. These tools need to get together, the distribution needs to get together. And we need to discuss common standards and design common standards so everybody can go into this equation with simple ideas of what's going on there. Without that common understanding, application developers will just find frustration and users will just find crash stuff and distributions will just keep on doing what they're doing at the context they're doing. And then we'll actually miss out on cool opportunities to use this stuff in the way like I was talking about. And security, sandboxing, the app isolation side of things is a complete mess right now. Everybody has cool ideas and no one's finished implementing anything. The app image side of things, well, I kind of understand that because they've gone into this with the approach of use fire jail or whatever the hell you want. That's cool. But whatever the hell you want to be a little more defined than that. SnapD obviously has the app armor side of things. We love app armor, but your patches aren't upstream yet. And I know yet that's cool. Fixing that. But let's get that done. Let's get that in. Because if I had my way of doing my way, I'd like to see app armor kind of become the single way of doing this. I think it makes more sense. I understand that armor or more than I can understand bubble wrap and what they're trying to do there. I think that bubble wrap stuff is a little bit too desktop application oriented, which is cool for desktop applications, but there's scope here for using it in other wonderful ways and IOT and stuff. So let's see what we can do about getting app armor in there, polished up, do it all the way. And with that, I just want to kind of say thank you. I mean, this is becoming a really good lesson for me of two months ago, I was screaming that I thought the world was ending and this would never work out all right. And I've become a convert. I want to help make this better now. So let's just get onto it and does anybody have any questions? Anybody? Go on then. Obviously. I was hoping you did. So I'm curious about security and licensing compliance you mentioned. So you say you have an app image built on OBS. What does actually check that maybe that app image is built from a Git tree somewhere else that you're still compliant that the Git tree does not contain stuff in a different and compatible license and it doesn't contain a bundled copy of a library that has security vulnerabilities? You're technically right. I mean, yeah, if someone's taking an OBS project, a random home OBS project and building app images from that, there's no magic license solution there. But like we were talking about earlier this week, like my other talk from Friday, if you look at what we're doing with tumbleweed, you've got a pool of software and tumbleweed where we are keeping up with upstreams. Heck, we were just about to publish a new GCC7 version that we did while we were here. The pace of changing tumbleweed is fine. So you've got this huge pool that is audited, that is there, that is done, that's keeping up with upstreams. So if you're building your app image based on tumbleweed packages, you've just got to worry then about your tiny little diff, your little part there, which means all of the other dependencies you're feeding into that, they've been audited, they've been checked, they work in the sense of tumbleweed done. Completely agree and snaps have exactly the same thing with the build snap. I talked about it. Yeah, but you're doing exactly the same thing with Ubuntu and only Ubuntu. Well, it's still we're doing it and exactly the same as you do. Yeah, but with the build service way, you can do it with everything I just said with tumbleweed and you can do it with Ubuntu and you can do it with that. And we're going to get there. Yeah, but you can do that your way or you can just copy what they didn't do with the build service. No, because the problems are not like that. The problems are technical below the stack. And it's not a problem of building, you can build it, you just can't run it yet. Once we get to the running, you can build it on top of anything and run it. And that's fine. So we have the same goal here. There's no disagreement. Okay, I'll believe it when I see it. Any more comments? Questions? Cool. So there's two issues actually don't really see solved yet. So first of all, you still end up with a certain amount of size, right? Yeah, because you need redundancy. And of course, one thing that always concerns me is the laziness of developers because they start to rely on compatibility or outdated libraries. So especially if you have something large like say, framework X, which depends on a lot of libraries, then you have a large footprint, which becomes outdated now. I don't want that to happen with stuff like an open SSL, for example. What do you think about that? It's a perfectly fair point. It's a perfectly fair problem. And the answers are exactly... I don't see that problem any different in the containerized application context than I do in the distribution one actually. We suffer that same pain. I'm kind of hoping that getting distributions around the tape, distributions and projects around the table and the kind of common framework idea that we're sort of coalescing might give a little bit of a push to help drive that problem away a little bit. Just like it did with like KDE and Leapneeding and LTS release and KDE, making a commitment to that. That's how they're going to do that and solve that problem for us. Yeah, it's a very real problem. We never really solved it on the dependence, on the distribution side of things on our own. This actually makes it potentially a little bit more... It gives us a second chance to do it right this time, hopefully. There's a question there. Yeah, if I will be developer of... I still feel that Android kick our ass because it's so much easier to develop for Android. And they have the same problem. They have, let's call it distributors, which is actually hardware vendors that provide their phones. And they have some stable base, which is versioned Android. And yeah, of course they have problems with security and such stuff, but I still feel that from developer point of view, it's much easier to develop for Android than any app images. Why? What's the difference there? Because ultimately, an Android app is just a bundle of a whole bunch of dependencies. It might be easy... How would that be easier, both... The question I've got to get to you then is, how is that easier, both getting the application app in the first place and then how is it easier maintaining it? Yeah, one part that's easier, they have a common base. They can easily... It's back... Somehow back what compatible. And you can say, I support this version of Android or newer. And it keeps working. But yeah, I think finding such base in Linux is, as you mentioned, in the standard base is basically missing. And there's a lot... Much more stuff in Android that's common than in Linux, which is... Yeah, that's because you have to bundle much more in Linux than on Android. Yeah, I can't argue with that part. That's why I think we need this. Because I think the problem with the Linux standard base is the scope was too broad. It always was, trying to define everything at every level of every bit of the stack. The nice thing with these technologies is they push that problem down to a certain amount. I mean, Snap tries to push it down to the kernel. App image tries to push it down quite low as well. Flatpack keeps on changing its mind because then the runtimes move the line all over the place. But at least the line tries to get defined further down the stack. So if we just figure out where that line is and define a common base below there, common... Let's say if it line gets drawn near the kernel, basically common standard configs of the kernel. What config is likely to be there? What is an LTS... Are we going to follow the upstream LTS kernel and move along at that kind of pace? Just so you can tag that with a version and say, okay, I'm supporting container-based version blah, and then you get that solution. You get that situation you have with Android. I think we need that in the Linux containerized app side of things as well, totally. Yeah, and another part that sucks for me as developer is that we don't have a common place where to distribute such stuff. They have the Android Store, how they call it, and you just upload it there and you are fine. Every Android user, even if it's from different distribution already having it. And we still miss it in Linux for many years. Yeah, I... Given the nature of the open-source word, I don't think you will have a single place. I mean, you might have a dominant place because someone's going to win the popularity war, but whether it will be a single one, I mean, even Snap has multiple store options already. OBS is now another one for App Image. There will be fragmentation there. I'd love to see a way of pulling it all together somehow because I think that will help in the long run. But that's a problem for the future, I think. Yeah, something we have to worry about, definitely. Yeah, so what I want to say is that we are still behind Android. We're still behind Android. No doubt about that. From developer point of view. And I think user point of view is also behind Android. And that's something that I... That's a drum I think I'll keep on beating because I think if we can get a bit more commonality between these different tool sets, it makes that a little bit easier to catch up with where Android are, makes it a little bit easier also to be honest, the kind of motivation behind this. I can see how this makes distribution's lives easier, less stuff, less to maintain. And we're all lazy. So I can see how we can actually use this to change everything in a rather nice way. But it's only going to work if we kind of find ways of applying focus to that and coming up with some common standards and then seeing how the technologies actually shake out in the long run. Cool. Any more questions? No? Okay then. Thank you very much.
here to talk about, well, dinosaurs or as most them in the real world. And this is a variation these technologies, looking at Snappy, looking at developers want to do is have a nice, well, then when there was a dependency or library maintenance nightmare as a user. How do you then is we care about the security of the operating those fixes out before issues start hitting your and it's kind of spawned by these new shared libraries are a problem. We're trying to mess. They just want to deliver their software in our own communities all here who care about this in the same way as the upstream maintainers care it's still a very real problem. Sometimes we just users at the pace the application developers want It will be portable. It will work everywhere promise. And then you have nice architecture like the flatback one, where you have this kind time it does, but there are still some very, very with Steam for the longest time. It just worked. these responsibilities to the various maintainers of App Image and Flapback and Snappy and start ask the question about, you know, what are we processes we have there. And it makes both sides applications being built from the tumbleweed of use fire jail or whatever the hell you want. That's cool. But whatever the hell you want to be with Ubuntu and you can do it with that. And problems are not like that. The problems are tape, distributions and projects around the table push to help drive that problem away a little that kind of pace? Just so you can tag that with already having it. And we still miss it in Linux the technologies actually shake out in the long Thank you very much.