How I Learned to Stop Worrying and Love Tumbleweed
Brown, Richard
Rolling Releases are the future of Linux distributions. They are already thebetter solution for power users & developers. The methodologies, techniques,and capabilities of Tumbleweed are opening up new doors, creatingpossibilities, and disrupting existing technologies beyond its borders. Thissession will explain how and why openSUSE Tumbleweed is paving the way forthat future, while already being "the reliable rolling release". The talk willdispel the fears, uncertainties and doubts that many have regarding rollingreleases in general and Tumbleweed specifically, and share how you can getinvolved both using, and improving, this exciting fast moving foundation ofthe openSUSE Project. The audience is anyone interested in using recent versions of the Linux kernelor other upstream projects. This includes not only enthusiasts, butdevelopers, especially Linux kernel developers and other developers of fastmoving stacks who often benefit from having a stable platform with all thelatest libraries and binaries available to them. Attendees can expect to get asolid understanding of how Tumbleweed is developed, it's strengths andbenefits, as well as a comparison with Tumbleweed to other similar Linux-basedrolling distributions. This presentation will encourage a broader adoption ofrolling releases and the mentality, methodologies and capabilities that comewith it. Tumbleweed and other similar rolling releases are key platforms toenable the high speed development of the Linux kernel and other key ecosystemcomponents which increasingly move at a faster pace than 'traditional' Linuxdistributions can keep up. Rolling releases are also the natural platform fornot only the development but the early adoption of key emerging technologiessuch as Containerisation and NFV.
 Thank you very much. Thank you very much. So, yeah, I'm basically here to convince you all that my... There we go. That my... My... My belief that rolling releases are the future of the next distributions. You know, I... The more I look at this, the more I've been working in this. So my history is I've been an open-suitor contributor now for 11 years since the project started. In the last three years, I've actually started working for SUSE, testing the enterprise distribution, so the regular stuff. So, you know, I've been a regular distribution guy for a really long time. But because of Tumbleweed, what we've been doing in Tumbleweed for years now, I see this as the trend for all distributions, where things are going. And of course, that comes with certain issues and certain problems, and, you know, that's what I'm going to talk about today. So, really need to start in the beginning with all of this stuff of, you know, how this began and, you know, what the heck is a Linux distribution anyway. So, you know, we're in the distro dev room. Most of us know what a regular release is. You know, most Linux distributions are a regular release, releasing every X months or X years, you know, on some kind of regular cadence. Generally speaking, once it's released, you're freezing the versions, or if you are upgrading, you're somewhat reluctantly upgrading those versions, you know, we're trying to keep a nice stable distribution for our users. We don't want to constantly be breaking everything all of the time. And then if we do have to fix something, we're generally talking about backporting individual fixes and narrowly fixing the distribution, not wholesale changing package versions, you know. So, generally speaking, this is, you know, the Fedora, the Debian, the traditional open-suzan model, and of course, the Ubuntu model. One of the problems with regular releases, okay, how do you then actually do a big step change? How do you develop that? How do you, you know, get ready for the next version? So, all these distributions pretty much have some kind of dev branch, you know, tumbleweed. It's an open-suzan we used to call that factory. In Fedora, they call it Rohide, Debian Cid. You know, a development branch where developers can just check in their stuff, build, you know, packages get built, maybe daily ISOs or something gets spun out, and you have the, you know, the next base for the next major release of your distribution. You know, so it's always released and, you know, nothing's frozen, nothing back-ported, just pure code going in, making packages, and without fail, it's normally broken, you know. You're lucky if it gets working in the last couple of weeks before your release, but even then, it's normally broken. But that's what us distro guys have to live with, you know, we're dealing with this lovely broken dev branches all the damn time. But that's kind of a big problem for not just distro developers, but any general developer looking to target your platform. I mean, you know, they need to be close to, you know, your latest and greatest, your latest upstream, and all of the latest versions of all those upstream packages you have in your dev branch, but it's completely unstable and unusable, and, you know, we've got work to do, but it's always breaking. So you end up with people coming up with wonderful hacky cludges of, you know, maybe taking the stable base and running some PPA or OBS repo on top of it, but, you know, then they have to rebase all of that. It's always a mess. So, you know, it's a nightmare to develop on, it's a nightmare to develop for distributions in particular. This becomes a big community and societal problem. I mean, if your dev branch is not stable, how are you going to get new contributors to your next release? Well, you're not. They're going to go away and play with something else. You know, you get less bugs being fired. You get less polish on your distribution. And then, even with your regular release cadence, you have a hard time actually getting that release out the door. And I think most distributions that have been around for the last few years have had these terrible problems. OpenSuser definitely has, of chugging along with a plan, trying to get a six or an eight month release, you know, release out, and then you're, you know, in the last month or so, and you realize our dev branch is totally messed up. We're going to have to delay for two months, three months, four months, and, you know, and, you know, we've done that. Fedora has problems like that as well. I've done two half. It always happens. And big part of this is the technical debt that comes with that, you know, because you haven't got people using that all the time, all these cludgy little hacks get in there. You're going to have to fix it sooner or later, but your dev branch is a complete mess. And from a project point of view, that generally ends up being a bit of a decline for that distribution. You know, less people are using you. You're not moving that fast. You're not that interesting for anybody. Your project starts stagnating. Everything starts going downhill. It's also a problem for upstreams, you know, they want their software in the hands of users. You know, that's the point. And having a regular release with all that freezing is way too slow for them. They're, you know, they're not going to be able to deliver their software to eight months or a year later or two years later. Dev branches don't do that at all. And regularly, this is not at all. It's a complete mess. There's no way of targeting for upstream developers. Everybody likes to talk about flatback and snappy and containerized apps. And, you know, promise that it's absolutely wonderful. I'm going to solve everything. It's solved some of the problems. And I'll be talking about that more in Janssen at one o'clock today. And then last but by no means least, of course, you know, users. But there are enthusiasts. They're here. They're the ones who want to run everything. See what we're doing. There we go. They don't want to wait. They want to get their latest software as soon as the upstreams published it. You know, new version of NoMor KDE, they want to be running it right away. But they also want a consistent experience. They don't just want some hack-together little bundle running on top of whatever distro they've chosen. They want that polished. They want it integrated with the entire experience. They want it easy to install and maintain. They want it well-patched. And when you start talking to a lot of, especially long-term Linux users, you know, ask them why they pick their distribution. That kind of, you know, somewhat untangible quality of, oh, it just felt right. It was put together properly. So that's why I'm still using it. Is the reason why we end up keeping those users and why we drive more people into our distributions is because we've got those things right. No user experience, but more than that. Just building it properly, building it consistently and convincing people that, you know, we're doing this the right way for the right reasons, but the right things. And they're the ones who are going to be the contributors of tomorrow. But they can't use our dev branches and our regular releases are too slow. So income-running releases. Now, everybody hopefully knows what a rolling release is, but generally speaking, a rolling release has no release schedule. Most, if not all packages are frequently updated. Some distributions like to be partially rolling. So, you know, have a frozen bit, but I'll be talking about more on that later. And updates are delivered when they're ready. And of course, when they're ready is a nice quirky definition. So, you know, when is something ever ready? And the main examples that I can think of with this are, you know, obviously Arch Linux and Gentoo kind of populized the whole thing. And I'm here to talk about Dumbledore. Quite often the feedback I hear about rolling releases though is it's unstable. You know, it changes too much. I can't trust it. Even if I do try it, it's unreliable. It breaks too often. Or it's just hard to live with. There's too much extra work I have to do to keep up with this rolling thing. And when it comes to the sort of the stability side of things, and in this case I'm really talking about stability in how fast is something moving. The, you know, I'm talking rich about, yeah, something always changing. You know, that's the point of a rolling release. You know, you want to get that new software that has to be moving at that pace. These up streams are moving faster and faster. We need to be keeping up with them. Or in some cases maybe even going faster than they are. That's the point. But then you've got to deliver it to your users and your developers who need to use the damn thing. So it has to be built and tested and then integrated together in a consistent way. And this is a problem that I don't think many rolling distributions really get a complete handle on. There's a tab in distribution sites to think of, oh, we're just putting together a bunch of packages and then it's a collection. You know, we're shipping an operating system that contains a bunch of packages, but we have to get all of that integration right. And that's more true with a rolling release than anything else. And also that also means finding a way to shield those users. Not from breakage. I'll talk about that next, but from those behavioral changes, you know, upstream has made some crazy decision to change something in their application. What if the user who's got his project to do tomorrow doesn't know how to do that yet? You know, doesn't know how to use that new feature or, you know, deal with those new defaults. It has to be some way of shielding those users from the unexpected behavioral changes just so they can get on with the work they've got to get on and then learn the new feature when they have the time to sit down and learn it. On the reliability side of things, rolling releases have typically get accused of always breaking. You know, it's got thousands of moving parts from thousands of different upstream projects. All that working together, especially when, you know, those upstream projects have dependencies on, you know, somewhere in between and they're moving at different places is a constant challenge. Just like I said, when it comes to the stability side of things, you know, we have to build this together. We have to test it together and we have to integrate it as one consistent distribution. This is not a piece of parts. The rolling release has to be everything done as a consistent distro. In practical terms, that means testing at the submission. So each submission being tested to make sure that isn't going to break everything. And then also not forgetting about testing as a whole. It doesn't matter if that one package is fine. If when you put that in, it breaks 20 other things. So testing the entire distribution, including all of those changes in one go. And obviously, users shouldn't be shipped something that doesn't work. You know, this isn't a dev branch we're talking about here and really dev branches that break aren't that useful anyway. The thing has to at least work in some basic functional sense. And then I looked at how other distributions deal with testing. Most of us deal with testing. We used to deal with it this way. The concept of sort of passive testing, you know, we put it in some testing branch. There'll be in testing, you know, whatever. And just leave it there for a couple of weeks and trust that the community will go away and test it. And then if they haven't found any bugs, that's good enough, we ship it. It's madness, but we all do it. But it's absolute madness. And it works better or worse depending on how big your community is. But it's still playing Russian roulette. You know, okay, you've got a huge pile of testers. Okay, you've got more chambers in the gun. You're slightly less likely to get your head shut off. But if you've got lots of developers, you've got more bullets in the gun. Eventually someone's going to get shot. So for a rolling release, you need to actively confirm this thing will work. Does it work in isolation? Does it work in a broader context? And does the behavior change in the user, you know, the way a user expects, you know, you've won it. You need to shield the users from that. You need to at least be able to warn them, okay, this package is going to completely change how these commands work. This UI element is going to change, you know, you need to be aware that this stuff is, you know, going to start hitting your users before it hits them. But you need to have all those answers, ideally before upstream have even finished their release. Or at least as fast as you can afterwards, because you need to be shipping this stuff as fast as you can after the upstream projects have finished what they're doing. And this is really the ultimate challenge of the whole thing. And well, yeah, the solutions are hopefully quite interesting. And then talking about things hard to live with, you know, when you look at rolling releases, you know, Arch have a great philosophy. That I really agree with, you know, do it yourself. It's a learning exercise for the arch. For Arch, it works really, really well. The Gen2 way, they don't have a nice definition, but that's kind of my variant of it. You know, do it yourself and then you've got time to read the ArchWiki to figure out what's going on while the whole thing's compiling. And with both of these distributions and a lot of others, you know, if something goes wrong, that's it. You're on your own. Good luck. Get out VM or Emacs. Yeah, you might get lucky to fix it. But who said rolling releases had to be difficult? You know, those fit for their users fine, but the problems are much broader than that. And I think the solution is actually much, much broader than that. And this is where Tumbleweed came in. Now, I'd love to be to take credit and say it was all my idea, but it wasn't. Tumbleweed initially started in the OpenSuser community by Greg Crow Hartman. He was working at SUSE at the time. And what SUSE was doing with our, and OpenSuser was doing with our traditional releases was way too boring for him. You know, upstream kernel hackers, you know, you need something completely different. And it originally started as effectively an add-on to the original OpenSuser distribution. So we started with the base system and did rolling stuff on top. Now, over the last couple of years, it's totally transformed from that model to what I'm going to talk about in a minute, where it's now moving at the pace of contribution. The entire code base will move as fast as our contributors can get those stuff packaged. And we have a process and the tools and the technology to move that incredibly quickly with those upstream projects as long as we have the contributors to do it. It's continuously tested. And when it comes to, you know, who were expecting to use it, well, basically anybody who comes to Faststamp. I mean, you know, you are the target audience for Tumbleweed perfectly. As I mentioned with the original Tumbleweed, so this is before 2014, it was a rolling update repository that we added to an existing, the existing OpenSuser distribution. So you started with the base system. We released that every six months or eight months or a year, depending on how things were going. And then you'd, you know, get a roll on top of that. The focus obviously with Greg, he was interested in the kernel, his desktop environment of choice. And, you know, the community added other things as well. So it's basically sort of a bit of user space and a bit of kernel space. And most of the stuff in between, we just relied on the stability, the stable base to work fine. And if it didn't work fine, then Tumbleweed would quite often just overwrite those packages with whatever it needed in that, in that rolling bit. So, you know, it was a FrankenSuser. It really, really was, you know, it was built that way by design, partly because it was easy to maintain that way. And then of course, every, every eight months, we'd do another OpenSuser release and then have to reset the whole thing to zero again and start from scratch, which was a huge problem. So the partially rolling, the FrankenSuser approach, it just did not work. We tried it for several years. The main problems we had were packages breaking sort of over that chasm between the stable base and the rolling part. You know, some dependency requirements or whatever, you know, just weird behavior issues appearing out of nowhere, weird dependency issues, seg faults you couldn't believe. And of course, you can deal with those issues one by one and then you ad hoc replace those broken parts. So the tumbleweed rolling stack on top gets a little bit bigger and replaces more from your base system, but then your stable base is no longer stable. So you've defeated the point of trying to be partially rolling. And then this reset to zero meant you'd be happily rolling along and then we do a new major release and you're back to square one and starting again, which, yeah, it was just painfully disruptive for users, you know. So often we avoided rolling back package versions, but it was still a complete script because he really had two different communities working on it. The stable guys working on the stable distro and the tumbleweed guys packaging things slightly differently and, you know, targeting their audience slightly differently. And looking at this, the open SUSE team at SUSE spent a really long time analyzing how to actually fix sort of all of these problems properly. I mean, they started first looking at our dev branch and looking at how do we get that more stable. I kind of started looking at it from the other side of, okay, how do we get tumbleweed more stable? And we kind of ended up meeting in the middle. And we found what I really think is a universal truth that when you're building a distribution with, you know, several thousand moving parts, if you need to be ready to move any part of that, you know, any package anywhere, you've got to be prepared to move everything. You've got to have the tools, the techniques, the processes that, okay, that new dependency from upstream is going to require you to, yes, change your init system and your kernel and God knows what else. And you have to be prepared to do that at a moment's notice. And if you're not able to do that, you're not able to do a rolling release consistently. You might be to start for a short period of time. Sooner or later, you'll hit a brick wall and stop and have real pain. So to get this solved properly, you need to always be ready to change everything at a moment's notice. Thankfully, in open SUSE, we had two ACs up our sleeve. The open SUSE build service, which is what we've been using for years for building all of our distributions. And it really has always embraced and tackled this idea of distribution is a cohesive product. You know, we build all our packages there, but the build services are aware of all those dependencies. If it needs to rebuild part of those packages, it does. So you can rebuild part of the dependency tree. It's dependency aware. It's what open SUSE and SUSE have been using for years for building our distributions. It's also cross-distribution support, which, you know, that kind of challenge of being able to build for other distros as well really taught us a heck of a lot of how to handle dependency trees, because, you know, Debian does it way different from RPM and, you know, Arch. We even had one guy who passed in Gen2 support for the build service, which was just pointless but beautiful. And, yeah, it's used by, not used by others, you know, on cloud Linux foundation, use it for Tizen, VLC, plenty of other people use it too. So we had the tools for building and, you know, really kind of helped with that rolling stuff already, but you've got to test the thing as well. And, yeah, my other job when I'm not doing open SUSE stuff is working in QA. And, yeah, open QA is a really nice testing tool for this. We have a boost-out test for this weekend. Unlike every other testing tool I've ever found, this tests from a user's perspective. You know, we're not necessarily caring about the code. It's API neutral. You don't have to teach it what the API needs to do. You tell open QA what is the user expected to do with this software. What commands are they going to type? What buttons are they going to click? Because open QA is also graphically aware, so we can actually look at the screen. We have open CV running on there. It can even actually do screenshot analysis or partial screenshot analysis. So is this UI element on the screen, whatever, or where it needs to be? And, yeah, therefore you can test the entire distribution end-to-end from the way the users are going to use the damn thing. Without this, we couldn't do tumbleweed the way we're doing today. And, in fact, it's adoption in a serious sense really began with tumbleweed. And since then, we've now started actually using it for the more stable distribution of open SUSELEAP and for the enterprise products at SUSE, which are all testing using the same framework. In terms of what it can test, it's pretty much anything. You have a nice dashboard with hundreds and hundreds of scenarios. I mean, I got to A to E here on one example. And, in fact, it's a really bad screenshot because I've missed out four of the columns, but it's also got support from multiple architectures. So here we're testing Intel, but we can also test ARM, S390, Power, 32-bit Intel if you're interested. Mostly based on VMs, although we have support now for testing on bare metal as well. So what it normally does, we'll start up with a distribution, fire it up in a VM, start testing it the way a user is going to do it, start the installation, run through the install, installs finished, start up the applications. If you want to feed it a disk image already of an existing installation, then you can do nice, fancy stuff like doing upgrade testing and migrations and all that kind of stuff. Has support for multiple machines, so you can do cluster tests and other weird and wonderful stuff with that. And, like I said, it's graphically aware. So when something goes horribly wrong, you can actually see, okay, I was expecting a tumbleweed, a nice dark blue screen on the left there, but in fact, the system came up with the wrong graphic. So even those tiny behavioral changes, which they're not bugs, it's just something that's changed upstream, get caught by OpenQA. If everything else has stayed the same way, if the UI is still working in the same way, OpenQA would just ignore it, it's fine, but if something's practically changed, if the buttons moved, the commands changed its function, OpenQA will always find it. So with those tools, we've developed this process for effectively continuous integration for an entire distribution. When you're submitting something to OpenSuser Tumbleweed, you're submitting it to an OBS project called Factory. That then goes into an automatic review queue, the usual kind of RPM-linked stuff of, you know, checking basic sanity checks and unit tests and that kind of stuff, nothing too fancy. But for any package that has a potential system-wide breaking impact, which is pretty much every package, we have a process called staging, where OpenQA will, sorry, OBS will spin up a variant of factory, well, variant of Tumbleweed, with everything that was working in the last release, plus these changes and nothing else. So basically, what if distribution, what if we merged this and we run a bunch of sanity tests in OpenQA. So actual proper smoke test, booting this thing up and installing it to make sure the distribution still works. And that takes about 20 minutes. It's not a long, complicated process, so something gets checked in. But this is before any human being has really actually looked at the code. No one's actually reviewed anything yet. No one's seen it. Does it work? Is it worth looking at? If it doesn't, it gets rejected. Dev has to go away and start again. If it passes, though, then it ends up in the review queue. Then, you know, your usual kind of code reviews, spec file review, checking everything works. Then it ends up in factory, where all of this stuff is then put in a single repository. If we have lots of check-ins at the same time, you end up with a big pile of changes. And then we test the entire distribution end-to-end in a few hundred different combinations on every architecture that we're interested in. That takes about, depending on the hardware, six to seven hours now. We've got more hardware, maybe shorter. So it's all very highly parallelized. And that will run through all those tests. The QA is therefore totally automatic. We actually tell OpenQA what the last acceptable failures were. Nothing's perfect, so there are sometimes acceptable failures in there. And if that new build doesn't have any new failures, it ships. It becomes tumbleweed for in the morning whenever. It doesn't matter. So that whole process is completely pipelined and automatically done. So writing these automated tests for OpenQA totally gate the quality of our distribution. So it's great. If a user reports a bug now saying, so-and-so broke, and I was doing this, well, just take that, translate it to the OpenQA descriptive language. You're never going to break that again. So every bug report becomes a nice easy test case in theory, and you can shield yourself from ever making the same mistake twice. At a relatively light cost, because you're already booting up the VM, so it doesn't really take that long to pop in, do what the user's doing, and done. So functionality checks, all that kind of stuff, totally able to be shielded with it. But that also means if you do listen to my talk now and do download tumbleweed and find something you don't like, contributing to OpenQA is the easiest way of fixing that. Because as soon as you put a test into OpenQA, it's never going to happen again. So the main website for OpenQA is open.qa. Documentation's there, and we have a nice little bug tracker, feature tracker for that. If you want to know more about the whole sort of pipelining and using OBS and OpenQA for continuous integration at 1.30 today, which of course clashes with my other talk in Janssen, but you can pick which one you like more. My colleague Christian is talking here about how you can use this approach in your projects, so not necessarily on a full distro level, but even on just a smaller upstream project kind of thing, you can use these tools to build your package in OBS, test it in OpenQA, follow the same kind of pipeline. Normally when I talk to people about this, they say, that's really fancy, but I don't want to wait for all that testing. I just want it now. It's too long. And in fact, there is one guy working at SUSE who does rebuild all of factory just so he can avoid OpenQA. He is crazy, but he does do it. In reality though, let's say working with upstream projects, especially these really demanding ones like GNOME and KDE, the process keeps pace with them. Upstream GNOME releases their tarbles, and within 48 hours, upstream GNOME is in Open SUSE Tumbleweed. Faster than ARCH, faster than Gen2 even. Most people don't get it compiled that quickly. So the process works even with these big upstream projects with hundreds of moving parts. In the case of KDE Plasma 4.9, which came out last week, we actually shipped it on the same day. That sounds really impressive. In reality, KDE kind of cheap because they give the tarbles out five days before. But yeah, we shipped on release day. It's kind of cool. We've also taken that process, and of course we've got the build service so we can do all this fancy stuff. And we actually have derivatives of Tumbleweed that kind of make that constant what if all the damn time. Because OBS is get aware so we can actually attach it to GNOME Git projects or KDE Git, and then build GNOME Next or Open SUSE Krypton, which are constantly building those packages as soon as something gets checked in, and constantly testing those packages as soon as something gets checked in, and then testing the entire distribution. So we get to file all these upstream bugs before anyone else does because we're seeing it in a context of an actual distribution all of the damn time. But on the border Tumbleweed side, every week now our Tumbleweed release manager Dominic writes a report of what's changed this week. So the bots that do all the magic of the pipelining automatically pump out release reports and change logs of everything that changed. But that's just ridiculously long. So he tries to summarize it to something humans can do. And he made the mistake last year of saying it's been a really quiet week this week. So then I went through those logs and figured out what does quiet mean. This was last year. We did three snapshots. Snapshots are basically Tumbleweed releases. So that whole pipelining end to end, a new build, three types. So three new sets of ISOs, three new repository versions, all in there. Those versions all together contain 146 updates, restore shipping DVDs or ISOs at least. So a bit of shuffling around of packages, 15 new packages on there, 38 packages out to make room for it, and one new kernel. That's quiet. That was a year ago. Six months ago, this was it. Five snapshots. Every single workday we shipped another snapshot. Twice as many packages. Twice as many packages on the DVD. Twice as many we had to take out and twice as many kernels. It's now six months later. The last decent run through we had. We had one run through 15 days of nonstop Tumbleweed releases. You know, every day a new snapshot. I didn't bother doing a slide for that because I can't be bothered to read that many logs. I mean, it's just crazy. The process really, really works and you can do a voting distribution across the entire code base without any problems. It just works from the user side of things, though that can then be terrifying because they're like, okay, all this stuff is changing. How did they deal with that? And okay, open QA is testing it for a practical point of view. It works. But it's still changing. I mean, you know, the behavior is still going to change. Luckily with openSUSA, we've got SQL source. We use BTFS by default and we have it by default with a tool called snapper set up. So on a standard Tumbleweed machine, of course you can disable it, use whatever file system you want. But in our normal configuration, every single time you install a package on a Tumbleweed machine, you have a snapshot before and after. So if that package changes something you don't like, just roll back to the snapshot before. Keep on working. You know, go back and learn it, how it works, you know, whenever you have some time. You're completely immunized from those, from those behavioral changes. You're also completely immunized from us screwing up your package accidentally because, you know, you can always roll back to how it worked yesterday. And even if we completely balls everything up and open QA completely screws up and your machine doesn't boot anymore, you can still boot to that snapshot from Grubb. So there's no risk as long as you're using BTFS and snapper. And there's very little risk of us with that because we're using it so much it just doesn't break on our systems. So Austin will be talking about this more this afternoon, actually more talking about sort of the next generation of this idea of not just using snapshots as a safety net, but actually, you know, more of as a atomic way of delivering updates. So actually, you know, doing transactional updates. So, you know, this concept plus plus, but still using RPMs. So that's tumbleweed. What about the Dev branch? What about open-suitor factory? You know, I've mentioned it's got a part in the process. But in reality, once we started doing this and delivering tumbleweed to users, we realized we don't need a Dev branch anymore. Factory no longer can be used by a human being. It exists in OBS tucked away and we never publish it. We don't have a traditional Dev branch anymore. It might be broken. Open QA or tell us if it is. We won't make it tumbleweed. If it worked, it's tumbleweed. So the old Greg's tumbleweed, we basically took the name, the old factory, we put added testing and then we called this a new tumbleweed. So it's a bit of a new distribution. And yeah, no more Dev branches. We're rolling on for that. And it serves that same purpose while also serving all those other purposes of rolling release does of, you know, hitting a bigger user base, being more useful, dealing with upstream projects, etc. But what about regular releases? I mean, you know, there's still a use case for that. Maybe not for the typical FOSTEM audience, but, you know, other people, you know, don't want to necessarily have everything changing all the time. Well, in Open Sousa, we kind of killed that idea off as well. Partly because of declining interest in the concept, but also because of an alternative concept that's way more interesting. We now have a distribution called LEAP, which starts with the Susan & Lytzen enterprise code base, which is now totally in OBS. So we build that, you know, kind of like CentOS, I guess, but we don't limit it to be a one-to-one match. The community then contributes to that code base as well and adds whatever packages they want in addition to. So you end up with a full-blown community distribution, all fully tested or fully integrated, but with that nice enterprise code base underneath it, so it's rock, rock, rock solid and stable and much more conservative with its pace of change. So in the past, you know, everybody knew as is Open Sousa the one distribution disc project. We now have two. Tumbleweed rolling ahead, continuously updated, continuously tested, like I said, perfect for the FOSTEM audience. And then LEAP with a shared core underneath it, sharing it with SLEE and, you know, much more conservative, you know, users. Anybody who doesn't like things changing often, you know, who wants to update once a year, you know, perfect for them, Sysadmin's your little server in the corner that you don't want to touch that often for patching. I mean, you can run a rolling release on a server, nothing's wrong with that, but, you know, you're going to have to find some nice way of automating those updates. You don't have to worry about that with LEAP. Like I said, it's kind of a combination of the Open Sousa project and the Sousa enterprise side of things. And this is roughly kind of, I guess, 50 split halfway, roughly how the whole thing looks. So, you know, there is a shared core of a nice stable base system, which is also what the enterprise side is using. There's a small amount of enterprise packages which just don't make any sense in a community sense or, you know, covered by licenses, we don't want to ship or, you know, whatever, just, you know, doesn't make sense there. And then Open Sousa, LEAP has thousands of community packages sitting on top of that shared base. And most of those community packages, if not all of them, actually originate in some form or another from Tumbleweed, where, you know, it's a pure community distribution, all those packages always moving, always rolling on. The interesting question I get normally get asked at this point is, well, what about SLEE? Because, you know, everyone knows this model. It's like, you know, Fedora with Red Hat, you know, you do a Fedora release and then there's a Red Hat Enterprise release based on it sometime after. You're not doing an Open Sousa release anymore. You can't really base SLEE on something based on SLEE that all gets a little bit circular. You know, what is Sousa going to do for their next release of the enterprise product? Well, they're going to base it on Tumbleweed, not in the sense that they're going to do a rolling release, but the Tumbleweed base system that's constantly rolling is stable enough and usable enough that Sousa Linux will be basing SLEE 13 on Tumbleweed, forking from it at some point for their next release for SLEE 13. So, not only does the process work for, you know, other guys at FOSTA, even enterprises are looking at this and like, okay, that's the best way of us developing our next major releases. We don't need this old model of a regular release in between and all those hassles with that, all the work that comes with that. And when you then also tie it with the leap side of things, where leap, of course, is based on the enterprise service packs and that stuff as well, but with community stuff on top, you end up with this weird, nice, complicated sort of infinity loop of open Sousa is simultaneously downstream and upstream of SLEE, and something that the community does in the last leap version is a candidate for the next service pack, and it makes things actually way more interesting with that whole working with a corporate partner side of things, because they have a lot more options on the table and less things get less stagnant, and it also means you end up with more enterprise developers working on your code base, which is, you know, good for polish and everything working nice. So in review, if you're a developer and you want to deal with, you know, if you're dealing with any upstream project, Tumbleweed is a perfect option for you. It's going to keep up with those upstream projects, more of upstream projects than you're interested in. You're going to get the latest and greatest of everything. It just works. And if it doesn't work exactly the way you want it to, it's really easy to contribute to. You know, we've got OBS and we've got Open QA, so you can make sure we don't break it the same way. So, yeah, and of course you've got Snapper, so you can just get back on when you work in the meantime when you haven't got the time to deal with that. So it's a perfect platform to work on and work with us on, especially if you're an upstream developer as well. Tumbleweed seems like a natural choice for quite a few projects. You know, it's a distribution there. We have all these tools. You can do stuff with us first, add a nice pace, get everything checked with the latest kernel and the latest system D and the latest Python or whatever. And then worry about how those other slower distributions are going to have to deal with it. And if you want to then use our tools like OBS to actually do that building, it's perfect because you can actually build originally for OpenSuser and then flick a few switches and build for a different RPM distribution. And then, you know, add a Debian file and flick a few files and you're building for Debian. I mean, the build service takes care of all of that. So it's a perfect platform for that. And like I say, containerized applications are cool for those other distributions, but there's actually more work for you in the long run. That's what I'm talking about later. This is actually less work. The community takes care of a good chunk of this. You just have to worry about your little part on top. And if you're a user, you manage to get the latest and greatest of everything. It just works if it doesn't snap us there to take care of it. And it's a perfect place to start contributing because that barrier of entry so low, it's easy to destroy something at the build service and say, hey, I'm going to submit this. If it doesn't work, it would be open QA or kick it out right away. It's easy to get your feet wet, easy to start contributing with us. So it's a perfect platform to start with that. You don't just have to take my word for it because we've actually been looking at the statistics of how many people are using Cumbleweed since this transition from partially rolling and the dev base, the dev branch to just having a pure rolling release and no dev branch. And our user numbers are looking quite good. If you look purely from a dev branch perspective, our dev branch now has 10 times more users than we ever had. If you factor in the partially rolling distribution, it's growing at 67% a year. And really weirdly, those weird little peaks around April seem to match up with the Buntu releases. And I'm not entirely sure what that means. But yeah, kind of fun. Yeah, that's it for me. I have a bit of time, I think. So yeah, 10 minutes. So 10 minutes for questions. Yes, Lan. How do I report it back against a rolling release so that you can reproduce it? Where's the reference that I can give you so you are the same code base to reproduce what I'm trying to report? Very rarely does a problem actually live in just that. But if it happens, because we have this snapshot model, so effectively every single tumbleweed version you're using is an entirely cohesive distro, you actually have a version number for your distro in ETCOS release. It's ISO date for the day of the snapshot. So you can mention that there. We can then fish it out of OBS and run that there. What we'll normally do is just see, does this happen on the latest version? Because if we fixed it already, that makes it really easy to close the bug. So why is it rolling? You still basically have a distribution release. Yes, and another release. That's part of the whole building it consistently. Yes. Sometimes maintainers of development projects refuse to enable the tumbleweed build target. And they say, oh, that's already factory enabled. But when I contribute packages, I'd like to make sure, okay, that there are two purposes, contribute back, but my main purpose is to get the package built for OBS and tumbleweed for my running system very soon. And so what's your perspective? Should every seven projects enable OBS with a tumbleweed, this snapshot as a build target? Generally speaking, most developed projects should, well, most developed projects shouldn't need to enable it, because they should be submitting everything into tumbleweed already. So this shouldn't really be an issue in the sense of OBS in the build service. You know, you are fast, but I want to have it now. You want to forget the testing and you just want it now? Usually I contribute when I make sure that it works for me. So I'd like to have the package now for this particular template with snapshot. Not factory with that diff of potential day. Yeah, in that case, persuade them. If they get stuck, call the board. I mean, we can help out with that. There's no problem from a build side of things. Yes? We were producing to test passively with the users and now you use OBS with the writing of the test and what the professional contribution versus what Susie is doing. Well, yeah, the question was, you know, we were passively testing in the past. We're now actively testing who's writing those tests. You know, is it Susie or the community or whatever? Because open QA adoption really started in the community, most of the tests were open SUSE tests first. And it's only in the last year or two where SUSE has actually been sort of actively contributing to open QA. So it's a community project first, which now has enterprise guys working on it too. So most of the tests come from that kind of original group of original contributors and now the enterprise side of things. The process is totally open. All the tests are in GitHub. So even for the enterprise stuff, actually, we've left that all open as well. So relatively easy to write a test submitted in. Generally speaking, I think there's no objection to this, but generally speaking, we don't get that many contributions from individuals outside kind of, I guess, the kind of tumbleweed release management loop. Like people who are actively sitting there working on the factory process contributing a lot, maybe the release manager for tumbleweed. They're the ones who see the things break initially before anyone's actually got it because we haven't shipped it yet. And they'll just throw together a test and patch it in. So I'd like to get that a bit more broad. So that's kind of one of the reasons I'm here is if there's stuff in tumbleweed, you download it, you try it and you just don't like it. And it's missed in the current process. That's why I've got those links in the slides so you can broaden that out a little bit because we don't scale. But the process does. Yes, Len. As someone working on an open source project that tries hard to support the most prominent distribution, I'm not sure if we would be able to cope with that pace and making sure that on the one hand, our project and code based on this rolling release is still being able to support all the other more common and slower distributions. If you're a project depending on certain frameworks and they change, you have to come up with lots of quirks to be able to support the latest version of the framework for tumbleweed, but still being backwards compatible to the old stuff in those traditional distributions that have a large use of this sort of... It's a challenge. It's a challenge. But then when you also look at the trend of even enterprise distributions playing around with the idea of things like modules and certain parts of enterprise distributions moving faster than others, it's a problem that everybody's going to have to deal with at some point. The reality is we can't... The regular model doesn't quite work. What I actually think the solution might be is taking this process and finding some way of defining a more moderate cadence. Tumbleweed runs as fast as we can contribute. Maybe there's a rolling release model that fits that problem nicer of something that runs a little bit slower to fit, but everybody else can keep up with it. Yes. How big is the gap between the traditional releases and the tumbleweed, especially when you want to distribute in 42 or 3? Yes. The stable kernel based on one's less curve. Yes. And tumbleweed is constantly moving. Yes. Everything. So I'm more interested in the next stable release or to the future tumbleweed. The gap is large and admittedly does get larger. It gets larger the more of these service packs are. It's a fact of life. That stable base isn't going to move that fast. However, if you look at, for example, how SUSE has done SLEE 12 compared to SLEE 11, the amount of change in each enterprise service pack has been significantly more and faster than in the traditional sort of SLEE 11, SLEE 10 and earlier era. Part of the reason SUSE has been able to do that is because Open SUSE is doing what we're doing with LEAP and tumbleweed. So, you know, things like jumping a new kernel version in service pack 2 isn't quite so terrifying when tumbleweed already was running it for a year. So, yeah, it's, so you see this kind of balance of enterprise distros catching up because the rolling release is going really fast. Him first. Sorry. Yes. So, you talked a lot about the advantages of this release model. Yes. If you have to change one thing or improve one thing in the tumbleweed release model, what would it be? That's a really good question. The one thing I would improve right now is actually have a delay between building and actually publishing. Right now, as soon as it's built and tested and everything's fine, we're publishing it. We need to put a delay in there because now we have all that crazy number of users. Our mirror infrastructure can't keep up. So, if you're one of those poor guys who hits that new snapshot first thing before our mirrors have synced, yeah, it's going to be slow. So, we've really got to find a way of kind of teaching the build service to, okay, it's ready, push it out to all the mirrors and then flag it up for everybody to use. And that's on our list at the moment because it's painful at the moment. Okay, I'd like to add something to this because I run several VMs with tumbleweed and I think one improvement could be to have an easy way to have a caching proxy. If you run several systems and you have a caching proxy to save your bandwidth. We actually have instructions on the wiki how to do that already. In the mirror guides actually. We have an asking server you can do all of that. Another question would be this new release model with the core 13, change the or improve the confusion about the because on the factory mailing list, even others, even people more familiar with all this are sometimes confused of what to take from leave and what to take from, you know, and then they say, okay, there's a list. It's called from sleep. Yeah, from factory, but then you also see, oh, there's a full from older. Yeah, there are pretty much confused. Part of the reason for that confusion when we, you know, this is all a relatively new concept when we first started leap, we purposefully didn't define where this line was between the shared core and the community stuff because we want the community to help define that. So, yeah, we, you know, 40 that's the safe bet push the factory push in tumbleweed it'll always work in it'll always get to where it needs to go. But the in the case of 42.1. Yeah, we were purposefully leaving it very open to divide that line around 42.2. We made it way.
 Thank you very much.  Thank you very much.  So, yeah, I'm basically here to convince you all that my...  There we go.  That my...  My...  My belief that rolling releases are the future of the next distributions.  You know, I...  The more I look at this, the more I've been working in this.  So my history is I've been an open-suitor contributor now for 11 years  since the project started.  In the last three years, I've actually started working for SUSE,  testing the enterprise distribution, so the regular stuff.  So, you know, I've been a regular distribution guy for a really long time.  But because of Tumbleweed, what we've been doing in Tumbleweed for years now,  I see this as the trend for all distributions, where things are going.  And of course, that comes with certain issues and certain problems,  and, you know, that's what I'm going to talk about today.  So, really need to start in the beginning with all of this stuff of, you know,  how this began and, you know, what the heck is a Linux distribution anyway.  So, you know, we're in the distro dev room.  Most of us know what a regular release is.  You know, most Linux distributions are a regular release,  releasing every X months or X years, you know, on some kind of regular cadence.  Generally speaking, once it's released, you're freezing the versions,  or if you are upgrading, you're somewhat reluctantly upgrading those versions,  you know, we're trying to keep a nice stable distribution for our users.  We don't want to constantly be breaking everything all of the time.  And then if we do have to fix something, we're generally talking about  backporting individual fixes and narrowly fixing the distribution,  not wholesale changing package versions, you know.  So, generally speaking, this is, you know, the Fedora, the Debian,  the traditional open-suzan model, and of course, the Ubuntu model.  One of the problems with regular releases, okay, how do you then actually do a big step change?  How do you develop that? How do you, you know, get ready for the next version?  So, all these distributions pretty much have some kind of dev branch, you know, tumbleweed.  It's an open-suzan we used to call that factory.  In Fedora, they call it Rohide, Debian Cid.  You know, a development branch where developers can just check in their stuff,  build, you know, packages get built, maybe daily ISOs or something gets spun out,  and you have the, you know, the next base for the next major release of your distribution.  You know, so it's always released and, you know, nothing's frozen,  nothing back-ported, just pure code going in, making packages,  and without fail, it's normally broken, you know.  You're lucky if it gets working in the last couple of weeks before your release,  but even then, it's normally broken.  But that's what us distro guys have to live with, you know, we're dealing with this lovely broken dev branches all the damn time.  But that's kind of a big problem for not just distro developers,  but any general developer looking to target your platform.  I mean, you know, they need to be close to, you know, your latest and greatest, your latest upstream,  and all of the latest versions of all those upstream packages you have in your dev branch,  but it's completely unstable and unusable, and, you know, we've got work to do, but it's always breaking.  So you end up with people coming up with wonderful hacky cludges of, you know,  maybe taking the stable base and running some PPA or OBS repo on top of it,  but, you know, then they have to rebase all of that. It's always a mess.  So, you know, it's a nightmare to develop on, it's a nightmare to develop for distributions in particular.  This becomes a big community and societal problem.  I mean, if your dev branch is not stable, how are you going to get new contributors to your next release?  Well, you're not. They're going to go away and play with something else.  You know, you get less bugs being fired. You get less polish on your distribution.  And then, even with your regular release cadence, you have a hard time actually getting that release out the door.  And I think most distributions that have been around for the last few years have had these terrible problems.  OpenSuser definitely has, of chugging along with a plan, trying to get a six or an eight month release, you know, release out,  and then you're, you know, in the last month or so, and you realize our dev branch is totally messed up.  We're going to have to delay for two months, three months, four months, and, you know, and, you know, we've done that.  Fedora has problems like that as well. I've done two half. It always happens.  And big part of this is the technical debt that comes with that, you know, because you haven't got people using that all the time,  all these cludgy little hacks get in there. You're going to have to fix it sooner or later, but your dev branch is a complete mess.  And from a project point of view, that generally ends up being a bit of a decline for that distribution.  You know, less people are using you. You're not moving that fast. You're not that interesting for anybody.  Your project starts stagnating. Everything starts going downhill.  It's also a problem for upstreams, you know, they want their software in the hands of users.  You know, that's the point. And having a regular release with all that freezing is way too slow for them.  They're, you know, they're not going to be able to deliver their software to eight months or a year later or two years later.  Dev branches don't do that at all. And regularly, this is not at all. It's a complete mess.  There's no way of targeting for upstream developers. Everybody likes to talk about flatback and snappy and containerized apps.  And, you know, promise that it's absolutely wonderful. I'm going to solve everything. It's solved some of the problems.  And I'll be talking about that more in Janssen at one o'clock today.  And then last but by no means least, of course, you know, users.  But there are enthusiasts. They're here. They're the ones who want to run everything. See what we're doing.  There we go. They don't want to wait. They want to get their latest software as soon as the upstreams published it.  You know, new version of NoMor KDE, they want to be running it right away.  But they also want a consistent experience. They don't just want some hack-together little bundle running on top of whatever distro they've chosen.  They want that polished. They want it integrated with the entire experience. They want it easy to install and maintain.  They want it well-patched. And when you start talking to a lot of, especially long-term Linux users, you know, ask them why they pick their distribution.  That kind of, you know, somewhat untangible quality of, oh, it just felt right. It was put together properly.  So that's why I'm still using it.  Is the reason why we end up keeping those users and why we drive more people into our distributions is because we've got those things right.  No user experience, but more than that.  Just building it properly, building it consistently and convincing people that, you know, we're doing this the right way for the right reasons, but the right things.  And they're the ones who are going to be the contributors of tomorrow.  But they can't use our dev branches and our regular releases are too slow.  So income-running releases.  Now, everybody hopefully knows what a rolling release is, but generally speaking, a rolling release has no release schedule.  Most, if not all packages are frequently updated.  Some distributions like to be partially rolling.  So, you know, have a frozen bit, but I'll be talking about more on that later.  And updates are delivered when they're ready.  And of course, when they're ready is a nice quirky definition.  So, you know, when is something ever ready?  And the main examples that I can think of with this are, you know, obviously Arch Linux and Gentoo kind of populized the whole thing.  And I'm here to talk about Dumbledore.  Quite often the feedback I hear about rolling releases though is it's unstable.  You know, it changes too much.  I can't trust it.  Even if I do try it, it's unreliable.  It breaks too often.  Or it's just hard to live with.  There's too much extra work I have to do to keep up with this rolling thing.  And when it comes to the sort of the stability side of things, and in this case I'm really talking about stability in how fast is something moving.  The, you know, I'm talking rich about, yeah, something always changing.  You know, that's the point of a rolling release.  You know, you want to get that new software that has to be moving at that pace.  These up streams are moving faster and faster.  We need to be keeping up with them.  Or in some cases maybe even going faster than they are.  That's the point.  But then you've got to deliver it to your users and your developers who need to use the damn thing.  So it has to be built and tested and then integrated together in a consistent way.  And this is a problem that I don't think many rolling distributions really get a complete handle on.  There's a tab in distribution sites to think of, oh, we're just putting together a bunch of packages and then it's a collection.  You know, we're shipping an operating system that contains a bunch of packages, but we have to get all of that integration right.  And that's more true with a rolling release than anything else.  And also that also means finding a way to shield those users.  Not from breakage.  I'll talk about that next, but from those behavioral changes, you know, upstream has made some crazy decision to change something in their application.  What if the user who's got his project to do tomorrow doesn't know how to do that yet?  You know, doesn't know how to use that new feature or, you know, deal with those new defaults.  It has to be some way of shielding those users from the unexpected behavioral changes just so they can get on with the work they've got to get on  and then learn the new feature when they have the time to sit down and learn it.  On the reliability side of things, rolling releases have typically get accused of always breaking.  You know, it's got thousands of moving parts from thousands of different upstream projects.  All that working together, especially when, you know, those upstream projects have dependencies on, you know, somewhere in between  and they're moving at different places is a constant challenge.  Just like I said, when it comes to the stability side of things, you know, we have to build this together.  We have to test it together and we have to integrate it as one consistent distribution.  This is not a piece of parts.  The rolling release has to be everything done as a consistent distro.  In practical terms, that means testing at the submission.  So each submission being tested to make sure that isn't going to break everything.  And then also not forgetting about testing as a whole.  It doesn't matter if that one package is fine.  If when you put that in, it breaks 20 other things.  So testing the entire distribution, including all of those changes in one go.  And obviously, users shouldn't be shipped something that doesn't work.  You know, this isn't a dev branch we're talking about here and really dev branches that break aren't that useful anyway.  The thing has to at least work in some basic functional sense.  And then I looked at how other distributions deal with testing.  Most of us deal with testing.  We used to deal with it this way.  The concept of sort of passive testing, you know, we put it in some testing branch.  There'll be in testing, you know, whatever.  And just leave it there for a couple of weeks and trust that the community will go away and test it.  And then if they haven't found any bugs, that's good enough, we ship it.  It's madness, but we all do it.  But it's absolute madness.  And it works better or worse depending on how big your community is.  But it's still playing Russian roulette.  You know, okay, you've got a huge pile of testers.  Okay, you've got more chambers in the gun.  You're slightly less likely to get your head shut off.  But if you've got lots of developers, you've got more bullets in the gun.  Eventually someone's going to get shot.  So for a rolling release, you need to actively confirm this thing will work.  Does it work in isolation?  Does it work in a broader context?  And does the behavior change in the user, you know, the way a user expects, you know, you've won it.  You need to shield the users from that.  You need to at least be able to warn them, okay, this package is going to completely change how these commands work.  This UI element is going to change, you know, you need to be aware that this stuff is, you know, going to start hitting your users before it hits them.  But you need to have all those answers, ideally before upstream have even finished their release.  Or at least as fast as you can afterwards, because you need to be shipping this stuff as fast as you can after the upstream projects have finished what they're doing.  And this is really the ultimate challenge of the whole thing.  And well, yeah, the solutions are hopefully quite interesting.  And then talking about things hard to live with, you know, when you look at rolling releases, you know, Arch have a great philosophy.  That I really agree with, you know, do it yourself.  It's a learning exercise for the arch.  For Arch, it works really, really well.  The Gen2 way, they don't have a nice definition, but that's kind of my variant of it.  You know, do it yourself and then you've got time to read the ArchWiki to figure out what's going on while the whole thing's compiling.  And with both of these distributions and a lot of others, you know, if something goes wrong, that's it.  You're on your own.  Good luck. Get out VM or Emacs.  Yeah, you might get lucky to fix it.  But who said rolling releases had to be difficult?  You know, those fit for their users fine, but the problems are much broader than that.  And I think the solution is actually much, much broader than that.  And this is where Tumbleweed came in.  Now, I'd love to be to take credit and say it was all my idea, but it wasn't.  Tumbleweed initially started in the OpenSuser community by Greg Crow Hartman.  He was working at SUSE at the time.  And what SUSE was doing with our, and OpenSuser was doing with our traditional releases was way too boring for him.  You know, upstream kernel hackers, you know, you need something completely different.  And it originally started as effectively an add-on to the original OpenSuser distribution.  So we started with the base system and did rolling stuff on top.  Now, over the last couple of years, it's totally transformed from that model to what I'm going to talk about in a minute,  where it's now moving at the pace of contribution.  The entire code base will move as fast as our contributors can get those stuff packaged.  And we have a process and the tools and the technology to move that incredibly quickly with those upstream projects  as long as we have the contributors to do it.  It's continuously tested.  And when it comes to, you know, who were expecting to use it, well, basically anybody who comes to Faststamp.  I mean, you know, you are the target audience for Tumbleweed perfectly.  As I mentioned with the original Tumbleweed, so this is before 2014,  it was a rolling update repository that we added to an existing, the existing OpenSuser distribution.  So you started with the base system.  We released that every six months or eight months or a year, depending on how things were going.  And then you'd, you know, get a roll on top of that.  The focus obviously with Greg, he was interested in the kernel, his desktop environment of choice.  And, you know, the community added other things as well.  So it's basically sort of a bit of user space and a bit of kernel space.  And most of the stuff in between, we just relied on the stability, the stable base to work fine.  And if it didn't work fine, then Tumbleweed would quite often just overwrite those packages with whatever it needed in that, in that rolling bit.  So, you know, it was a FrankenSuser. It really, really was, you know, it was built that way by design,  partly because it was easy to maintain that way.  And then of course, every, every eight months, we'd do another OpenSuser release  and then have to reset the whole thing to zero again and start from scratch, which was a huge problem.  So the partially rolling, the FrankenSuser approach, it just did not work.  We tried it for several years. The main problems we had were packages breaking sort of over that chasm between the stable base and the rolling part.  You know, some dependency requirements or whatever, you know, just weird behavior issues appearing out of nowhere, weird dependency issues, seg faults you couldn't believe.  And of course, you can deal with those issues one by one and then you ad hoc replace those broken parts.  So the tumbleweed rolling stack on top gets a little bit bigger and replaces more from your base system,  but then your stable base is no longer stable.  So you've defeated the point of trying to be partially rolling.  And then this reset to zero meant you'd be happily rolling along and then we do a new major release and you're back to square one and starting again,  which, yeah, it was just painfully disruptive for users, you know.  So often we avoided rolling back package versions, but it was still a complete script because he really had two different communities working on it.  The stable guys working on the stable distro and the tumbleweed guys packaging things slightly differently and, you know, targeting their audience slightly differently.  And looking at this, the open SUSE team at SUSE spent a really long time analyzing how to actually fix sort of all of these problems properly.  I mean, they started first looking at our dev branch and looking at how do we get that more stable.  I kind of started looking at it from the other side of, okay, how do we get tumbleweed more stable?  And we kind of ended up meeting in the middle.  And we found what I really think is a universal truth that when you're building a distribution with, you know, several thousand moving parts,  if you need to be ready to move any part of that, you know, any package anywhere, you've got to be prepared to move everything.  You've got to have the tools, the techniques, the processes that, okay, that new dependency from upstream is going to require you to, yes,  change your init system and your kernel and God knows what else.  And you have to be prepared to do that at a moment's notice.  And if you're not able to do that, you're not able to do a rolling release consistently.  You might be to start for a short period of time.  Sooner or later, you'll hit a brick wall and stop and have real pain.  So to get this solved properly, you need to always be ready to change everything at a moment's notice.  Thankfully, in open SUSE, we had two ACs up our sleeve.  The open SUSE build service, which is what we've been using for years for building all of our distributions.  And it really has always embraced and tackled this idea of distribution is a cohesive product.  You know, we build all our packages there, but the build services are aware of all those dependencies.  If it needs to rebuild part of those packages, it does.  So you can rebuild part of the dependency tree.  It's dependency aware.  It's what open SUSE and SUSE have been using for years for building our distributions.  It's also cross-distribution support, which, you know, that kind of challenge of being able to build for other distros as well  really taught us a heck of a lot of how to handle dependency trees, because, you know, Debian does it way different from RPM and, you know, Arch.  We even had one guy who passed in Gen2 support for the build service, which was just pointless but beautiful.  And, yeah, it's used by, not used by others, you know, on cloud Linux foundation, use it for Tizen, VLC, plenty of other people use it too.  So we had the tools for building and, you know, really kind of helped with that rolling stuff already, but you've got to test the thing as well.  And, yeah, my other job when I'm not doing open SUSE stuff is working in QA.  And, yeah, open QA is a really nice testing tool for this. We have a boost-out test for this weekend.  Unlike every other testing tool I've ever found, this tests from a user's perspective.  You know, we're not necessarily caring about the code. It's API neutral.  You don't have to teach it what the API needs to do.  You tell open QA what is the user expected to do with this software.  What commands are they going to type? What buttons are they going to click?  Because open QA is also graphically aware, so we can actually look at the screen.  We have open CV running on there. It can even actually do screenshot analysis or partial screenshot analysis.  So is this UI element on the screen, whatever, or where it needs to be?  And, yeah, therefore you can test the entire distribution end-to-end from the way the users are going to use the damn thing.  Without this, we couldn't do tumbleweed the way we're doing today.  And, in fact, it's adoption in a serious sense really began with tumbleweed.  And since then, we've now started actually using it for the more stable distribution of open SUSELEAP  and for the enterprise products at SUSE, which are all testing using the same framework.  In terms of what it can test, it's pretty much anything.  You have a nice dashboard with hundreds and hundreds of scenarios.  I mean, I got to A to E here on one example.  And, in fact, it's a really bad screenshot because I've missed out four of the columns,  but it's also got support from multiple architectures.  So here we're testing Intel, but we can also test ARM, S390, Power, 32-bit Intel if you're interested.  Mostly based on VMs, although we have support now for testing on bare metal as well.  So what it normally does, we'll start up with a distribution, fire it up in a VM,  start testing it the way a user is going to do it, start the installation, run through the install,  installs finished, start up the applications.  If you want to feed it a disk image already of an existing installation,  then you can do nice, fancy stuff like doing upgrade testing and migrations and all that kind of stuff.  Has support for multiple machines, so you can do cluster tests and other weird and wonderful stuff with that.  And, like I said, it's graphically aware.  So when something goes horribly wrong, you can actually see, okay, I was expecting a tumbleweed,  a nice dark blue screen on the left there, but in fact, the system came up with the wrong graphic.  So even those tiny behavioral changes, which they're not bugs, it's just something that's changed upstream,  get caught by OpenQA.  If everything else has stayed the same way, if the UI is still working in the same way,  OpenQA would just ignore it, it's fine, but if something's practically changed,  if the buttons moved, the commands changed its function, OpenQA will always find it.  So with those tools, we've developed this process for effectively continuous integration for an entire distribution.  When you're submitting something to OpenSuser Tumbleweed, you're submitting it to an OBS project called Factory.  That then goes into an automatic review queue, the usual kind of RPM-linked stuff of, you know,  checking basic sanity checks and unit tests and that kind of stuff, nothing too fancy.  But for any package that has a potential system-wide breaking impact, which is pretty much every package,  we have a process called staging, where OpenQA will, sorry, OBS will spin up a variant of factory,  well, variant of Tumbleweed, with everything that was working in the last release,  plus these changes and nothing else.  So basically, what if distribution, what if we merged this and we run a bunch of sanity tests in OpenQA.  So actual proper smoke test, booting this thing up and installing it to make sure the distribution still works.  And that takes about 20 minutes.  It's not a long, complicated process, so something gets checked in.  But this is before any human being has really actually looked at the code.  No one's actually reviewed anything yet.  No one's seen it.  Does it work?  Is it worth looking at?  If it doesn't, it gets rejected.  Dev has to go away and start again.  If it passes, though, then it ends up in the review queue.  Then, you know, your usual kind of code reviews, spec file review, checking everything works.  Then it ends up in factory, where all of this stuff is then put in a single repository.  If we have lots of check-ins at the same time, you end up with a big pile of changes.  And then we test the entire distribution end-to-end in a few hundred different combinations on every architecture that we're interested in.  That takes about, depending on the hardware, six to seven hours now.  We've got more hardware, maybe shorter.  So it's all very highly parallelized.  And that will run through all those tests.  The QA is therefore totally automatic.  We actually tell OpenQA what the last acceptable failures were.  Nothing's perfect, so there are sometimes acceptable failures in there.  And if that new build doesn't have any new failures, it ships.  It becomes tumbleweed for in the morning whenever.  It doesn't matter.  So that whole process is completely pipelined and automatically done.  So writing these automated tests for OpenQA totally gate the quality of our distribution.  So it's great.  If a user reports a bug now saying, so-and-so broke, and I was doing this, well, just take that, translate it to the OpenQA descriptive language.  You're never going to break that again.  So every bug report becomes a nice easy test case in theory, and you can shield yourself from ever making the same mistake twice.  At a relatively light cost, because you're already booting up the VM, so it doesn't really take that long to pop in, do what the user's doing, and done.  So functionality checks, all that kind of stuff, totally able to be shielded with it.  But that also means if you do listen to my talk now and do download tumbleweed and find something you don't like, contributing to OpenQA is the easiest way of fixing that.  Because as soon as you put a test into OpenQA, it's never going to happen again.  So the main website for OpenQA is open.qa.  Documentation's there, and we have a nice little bug tracker, feature tracker for that.  If you want to know more about the whole sort of pipelining and using OBS and OpenQA for continuous integration at 1.30 today, which of course clashes with my other talk in Janssen,  but you can pick which one you like more.  My colleague Christian is talking here about how you can use this approach in your projects, so not necessarily on a full distro level,  but even on just a smaller upstream project kind of thing, you can use these tools to build your package in OBS, test it in OpenQA, follow the same kind of pipeline.  Normally when I talk to people about this, they say, that's really fancy, but I don't want to wait for all that testing.  I just want it now. It's too long.  And in fact, there is one guy working at SUSE who does rebuild all of factory just so he can avoid OpenQA.  He is crazy, but he does do it.  In reality though, let's say working with upstream projects, especially these really demanding ones like GNOME and KDE, the process keeps pace with them.  Upstream GNOME releases their tarbles, and within 48 hours, upstream GNOME is in Open SUSE Tumbleweed.  Faster than ARCH, faster than Gen2 even. Most people don't get it compiled that quickly.  So the process works even with these big upstream projects with hundreds of moving parts.  In the case of KDE Plasma 4.9, which came out last week, we actually shipped it on the same day.  That sounds really impressive. In reality, KDE kind of cheap because they give the tarbles out five days before.  But yeah, we shipped on release day. It's kind of cool.  We've also taken that process, and of course we've got the build service so we can do all this fancy stuff.  And we actually have derivatives of Tumbleweed that kind of make that constant what if all the damn time.  Because OBS is get aware so we can actually attach it to GNOME Git projects or KDE Git, and then build GNOME Next or Open SUSE Krypton,  which are constantly building those packages as soon as something gets checked in,  and constantly testing those packages as soon as something gets checked in, and then testing the entire distribution.  So we get to file all these upstream bugs before anyone else does because we're seeing it in a context of an actual distribution all of the damn time.  But on the border Tumbleweed side, every week now our Tumbleweed release manager Dominic writes a report of what's changed this week.  So the bots that do all the magic of the pipelining automatically pump out release reports and change logs of everything that changed.  But that's just ridiculously long. So he tries to summarize it to something humans can do.  And he made the mistake last year of saying it's been a really quiet week this week.  So then I went through those logs and figured out what does quiet mean.  This was last year. We did three snapshots.  Snapshots are basically Tumbleweed releases. So that whole pipelining end to end, a new build, three types.  So three new sets of ISOs, three new repository versions, all in there.  Those versions all together contain 146 updates, restore shipping DVDs or ISOs at least.  So a bit of shuffling around of packages, 15 new packages on there, 38 packages out to make room for it, and one new kernel.  That's quiet. That was a year ago.  Six months ago, this was it. Five snapshots. Every single workday we shipped another snapshot.  Twice as many packages. Twice as many packages on the DVD. Twice as many we had to take out and twice as many kernels.  It's now six months later. The last decent run through we had. We had one run through 15 days of nonstop Tumbleweed releases.  You know, every day a new snapshot. I didn't bother doing a slide for that because I can't be bothered to read that many logs.  I mean, it's just crazy. The process really, really works and you can do a voting distribution across the entire code base without any problems.  It just works from the user side of things, though that can then be terrifying because they're like, okay, all this stuff is changing.  How did they deal with that? And okay, open QA is testing it for a practical point of view. It works.  But it's still changing. I mean, you know, the behavior is still going to change.  Luckily with openSUSA, we've got SQL source. We use BTFS by default and we have it by default with a tool called snapper set up.  So on a standard Tumbleweed machine, of course you can disable it, use whatever file system you want.  But in our normal configuration, every single time you install a package on a Tumbleweed machine, you have a snapshot before and after.  So if that package changes something you don't like, just roll back to the snapshot before. Keep on working.  You know, go back and learn it, how it works, you know, whenever you have some time. You're completely immunized from those, from those behavioral changes.  You're also completely immunized from us screwing up your package accidentally because, you know, you can always roll back to how it worked yesterday.  And even if we completely balls everything up and open QA completely screws up and your machine doesn't boot anymore, you can still boot to that snapshot from Grubb.  So there's no risk as long as you're using BTFS and snapper. And there's very little risk of us with that because we're using it so much it just doesn't break on our systems.  So Austin will be talking about this more this afternoon, actually more talking about sort of the next generation of this idea of not just using snapshots as a safety net,  but actually, you know, more of as a atomic way of delivering updates. So actually, you know, doing transactional updates.  So, you know, this concept plus plus, but still using RPMs.  So that's tumbleweed. What about the Dev branch? What about open-suitor factory? You know, I've mentioned it's got a part in the process.  But in reality, once we started doing this and delivering tumbleweed to users, we realized we don't need a Dev branch anymore.  Factory no longer can be used by a human being. It exists in OBS tucked away and we never publish it.  We don't have a traditional Dev branch anymore. It might be broken. Open QA or tell us if it is. We won't make it tumbleweed.  If it worked, it's tumbleweed. So the old Greg's tumbleweed, we basically took the name, the old factory, we put added testing and then we called this a new tumbleweed.  So it's a bit of a new distribution. And yeah, no more Dev branches. We're rolling on for that.  And it serves that same purpose while also serving all those other purposes of rolling release does of, you know, hitting a bigger user base,  being more useful, dealing with upstream projects, etc.  But what about regular releases? I mean, you know, there's still a use case for that.  Maybe not for the typical FOSTEM audience, but, you know, other people, you know, don't want to necessarily have everything changing all the time.  Well, in Open Sousa, we kind of killed that idea off as well.  Partly because of declining interest in the concept, but also because of an alternative concept that's way more interesting.  We now have a distribution called LEAP, which starts with the Susan & Lytzen enterprise code base, which is now totally in OBS.  So we build that, you know, kind of like CentOS, I guess, but we don't limit it to be a one-to-one match.  The community then contributes to that code base as well and adds whatever packages they want in addition to.  So you end up with a full-blown community distribution, all fully tested or fully integrated,  but with that nice enterprise code base underneath it, so it's rock, rock, rock solid and stable and much more conservative with its pace of change.  So in the past, you know, everybody knew as is Open Sousa the one distribution disc project.  We now have two.  Tumbleweed rolling ahead, continuously updated, continuously tested, like I said, perfect for the FOSTEM audience.  And then LEAP with a shared core underneath it, sharing it with SLEE and, you know, much more conservative, you know, users.  Anybody who doesn't like things changing often, you know, who wants to update once a year, you know, perfect for them,  Sysadmin's your little server in the corner that you don't want to touch that often for patching.  I mean, you can run a rolling release on a server, nothing's wrong with that,  but, you know, you're going to have to find some nice way of automating those updates.  You don't have to worry about that with LEAP.  Like I said, it's kind of a combination of the Open Sousa project and the Sousa enterprise side of things.  And this is roughly kind of, I guess, 50 split halfway, roughly how the whole thing looks.  So, you know, there is a shared core of a nice stable base system, which is also what the enterprise side is using.  There's a small amount of enterprise packages which just don't make any sense in a community sense or, you know,  covered by licenses, we don't want to ship or, you know, whatever, just, you know, doesn't make sense there.  And then Open Sousa, LEAP has thousands of community packages sitting on top of that shared base.  And most of those community packages, if not all of them, actually originate in some form or another from Tumbleweed,  where, you know, it's a pure community distribution, all those packages always moving, always rolling on.  The interesting question I get normally get asked at this point is, well, what about SLEE?  Because, you know, everyone knows this model.  It's like, you know, Fedora with Red Hat, you know, you do a Fedora release and then there's a Red Hat Enterprise release based on it sometime after.  You're not doing an Open Sousa release anymore.  You can't really base SLEE on something based on SLEE that all gets a little bit circular.  You know, what is Sousa going to do for their next release of the enterprise product?  Well, they're going to base it on Tumbleweed, not in the sense that they're going to do a rolling release,  but the Tumbleweed base system that's constantly rolling is stable enough and usable enough that Sousa Linux will be basing SLEE 13 on Tumbleweed,  forking from it at some point for their next release for SLEE 13.  So, not only does the process work for, you know, other guys at FOSTA,  even enterprises are looking at this and like, okay, that's the best way of us developing our next major releases.  We don't need this old model of a regular release in between and all those hassles with that, all the work that comes with that.  And when you then also tie it with the leap side of things,  where leap, of course, is based on the enterprise service packs and that stuff as well, but with community stuff on top,  you end up with this weird, nice, complicated sort of infinity loop of open Sousa is simultaneously downstream and upstream of SLEE,  and something that the community does in the last leap version is a candidate for the next service pack,  and it makes things actually way more interesting with that whole working with a corporate partner side of things,  because they have a lot more options on the table and less things get less stagnant,  and it also means you end up with more enterprise developers working on your code base,  which is, you know, good for polish and everything working nice.  So in review, if you're a developer and you want to deal with, you know, if you're dealing with any upstream project,  Tumbleweed is a perfect option for you.  It's going to keep up with those upstream projects, more of upstream projects than you're interested in.  You're going to get the latest and greatest of everything.  It just works.  And if it doesn't work exactly the way you want it to, it's really easy to contribute to.  You know, we've got OBS and we've got Open QA, so you can make sure we don't break it the same way.  So, yeah, and of course you've got Snapper, so you can just get back on when you work in the meantime when you haven't got the time to deal with that.  So it's a perfect platform to work on and work with us on, especially if you're an upstream developer as well.  Tumbleweed seems like a natural choice for quite a few projects.  You know, it's a distribution there.  We have all these tools.  You can do stuff with us first, add a nice pace, get everything checked with the latest kernel and the latest system D and the latest Python or whatever.  And then worry about how those other slower distributions are going to have to deal with it.  And if you want to then use our tools like OBS to actually do that building, it's perfect because you can actually build originally for OpenSuser and then flick a few switches and build for a different RPM distribution.  And then, you know, add a Debian file and flick a few files and you're building for Debian.  I mean, the build service takes care of all of that.  So it's a perfect platform for that.  And like I say, containerized applications are cool for those other distributions, but there's actually more work for you in the long run.  That's what I'm talking about later.  This is actually less work.  The community takes care of a good chunk of this.  You just have to worry about your little part on top.  And if you're a user, you manage to get the latest and greatest of everything.  It just works if it doesn't snap us there to take care of it.  And it's a perfect place to start contributing because that barrier of entry so low, it's easy to destroy something at the build service and say, hey, I'm going to submit this.  If it doesn't work, it would be open QA or kick it out right away.  It's easy to get your feet wet, easy to start contributing with us.  So it's a perfect platform to start with that.  You don't just have to take my word for it because we've actually been looking at the statistics of how many people are using Cumbleweed since this transition from partially rolling and the dev base, the dev branch to just having a pure rolling release and no dev branch.  And our user numbers are looking quite good.  If you look purely from a dev branch perspective, our dev branch now has 10 times more users than we ever had.  If you factor in the partially rolling distribution, it's growing at 67% a year.  And really weirdly, those weird little peaks around April seem to match up with the Buntu releases.  And I'm not entirely sure what that means.  But yeah, kind of fun.  Yeah, that's it for me. I have a bit of time, I think.  So yeah, 10 minutes. So 10 minutes for questions.  Yes, Lan.  How do I report it back against a rolling release so that you can reproduce it?  Where's the reference that I can give you so you are the same code base to reproduce what I'm trying to report?  Very rarely does a problem actually live in just that.  But if it happens, because we have this snapshot model, so effectively every single tumbleweed version you're using is an entirely cohesive distro,  you actually have a version number for your distro in ETCOS release.  It's ISO date for the day of the snapshot.  So you can mention that there. We can then fish it out of OBS and run that there.  What we'll normally do is just see, does this happen on the latest version?  Because if we fixed it already, that makes it really easy to close the bug.  So why is it rolling? You still basically have a distribution release.  Yes, and another release. That's part of the whole building it consistently.  Yes.  Sometimes maintainers of development projects refuse to enable the tumbleweed build target.  And they say, oh, that's already factory enabled. But when I contribute packages, I'd like to make sure, okay,  that there are two purposes, contribute back, but my main purpose is to get the package built for OBS and tumbleweed for my running system very soon.  And so what's your perspective? Should every seven projects enable OBS with a tumbleweed, this snapshot as a build target?  Generally speaking, most developed projects should, well, most developed projects shouldn't need to enable it,  because they should be submitting everything into tumbleweed already.  So this shouldn't really be an issue in the sense of OBS in the build service.  You know, you are fast, but I want to have it now.  You want to forget the testing and you just want it now?  Usually I contribute when I make sure that it works for me.  So I'd like to have the package now for this particular template with snapshot.  Not factory with that diff of potential day.  Yeah, in that case, persuade them. If they get stuck, call the board.  I mean, we can help out with that. There's no problem from a build side of things.  Yes?  We were producing to test passively with the users and now you use OBS with the writing of the test and what the professional contribution versus what Susie is doing.  Well, yeah, the question was, you know, we were passively testing in the past.  We're now actively testing who's writing those tests. You know, is it Susie or the community or whatever?  Because open QA adoption really started in the community, most of the tests were open SUSE tests first.  And it's only in the last year or two where SUSE has actually been sort of actively contributing to open QA.  So it's a community project first, which now has enterprise guys working on it too.  So most of the tests come from that kind of original group of original contributors and now the enterprise side of things.  The process is totally open. All the tests are in GitHub.  So even for the enterprise stuff, actually, we've left that all open as well.  So relatively easy to write a test submitted in.  Generally speaking, I think there's no objection to this, but generally speaking, we don't get that many contributions from individuals outside kind of, I guess, the kind of tumbleweed release management loop.  Like people who are actively sitting there working on the factory process contributing a lot, maybe the release manager for tumbleweed.  They're the ones who see the things break initially before anyone's actually got it because we haven't shipped it yet.  And they'll just throw together a test and patch it in.  So I'd like to get that a bit more broad.  So that's kind of one of the reasons I'm here is if there's stuff in tumbleweed, you download it, you try it and you just don't like it.  And it's missed in the current process.  That's why I've got those links in the slides so you can broaden that out a little bit because we don't scale.  But the process does.  Yes, Len.  As someone working on an open source project that tries hard to support the most prominent distribution, I'm not sure if we would be able to cope with that pace and making sure that on the one hand, our project and code based on this rolling release is still being able to support all the other more common and slower distributions.  If you're a project depending on certain frameworks and they change, you have to come up with lots of quirks to be able to support the latest version of the framework for tumbleweed, but still being backwards compatible to the old stuff in those traditional distributions that have a large use of this sort of...  It's a challenge.  It's a challenge.  But then when you also look at the trend of even enterprise distributions playing around with the idea of things like modules and certain parts of enterprise distributions moving faster than others, it's a problem that everybody's going to have to deal with at some point.  The reality is we can't...  The regular model doesn't quite work.  What I actually think the solution might be is taking this process and finding some way of defining a more moderate cadence. Tumbleweed runs as fast as we can contribute.  Maybe there's a rolling release model that fits that problem nicer of something that runs a little bit slower to fit, but everybody else can keep up with it.  Yes.  How big is the gap between the traditional releases and the tumbleweed, especially when you want to distribute in 42 or 3?  Yes.  The stable kernel based on one's less curve.  Yes.  And tumbleweed is constantly moving.  Yes.  Everything.  So I'm more interested in the next stable release or to the future tumbleweed.  The gap is large and admittedly does get larger.  It gets larger the more of these service packs are.  It's a fact of life.  That stable base isn't going to move that fast.  However, if you look at, for example, how SUSE has done SLEE 12 compared to SLEE 11, the amount of change in each enterprise service pack has been significantly more and faster than in the traditional sort of SLEE 11, SLEE 10 and earlier era.  Part of the reason SUSE has been able to do that is because Open SUSE is doing what we're doing with LEAP and tumbleweed.  So, you know, things like jumping a new kernel version in service pack 2 isn't quite so terrifying when tumbleweed already was running it for a year.  So, yeah, it's, so you see this kind of balance of enterprise distros catching up because the rolling release is going really fast.  Him first.  Sorry.  Yes.  So, you talked a lot about the advantages of this release model.  Yes.  If you have to change one thing or improve one thing in the tumbleweed release model, what would it be?  That's a really good question.  The one thing I would improve right now is actually have a delay between building and actually publishing.  Right now, as soon as it's built and tested and everything's fine, we're publishing it.  We need to put a delay in there because now we have all that crazy number of users.  Our mirror infrastructure can't keep up.  So, if you're one of those poor guys who hits that new snapshot first thing before our mirrors have synced, yeah, it's going to be slow.  So, we've really got to find a way of kind of teaching the build service to, okay, it's ready, push it out to all the mirrors and then flag it up for everybody to use.  And that's on our list at the moment because it's painful at the moment.  Okay, I'd like to add something to this because I run several VMs with tumbleweed and I think one improvement could be to have an easy way to have a caching proxy.  If you run several systems and you have a caching proxy to save your bandwidth.  We actually have instructions on the wiki how to do that already.  In the mirror guides actually.  We have an asking server you can do all of that.  Another question would be this new release model with the core 13, change the or improve the confusion about the because on the factory mailing list, even others, even people more familiar with all this are sometimes confused of what to take from leave and what to take from, you know, and then they say, okay, there's a list.  It's called from sleep.  Yeah, from factory, but then you also see, oh, there's a full from older.  Yeah, there are pretty much confused.  Part of the reason for that confusion when we, you know, this is all a relatively new concept when we first started leap, we purposefully didn't define where this line was between the shared core and the community stuff because we want the community to help define that.  So, yeah, we, you know,  40 that's the safe bet push the factory push in tumbleweed it'll always work in it'll always get to where it needs to go.  But the in the case of 42.1. Yeah, we were purposefully leaving it very open to divide that line around 42.2. We made it way.
How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_0_131.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_132_215.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_216_263.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_264_383.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_384_659.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_660_767.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_768_1031.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_1032_1307.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_1308_1523.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_1524_2855.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_2856_4079.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_4080_5099.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_5100_7103.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_7104_8135.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_8136_10259.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_10260_10355.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_10356_11267.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_11268_11315.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_11316_11375.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_11376_11711.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_11712_14099.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_14100_15947.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_15948_17123.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_17124_18431.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_18432_19427.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_19428_19715.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_19716_19907.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_19908_21527.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_21528_23051.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_23052_25955.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_25956_26963.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_26964_28643.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_28644_30671.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_30672_31463.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_31464_32147.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_32148_33143.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_33144_35087.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_35088_35135.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_35136_37835.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_37836_38387.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_38388_38663.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_38664_38927.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_38928_39047.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_39048_39167.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_39168_39299.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_39300_39431.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_39432_39551.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_39552_39635.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_39636_41195.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_41196_42539.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_42540_43247.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_43248_43415.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_43416_44411.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_44412_44903.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_44904_47171.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_47172_47327.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_47328_47471.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_47472_48671.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_48672_48767.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_48768_48887.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_48888_48995.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_48996_50351.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_50352_51323.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_51324_51539.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_51540_53387.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_53388_55475.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_55476_56459.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_56460_57995.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_57996_59111.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_59112_59999.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_60000_69599.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_69600_69647.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_69648_74003.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_74004_75119.png How_I_Learned_to_Stop_Worrying_and_Love_Tumbleweed_Why_everyone_should_be_running_a_rol_75120_75237.png
Thank you very much. Thank you very much. So, yeah, I'm basically here to convince you all that my... There we go. That my... My... My belief that rolling releases are the future of the next distributions. You know, I... The more I look at this, the more I've been working in this. So my history is I've been an open-suitor contributor now for 11 years since the project started. In the last three years, I've actually started working for SUSE, testing the enterprise distribution, so the regular stuff. So, you know, I've been a regular distribution guy for a really long time. But because of Tumbleweed, what we've been doing in Tumbleweed for years now, I see this as the trend for all distributions, where things are going. And of course, that comes with certain issues and certain problems, and, you know, that's what I'm going to talk about today. So, really need to start in the beginning with all of this stuff of, you know, how this began and, you know, what the heck is a Linux distribution anyway. So, you know, we're in the distro dev room. Most of us know what a regular release is. You know, most Linux distributions are a regular release, releasing every X months or X years, you know, on some kind of regular cadence. Generally speaking, once it's released, you're freezing the versions, or if you are upgrading, you're somewhat reluctantly upgrading those versions, you know, we're trying to keep a nice stable distribution for our users. We don't want to constantly be breaking everything all of the time. And then if we do have to fix something, we're generally talking about backporting individual fixes and narrowly fixing the distribution, not wholesale changing package versions, you know. So, generally speaking, this is, you know, the Fedora, the Debian, the traditional open-suzan model, and of course, the Ubuntu model. One of the problems with regular releases, okay, how do you then actually do a big step change? How do you develop that? How do you, you know, get ready for the next version? So, all these distributions pretty much have some kind of dev branch, you know, tumbleweed. It's an open-suzan we used to call that factory. In Fedora, they call it Rohide, Debian Cid. You know, a development branch where developers can just check in their stuff, build, you know, packages get built, maybe daily ISOs or something gets spun out, and you have the, you know, the next base for the next major release of your distribution. You know, so it's always released and, you know, nothing's frozen, nothing back-ported, just pure code going in, making packages, and without fail, it's normally broken, you know. You're lucky if it gets working in the last couple of weeks before your release, but even then, it's normally broken. But that's what us distro guys have to live with, you know, we're dealing with this lovely broken dev branches all the damn time. But that's kind of a big problem for not just distro developers, but any general developer looking to target your platform. I mean, you know, they need to be close to, you know, your latest and greatest, your latest upstream, and all of the latest versions of all those upstream packages you have in your dev branch, but it's completely unstable and unusable, and, you know, we've got work to do, but it's always breaking. So you end up with people coming up with wonderful hacky cludges of, you know, maybe taking the stable base and running some PPA or OBS repo on top of it, but, you know, then they have to rebase all of that. It's always a mess. So, you know, it's a nightmare to develop on, it's a nightmare to develop for distributions in particular. This becomes a big community and societal problem. I mean, if your dev branch is not stable, how are you going to get new contributors to your next release? Well, you're not. They're going to go away and play with something else. You know, you get less bugs being fired. You get less polish on your distribution. And then, even with your regular release cadence, you have a hard time actually getting that release out the door. And I think most distributions that have been around for the last few years have had these terrible problems. OpenSuser definitely has, of chugging along with a plan, trying to get a six or an eight month release, you know, release out, and then you're, you know, in the last month or so, and you realize our dev branch is totally messed up. We're going to have to delay for two months, three months, four months, and, you know, and, you know, we've done that. Fedora has problems like that as well. I've done two half. It always happens. And big part of this is the technical debt that comes with that, you know, because you haven't got people using that all the time, all these cludgy little hacks get in there. You're going to have to fix it sooner or later, but your dev branch is a complete mess. And from a project point of view, that generally ends up being a bit of a decline for that distribution. You know, less people are using you. You're not moving that fast. You're not that interesting for anybody. Your project starts stagnating. Everything starts going downhill. It's also a problem for upstreams, you know, they want their software in the hands of users. You know, that's the point. And having a regular release with all that freezing is way too slow for them. They're, you know, they're not going to be able to deliver their software to eight months or a year later or two years later. Dev branches don't do that at all. And regularly, this is not at all. It's a complete mess. There's no way of targeting for upstream developers. Everybody likes to talk about flatback and snappy and containerized apps. And, you know, promise that it's absolutely wonderful. I'm going to solve everything. It's solved some of the problems. And I'll be talking about that more in Janssen at one o'clock today. And then last but by no means least, of course, you know, users. But there are enthusiasts. They're here. They're the ones who want to run everything. See what we're doing. There we go. They don't want to wait. They want to get their latest software as soon as the upstreams published it. You know, new version of NoMor KDE, they want to be running it right away. But they also want a consistent experience. They don't just want some hack-together little bundle running on top of whatever distro they've chosen. They want that polished. They want it integrated with the entire experience. They want it easy to install and maintain. They want it well-patched. And when you start talking to a lot of, especially long-term Linux users, you know, ask them why they pick their distribution. That kind of, you know, somewhat untangible quality of, oh, it just felt right. It was put together properly. So that's why I'm still using it. Is the reason why we end up keeping those users and why we drive more people into our distributions is because we've got those things right. No user experience, but more than that. Just building it properly, building it consistently and convincing people that, you know, we're doing this the right way for the right reasons, but the right things. And they're the ones who are going to be the contributors of tomorrow. But they can't use our dev branches and our regular releases are too slow. So income-running releases. Now, everybody hopefully knows what a rolling release is, but generally speaking, a rolling release has no release schedule. Most, if not all packages are frequently updated. Some distributions like to be partially rolling. So, you know, have a frozen bit, but I'll be talking about more on that later. And updates are delivered when they're ready. And of course, when they're ready is a nice quirky definition. So, you know, when is something ever ready? And the main examples that I can think of with this are, you know, obviously Arch Linux and Gentoo kind of populized the whole thing. And I'm here to talk about Dumbledore. Quite often the feedback I hear about rolling releases though is it's unstable. You know, it changes too much. I can't trust it. Even if I do try it, it's unreliable. It breaks too often. Or it's just hard to live with. There's too much extra work I have to do to keep up with this rolling thing. And when it comes to the sort of the stability side of things, and in this case I'm really talking about stability in how fast is something moving. The, you know, I'm talking rich about, yeah, something always changing. You know, that's the point of a rolling release. You know, you want to get that new software that has to be moving at that pace. These up streams are moving faster and faster. We need to be keeping up with them. Or in some cases maybe even going faster than they are. That's the point. But then you've got to deliver it to your users and your developers who need to use the damn thing. So it has to be built and tested and then integrated together in a consistent way. And this is a problem that I don't think many rolling distributions really get a complete handle on. There's a tab in distribution sites to think of, oh, we're just putting together a bunch of packages and then it's a collection. You know, we're shipping an operating system that contains a bunch of packages, but we have to get all of that integration right. And that's more true with a rolling release than anything else. And also that also means finding a way to shield those users. Not from breakage. I'll talk about that next, but from those behavioral changes, you know, upstream has made some crazy decision to change something in their application. What if the user who's got his project to do tomorrow doesn't know how to do that yet? You know, doesn't know how to use that new feature or, you know, deal with those new defaults. It has to be some way of shielding those users from the unexpected behavioral changes just so they can get on with the work they've got to get on and then learn the new feature when they have the time to sit down and learn it. On the reliability side of things, rolling releases have typically get accused of always breaking. You know, it's got thousands of moving parts from thousands of different upstream projects. All that working together, especially when, you know, those upstream projects have dependencies on, you know, somewhere in between and they're moving at different places is a constant challenge. Just like I said, when it comes to the stability side of things, you know, we have to build this together. We have to test it together and we have to integrate it as one consistent distribution. This is not a piece of parts. The rolling release has to be everything done as a consistent distro. In practical terms, that means testing at the submission. So each submission being tested to make sure that isn't going to break everything. And then also not forgetting about testing as a whole. It doesn't matter if that one package is fine. If when you put that in, it breaks 20 other things. So testing the entire distribution, including all of those changes in one go. And obviously, users shouldn't be shipped something that doesn't work. You know, this isn't a dev branch we're talking about here and really dev branches that break aren't that useful anyway. The thing has to at least work in some basic functional sense. And then I looked at how other distributions deal with testing. Most of us deal with testing. We used to deal with it this way. The concept of sort of passive testing, you know, we put it in some testing branch. There'll be in testing, you know, whatever. And just leave it there for a couple of weeks and trust that the community will go away and test it. And then if they haven't found any bugs, that's good enough, we ship it. It's madness, but we all do it. But it's absolute madness. And it works better or worse depending on how big your community is. But it's still playing Russian roulette. You know, okay, you've got a huge pile of testers. Okay, you've got more chambers in the gun. You're slightly less likely to get your head shut off. But if you've got lots of developers, you've got more bullets in the gun. Eventually someone's going to get shot. So for a rolling release, you need to actively confirm this thing will work. Does it work in isolation? Does it work in a broader context? And does the behavior change in the user, you know, the way a user expects, you know, you've won it. You need to shield the users from that. You need to at least be able to warn them, okay, this package is going to completely change how these commands work. This UI element is going to change, you know, you need to be aware that this stuff is, you know, going to start hitting your users before it hits them. But you need to have all those answers, ideally before upstream have even finished their release. Or at least as fast as you can afterwards, because you need to be shipping this stuff as fast as you can after the upstream projects have finished what they're doing. And this is really the ultimate challenge of the whole thing. And well, yeah, the solutions are hopefully quite interesting. And then talking about things hard to live with, you know, when you look at rolling releases, you know, Arch have a great philosophy. That I really agree with, you know, do it yourself. It's a learning exercise for the arch. For Arch, it works really, really well. The Gen2 way, they don't have a nice definition, but that's kind of my variant of it. You know, do it yourself and then you've got time to read the ArchWiki to figure out what's going on while the whole thing's compiling. And with both of these distributions and a lot of others, you know, if something goes wrong, that's it. You're on your own. Good luck. Get out VM or Emacs. Yeah, you might get lucky to fix it. But who said rolling releases had to be difficult? You know, those fit for their users fine, but the problems are much broader than that. And I think the solution is actually much, much broader than that. And this is where Tumbleweed came in. Now, I'd love to be to take credit and say it was all my idea, but it wasn't. Tumbleweed initially started in the OpenSuser community by Greg Crow Hartman. He was working at SUSE at the time. And what SUSE was doing with our, and OpenSuser was doing with our traditional releases was way too boring for him. You know, upstream kernel hackers, you know, you need something completely different. And it originally started as effectively an add-on to the original OpenSuser distribution. So we started with the base system and did rolling stuff on top. Now, over the last couple of years, it's totally transformed from that model to what I'm going to talk about in a minute, where it's now moving at the pace of contribution. The entire code base will move as fast as our contributors can get those stuff packaged. And we have a process and the tools and the technology to move that incredibly quickly with those upstream projects as long as we have the contributors to do it. It's continuously tested. And when it comes to, you know, who were expecting to use it, well, basically anybody who comes to Faststamp. I mean, you know, you are the target audience for Tumbleweed perfectly. As I mentioned with the original Tumbleweed, so this is before 2014, it was a rolling update repository that we added to an existing, the existing OpenSuser distribution. So you started with the base system. We released that every six months or eight months or a year, depending on how things were going. And then you'd, you know, get a roll on top of that. The focus obviously with Greg, he was interested in the kernel, his desktop environment of choice. And, you know, the community added other things as well. So it's basically sort of a bit of user space and a bit of kernel space. And most of the stuff in between, we just relied on the stability, the stable base to work fine. And if it didn't work fine, then Tumbleweed would quite often just overwrite those packages with whatever it needed in that, in that rolling bit. So, you know, it was a FrankenSuser. It really, really was, you know, it was built that way by design, partly because it was easy to maintain that way. And then of course, every, every eight months, we'd do another OpenSuser release and then have to reset the whole thing to zero again and start from scratch, which was a huge problem. So the partially rolling, the FrankenSuser approach, it just did not work. We tried it for several years. The main problems we had were packages breaking sort of over that chasm between the stable base and the rolling part. You know, some dependency requirements or whatever, you know, just weird behavior issues appearing out of nowhere, weird dependency issues, seg faults you couldn't believe. And of course, you can deal with those issues one by one and then you ad hoc replace those broken parts. So the tumbleweed rolling stack on top gets a little bit bigger and replaces more from your base system, but then your stable base is no longer stable. So you've defeated the point of trying to be partially rolling. And then this reset to zero meant you'd be happily rolling along and then we do a new major release and you're back to square one and starting again, which, yeah, it was just painfully disruptive for users, you know. So often we avoided rolling back package versions, but it was still a complete script because he really had two different communities working on it. The stable guys working on the stable distro and the tumbleweed guys packaging things slightly differently and, you know, targeting their audience slightly differently. And looking at this, the open SUSE team at SUSE spent a really long time analyzing how to actually fix sort of all of these problems properly. I mean, they started first looking at our dev branch and looking at how do we get that more stable. I kind of started looking at it from the other side of, okay, how do we get tumbleweed more stable? And we kind of ended up meeting in the middle. And we found what I really think is a universal truth that when you're building a distribution with, you know, several thousand moving parts, if you need to be ready to move any part of that, you know, any package anywhere, you've got to be prepared to move everything. You've got to have the tools, the techniques, the processes that, okay, that new dependency from upstream is going to require you to, yes, change your init system and your kernel and God knows what else. And you have to be prepared to do that at a moment's notice. And if you're not able to do that, you're not able to do a rolling release consistently. You might be to start for a short period of time. Sooner or later, you'll hit a brick wall and stop and have real pain. So to get this solved properly, you need to always be ready to change everything at a moment's notice. Thankfully, in open SUSE, we had two ACs up our sleeve. The open SUSE build service, which is what we've been using for years for building all of our distributions. And it really has always embraced and tackled this idea of distribution is a cohesive product. You know, we build all our packages there, but the build services are aware of all those dependencies. If it needs to rebuild part of those packages, it does. So you can rebuild part of the dependency tree. It's dependency aware. It's what open SUSE and SUSE have been using for years for building our distributions. It's also cross-distribution support, which, you know, that kind of challenge of being able to build for other distros as well really taught us a heck of a lot of how to handle dependency trees, because, you know, Debian does it way different from RPM and, you know, Arch. We even had one guy who passed in Gen2 support for the build service, which was just pointless but beautiful. And, yeah, it's used by, not used by others, you know, on cloud Linux foundation, use it for Tizen, VLC, plenty of other people use it too. So we had the tools for building and, you know, really kind of helped with that rolling stuff already, but you've got to test the thing as well. And, yeah, my other job when I'm not doing open SUSE stuff is working in QA. And, yeah, open QA is a really nice testing tool for this. We have a boost-out test for this weekend. Unlike every other testing tool I've ever found, this tests from a user's perspective. You know, we're not necessarily caring about the code. It's API neutral. You don't have to teach it what the API needs to do. You tell open QA what is the user expected to do with this software. What commands are they going to type? What buttons are they going to click? Because open QA is also graphically aware, so we can actually look at the screen. We have open CV running on there. It can even actually do screenshot analysis or partial screenshot analysis. So is this UI element on the screen, whatever, or where it needs to be? And, yeah, therefore you can test the entire distribution end-to-end from the way the users are going to use the damn thing. Without this, we couldn't do tumbleweed the way we're doing today. And, in fact, it's adoption in a serious sense really began with tumbleweed. And since then, we've now started actually using it for the more stable distribution of open SUSELEAP and for the enterprise products at SUSE, which are all testing using the same framework. In terms of what it can test, it's pretty much anything. You have a nice dashboard with hundreds and hundreds of scenarios. I mean, I got to A to E here on one example. And, in fact, it's a really bad screenshot because I've missed out four of the columns, but it's also got support from multiple architectures. So here we're testing Intel, but we can also test ARM, S390, Power, 32-bit Intel if you're interested. Mostly based on VMs, although we have support now for testing on bare metal as well. So what it normally does, we'll start up with a distribution, fire it up in a VM, start testing it the way a user is going to do it, start the installation, run through the install, installs finished, start up the applications. If you want to feed it a disk image already of an existing installation, then you can do nice, fancy stuff like doing upgrade testing and migrations and all that kind of stuff. Has support for multiple machines, so you can do cluster tests and other weird and wonderful stuff with that. And, like I said, it's graphically aware. So when something goes horribly wrong, you can actually see, okay, I was expecting a tumbleweed, a nice dark blue screen on the left there, but in fact, the system came up with the wrong graphic. So even those tiny behavioral changes, which they're not bugs, it's just something that's changed upstream, get caught by OpenQA. If everything else has stayed the same way, if the UI is still working in the same way, OpenQA would just ignore it, it's fine, but if something's practically changed, if the buttons moved, the commands changed its function, OpenQA will always find it. So with those tools, we've developed this process for effectively continuous integration for an entire distribution. When you're submitting something to OpenSuser Tumbleweed, you're submitting it to an OBS project called Factory. That then goes into an automatic review queue, the usual kind of RPM-linked stuff of, you know, checking basic sanity checks and unit tests and that kind of stuff, nothing too fancy. But for any package that has a potential system-wide breaking impact, which is pretty much every package, we have a process called staging, where OpenQA will, sorry, OBS will spin up a variant of factory, well, variant of Tumbleweed, with everything that was working in the last release, plus these changes and nothing else. So basically, what if distribution, what if we merged this and we run a bunch of sanity tests in OpenQA. So actual proper smoke test, booting this thing up and installing it to make sure the distribution still works. And that takes about 20 minutes. It's not a long, complicated process, so something gets checked in. But this is before any human being has really actually looked at the code. No one's actually reviewed anything yet. No one's seen it. Does it work? Is it worth looking at? If it doesn't, it gets rejected. Dev has to go away and start again. If it passes, though, then it ends up in the review queue. Then, you know, your usual kind of code reviews, spec file review, checking everything works. Then it ends up in factory, where all of this stuff is then put in a single repository. If we have lots of check-ins at the same time, you end up with a big pile of changes. And then we test the entire distribution end-to-end in a few hundred different combinations on every architecture that we're interested in. That takes about, depending on the hardware, six to seven hours now. We've got more hardware, maybe shorter. So it's all very highly parallelized. And that will run through all those tests. The QA is therefore totally automatic. We actually tell OpenQA what the last acceptable failures were. Nothing's perfect, so there are sometimes acceptable failures in there. And if that new build doesn't have any new failures, it ships. It becomes tumbleweed for in the morning whenever. It doesn't matter. So that whole process is completely pipelined and automatically done. So writing these automated tests for OpenQA totally gate the quality of our distribution. So it's great. If a user reports a bug now saying, so-and-so broke, and I was doing this, well, just take that, translate it to the OpenQA descriptive language. You're never going to break that again. So every bug report becomes a nice easy test case in theory, and you can shield yourself from ever making the same mistake twice. At a relatively light cost, because you're already booting up the VM, so it doesn't really take that long to pop in, do what the user's doing, and done. So functionality checks, all that kind of stuff, totally able to be shielded with it. But that also means if you do listen to my talk now and do download tumbleweed and find something you don't like, contributing to OpenQA is the easiest way of fixing that. Because as soon as you put a test into OpenQA, it's never going to happen again. So the main website for OpenQA is open.qa. Documentation's there, and we have a nice little bug tracker, feature tracker for that. If you want to know more about the whole sort of pipelining and using OBS and OpenQA for continuous integration at 1.30 today, which of course clashes with my other talk in Janssen, but you can pick which one you like more. My colleague Christian is talking here about how you can use this approach in your projects, so not necessarily on a full distro level, but even on just a smaller upstream project kind of thing, you can use these tools to build your package in OBS, test it in OpenQA, follow the same kind of pipeline. Normally when I talk to people about this, they say, that's really fancy, but I don't want to wait for all that testing. I just want it now. It's too long. And in fact, there is one guy working at SUSE who does rebuild all of factory just so he can avoid OpenQA. He is crazy, but he does do it. In reality though, let's say working with upstream projects, especially these really demanding ones like GNOME and KDE, the process keeps pace with them. Upstream GNOME releases their tarbles, and within 48 hours, upstream GNOME is in Open SUSE Tumbleweed. Faster than ARCH, faster than Gen2 even. Most people don't get it compiled that quickly. So the process works even with these big upstream projects with hundreds of moving parts. In the case of KDE Plasma 4.9, which came out last week, we actually shipped it on the same day. That sounds really impressive. In reality, KDE kind of cheap because they give the tarbles out five days before. But yeah, we shipped on release day. It's kind of cool. We've also taken that process, and of course we've got the build service so we can do all this fancy stuff. And we actually have derivatives of Tumbleweed that kind of make that constant what if all the damn time. Because OBS is get aware so we can actually attach it to GNOME Git projects or KDE Git, and then build GNOME Next or Open SUSE Krypton, which are constantly building those packages as soon as something gets checked in, and constantly testing those packages as soon as something gets checked in, and then testing the entire distribution. So we get to file all these upstream bugs before anyone else does because we're seeing it in a context of an actual distribution all of the damn time. But on the border Tumbleweed side, every week now our Tumbleweed release manager Dominic writes a report of what's changed this week. So the bots that do all the magic of the pipelining automatically pump out release reports and change logs of everything that changed. But that's just ridiculously long. So he tries to summarize it to something humans can do. And he made the mistake last year of saying it's been a really quiet week this week. So then I went through those logs and figured out what does quiet mean. This was last year. We did three snapshots. Snapshots are basically Tumbleweed releases. So that whole pipelining end to end, a new build, three types. So three new sets of ISOs, three new repository versions, all in there. Those versions all together contain 146 updates, restore shipping DVDs or ISOs at least. So a bit of shuffling around of packages, 15 new packages on there, 38 packages out to make room for it, and one new kernel. That's quiet. That was a year ago. Six months ago, this was it. Five snapshots. Every single workday we shipped another snapshot. Twice as many packages. Twice as many packages on the DVD. Twice as many we had to take out and twice as many kernels. It's now six months later. The last decent run through we had. We had one run through 15 days of nonstop Tumbleweed releases. You know, every day a new snapshot. I didn't bother doing a slide for that because I can't be bothered to read that many logs. I mean, it's just crazy. The process really, really works and you can do a voting distribution across the entire code base without any problems. It just works from the user side of things, though that can then be terrifying because they're like, okay, all this stuff is changing. How did they deal with that? And okay, open QA is testing it for a practical point of view. It works. But it's still changing. I mean, you know, the behavior is still going to change. Luckily with openSUSA, we've got SQL source. We use BTFS by default and we have it by default with a tool called snapper set up. So on a standard Tumbleweed machine, of course you can disable it, use whatever file system you want. But in our normal configuration, every single time you install a package on a Tumbleweed machine, you have a snapshot before and after. So if that package changes something you don't like, just roll back to the snapshot before. Keep on working. You know, go back and learn it, how it works, you know, whenever you have some time. You're completely immunized from those, from those behavioral changes. You're also completely immunized from us screwing up your package accidentally because, you know, you can always roll back to how it worked yesterday. And even if we completely balls everything up and open QA completely screws up and your machine doesn't boot anymore, you can still boot to that snapshot from Grubb. So there's no risk as long as you're using BTFS and snapper. And there's very little risk of us with that because we're using it so much it just doesn't break on our systems. So Austin will be talking about this more this afternoon, actually more talking about sort of the next generation of this idea of not just using snapshots as a safety net, but actually, you know, more of as a atomic way of delivering updates. So actually, you know, doing transactional updates. So, you know, this concept plus plus, but still using RPMs. So that's tumbleweed. What about the Dev branch? What about open-suitor factory? You know, I've mentioned it's got a part in the process. But in reality, once we started doing this and delivering tumbleweed to users, we realized we don't need a Dev branch anymore. Factory no longer can be used by a human being. It exists in OBS tucked away and we never publish it. We don't have a traditional Dev branch anymore. It might be broken. Open QA or tell us if it is. We won't make it tumbleweed. If it worked, it's tumbleweed. So the old Greg's tumbleweed, we basically took the name, the old factory, we put added testing and then we called this a new tumbleweed. So it's a bit of a new distribution. And yeah, no more Dev branches. We're rolling on for that. And it serves that same purpose while also serving all those other purposes of rolling release does of, you know, hitting a bigger user base, being more useful, dealing with upstream projects, etc. But what about regular releases? I mean, you know, there's still a use case for that. Maybe not for the typical FOSTEM audience, but, you know, other people, you know, don't want to necessarily have everything changing all the time. Well, in Open Sousa, we kind of killed that idea off as well. Partly because of declining interest in the concept, but also because of an alternative concept that's way more interesting. We now have a distribution called LEAP, which starts with the Susan & Lytzen enterprise code base, which is now totally in OBS. So we build that, you know, kind of like CentOS, I guess, but we don't limit it to be a one-to-one match. The community then contributes to that code base as well and adds whatever packages they want in addition to. So you end up with a full-blown community distribution, all fully tested or fully integrated, but with that nice enterprise code base underneath it, so it's rock, rock, rock solid and stable and much more conservative with its pace of change. So in the past, you know, everybody knew as is Open Sousa the one distribution disc project. We now have two. Tumbleweed rolling ahead, continuously updated, continuously tested, like I said, perfect for the FOSTEM audience. And then LEAP with a shared core underneath it, sharing it with SLEE and, you know, much more conservative, you know, users. Anybody who doesn't like things changing often, you know, who wants to update once a year, you know, perfect for them, Sysadmin's your little server in the corner that you don't want to touch that often for patching. I mean, you can run a rolling release on a server, nothing's wrong with that, but, you know, you're going to have to find some nice way of automating those updates. You don't have to worry about that with LEAP. Like I said, it's kind of a combination of the Open Sousa project and the Sousa enterprise side of things. And this is roughly kind of, I guess, 50 split halfway, roughly how the whole thing looks. So, you know, there is a shared core of a nice stable base system, which is also what the enterprise side is using. There's a small amount of enterprise packages which just don't make any sense in a community sense or, you know, covered by licenses, we don't want to ship or, you know, whatever, just, you know, doesn't make sense there. And then Open Sousa, LEAP has thousands of community packages sitting on top of that shared base. And most of those community packages, if not all of them, actually originate in some form or another from Tumbleweed, where, you know, it's a pure community distribution, all those packages always moving, always rolling on. The interesting question I get normally get asked at this point is, well, what about SLEE? Because, you know, everyone knows this model. It's like, you know, Fedora with Red Hat, you know, you do a Fedora release and then there's a Red Hat Enterprise release based on it sometime after. You're not doing an Open Sousa release anymore. You can't really base SLEE on something based on SLEE that all gets a little bit circular. You know, what is Sousa going to do for their next release of the enterprise product? Well, they're going to base it on Tumbleweed, not in the sense that they're going to do a rolling release, but the Tumbleweed base system that's constantly rolling is stable enough and usable enough that Sousa Linux will be basing SLEE 13 on Tumbleweed, forking from it at some point for their next release for SLEE 13. So, not only does the process work for, you know, other guys at FOSTA, even enterprises are looking at this and like, okay, that's the best way of us developing our next major releases. We don't need this old model of a regular release in between and all those hassles with that, all the work that comes with that. And when you then also tie it with the leap side of things, where leap, of course, is based on the enterprise service packs and that stuff as well, but with community stuff on top, you end up with this weird, nice, complicated sort of infinity loop of open Sousa is simultaneously downstream and upstream of SLEE, and something that the community does in the last leap version is a candidate for the next service pack, and it makes things actually way more interesting with that whole working with a corporate partner side of things, because they have a lot more options on the table and less things get less stagnant, and it also means you end up with more enterprise developers working on your code base, which is, you know, good for polish and everything working nice. So in review, if you're a developer and you want to deal with, you know, if you're dealing with any upstream project, Tumbleweed is a perfect option for you. It's going to keep up with those upstream projects, more of upstream projects than you're interested in. You're going to get the latest and greatest of everything. It just works. And if it doesn't work exactly the way you want it to, it's really easy to contribute to. You know, we've got OBS and we've got Open QA, so you can make sure we don't break it the same way. So, yeah, and of course you've got Snapper, so you can just get back on when you work in the meantime when you haven't got the time to deal with that. So it's a perfect platform to work on and work with us on, especially if you're an upstream developer as well. Tumbleweed seems like a natural choice for quite a few projects. You know, it's a distribution there. We have all these tools. You can do stuff with us first, add a nice pace, get everything checked with the latest kernel and the latest system D and the latest Python or whatever. And then worry about how those other slower distributions are going to have to deal with it. And if you want to then use our tools like OBS to actually do that building, it's perfect because you can actually build originally for OpenSuser and then flick a few switches and build for a different RPM distribution. And then, you know, add a Debian file and flick a few files and you're building for Debian. I mean, the build service takes care of all of that. So it's a perfect platform for that. And like I say, containerized applications are cool for those other distributions, but there's actually more work for you in the long run. That's what I'm talking about later. This is actually less work. The community takes care of a good chunk of this. You just have to worry about your little part on top. And if you're a user, you manage to get the latest and greatest of everything. It just works if it doesn't snap us there to take care of it. And it's a perfect place to start contributing because that barrier of entry so low, it's easy to destroy something at the build service and say, hey, I'm going to submit this. If it doesn't work, it would be open QA or kick it out right away. It's easy to get your feet wet, easy to start contributing with us. So it's a perfect platform to start with that. You don't just have to take my word for it because we've actually been looking at the statistics of how many people are using Cumbleweed since this transition from partially rolling and the dev base, the dev branch to just having a pure rolling release and no dev branch. And our user numbers are looking quite good. If you look purely from a dev branch perspective, our dev branch now has 10 times more users than we ever had. If you factor in the partially rolling distribution, it's growing at 67% a year. And really weirdly, those weird little peaks around April seem to match up with the Buntu releases. And I'm not entirely sure what that means. But yeah, kind of fun. Yeah, that's it for me. I have a bit of time, I think. So yeah, 10 minutes. So 10 minutes for questions. Yes, Lan. How do I report it back against a rolling release so that you can reproduce it? Where's the reference that I can give you so you are the same code base to reproduce what I'm trying to report? Very rarely does a problem actually live in just that. But if it happens, because we have this snapshot model, so effectively every single tumbleweed version you're using is an entirely cohesive distro, you actually have a version number for your distro in ETCOS release. It's ISO date for the day of the snapshot. So you can mention that there. We can then fish it out of OBS and run that there. What we'll normally do is just see, does this happen on the latest version? Because if we fixed it already, that makes it really easy to close the bug. So why is it rolling? You still basically have a distribution release. Yes, and another release. That's part of the whole building it consistently. Yes. Sometimes maintainers of development projects refuse to enable the tumbleweed build target. And they say, oh, that's already factory enabled. But when I contribute packages, I'd like to make sure, okay, that there are two purposes, contribute back, but my main purpose is to get the package built for OBS and tumbleweed for my running system very soon. And so what's your perspective? Should every seven projects enable OBS with a tumbleweed, this snapshot as a build target? Generally speaking, most developed projects should, well, most developed projects shouldn't need to enable it, because they should be submitting everything into tumbleweed already. So this shouldn't really be an issue in the sense of OBS in the build service. You know, you are fast, but I want to have it now. You want to forget the testing and you just want it now? Usually I contribute when I make sure that it works for me. So I'd like to have the package now for this particular template with snapshot. Not factory with that diff of potential day. Yeah, in that case, persuade them. If they get stuck, call the board. I mean, we can help out with that. There's no problem from a build side of things. Yes? We were producing to test passively with the users and now you use OBS with the writing of the test and what the professional contribution versus what Susie is doing. Well, yeah, the question was, you know, we were passively testing in the past. We're now actively testing who's writing those tests. You know, is it Susie or the community or whatever? Because open QA adoption really started in the community, most of the tests were open SUSE tests first. And it's only in the last year or two where SUSE has actually been sort of actively contributing to open QA. So it's a community project first, which now has enterprise guys working on it too. So most of the tests come from that kind of original group of original contributors and now the enterprise side of things. The process is totally open. All the tests are in GitHub. So even for the enterprise stuff, actually, we've left that all open as well. So relatively easy to write a test submitted in. Generally speaking, I think there's no objection to this, but generally speaking, we don't get that many contributions from individuals outside kind of, I guess, the kind of tumbleweed release management loop. Like people who are actively sitting there working on the factory process contributing a lot, maybe the release manager for tumbleweed. They're the ones who see the things break initially before anyone's actually got it because we haven't shipped it yet. And they'll just throw together a test and patch it in. So I'd like to get that a bit more broad. So that's kind of one of the reasons I'm here is if there's stuff in tumbleweed, you download it, you try it and you just don't like it. And it's missed in the current process. That's why I've got those links in the slides so you can broaden that out a little bit because we don't scale. But the process does. Yes, Len. As someone working on an open source project that tries hard to support the most prominent distribution, I'm not sure if we would be able to cope with that pace and making sure that on the one hand, our project and code based on this rolling release is still being able to support all the other more common and slower distributions. If you're a project depending on certain frameworks and they change, you have to come up with lots of quirks to be able to support the latest version of the framework for tumbleweed, but still being backwards compatible to the old stuff in those traditional distributions that have a large use of this sort of... It's a challenge. It's a challenge. But then when you also look at the trend of even enterprise distributions playing around with the idea of things like modules and certain parts of enterprise distributions moving faster than others, it's a problem that everybody's going to have to deal with at some point. The reality is we can't... The regular model doesn't quite work. What I actually think the solution might be is taking this process and finding some way of defining a more moderate cadence. Tumbleweed runs as fast as we can contribute. Maybe there's a rolling release model that fits that problem nicer of something that runs a little bit slower to fit, but everybody else can keep up with it. Yes. How big is the gap between the traditional releases and the tumbleweed, especially when you want to distribute in 42 or 3? Yes. The stable kernel based on one's less curve. Yes. And tumbleweed is constantly moving. Yes. Everything. So I'm more interested in the next stable release or to the future tumbleweed. The gap is large and admittedly does get larger. It gets larger the more of these service packs are. It's a fact of life. That stable base isn't going to move that fast. However, if you look at, for example, how SUSE has done SLEE 12 compared to SLEE 11, the amount of change in each enterprise service pack has been significantly more and faster than in the traditional sort of SLEE 11, SLEE 10 and earlier era. Part of the reason SUSE has been able to do that is because Open SUSE is doing what we're doing with LEAP and tumbleweed. So, you know, things like jumping a new kernel version in service pack 2 isn't quite so terrifying when tumbleweed already was running it for a year. So, yeah, it's, so you see this kind of balance of enterprise distros catching up because the rolling release is going really fast. Him first. Sorry. Yes. So, you talked a lot about the advantages of this release model. Yes. If you have to change one thing or improve one thing in the tumbleweed release model, what would it be? That's a really good question. The one thing I would improve right now is actually have a delay between building and actually publishing. Right now, as soon as it's built and tested and everything's fine, we're publishing it. We need to put a delay in there because now we have all that crazy number of users. Our mirror infrastructure can't keep up. So, if you're one of those poor guys who hits that new snapshot first thing before our mirrors have synced, yeah, it's going to be slow. So, we've really got to find a way of kind of teaching the build service to, okay, it's ready, push it out to all the mirrors and then flag it up for everybody to use. And that's on our list at the moment because it's painful at the moment. Okay, I'd like to add something to this because I run several VMs with tumbleweed and I think one improvement could be to have an easy way to have a caching proxy. If you run several systems and you have a caching proxy to save your bandwidth. We actually have instructions on the wiki how to do that already. In the mirror guides actually. We have an asking server you can do all of that. Another question would be this new release model with the core 13, change the or improve the confusion about the because on the factory mailing list, even others, even people more familiar with all this are sometimes confused of what to take from leave and what to take from, you know, and then they say, okay, there's a list. It's called from sleep. Yeah, from factory, but then you also see, oh, there's a full from older. Yeah, there are pretty much confused. Part of the reason for that confusion when we, you know, this is all a relatively new concept when we first started leap, we purposefully didn't define where this line was between the shared core and the community stuff because we want the community to help define that. So, yeah, we, you know, 40 that's the safe bet push the factory push in tumbleweed it'll always work in it'll always get to where it needs to go. But the in the case of 42.1. Yeah, we were purposefully leaving it very open to divide that line around 42.2. We made it way.
that rolling releases are the future of the next release is. You know, most Linux distributions for not just distro developers, but any general upstream, and all of the latest versions of all develop for distributions in particular. This And having a regular release with all that upstream developers. Everybody likes to talk you know, users. But there are enthusiasts. why we end up keeping those users and why we them, okay, this package is going to completely fast as you can after the upstream projects have broader than that. And I think the solution is now moving at the pace of contribution. The entire code base will move as fast as our audience slightly differently. And looking at okay, that new dependency from upstream is going of being able to build for other distros as well distribution end-to-end from the way the users already of an existing installation, then you can with upstream projects, especially these really just using snapshots as a safety net, but Anybody who doesn't like things changing often, stable base system, which is also what the going to do a rolling release, but the Tumbleweed So, not only does the process work for, you know, interested in. You're going to get the latest and everything checked with the latest kernel and the perfect platform for that. And like I say, been looking at the statistics of how many people development projects refuse to enable the most developed projects should, well, most faster than others, it's a problem that else can keep up with it. Yes. How big is the gap between the traditional releases and the