Stable kernel maintenance for distributions
Hutchings, Ben
Each Linux kernel release is followed by a series of stable updates fixing bugs, hosted at kernel.org. Linux distributions generally use these as a basis for their packages. I describe how the stable update process works and how distributions can be more involved.
 Hello, I'm a member of the Debian kernel team and I'm also involved in kernel development upstream and currently I maintain the 3.2 stable branch. So it's sort of a summary of how Linux stable releases work at the moment, probably most of you know most of this. After every stable release from Linux 3.x there are a series of updates to that, usually by Greg Crow Hartman, number 3.x.y. And these will have cherry picked bug fixes, sometimes new hardware support if it doesn't take a lot of, it doesn't take a very big change to do that. Sometimes the fixes need to be back ported because the kernel APIs have changed. And these updates continue at least until Linux releases the next, then it makes another release. And some versions are used as a basis, some Linux versions become the basis for a long term stable branch which can be maintained for several years. So that's how the stable update process works. The mailing list for this is quite high traffic, stable.vgo.kernel.org. We commit to Linux that has a CC to this list in its commit message, ends up being, well it's a candidate for inclusion in stable updates. Greg runs scripts that will actually copy those to your list once Linux has pulled those changes into history. And it also reports, did it get applied to certain branches, could it, and if it failed because the code around this, the code context has changed. And anyone can nominate a change that ought to go into stable updates once it's reached Linux's tree as a file in the source tree documentation slash stable.kernel.txt and that explains what the rules are for what's acceptable. And occasionally there are regressions because some change that got applied to a stable branch didn't really belong there, or it should have been adjusted and it wasn't. But otherwise everything has to be in Linux's tree first. So the maintainable stable branch will collect up changes for some period of weeks and will then send these out as a batch for review before making a release. Those go to the mailing list, they also get sent to everyone named in the original commit. And the review is supposed to catch things like this isn't needed here because the bug doesn't exist or this change is only going to work if you take these other changes that it depends on. Or this is called regression when we applied it in mainline and it's not yet fixed there, in which case once the second fix is available we can take both of those into a stable branch. Following the review that list of changes might need to be revised and then the maintainer can make a, can apply them to the git branch, tag it and if the maintainer is not Greg then they need to get Greg to pull and he then uploads to kind of org. So what does this mean for distributions? How many people here work on their distributions kernel packaging? That's a little bit. Okay. So as a package maintainer of course you want to get bug fixes as soon as possible, at least if they're important enough for the stable updates. And also probably you get bug reports directly from your users, you report them upstream, maybe you find a fix that's already been made and needs to be back ported, maybe you can write fix and stable updates should be a way of showing those fixes between distributions. So you, do you send fixes to, do you nominate fixes for stable updates? Do you know about the process? Yeah. So I once had a fix but it didn't apply to the stable kernel because we had another patch that was needed before. And was it clear how to fix that up? You know, you should fix them or you fix too big. Possibly but I didn't invest them for doing it because I'm not really a kernel developer. Okay. Yeah. Was there anyone else who's actually active in kernel packaging? Okay. This isn't going to be very interesting talk then, isn't it? We're hoping to have more response from the audience. So yes, so distributions have stable releases that normally get maintained for a period of years and not for the two or three months between kernel releases. So the, that means though that a long term stable branch is quite important or useful at least. In fact, still active today, the Linux 2.6.32 branch was used as the basis for a bunch of long term supported distributions. And that seems to have worked out quite well. Lots of fixes. I counted over 3,500 changes went into that branch. Mostly about fixes. But there hasn't been quite as much. The other long term branches don't have, that seems to have been shared between so many distributions probably because release schedules didn't really line up. But also because there's a lot of, maybe they could have lined up if we'd actually talked about plans and tried to settle on a common base version for releases. So I have no one to ask now except for myself. I certainly want there to be a long term stable branch for Debian releases, which is why I wanted to maintain the 3.2 branch long term. And I'm not the only person doing that. So it's open to anyone who's involved upstream, who's involved in the stable process to take on one of those branches if it's useful for their, whatever they're working on, whether they're in a distribution or embedded systems or wherever they need as long term support. So the documentation in the kernel of source tree is kind of the FAQ for stable updates. Anyone else have questions? So how often do you expect that long term stable kernels are forked off the main line? Well, the long term branch is then just a continuation of the usual stable branch. So whenever, so Greg will pick one per year, but anyone who wants a long term, who wants a long term, once another stable branch to be long term maintained can volunteer to do that. They wouldn't necessarily be accepted. They have to be seen to be someone who's competent to do that. But if you're seeing actively working upstream, then you should be okay. Josh? The number of changes in the 2.6.32 branch is pretty impressive. I mean several commits per day on average, which is really large for a project in itself. It's locked down quite a bit recently. What do you expect for the 3.2 we use for WISI? Do you expect a big flow of changes that will lead to good usage for this kernel in the long term or should it be less useful? There's certainly quite a high rate of changes at the moment. I have 100 also pending for the next release. That's been collected over a period of two to three weeks, I think. At the moment, there are actually a lot of distributions using 3.2. That's, I think, mostly debilitating the Buntu derivatives. I believe the Buntu is in fact decided to move to 3.5 probably because of hardware, new hardware support, which if any sort of trivial changes like adding a new device ID to a table, they're 5 a stable, but quite often new hardware support is going to involve writing a lot of new code that doesn't go in there. That's one reason why that's one thing that stable Buntus don't cover and which distributions currently have to do, always have to do for themselves unless they're going to move forward to a new release. Currently, the BN is alone with 3.2? No, currently, the BN is alone with 3.2. Currently, the Buntu is still at 3.2, but I believe they're going to change in a few months. Yes, they're still going to change. So they're still going to move the LTS to a new kind of version. In fact, this is also something that Susie has been doing with ServicePacks. Question on whether, say, hardware developers say designing new SOC support, should they go with a stable kernel or follow the upstream? I'm not sure what you mean. It sounds like you're talking about addings. Are you talking about the development? Obviously, development has to go into this as tree. New development has to go into this as tree first. Are you talking about something like what you would use in a board support package? It's just that I'm seeing a lot of the work I do is with vendor-supplied boards, and a lot of them are taking the long-term support and putting their stuff on that as a start and then supplying that to their customers. Are the patrangers not going upstream? In some cases, the stuff doesn't get upstream in the end. I saw this a lot, my last example is with the Marvell stuff, which they are finally getting upstream, but I've seen several vendors do this with taking a long-term supported kernel and providing that to their customers. This is what you should use on our hardware. That's not terrible, but they really should be getting the changes upstream as well. Comment on the... As we're working with some ARM SUCs and seeing some community boards there about every vendor that really cares about getting his stuff not only sold to the industry, I think should really go for the mainline because those vendor-supplied BSP packages tend to be horribly outdated after a year or so. You cannot reproduce it. You will never get that upstream. Remember Thomas talked from yesterday very good concerning the Marvell socks, where he said Marvell hired us to get their support into mainline, and we never ever even looked at their BSPs. We just redid everything from scratch because the vendor BSPs are outdated, broken, whatever. For me, the only way is to really stick to upstream and get things in one by one. Sorry for... That's not meant to disregard your work concerning the stable currency. No, I'm honest. I'm in agreement with you. Thanks. Hey. I know there's at least one maintainer who actually maintains his own stable tree, which is the networking maintainer. Do you are not too afraid there's going to be... Who maintainer? David Miller. Oh, yes, yes. Do you... Well, he maintains his own stable queue. He takes responsibility for nominating networking fixes that should go to stable. You can also write to him to nominate him, but I mean, he's taking this sort of authoritative in that area. Any subsystem maintainer could insist on doing that if they wanted. So is that something that you welcome as an approach? Does that offload you from a lot of work? Somewhat, yes, because he's taking care of backforting. That does take some work off me. Any more questions? All right, then I think I might be done here. Okay. Yeah. Yeah. Yeah. Thank you.
 Hello, I'm a member of the Debian kernel team and I'm also involved in kernel development  upstream and currently I maintain the 3.2 stable branch.  So it's sort of a summary of how Linux stable releases work at the moment, probably most  of you know most of this.  After every stable release from Linux 3.x there are a series of updates to that, usually  by Greg Crow Hartman, number 3.x.y.  And these will have cherry picked bug fixes, sometimes new hardware support if it doesn't  take a lot of, it doesn't take a very big change to do that.  Sometimes the fixes need to be back ported because the kernel APIs have changed.  And these updates continue at least until Linux releases the next, then it makes another  release.  And some versions are used as a basis, some Linux versions become the basis for a long  term stable branch which can be maintained for several years.  So that's how the stable update process works.  The mailing list for this is quite high traffic, stable.vgo.kernel.org.  We commit to Linux that has a CC to this list in its commit message, ends up being, well  it's a candidate for inclusion in stable updates.  Greg runs scripts that will actually copy those to your list once Linux has pulled those  changes into history.  And it also reports, did it get applied to certain branches, could it, and if it failed  because the code around this, the code context has changed.  And anyone can nominate a change that ought to go into stable updates once it's reached  Linux's tree as a file in the source tree documentation slash stable.kernel.txt and  that explains what the rules are for what's acceptable.  And occasionally there are regressions because some change that got applied to a stable branch  didn't really belong there, or it should have been adjusted and it wasn't.  But otherwise everything has to be in Linux's tree first.  So the maintainable stable branch will collect up changes for some period of weeks and will  then send these out as a batch for review before making a release.  Those go to the mailing list, they also get sent to everyone named in the original commit.  And the review is supposed to catch things like this isn't needed here because the bug  doesn't exist or this change is only going to work if you take these other changes that  it depends on.  Or this is called regression when we applied it in mainline and it's not yet fixed there,  in which case once the second fix is available we can take both of those into a stable branch.  Following the review that list of changes might need to be revised and then the maintainer  can make a, can apply them to the git branch, tag it and if the maintainer is not Greg then  they need to get Greg to pull and he then uploads to kind of org.  So what does this mean for distributions?  How many people here work on their distributions kernel packaging?  That's a little bit.  Okay.  So as a package maintainer of course you want to get bug fixes as soon as possible, at least  if they're important enough for the stable updates.  And also probably you get bug reports directly from your users, you report them upstream,  maybe you find a fix that's already been made and needs to be back ported, maybe you can  write fix and stable updates should be a way of showing those fixes between distributions.  So you, do you send fixes to, do you nominate fixes for stable updates?  Do you know about the process?  Yeah.  So I once had a fix but it didn't apply to the stable kernel because we had another patch  that was needed before.  And was it clear how to fix that up?  You know, you should fix them or you fix too big.  Possibly but I didn't invest them for doing it because I'm not really a kernel developer.  Okay.  Yeah.  Was there anyone else who's actually active in kernel packaging?  Okay.  This isn't going to be very interesting talk then, isn't it?  We're hoping to have more response from the audience.  So yes, so distributions have stable releases that normally get maintained for a period  of years and not for the two or three months between kernel releases.  So the, that means though that a long term stable branch is quite important or useful  at least.  In fact, still active today, the Linux 2.6.32 branch was used as the basis for a bunch of  long term supported distributions.  And that seems to have worked out quite well.  Lots of fixes.  I counted over 3,500 changes went into that branch.  Mostly about fixes.  But there hasn't been quite as much.  The other long term branches don't have, that seems to have been shared between so many  distributions probably because release schedules didn't really line up.  But also because there's a lot of, maybe they could have lined up if we'd actually talked  about plans and tried to settle on a common base version for releases.  So I have no one to ask now except for myself.  I certainly want there to be a long term stable branch for Debian releases, which is why I  wanted to maintain the 3.2 branch long term.  And I'm not the only person doing that.  So it's open to anyone who's involved upstream, who's involved in the stable process to take  on one of those branches if it's useful for their, whatever they're working on, whether  they're in a distribution or embedded systems or wherever they need as long term support.  So the documentation in the kernel of source tree is kind of the FAQ for stable updates.  Anyone else have questions?  So how often do you expect that long term stable kernels are forked off the main line?  Well, the long term branch is then just a continuation of the usual stable branch.  So whenever, so Greg will pick one per year, but anyone who wants a long term, who wants  a long term, once another stable branch to be long term maintained can volunteer to do  that.  They wouldn't necessarily be accepted.  They have to be seen to be someone who's competent to do that.  But if you're seeing actively working upstream, then you should be okay.  Josh?  The number of changes in the 2.6.32 branch is pretty impressive.  I mean several commits per day on average, which is really large for a project in itself.  It's locked down quite a bit recently.  What do you expect for the 3.2 we use for WISI?  Do you expect a big flow of changes that will lead to good usage for this kernel in the long  term or should it be less useful?  There's certainly quite a high rate of changes at the moment.  I have 100 also pending for the next release.  That's been collected over a period of two to three weeks, I think.  At the moment, there are actually a lot of distributions using 3.2.  That's, I think, mostly debilitating the Buntu derivatives.  I believe the Buntu is in fact decided to move to 3.5 probably because of hardware, new  hardware support, which if any sort of trivial changes like adding a new device ID to a table,  they're 5 a stable, but quite often new hardware support is going to involve writing a lot  of new code that doesn't go in there.  That's one reason why that's one thing that stable Buntus don't cover and which distributions  currently have to do, always have to do for themselves unless they're going to move forward  to a new release.  Currently, the BN is alone with 3.2?  No, currently, the BN is alone with 3.2.  Currently, the Buntu is still at 3.2, but I believe they're going to change in a few  months.  Yes, they're still going to change.  So they're still going to move the LTS to a new kind of version.  In fact, this is also something that Susie has been doing with ServicePacks.  Question on whether, say, hardware developers say designing new SOC support, should they  go with a stable kernel or follow the upstream?  I'm not sure what you mean.  It sounds like you're talking about addings.  Are you talking about the development?  Obviously, development has to go into this as tree.  New development has to go into this as tree first.  Are you talking about something like what you would use in a board support package?  It's just that I'm seeing a lot of the work I do is with vendor-supplied boards, and a  lot of them are taking the long-term support and putting their stuff on that as a start  and then supplying that to their customers.  Are the patrangers not going upstream?  In some cases, the stuff doesn't get upstream in the end.  I saw this a lot, my last example is with the Marvell stuff, which they are finally getting  upstream, but I've seen several vendors do this with taking a long-term supported kernel  and providing that to their customers.  This is what you should use on our hardware.  That's not terrible, but they really should be getting the changes upstream as well.  Comment on the...  As we're working with some ARM SUCs and seeing some community boards there about every vendor  that really cares about getting his stuff not only sold to the industry, I think should  really go for the mainline because those vendor-supplied BSP packages tend to be horribly outdated  after a year or so.  You cannot reproduce it.  You will never get that upstream.  Remember Thomas talked from yesterday very good concerning the Marvell socks, where he  said Marvell hired us to get their support into mainline, and we never ever even looked  at their BSPs.  We just redid everything from scratch because the vendor BSPs are outdated, broken, whatever.  For me, the only way is to really stick to upstream and get things in one by one.  Sorry for...  That's not meant to disregard your work concerning the stable currency.  No, I'm honest.  I'm in agreement with you.  Thanks.  Hey.  I know there's at least one maintainer who actually maintains his own stable tree, which  is the networking maintainer.  Do you are not too afraid there's going to be...  Who maintainer?  David Miller.  Oh, yes, yes.  Do you...  Well, he maintains his own stable queue.  He takes responsibility for nominating networking fixes that should go to stable.  You can also write to him to nominate him, but I mean, he's taking this sort of authoritative  in that area.  Any subsystem maintainer could insist on doing that if they wanted.  So is that something that you welcome as an approach?  Does that offload you from a lot of work?  Somewhat, yes, because he's taking care of backforting.  That does take some work off me.  Any more questions?  All right, then I think I might be done here.  Okay.  Yeah.  Yeah.  Yeah.  Thank you.
Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_0_131.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_132_419.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_420_9371.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_9372_11063.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_11064_15503.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_15504_18947.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_18948_19631.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_19632_19703.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_19704_19823.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_19824_20663.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_20664_20759.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_20760_22211.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_22212_24311.png Stable_kernel_maintenance_for_distributions_wnWZ8GOxY2s_2_24312_26871.png
Hello, I'm a member of the Debian kernel team and I'm also involved in kernel development upstream and currently I maintain the 3.2 stable branch. So it's sort of a summary of how Linux stable releases work at the moment, probably most of you know most of this. After every stable release from Linux 3.x there are a series of updates to that, usually by Greg Crow Hartman, number 3.x.y. And these will have cherry picked bug fixes, sometimes new hardware support if it doesn't take a lot of, it doesn't take a very big change to do that. Sometimes the fixes need to be back ported because the kernel APIs have changed. And these updates continue at least until Linux releases the next, then it makes another release. And some versions are used as a basis, some Linux versions become the basis for a long term stable branch which can be maintained for several years. So that's how the stable update process works. The mailing list for this is quite high traffic, stable.vgo.kernel.org. We commit to Linux that has a CC to this list in its commit message, ends up being, well it's a candidate for inclusion in stable updates. Greg runs scripts that will actually copy those to your list once Linux has pulled those changes into history. And it also reports, did it get applied to certain branches, could it, and if it failed because the code around this, the code context has changed. And anyone can nominate a change that ought to go into stable updates once it's reached Linux's tree as a file in the source tree documentation slash stable.kernel.txt and that explains what the rules are for what's acceptable. And occasionally there are regressions because some change that got applied to a stable branch didn't really belong there, or it should have been adjusted and it wasn't. But otherwise everything has to be in Linux's tree first. So the maintainable stable branch will collect up changes for some period of weeks and will then send these out as a batch for review before making a release. Those go to the mailing list, they also get sent to everyone named in the original commit. And the review is supposed to catch things like this isn't needed here because the bug doesn't exist or this change is only going to work if you take these other changes that it depends on. Or this is called regression when we applied it in mainline and it's not yet fixed there, in which case once the second fix is available we can take both of those into a stable branch. Following the review that list of changes might need to be revised and then the maintainer can make a, can apply them to the git branch, tag it and if the maintainer is not Greg then they need to get Greg to pull and he then uploads to kind of org. So what does this mean for distributions? How many people here work on their distributions kernel packaging? That's a little bit. Okay. So as a package maintainer of course you want to get bug fixes as soon as possible, at least if they're important enough for the stable updates. And also probably you get bug reports directly from your users, you report them upstream, maybe you find a fix that's already been made and needs to be back ported, maybe you can write fix and stable updates should be a way of showing those fixes between distributions. So you, do you send fixes to, do you nominate fixes for stable updates? Do you know about the process? Yeah. So I once had a fix but it didn't apply to the stable kernel because we had another patch that was needed before. And was it clear how to fix that up? You know, you should fix them or you fix too big. Possibly but I didn't invest them for doing it because I'm not really a kernel developer. Okay. Yeah. Was there anyone else who's actually active in kernel packaging? Okay. This isn't going to be very interesting talk then, isn't it? We're hoping to have more response from the audience. So yes, so distributions have stable releases that normally get maintained for a period of years and not for the two or three months between kernel releases. So the, that means though that a long term stable branch is quite important or useful at least. In fact, still active today, the Linux 2.6.32 branch was used as the basis for a bunch of long term supported distributions. And that seems to have worked out quite well. Lots of fixes. I counted over 3,500 changes went into that branch. Mostly about fixes. But there hasn't been quite as much. The other long term branches don't have, that seems to have been shared between so many distributions probably because release schedules didn't really line up. But also because there's a lot of, maybe they could have lined up if we'd actually talked about plans and tried to settle on a common base version for releases. So I have no one to ask now except for myself. I certainly want there to be a long term stable branch for Debian releases, which is why I wanted to maintain the 3.2 branch long term. And I'm not the only person doing that. So it's open to anyone who's involved upstream, who's involved in the stable process to take on one of those branches if it's useful for their, whatever they're working on, whether they're in a distribution or embedded systems or wherever they need as long term support. So the documentation in the kernel of source tree is kind of the FAQ for stable updates. Anyone else have questions? So how often do you expect that long term stable kernels are forked off the main line? Well, the long term branch is then just a continuation of the usual stable branch. So whenever, so Greg will pick one per year, but anyone who wants a long term, who wants a long term, once another stable branch to be long term maintained can volunteer to do that. They wouldn't necessarily be accepted. They have to be seen to be someone who's competent to do that. But if you're seeing actively working upstream, then you should be okay. Josh? The number of changes in the 2.6.32 branch is pretty impressive. I mean several commits per day on average, which is really large for a project in itself. It's locked down quite a bit recently. What do you expect for the 3.2 we use for WISI? Do you expect a big flow of changes that will lead to good usage for this kernel in the long term or should it be less useful? There's certainly quite a high rate of changes at the moment. I have 100 also pending for the next release. That's been collected over a period of two to three weeks, I think. At the moment, there are actually a lot of distributions using 3.2. That's, I think, mostly debilitating the Buntu derivatives. I believe the Buntu is in fact decided to move to 3.5 probably because of hardware, new hardware support, which if any sort of trivial changes like adding a new device ID to a table, they're 5 a stable, but quite often new hardware support is going to involve writing a lot of new code that doesn't go in there. That's one reason why that's one thing that stable Buntus don't cover and which distributions currently have to do, always have to do for themselves unless they're going to move forward to a new release. Currently, the BN is alone with 3.2? No, currently, the BN is alone with 3.2. Currently, the Buntu is still at 3.2, but I believe they're going to change in a few months. Yes, they're still going to change. So they're still going to move the LTS to a new kind of version. In fact, this is also something that Susie has been doing with ServicePacks. Question on whether, say, hardware developers say designing new SOC support, should they go with a stable kernel or follow the upstream? I'm not sure what you mean. It sounds like you're talking about addings. Are you talking about the development? Obviously, development has to go into this as tree. New development has to go into this as tree first. Are you talking about something like what you would use in a board support package? It's just that I'm seeing a lot of the work I do is with vendor-supplied boards, and a lot of them are taking the long-term support and putting their stuff on that as a start and then supplying that to their customers. Are the patrangers not going upstream? In some cases, the stuff doesn't get upstream in the end. I saw this a lot, my last example is with the Marvell stuff, which they are finally getting upstream, but I've seen several vendors do this with taking a long-term supported kernel and providing that to their customers. This is what you should use on our hardware. That's not terrible, but they really should be getting the changes upstream as well. Comment on the... As we're working with some ARM SUCs and seeing some community boards there about every vendor that really cares about getting his stuff not only sold to the industry, I think should really go for the mainline because those vendor-supplied BSP packages tend to be horribly outdated after a year or so. You cannot reproduce it. You will never get that upstream. Remember Thomas talked from yesterday very good concerning the Marvell socks, where he said Marvell hired us to get their support into mainline, and we never ever even looked at their BSPs. We just redid everything from scratch because the vendor BSPs are outdated, broken, whatever. For me, the only way is to really stick to upstream and get things in one by one. Sorry for... That's not meant to disregard your work concerning the stable currency. No, I'm honest. I'm in agreement with you. Thanks. Hey. I know there's at least one maintainer who actually maintains his own stable tree, which is the networking maintainer. Do you are not too afraid there's going to be... Who maintainer? David Miller. Oh, yes, yes. Do you... Well, he maintains his own stable queue. He takes responsibility for nominating networking fixes that should go to stable. You can also write to him to nominate him, but I mean, he's taking this sort of authoritative in that area. Any subsystem maintainer could insist on doing that if they wanted. So is that something that you welcome as an approach? Does that offload you from a lot of work? Somewhat, yes, because he's taking care of backforting. That does take some work off me. Any more questions? All right, then I think I might be done here. Okay. Yeah. Yeah. Yeah. Thank you.
from Linux 3.x there are a series of updates to versions are used as a basis, some Linux versions that's how the stable update process works. The can write fix and stable updates should be a way distributions probably because release schedules branches if it's useful for their, whatever