Maintaining a kernel subsystem
Sang, Wolfram
Maintaining a kernel subsystem ...at least how I do it This talk provides some insight how Wolfram handles the I2C subsystem of the Linux kernel, which is one puzzle piece which simply has to work. In the Linux ecosystem, maintainers are gatekeepers. They take the ultimate decision as to which patches are accepted and they are entrusted to lead their subsystem in a future-proof direction. Discussions and decisions are public via mailing lists, but what does an actual workflow look like? Are there any special tools involved? What equipment is used? What helps in handling difficult corner cases? How do maintainers collaborate? How much time does it take? This talk provides some insight how Wolfram handles this for the I2C subsystem, which is one puzzle piece of the kernel which simply has to work.
 Well, from I'm the current kernel I2C maintainer, and I wanted to talk about how I maintain the subsystem in the Linux kernel, because, yeah, I wanted to give some transparency to that topic. And I myself thought, before I was a maintainer, I was interested in how that could work and how it could look like. And so here I am. First thing, just a few words about what I2C itself is. It is a bus used since quite some time, 82. What I found out, it was originally used inside televisions to communicate between the different chips in there. It's pretty well established. It's very simple. You have just two wires, one clock, one data line. It's not designed for high-speed transfers, but since it's so well established and so simple, it is still widely used. And just to give you an idea, here you have a clock line. So that's a master. It's always a master-slave relationship. You have a master who says how fast to talk and who's allowed to talk. And then you have a data line. And whenever the clock line is on the high level, you look on the data line, OK, here it's low. So we have a zero. Here it's high. We have a one. And again and again. So you get a stream of zeros and ones. And this is enough to communicate. You can put a protocol on that. So this first here is the address, which device to I want to talk to. Then I say I want to write. The device acknowledges that. And then I can send data. Pretty simple. It's a bus used on today's mobile phones as well. I guess it's still used in televisions. It's cheap and it's robust and it does its thing. And we have support for that in the Linux kernel, obviously. So what does the i-square-c subsystem control in the Linux kernel? It is there is a core involved, which is handling all the protocol, how messages have to be formatted, in what row they have to be transmitted. All that kind of core mechanism I have to take care about. There are buses which are similar to i-square-c, namely SM bus. We have a compatibility layer in the kernel. I also look for that. And as I said before, we have a master-slave relationship. There's a master requesting data and a slave, a sensor, a temperature sensor, or a data storage device. I take also care about the master drivers. I'm not taking care. And we have currently approximately 70 different master drivers in the Linux kernel. Because mainly driven, we just heard talk about ARM chips. And every ARM chip usually has its own i-square-c controller. They are all handled in the i-square-c subsystem. Not handled are the slave devices, the sensors. I mentioned just a few. Those are located in drivers HW and the hardware monitoring section of the Linux kernel. If you want to talk to audio codecs, you will find them here. EEPROMs, you will find them here, and so on, and so on. Currently, according to my research, more than 400 i-square-c drivers. The thing to note is, of course, there are other audio codecs or other sensors. They can be wired by SPI, USB, whatever. So it makes sense to group them into that functionality. And I don't have to take care about devices I don't know otherwise. So all sensors are here. All the codecs are here. And I just take care about the master drivers. That's a rough overview of a typical ARM system on ship, like it is used in mobile phones, tablets, whatever. And it has lots of components. We don't need to check them all. What I want to point out is i-square-c or two-wire interface, how it's also sometimes named, is one little part of it. And like in the Linux kernel itself, it's one little part of it. But we surely wanted to work to have proper to be in place and maintain support. It's not rocket science. It's not bleeding edge. But it's one important piece which should be there, if we want it. So this talk is not so much about i-square-c. It's more about maintaining a subsystem. And the question how to become a maintainer is, for me, really answered by meritocracy. It's about merit. If you start participating, if you start discussing in a certain subsystem, you can read that for yourself. Then you get entrusted. Entrusted to your opinion gets entrusted. And then you are offered, if you want that, some responsibility. And this is really, from what I think, a key distinguishment between potential maintainer and a developer. It's taking over responsibility not only for the current problem you are working on, and as soon as that problem is gone, you're moving on to the next. But some general responsibility for a subsystem or that other parts in that subsystem could be improved or cleaned up, stuff like this. My suggestion, what I wrote with Care About the Kernel, is I think it's in the long term more sustainable. If you do that because you have a feeling that this should be properly handled in the kernel, so you want to do the maintenance because you want that to be proper in the kernel. If you do that, let's say for egoistic reasons, because you want to be a maintainer, just to be a maintainer, that's a motivation which can go away if things go wrong. But if you still are interested in, hey, I want the kernel to be the best operating system in the world, and I need to go through that, that's what I figured out the more sustainable way of thinking in that regard. So the case for me is pretty boring, pretty standard. When I started hacking the Linux kernel, by accident started in the I-square-C subsystem. That was the first talk I was given by my former employer. And I was lucky to meet Jean de Veyre in that subsystem, who was some kind of mentor to me, a good person guiding me how to get into the kernel development. And I got not only interested, as I mentioned before, in the driver I was working on, I was interested in the subsystem itself. I was seeing, OK, according to what you just taught me, this part doesn't look proper. I said, yeah, that's all craft. I don't have time to clean that up. So I did. And over the time, over years, I was sending cleanup patches, start to look at other people patches. Although I didn't have personal interest in that, just to guide other people. This was the next step. And keeping at this, I started becoming a co-maintenor. And since last year, I was interested to be the maintainer of the iSquare-C subsystem, which is an interesting challenge, really. I can recommend it after all. So there was I, a maintainer. And so I was responsible to accept patches from the mailing list, collect patches from the mailing list, review them, and pass them onto Linus. So how do I do that? How do you do that? I was looking for some kind of how-to, but I couldn't really find one. This is also one reason why I'm standing here. There were some talks and some articles by Greg Rohartman, which were helpful in pointing potential issues I have to deal with. But when I looked at them, how things were handled in the scripts, it came to the conclusion, well, maybe I would do that different. And I think this is the answer why there's no proper how-to. This is a lot about workflow, how you do things, and what form, and like with most workflows, this is so subjective that I also would not know how to write a how-to do this, but I just can present how I do this. I will answer questions, but I would recommend every maintainer to just think about the things involved and find their own way, how they handle this amount of patches, and all the things involved. So what is involved? First, you need to track the patches. Usually, every subsystem in the Linux kernel has a dedicated mailing list to discuss the issues there or to discuss patches. And you need to track what's happening there and work on them. I used to do this with standard mail software. I used mud for handling patches, and I would really recommend mud for handling patches because other mailers tend to break the formatting and make applying harder. I used iStuff for priority settings and stuff like this that maybe could have done with mud as well, but that was the most easy solution for me, so I took it and it worked OK for a while. But I felt that it was not perfect, and I am now starting to see that I am using something called patchwork. I've seen that before, but I now have a different view of it now that I'm a maintainer. That's a net-based patch management tool. It has a command line utility, so I can use it for scripting. It has a web interface, so I can change settings there. Or users can look what the status of a patch is, and I have the impression that it will work a lot better. I want to give a short view of how it looks. Is that readable in the last rows? Better? No reaction. OK, I will just keep to this. Here I have a list of the patches which came in recently. So that released me from forgetting a patch because I didn't care about my mail folder enough, deleting accidentally. This software does not delete accidentally, so no patch gets lost. That is very convenient. And then you see on the right side, they have a status, which are all new. I have not processed them. And I just can click onto one patch. I assume, oh no, I'm not logged in. I should do that. I hate typing passwords when I'm recorded, but I can't help it right now. So what you saw before is that what users could do is that users can see the status of their own patch. Am I working on it? Have I rejected it? What not? And if I now go onto a patch like this one, I am able to change the status here to whatever I think is appropriate. And the nice thing is that I just have configured the software in a way that whenever I change it, the original patch author gets a mail. So there's also response. And for me, left is the burdens to write a mail. It's all automated. That's really helpful. And I don't want to show that. Trust me that before, this is a patch which got accidentally on the mailing list. It has nothing to do with I2C. So I don't have this does not fall under my control. So I would just set the patch to not applicable update. And the author will get a mail. And for me, the patch is marked as done more or less, because I know I don't have to care about it. Another nice feature is that I have a very good patch, another nice, see it has gone only three pages we had four before. Another nice thing is that it just not collects only the patch, but also, that's the wrong one. Sorry. This is a very simple patch. And the nice thing is it does not only collect the patch, but all the commands which were made by this patch, which is in this case just reviewed by from Jean de Verre, which is a former maintainer. I trust him very much. So I assume that the patch is OK. I have looked myself at it. And since it is so simple, I also agree it is OK. It just adds a bit of documentation and basically adds just a new PCI device ID to an existing driver. So the new item, what not, will support iSquashD as well. So I have the patch. I have all the correspondent males all in one place. What I now want to show is that there's also this command line interface. If I use the patchwork client and let it list, I hope not too much of you are using the Wi-Fi here, because it will slow down my presentation. I would get a list of all the patches, like I have had that in the, you've seen that on the web. I was, OK, here it is. And you can see here, there's the patch I've just seen in the web, and with the simple commands client gets. And this ID number, I could get the patch from a script. And with all the hooks, you can imagine, you can do wonderful things automatically, getting, accepting, setting status. This is very convenient. So if you are maybe not a Linux kernel maintainer, but some other person handling lots of patches, you need to track, this is a good idea. This is a good software. Kernel Org offers patchwork, and ozileps.org offers that, but you can also install that on your own. This is something I can really recommend. The next thing is about, I've collected all the patches. Now I need to comment on them. I need to review them to give you advices. And the first thing is especially, well, you have to learn that as a maintainer as well, but also for the users, the key idea is, well, let me exaggerate. Kernel is the best thing we have, and there must be a very, very good reason to change it. So the default is, I'm not accepting a patch, unless I understand the reason why you send it. It must be clear to me. I appreciate all the effort that went into a patch, and I appreciate the idea of improving the Linux kernel, but just making a patch alone does not justify the inclusion. And if I can't see the reason to include the patch, then you or me or we both, we have to work on that until it's clear if it's acceptable or not. This is one key thing to remember. Of course, I start with a visual review. I just read the mail and reply to it and check if the concepts in that patch are proper, if the formal requirements are OK. Because I don't need to do a low level, picky review if the basics are not correct. That would be waste of time. So I check for the big picture. And if I find something there which is not OK, then I will say, OK, I will stop reviewing here. Please fix that first, or start a discussion on what the proper mechanism is. The thing which causes most arguments is what I called impression and put it in quotes. This is something developed over the years, over the time where sometimes I just have the feeling it's not cutting it. And then I may be wrong. This usually needs to discussion what the proper way to do it is. I may be convinced if there are arguments. But I dare to say most times there's a reason why I have these feelings that it's not proper. And it's not always ending up in an argument. Usually you work together and find out a really clean, elegant way of finding a problem and finding a better solution to that. The next thing is obviously compile tests. If the first part went OK, then does it even apply to the current Linux kernel? Does it build? I can only test if I have the compiler at hand for very, very strange platforms. I just skip it and trust on Linux next. But usually I have the compiler. And I run a few code checkers on the code resulted from the patch. I will say some more words about that later. Runtime tests is a critical, no, not critical. But I would like to do a lot more runtime tests. But obviously, as I said, I have 70 different drivers. And I don't have 70 different hardware at home to just check the drivers. And even if I would have plucking them in and putting them onto the right place, it would be just too much of work. So what I'm really testing myself on real hardware is changes to the core, to the iScrashy core. But for driver-specific, processor-specifics, I have to trust the patch authors that they did the right thing, or the community who tests the patches sent out. So I'm really, really appreciating text tested by even if you're not a developer, not even understanding the patch. If you're using it and it does solve a problem for you, and you just, OK, I took the patch and it works for me, although I don't know what it really does. That's for me, it's helping. And I know that it gives more trust to the patch and makes me get a better feeling to apply it. Code checkers. Of course, big bit of warning at the top is meant very seriously, they are just tools. They do things wrong. They give false positives, or they give output which is not applicable, which does not really fit to your current patch or what. So my suggestion is use them and use them with care. On that being said, here are the ones I use. Mandatory should be check patch from the kernel tree. I also suggest to use a strict option so you get more information. I highly recommend sparse and smash. They are pretty easy to get and pretty easy to include into the kernel build. I recommend very much coxinell or coquichec, how it's called in the Linux kernel. There's a special script for it. It's pretty slow because it's kind of abused. Coxinell is a tool for different kind of work, but it can be used for that and finds also pretty subtle things. So I recommend that, spend that time. And there's some more generic code checkers. I added cppcheck. It's not really targeted on kernel code. Gives lots of output and false positives. But it finds a few things. So I have it included. And really outdated are floor finder and rats. But they are pretty good at finding problems with the string handling, which is always fun in the C language, as you know. So I let them run. I know how to pass their output. Once in a while, if they find something, it's something worth fixing. I need to put all this together somehow. So I have a bunch of scripts. Surprisingly, it's all scripts. When deciding on a namespace, I use the ninja approach because on the Commodore 64 demo scene, my nickname is ninja. So I thought I will do all the ninja style and all my scripts start with ninja. And so we have ninja get arc. For easing the kernel build, it will automatically detect for what architecture I want to build the kernel and set up the cross compiler and what else is needed. I have a script called ninja check, which runs all the code checkers I just mentioned in a row. And I can easily just build the kernel with make and call ninja check. And it will run all the code. Code analyzers. And my key script is ninja apply mailbox, which builds the unpatched kernel, applies all the patches I got on top of that. And for every patch I got in that mailbox, we'll build the kernel and run all the code checkers. So if I get a huge set of patches from one person, I can ensure that the kernel always builds the code checkers. The kernel always builds, and I will get a separate report for every change this person makes. And based on that output, I will include that in my review. The problem, there's a slight problem when building the kernels. For a kernel, you need a configuration, the.config file. And having 70 different drivers, I had to ask myself, so how would I deal with that? Do I have to have 70 different config files and keep them up to date whenever the kernel updates? And I decided to do that not. The next option was to use the def default configs, which are already in the kernel. But they also have problems. And I didn't want to become the def config maintainer, so I dropped that idea too. My idea is to always start with an all-no config to have a minimal configuration and then just activate the few things by hand. Meanwhile, I know what to activate, so I'm pretty fast at that. And the good thing about that is it seems to be quite unusual configuration, which is not tested very well. So I find lots of built errors which don't have necessary to do with i-square-c. So I'm working on a script, ninja-complain-build-error, which whenever I get the build error, I'm not responsible for, but just write a mail to the appropriate people and ask them to fix that. I don't know if that's good for my karma or not, but we'll do it nonetheless. So how does it look like in action? Well, it makes this a bit bigger. I'm basically now applying the patch I've shown before, which just adds this PCI ID to the driver. It was a simple patch where it said, OK, it looks good to me, so let's see if it builds. And the only thing I do is lose my ninja-apply mailbox. This is a patch I just downloaded with the command line utility via patchwork. And now it won't take terribly long, but it's a good time to drink some water or relax or exchange a kiss or whatever. OK. We're just seeing here now it's using the code checker S patch, and now you know why I said it's a bit slow. This is really taking most of the time during this process. But I still think it's worth it. And it just makes a few comments. Here, check patch has been run, and you can see that it has some warnings. And this is a good example of what I said before. Tools are just tools. They can go wrong. It's interpreting this as code, although it's just a comment, so you can really ignore. In this case, you can ignore the warnings and keep going. Oh, where's my console? If you look here, sparse was happy. Smash was happy. That's also a good indication. Cpp check, as I said, is pretty producing a lot of things. Those not in it, exit in it, not used, might be worth looking something, might be wrong something in the driver. But in general, I have some experience of these code checkers. I would say this patch is OK, although the code checkers did produce a lot of output. That's what I said. They're just tools. You have to learn to work with them. There might be things worth looking at, but in general, I get the impression the patch is OK, and I will probably apply it. And then the next Linux kernel has support for iSql-C on that chip, too. And that's all which is done for the built and code review checking. So one script, and I'm done. The next thing is that somewhere I have to publish what I collected. I have my own branches, which are soon, hopefully tomorrow, move to kernel.org. Because my job situation changed. I'm a freelancer. I became a freelancer. So I'm moving my branches to kernel.org where they belong. I have currently two branches for current, for the current release, and for next, for the next kernel release. I'm not perfectly happy with that, but I've not found yet a better solution. The key thing is that other people rely. If you're a maintainer, people rely on your for next branch to develop. You should never, never rebase it. I was a bit naive in the beginning and thought, once in a while, that won't hurt. But it will hurt. So never read. Once you push stuff out, this is set in stone, and you have to work with incremental patches on that. Then they go into Linux Next for broader testing. As I said before, I can't test all the things myself. So if they go out to the public into Linux Next, they're opposed, well, offered to broader testing. And if I miss to check something, there are enough build compiled checkers out there in the world who will notify me if I did something wrong. If all went well, and I want Linux to pull that, I have also a script called ninja pull request, which will set up a pull request properly formatted with the right people on the CC list. So I just have to configure this once and not every time. Time is a good topic. Sadly, I'm not a full-time maintainer. I have to do this somehow besides what I do regularly do. I would love to have a rolling process. So whenever a patch comes in, I'd love to, as soon as possible, respond to that. So the person working on the patch can continue working based on my review. That I totally understand that this would be the most effective way and most desired way. Sadly, at least in the last one or two kernel revisions, my workload simply did not allow that. Because that would have meant context switches from totally different topics. I was working on bootloaders, NAND support in bootloaders. And then I would have to switch to I-square-C again, which is a different way of thinking. So what I did in the last time was block-wise. I just reserved one or two weekends and just did only I-square-C, really focused, concentrated. That was, for me, the only way to handle the amount of patches, which fluctuates. Some last kernel was pretty calm. That was OK. The kernels, therefore, have been pretty, pretty busy because all the device-tree conversions for ARM caused a huge stress, both in amount of patches and both in, there were lots of unclear situations which had never been discussed before. And now I was there and have to make a decision on an unclear situation that was pretty stressful. But OK, I hope that I will have more influence on my workload and on my schedule. So I hope that in the future, the situation will change a little bit. Nonetheless, I think we have a bit of a scaling problem when it comes to maintaining. Because it's really nice to see that lots of chip vendors and more and more people are joining the upstream development. So we have lots of new developers sending patches, but we don't have so many people taking the responsibility wanting to be maintainers, wanting to review patches. And most maintainers are totally overworked and overloaded with work. One possible solution I see for that is that more maintainers should be doing this work full-time and not part-time or in their free time. They should still be independent. I think it would help if they were not hired by a company who has interest in that particular subsystem. But my hope would be that maybe more full-time paid maintainers can do concentrated work on that. I think that would be one solution to handle the massive amount of patches coming in recently. When it comes to collaboration, I don't think it's that hard if you follow a few guidelines I wrote here or have a few rules stick to them. Most of the time, I have to admit, it's pretty fun working with people from all over the world, from all over the doesn't matter which company they work for. They have just interest to get this problem solved to make the kernel a bit better. I suggest what usually cause problems if people have too much of an ego. My code must go in. And I don't want to change it. My solution is the best one. Or I spend so much time on it, this simply has to go in. As I said before, I appreciate all the effort. But this alone can't be a reason. The rationale, as I wrote down there, what is best for the kernel? And sometimes after some discussion, you find out, OK, I see your problem. I know that your solution does work somehow, but for a proper solution, we need to restart. Or you have to restart. And if you drop your ego and start thinking what is best for the kernel, it's easier for you to agree on that than saying, OK, no, I want to do that. So this helps a lot. And in discussions of stay technical, no personal flame, I don't see any benefit in attacking people personally. So better stay on a technical level. With Be Clear, I might really read your messages again and try to think of ways if they could be misinterpreted and if so, fix that. I've seen great flame wars which were totally, well, in my book, they are all useless. But they were based on a simple misunderstanding. And people were fighting over each other, although they had the same opinion. So it was not really needed. So Be Clear, make sure that what you write is really understandable, not misinterpretable. And keep seeing the fun it is. We have a very, very special project. We have a very special setup with all the people working collaboratively together. Even if it's sometimes difficult, it's a great thing to have. And if you just visualize that again, what you're doing, or what all these people are doing, and still feel the magic in it, then that's a good base to work on. Then there's this, is it? Well, Linus Torvald said something like, yay, making fun of people is half of the fun of open source development or something like that. I think Greg Rohartman said something like, I need to be harsh, so I get the best quote from the people. Something like this, I hope I quoted somehow correctly. Maintainers sometimes are said to be harsh and grumpy. And some people really like to play that role. I like to split that up a little bit. I see two parts in that. The first one is strictness, which is definitely needed. Follow the rules. Try to get the best solution to the main line. And do not just accept everything. This makes a kernel better. I agree with that. This makes you as a coder better if you really find the proper solution and think about the best solution. And yeah, obviously it makes me as a maintainer better because I understand the needs of the users better and can shape the subsystem in that way. About grumpiness, I don't know. I see people who really like to start a flame war and they have fun doing a flame war. OK, I can't follow that. But if they do so, then they may do so as long as they leave me out because I think it's a waste of energy. I prefer to write patches instead of users' mails. It can happen that I'm in a more grumpy mood, but this is more related to too much work. If I've overloaded with work and then get another patch and people are starting a comment with me or whatever, it can happen. So I have sympathy if that happens. So I'm not personally offended if some other maintainer gets grumpy to me. But my preference would be that this is caused by too much work or something like this. And it should not be inherent to be a maintainer that grumpiness is needed. Well, I would rather see it. It can happen, but I would be happy if it shouldn't. It wouldn't. And I try to follow that. I don't want to be grumpy. OK, these were the suggestions, demonstrations, annotations I had for maintaining a kernel subsystem. I wanted to reserve quite some time for questions, if you have questions, and answer them. So I'm open for them. Do we have a question?
 Well, from I'm the current kernel I2C maintainer,  and I wanted to talk about how I maintain the subsystem  in the Linux kernel, because, yeah,  I wanted to give some transparency to that topic.  And I myself thought, before I was a maintainer,  I was interested in how that could work  and how it could look like.  And so here I am.  First thing, just a few words about what I2C itself is.  It is a bus used since quite some time, 82.  What I found out, it was originally used inside televisions  to communicate between the different chips in there.  It's pretty well established.  It's very simple.  You have just two wires, one clock, one data line.  It's not designed for high-speed transfers,  but since it's so well established and so simple,  it is still widely used.  And just to give you an idea, here you have a clock line.  So that's a master.  It's always a master-slave relationship.  You have a master who says how fast to talk  and who's allowed to talk.  And then you have a data line.  And whenever the clock line is on the high level,  you look on the data line, OK, here it's low.  So we have a zero.  Here it's high.  We have a one.  And again and again.  So you get a stream of zeros and ones.  And this is enough to communicate.  You can put a protocol on that.  So this first here is the address, which device  to I want to talk to.  Then I say I want to write.  The device acknowledges that.  And then I can send data.  Pretty simple.  It's a bus used on today's mobile phones as well.  I guess it's still used in televisions.  It's cheap and it's robust and it does its thing.  And we have support for that in the Linux kernel, obviously.  So what does the i-square-c subsystem control  in the Linux kernel?  It is there is a core involved, which  is handling all the protocol, how messages have  to be formatted, in what row they have to be transmitted.  All that kind of core mechanism I have to take care about.  There are buses which are similar to i-square-c,  namely SM bus.  We have a compatibility layer in the kernel.  I also look for that.  And as I said before, we have a master-slave relationship.  There's a master requesting data and a slave,  a sensor, a temperature sensor, or a data storage device.  I take also care about the master drivers.  I'm not taking care.  And we have currently approximately 70 different master  drivers in the Linux kernel.  Because mainly driven, we just heard  talk about ARM chips.  And every ARM chip usually has its own i-square-c controller.  They are all handled in the i-square-c subsystem.  Not handled are the slave devices, the sensors.  I mentioned just a few.  Those are located in drivers HW and the hardware monitoring  section of the Linux kernel.  If you want to talk to audio codecs, you will find them here.  EEPROMs, you will find them here, and so on, and so on.  Currently, according to my research,  more than 400 i-square-c drivers.  The thing to note is, of course, there  are other audio codecs or other sensors.  They can be wired by SPI, USB, whatever.  So it makes sense to group them into that functionality.  And I don't have to take care about devices I don't know  otherwise.  So all sensors are here.  All the codecs are here.  And I just take care about the master drivers.  That's a rough overview of a typical ARM system on ship,  like it is used in mobile phones, tablets, whatever.  And it has lots of components.  We don't need to check them all.  What I want to point out is i-square-c or two-wire  interface, how it's also sometimes named,  is one little part of it.  And like in the Linux kernel itself,  it's one little part of it.  But we surely wanted to work to have proper to be in place  and maintain support.  It's not rocket science.  It's not bleeding edge.  But it's one important piece which should be there,  if we want it.  So this talk is not so much about i-square-c.  It's more about maintaining a subsystem.  And the question how to become a maintainer is, for me,  really answered by meritocracy.  It's about merit.  If you start participating, if you start discussing  in a certain subsystem, you can read that for yourself.  Then you get entrusted.  Entrusted to your opinion gets entrusted.  And then you are offered, if you want that,  some responsibility.  And this is really, from what I think,  a key distinguishment between potential maintainer  and a developer.  It's taking over responsibility not only  for the current problem you are working on,  and as soon as that problem is gone,  you're moving on to the next.  But some general responsibility for a subsystem  or that other parts in that subsystem  could be improved or cleaned up, stuff like this.  My suggestion, what I wrote with Care About the Kernel,  is I think it's in the long term more sustainable.  If you do that because you have a feeling  that this should be properly handled in the kernel,  so you want to do the maintenance because you  want that to be proper in the kernel.  If you do that, let's say for egoistic reasons,  because you want to be a maintainer, just to be a maintainer,  that's a motivation which can go away if things go wrong.  But if you still are interested in, hey,  I want the kernel to be the best operating system in the world,  and I need to go through that, that's  what I figured out the more sustainable way of thinking  in that regard.  So the case for me is pretty boring, pretty standard.  When I started hacking the Linux kernel,  by accident started in the I-square-C subsystem.  That was the first talk I was given by my former employer.  And I was lucky to meet Jean de Veyre in that subsystem, who  was some kind of mentor to me, a good person guiding me  how to get into the kernel development.  And I got not only interested, as I mentioned before,  in the driver I was working on, I  was interested in the subsystem itself.  I was seeing, OK, according to what you just taught me,  this part doesn't look proper.  I said, yeah, that's all craft.  I don't have time to clean that up.  So I did.  And over the time, over years, I was sending cleanup patches,  start to look at other people patches.  Although I didn't have personal interest in that,  just to guide other people.  This was the next step.  And keeping at this, I started becoming a co-maintenor.  And since last year, I was interested  to be the maintainer of the iSquare-C subsystem, which  is an interesting challenge, really.  I can recommend it after all.  So there was I, a maintainer.  And so I was responsible to accept patches from the mailing  list, collect patches from the mailing list,  review them, and pass them onto Linus.  So how do I do that?  How do you do that?  I was looking for some kind of how-to,  but I couldn't really find one.  This is also one reason why I'm standing here.  There were some talks and some articles  by Greg Rohartman, which were helpful in pointing  potential issues I have to deal with.  But when I looked at them, how things were handled in the  scripts, it came to the conclusion, well, maybe I  would do that different.  And I think this is the answer why there's no proper how-to.  This is a lot about workflow, how you do things, and what  form, and like with most workflows,  this is so subjective that I also would not know how to  write a how-to do this, but I just can present how I do  this.  I will answer questions, but I would recommend every  maintainer to just think about the things involved and find  their own way, how they handle this amount of patches, and  all the things involved.  So what is involved?  First, you need to track the patches.  Usually, every subsystem in the Linux kernel has a  dedicated mailing list to discuss the issues there or to  discuss patches.  And you need to track what's happening there and work on  them.  I used to do this with standard mail software.  I used mud for handling patches, and I would really  recommend mud for handling patches because other mailers  tend to break the formatting and make applying harder.  I used iStuff for priority settings and stuff like this  that maybe could have done with mud as well, but that was the  most easy solution for me, so I took it and it worked OK for  a while.  But I felt that it was not perfect, and I am now starting  to see that I am using something called patchwork.  I've seen that before, but I now have a different view of  it now that I'm a maintainer.  That's a net-based patch management tool.  It has a command line utility, so I can use it for  scripting.  It has a web interface, so I can change settings there.  Or users can look what the status of a patch is, and I  have the impression that it will work a lot better.  I want to give a short view of how it looks.  Is that readable in the last rows?  Better?  No reaction.  OK, I will just keep to this.  Here I have a list of the patches which came in recently.  So that released me from forgetting a patch because I  didn't care about my mail folder enough, deleting  accidentally.  This software does not delete accidentally, so no patch  gets lost.  That is very convenient.  And then you see on the right side, they have a status,  which are all new.  I have not processed them.  And I just can click onto one patch.  I assume, oh no, I'm not logged in.  I should do that.  I hate typing passwords when I'm recorded, but I can't  help it right now.  So what you saw before is that what users could do is  that users can see the status of their own patch.  Am I working on it?  Have I rejected it?  What not?  And if I now go onto a patch like this one, I am able to  change the status here to whatever I think is appropriate.  And the nice thing is that I just have  configured the software in a way that whenever I change it,  the original patch author gets a mail.  So there's also response.  And for me, left is the burdens to write a mail.  It's all automated.  That's really helpful.  And I don't want to show that.  Trust me that before, this is a patch which got accidentally  on the mailing list.  It has nothing to do with I2C.  So I don't have this does not fall under my control.  So I would just set the patch to not applicable update.  And the author will get a mail.  And for me, the patch is marked as done more or less,  because I know I don't have to care about it.  Another nice feature is that I have a very good  patch, another nice, see it has gone only three pages we had  four before.  Another nice thing is that it just not collects only the  patch, but also, that's the wrong one.  Sorry.  This is a very simple patch.  And the nice thing is it does not only collect the patch,  but all the commands which were made by this patch, which is  in this case just reviewed by from Jean de Verre, which is a  former maintainer.  I trust him very much.  So I assume that the patch is OK.  I have looked myself at it.  And since it is so simple, I also agree it is OK.  It just adds a bit of documentation and basically  adds just a new PCI device ID to an existing driver.  So the new item, what not, will support  iSquashD as well.  So I have the patch.  I have all the correspondent males all in one place.  What I now want to show is that there's also this command  line interface.  If I use the patchwork client and let it list, I hope not  too much of you are using the Wi-Fi here, because it will  slow down my presentation.  I would get a list of all the patches, like I have had that  in the, you've seen that on the web.  I was, OK, here it is.  And you can see here, there's the patch I've just seen in  the web, and with the simple commands client gets.  And this ID number, I could get the patch from a script.  And with all the hooks, you can imagine, you can do  wonderful things automatically, getting, accepting,  setting status.  This is very convenient.  So if you are maybe not a Linux kernel maintainer, but  some other person handling lots of patches, you need to  track, this is a good idea.  This is a good software.  Kernel Org offers patchwork, and ozileps.org offers that,  but you can also install that on your own.  This is something I can really recommend.  The next thing is about, I've collected all the patches.  Now I need to comment on them.  I need to review them to give you advices.  And the first thing is especially, well, you have  to learn that as a maintainer as well, but also for the  users, the key idea is, well, let me exaggerate.  Kernel is the best thing we have, and there must be a  very, very good reason to change it.  So the default is, I'm not accepting a patch, unless I  understand the reason why you send it.  It must be clear to me.  I appreciate all the effort that went into a patch, and I  appreciate the idea of improving the Linux kernel,  but just making a patch alone does not justify the  inclusion.  And if I can't see the reason to include the patch, then you  or me or we both, we have to work on that until it's clear  if it's acceptable or not.  This is one key thing to remember.  Of course, I start with a visual review.  I just read the mail and reply to it and check if the  concepts in that patch are proper, if the formal  requirements are OK.  Because I don't need to do a low level, picky review if the  basics are not correct.  That would be waste of time.  So I check for the big picture.  And if I find something there which is not OK, then I will  say, OK, I will stop reviewing here.  Please fix that first, or start a discussion on what the  proper mechanism is.  The thing which causes most arguments is what I called  impression and put it in quotes.  This is something developed over the years, over the time  where sometimes I just have the feeling it's not cutting it.  And then I may be wrong.  This usually needs to discussion what the proper way to do  it is.  I may be convinced if there are arguments.  But I dare to say most times there's a reason why I have  these feelings that it's not proper.  And it's not always ending up in an argument.  Usually you work together and find out a really clean,  elegant way of finding a problem and finding a better  solution to that.  The next thing is obviously compile tests.  If the first part went OK, then does it even apply to the  current Linux kernel?  Does it build?  I can only test if I have the compiler at hand for very,  very strange platforms.  I just skip it and trust on Linux next.  But usually I have the compiler.  And I run a few code checkers on the code  resulted from the patch.  I will say some more words about that later.  Runtime tests is a critical, no, not critical.  But I would like to do a lot more runtime tests.  But obviously, as I said, I have 70 different drivers.  And I don't have 70 different hardware at home to just  check the drivers.  And even if I would have plucking them in and putting  them onto the right place, it would be just too much of work.  So what I'm really testing myself on real hardware is  changes to the core, to the iScrashy core.  But for driver-specific, processor-specifics, I  have to trust the patch authors that they did the right  thing, or the community who tests the patches sent out.  So I'm really, really appreciating text tested  by even if you're not a developer, not even  understanding the patch.  If you're using it and it does solve a problem for you,  and you just, OK, I took the patch and it works for me,  although I don't know what it really does.  That's for me, it's helping.  And I know that it gives more trust to the patch and makes  me get a better feeling to apply it.  Code checkers.  Of course, big bit of warning at the top is meant very  seriously, they are just tools.  They do things wrong.  They give false positives, or they give output which is not  applicable, which does not really fit to your current  patch or what.  So my suggestion is use them and use them with care.  On that being said, here are the ones I use.  Mandatory should be check patch from the kernel tree.  I also suggest to use a strict option so you get more  information.  I highly recommend sparse and smash.  They are pretty easy to get and pretty easy to include into  the kernel build.  I recommend very much coxinell or coquichec, how it's  called in the Linux kernel.  There's a special script for it.  It's pretty slow because it's kind of abused.  Coxinell is a tool for different kind of work, but it  can be used for that and finds also pretty subtle things.  So I recommend that, spend that time.  And there's some more generic code checkers.  I added cppcheck.  It's not really targeted on kernel code.  Gives lots of output and false positives.  But it finds a few things.  So I have it included.  And really outdated are floor finder and rats.  But they are pretty good at finding problems with the  string handling, which is always fun in the C language,  as you know.  So I let them run.  I know how to pass their output.  Once in a while, if they find something, it's  something worth fixing.  I need to put all this together somehow.  So I have a bunch of scripts.  Surprisingly, it's all scripts.  When deciding on a namespace, I use the ninja approach  because on the Commodore 64 demo scene, my nickname is  ninja.  So I thought I will do all the ninja style and all my  scripts start with ninja.  And so we have ninja get arc.  For easing the kernel build, it will automatically detect  for what architecture I want to build the kernel and set up  the cross compiler and what else is needed.  I have a script called ninja check, which runs all the  code checkers I just mentioned in a row.  And I can easily just build the kernel with make and call  ninja check.  And it will run all the code.  Code analyzers.  And my key script is ninja apply mailbox, which builds the  unpatched kernel, applies all the patches I got on top of  that.  And for every patch I got in that mailbox, we'll build the  kernel and run all the code checkers.  So if I get a huge set of patches from one person, I can  ensure that the kernel always builds the code checkers.  The kernel always builds, and I will get a separate report  for every change this person makes.  And based on that output, I will include that in my review.  The problem, there's a slight problem when building the  kernels.  For a kernel, you need a configuration, the.config file.  And having 70 different drivers, I had to ask myself, so  how would I deal with that?  Do I have to have 70 different config files and keep them up  to date whenever the kernel updates?  And I decided to do that not.  The next option was to use the def default configs, which are  already in the kernel.  But they also have problems.  And I didn't want to become the def config maintainer, so I  dropped that idea too.  My idea is to always start with an all-no config to have a  minimal configuration and then just activate the few things  by hand.  Meanwhile, I know what to activate, so I'm pretty fast  at that.  And the good thing about that is it seems to be quite  unusual configuration, which is not tested very well.  So I find lots of built errors which don't have  necessary to do with i-square-c.  So I'm working on a script, ninja-complain-build-error,  which whenever I get the build error, I'm not  responsible for, but just write a mail to the appropriate  people and ask them to fix that.  I don't know if that's good for my karma or not, but we'll do  it nonetheless.  So how does it look like in action?  Well, it makes this a bit bigger.  I'm basically now applying the patch I've shown before, which  just adds this PCI ID to the driver.  It was a simple patch where it said, OK, it looks good to me,  so let's see if it builds.  And the only thing I do is lose my ninja-apply mailbox.  This is a patch I just downloaded with the command  line utility via patchwork.  And now it won't take terribly long, but it's a good time  to drink some water or relax or exchange a kiss or whatever.  OK.  We're just seeing here now it's using the code checker S  patch, and now you know why I said it's a bit slow.  This is really taking most of the time during this process.  But I still think it's worth it.  And it just makes a few comments.  Here, check patch has been run, and you can see that it has  some warnings.  And this is a good example of what I said before.  Tools are just tools.  They can go wrong.  It's interpreting this as code, although it's just a  comment, so you can really ignore.  In this case, you can ignore the warnings and keep going.  Oh, where's my console?  If you look here, sparse was happy.  Smash was happy.  That's also a good indication.  Cpp check, as I said, is pretty producing a lot of things.  Those not in it, exit in it, not used, might be worth  looking something, might be wrong something in the driver.  But in general, I have some experience of these code  checkers.  I would say this patch is OK, although the code checkers did  produce a lot of output.  That's what I said.  They're just tools.  You have to learn to work with them.  There might be things worth looking at, but in general, I  get the impression the patch is OK, and I will probably  apply it.  And then the next Linux kernel has support for  iSql-C on that chip, too.  And that's all which is done for the built and code review  checking.  So one script, and I'm done.  The next thing is that somewhere I have to publish what I  collected.  I have my own branches, which are soon, hopefully tomorrow,  move to kernel.org.  Because my job situation changed.  I'm a freelancer.  I became a freelancer.  So I'm moving my branches to kernel.org where they belong.  I have currently two branches for current, for the current  release, and for next, for the next kernel release.  I'm not perfectly happy with that, but I've not found yet  a better solution.  The key thing is that other people rely.  If you're a maintainer, people rely on your for next branch  to develop.  You should never, never rebase it.  I was a bit naive in the beginning and thought, once  in a while, that won't hurt.  But it will hurt.  So never read.  Once you push stuff out, this is set in stone, and you have  to work with incremental patches on that.  Then they go into Linux Next for broader testing.  As I said before, I can't test all the things myself.  So if they go out to the public into Linux Next, they're  opposed, well, offered to broader testing.  And if I miss to check something, there are enough  build compiled checkers out there in the world who will  notify me if I did something wrong.  If all went well, and I want Linux to pull that, I have  also a script called ninja pull request, which will set up a  pull request properly formatted with the right  people on the CC list.  So I just have to configure this once and not every time.  Time is a good topic.  Sadly, I'm not a full-time maintainer.  I have to do this somehow besides what I do regularly do.  I would love to have a rolling process.  So whenever a patch comes in, I'd love to, as soon as  possible, respond to that.  So the person working on the patch can continue working  based on my review.  That I totally understand that this would be the most  effective way and most desired way.  Sadly, at least in the last one or two kernel revisions, my  workload simply did not allow that.  Because that would have meant context switches from  totally different topics.  I was working on bootloaders, NAND support in bootloaders.  And then I would have to switch to I-square-C again,  which is a different way of thinking.  So what I did in the last time was block-wise.  I just reserved one or two weekends and just did only  I-square-C, really focused, concentrated.  That was, for me, the only way to handle the amount of  patches, which fluctuates.  Some last kernel was pretty calm.  That was OK.  The kernels, therefore, have been pretty, pretty busy  because all the device-tree conversions for ARM caused a  huge stress, both in amount of patches and both in, there  were lots of unclear situations which had never been  discussed before.  And now I was there and have to make a decision on an  unclear situation that was pretty stressful.  But OK, I hope that I will have more influence on my  workload and on my schedule.  So I hope that in the future, the situation will change a  little bit.  Nonetheless, I think we have a bit of a scaling problem when  it comes to maintaining.  Because it's really nice to see that lots of chip vendors  and more and more people are joining the upstream  development.  So we have lots of new developers sending patches, but  we don't have so many people taking the responsibility  wanting to be maintainers, wanting to review patches.  And most maintainers are totally overworked and  overloaded with work.  One possible solution I see for that is that more  maintainers should be doing this work full-time and not  part-time or in their free time.  They should still be independent.  I think it would help if they were not hired by a company  who has interest in that particular subsystem.  But my hope would be that maybe more full-time paid  maintainers can do concentrated work on that.  I think that would be one solution to handle the  massive amount of patches coming in recently.  When it comes to collaboration, I don't think it's that  hard if you follow a few guidelines I wrote here or  have a few rules stick to them.  Most of the time, I have to admit, it's pretty fun working  with people from all over the world, from all over the  doesn't matter which company they work for.  They have just interest to get this problem solved to make  the kernel a bit better.  I suggest what usually cause problems if people have too  much of an ego.  My code must go in.  And I don't want to change it.  My solution is the best one.  Or I spend so much time on it, this simply has to go in.  As I said before, I appreciate all the effort.  But this alone can't be a reason.  The rationale, as I wrote down there, what is best for the  kernel?  And sometimes after some discussion, you find out,  OK, I see your problem.  I know that your solution does work somehow, but for a  proper solution, we need to restart.  Or you have to restart.  And if you drop your ego and start thinking what is best  for the kernel, it's easier for you to agree on that than  saying, OK, no, I want to do that.  So this helps a lot.  And in discussions of stay technical, no personal flame,  I don't see any benefit in attacking people personally.  So better stay on a technical level.  With Be Clear, I might really read your messages again and  try to think of ways if they could be misinterpreted and  if so, fix that.  I've seen great flame wars which were totally, well, in  my book, they are all useless.  But they were based on a simple misunderstanding.  And people were fighting over each other, although they  had the same opinion.  So it was not really needed.  So Be Clear, make sure that what you write is really  understandable, not misinterpretable.  And keep seeing the fun it is.  We have a very, very special project.  We have a very special setup with all the people working  collaboratively together.  Even if it's sometimes difficult, it's a great thing  to have.  And if you just visualize that again, what you're doing, or  what all these people are doing, and still feel the magic  in it, then that's a good base to work on.  Then there's this, is it?  Well, Linus Torvald said something like, yay, making  fun of people is half of the fun of open source development  or something like that.  I think Greg Rohartman said something like, I need to be  harsh, so I get the best quote from the people.  Something like this, I hope I quoted  somehow correctly.  Maintainers sometimes are said to be harsh and grumpy.  And some people really like to play that role.  I like to split that up a little bit.  I see two parts in that.  The first one is strictness, which is definitely needed.  Follow the rules.  Try to get the best solution to the main line.  And do not just accept everything.  This makes a kernel better.  I agree with that.  This makes you as a coder better if you really find the  proper solution and think about the best solution.  And yeah, obviously it makes me as a maintainer better  because I understand the needs of the users better and can  shape the subsystem in that way.  About grumpiness, I don't know.  I see people who really like to start a flame war and they  have fun doing a flame war.  OK, I can't follow that.  But if they do so, then they may do so as long as they leave  me out because I think it's a waste of energy.  I prefer to write patches instead of users' mails.  It can happen that I'm in a more grumpy mood, but this is  more related to too much work.  If I've overloaded with work and then get another patch and  people are starting a comment with me or whatever, it can  happen.  So I have sympathy if that happens.  So I'm not personally offended if some other  maintainer gets grumpy to me.  But my preference would be that this is caused by too much  work or something like this.  And it should not be inherent to be a maintainer that  grumpiness is needed.  Well, I would rather see it.  It can happen, but I would be happy if it shouldn't.  It wouldn't.  And I try to follow that.  I don't want to be grumpy.  OK, these were the suggestions, demonstrations,  annotations I had for maintaining a kernel  subsystem.  I wanted to reserve quite some time for questions, if you  have questions, and answer them.  So I'm open for them.  Do we have a question?
Maintaining_a_kernel_subsystem_KSgFgO3388g_2_0_47.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_48_95.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_96_143.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_144_191.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_192_239.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_240_287.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_288_335.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_336_395.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_396_455.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_456_515.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_516_563.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_564_623.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_624_671.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_672_2831.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_2832_6275.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_6276_7463.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_7464_10763.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_10764_17615.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_17616_18887.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_18888_18995.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_18996_19751.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_19752_21335.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_21336_21587.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_21588_21695.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_21696_21779.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_21780_25199.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_25200_32075.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_32076_32435.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_32436_32843.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_32844_35411.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_35412_35567.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_35568_37787.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_37788_40211.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_40212_41207.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_41208_41795.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_41796_41891.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_41892_41951.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_41952_42299.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_42300_42947.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_42948_43103.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_43104_43187.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_43188_43259.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_43260_43475.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_43476_44375.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_44376_44495.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_44496_44663.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_44664_47111.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_47112_51935.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_51936_56099.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_56100_60815.png Maintaining_a_kernel_subsystem_KSgFgO3388g_2_60816_61366.png
Well, from I'm the current kernel I2C maintainer, and I wanted to talk about how I maintain the subsystem in the Linux kernel, because, yeah, I wanted to give some transparency to that topic. And I myself thought, before I was a maintainer, I was interested in how that could work and how it could look like. And so here I am. First thing, just a few words about what I2C itself is. It is a bus used since quite some time, 82. What I found out, it was originally used inside televisions to communicate between the different chips in there. It's pretty well established. It's very simple. You have just two wires, one clock, one data line. It's not designed for high-speed transfers, but since it's so well established and so simple, it is still widely used. And just to give you an idea, here you have a clock line. So that's a master. It's always a master-slave relationship. You have a master who says how fast to talk and who's allowed to talk. And then you have a data line. And whenever the clock line is on the high level, you look on the data line, OK, here it's low. So we have a zero. Here it's high. We have a one. And again and again. So you get a stream of zeros and ones. And this is enough to communicate. You can put a protocol on that. So this first here is the address, which device to I want to talk to. Then I say I want to write. The device acknowledges that. And then I can send data. Pretty simple. It's a bus used on today's mobile phones as well. I guess it's still used in televisions. It's cheap and it's robust and it does its thing. And we have support for that in the Linux kernel, obviously. So what does the i-square-c subsystem control in the Linux kernel? It is there is a core involved, which is handling all the protocol, how messages have to be formatted, in what row they have to be transmitted. All that kind of core mechanism I have to take care about. There are buses which are similar to i-square-c, namely SM bus. We have a compatibility layer in the kernel. I also look for that. And as I said before, we have a master-slave relationship. There's a master requesting data and a slave, a sensor, a temperature sensor, or a data storage device. I take also care about the master drivers. I'm not taking care. And we have currently approximately 70 different master drivers in the Linux kernel. Because mainly driven, we just heard talk about ARM chips. And every ARM chip usually has its own i-square-c controller. They are all handled in the i-square-c subsystem. Not handled are the slave devices, the sensors. I mentioned just a few. Those are located in drivers HW and the hardware monitoring section of the Linux kernel. If you want to talk to audio codecs, you will find them here. EEPROMs, you will find them here, and so on, and so on. Currently, according to my research, more than 400 i-square-c drivers. The thing to note is, of course, there are other audio codecs or other sensors. They can be wired by SPI, USB, whatever. So it makes sense to group them into that functionality. And I don't have to take care about devices I don't know otherwise. So all sensors are here. All the codecs are here. And I just take care about the master drivers. That's a rough overview of a typical ARM system on ship, like it is used in mobile phones, tablets, whatever. And it has lots of components. We don't need to check them all. What I want to point out is i-square-c or two-wire interface, how it's also sometimes named, is one little part of it. And like in the Linux kernel itself, it's one little part of it. But we surely wanted to work to have proper to be in place and maintain support. It's not rocket science. It's not bleeding edge. But it's one important piece which should be there, if we want it. So this talk is not so much about i-square-c. It's more about maintaining a subsystem. And the question how to become a maintainer is, for me, really answered by meritocracy. It's about merit. If you start participating, if you start discussing in a certain subsystem, you can read that for yourself. Then you get entrusted. Entrusted to your opinion gets entrusted. And then you are offered, if you want that, some responsibility. And this is really, from what I think, a key distinguishment between potential maintainer and a developer. It's taking over responsibility not only for the current problem you are working on, and as soon as that problem is gone, you're moving on to the next. But some general responsibility for a subsystem or that other parts in that subsystem could be improved or cleaned up, stuff like this. My suggestion, what I wrote with Care About the Kernel, is I think it's in the long term more sustainable. If you do that because you have a feeling that this should be properly handled in the kernel, so you want to do the maintenance because you want that to be proper in the kernel. If you do that, let's say for egoistic reasons, because you want to be a maintainer, just to be a maintainer, that's a motivation which can go away if things go wrong. But if you still are interested in, hey, I want the kernel to be the best operating system in the world, and I need to go through that, that's what I figured out the more sustainable way of thinking in that regard. So the case for me is pretty boring, pretty standard. When I started hacking the Linux kernel, by accident started in the I-square-C subsystem. That was the first talk I was given by my former employer. And I was lucky to meet Jean de Veyre in that subsystem, who was some kind of mentor to me, a good person guiding me how to get into the kernel development. And I got not only interested, as I mentioned before, in the driver I was working on, I was interested in the subsystem itself. I was seeing, OK, according to what you just taught me, this part doesn't look proper. I said, yeah, that's all craft. I don't have time to clean that up. So I did. And over the time, over years, I was sending cleanup patches, start to look at other people patches. Although I didn't have personal interest in that, just to guide other people. This was the next step. And keeping at this, I started becoming a co-maintenor. And since last year, I was interested to be the maintainer of the iSquare-C subsystem, which is an interesting challenge, really. I can recommend it after all. So there was I, a maintainer. And so I was responsible to accept patches from the mailing list, collect patches from the mailing list, review them, and pass them onto Linus. So how do I do that? How do you do that? I was looking for some kind of how-to, but I couldn't really find one. This is also one reason why I'm standing here. There were some talks and some articles by Greg Rohartman, which were helpful in pointing potential issues I have to deal with. But when I looked at them, how things were handled in the scripts, it came to the conclusion, well, maybe I would do that different. And I think this is the answer why there's no proper how-to. This is a lot about workflow, how you do things, and what form, and like with most workflows, this is so subjective that I also would not know how to write a how-to do this, but I just can present how I do this. I will answer questions, but I would recommend every maintainer to just think about the things involved and find their own way, how they handle this amount of patches, and all the things involved. So what is involved? First, you need to track the patches. Usually, every subsystem in the Linux kernel has a dedicated mailing list to discuss the issues there or to discuss patches. And you need to track what's happening there and work on them. I used to do this with standard mail software. I used mud for handling patches, and I would really recommend mud for handling patches because other mailers tend to break the formatting and make applying harder. I used iStuff for priority settings and stuff like this that maybe could have done with mud as well, but that was the most easy solution for me, so I took it and it worked OK for a while. But I felt that it was not perfect, and I am now starting to see that I am using something called patchwork. I've seen that before, but I now have a different view of it now that I'm a maintainer. That's a net-based patch management tool. It has a command line utility, so I can use it for scripting. It has a web interface, so I can change settings there. Or users can look what the status of a patch is, and I have the impression that it will work a lot better. I want to give a short view of how it looks. Is that readable in the last rows? Better? No reaction. OK, I will just keep to this. Here I have a list of the patches which came in recently. So that released me from forgetting a patch because I didn't care about my mail folder enough, deleting accidentally. This software does not delete accidentally, so no patch gets lost. That is very convenient. And then you see on the right side, they have a status, which are all new. I have not processed them. And I just can click onto one patch. I assume, oh no, I'm not logged in. I should do that. I hate typing passwords when I'm recorded, but I can't help it right now. So what you saw before is that what users could do is that users can see the status of their own patch. Am I working on it? Have I rejected it? What not? And if I now go onto a patch like this one, I am able to change the status here to whatever I think is appropriate. And the nice thing is that I just have configured the software in a way that whenever I change it, the original patch author gets a mail. So there's also response. And for me, left is the burdens to write a mail. It's all automated. That's really helpful. And I don't want to show that. Trust me that before, this is a patch which got accidentally on the mailing list. It has nothing to do with I2C. So I don't have this does not fall under my control. So I would just set the patch to not applicable update. And the author will get a mail. And for me, the patch is marked as done more or less, because I know I don't have to care about it. Another nice feature is that I have a very good patch, another nice, see it has gone only three pages we had four before. Another nice thing is that it just not collects only the patch, but also, that's the wrong one. Sorry. This is a very simple patch. And the nice thing is it does not only collect the patch, but all the commands which were made by this patch, which is in this case just reviewed by from Jean de Verre, which is a former maintainer. I trust him very much. So I assume that the patch is OK. I have looked myself at it. And since it is so simple, I also agree it is OK. It just adds a bit of documentation and basically adds just a new PCI device ID to an existing driver. So the new item, what not, will support iSquashD as well. So I have the patch. I have all the correspondent males all in one place. What I now want to show is that there's also this command line interface. If I use the patchwork client and let it list, I hope not too much of you are using the Wi-Fi here, because it will slow down my presentation. I would get a list of all the patches, like I have had that in the, you've seen that on the web. I was, OK, here it is. And you can see here, there's the patch I've just seen in the web, and with the simple commands client gets. And this ID number, I could get the patch from a script. And with all the hooks, you can imagine, you can do wonderful things automatically, getting, accepting, setting status. This is very convenient. So if you are maybe not a Linux kernel maintainer, but some other person handling lots of patches, you need to track, this is a good idea. This is a good software. Kernel Org offers patchwork, and ozileps.org offers that, but you can also install that on your own. This is something I can really recommend. The next thing is about, I've collected all the patches. Now I need to comment on them. I need to review them to give you advices. And the first thing is especially, well, you have to learn that as a maintainer as well, but also for the users, the key idea is, well, let me exaggerate. Kernel is the best thing we have, and there must be a very, very good reason to change it. So the default is, I'm not accepting a patch, unless I understand the reason why you send it. It must be clear to me. I appreciate all the effort that went into a patch, and I appreciate the idea of improving the Linux kernel, but just making a patch alone does not justify the inclusion. And if I can't see the reason to include the patch, then you or me or we both, we have to work on that until it's clear if it's acceptable or not. This is one key thing to remember. Of course, I start with a visual review. I just read the mail and reply to it and check if the concepts in that patch are proper, if the formal requirements are OK. Because I don't need to do a low level, picky review if the basics are not correct. That would be waste of time. So I check for the big picture. And if I find something there which is not OK, then I will say, OK, I will stop reviewing here. Please fix that first, or start a discussion on what the proper mechanism is. The thing which causes most arguments is what I called impression and put it in quotes. This is something developed over the years, over the time where sometimes I just have the feeling it's not cutting it. And then I may be wrong. This usually needs to discussion what the proper way to do it is. I may be convinced if there are arguments. But I dare to say most times there's a reason why I have these feelings that it's not proper. And it's not always ending up in an argument. Usually you work together and find out a really clean, elegant way of finding a problem and finding a better solution to that. The next thing is obviously compile tests. If the first part went OK, then does it even apply to the current Linux kernel? Does it build? I can only test if I have the compiler at hand for very, very strange platforms. I just skip it and trust on Linux next. But usually I have the compiler. And I run a few code checkers on the code resulted from the patch. I will say some more words about that later. Runtime tests is a critical, no, not critical. But I would like to do a lot more runtime tests. But obviously, as I said, I have 70 different drivers. And I don't have 70 different hardware at home to just check the drivers. And even if I would have plucking them in and putting them onto the right place, it would be just too much of work. So what I'm really testing myself on real hardware is changes to the core, to the iScrashy core. But for driver-specific, processor-specifics, I have to trust the patch authors that they did the right thing, or the community who tests the patches sent out. So I'm really, really appreciating text tested by even if you're not a developer, not even understanding the patch. If you're using it and it does solve a problem for you, and you just, OK, I took the patch and it works for me, although I don't know what it really does. That's for me, it's helping. And I know that it gives more trust to the patch and makes me get a better feeling to apply it. Code checkers. Of course, big bit of warning at the top is meant very seriously, they are just tools. They do things wrong. They give false positives, or they give output which is not applicable, which does not really fit to your current patch or what. So my suggestion is use them and use them with care. On that being said, here are the ones I use. Mandatory should be check patch from the kernel tree. I also suggest to use a strict option so you get more information. I highly recommend sparse and smash. They are pretty easy to get and pretty easy to include into the kernel build. I recommend very much coxinell or coquichec, how it's called in the Linux kernel. There's a special script for it. It's pretty slow because it's kind of abused. Coxinell is a tool for different kind of work, but it can be used for that and finds also pretty subtle things. So I recommend that, spend that time. And there's some more generic code checkers. I added cppcheck. It's not really targeted on kernel code. Gives lots of output and false positives. But it finds a few things. So I have it included. And really outdated are floor finder and rats. But they are pretty good at finding problems with the string handling, which is always fun in the C language, as you know. So I let them run. I know how to pass their output. Once in a while, if they find something, it's something worth fixing. I need to put all this together somehow. So I have a bunch of scripts. Surprisingly, it's all scripts. When deciding on a namespace, I use the ninja approach because on the Commodore 64 demo scene, my nickname is ninja. So I thought I will do all the ninja style and all my scripts start with ninja. And so we have ninja get arc. For easing the kernel build, it will automatically detect for what architecture I want to build the kernel and set up the cross compiler and what else is needed. I have a script called ninja check, which runs all the code checkers I just mentioned in a row. And I can easily just build the kernel with make and call ninja check. And it will run all the code. Code analyzers. And my key script is ninja apply mailbox, which builds the unpatched kernel, applies all the patches I got on top of that. And for every patch I got in that mailbox, we'll build the kernel and run all the code checkers. So if I get a huge set of patches from one person, I can ensure that the kernel always builds the code checkers. The kernel always builds, and I will get a separate report for every change this person makes. And based on that output, I will include that in my review. The problem, there's a slight problem when building the kernels. For a kernel, you need a configuration, the.config file. And having 70 different drivers, I had to ask myself, so how would I deal with that? Do I have to have 70 different config files and keep them up to date whenever the kernel updates? And I decided to do that not. The next option was to use the def default configs, which are already in the kernel. But they also have problems. And I didn't want to become the def config maintainer, so I dropped that idea too. My idea is to always start with an all-no config to have a minimal configuration and then just activate the few things by hand. Meanwhile, I know what to activate, so I'm pretty fast at that. And the good thing about that is it seems to be quite unusual configuration, which is not tested very well. So I find lots of built errors which don't have necessary to do with i-square-c. So I'm working on a script, ninja-complain-build-error, which whenever I get the build error, I'm not responsible for, but just write a mail to the appropriate people and ask them to fix that. I don't know if that's good for my karma or not, but we'll do it nonetheless. So how does it look like in action? Well, it makes this a bit bigger. I'm basically now applying the patch I've shown before, which just adds this PCI ID to the driver. It was a simple patch where it said, OK, it looks good to me, so let's see if it builds. And the only thing I do is lose my ninja-apply mailbox. This is a patch I just downloaded with the command line utility via patchwork. And now it won't take terribly long, but it's a good time to drink some water or relax or exchange a kiss or whatever. OK. We're just seeing here now it's using the code checker S patch, and now you know why I said it's a bit slow. This is really taking most of the time during this process. But I still think it's worth it. And it just makes a few comments. Here, check patch has been run, and you can see that it has some warnings. And this is a good example of what I said before. Tools are just tools. They can go wrong. It's interpreting this as code, although it's just a comment, so you can really ignore. In this case, you can ignore the warnings and keep going. Oh, where's my console? If you look here, sparse was happy. Smash was happy. That's also a good indication. Cpp check, as I said, is pretty producing a lot of things. Those not in it, exit in it, not used, might be worth looking something, might be wrong something in the driver. But in general, I have some experience of these code checkers. I would say this patch is OK, although the code checkers did produce a lot of output. That's what I said. They're just tools. You have to learn to work with them. There might be things worth looking at, but in general, I get the impression the patch is OK, and I will probably apply it. And then the next Linux kernel has support for iSql-C on that chip, too. And that's all which is done for the built and code review checking. So one script, and I'm done. The next thing is that somewhere I have to publish what I collected. I have my own branches, which are soon, hopefully tomorrow, move to kernel.org. Because my job situation changed. I'm a freelancer. I became a freelancer. So I'm moving my branches to kernel.org where they belong. I have currently two branches for current, for the current release, and for next, for the next kernel release. I'm not perfectly happy with that, but I've not found yet a better solution. The key thing is that other people rely. If you're a maintainer, people rely on your for next branch to develop. You should never, never rebase it. I was a bit naive in the beginning and thought, once in a while, that won't hurt. But it will hurt. So never read. Once you push stuff out, this is set in stone, and you have to work with incremental patches on that. Then they go into Linux Next for broader testing. As I said before, I can't test all the things myself. So if they go out to the public into Linux Next, they're opposed, well, offered to broader testing. And if I miss to check something, there are enough build compiled checkers out there in the world who will notify me if I did something wrong. If all went well, and I want Linux to pull that, I have also a script called ninja pull request, which will set up a pull request properly formatted with the right people on the CC list. So I just have to configure this once and not every time. Time is a good topic. Sadly, I'm not a full-time maintainer. I have to do this somehow besides what I do regularly do. I would love to have a rolling process. So whenever a patch comes in, I'd love to, as soon as possible, respond to that. So the person working on the patch can continue working based on my review. That I totally understand that this would be the most effective way and most desired way. Sadly, at least in the last one or two kernel revisions, my workload simply did not allow that. Because that would have meant context switches from totally different topics. I was working on bootloaders, NAND support in bootloaders. And then I would have to switch to I-square-C again, which is a different way of thinking. So what I did in the last time was block-wise. I just reserved one or two weekends and just did only I-square-C, really focused, concentrated. That was, for me, the only way to handle the amount of patches, which fluctuates. Some last kernel was pretty calm. That was OK. The kernels, therefore, have been pretty, pretty busy because all the device-tree conversions for ARM caused a huge stress, both in amount of patches and both in, there were lots of unclear situations which had never been discussed before. And now I was there and have to make a decision on an unclear situation that was pretty stressful. But OK, I hope that I will have more influence on my workload and on my schedule. So I hope that in the future, the situation will change a little bit. Nonetheless, I think we have a bit of a scaling problem when it comes to maintaining. Because it's really nice to see that lots of chip vendors and more and more people are joining the upstream development. So we have lots of new developers sending patches, but we don't have so many people taking the responsibility wanting to be maintainers, wanting to review patches. And most maintainers are totally overworked and overloaded with work. One possible solution I see for that is that more maintainers should be doing this work full-time and not part-time or in their free time. They should still be independent. I think it would help if they were not hired by a company who has interest in that particular subsystem. But my hope would be that maybe more full-time paid maintainers can do concentrated work on that. I think that would be one solution to handle the massive amount of patches coming in recently. When it comes to collaboration, I don't think it's that hard if you follow a few guidelines I wrote here or have a few rules stick to them. Most of the time, I have to admit, it's pretty fun working with people from all over the world, from all over the doesn't matter which company they work for. They have just interest to get this problem solved to make the kernel a bit better. I suggest what usually cause problems if people have too much of an ego. My code must go in. And I don't want to change it. My solution is the best one. Or I spend so much time on it, this simply has to go in. As I said before, I appreciate all the effort. But this alone can't be a reason. The rationale, as I wrote down there, what is best for the kernel? And sometimes after some discussion, you find out, OK, I see your problem. I know that your solution does work somehow, but for a proper solution, we need to restart. Or you have to restart. And if you drop your ego and start thinking what is best for the kernel, it's easier for you to agree on that than saying, OK, no, I want to do that. So this helps a lot. And in discussions of stay technical, no personal flame, I don't see any benefit in attacking people personally. So better stay on a technical level. With Be Clear, I might really read your messages again and try to think of ways if they could be misinterpreted and if so, fix that. I've seen great flame wars which were totally, well, in my book, they are all useless. But they were based on a simple misunderstanding. And people were fighting over each other, although they had the same opinion. So it was not really needed. So Be Clear, make sure that what you write is really understandable, not misinterpretable. And keep seeing the fun it is. We have a very, very special project. We have a very special setup with all the people working collaboratively together. Even if it's sometimes difficult, it's a great thing to have. And if you just visualize that again, what you're doing, or what all these people are doing, and still feel the magic in it, then that's a good base to work on. Then there's this, is it? Well, Linus Torvald said something like, yay, making fun of people is half of the fun of open source development or something like that. I think Greg Rohartman said something like, I need to be harsh, so I get the best quote from the people. Something like this, I hope I quoted somehow correctly. Maintainers sometimes are said to be harsh and grumpy. And some people really like to play that role. I like to split that up a little bit. I see two parts in that. The first one is strictness, which is definitely needed. Follow the rules. Try to get the best solution to the main line. And do not just accept everything. This makes a kernel better. I agree with that. This makes you as a coder better if you really find the proper solution and think about the best solution. And yeah, obviously it makes me as a maintainer better because I understand the needs of the users better and can shape the subsystem in that way. About grumpiness, I don't know. I see people who really like to start a flame war and they have fun doing a flame war. OK, I can't follow that. But if they do so, then they may do so as long as they leave me out because I think it's a waste of energy. I prefer to write patches instead of users' mails. It can happen that I'm in a more grumpy mood, but this is more related to too much work. If I've overloaded with work and then get another patch and people are starting a comment with me or whatever, it can happen. So I have sympathy if that happens. So I'm not personally offended if some other maintainer gets grumpy to me. But my preference would be that this is caused by too much work or something like this. And it should not be inherent to be a maintainer that grumpiness is needed. Well, I would rather see it. It can happen, but I would be happy if it shouldn't. It wouldn't. And I try to follow that. I don't want to be grumpy. OK, these were the suggestions, demonstrations, annotations I had for maintaining a kernel subsystem. I wanted to reserve quite some time for questions, if you have questions, and answer them. So I'm open for them. Do we have a question?
and I wanted to talk about how I maintain the on ship, like it is used in mobile phones, maintainer of the iSquare-C subsystem, which is how do I do that? How do you do that? I was workflow, how you do things, and what form, and involved and find their own way, how they handle track the patches. Usually, every subsystem in the Linux kernel has a dedicated mailing list to kernel maintainer, but some other person handling an argument. Usually you work together and find seriously, they are just tools. They do things So how does it look like in action? Well, it at least in the last one or two kernel revisions, most maintainers are totally overworked and the best one. Or I spend so much time on it, this simply has to go in. As I said before, I there, what is best for the kernel? And sometimes a kernel subsystem. I wanted to reserve quite